Incremental Update 3

Almost everything a Feldera user interacts with has changed in v0.23. The most consistent feedback we got from you was that our API and UI was too complicated. So we made some dramatic changes.

First of all, the REST API now only has one concept: a pipeline. You can create a pipeline, update it, start, pause, stop, and delete it. That's it. No more connectors, no more programs. Just pipelines. So you might wonder where did the SQL code go? It's still there of course, but it is now associated with the pipeline directly.

And what about connectors? They are still there too, but they are now embedded in the SQL. You specify the configuration for a connector in your SQL directly. This makes it much easier to understand what your pipeline is doing. Here is an example taken from our tutorial (all our docs and the Python SDK were also updated to reflect the new API):

create table VENDOR (
    id bigint not null primary key,
    name varchar,
    address varchar
) WITH ('connectors' = '[{
    "transport": {
        "name": "url_input", "config": {"path": "https://feldera-basics-tutorial.s3.amazonaws.com/vendor.json"}
    },
    "format": { "name": "json" }
}]');

This snippet specifies a vendor table and directly associates a connector with it. The connector reads data from a URL and expects JSON data.

We've also removed the old UI and replaced it with a new one that is much easier to use. With the connector and program concepts gone from the API, the new UI is now focused on editing SQL and managing pipeline life-cycles. It still lacks some features (like browsing materialized tables and views), but we're working on adding them back in a more flexible way by supporting a subset of SQL on them. Meanwhile, we added a way to view the change-stream of a pipeline directly in the UI. This has the advantage that you can see the changes flowing through Feldera in real-time even for views or tables that are not materialized.

Another addition is a new connector called datagen, it does what the name suggests: you can configure it to generate random data for your tables. This is useful for getting bootstrapped with a new pipeline, testing it and benchmarking the performance. Check out the new tutorial and documentation for it.