Incremental Update 22

It's time for another bi-weekly incremental update. Here are the biggest user-facing changes we've done to our software in the last two weeks.

PostgreSQL Connector Updates

We already had support to ingest data from PostgreSQL to Feldera, but now we also added support to write incrementally computed results from Feldera to a PostgreSQL database.

Storage is now turned on by default

We've had our storage feature for quite a while now, but with v0.44 we are now turning it on by default (for any newly created pipeline). You can still run jobs in-memory, but you have to explicitly disable storage e.g., by running fda set-config $pipeline-name storage false.

feldera-rest-api crate

We've released a new Rust crate, feldera-rest-api, which contains helper code that makes it easier to interact with our API.

More configuration settings for our delta lake connector

Our delta-lake connector got two new settings:

• max_concurrent_readers: To limit the amount of concurrent tokio tasks reading data from the delta-lake during ingestion
• skip_unused_columns: Which (as the name suggests) avoids ingesting columns which are not used by any views.

You will find more details about these settings in our docs page.

Feldera SQL now supports lateral column aliasing

Due to popular demand we also added support for "lateral column aliasing" in our SQL compiler. Lateral column aliasing refers to the ability to define a column alias and then reuse that alias within the same SELECT clause. Here is a simple example to better understand how it works:

SELECT
    price,
    price * 0.25 AS tax,  -- alias 'tax'
    price + tax AS total, -- alias 'total' (price + tax)
FROM products;