Incremental Update 7

Today, we’re excited to announce the release of feldera v0.27! This version is packed with new features, performance improvements, and bug fixes—topped off with a highly anticipated feature that our community has been asking for. Let’s dive in.

Log Endpoint

One of the most requested features from the operation teams of our users is finally here: pipeline log inspection. Thanks to the hard work of Simon and George, you can now inspect pipeline logs directly from the UI or via the CLI (fda log pipeline-name).

Here’s a peek at the new Logs Tab in action:

A fresh look for Feldera

If you’re reading this on feldera.com, you’ve already seen our new branding. We also updated the Webconsole to make it consistent with the new brand. It is live on try.feldera.com.

N-Way merging

Our batch-merger in DBSP just got a significant boost: Previously, merges only kicked off at the start of a circuit step. In v0.27, we’ve made merges more asynchronous, meaning merges can now start at any time during the life of a pipeline.

We also added support for n-way merging. This is a well known optimization in LSM trees, so unsurprisingly it helped in making things faster. For instance, on one of our machines, the time to process 100M events for Nexmark Q9 dropped from 133 seconds to 114 seconds.

Improved Variant Support

We fixed several issues in the SQL compiler and enhanced support for the VARIANT type which we introduced in the previous incremental update. You can now use VARIANT to represent BINARY types, and seamlessly cast between MAP, ARRAY and VARIANT types.

Here is an example illustrating a cast from a VARIANT to a MAP:

CREATE TABLE t (
    s string
) with (
  'materialized' = 'true',
  'connectors' = '[{
    "transport": {
      "name": "datagen",
      "config": {
        "plan": [{
            "limit": 1,
            "fields": {
              "s": { "values": ["{\"a\": 1}"]}
            }
        }]
      }
    }
  }]'
);

CREATE MATERIALIZED VIEW v AS
SELECT
    CAST(PARSE_JSON(s) AS MAP<STRING, INTEGER>)
FROM
    t;

Run this in our sandbox, and you’ll see the following output:

New Packages on PyPI and crates.io

Starting with v0.27, we’re expanding Feldera’s ecosystem. We’re now publishing two new packages to popular package indexes:

• The Feldera Python SDK is now available on PyPI.
• The Feldera SQL Library is now available on crates.io.

These, along with the dbsp library and fda CLI (also on crates.io), will be updated bi-weekly as part of our regular release cycle.

The release of the Feldera SQL library is a pre-cursor for a new feature we've been working on behind the scenes: User defined functions (UDFs) will land soon, ideally in the next release, so stay tuned for the next update 🙂