Incremental Update 12

Incremental Update 12

Gerd Zellweger
Gerd ZellwegerHead of Engineering / Co-Founder
| December 10, 2024

Today, we’re happy to announce the release of feldera v0.32! This update brings a series of bug-fixes and improved caching for pipeline compilation - a feature that was requested very often by our users.

Compilation Cache

In the past, we would trigger recompilations for pipelines even in cases where the rust code didn't change (e.g., when connector configurations were updated). With this update, thanks to the hard work of Simon, we're happy to report that only updates to the code that end up changing the SQL code will lead to a recompilation.

We hope this makes it easier for you to iterate faster!

SQL Lancer integration

If you follow our changelog more closely, you might have also noticed a significant uptick in bug fixes in the SQL compiler in the last weeks. This is thanks to the work Abhinav has been doing on integrating our software with sqlancer, an amazing tool from academia for finding bugs in databases. Of course you also need someone that fixes the bugs, in our case that happens thanks to Mihai.

Here is a picture of our bugtracker, showing the bugs the two have found and fixed over the last two weeks:

Abhinav and Mihai fixing bugs/

Deltalake connector improvements

In the past, our delta lake connector would try to sort the entire input data by a timestamp in case the table you are ingesting into had a lateness constraints (to make sure input does not appear out of order and accidentally gets dropped). However, sorting the entire input is problematic in case you're ingesting large amounts of data. With this update, we're happy to report that we can now ingest more efficiently in such cases by partitioning the ingestion with multiple range queries. You can read more about in our updated deltalake documentation.

As usual, the sandbox has been updated with our latest release. Try it out and let us know your suggestions and feedback.

Other articles you may like

Incremental Update 6 at Feldera

We’re excited to announce the release of v0.26, which represents a significant step forward for Feldera. This release includes over 200 commits, adding 25,000 lines of new code and documentation. Let's dive into the highlights!

Database computations on Z-sets

How can Z-sets be used to implement database computations

Incremental Update 5 at Feldera

A quick overview of what's new in v0.25.