Incremental Update 12
Today, we’re happy to announce the release of feldera v0.32! This update brings a series of bug-fixes and improved caching for pipeline compilation - a feature that was requested very often by our users.
Compilation Cache
In the past, we would trigger recompilations for pipelines even in cases where the rust code didn't change (e.g., when connector configurations were updated). With this update, thanks to the hard work of Simon, we're happy to report that only updates to the code that end up changing the SQL code will lead to a recompilation.
We hope this makes it easier for you to iterate faster!
SQL Lancer integration
If you follow our changelog more closely, you might have also noticed a significant uptick in bug fixes in the SQL compiler in the last weeks. This is thanks to the work Abhinav has been doing on integrating our software with sqlancer, an amazing tool from academia for finding bugs in databases. Of course you also need someone that fixes the bugs, in our case that happens thanks to Mihai.
Here is a picture of our bugtracker, showing the bugs the two have found and fixed over the last two weeks:
Deltalake connector improvements
In the past, our delta lake connector would try to sort the entire input data by a timestamp in case the table you are ingesting into had a lateness constraints (to make sure input does not appear out of order and accidentally gets dropped). However, sorting the entire input is problematic in case you're ingesting large amounts of data. With this update, we're happy to report that we can now ingest more efficiently in such cases by partitioning the ingestion with multiple range queries. You can read more about in our updated deltalake documentation.
As usual, the sandbox has been updated with our latest release. Try it out and let us know your suggestions and feedback.
Other articles you may like
Incremental Update 6 at Feldera
We’re excited to announce the release of v0.26, which represents a significant step forward for Feldera. This release includes over 200 commits, adding 25,000 lines of new code and documentation. Let's dive into the highlights!
Database computations on Z-sets
How can Z-sets be used to implement database computations
Incremental Update 5 at Feldera
A quick overview of what's new in v0.25.