Incremental Update 21

It's time for another bi-weekly incremental update. A lot of things have changed for feldera in these two weeks! We'll share the highlights here:

Rust Compilation Time Improvements

You'll most likely see much better compilation times with the new release. Especially if you make small, incremental changes to your SQL. This is due to some incredible work going on behind the scenes from our team. We'll blog about it separately more, but in short we changed the way rust is generated. If you look under the hood e.g., by browsing the ~/.feldera/rust-compilation/crates directory you will now see a lot more crates:

This allows two things:

• The rust compiler is able to leverage many cores much better
• Thanks to "smart naming" of these crates (using merkle tree hashes) we are able to re-use already compiled code in new or modified pipelines

SQL Time Compilation Improvements

Improving the rust compilation by such a big amount meant that now (for some programs) our own SQL-to-Rust compiler became the bottleneck. In turn, we also improved its performance by relying on memoization for some of the compiler passes.

Better Ad-Hoc Query Support

We also applied a few bug-fixes to our adhoc query engine and added support for a new format: You can now request query results in the Arrow-IPC format, which is more efficient and has less restriction than e.g., JSON.