Incremental Update 16

We’ve just shipped feldera v0.36, focusing on performance optimizations for large pipelines and data-intensive workflows. Many of these changes respond directly to real-world demands from our customers, and we’re excited to share the highlights.

Big Programs

• Faster Rust Builds: When dealing with large SQL schemas (hundreds of fields), previous versions could balloon Rust compilation times. Now, while SQL compilation itself may take slightly longer, compiling the final pipeline is significantly faster, thanks to smarter dependency tracking. In other words, we trim away unused parts of your program.
• Code Size Reductions: We’ve also introduced optimizations that shrink the generated Rust binaries. Though Rust compile times remain an ongoing concern, expect even bigger improvements in upcoming releases.

On the UI side, massive SQL and Rust programs sometimes caused sluggishness because we were re-fetching them too frequently. We’ve resolved that in v0.36, so you should see a faster web-console with these huge programs.

Big Joins

We focused on LEFT JOIN optimizations to reduce the state needed by about 25% per join. As a result, any pipeline with multiple chained LEFT JOINs should see lower memory usage and higher throughput. If you rely heavily on complex joins in a single view, you’ll likely notice a tangible speedup.

Big Data

Finally, we also made two exciting improvements to the storage layer:

• Data Compression: You can now enable compression for your stored data, yielding substantial storage savings—some tests show up to 10× reduction.
• Multiway Merging: We added a multiway-merge strategy for data batches (building on existing multiway merges in other parts of feldera). This significantly boosts performance for pipelines that generate and maintain large amounts of state.

What’s Next?

We’re steadily moving feldera toward smooth operation at massive scales. A key part of that vision is S3 storage support, which is coming soon. Stay tuned for more announcements and let us know what you think of the new release in the meantime!