We released v0.21 today. This release features several exciting changes, all based on user input.
We resolved a significant performance bottleneck in our storage engine. A key issue was that we use the metrics-rs library to gather metrics about a pipeline and provide them to Prometheus/Grafana. Unfortunately, we encountered high contention overheads when updating the same metric from multiple threads. The solution? We reduced the metrics recorded on the read-path1. This PR yielded a dramatic performance boost—one of our workloads dropped from 6 minutes to just 35 seconds. We expect all workloads that need to maintain state will benefit from this change.
Speaking of performance, our summer intern Matei has been tirelessly working over the last two weeks on building out our benchmarking infrastructure. You can view the fruits of his labor on benchmarks.feldera.io. Every PR that is merged into main gets benchmarked (and graphed) which allows us to spot performance regressions (and improvements) quickly.
Over in SQL compiler land, we've been busy with garbage collection and optimizations to reduce memory usage. For example, if you have queries with joins followed by filters, you will likely see lower memory usage thanks to this PR.
We are also working on a new and simplified web UI. The first, early prototype
for it is now included in this release. This isn't something you want to use yet
as it isn't complete -- for now we just focused on getting the build and CI
system to work with the new framework. But, if you're really curious the
prototype is already served under <manager-url>/new.
Try it out and let us know what you think! We're always looking for feedback.
- We will address the contention in the metrics library in a future release so we can hopefully bring all metrics back.↩