Incremental Update 8

Today, we’re happy to announce the release of feldera v0.28! The highlight of this new version is the addition of UDFs (User defined functions) to Feldera. This is paired with our usual long list of bugfixes and other, smaller improvements. So let's dive right into it.

User defined functions

SQL is great at describing data and asking+answering questions about it. However, our advanced users often wish they had the flexibility to do arbitrary computations and transformation in feldera. UDFs are all about giving you that extra expressiveness.

But don't believe it until you see it, so let's write a pipeline with an UDF together:

create function morse_code(text varchar not null) returns varchar not null;

create table inputs (
    text varchar not null
) with ('materialized' = 'true');

create materialized view outputs as
select
    morse_code(text) as output
from
    inputs;

We declare the UDF morse_code and add a table inputs that takes a string, and a view outputs that returns the strings from inputs - with a twist that inputs get transformed by applying our morse_code function, which we'll write next.

The Webconsole now shows three new tabs stubs.rs, udf.rs and udf.toml in addition to program.sql. stubs.rs helps to determine the type signature for the rust function morse_code that we'll have to implement:

To implement our UDF (a function that transforms text to it's equivalent in Morse code), we'll leverage an existing rust library called morse. We can add it to the udf.toml tab such that during the build, the manager will fetch it from crates.io for us:

morse = "0.1"

By using the library, writing a function that converts text to morse code in udf.rs can be done in just a few lines of code:

use morse::encode::encode;

pub fn morse_code(text: String) -> Result<String, Box<dyn std::error::Error>> {
    Ok(encode(&text).unwrap_or_else(|_| String::from("unable to translate")))
}

That's it, now anything that gets inserted into inputs, will show up in outputs in morse code. Want to try it out yourself? Then you should give the new ad-hoc query support in the Webconsole a spin:

Ad-hoc Query Support in Webconsole and Python

We had ad-hoc queries for a while in fda, but now we're happy to report that they're also supported in our Python SDK and the Webconsole. Here is a screenshot of our earlier morse-code pipeline. We executed two ad-hoc SQL statements to insert "Hello feldera" and inspect the output where it got converted to morse code:

Outlook

We're happy to hear feedback on how UDFs and ad-hoc queries are working out for you. Meanwhile we're working behind the scenes on adding fault-tolerance for pipelines and ways to handle errors/panics during data processing gracefully. So stay tuned for some exciting new features in the upcoming releases!