SmartModules are one of Fluvio’s unique features, allowing users to have direct control over their streaming data by providing a programmable API for inline data manipulation.
SmartModules are user-defined functions, written in Rust and compiled to WebAssembly - that means they’re lightweight and portable, and enable enormous flexibility for being integrated at many points in a streaming pipeline.
They enhance the entire streaming platform: reducing network costs by allowing precision filtering along the stream and encouraging code-reuse and collaboration.
The following diagram shows common components which may be configured with SmartModules performing inline computation.
The diagram shows five places where SmartModules may currently be used:
SmartModules are typically applied to data before the data is sent from one location to another.
This is so that any filtering that happens will result in reduced network traffic and more savings.
For Stream inputs (such Source Connectors and Producers)
For Stream outputs (such as Sink Connectors, Consumers, and the Fluvio CLI)
This helps to overcome “Data Gravity” by moving only the minimum amount of data necessary.
Fluvio supports the following types of SmartModules:
A Filter SmartModule takes an input record and allows you to check if the record value meets certain criteria.
If false the record is discarded, or true if the record is saved, and continues downstream.
A Map SmartModule takes an input record allows you to apply any data transformations to the record before it continues downstream.
“Map” refers to the programming language term, which simply is a function that is applied to all input data.
A FilterMap SmartModule takes one input record and returns zero or one output record.
As the name may imply, FilterMap is the combination of filter and map
You can check for conditions in the data and if met, apply transformations. Or if the conditions are not met, discard the record.
An ArrayMap SmartModule takes one input record and returns zero or many output records.
The Array in ArrayMap refers to a JSON array.
Given a single record that is a JSON array, you may flatten the single input array. The result is the creation of several individual records, which you may additionally apply transformations before returning.
An Aggregate SmartModule ability to create a feedback loop, by providing a Accumulator Record that is persistent per stream. Each event may modify the Accumulator, and the changes will be available for next input to use.
The value of the Accumulator Record is also returned after each input.
For example, if you’re trying to Sum up stream of numbers, you would add each input value to the current value of the Accumulator Record.
Currently the SmartModule development is limited to the Rust programming language, but you can be a Rust beginner and still take advantage of custom SmartModules!
In addition to the basic Rust development environment, we need the following Rust tools installed:
wasm32-unknown-unknownThis is needed to compile the SmartModule Rust code into a WebAssembly module
$ rustup target install wasm32-unknown-unknown
This will be used for creating new Rust SmartModule projects
$ cargo install cargo-generate
Using cargo generate, you can answer a few prompts and generate the code for a SmartModule
$ cargo generate gh:infinyon/fluvio-smartmodule-template
Example:
We are creating a filter type of SmartModule, named my-filter
$ cargo generate gh:infinyon/fluvio-smartmodule-template
🤷 Project Name : my-filter
🔧 Generating template ...
✔ 🤷 Which type of SmartModule would you like? · filter
✔ 🤷 Want to use SmartModule parameters? · true
[1/7] Done: .cargo/config.toml
[2/7] Done: .cargo
[3/7] Done: .gitignore
[4/7] Done: Cargo.toml
[5/7] Done: README.md
[6/7] Done: src/lib.rs
[7/7] Done: src
🔧 Moving generated files into: `/home/User/my-filter`...
💡 Initializing a fresh Git repository
✨ Done! New project created /home/User/my-filter
Navigate to your SmartModule directory, make your changes, then compile:
$ cargo build --release
Now that we have the SmartModule binary compiled let’s see it in action.
After building a SmartModule as a WASM binary, it may be registered with Fluvio using
the fluvio smart-module command, providing a name and a path to the binary.
Use SmartModule filters to build a WASM file.
$ fluvio smart-module create my-filter --wasm-file ./target/wasm32-unknown-unknown/release/my_filter.wasm
After creating one or more SmartModules, one may use the fluvio smart-module list command
to see the available SmartModules:
$ fluvio smart-module list
NAME STATUS SIZE
my-filter SmartModuleStatus 90442
You may use a Registered SmartModule anywhere that SmartModules may be used. To use them, you’ll need to provide the name of the SmartModule as well as its type.
For example, if we want to apply our registered SmartModule my-filter while consuming from our topic my-topic,
provide it’s name to the --filter argument.
$ fluvio consume my-topic -B --filter my-filter
During the development you may find it convenient for testing to skip the SmartModule registration step.
With the Fluvio CLI Consumer, you may pass a file path to --filter that points to the SmartModule WASM file,
like this:
$ fluvio consume my-topic -B --filter=target/wasm32-unknown-unknown/release/my_filter_in_development.wasm
For our official source and sink connectors you can apply SmartModules can be applied to any source or sink connector.
You just need to provide the type of module (filter, map, array-map, filter-map, aggregate) and it’s registered name as a parameter.
For example, this is how you would define a filter type SmartModule named my-filter to use with our http source connector, to apply the filter to each HTTP response before publishing to the topic:
# cat-facts-connector.yml
name: cat-facts-connector
type: http
topic: cat-facts
create_topic: true
direction: source
parameters:
endpoint: https://catfact.ninja/fact
interval: 30
filter: my-filter
