SmartModules are one of Fluvio’s premiere features, allowing users to have direct control over their streaming data by providing a programmable API for inline data manipulation. SmartModules are user-defined functions, written in Rust and compiled to WebAssembly - that means they’re lightweight and portable, and enable enormous flexibility for being integrated at many points in a streaming pipeline. They enhance the entire streaming platform: reducing network costs by allowing precision filtering along the stream and encouraging code-reuse and collaboration.
The following diagram shows common components which may be configured with SmartModules performing inline computation.
The diagram shows five places where SmartModules may currently be used:
SmartModules are typically applied to data before the data is sent from one location to another. This is so that any filtering that happens will result in less network traffic and more savings. For example, stream inputs (i.e. Source Connectors and Producers) will apply SmartModules before sending data to the Fluvio cluster, and stream outputs (such as Sink Connectors, Consumers, and the Fluvio CLI) will upload SmartModules to the Fluvio cluster and process data before the stream is sent over the network. This helps to overcome “Data Gravity” by moving only the most minimal amount of data necessary.
Fluvio features the following types of SmartModules:
A Filter SmartModule takes an input record and returns
false if the record should
be discarded, or
true if the record should be kept and sent downstream.
A Map SmartModule takes an input record and may edit and transform it, returning a new record that is sent downstream.
A FilterMap SmartModule takes one input record and returns zero or one output record. This effectively means that it may decide to discard or “filter” any input records, or to keep them and apply a transformation to them at the same time.
An ArrayMap SmartModule takes one input record and returns zero or many output records. This means that the output stream may have more records than the input stream.
An Aggregate SmartModule takes input records and “accumulates” them, essentially adding each input record to a rolling sum. Each input record gets “added” to the sum after the previous record, and the output stream is full of the “summed” records.
SmartModules may be used in two ways: by registering them by name with the Fluvio cluster, or by providing them “ad hoc” right when you want to use them. In development, it is useful to use ad-hoc SmartModules, since the compiled WASM code is changing between each run. However, for most actual use-cases, it typically becomes much easier to register the SmartModule and later refer to them by name.
After building a SmartModule as a WASM binary, it may be registered with Fluvio using
fluvio smart-module command, providing a name and a path to the binary. Use SmartModule filters to build a WASM file.
$ fluvio smart-module create my-filter --wasm-file=target/wasm32-unknown-unknown/release/my_filter.wasm
After creating one or more SmartModules, one may use the
fluvio smart-module list command
to see the available SmartModules:
$ fluvio smart-module list NAME STATUS SIZE my-filter SmartModuleStatus 108754
Finally, to delete a SmartModule, use
fluvio smart-module delete and provide the name of the
SmartModule to delete.
$ fluvio smart-module delete my-filter
You may use a Registered SmartModule anywhere that SmartModules may be used. To use them,
you’ll need to provide the name of the SmartModule as well as its type. For example, when
using a registered filter SmartModule with the Fluvio CLI Consumer, provide its name
--filter argument, like so:
$ fluvio consume my-topic -B --filter=my-filter
You may still use SmartModules even without registering them with Fluvio. When using
SmartModules this way, you simply provide a path to the WASM file directly when you
use it. For example, when using an ad-hoc SmartModule with the Fluvio CLI Consumer,
you may pass a file path to
--filter that points to the SmartModule WASM file,
$ fluvio consume my-topic -B --filter=target/wasm32-unknown-unknown/release/my_filter.wasm