Fluvio - Fluvio - The programmable data streaming platform

Programmable platform for data in motion

An open source-data streaming platform with in-line computation capabilities. Apply your custom programs to aggregate, correlate, and transform data records in real-time as they move over the network.

Download

Contribute

The Stack

Smart Streams

Programmable stream processing to clean, transform, correlate, and derive insights from data in real-time.

Data Streams

Distribured stream processing with redundancy and failover to prevent data loss and minimize downtime.

Immutable Store

Long-lived immutable storage layer that persists data without compromising latency.

Clients & Connectors

Native API support to most popular programming languages & connector catalog (in preview).

The Technology

Blazing fast and memory-efficient, no runtime or garbage collector.

Async End-to-End

Scales to many concurrent clients without blocking threads.

Cloud Native

Native integration with Kubernetes, to start small and scale on demand.

Universal Client API

Programming languages are bound natively, for optimal performance.

WebAssembly

Apply powerful custom operations to real-time data streaming.

Declarative Management

Declare desired state and the system applies the difference.

Connect Anywhere

Small footprint binary that runs anywhere from IOT device to multi-core servers.

Horizontal Scale

Scalable streaming processing units (SPU) with built-in replicas, partitions and failover.

Low Latency

Single digit millisecond latency at high throughput and consistent variance up to p99.

Hybrid SPU

Deploy your SPUs anywhere: cloud, data center, VMs, desktop, etc.

Immutable Store

All records are written in the order received and cannot be altered.

Self Service

Powerful CLI with built-in cluster management for hands free opertions.

The Platform

Unified Cluster

A unified cluster for streaming and stateful computation minimizes delay, reduces operational complexity, and boosts security. When streaming and stateful computation are combined, it sets the foundation for a new class of real-time streaming use cases unique to Fluvio. For example:

Normalize: remove null values, map invalid fields, filter out records, and more.
Protect: strip out personally identifiable information (PII), and encrypt fields.
Refine: compute aggregates, derive substreams, and more.

Extensible

Fluvio uses WebAssembly(WASM) to extend real-time stateful computation to a countless number of use cases. Custom modules are loaded dynamically and applied to any number of data streams. For example:

Filter patients with high blood sugar in real-time.
Count all users that have a declined credit card transactions.
Transcode video file when played to a mobile device with insufficient network bandwidth.

Efficient

Fluvio is highly optimized for machine code, and it does not require a virtual machine or garbage collection. It can scale from IOT devices such as Raspberry Pi to multi-core servers.

Low latency: single-digit milliseconds response at high throughput and consistent variance.
Low footprint: small executable and low memory.
Asynchronous architecture: non-blocking calls to reduce latency and scale to a large number of concurrent streams.

Self Healing

Fluvio operates the cluster automatically with minimum human intervention. The platform implements self-healing by combining a variety of technologies:

Declarative Management: a technology pattern where users declare intent and the system provisions resources as they become available.
Reconciliation: the cluster continuously checks system components and brings them to a stable state.
Replication: all data streams can have multiple copies to reduce the possibility of data loss during outages.

Developer Friendly

Fluvio was built with the development community in mind. It offers a powerful CLI for operational efficiency and native language bindings for most common programming languages. For example:

One-Click Deploy: create a cluster locally fluvio cluster start or login to cloud fluvio cloud login with one simple command.
Simple CLI: provision streams, apply stateful computations, produce, consume and more.
Native APIs: support for Rust, Node.js, Python, and Java with other languages coming soon.