Architecture

Fluvio Data Streaming is a modern Cloud Native software stack designed for high speed, real-time data processing. Fluvio is fast, scalable, self-healing, pluggable, and user-friendly.

Built in Rust

Fluvio is built in Rust, a systems programming language with higher performance than Java and better code safety than C/C++. Rust has a powerful multi-threaded asynchronous engine that runs natively in multi-core and low powered embedded systems. Zero cost abstractions and no garbage collection makes this language ideal for low network latency and high IO throughput systems.

The choice of programming language makes Fluvio a low memory, high performance product that compiles natively in many software distributions such as MacOS, Linux, Windows, and small footprint embedded systems such as Raspberry Pi.

Cloud Native by Design

Fluvio is a Cloud Native platform designed to work with any infrastructure type from bare bones hardware to containerized platforms. As a Cloud Native first product, Fluvio is natively integrated with Kubernetes. Any infrastructure running Kubernetes can install the Fluvio Helm Chart and get up and running in a matter of minutes. For additional details, check out the Kubernetes integration section.

Fluvio Cloud

If you don’t have Kubernetes installed or prefer to run Fluvio as a Service, you can use Fluvio Cloud. The cloud installation hides all the complexity associated with the infrastructure and exposes only relevant streaming APIs. Use the Getting Started guide to set up your dedicated cloud environment.

High Level Architecture 

Fluvio’s architecture centers around real time streaming, and the platform can scale horizontally to accommodate large volumes of data.

Architecture Components - SC/SPUs

A Streaming Controller (SC) manages the SPU life cycle and optimizes the distribution of data streams across the cluster. The Streaming Processing Units (SPUs) are responsible for data streaming.

SCs and SPUs are independent, loosely coupled services. Each service can be restarted, upgraded, or scaled independently without impacting traffic.

Streaming Controller (SC) 

Fluvio is designed to address a variety of deployment scenarios from public clouds to private data centers, edge networks and IOT devices. SC maintains the topology map of the SPUs and serves as the first point of contact for producers and consumers.

DC, Cloud, Edge, IoT

The SC handles topology map dynamically to simplify complex tasks such as increasing capacity, adding new infrastructure, or attaching a new geo-locations.

For a deep dive in the SC design, checkout SC Architecture section.

Streaming Processing Unit (SPU) 

Streaming Processing Units (SPUs) are responsible for all data streaming related matters. Each SPU receives data from producers, sends data to consumers, and saves copies of the data to local storage.

SPU produce/consume & replication

SPUs are also responsible for data replication. Data streams that are created with a replication factor of 2 or more are managed by a cluster of SPUs. One SPU is elected as leader and all others are followers. The leader receives the data from consumers and forwards a copy to followers. Followers save a copy in their local storage. If the leader goes offline, one of the followers takes over as leader. For additional information, check out Replica Election.

Each SPU performs leader and follower duties on multiple data streams in parallel. For optimal performance, Fluvio utilizes all available CPU cores.

For a deep dive into the SPU design, check out the SPU Architecture section.

Topic/Partitions 

A Topic in a streaming platform is like a Table in a database. Suppose you’re building an online chat service. You may have a “Chatroom” topic that streams events such as “Sent Message” and “Viewed Message”, as well as other related events. You can think of a Topic as a category of events that are related under your domain.

Now suppose that your chat service becomes a wild success and you have thousands of users. In order to keep up with the increased traffic, you can divide your topic into multiple Partitions. This allows the events in your topic to be distributed between multiple SPUs in parallel, increasing your traffic capacity by just changing a setting.

For example, a configuration with the 2 topics generates the replication map in the diagram:

  • topic-a => 2 partitions, 2 replicas
  • topic-b => 1 partition, 3 replicas

SPU-1 is the leader for topic-a/0 , SPU-2 for topic-a/1, and SPU-3 for topic-b/0.

Topic/Partitions

For additional information on partitions and replica assignments, checkout Replica Assignment.

Data Persistence 

SPU leaders save all data stream messages received from producers on local storage. Based on platform availability, SPUs use zero-copy IO to transfer data from disk to network. Messages on local storage are immutable and ordered. Fluvio guarantees in-order writes for all messages received on the same replica.

Data Storage

SPU persistence is designed as single-writer, multi-reader with zero-copy writes. Each SPU can save large volumes of data at wire speed, and serve consumers and producers in near real-time.

Fluvio adds messages local storage until the retention period is met. The retention periods should be set to cover up to 80% of the disk size. If the disk is full before the retention period is triggered, the SPU stops accepting messages and the overall health of the system may be compromised.

APIs 

The Fluvio architecture places heavy emphasis on clean, user-friendly APIs. There are two types of APIs, external and internal. The APIs use TLS to ensure secure communication.

External APIs 

External APIs are used by the Fluvio CLI and a growing number of programming language interfaces, such as Node and Rust. There are two categories of APIs, control plane APIs and data plane APIs. Control Plane APIs manage the life cycle of the cluster objects such as SPUs, topics, and replicas. Data Plane APIs handle data access for producers and consumers.

External APIs

API reference guides for programming languages are available at:

Internal APIs 

Internal APIs are used by the SC communicate with the SPUs and for the SPUs to communicate with their peers to elect leaders and exchange replica information.

Internal APIs

For additional details about Internal APIs checkout Fluvio development guide on github.

Next Steps