Welcome to This Week in Fluvio, our weekly newsletter for development updates to Fluvio open source. Fluvio is a distributed, programmable streaming platform written in Rust.
We have added time-based retention policy for data stored in topics. When records are created, we keep track of its age. When a record’s age reaches the same duration as the retention policy, it is purged.
You can configure the retention duration time when you create a topic.
# Some example durations: '1h', '2d 10s', '7 days' $ fluvio topic create <topic-name> --retention-time <time>
Along with the introduction of retention policy, new topics will be created with a default
7 day retention.
Docs about retention policy coming soon.
For processing live data, using a batching workflow for sending records improves the efficiency of your data transfers by increasing throughput and reducing latency for each producer send. (As opposed to sending records individually)
Producer batch support already exists in the CLI using
fluvio produce, but you can realistically only use this CLI feature if you produce with the
At the end of the day, it meant that if you want time-based or size-based batching, this was extra effort for the developer to implement themselves.
In this release, we make it easier to use batching in the Rust API. To use create an auto-batching Producer, you need to create your
TopicProducer configured with batch and/or linger.
let fluvio_client = Fluvio::connect().await?; let config = TopicProducerConfigBuilder::default() .linger(Duration::from_millis(600000)) .batch_size(17) .build() .expect("failed to build config"); let producer: TopicProducer = fluvio_client .topic_producer_with_config(topic, config) .await;
For more detail on the available config options, see the Rust docs
The ability to test pre-release changes in CLI is now easier to do with CLI channels.
More documentation is coming soon, but if you’re familiar with Rust’s release channels, you’ll be familiar with Fluvio’s CLI channels.
New Fluvio installations support the ability to switch back and forth between the most recent
stable release or the
latest development builds of the Fluvio CLI.
CLI channels will be especially useful for the current users who have reached out to us on Discord. Now we can more easily work together to quickly validate fixes to issues without the need to build the Fluvio code locally.
To try out channels now, you will need to re-install Fluvio with the instructions on the download page. This will download the channel-enabled frontend and the most recent
# Switch to the `latest` channel $ fluvio version switch latest # Switch to the `stable` channel $ fluvio version switch stable
In the CLI, to start consuming records for a specific starting offset, you would use the
--offset flag. Now you can also provide a final offset to close the Consumer stream when reached with the
twifwith an ending offset of
fluvio produceto send over
10records, which we will show first.
Terminal 1 - Producer:
$ fluvio produce twif > 0 Ok! > 1 Ok! > 2 Ok! > 3 Ok! > 4 Ok! > 5 Ok! > 6 Ok! > 7 Ok! > 8 Ok! > 9 Ok! > 10 Ok!
Terminal 2 - Record indexing is 0-based, so we expect the stream to close when we receive the 6th record.
$ fluvio consume -B --end-offset 5 twif Consuming records from the beginning of topic 'twif' 0 1 2 3 4 5 ⠋ Consumer stream has closed
We can also use a starting offset and ending offset together. As a result you can capture chunks of continuous blocks of records.
Here we use the existing
twif topic, and consume a small subset of the records we produced earlier between offset 3-7 (inclusive).
$ fluvio consume --offset 3 --end-offset 7 twif Consuming records from offset 3 in topic 'twif' 3 4 5 6 7 ⠁ Consumer stream has closed
For the full list of changes this week, be sure to check out our CHANGELOG.
Until next week!