Generate
 

Generate a Connector

CDK generate helps developers build a sample Connector project by answering a few simple questions.

Use cdk generate to create a new connector project:

$ cdk generate
๐Ÿคท   Project Name: my-connector
๐Ÿ”ง   Destination: ~/my-connector ...
๐Ÿ”ง   project-name: my-connector ...
๐Ÿ”ง   Generating template ...
โœ” ๐Ÿคท   Will your Connector be public? ยท false
๐Ÿคท   Please set a group name: acme
โœ” ๐Ÿคท   Which type of Connector would you like [source/sink]? ยท source
Ignoring: /var/folders/r8/4x6_d2rn283946frzd1gc1pr0000gn/T/.tmptToFV3/cargo-generate.toml
[1/6]   Done: Cargo.toml             
[2/6]   Done: Connector.toml
[3/6]   Done: sample-config.yaml
[4/6]   Done: src/config.rs
[5/6]   Done: src/main.rs
[6/6]   Done: src
๐Ÿ”ง   Moving generated files into: `~/my-connector`...
๐Ÿ’ก   Initializing a fresh Git repository
โœจ   Done! New project created ~/my-connector

The generator created Rust project ready to compile:

$ tree 
.
โ”œโ”€โ”€ Cargo.toml
โ”œโ”€โ”€ Connector.toml
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ sample-config.yaml
โ””โ”€โ”€ src
    โ”œโ”€โ”€ config.rs
    โ””โ”€โ”€ main.rs

This a simple connector with the code in src/main.rs:

mod config;
use config::CustomConfig;

use fluvio::{RecordKey, TopicProducer};
use fluvio_connector_common::{
    connector,
    Result
};

#[connector(source)]
async fn start(config: CustomConfig, producer: TopicProducer) -> Result<()> {
    println!("Starting my-connector source connector with {config:?}");
    for i in 1..1000 {
        let value = format!("Hello, Fluvio - {i}");
        producer.send(RecordKey::NULL, value).await?;
        producer.flush().await?;
        std::thread::sleep(std::time::Duration::from_millis(1000));
    }
    Ok(())
}

Connectors may also have can have configuration parameters as defined in src/config.rs:

use fluvio_connector_common::connector;

#[connector(config)]
#[derive(Debug)]
pub(crate) struct CustomConfig {
    #[allow(dead_code)]
    foo: String,
}

The Connector.toml file contains the definition of the Connector parameters required to load the file in the Cluster and publish it to Connector Hub.

$ cat Connector.toml
[package]
name = "my-connector"
group = "acme"
version = "0.1.0"
apiVersion = "0.1.0"
fluvio = "0.21.3"
description = ""
license = "Apache-2.0"
visibility = "private"

[direction]
source = true

[deployment]
binary = "my-connector"

[custom.properties.foo]
title = "Foo"
description = "Foo"
type = "string"
 

Sections

  • package is used to build the connector FQDN acme/my-connector@0.1.0, and the description to publish to Hub. The group name is equivalent to the package owner in the Hub.
  • direction is used to declare the direction data flows through the connector, with respect to the Fluvio cluster. An inbound connector uses source = true, and and outbound connectdor uses sink = true
  • custom.properties.foo defines a user configuration key foo that can be used in the logic of the connector.

The project is ready to build and test. Checkout the next section for instructions.

 

Steps

  1. Generate a Connector
  2. Build and Test
  3. Start and Shutdown
  4. Troubleshooting
  5. Secrets
  6. Publish to Connector Hub
  7. Use Examples in Github