Skip to content

Kafka Sink

License Required

The Kafka sink writes records from a KurrentDB stream to a Kafka topic, using an idempotent producer for reliable delivery. It can extract the partition key from the record based on specific sources such as the stream ID, headers, or record key. It also supports basic authentication and resilience features that handle transient errors.

Before using the Kafka sink connector, ensure you have:

  • A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords)
  • A Kafka cluster with accessible broker addresses
  • Appropriate authentication credentials if using SASL

Create the Kafka sink with client.Connectors. Replace example-kafka with your own connector id:

Dictionary<string, string> settings = new() {
["instanceTypeName"] = "kafka-sink",
["bootstrapServers"] = "localhost:9092",
["topic"] = "customers",
["subscription:filter:scope"] = "stream",
["subscription:filter:filterType"] = "streamId",
["subscription:filter:expression"] = "example-stream"
};
await client.Connectors.Create("example-kafka", settings);
await client.Connectors.Start("example-kafka");

Once started, the connector sends each record appended to example-stream to the Kafka topic. Managing Connectors covers the rest of the lifecycle.

Adjust these settings to control how the Kafka sink connector behaves and how it interacts with KurrentDB.

The Kafka sink can be configured with the following options:

Name Details
topic required

Description:
The Kafka topic to produce records to.
bootstrapServers Description:
Comma-separated list of Kafka broker addresses.

Default: "localhost:9092"
defaultHeaders Description:
Headers included in all produced messages.

Default: Empty
Name Details
authentication:securityProtocol Description:
Protocol used for Kafka broker communication.

Default: "plaintext"

Accepted Values:
- "plaintext", "saslPlaintext" or "saslSsl"
authentication:saslMechanism Description:
SASL mechanism to use for authentication.

Default: "plain"

Accepted Values:
- "plain", "scramSha256", or "scramSha512"
authentication:username Description:
SASL username
authentication:password Description:
SASL password
Name Details
partitionKeyExtraction:enabled Description:
Enables partition key extraction.

Default: "false"
partitionKeyExtraction:source Description:
Source for extracting the partition key. See Partitioning

Accepted Values:"partitionKey", "stream", "streamSuffix", or "headers"

Default: "partitionKey"
partitionKeyExtraction:expression Description:
Regular expression for extracting the partition key.

See the Partitioning section for examples.

The Kafka sink uses the shared resilience settings, including resilience:enabled and the three delay bounds. It adds the keys below on top of them. The last two are Kafka producer settings that the connector passes straight to the client.

Name Details
waitForBrokerAck Description:
Whether the producer waits for broker acknowledgment before considering the send operation complete. See Broker Acknowledgment

Default: "false"
resilience:requestTimeoutMs Description:
The timeout in milliseconds for a produce request.

Default: "3000"
resilience:maxRetries Description:
The maximum number of retry attempts. "-1" means retry without a limit.

Default: "-1"
resilience:transientErrorDelay Description:
The delay before retrying a transient error.

Default: "00:00:00"
resilience:reconnectBackoffMaxMs Description:
The maximum time to wait before reconnecting to a broker after the connection has been closed.

Default: "20000"
resilience:messageSendMaxRetries Description:
How many times to retry sending a failing Message.

Default: "2147483647"
Name Details
brokerAddressFamily Description:
Allowed broker IP address families.

Default: "V4"

Accepted Values: "Any","V4", or "V6"
compression:type Description:
Kafka compression type.

Default: "Zstd"

Accepted Values: "None", "Gzip","Lz4", "Zstd", or "Snappy"
compression:level Description:
Kafka compression level.

Default: "6"

The Kafka sink provides at-least-once delivery by using Kafka’s idempotent producer and retry settings. The connector checkpoints only after Kafka confirms successful delivery.

The waitForBrokerAck setting controls when the connector waits for the broker acknowledgment:

  • If enabled, the connector waits for the broker to confirm delivery before moving its checkpoint. This reduces throughput but increases backpressure visibility.
  • If disabled, the connector does not block while waiting for the broker response. However, checkpointing still occurs only after the broker confirms delivery.

The Kafka sink uses an idempotent producer by default, so when writing to a single partition, Kafka preserves message order regardless of which produce API is used.

If a failure occurs before acknowledgment, the retry mechanism redelivers. After a restart, the connector resumes from the last checkpointed position, which may cause previously sent—but uncheckpointed—messages to be reprocessed.

The Kafka sink connector lets you include custom headers in the message headers it sends to your topic. To add custom headers, use the defaultHeaders setting in your connector configuration. Specify each custom header with the prefix defaultHeaders: followed by the header name.

Example:

{
"defaultHeaders:X-API-Key": "your-api-key-here",
"defaultHeaders:X-Tenant-ID": "production-tenant",
"defaultHeaders:X-Source-System": "KurrentDB"
}

The connector includes these headers in every message it sends, alongside the default headers its plugin adds automatically.

The Kafka sink connector supports secure communication with Kafka brokers using SASL authentication. By default, the connector communicates in plaintext without authentication. However, you can configure it to use SASL with different security protocols and authentication mechanisms.

{
"authentication:securityProtocol": "saslPlaintext",
"authentication:saslMechanism": "plain",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

SASL/PLAINTEXT with SCRAM-SHA-256 authentication

Section titled “SASL/PLAINTEXT with SCRAM-SHA-256 authentication”
{
"authentication:securityProtocol": "saslPlaintext",
"authentication:saslMechanism": "scramSha256",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

SASL/PLAINTEXT with SCRAM-SHA-512 authentication

Section titled “SASL/PLAINTEXT with SCRAM-SHA-512 authentication”
{
"authentication:securityProtocol": "saslPlaintext",
"authentication:saslMechanism": "scramSha512",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

For production environments with encryption (recommended for managed Kafka services):

{
"authentication:securityProtocol": "saslSsl",
"authentication:saslMechanism": "plain",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

SASL/SSL with SCRAM-SHA-256 authentication

Section titled “SASL/SSL with SCRAM-SHA-256 authentication”
{
"authentication:securityProtocol": "saslSsl",
"authentication:saslMechanism": "scramSha256",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

SASL/SSL with SCRAM-SHA-512 authentication

Section titled “SASL/SSL with SCRAM-SHA-512 authentication”
{
"authentication:securityProtocol": "saslSsl",
"authentication:saslMechanism": "scramSha512",
"authentication:username": "my-username",
"authentication:password": "my-password"
}

The Kafka sink connector lets you customize the partition keys sent with the message.

partitionKeyExtraction:enabled defaults to "false", so the connector extracts no key unless you enable it. Without a key, Kafka distributes messages round-robin across the available partitions in the topic.

Partition using Stream ID

You can extract part of the stream name using a regular expression (regex) to define the partition key. The expression is optional, and you can adapt it to your naming convention. In this example, the expression captures the stream name up to _data.

{
"partitionKeyExtraction:enabled": "true",
"partitionKeyExtraction:source": "stream",
"partitionKeyExtraction:expression": "^(.*)_data$"
}

Alternatively, if you need only the last segment of the stream name (after a hyphen), you can use the streamSuffix source. This doesn’t require an expression since it automatically extracts the suffix.

{
"partitionKeyExtraction:enabled": "true",
"partitionKeyExtraction:source": "streamSuffix"
}

The streamSuffix source is useful when stream names follow a structured format, and you want to use only the trailing part as the partition key. For example, if the stream is named user-123, the partition key is 123.

Partition using header values

You can create partition keys by combining values from a record’s metadata.

{
"partitionKeyExtraction:enabled": "true",
"partitionKeyExtraction:source": "headers",
"partitionKeyExtraction:expression": "key1,key2"
}

Specify the header keys you want to use in the partitionKeyExtraction:expression field (e.g., key1,key2). The connector concatenates the header values with a hyphen (-) to create the partition key.

For example, if your record has headers key1: regionA and key2: zone1, the partition key is regionA-zone1.

Learn how to set up and use a Kafka Sink connector in KurrentDB through a tutorial.