Kafka Source
The Kafka Source Connector consumes messages from an existing Kafka topic and appends them to a KurrentDB stream.
Prerequisites
Section titled “Prerequisites”Before using the Kafka source connector, ensure you have:
- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords)
- A Kafka cluster with accessible broker addresses
- Appropriate authentication credentials if using SASL
Quickstart
Section titled “Quickstart”Create the Kafka source with client.Connectors. Replace example-kafka-source with your own
connector id, and topic and schemaName with your own values. Both are required:
Dictionary<string, string> settings = new() { ["instanceTypeName"] = "kafka-source", ["topic"] = "customers", ["consumer:bootstrapServers"] = "localhost:9092", ["schemaName"] = "customer"};
await client.Connectors.Create("example-kafka-source", settings);await client.Connectors.Start("example-kafka-source");Once started, the connector consumes from the topic and appends each message to a KurrentDB stream. Managing Connectors covers the rest of the lifecycle.
Settings
Section titled “Settings”Core options
Section titled “Core options”| Name | Details |
|---|---|
topic |
required Description: Kafka topic to consume from. |
schemaName |
required Description: Name of the schema used for the message. |
partition |
Description: Specific partition to consume from (leave empty for all partitions). |
offset |
Description: Starting offset position (leave empty for default behavior). |
preserveMagicByte |
Description: Whether to preserve Kafka’s schema registry magic byte. Default: false |
Concurrency configuration
Section titled “Concurrency configuration”The Kafka Source Connector processes messages in parallel with multiple consumer tasks. Each task has its own consumer
and buffer, all sharing the group kurrentdb-{ConnectorId}. Their channels run independently and merge into one output
stream for efficient multi-partition handling.
| Name | Details |
|---|---|
concurrency:channelCapacity |
Description: Capacity of each consumer’s bounded channel. Higher values provide more buffering but consume more memory. Default: 10000 |
concurrency:tasks |
Description: The maximum number of tasks to create for this connector. Default: 1 |
Consumer configuration
Section titled “Consumer configuration”The Consumer section accepts standard Confluent Kafka consumer configuration options. Key options include:
| Name | Details |
|---|---|
consumer:bootstrapServers |
Description: Comma-separated list of Kafka broker addresses. Default: "localhost:9092" |
consumer:autoOffsetReset |
Description: Offset reset behavior when there is no initial offset. Default: Earliest |
consumer:enableAutoCommit |
Description: Enable automatic offset commits. Default: true |
consumer:autoCommitIntervalMs |
Description: Auto-commit interval in milliseconds. Default: 5000 |
consumer:sessionTimeoutMs |
Description: Session timeout in milliseconds. Default: 45000 |
consumer:enablePartitionEof |
Description: Enable end-of-partition notifications. Default: true |
consumer:securityProtocol |
Description: Protocol used to communicate with brokers. Accepted Values: plaintext, ssl, saslPlaintextDefault: plaintext |
consumer:saslMechanism |
Description: SASL mechanism to use for authentication. Accepted Values: plainDefault: "" |
consumer:sslCaPem |
Description: CA certificate string (PEM format) for verifying the broker’s key. |
consumer:sslCertificatePem |
Description: Client’s public key string (PEM format) used for authentication. |
consumer:sslKeyPem |
Description: Client’s private key string (PEM format) used for authentication. |
consumer:saslUsername |
Description: SASL username for use with the PLAIN mechanism. |
consumer:saslPassword |
Description: SASL password for use with the PLAIN mechanism. |
Stream routing configuration
Section titled “Stream routing configuration”The stream section configures how the connector routes Kafka messages to KurrentDB streams:
| Name | Details |
|---|---|
stream:strategy |
Description: Stream routing strategy. Accepted Values: - topic: Route to stream named after the Kafka topic- partitionKey: Route based on Kafka message key- fixed: Route all messages to a single stream- header: Extract stream name from message headersDefault: topic |
stream:expression |
Description: Expression for custom routing, depending on strategy. - For fixed: stream name (defaults to topic name if not provided)- For header: comma-separated header names to check- Otherwise: not used |
Authentication
Section titled “Authentication”The default authentication method is plaintext. Set the consumer:securityProtocol option to choose another
method. See Examples for worked configurations.
Examples
Section titled “Examples”Authentication
Section titled “Authentication”SSL/TLS
Section titled “SSL/TLS”{ "instanceTypeName": "kafka-source", "topic": "customers",
"consumer:securityProtocol": "ssl", "consumer:sslCaPem": "...", "consumer:sslCertificatePem": "...", "consumer:sslKeyPem": "..."}SASL/PLAIN
Section titled “SASL/PLAIN”{ "instanceTypeName": "kafka-source", "topic": "customers",
"consumer:securityProtocol": "saslPlaintext", "consumer:saslMechanism": "plain", "consumer:saslUsername": "user", "consumer:saslPassword": "pass"}