Skip to content

Kafka Source

License Required

The Kafka Source Connector consumes messages from an existing Kafka topic and appends them to a KurrentDB stream.

Before using the Kafka source connector, ensure you have:

  • A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords)
  • A Kafka cluster with accessible broker addresses
  • Appropriate authentication credentials if using SASL

Create the Kafka source with client.Connectors. Replace example-kafka-source with your own connector id, and topic and schemaName with your own values. Both are required:

Dictionary<string, string> settings = new() {
["instanceTypeName"] = "kafka-source",
["topic"] = "customers",
["consumer:bootstrapServers"] = "localhost:9092",
["schemaName"] = "customer"
};
await client.Connectors.Create("example-kafka-source", settings);
await client.Connectors.Start("example-kafka-source");

Once started, the connector consumes from the topic and appends each message to a KurrentDB stream. Managing Connectors covers the rest of the lifecycle.

Name Details
topic required

Description: Kafka topic to consume from.
schemaName required

Description: Name of the schema used for the message.
partition Description: Specific partition to consume from (leave empty for all partitions).
offset Description: Starting offset position (leave empty for default behavior).
preserveMagicByte Description: Whether to preserve Kafka’s schema registry magic byte.

Default: false

The Kafka Source Connector processes messages in parallel with multiple consumer tasks. Each task has its own consumer and buffer, all sharing the group kurrentdb-{ConnectorId}. Their channels run independently and merge into one output stream for efficient multi-partition handling.

Name Details
concurrency:channelCapacity Description: Capacity of each consumer’s bounded channel. Higher values provide more buffering but consume more memory.

Default: 10000
concurrency:tasks Description: The maximum number of tasks to create for this connector.

Default: 1

The Consumer section accepts standard Confluent Kafka consumer configuration options. Key options include:

Name Details
consumer:bootstrapServers Description: Comma-separated list of Kafka broker addresses.

Default: "localhost:9092"
consumer:autoOffsetReset Description: Offset reset behavior when there is no initial offset.

Default: Earliest
consumer:enableAutoCommit Description: Enable automatic offset commits.

Default: true
consumer:autoCommitIntervalMs Description: Auto-commit interval in milliseconds.

Default: 5000
consumer:sessionTimeoutMs Description: Session timeout in milliseconds.

Default: 45000
consumer:enablePartitionEof Description: Enable end-of-partition notifications.

Default: true
consumer:securityProtocol Description: Protocol used to communicate with brokers.

Accepted Values:plaintext, ssl, saslPlaintext

Default: plaintext
consumer:saslMechanism Description: SASL mechanism to use for authentication.

Accepted Values:plain

Default: ""
consumer:sslCaPem Description: CA certificate string (PEM format) for verifying the broker’s key.
consumer:sslCertificatePem Description: Client’s public key string (PEM format) used for authentication.
consumer:sslKeyPem Description: Client’s private key string (PEM format) used for authentication.
consumer:saslUsername Description: SASL username for use with the PLAIN mechanism.
consumer:saslPassword Description: SASL password for use with the PLAIN mechanism.

The stream section configures how the connector routes Kafka messages to KurrentDB streams:

Name Details
stream:strategy Description: Stream routing strategy.

Accepted Values:
- topic: Route to stream named after the Kafka topic
- partitionKey: Route based on Kafka message key
- fixed: Route all messages to a single stream
- header: Extract stream name from message headers

Default: topic
stream:expression Description: Expression for custom routing, depending on strategy.
- For fixed: stream name (defaults to topic name if not provided)
- For header: comma-separated header names to check
- Otherwise: not used

The default authentication method is plaintext. Set the consumer:securityProtocol option to choose another method. See Examples for worked configurations.

{
"instanceTypeName": "kafka-source",
"topic": "customers",
"consumer:securityProtocol": "ssl",
"consumer:sslCaPem": "...",
"consumer:sslCertificatePem": "...",
"consumer:sslKeyPem": "..."
}
{
"instanceTypeName": "kafka-source",
"topic": "customers",
"consumer:securityProtocol": "saslPlaintext",
"consumer:saslMechanism": "plain",
"consumer:saslUsername": "user",
"consumer:saslPassword": "pass"
}