# Kafka Source

<ConnectorLogo name="kafka" alt="Apache Kafka" />

The Kafka Source Connector consumes messages from an existing Kafka topic and
appends them to a KurrentDB stream.

## Prerequisites

Before using the Kafka source connector, ensure you have:

- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords)
- A Kafka cluster with accessible broker addresses
- Appropriate authentication credentials if using SASL

:::tip
See the [Data Protection](/kurrent-surge/client/connectors/data-protection/) documentation for instructions on configuring the encryption token.
:::

## Quickstart

Create the Kafka source with `client.Connectors`. Replace `example-kafka-source` with your own
connector id, and `topic` and `schemaName` with your own values. Both are required:

```csharp
Dictionary<string, string> settings = new() {
    ["instanceTypeName"]          = "kafka-source",
    ["topic"]                     = "customers",
    ["consumer:bootstrapServers"] = "localhost:9092",
    ["schemaName"]                = "customer"
};

await client.Connectors.Create("example-kafka-source", settings);
await client.Connectors.Start("example-kafka-source");
```

Once started, the connector consumes from the topic and appends each message to a KurrentDB
stream. [Managing Connectors](/kurrent-surge/client/connectors/manage/) covers the rest of the
lifecycle.

## Settings

### Core options

| Name                | Details                                                                                                    |
|---------------------|------------------------------------------------------------------------------------------------------------|
| `topic`             | _required_<br/><br/>**Description:** Kafka topic to consume from.                                            |
| `schemaName`        | _required_<br/><br/>**Description:** Name of the schema used for the message.                                |
| `partition`         | **Description:** Specific partition to consume from (leave empty for all partitions).                      |
| `offset`            | **Description:** Starting offset position (leave empty for default behavior).                              |
| `preserveMagicByte` | **Description:** Whether to preserve Kafka's schema registry magic byte.<br/><br/>**Default**: `false`       |

### Concurrency configuration

The Kafka Source Connector processes messages in parallel with multiple consumer tasks. Each task has its own consumer
and buffer, all sharing the group `kurrentdb-{ConnectorId}`. Their channels run independently and merge into one output
stream for efficient multi-partition handling.

| Name                          | Details                                                                                                                                                 |
|-------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------|
| `concurrency:channelCapacity` | **Description:** Capacity of each consumer's bounded channel. Higher values provide more buffering but consume more memory.<br/><br/>**Default**: `10000` |
| `concurrency:tasks`           | **Description:** The maximum number of tasks to create for this connector.<br/><br/>**Default**: `1`                                                      |

### Consumer configuration

The `Consumer` section accepts standard Confluent Kafka consumer configuration options. Key options include:

| Name                            | Details                                                                                                                                                   |
|---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| `consumer:bootstrapServers`     | **Description:** Comma-separated list of Kafka broker addresses.<br/><br/>**Default**: `"localhost:9092"`                                                   |
| `consumer:autoOffsetReset`      | **Description:** Offset reset behavior when there is no initial offset.<br/><br/>**Default**: `Earliest`                                                    |
| `consumer:enableAutoCommit`     | **Description:** Enable automatic offset commits.<br/><br/>**Default**: `true`                                                                              |
| `consumer:autoCommitIntervalMs` | **Description:** Auto-commit interval in milliseconds.<br/><br/>**Default**: `5000`                                                                         |
| `consumer:sessionTimeoutMs`     | **Description:** Session timeout in milliseconds.<br/><br/>**Default**: `45000`                                                                             |
| `consumer:enablePartitionEof`   | **Description:** Enable end-of-partition notifications.<br/><br/>**Default**: `true`                                                                        |
| `consumer:securityProtocol`     | **Description:** Protocol used to communicate with brokers.<br/><br/>**Accepted Values:**`plaintext`, `ssl`, `saslPlaintext`<br/><br/>**Default**: `plaintext` |
| `consumer:saslMechanism`        | **Description:** SASL mechanism to use for authentication.<br/><br/>**Accepted Values:**`plain`<br/><br/>**Default**: `""`                                   |
| `consumer:sslCaPem`             | **Description:** CA certificate string (PEM format) for verifying the broker's key.                                                                       |
| `consumer:sslCertificatePem`    | **Description:** Client's public key string (PEM format) used for authentication.                                                                         |
| `consumer:sslKeyPem`            | **Description:** Client's private key string (PEM format) used for authentication.                                                                        |
| `consumer:saslUsername`         | **Description:** SASL username for use with the PLAIN mechanism.                                                                                                    |
| `consumer:saslPassword`         | **Description:** SASL password for use with the PLAIN mechanism.                                                                                                    |

### Stream routing configuration

The `stream` section configures how the connector routes Kafka messages to KurrentDB streams:

| Name                | Details                                                                                                                                                                                                                                                                                                                        |
|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `stream:strategy`   | **Description:** Stream routing strategy. <br/><br/>**Accepted Values:**<br/>- `topic`: Route to stream named after the Kafka topic<br/>- `partitionKey`: Route based on Kafka message key<br/>- `fixed`: Route all messages to a single stream<br/>- `header`: Extract stream name from message headers<br/><br/>**Default**: `topic` |
| `stream:expression` | **Description:** Expression for custom routing, depending on strategy.<br/>- For `fixed`: stream name (defaults to topic name if not provided)<br/>- For `header`: comma-separated header names to check<br/>- Otherwise: not used                                                                                                |

### Authentication

The default authentication method is `plaintext`. Set the `consumer:securityProtocol` option to choose another
method. See [Examples](#examples) for worked configurations.

## Examples

### Authentication

#### SSL/TLS

```json
{
    "instanceTypeName": "kafka-source",
    "topic": "customers",

    "consumer:securityProtocol": "ssl",
    "consumer:sslCaPem": "...",
    "consumer:sslCertificatePem": "...",
    "consumer:sslKeyPem": "..."
}
```

#### SASL/PLAIN

```json
{
    "instanceTypeName": "kafka-source",
    "topic": "customers",

    "consumer:securityProtocol": "saslPlaintext",
    "consumer:saslMechanism": "plain",
    "consumer:saslUsername": "user",
    "consumer:saslPassword": "pass"
}
```