# Kafka Sink

<ConnectorLogo name="kafka" alt="Apache Kafka" />

The Kafka sink writes records from a KurrentDB stream to a Kafka topic, using
an idempotent producer for reliable delivery. It can extract the partition key
from the record based on specific sources such as the stream ID, headers, or
record key. It also supports basic authentication and resilience features that
handle transient errors.

## Prerequisites

Before using the Kafka sink connector, ensure you have:

- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords)
- A Kafka cluster with accessible broker addresses
- Appropriate authentication credentials if using SASL

:::tip
See the [Data Protection](/kurrent-surge/client/connectors/data-protection/) documentation for instructions on configuring the encryption token.
:::

## Quickstart

Create the Kafka sink with `client.Connectors`. Replace `example-kafka` with your own
connector id:

```csharp
Dictionary<string, string> settings = new() {
    ["instanceTypeName"]               = "kafka-sink",
    ["bootstrapServers"]               = "localhost:9092",
    ["topic"]                          = "customers",
    ["subscription:filter:scope"]      = "stream",
    ["subscription:filter:filterType"] = "streamId",
    ["subscription:filter:expression"] = "example-stream"
};

await client.Connectors.Create("example-kafka", settings);
await client.Connectors.Start("example-kafka");
```

Once started, the connector sends each record appended to `example-stream` to the Kafka topic.
[Managing Connectors](/kurrent-surge/client/connectors/manage/) covers the rest of the lifecycle.

## Settings

Adjust these settings to control how the Kafka sink connector behaves and how it interacts with KurrentDB.

:::tip
The Kafka sink inherits a set of common settings that are used to configure the connector. The settings can be found in
the [Sink Options](/kurrent-surge/client/connectors/sinks/#settings) page.
:::

The Kafka sink can be configured with the following options:

| Name               | Details                                                                                                    |
| ------------------ | ---------------------------------------------------------------------------------------------------------- |
| `topic`            | _required_<br/><br/>**Description:**<br/>The Kafka topic to produce records to.                               |
| `bootstrapServers` | **Description:**<br/>Comma-separated list of Kafka broker addresses.<br/><br/>**Default**: `"localhost:9092"` |
| `defaultHeaders`   | **Description:**<br/>Headers included in all produced messages.<br/><br/>**Default**: Empty                   |

### Authentication

| Name                              | Details                                                                                                                                                                         |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `authentication:securityProtocol` | **Description:**<br/>Protocol used for Kafka broker communication.<br/><br/>**Default**: `"plaintext"`<br/><br/>**Accepted Values:**<br/> - `"plaintext"`, `"saslPlaintext"` or `"saslSsl"` |
| `authentication:saslMechanism`    | **Description:**<br/>SASL mechanism to use for authentication.<br/><br/>**Default**: `"plain"`<br/><br/>**Accepted Values:**<br/> - `"plain"`, `"scramSha256"`, or `"scramSha512"`          |
| `authentication:username`         | **Description:**<br/>SASL username                                                                                                                                               |
| `authentication:password`         | **Description:**<br/>SASL password                                                                                                                                               |

### Partitioning

| Name                                | Details                                                                                                                                                                                     |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `partitionKeyExtraction:enabled`    | **Description:**<br/>Enables partition key extraction.<br/><br/>**Default**: `"false"`                                                                                                         |
| `partitionKeyExtraction:source`     | **Description:**<br/>Source for extracting the partition key. See [Partitioning](#partitioning-1)<br/><br/>**Accepted Values:**`"partitionKey"`, `"stream"`, `"streamSuffix"`, or `"headers"`<br/><br/>**Default**: `"partitionKey"` |
| `partitionKeyExtraction:expression` | **Description:**<br/>Regular expression for extracting the partition key.                                                                                                                    |

See the [Partitioning](#partitioning-1) section for examples.

### Resilience

The Kafka sink uses the shared
[resilience settings](/kurrent-surge/client/connectors/sinks/#resilience-settings), including
`resilience:enabled` and the three delay bounds. It adds the keys below on top of them. The last two
are Kafka producer settings that the connector passes straight to the client.

| Name                               | Details                                                                                                                                                                                                    |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `waitForBrokerAck`                 | **Description:**<br/>Whether the producer waits for broker acknowledgment before considering the send operation complete. See [Broker Acknowledgment](#delivery-guarantees)<br/><br/>**Default**: `"false"` |
| `resilience:requestTimeoutMs`      | **Description:**<br/>The timeout in milliseconds for a produce request.<br/><br/>**Default**: `"3000"`                                                                                                        |
| `resilience:maxRetries`            | **Description:**<br/>The maximum number of retry attempts. `"-1"` means retry without a limit.<br/><br/>**Default**: `"-1"`                                                                                   |
| `resilience:transientErrorDelay`   | **Description:**<br/>The delay before retrying a transient error.<br/><br/>**Default**: `"00:00:00"`                                                                                                          |
| `resilience:reconnectBackoffMaxMs` | **Description:**<br/>The maximum time to wait before reconnecting to a broker after the connection has been closed.<br/><br/>**Default**: `"20000"`                                                           |
| `resilience:messageSendMaxRetries` | **Description:**<br/>How many times to retry sending a failing Message.<br/><br/>**Default**: `"2147483647"`                                                                                                  |

### Miscellaneous

| Name                  | Details                                                                                                                                                  |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `brokerAddressFamily` | **Description:**<br/>Allowed broker IP address families.<br/><br/>**Default**: `"V4"`<br/><br/>**Accepted Values:** `"Any"`,`"V4"`, or `"V6"`                 |
| `compression:type`    | **Description:**<br/>Kafka compression type.<br/><br/>**Default**: `"Zstd"`<br/><br/>**Accepted Values:** `"None"`, `"Gzip"`,`"Lz4"`, `"Zstd"`, or `"Snappy"` |
| `compression:level`   | **Description:**<br/>Kafka compression level.<br/><br/>**Default**: `"6"`                                                                                   |

## Delivery guarantees

The Kafka sink provides at-least-once delivery by using Kafka's idempotent
producer and retry settings. The connector checkpoints only after Kafka
confirms successful delivery.

The `waitForBrokerAck` setting controls *when the connector waits* for the
broker acknowledgment:

- If enabled, the connector waits for the broker to confirm
  delivery before moving its checkpoint. This reduces throughput but increases
  backpressure visibility.
- If disabled, the connector does not block while waiting for the broker response. However,
  checkpointing still occurs only after the broker confirms delivery.

The Kafka sink uses an idempotent producer by default, so when writing to a
single partition, Kafka preserves message order regardless of which produce API
is used.

If a failure occurs before acknowledgment, the retry mechanism redelivers.
After a restart, the connector resumes from the last checkpointed position,
which may cause previously sent—but uncheckpointed—messages to be reprocessed.

## Headers

The Kafka sink connector lets you include custom headers in the message headers
it sends to your topic. To add custom headers, use the `defaultHeaders` setting
in your connector configuration. Specify each custom header with the prefix
`defaultHeaders:` followed by the header name.

Example:

```json
{
  "defaultHeaders:X-API-Key": "your-api-key-here",
  "defaultHeaders:X-Tenant-ID": "production-tenant",
  "defaultHeaders:X-Source-System": "KurrentDB"
}
```

The connector includes these headers in every message it sends, alongside the [default headers](/kurrent-surge/client/connectors/sinks/#headers) its plugin adds automatically.

## Examples

### Authentication

The Kafka sink connector supports secure communication with Kafka brokers using
**SASL authentication**. By default, the connector communicates in **plaintext**
without authentication. However, you can configure it to use SASL with different
security protocols and authentication mechanisms.

:::note
When using `saslSsl`, the connector uses your system's trusted CA certificates
for SSL/TLS encryption. This works with managed services like **AWS MSK**,
**Confluent Cloud**, and **Azure Event Hubs**. For self-signed or private CA
certificates, add them to your system's trust store first.
:::

#### SASL/PLAINTEXT with PLAIN authentication
```json
{
  "authentication:securityProtocol": "saslPlaintext",
  "authentication:saslMechanism": "plain",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

#### SASL/PLAINTEXT with SCRAM-SHA-256 authentication
```json
{
  "authentication:securityProtocol": "saslPlaintext",
  "authentication:saslMechanism": "scramSha256",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

#### SASL/PLAINTEXT with SCRAM-SHA-512 authentication
```json
{
  "authentication:securityProtocol": "saslPlaintext",
  "authentication:saslMechanism": "scramSha512",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

#### SASL/SSL with PLAIN authentication

For production environments with encryption (recommended for managed Kafka services):
```json
{
  "authentication:securityProtocol": "saslSsl",
  "authentication:saslMechanism": "plain",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

#### SASL/SSL with SCRAM-SHA-256 authentication
```json
{
  "authentication:securityProtocol": "saslSsl",
  "authentication:saslMechanism": "scramSha256",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

#### SASL/SSL with SCRAM-SHA-512 authentication
```json
{
  "authentication:securityProtocol": "saslSsl",
  "authentication:saslMechanism": "scramSha512",
  "authentication:username": "my-username",
  "authentication:password": "my-password"
}
```

### Additional resources
- [Azure Event Hub Security and Authentication](https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-apache-kafka-overview#security-and-authentication)
- [Set up SASL/SCRAM authentication for an Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial.html)
- [Use SASL/SCRAM authentication in Confluent Platform](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html#use-sasl-scram-authentication-in-cp)

### Partitioning

The Kafka sink connector lets you customize the partition keys sent with the
message.

`partitionKeyExtraction:enabled` defaults to `"false"`, so the connector
extracts no key unless you enable it. Without a key, Kafka distributes messages
round-robin across the available partitions in the topic.

**Partition using Stream ID**

You can extract part of the stream name using a regular expression (regex) to
define the partition key. The expression is optional, and you can adapt it to
your naming convention. In this example, the expression captures the stream
name up to `_data`.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "stream",
  "partitionKeyExtraction:expression": "^(.*)_data$"
}
```

Alternatively, if you need only the last segment of the stream name (after a
hyphen), you can use the `streamSuffix` source. This
doesn't require an expression since it automatically extracts the suffix.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "streamSuffix"
}
```

The `streamSuffix` source is useful when stream names follow a structured
format, and you want to use only the trailing part as the partition key. For
example, if the stream is named `user-123`, the partition key is `123`.

**Partition using header values**

You can create partition keys by combining values from a record's metadata.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "headers",
  "partitionKeyExtraction:expression": "key1,key2"
}
```

Specify the header keys you want to use in the `partitionKeyExtraction:expression` field (e.g., `key1,key2`). The connector concatenates the header values with a hyphen (`-`) to create the partition key.

For example, if your record has headers `key1: regionA` and `key2: zone1`, the partition key is `regionA-zone1`.

## Tutorial

[Learn how to set up and use a Kafka Sink connector in KurrentDB through a tutorial.](https://docs.kurrent.io/server/v26.1/tutorials/Kafka_Sink)