# Pulsar Sink

<ConnectorLogo name="pulsar" alt="Apache Pulsar" />

The Apache Pulsar Sink connector writes records from your KurrentDB stream to a specified Pulsar topic.

## Prerequisites

Before using the Pulsar sink connector, ensure you have:

- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like authentication tokens)
- A Pulsar cluster with an accessible service URL
- Appropriate authentication token if your Pulsar cluster requires authentication

:::tip
See the [Data Protection](/kurrent-surge/client/connectors/data-protection/) documentation for instructions on configuring the encryption token.
:::

## Quickstart

Create the Pulsar sink with `client.Connectors`. Replace `example-pulsar` with your own
connector id:

```csharp
Dictionary<string, string> settings = new() {
    ["instanceTypeName"]               = "pulsar-sink",
    ["topic"]                          = "customers",
    ["url"]                            = "pulsar://localhost:6650",
    ["subscription:filter:scope"]      = "stream",
    ["subscription:filter:filterType"] = "streamId",
    ["subscription:filter:expression"] = "example-stream"
};

await client.Connectors.Create("example-pulsar", settings);
await client.Connectors.Start("example-pulsar");
```

Check the connector's state with `client.Connectors.List`, or read the connector logs.
[Managing Connectors](/kurrent-surge/client/connectors/manage/) covers the rest of the lifecycle.

## Settings

These settings control how the connector interacts with Pulsar, partitions messages, and stays resilient while handling them.

:::tip
The Pulsar sink inherits a set of common settings that are used to configure the connector. The settings can be found in
the [Sink Options](/kurrent-surge/client/connectors/sinks/#settings) page.
:::

### General settings 

| Name                   | Details                                                                                                     |
| ---------------------- | ----------------------------------------------------------------------------------------------------------- |
| `topic`                | _required_<br/><br/>**Description:**<br/>The Pulsar topic that the connector publishes records to.            |
| `url`                  | **Description:**<br/>The service URL for the Pulsar cluster.<br/><br/>**Default**: `"pulsar://localhost:6650"` |
| `defaultHeaders`       | **Description:**<br/>Headers included in all produced messages.<br/><br/>**Default**: Empty                    |
| `authentication:token` | **Description:**<br/>A JSON web token for authenticating the connector with Pulsar.                          |

### Partitioning

Partitioning options determine how the connector assigns partition keys, which affect message routing and topic compaction.

| Name                                | Details                                                                                                                                                                                                                                                           |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `partitionKeyExtraction:enabled`    | **Description:**<br/>Enables partition key extraction.<br/><br/>**Default**: `"false"`                                                                                                                                                                                      |
| `partitionKeyExtraction:source`     | **Description:**<br/>The source for extracting the partition key.<br/><br/>**Accepted Values:**`"partitionKey"`, `"stream"`, `"streamSuffix"`, `"headers"`<br/><br/>**Default**: `"partitionKey"`                                                                                                   |
| `partitionKeyExtraction:expression` | **Description:**<br/>A regex (for `stream` source) or a comma-separated list of header keys (for `headers` source) used to extract or combine values for the partition key. When using headers, values are concatenated with a hyphen (for example, `value1-value2`). |

### Resilience

The Pulsar sink connector relies on its own retry mechanism and doesn't include the configuration from [Resilience configuration](/kurrent-surge/client/connectors/sinks/#resilience-settings).

| Name                       | Details                                                                                                                                           |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `resilience:enabled`       | **Description:**<br/>Enables resilience features for message handling.<br/><br/>**Default**: `"true"`                                                   |
| `resilience:retryInterval` | **Description:**<br/>Retry interval in seconds. Must be greater than 0.<br/><br/>**Format:** seconds or `"HH:MM:SS"`.<br/><br/> **Default:** `"00:00:03"` |

## Headers

The Pulsar sink connector lets you include custom headers in the message headers
it sends to your topic. To add custom headers, use the `defaultHeaders` setting
in your connector configuration. Specify each custom header with the prefix
`defaultHeaders:` followed by the header name.

Example:

```json
{
  "defaultHeaders:X-API-Key": "your-api-key-here",
  "defaultHeaders:X-Tenant-ID": "production-tenant",
  "defaultHeaders:X-Source-System": "KurrentDB"
}
```

The connector includes these headers in every message it sends, alongside the [default headers](/kurrent-surge/client/connectors/sinks/#headers) its plugin adds automatically.

## Examples

### Partitioning

The Pulsar sink connector lets you customize the partition keys sent with the
message.

`partitionKeyExtraction:enabled` defaults to `"false"`, so the connector extracts no key unless you enable it.
Without a key, Pulsar distributes messages round-robin across the available partitions in the topic.

**Partition using Stream ID**

You can extract part of the stream name using a regular expression (regex) to
define the partition key. The expression is optional, and you can adapt it to
your naming convention. In this example, the expression captures the stream
name up to `_data`.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "stream",
  "partitionKeyExtraction:expression": "^(.*)_data$"
}
```

Alternatively, if you need only the last segment of the stream name (after a
hyphen), you can use the `streamSuffix` source. This
doesn't require an expression since it automatically extracts the suffix.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "streamSuffix"
}
```

The `streamSuffix` source is useful when stream names follow a structured
format, and you want to use only the trailing part as the partition key. For
example, if the stream is named `user-123`, the partition key is `123`.

**Partition using header values**

You can create partition keys by combining values from a record's metadata.

```json
{
  "partitionKeyExtraction:enabled": "true",
  "partitionKeyExtraction:source": "headers",
  "partitionKeyExtraction:expression": "key1,key2"
}
```

Specify the header keys you want to use in the `partitionKeyExtraction:expression` field (e.g., `key1,key2`). The connector concatenates the header values with a hyphen (`-`) to create the partition key.

For example, if your record has headers `key1: regionA` and `key2: zone1`, the partition key is `regionA-zone1`.