# Elasticsearch Sink

<ConnectorLogo name="elasticsearch" alt="Elasticsearch" />

The Elasticsearch sink reads records from a KurrentDB stream and stores them in
an Elasticsearch index. The connector serializes each record as a JSON document
compatible with Elasticsearch's document structure.

## Prerequisites

Before using the Elasticsearch sink connector, ensure you have:

- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords and API keys)
- An Elasticsearch cluster with an accessible URL
- Appropriate authentication credentials (basic auth, token, or API key)

:::tip
See the [Data Protection](/kurrent-surge/client/connectors/data-protection/) documentation for instructions on configuring the encryption token.
:::

## Quickstart

Create the Elasticsearch sink with `client.Connectors`. Replace `example-elastic` with your own
connector id:

```csharp
Dictionary<string, string> settings = new() {
    ["instanceTypeName"]               = "elasticsearch-sink",
    ["url"]                            = "http://localhost:9200",
    ["indexName"]                      = "sample-index",
    ["subscription:filter:scope"]      = "stream",
    ["subscription:filter:filterType"] = "streamId",
    ["subscription:filter:expression"] = "example-stream"
};

await client.Connectors.Create("example-elastic", settings);
await client.Connectors.Start("example-elastic");
```

Once started, the connector writes each record appended to `example-stream` into the index.
[Managing Connectors](/kurrent-surge/client/connectors/manage/) covers the rest of the lifecycle.

## Settings

Adjust these settings to control how the Elasticsearch sink connector behaves
and how it interacts with KurrentDB.

:::tip
The Elasticsearch sink inherits a set of common settings that are used to
configure the connector. The settings can be found in the [Sink Options](/kurrent-surge/client/connectors/sinks/#settings) page.
:::

The Elasticsearch sink can be configured with the following options:

| Name                                        | Details                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                                       | _required_<br/><br/>**Description:**<br/>The URL of the Elasticsearch cluster to which the connector connects.<br/><br/>**Default**: `"http://localhost:9200"`                                                                                                                                                                                                                                          |
| `indexName`                                 | _required_<br/><br/>**Description:**<br/>The index name that the connector writes records to.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                                                              |
| `refresh`                                   | **Description:**<br/>Specifies whether Elasticsearch should refresh the affected shards to make this operation visible to search. If set to `"true"`, Elasticsearch refreshes the affected shards. If set to `"wait_for"`, Elasticsearch waits for a refresh to make this operation visible to search. If set to `"false"`, Elasticsearch does nothing with refreshes.<br/><br/>**Default**: `"true"` |
| `batching:batchSize`                        | **Description:**<br/>Threshold batch size at which the sink pushes the batch of records to the Elasticsearch index.<br/><br/>**Default**: `"1000"`                                                                                                                                                                                                                        |
| `batching:batchTimeoutMs`                   | **Description:**<br/>Threshold time in milliseconds at which the sink pushes the current batch of records to the Elasticsearch index, regardless of the batch size.<br/><br/>**Default**: `"250"`                                                                                                                                                                         |

### Authentication

| Name                                        | Details                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `authentication:method`                     | **Description:**<br/>The authentication method used by the connector to connect to the Elasticsearch cluster.<br/><br/>**Default**: `"basic"`<br/><br/>**Accepted Values:** `"basic"`, `"token"`, `"apiKey"`                                                                                                                                                                                            |
| `authentication:username`                   | **Description:**<br/>The username used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided.<br/><br/>**Default**: `"elastic"`                                                                                                                                                                                      |
| `authentication:password`                   | **Description:**<br/>The password used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided.<br/><br/>**Default**: `"changeme"`                                                                                                                                                                                     |
| `authentication:apiKey`                     | **Description:**<br/>The API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to ApiKey.<br/><br/>**Default**: `""`                                                                                                                                                                                                                    |
| `authentication:base64ApiKey`               | **Description:**<br/>The Base64 Encoded API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to Token or Base64ApiKey.<br/><br/>**Default**: `""`                                                                                                                                                                                      |
| `authentication:clientCertificate:rawData`  | **Description:**<br/>Base64 encoded x509 client certificate for Mutual TLS authentication with Elasticsearch.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                           |
| `authentication:clientCertificate:password` | **Description:**<br/>The password for the client certificate, if required.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                                                              |
| `authentication:rootCertificate:rawData`    | **Description:**<br/>Base64 encoded x509 root certificate for TLS authentication with Elasticsearch.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                                    |
| `authentication:rootCertificate:password`   | **Description:**<br/>The password for the root certificate, if required.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                                                                |

### Document ID

| Name                                        | Details                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `documentId:source`                         | **Description:**<br/>The attribute used to generate the document id.<br/><br/>**Default**: `"recordId"`<br/><br/>**Accepted Values:** `"recordId"`, `"stream"`, `"headers"` or `"streamSuffix"`.                                                                                                                                                                                     |
| `documentId:expression`                     | **Description:**<br/>The expression used to format the document id based on the selected source. This allows for custom id generation logic.<br/><br/>**Default**: `""`                                                                                                                                                                                                                               |

### Resilience

| Name                                        | Details                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `resilience:enabled`                        | **Description:**<br/>Enables resilience mechanisms for handling connection failures and retries.<br/><br/>**Default**: `"true"`                                                                                                                                                                                                                                                                       |
| `resilience:connectionLimit`                | **Description:**<br/>The maximum number of concurrent connections to Elasticsearch.<br/><br/>**Default**: `"80"`                                                                                                                                                                                                                                                             |
| `resilience:maxRetries`                     | **Description:**<br/>The maximum number of retries for a request.<br/><br/>**Default**: `"2147483647"`                                                                                                                                                                                                                                                                       |
| `resilience:maxRetryTimeout`                | **Description:**<br/>The maximum timeout in milliseconds for retries.<br/><br/>**Default**: `"60"`                                                                                                                                                                                                                                                                              |
| `resilience:requestTimeout`                 | **Description:**<br/>The timeout in milliseconds for a request.<br/><br/>**Default**: `"60000"`                                                                                                                                                                                                                                                                                 |
| `resilience:dnsRefreshTimeout`              | **Description:**<br/>The time in milliseconds after which to refresh the DNS information.<br/><br/>**Default**: `"300000"`                                                                                                                                                                                                                                                      |
| `resilience:pingTimeout`                    | **Description:**<br/>The timeout in milliseconds for a ping request.<br/><br/>**Default**: `"120000"`                                                                                                                                                                                                                                                                           |
| `resilience:retryOnErrorTypes`              | **Description:**<br/>Comma-separated list of error types to retry on.<br/><br/>**Default**: `""`                                                                                                                                                                                                                                                                                                      |

Two more keys tune how long a node stays marked dead: `resilience:deadTimeout` and
`resilience:maxDeadTimeout`. Both are unset by default, which leaves the Elasticsearch client's own
defaults in place.

The Elasticsearch sink also uses the shared
[resilience settings](/kurrent-surge/client/connectors/sinks/#resilience-settings): the three delay
bounds that govern how it escalates between retries.

## Metadata

The Elasticsearch sink connector automatically includes these [default headers](/kurrent-surge/client/connectors/sinks/#headers) in each document sent to Elasticsearch. The connector stores them in a `_metadata` field. You can exclude that field from indexing through your Elasticsearch index configuration.

## Examples

### Document ID strategy

The connector can generate the document ID automatically from the source you specify, and from an expression if needed.

By default, the Elasticsearch sink uses the `recordId` as the document ID. This is the unique identifier generated for every record in KurrentDB.

If you do not use the default `recordId`, these options generate document IDs from other sources:

**Set Document ID using Stream ID**

You can extract part of the stream name using a regular expression (regex) to
define the document id. The expression is optional, and you can adapt it to
your naming convention. In this example, the expression captures the stream
name up to `_data`.

```json
{
  "documentId:source": "stream",
  "documentId:expression": "^(.*)_data$"
}
```

Alternatively, if you need only the last segment of the stream name (after a
hyphen), you can use the `streamSuffix` source. This doesn't require an
expression since it automatically extracts the suffix.

```json
{
  "documentId:source": "streamSuffix"
}
```

The `streamSuffix` source is useful when stream names follow a structured
format, and you want to use only the trailing part as the document ID. For
example, if the stream is named `user-123`, the document ID is `123`.

**Set Document ID from Headers**

You can create the document ID by combining values from a record's metadata.

```json
{
  "documentId:source": "headers",
  "documentId:expression": "key1,key2"
}
```

Specify the header keys you want to use in the `documentId:expression` field (e.g., `key1,key2`). The connector concatenates the header values with a hyphen (`-`) to create the document ID.

For example, if your record has headers `key1: regionA` and `key2: zone1`, the document ID is `regionA-zone1`.

### Authentication

The Elasticsearch sink connector supports multiple authentication methods.

#### Basic authentication

To use Basic Authentication, set the method to `basic` and provide the username and password:

```json
{
  "authentication:method": "basic",
  "authentication:username": "elastic",
  "authentication:password": "your_secure_password"
}
```

#### API key authentication

To use API Key authentication, set the method to `apiKey` and provide the API key:

```json
{
  "authentication:method": "apiKey",
  "authentication:apiKey": "your_api_key"
}
```

#### Token authentication

To use Token authentication, set the method to `token` and provide the Base64 encoded API key:

```json
{
  "authentication:method": "token",
  "authentication:base64ApiKey": ".."
}
```

#### Certificate authentication

To configure Mutual TLS authentication, provide the client and root certificates:

```json
{
  "authentication:clientCertificate:rawData": "..",
  "authentication:clientCertificate:password": "..",
  "authentication:rootCertificate:rawData": "..",
  "authentication:rootCertificate:password": ".."
}
```