Skip to content

Elasticsearch Sink

License Required

The Elasticsearch sink reads records from a KurrentDB stream and stores them in an Elasticsearch index. The connector serializes each record as a JSON document compatible with Elasticsearch’s document structure.

Before using the Elasticsearch sink connector, ensure you have:

  • A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords and API keys)
  • An Elasticsearch cluster with an accessible URL
  • Appropriate authentication credentials (basic auth, token, or API key)

Create the Elasticsearch sink with client.Connectors. Replace example-elastic with your own connector id:

Dictionary<string, string> settings = new() {
["instanceTypeName"] = "elasticsearch-sink",
["url"] = "http://localhost:9200",
["indexName"] = "sample-index",
["subscription:filter:scope"] = "stream",
["subscription:filter:filterType"] = "streamId",
["subscription:filter:expression"] = "example-stream"
};
await client.Connectors.Create("example-elastic", settings);
await client.Connectors.Start("example-elastic");

Once started, the connector writes each record appended to example-stream into the index. Managing Connectors covers the rest of the lifecycle.

Adjust these settings to control how the Elasticsearch sink connector behaves and how it interacts with KurrentDB.

The Elasticsearch sink can be configured with the following options:

Name Details
url required

Description:
The URL of the Elasticsearch cluster to which the connector connects.

Default: "http://localhost:9200"
indexName required

Description:
The index name that the connector writes records to.

Default: ""
refresh Description:
Specifies whether Elasticsearch should refresh the affected shards to make this operation visible to search. If set to "true", Elasticsearch refreshes the affected shards. If set to "wait_for", Elasticsearch waits for a refresh to make this operation visible to search. If set to "false", Elasticsearch does nothing with refreshes.

Default: "true"
batching:batchSize Description:
Threshold batch size at which the sink pushes the batch of records to the Elasticsearch index.

Default: "1000"
batching:batchTimeoutMs Description:
Threshold time in milliseconds at which the sink pushes the current batch of records to the Elasticsearch index, regardless of the batch size.

Default: "250"
Name Details
authentication:method Description:
The authentication method used by the connector to connect to the Elasticsearch cluster.

Default: "basic"

Accepted Values: "basic", "token", "apiKey"
authentication:username Description:
The username used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided.

Default: "elastic"
authentication:password Description:
The password used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided.

Default: "changeme"
authentication:apiKey Description:
The API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to ApiKey.

Default: ""
authentication:base64ApiKey Description:
The Base64 Encoded API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to Token or Base64ApiKey.

Default: ""
authentication:clientCertificate:rawData Description:
Base64 encoded x509 client certificate for Mutual TLS authentication with Elasticsearch.

Default: ""
authentication:clientCertificate:password Description:
The password for the client certificate, if required.

Default: ""
authentication:rootCertificate:rawData Description:
Base64 encoded x509 root certificate for TLS authentication with Elasticsearch.

Default: ""
authentication:rootCertificate:password Description:
The password for the root certificate, if required.

Default: ""
Name Details
documentId:source Description:
The attribute used to generate the document id.

Default: "recordId"

Accepted Values: "recordId", "stream", "headers" or "streamSuffix".
documentId:expression Description:
The expression used to format the document id based on the selected source. This allows for custom id generation logic.

Default: ""
Name Details
resilience:enabled Description:
Enables resilience mechanisms for handling connection failures and retries.

Default: "true"
resilience:connectionLimit Description:
The maximum number of concurrent connections to Elasticsearch.

Default: "80"
resilience:maxRetries Description:
The maximum number of retries for a request.

Default: "2147483647"
resilience:maxRetryTimeout Description:
The maximum timeout in milliseconds for retries.

Default: "60"
resilience:requestTimeout Description:
The timeout in milliseconds for a request.

Default: "60000"
resilience:dnsRefreshTimeout Description:
The time in milliseconds after which to refresh the DNS information.

Default: "300000"
resilience:pingTimeout Description:
The timeout in milliseconds for a ping request.

Default: "120000"
resilience:retryOnErrorTypes Description:
Comma-separated list of error types to retry on.

Default: ""

Two more keys tune how long a node stays marked dead: resilience:deadTimeout and resilience:maxDeadTimeout. Both are unset by default, which leaves the Elasticsearch client’s own defaults in place.

The Elasticsearch sink also uses the shared resilience settings: the three delay bounds that govern how it escalates between retries.

The Elasticsearch sink connector automatically includes these default headers in each document sent to Elasticsearch. The connector stores them in a _metadata field. You can exclude that field from indexing through your Elasticsearch index configuration.

The connector can generate the document ID automatically from the source you specify, and from an expression if needed.

By default, the Elasticsearch sink uses the recordId as the document ID. This is the unique identifier generated for every record in KurrentDB.

If you do not use the default recordId, these options generate document IDs from other sources:

Set Document ID using Stream ID

You can extract part of the stream name using a regular expression (regex) to define the document id. The expression is optional, and you can adapt it to your naming convention. In this example, the expression captures the stream name up to _data.

{
"documentId:source": "stream",
"documentId:expression": "^(.*)_data$"
}

Alternatively, if you need only the last segment of the stream name (after a hyphen), you can use the streamSuffix source. This doesn’t require an expression since it automatically extracts the suffix.

{
"documentId:source": "streamSuffix"
}

The streamSuffix source is useful when stream names follow a structured format, and you want to use only the trailing part as the document ID. For example, if the stream is named user-123, the document ID is 123.

Set Document ID from Headers

You can create the document ID by combining values from a record’s metadata.

{
"documentId:source": "headers",
"documentId:expression": "key1,key2"
}

Specify the header keys you want to use in the documentId:expression field (e.g., key1,key2). The connector concatenates the header values with a hyphen (-) to create the document ID.

For example, if your record has headers key1: regionA and key2: zone1, the document ID is regionA-zone1.

The Elasticsearch sink connector supports multiple authentication methods.

To use Basic Authentication, set the method to basic and provide the username and password:

{
"authentication:method": "basic",
"authentication:username": "elastic",
"authentication:password": "your_secure_password"
}

To use API Key authentication, set the method to apiKey and provide the API key:

{
"authentication:method": "apiKey",
"authentication:apiKey": "your_api_key"
}

To use Token authentication, set the method to token and provide the Base64 encoded API key:

{
"authentication:method": "token",
"authentication:base64ApiKey": ".."
}

To configure Mutual TLS authentication, provide the client and root certificates:

{
"authentication:clientCertificate:rawData": "..",
"authentication:clientCertificate:password": "..",
"authentication:rootCertificate:rawData": "..",
"authentication:rootCertificate:password": ".."
}