Elasticsearch Sink

The Elasticsearch sink reads records from a KurrentDB stream and stores them in an Elasticsearch index. The connector serializes each record as a JSON document compatible with Elasticsearch’s document structure.
Prerequisites
Section titled “Prerequisites”Before using the Elasticsearch sink connector, ensure you have:
- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords and API keys)
- An Elasticsearch cluster with an accessible URL
- Appropriate authentication credentials (basic auth, token, or API key)
Quickstart
Section titled “Quickstart”Create the Elasticsearch sink with client.Connectors. Replace example-elastic with your own
connector id:
Dictionary<string, string> settings = new() { ["instanceTypeName"] = "elasticsearch-sink", ["url"] = "http://localhost:9200", ["indexName"] = "sample-index", ["subscription:filter:scope"] = "stream", ["subscription:filter:filterType"] = "streamId", ["subscription:filter:expression"] = "example-stream"};
await client.Connectors.Create("example-elastic", settings);await client.Connectors.Start("example-elastic");Once started, the connector writes each record appended to example-stream into the index.
Managing Connectors covers the rest of the lifecycle.
Settings
Section titled “Settings”Adjust these settings to control how the Elasticsearch sink connector behaves and how it interacts with KurrentDB.
The Elasticsearch sink can be configured with the following options:
| Name | Details |
|---|---|
url |
required Description: The URL of the Elasticsearch cluster to which the connector connects. Default: "http://localhost:9200" |
indexName |
required Description: The index name that the connector writes records to. Default: "" |
refresh |
Description: Specifies whether Elasticsearch should refresh the affected shards to make this operation visible to search. If set to "true", Elasticsearch refreshes the affected shards. If set to "wait_for", Elasticsearch waits for a refresh to make this operation visible to search. If set to "false", Elasticsearch does nothing with refreshes.Default: "true" |
batching:batchSize |
Description: Threshold batch size at which the sink pushes the batch of records to the Elasticsearch index. Default: "1000" |
batching:batchTimeoutMs |
Description: Threshold time in milliseconds at which the sink pushes the current batch of records to the Elasticsearch index, regardless of the batch size. Default: "250" |
Authentication
Section titled “Authentication”| Name | Details |
|---|---|
authentication:method |
Description: The authentication method used by the connector to connect to the Elasticsearch cluster. Default: "basic"Accepted Values: "basic", "token", "apiKey" |
authentication:username |
Description: The username used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided. Default: "elastic" |
authentication:password |
Description: The password used by the connector to connect to the Elasticsearch cluster. If username is set, then password should also be provided. Default: "changeme" |
authentication:apiKey |
Description: The API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to ApiKey. Default: "" |
authentication:base64ApiKey |
Description: The Base64 Encoded API key used by the connector to connect to the Elasticsearch cluster. Used if the method is set to Token or Base64ApiKey. Default: "" |
authentication:clientCertificate:rawData |
Description: Base64 encoded x509 client certificate for Mutual TLS authentication with Elasticsearch. Default: "" |
authentication:clientCertificate:password |
Description: The password for the client certificate, if required. Default: "" |
authentication:rootCertificate:rawData |
Description: Base64 encoded x509 root certificate for TLS authentication with Elasticsearch. Default: "" |
authentication:rootCertificate:password |
Description: The password for the root certificate, if required. Default: "" |
Document ID
Section titled “Document ID”| Name | Details |
|---|---|
documentId:source |
Description: The attribute used to generate the document id. Default: "recordId"Accepted Values: "recordId", "stream", "headers" or "streamSuffix". |
documentId:expression |
Description: The expression used to format the document id based on the selected source. This allows for custom id generation logic. Default: "" |
Resilience
Section titled “Resilience”| Name | Details |
|---|---|
resilience:enabled |
Description: Enables resilience mechanisms for handling connection failures and retries. Default: "true" |
resilience:connectionLimit |
Description: The maximum number of concurrent connections to Elasticsearch. Default: "80" |
resilience:maxRetries |
Description: The maximum number of retries for a request. Default: "2147483647" |
resilience:maxRetryTimeout |
Description: The maximum timeout in milliseconds for retries. Default: "60" |
resilience:requestTimeout |
Description: The timeout in milliseconds for a request. Default: "60000" |
resilience:dnsRefreshTimeout |
Description: The time in milliseconds after which to refresh the DNS information. Default: "300000" |
resilience:pingTimeout |
Description: The timeout in milliseconds for a ping request. Default: "120000" |
resilience:retryOnErrorTypes |
Description: Comma-separated list of error types to retry on. Default: "" |
Two more keys tune how long a node stays marked dead: resilience:deadTimeout and
resilience:maxDeadTimeout. Both are unset by default, which leaves the Elasticsearch client’s own
defaults in place.
The Elasticsearch sink also uses the shared resilience settings: the three delay bounds that govern how it escalates between retries.
Metadata
Section titled “Metadata”The Elasticsearch sink connector automatically includes these default headers in each document sent to Elasticsearch. The connector stores them in a _metadata field. You can exclude that field from indexing through your Elasticsearch index configuration.
Examples
Section titled “Examples”Document ID strategy
Section titled “Document ID strategy”The connector can generate the document ID automatically from the source you specify, and from an expression if needed.
By default, the Elasticsearch sink uses the recordId as the document ID. This is the unique identifier generated for every record in KurrentDB.
If you do not use the default recordId, these options generate document IDs from other sources:
Set Document ID using Stream ID
You can extract part of the stream name using a regular expression (regex) to
define the document id. The expression is optional, and you can adapt it to
your naming convention. In this example, the expression captures the stream
name up to _data.
{ "documentId:source": "stream", "documentId:expression": "^(.*)_data$"}Alternatively, if you need only the last segment of the stream name (after a
hyphen), you can use the streamSuffix source. This doesn’t require an
expression since it automatically extracts the suffix.
{ "documentId:source": "streamSuffix"}The streamSuffix source is useful when stream names follow a structured
format, and you want to use only the trailing part as the document ID. For
example, if the stream is named user-123, the document ID is 123.
Set Document ID from Headers
You can create the document ID by combining values from a record’s metadata.
{ "documentId:source": "headers", "documentId:expression": "key1,key2"}Specify the header keys you want to use in the documentId:expression field (e.g., key1,key2). The connector concatenates the header values with a hyphen (-) to create the document ID.
For example, if your record has headers key1: regionA and key2: zone1, the document ID is regionA-zone1.
Authentication
Section titled “Authentication”The Elasticsearch sink connector supports multiple authentication methods.
Basic authentication
Section titled “Basic authentication”To use Basic Authentication, set the method to basic and provide the username and password:
{ "authentication:method": "basic", "authentication:username": "elastic", "authentication:password": "your_secure_password"}API key authentication
Section titled “API key authentication”To use API Key authentication, set the method to apiKey and provide the API key:
{ "authentication:method": "apiKey", "authentication:apiKey": "your_api_key"}Token authentication
Section titled “Token authentication”To use Token authentication, set the method to token and provide the Base64 encoded API key:
{ "authentication:method": "token", "authentication:base64ApiKey": ".."}Certificate authentication
Section titled “Certificate authentication”To configure Mutual TLS authentication, provide the client and root certificates:
{ "authentication:clientCertificate:rawData": "..", "authentication:clientCertificate:password": "..", "authentication:rootCertificate:rawData": "..", "authentication:rootCertificate:password": ".."}