Skip to content

How Sinks Work

A sink connector moves each record through four stages. This page documents them in order, starting with the shape of the record itself.

Everything on this page applies to sinks only. A source takes instanceTypeName and logging:enabled, and nothing else described here. See How Sources Work.

DiagramKurrentDBTargetStreamsCheckpointsSink connectorSubscriptionFilterTransformSink records positionwrites

Every sink takes these two keys. The other shared keys sit next to the section that explains them: subscription under Filters, transformer under Transformations, auto-commit under Checkpointing, and resilience under Resilience. Keys that belong to one connector are on that connector’s own page.

Name Details
instanceTypeName required

Description:
The type of connector instance.

Accepted Values:
  • elasticsearch-sink
  • http-sink
  • kafka-sink
  • mongo-db-sink
  • pulsar-sink
  • rabbit-mq-sink
  • serilog-sink
  • sql-sink
logging:enabled Description:
Enables or disables logging.

Default: "true"

When a sink connector consumes records from KurrentDB, each record has the following structure:

{
"recordId": "string",
"position": {
"streamId": "string",
"partitionId": "number",
"logPosition": "number"
},
"sequenceId": "number",
"isRedacted": "boolean",
"schemaInfo": {
"subject": "string",
"type": "number"
},
"headers": {
"key1": "string"
},
"value": "object"
}

Use this schema as a reference when you write filters or transformation functions. Key components:

  • value: Contains the record data
  • schemaInfo: Contains the event type and the subject
  • streamId: Identifies the stream the record came from
  • headers: Contains additional metadata, both internal properties and custom properties

When sink connectors process records, they inject internal headers alongside any custom headers from your records. These headers carry additional metadata about the record. The esdb- prefix distinguishes them from user-defined headers.

The connector adds the following internal headers to each record:

Header Sent by Description
esdb-connector-id Every sink The unique identifier of the connector processing the record
esdb-record-id Every sink The unique identifier of the record
esdb-record-timestamp Every sink The timestamp of the record
esdb-record-is-redacted Every sink A boolean value that indicates if the record has been redacted
esdb-record-stream-id Every sink The stream ID where the record originated
esdb-record-stream-revision Every sink The revision number of the record in its stream
esdb-record-log-position Every sink The global log position of the record
esdb-record-schema-subject Every sink The schema subject/name of the record
esdb-record-schema-type Every sink The schema data format type
esdb-request-id HTTP sink A unique identifier for the processing request
esdb-request-date HTTP sink The timestamp when the request was processed

If the connector sends headers to the sink, it prefixes user-defined headers from your records with esdb-record-headers-, followed by the header name.

By default, the connector treats headers prefixed with $ as system headers and excludes them from the output. The "headers:ignoreSystem" configuration setting changes that:

{
"headers:ignoreSystem": "true"
}

When set to "false", the connector output includes system headers, those prefixed with $.

All sink connectors filter records using regular expressions, JsonPath expressions, or prefixes. The connector checks the expression first as a JsonPath, then as a regex. If it is neither, the connector uses it as a prefix.

Name Details
subscription:filter:scope Description:
The scope of record filtering. Use "stream" to filter by stream ID, or "record" to filter by event type.

Accepted Values:
- "stream", "record"

Default: unspecified
subscription:filter:filterType Description:
The method used to filter records.

Accepted Values:
- "streamId", "regex", "prefix", "jsonPath"

Default: unspecified
subscription:filter:expression Description:
A filter expression (regex, JsonPath, or prefix). If scope is set and the expression is empty, the connector consumes $all including system events. If scope is not set, it consumes $all excluding system events.

Default: ""
subscription:initialPosition Description:
Where a connector starts reading when it has no checkpoint.

Accepted Values:
- "latest", "earliest"

Default: "latest"

The stream ID filter matches records by their stream ID. It applies at the stream scope. For example:

{
"subscription:filter:scope": "stream",
"subscription:filter:filterType": "streamId",
"subscription:filter:expression": "some-stream"
}

The filter matches only records on the some-stream stream.

Regular expressions are the simplest and fastest way to filter records. A regex filter applies at the stream scope, which matches the stream ID, or at the record scope, which matches the event type.

An example of a Regex expression:

{
"subscription:filter:scope": "record",
"subscription:filter:filterType": "regex",
"subscription:filter:expression": "^eventType.*"
}

The filter matches only records whose event type starts with eventType. This filter type works at both the stream and record scopes.

You can also filter records by prefix using the following configuration:

{
"subscription:filter:scope": "stream",
"subscription:filter:filterType": "prefix",
"subscription:filter:expression": "prefix1,prefix2"
}

The filter matches only records whose stream ID starts with prefix1 or prefix2. This filter type works at both the stream and record scopes.

JSONPath is a standardized string syntax for selecting and extracting JSON values from KurrentDB records. It follows the RFC 9535 standard. The connector applies JSONPath filters itself, and only at the record scope. They match only records with the application/json content type.

An example of a JsonPath filter:

{
"subscription:filter:scope": "record",
"subscription:filter:filterType": "jsonPath",
"subscription:filter:expression": "$[?($.value.vehicle.year==2018)]"
}

The filter matches only records where the year field equals 2018.

Sink connectors support transformations written in JavaScript. A transformation modifies a record from a KurrentDB stream before the connector sends it to the destination. Use one to combine fields, convert values, or enrich records.

Name Details
transformer:enabled Description:
Enables or disables the transformer.

Default: "false"
transformer:function required when enabled

Description:
Base64 encoded JavaScript function for transforming records.

Default: ""

A transformation is a JavaScript function, encoded in base64 and configured on the connector. The function receives a KurrentDB record that follows the record structure. It can modify any part of that record before the sink processes it.

Follow these steps to create and apply a transformation to your connector:

1. Write the Transformation Function in JavaScript

Section titled “1. Write the Transformation Function in JavaScript”

The transformation function must meet the following requirements:

  • Function name: Must be named transform
  • Parameters: Must accept a single parameter called record that follows the record structure
  • Mutation style: Mutate the record object passed as the argument. The connector also accepts a returned object and uses it in place of the record, but mutating in place is the supported style.

Here’s an example that combines first and last names into a full name and normalizes the email address:

function transform(record) {
const { firstName, lastName, email } = record.value;
record.value.fullName = `${firstName} ${lastName}`;
record.value.email = email.toLowerCase();
}

Input:

This function assumes the record.value is a JSON object with the following structure:

{
"firstName": "John",
"lastName": "Doe",
"email": "John.Doe@example.com"
}

Output:

After transformation, the record contains:

{
"email": "john.doe@example.com",
"firstName": "John",
"fullName": "John Doe",
"lastName": "Doe"
}

2. Encode the Transformation Function to Base64

Section titled “2. Encode the Transformation Function to Base64”

Convert your JavaScript function to a base64-encoded string. You must encode the transformation before you configure it on the connector.

3. Configure the Connector With the Encoded Transformation

Section titled “3. Configure the Connector With the Encoded Transformation”

Two settings turn the transformer on and supply the function:

{
"transformer:enabled": "true",
"transformer:function": "ZnVuY3Rpb24gdHJhbnNmb3JtKHJlY29yZCkgewogIGNvbnN0IHsgZmlyc3ROYW1lLCBsYXN0TmFtZSwgZW1haWwgfSA9IHJlY29yZC52YWx1ZTsKICByZWNvcmQudmFsdWUuZnVsbE5hbWUgPSBgJHtmaXJzdE5hbWV9ICR7bGFzdE5hbWV9YDsKICByZWNvcmQudmFsdWUuZW1haWwgPSBlbWFpbC50b0xvd2VyQ2FzZSgpOwp9"
}

Set them when you create the connector. To add them to a connector that already exists, use Reconfigure, which replaces the settings wholesale — send the connector’s existing settings alongside these two, or the rest are lost. See Managing Connectors.

Once configured, start the connector to process records with the transformation applied:

await client.Connectors.Start("example-connector");

When you append a record to the target KurrentDB stream, the connector applies the transformation function before sending the record to the destination.

Connectors periodically store the position of the last record they processed successfully. If the connector host restarts, each connector continues from close to where it got to. KurrentDB stores the checkpoint in the $connectors/{connector-id}/checkpoints system stream. Each connector has its own dedicated stream for storing checkpoints.

By default, a connector with no checkpoints starts from the latest position in the stream. The initialPosition property in the settings changes that position. To start from a specific position, use Start and Stop with the position parameter.

The connector commits a checkpoint on an interval, on a record count, or on both.

Name Details
autocommit:enabled Description:
Enables or disables auto-commit.

Default: "true"
autocommit:interval Description:
The interval, in milliseconds, at which auto-commit occurs.

Default: "5000"
autocommit:recordsThreshold Description:
The number of records that triggers an auto-commit.

Default: "1000"

Connectors currently support only at least once delivery guarantees, so a record may be delivered more than once. Records are delivered in order: record x is not delivered until every preceding record has been delivered.

Each sink declares its own retry strategy. There is no shared default, so what happens after a failed write depends on the connector you chose.

Five sinks escalate through three phases with identical defaults: Kafka, HTTP, Elasticsearch, MongoDB, and SQL. Each of them adds keys of its own, so check that connector’s page too. Pulsar retries at one fixed interval instead, and RabbitMQ does not retry at all.

The three phases escalate like this:

DiagramPhase 1 · every 5sPhase 2 · every 10mPhase 3 · every hour after 1 minafter 1 hourforever

These five sinks retry a failed write automatically. Retries run at three fixed delays, and an elapsed-time limit bounds each one. The delay never doubles between attempts. All five sinks share the default values below.

The retry process unfolds in three phases:

  1. Phase 1 – Rapid Retries: Right after a failure, the connector retries every 5 seconds for up to 1 minute. This phase covers minor issues that resolve on their own.

  2. Phase 2 – Moderate Delays: If the failure continues, retries slow to every 10 minutes for the next 1 hour. The wider spacing eases the strain on both systems.

  3. Phase 3 – Extended Delays: If the failure persists, the final phase retries every 1 hour indefinitely. This phase reduces unnecessary attempts, though manual intervention may eventually be required.

Each phase retries less often than the one before it.

Name Details
resilience:enabled Description:
Enables or disables resilience.

Default: "true"
resilience:firstDelayBound:upperLimitMs Description:
The upper limit for the first delay bound, in milliseconds.

Default: "60000"
resilience:firstDelayBound:delayMs Description:
The delay for the first bound, in milliseconds.

Default: "5000"
resilience:secondDelayBound:upperLimitMs Description:
The upper limit for the second delay bound, in milliseconds.

Default: "3600000"
resilience:secondDelayBound:delayMs Description:
The delay for the second bound, in milliseconds.

Default: "600000"
resilience:thirdDelayBound:upperLimitMs Description:
The upper limit for the third delay bound, in milliseconds. "-1" means forever.

Default: "-1"
resilience:thirdDelayBound:delayMs Description:
The delay for the third bound, in milliseconds.

Default: "3600000"