How Sinks Work
A sink connector moves each record through four stages. This page documents them in order, starting with the shape of the record itself.
Everything on this page applies to sinks only. A source takes instanceTypeName and
logging:enabled, and nothing else described here. See
How Sources Work.
Settings
Section titled “Settings”Every sink takes these two keys. The other shared keys sit next to the section that explains them: subscription under Filters, transformer under Transformations, auto-commit under Checkpointing, and resilience under Resilience. Keys that belong to one connector are on that connector’s own page.
| Name | Details |
|---|---|
instanceTypeName |
required Description: The type of connector instance. Accepted Values:
|
logging:enabled |
Description: Enables or disables logging. Default: "true" |
KurrentDB Record
Section titled “KurrentDB Record”When a sink connector consumes records from KurrentDB, each record has the following structure:
{ "recordId": "string", "position": { "streamId": "string", "partitionId": "number", "logPosition": "number" }, "sequenceId": "number", "isRedacted": "boolean", "schemaInfo": { "subject": "string", "type": "number" }, "headers": { "key1": "string" }, "value": "object"}Use this schema as a reference when you write filters or transformation functions. Key components:
value: Contains the record dataschemaInfo: Contains the event type and the subjectstreamId: Identifies the stream the record came fromheaders: Contains additional metadata, both internal properties and custom properties
Headers
Section titled “Headers”When sink connectors process records, they inject internal headers alongside any custom headers from your records. These headers carry additional metadata about the record. The esdb- prefix distinguishes them from user-defined headers.
The connector adds the following internal headers to each record:
| Header | Sent by | Description |
|---|---|---|
esdb-connector-id |
Every sink | The unique identifier of the connector processing the record |
esdb-record-id |
Every sink | The unique identifier of the record |
esdb-record-timestamp |
Every sink | The timestamp of the record |
esdb-record-is-redacted |
Every sink | A boolean value that indicates if the record has been redacted |
esdb-record-stream-id |
Every sink | The stream ID where the record originated |
esdb-record-stream-revision |
Every sink | The revision number of the record in its stream |
esdb-record-log-position |
Every sink | The global log position of the record |
esdb-record-schema-subject |
Every sink | The schema subject/name of the record |
esdb-record-schema-type |
Every sink | The schema data format type |
esdb-request-id |
HTTP sink | A unique identifier for the processing request |
esdb-request-date |
HTTP sink | The timestamp when the request was processed |
User Defined Headers
Section titled “User Defined Headers”If the connector sends headers to the sink, it prefixes user-defined headers
from your records with esdb-record-headers-, followed by the header name.
By default, the connector treats headers prefixed with $ as system headers and excludes them from the output. The "headers:ignoreSystem" configuration setting changes that:
{ "headers:ignoreSystem": "true"}When set to "false", the connector output includes system headers, those prefixed with $.
Filters
Section titled “Filters”All sink connectors filter records using regular expressions, JsonPath expressions, or prefixes. The connector checks the expression first as a JsonPath, then as a regex. If it is neither, the connector uses it as a prefix.
Subscription Settings
Section titled “Subscription Settings”| Name | Details |
|---|---|
subscription:filter:scope |
Description: The scope of record filtering. Use "stream" to filter by stream ID, or "record" to filter by event type.Accepted Values: - "stream", "record"Default: unspecified |
subscription:filter:filterType |
Description: The method used to filter records. Accepted Values: - "streamId", "regex", "prefix", "jsonPath"Default: unspecified |
subscription:filter:expression |
Description: A filter expression (regex, JsonPath, or prefix). If scope is set and the expression is empty, the connector consumes $all including system events. If scope is not set, it consumes $all excluding system events.Default: "" |
subscription:initialPosition |
Description: Where a connector starts reading when it has no checkpoint. Accepted Values: - "latest", "earliest"Default: "latest" |
Stream ID Filter
Section titled “Stream ID Filter”The stream ID filter matches records by their stream ID. It applies at the stream scope. For example:
{ "subscription:filter:scope": "stream", "subscription:filter:filterType": "streamId", "subscription:filter:expression": "some-stream"}The filter matches only records on the some-stream stream.
Regex Filters
Section titled “Regex Filters”Regular expressions are the simplest and fastest way to filter records. A regex filter applies at the stream scope, which matches the stream ID, or at the record scope, which matches the event type.
An example of a Regex expression:
{ "subscription:filter:scope": "record", "subscription:filter:filterType": "regex", "subscription:filter:expression": "^eventType.*"}The filter matches only records whose event type starts with eventType. This
filter type works at both the stream and record scopes.
Prefix Filters
Section titled “Prefix Filters”You can also filter records by prefix using the following configuration:
{ "subscription:filter:scope": "stream", "subscription:filter:filterType": "prefix", "subscription:filter:expression": "prefix1,prefix2"}The filter matches only records whose stream ID starts with prefix1 or
prefix2. This filter type works at both the stream and record scopes.
JsonPath Filters
Section titled “JsonPath Filters”JSONPath is a standardized string syntax for selecting and extracting JSON values from KurrentDB records. It follows the RFC 9535 standard. The connector applies JSONPath filters itself, and only at the record scope. They match only records with the application/json content type.
An example of a JsonPath filter:
{ "subscription:filter:scope": "record", "subscription:filter:filterType": "jsonPath", "subscription:filter:expression": "$[?($.value.vehicle.year==2018)]"}The filter matches only records where the year field equals 2018.
Transformations
Section titled “Transformations”Sink connectors support transformations written in JavaScript. A transformation modifies a record from a KurrentDB stream before the connector sends it to the destination. Use one to combine fields, convert values, or enrich records.
Transformer Settings
Section titled “Transformer Settings”| Name | Details |
|---|---|
transformer:enabled |
Description: Enables or disables the transformer. Default: "false" |
transformer:function |
required when enabled Description: Base64 encoded JavaScript function for transforming records. Default: "" |
How Transformations Work
Section titled “How Transformations Work”A transformation is a JavaScript function, encoded in base64 and configured on the connector. The function receives a KurrentDB record that follows the record structure. It can modify any part of that record before the sink processes it.
Creating a Transformation
Section titled “Creating a Transformation”Follow these steps to create and apply a transformation to your connector:
1. Write the Transformation Function in JavaScript
Section titled “1. Write the Transformation Function in JavaScript”The transformation function must meet the following requirements:
- Function name: Must be named
transform - Parameters: Must accept a single parameter called
recordthat follows the record structure - Mutation style: Mutate the
recordobject passed as the argument. The connector also accepts a returned object and uses it in place of the record, but mutating in place is the supported style.
Here’s an example that combines first and last names into a full name and normalizes the email address:
function transform(record) { const { firstName, lastName, email } = record.value; record.value.fullName = `${firstName} ${lastName}`; record.value.email = email.toLowerCase();}Input:
This function assumes the record.value is a JSON object with the following structure:
{ "firstName": "John", "lastName": "Doe", "email": "John.Doe@example.com"}Output:
After transformation, the record contains:
{ "email": "john.doe@example.com", "firstName": "John", "fullName": "John Doe", "lastName": "Doe"}2. Encode the Transformation Function to Base64
Section titled “2. Encode the Transformation Function to Base64”Convert your JavaScript function to a base64-encoded string. You must encode the transformation before you configure it on the connector.
3. Configure the Connector With the Encoded Transformation
Section titled “3. Configure the Connector With the Encoded Transformation”Two settings turn the transformer on and supply the function:
{ "transformer:enabled": "true", "transformer:function": "ZnVuY3Rpb24gdHJhbnNmb3JtKHJlY29yZCkgewogIGNvbnN0IHsgZmlyc3ROYW1lLCBsYXN0TmFtZSwgZW1haWwgfSA9IHJlY29yZC52YWx1ZTsKICByZWNvcmQudmFsdWUuZnVsbE5hbWUgPSBgJHtmaXJzdE5hbWV9ICR7bGFzdE5hbWV9YDsKICByZWNvcmQudmFsdWUuZW1haWwgPSBlbWFpbC50b0xvd2VyQ2FzZSgpOwp9"}Set them when you create the connector. To add them to a connector that already exists, use
Reconfigure, which replaces the settings wholesale — send the connector’s existing settings
alongside these two, or the rest are lost. See
Managing Connectors.
4. Start the Connector
Section titled “4. Start the Connector”Once configured, start the connector to process records with the transformation applied:
await client.Connectors.Start("example-connector");When you append a record to the target KurrentDB stream, the connector applies the transformation function before sending the record to the destination.
Checkpointing
Section titled “Checkpointing”Connectors periodically store the position of the last record they processed
successfully. If the connector host restarts, each connector continues from
close to where it got to. KurrentDB stores the checkpoint in the
$connectors/{connector-id}/checkpoints system stream. Each connector has its
own dedicated stream for storing checkpoints.
By default, a connector with no checkpoints starts from the latest position in
the stream. The initialPosition property in the settings changes that
position. To start from a specific position, use Start and Stop with the
position parameter.
Auto-commit Settings
Section titled “Auto-commit Settings”The connector commits a checkpoint on an interval, on a record count, or on both.
| Name | Details |
|---|---|
autocommit:enabled |
Description: Enables or disables auto-commit. Default: "true" |
autocommit:interval |
Description: The interval, in milliseconds, at which auto-commit occurs. Default: "5000" |
autocommit:recordsThreshold |
Description: The number of records that triggers an auto-commit. Default: "1000" |
Resilience
Section titled “Resilience”Connectors currently support only at least once delivery guarantees, so a
record may be delivered more than once. Records are delivered in order: record
x is not delivered until every preceding record has been delivered.
Each sink declares its own retry strategy. There is no shared default, so what happens after a failed write depends on the connector you chose.
Five sinks escalate through three phases with identical defaults: Kafka, HTTP, Elasticsearch, MongoDB, and SQL. Each of them adds keys of its own, so check that connector’s page too. Pulsar retries at one fixed interval instead, and RabbitMQ does not retry at all.
The three phases escalate like this:
Automatic Retries and Resilience Strategy
Section titled “Automatic Retries and Resilience Strategy”These five sinks retry a failed write automatically. Retries run at three fixed delays, and an elapsed-time limit bounds each one. The delay never doubles between attempts. All five sinks share the default values below.
The retry process unfolds in three phases:
-
Phase 1 – Rapid Retries: Right after a failure, the connector retries every 5 seconds for up to 1 minute. This phase covers minor issues that resolve on their own.
-
Phase 2 – Moderate Delays: If the failure continues, retries slow to every 10 minutes for the next 1 hour. The wider spacing eases the strain on both systems.
-
Phase 3 – Extended Delays: If the failure persists, the final phase retries every 1 hour indefinitely. This phase reduces unnecessary attempts, though manual intervention may eventually be required.
Each phase retries less often than the one before it.
Resilience Settings
Section titled “Resilience Settings”| Name | Details |
|---|---|
resilience:enabled |
Description: Enables or disables resilience. Default: "true" |
resilience:firstDelayBound:upperLimitMs |
Description: The upper limit for the first delay bound, in milliseconds. Default: "60000" |
resilience:firstDelayBound:delayMs |
Description: The delay for the first bound, in milliseconds. Default: "5000" |
resilience:secondDelayBound:upperLimitMs |
Description: The upper limit for the second delay bound, in milliseconds. Default: "3600000" |
resilience:secondDelayBound:delayMs |
Description: The delay for the second bound, in milliseconds. Default: "600000" |
resilience:thirdDelayBound:upperLimitMs |
Description: The upper limit for the third delay bound, in milliseconds. "-1" means forever.Default: "-1" |
resilience:thirdDelayBound:delayMs |
Description: The delay for the third bound, in milliseconds. Default: "3600000" |