MongoDB Sink

The MongoDB sink reads records from a KurrentDB stream and stores them in a collection. The connector serializes each record into a BSON document, so the data must be valid BSON.
Prerequisites
Section titled “Prerequisites”Before using the MongoDB sink connector, ensure you have:
- A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords and certificate passwords)
- A MongoDB instance with an accessible connection string
- Appropriate authentication credentials (username/password or X.509 certificate)
Quickstart
Section titled “Quickstart”Create the MongoDB sink with client.Connectors. Replace example-mongo with your own
connector id:
Dictionary<string, string> settings = new() { ["instanceTypeName"] = "mongo-db-sink", ["connectionString"] = "mongodb://127.0.0.1:27020", ["database"] = "my_database", ["collection"] = "my_collection", ["subscription:filter:scope"] = "stream", ["subscription:filter:filterType"] = "streamId", ["subscription:filter:expression"] = "example-stream"};
await client.Connectors.Create("example-mongo", settings);await client.Connectors.Start("example-mongo");Once started, the connector writes each record appended to example-stream into the
collection. Managing Connectors covers the rest of
the lifecycle.
Settings
Section titled “Settings”Adjust these settings to control how the MongoDB sink connector behaves and how it interacts with KurrentDB.
The MongoDB sink can be configured with the following options:
| Name | Details |
|---|---|
database |
required Description: The name of the database where the connector stores the records. |
collection |
required Description: The collection name that resides in the database to push records to. |
connectionString |
required Description: The MongoDB URI to which the connector connects. Default: "mongodb://mongoadmin:secret@localhost:27017/admin" |
documentId:source |
Description: The attribute used to generate the document id. Default: "recordId"Accepted Values: - "recordId", "stream", "headers", or "streamSuffix" |
documentId:expression |
Description: The expression used to format the document id based on the selected source. This allows for custom id generation logic. Default: "" |
certificate:rawData |
Description: Base64 encoded x509 certificate. Default: “” |
certificate:password |
Description: The password used to access the x509 certificate for secure connections. Default: “” |
batching:batchSize |
Description: Threshold batch size at which the sink pushes the batch of records to the MongoDB collection. Default: "1000" |
batching:batchTimeoutMs |
Description: Threshold time in milliseconds at which the sink pushes the current batch of records to the MongoDB collection, regardless of the batch size. Default: "250" |
Resilience options can be found in the Resilience Configuration section.
Metadata
Section titled “Metadata”The MongoDB sink connector automatically includes these default headers in each document sent to the collection. They
provide metadata about the record, and the connector stores them in a
_metadata field within the document.
Examples
Section titled “Examples”Authentication
Section titled “Authentication”This MongoDB sink connector currently supports only SCRAM and X.509 certificate authentication.
To use SCRAM for authentication, include the username and password in the
connection string. Set the authMechanism parameter in that same connection
string to either SCRAM-SHA-1 or SCRAM-SHA-256 to select the MongoDB
authentication mechanism. For more on the connection string URI, see the
official MongoDB documentation on Authentication Mechanism.
X.509 certificate authentication
Section titled “X.509 certificate authentication”To use X.509 certificate authentication, include the base64 encoded x509 certificate and the password in the settings. You can use an online tool like base64encode to encode your certificate.
{ "certificate:rawData": "base64encodedstring", "certificate:password": "password"}Document ID strategy
Section titled “Document ID strategy”The connector can generate the document ID automatically from the source you specify, and from an expression if needed. The following options are available:
By default, the MongoDB sink connector uses the record’s Id as the document ID. This is the unique identifier generated for every record in KurrentDB.
Set Document ID using Stream ID
You can extract part of the stream name using a regular expression (regex) to
define the document id. The expression is optional, and you can adapt it to
your naming convention. In this example, the expression captures the stream
name up to _data.
{ "documentId:source": "stream", "documentId:expression": "^(.*)_data$"}Alternatively, if you need only the last segment of the stream name (after a
hyphen), you can use the streamSuffix source. This
doesn’t require an expression since it automatically extracts the suffix.
{ "documentId:source": "streamSuffix"}The streamSuffix source is useful when stream names follow a structured
format, and you want to use only the trailing part as the document ID. For
example, if the stream is named user-123, the document ID is 123.
Set Document ID from Headers
You can create the document ID by combining values from a record’s metadata.
{ "documentId:source": "headers", "documentId:expression": "key1,key2"}Specify the header keys you want to use in the documentId:expression field (e.g., key1,key2). The connector concatenates the header values with a hyphen (-) to create the document ID.
For example, if your record has headers key1: regionA and key2: zone1, the document ID is regionA-zone1.
Tutorial
Section titled “Tutorial”Learn how to set up and use a MongoDB Sink connector in KurrentDB through a tutorial.