Skip to content

MongoDB Sink

License Required

The MongoDB sink reads records from a KurrentDB stream and stores them in a collection. The connector serializes each record into a BSON document, so the data must be valid BSON.

Before using the MongoDB sink connector, ensure you have:

  • A data protection token configured in your KurrentDB instance (required to encrypt sensitive fields like passwords and certificate passwords)
  • A MongoDB instance with an accessible connection string
  • Appropriate authentication credentials (username/password or X.509 certificate)

Create the MongoDB sink with client.Connectors. Replace example-mongo with your own connector id:

Dictionary<string, string> settings = new() {
["instanceTypeName"] = "mongo-db-sink",
["connectionString"] = "mongodb://127.0.0.1:27020",
["database"] = "my_database",
["collection"] = "my_collection",
["subscription:filter:scope"] = "stream",
["subscription:filter:filterType"] = "streamId",
["subscription:filter:expression"] = "example-stream"
};
await client.Connectors.Create("example-mongo", settings);
await client.Connectors.Start("example-mongo");

Once started, the connector writes each record appended to example-stream into the collection. Managing Connectors covers the rest of the lifecycle.

Adjust these settings to control how the MongoDB sink connector behaves and how it interacts with KurrentDB.

The MongoDB sink can be configured with the following options:

Name Details
database required

Description:
The name of the database where the connector stores the records.
collection required

Description:
The collection name that resides in the database to push records to.
connectionString required

Description:
The MongoDB URI to which the connector connects.

Default: "mongodb://mongoadmin:secret@localhost:27017/admin"
documentId:source Description:
The attribute used to generate the document id.

Default: "recordId"

Accepted Values:
- "recordId", "stream", "headers", or "streamSuffix"
documentId:expression Description:
The expression used to format the document id based on the selected source. This allows for custom id generation logic.

Default: ""
certificate:rawData Description:
Base64 encoded x509 certificate.

Default: “”
certificate:password Description:
The password used to access the x509 certificate for secure connections.

Default: “”
batching:batchSize Description:
Threshold batch size at which the sink pushes the batch of records to the MongoDB collection.

Default: "1000"
batching:batchTimeoutMs Description:
Threshold time in milliseconds at which the sink pushes the current batch of records to the MongoDB collection, regardless of the batch size.

Default: "250"

Resilience options can be found in the Resilience Configuration section.

The MongoDB sink connector automatically includes these default headers in each document sent to the collection. They provide metadata about the record, and the connector stores them in a _metadata field within the document.

This MongoDB sink connector currently supports only SCRAM and X.509 certificate authentication.

To use SCRAM for authentication, include the username and password in the connection string. Set the authMechanism parameter in that same connection string to either SCRAM-SHA-1 or SCRAM-SHA-256 to select the MongoDB authentication mechanism. For more on the connection string URI, see the official MongoDB documentation on Authentication Mechanism.

To use X.509 certificate authentication, include the base64 encoded x509 certificate and the password in the settings. You can use an online tool like base64encode to encode your certificate.

{
"certificate:rawData": "base64encodedstring",
"certificate:password": "password"
}

The connector can generate the document ID automatically from the source you specify, and from an expression if needed. The following options are available:

By default, the MongoDB sink connector uses the record’s Id as the document ID. This is the unique identifier generated for every record in KurrentDB.

Set Document ID using Stream ID

You can extract part of the stream name using a regular expression (regex) to define the document id. The expression is optional, and you can adapt it to your naming convention. In this example, the expression captures the stream name up to _data.

{
"documentId:source": "stream",
"documentId:expression": "^(.*)_data$"
}

Alternatively, if you need only the last segment of the stream name (after a hyphen), you can use the streamSuffix source. This doesn’t require an expression since it automatically extracts the suffix.

{
"documentId:source": "streamSuffix"
}

The streamSuffix source is useful when stream names follow a structured format, and you want to use only the trailing part as the document ID. For example, if the stream is named user-123, the document ID is 123.

Set Document ID from Headers

You can create the document ID by combining values from a record’s metadata.

{
"documentId:source": "headers",
"documentId:expression": "key1,key2"
}

Specify the header keys you want to use in the documentId:expression field (e.g., key1,key2). The connector concatenates the header values with a hyphen (-) to create the document ID.

For example, if your record has headers key1: regionA and key2: zone1, the document ID is regionA-zone1.

Learn how to set up and use a MongoDB Sink connector in KurrentDB through a tutorial.