Running a Processor
A processor runs inside the .NET host that you already use. AddKurrentSurge registers the
processor as a hosted service. It also registers the processor class in the container and adds a
health check for the processor. The host gives you configuration, logging, dependency injection,
and graceful shutdown. You write the processor class.
This page shows how to write the program, configure it, and read its health and its failures.
Write the smallest program
Section titled “Write the smallest program”This program runs OrderProcessor, the class from
Writing a Processor:
using Kurrent.Surge;using Microsoft.Extensions.Hosting;
var builder = Host.CreateApplicationBuilder(args);
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderProcessor>());
await builder.Build().RunAsync();RunAsync is an extension method in Microsoft.Extensions.Hosting, so the program must have that
using.
The program gets its connection from configuration. Add the connection string to
appsettings.json:
{ "ConnectionStrings": { "KurrentDB": "kurrentdb://localhost:2113" }}Register an IRecordProcessor with AddProcessor<T>. Register an IRecordBatchProcessor with
AddBatchProcessor<T>. The two methods take the same arguments.
To set an option in code, give AddProcessor a delegate:
using Kurrent.Surge;using Kurrent.Surge.Client.Streams;using Microsoft.Extensions.Hosting;
var builder = Host.CreateApplicationBuilder(args);
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderProcessor>(options => options.Subscription.Filter = RecordFilter.ByStream("orders")));
await builder.Build().RunAsync();Build and run the application
Section titled “Build and run the application”A program goes through these steps:
- Create the builder with
Host.CreateApplicationBuilder(args). For a web application, useWebApplication.CreateBuilder(args). - Call
builder.AddKurrentSurge(surge => surge.AddProcessor<T>("processor-id", options => ...)). The processor id and the delegate are optional. - Register the dependencies of the processor on
builder.Services. - Call
builder.Build(), then callRunAsync()on the host.
When the host starts, the scope reads the options from configuration, runs the delegate, validates the result, and starts the processor.
The QuickStart passes the processor id live-transfers, sets its options in the delegate, and
registers a seeder:
var builder = Host.CreateApplicationBuilder(args);
builder.AddKurrentSurge(surge => surge.AddProcessor<LiveTransferProcessor>("live-transfers", options => { options.Subscription.Filter = RecordFilter.ByExpression("stream like 'player-%'"); options.Subscription.InitialPosition = SubscriptionInitialPosition.Latest;
// Tailing live data only, so there is nothing to resume after a restart — checkpoint // rarely rather than never, so the drone never wastes writes catching up to itself. options.Checkpoint.Interval = TimeSpan.FromDays(1);}));
builder.Services.AddSerilog();builder.Services.AddHostedService<TransferSeeder>();
await builder.Build().RunAsync();Its processor is a MessageProcessor subclass. The constructor registers one handler for each
message type:
sealed class LiveTransferProcessor : MessageProcessor { public LiveTransferProcessor() { Process<PlayerTransferred>((transfer, ctx) => ctx.Logger.LogInformation( "Transfer: {PlayerName} from {FromClub} to {ToClub} for {Fee:C}", transfer.PlayerName, transfer.FromClubId.ToString(), transfer.ToClubId.ToString(), transfer.Fee.Amount));
Process<PlayerRegistered>((registered, ctx) => ctx.Logger.LogInformation( "Registration: {PlayerName} ({Nationality}) {Position}", registered.Name.ToString(), registered.Nationality.ToString(), registered.MainPosition.ToString())); }}The whole program runs:
examples/processors/Processor.QuickStart
seeds player transfers and logs the events that the processor receives.
Configure the processor
Section titled “Configure the processor”The scope reads the options once, from the section Kurrent:Surge:Processors:{id}. {id} is the
processor id. Then the scope runs the delegate on the values that it read. A value that the delegate
sets replaces the configured value.
The QuickStart passes the id live-transfers, so it reads this section:
{ "Kurrent": { "Surge": { "Processors": { "live-transfers": { "Batching": { "MaxBatchSize": 1000 }, "Checkpoint": { "Interval": "00:00:10" } } } } }}Its delegate sets Checkpoint.Interval to TimeSpan.FromDays(1) and does not set
Batching.MaxBatchSize. The processor runs with these values:
| Option | Section | Delegate | Result |
|---|---|---|---|
Batching.MaxBatchSize |
1000 |
Not set | 1000 |
Checkpoint.Interval |
00:00:10 |
TimeSpan.FromDays(1) |
One day |
The section follows these rules:
- When the scope compares the section name with the id, it ignores case,
-, and_.live-transfers,live_transfers, andLiveTransfersreach the same section. - The scope reads no other section for the processor.
- The scope validates the options. If a value is incorrect, the host does not start.
To change an option with no rebuild, add its key to the section.
Keys that have no effect
Section titled “Keys that have no effect”Configuration cannot set the processor id. The scope logs a ProcessorId key in the section as an
unrecognised setting. The processor id does not change.
| Key in the section | Warning that the scope logs |
|---|---|
| A key with a spelling error | Unrecognised setting. Check the spelling — it had no effect. |
ProcessorId |
Unrecognised setting. Check the spelling — it had no effect. |
Observers, MeterFactory, or Time |
This setting is not configurable and had no effect. Set it in code. |
DroneId, the id of this instance, can come from the section.
Configure the application
Section titled “Configure the application”The application reads the configuration sources of its host builder. For
Host.CreateApplicationBuilder and WebApplication.CreateBuilder, these sources include:
appsettings.json- Environment variables
- The command line, from
args
Set the connection
Section titled “Set the connection”The application uses the first source in this table that has a value.
| # | Source |
|---|---|
| 1 | options.Connection(...) in the delegate |
| 2 | ConnectionString or Connection in Kurrent:Surge:Processors:{ProcessorId} |
| 3 | surge.Connection(...) in builder.AddKurrentSurge(...) |
| 4 | ConnectionStrings:Kurrent:Surge |
| 5 | ConnectionStrings:KurrentDB |
The scope resolves sources 2 to 5 before it runs the delegate, so the delegate receives the connection the processor will use. A connection that the delegate sets replaces it.
When the application compares keys, it ignores case, -, and _. order-processor and
OrderProcessor find the same entry.
If no source has a value, the processor fails validation and the host does not start.
To configure the client, use options.Connection(connectionString, client => ...). To attach a
catalog that no assembly attribute activates, call client.Messages.UseCatalog(...) in that
delegate. See Catalogs.
To change the configured connection at the application level, call surge.Connection(client => ...)
in the same AddKurrentSurge call. This example attaches a catalog at that level:
var builder = Host.CreateApplicationBuilder(args);
builder.AddKurrentSurge(surge => { surge.Connection(client => client.Messages.UseCatalog(catalog)); surge.AddProcessor<OrderProcessor>();});
await builder.Build().RunAsync();The processor and each service that gets KurrentClient from the container, for example a seeder,
use the same connection options. The table shows what each level changes:
| Level | Call | Reaches |
|---|---|---|
| Processor | options.Connection(...) |
The processor only |
| Application | builder.AddKurrentSurge(surge => surge.Connection(...)) |
Each processor and each service that gets KurrentClient from the container |
If the processor’s own section sets ConnectionString or Connection, the processor uses that
value and not the application connection.
Processor identity
Section titled “Processor identity”The processor id is the string that you pass to AddProcessor. If you do not pass one, the scope
uses the type name in kebab-case: LiveTransferProcessor gives live-transfer-processor.
The runtime uses the processor id for these items:
- The configuration section
- The checkpoints
- The name of the health check
- The key of the processor class and of
IDronein the container - The
processor_idtag on the metrics - The start of the drone id, which is the id of this instance
The checkpoints use the processor id as their key. If the processor id changes, the processor starts again from its configured initial position. If you pass no id and rename the class, the processor id changes. To keep the same processor id, pass it:
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderProcessor>("orders"));Set the id at registration. If the delegate sets ProcessorId, the scope reads no section for the
new id. To choose the drone id, set DroneId.
Register the processor and its dependencies
Section titled “Register the processor and its dependencies”AddProcessor<T> registers T in the container as a singleton, with the processor id as its key.
The container creates T and resolves its constructor dependencies. You do not register T.
If you register T yourself, with the processor id as the key, your registration wins. Use this to
create the processor in your own code:
builder.Services.AddKeyedSingleton<OrderProcessor>("orders", (sp, _) => new OrderProcessor(sp.GetRequiredService<IOrderStore>()));
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderProcessor>("orders"));The scope registers T and nothing else. Register the dependencies of T on builder.Services:
using Kurrent.Surge;using Microsoft.Extensions.DependencyInjection;using Microsoft.Extensions.Hosting;
var builder = Host.CreateApplicationBuilder(args);
builder.AddServiceDefaults();builder.AddNpgsqlDataSource("orders-db");builder.Services.AddSingleton<IOrderStore, SqlOrderStore>();
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderProcessor>());
await builder.Build().RunAsync();Extension methods for IHostApplicationBuilder work on the builder as on any host builder. Examples
are the Aspire service defaults and Aspire client integrations such as AddNpgsqlDataSource.
Extension methods for IServiceCollection, such as Serilog’s AddSerilog, work on
builder.Services.
To seed data, register a hosted service. The QuickStart registers TransferSeeder with
AddHostedService<TransferSeeder>(), and the host runs it at the same time as the processor.
Resolve the processor
Section titled “Resolve the processor”To use the processor in other code, resolve it with the processor id as the key:
var processor = host.Services.GetRequiredKeyedService<OrderProcessor>("orders");In a minimal API, use [FromKeyedServices("orders")] on the parameter.
While a type runs under one id only, GetRequiredService<OrderProcessor>() also resolves it. When you
register a second id for the same type, the scope removes the registration with no key. Then use the
key.
If you call builder.Services.AddSingleton<OrderProcessor>(), the container holds a different
instance. The processor runtime does not use that instance.
After Build(), resolve these services from host.Services:
| Service | Key | Contains |
|---|---|---|
The processor class T |
The processor id | The processor instance, a singleton |
IDrone |
The processor id | ProcessorId, Status, and Options. Status is the DroneStatus of the processor. Options is the ProcessorOptions that the processor runs with |
HealthCheckService |
None | The health report. Each processor entry has the processor id as its key |
Run several processors
Section titled “Run several processors”Call AddProcessor or AddBatchProcessor one time for each processor in the AddKurrentSurge
delegate. The scope registers one hosted service for each processor id. The host runs all the
processors at the same time.
builder.AddKurrentSurge(surge => { surge.AddProcessor<OrderProcessor>(); surge.AddBatchProcessor<BulkIndexer>();});The same type can run under different ids, for example one processor for each tenant. The delegate
that takes an IServiceProvider gets services for the options:
builder.AddKurrentSurge(surge => { foreach (var tenant in tenants) surge.AddProcessor<OrderProcessor>($"orders-{tenant}", (options, sp) => { options.Subscription.Filter = RecordFilter.ByStream($"orders-{tenant}"); options.Checkpoint.Interval = sp.GetRequiredService<TenantPolicy>().CheckpointInterval(tenant); });});Each id has its own section, its own checkpoints, and its own health check.
If two registrations use the same id, the scope throws InvalidOperationException. The message
names the id.
Run processors in a web application
Section titled “Run processors in a web application”In a web application, call AddKurrentSurge on WebApplication.CreateBuilder. The processor runs
beside the HTTP endpoints. Surge adds no HTTP endpoint.
var builder = WebApplication.CreateBuilder(args);
builder.AddKurrentSurge(surge => surge.AddProcessor<OrderStatsProcessor>("orders"));
var app = builder.Build();
app.MapGet("/stats", ([FromKeyedServices("orders")] OrderStatsProcessor p) => p.Snapshot());
await app.RunAsync();Snapshot() is a method of the application’s processor class.
Processor.AspNetMinimal
hosts a processor in an ASP.NET Core application.
Read the health check
Section titled “Read the health check”AddProcessor registers one health check for each processor. The name of the health check is the
processor id. Its tags are kurrent, surge, and processor. HealthCheckService combines the
checks, and the worst status wins.
The table shows the health for each processor status:
| Processor status | Health |
|---|---|
| Not started | Degraded |
Starting |
Degraded |
Processing |
Healthy |
Processing, and the last checkpoint failed |
Degraded, with the exception |
Stopping or Stopped |
Unhealthy |
| Failed, at start-up or while processing | Unhealthy, with the exception |
A failed checkpoint keeps the check Degraded until the next checkpoint commits.
To read the health check, get IDrone with the processor id as the key, and HealthCheckService,
from the host services:
using Kurrent.Surge.Processors;using Microsoft.Extensions.DependencyInjection;using Microsoft.Extensions.Diagnostics.HealthChecks;
var host = builder.Build();
var drone = host.Services.GetRequiredKeyedService<IDrone>("orders");var health = host.Services.GetRequiredService<HealthCheckService>();var report = await health.CheckHealthAsync();
var processor = report.Entries[drone.ProcessorId];Surge adds no HTTP endpoint. To send the report to a different system, register an
IHealthCheckPublisher as a singleton. By default, the host runs each publisher at 30-second
intervals.
In a web application, app.MapHealthChecks("/health") exposes the checks over HTTP. A liveness
probe on that endpoint lets the orchestrator restart the container when a processor is Unhealthy.
The probe needs nothing from Surge.
Failure and exit code
Section titled “Failure and exit code”RunAsync runs the host until Ctrl+C, SIGTERM, or its cancellationToken stops it. Then it
disposes the host.
If a processor fails, the host logs the exception at the Error level and stops. On .NET 10,
RunAsync then returns without an exception, and the process exits with code 0. The SDK never sets
the exit code of the process.
To make the process fail, call FailProcessOnFatalError() in AddKurrentSurge:
builder.AddKurrentSurge(surge => { surge.FailProcessOnFatalError(); surge.AddProcessor<OrderProcessor>();});With this option, RunAsync throws the exception of the processor after the host stops. The process
then exits with a failure code, and a container orchestrator can start it again. The processor
continues at or before the record that failed.
On .NET 11, RunAsync throws the exception with no option. There, FailProcessOnFatalError() adds
nothing.
A stop by Ctrl+C or SIGTERM is not a failure. RunAsync returns, and the process exits with code 0,
with or without the option.
These items show a failure:
- The host log, which has an
Errorentry with the exception - The health check, which reports Unhealthy with the exception
- With
FailProcessOnFatalError(), the exception thatRunAsyncthrows and the failure exit code of the process
A liveness probe on the health endpoint is the other way to get a restart. See Read the health check.
Error Handling covers what counts as a failure and where the position stops.
Wiring Up Surge covers the full AddKurrentSurge
scope.