Skip to content

Observability

A processor emits metrics through System.Diagnostics.Metrics, so anything that reads a .NET meter reads these: OpenTelemetry, Prometheus through the OTLP exporter, or dotnet-counters at a terminal.

Meter name: Kurrent.Surge.Processors
Metric Kind Tags
kurrent_processor_records_processed counter processor_id, drone_id
kurrent_processor_record_duration_seconds histogram processor_id
kurrent_processor_batch_duration_seconds histogram processor_id
kurrent_processor_batch_size histogram processor_id
kurrent_processor_checkpoints_completed counter processor_id, drone_id, trigger
kurrent_processor_checkpoints_failed counter processor_id, drone_id, trigger
kurrent_processor_checkpoint_duration_seconds histogram processor_id
kurrent_processor_drone_errors counter processor_id
kurrent_processor_uncommitted_records gauge processor_id, drone_id
kurrent_processor_uncommitted_bytes gauge processor_id, drone_id
kurrent_processor_source_lag_bytes gauge processor_id, drone_id
kurrent_processor_source_lag_seconds gauge processor_id, drone_id

The two source_lag_* gauges are the only ones you opt into. See Lag.

record_duration_seconds means different things on the two dispatch paths. On the per-record path it is the true per-record time. On the batch path it is the batch duration divided by the committed count, so it stays comparable across both.

The trigger tag on the checkpoint counters carries which condition fired: an interval, a filter, a caught-up signal, a manual call, a shutdown, or a rebalance.

The uncommitted_* gauges are free, because the processor already knows how much it has taken and not yet committed.

Source lag is opt in.

opts.TrackSourceLag = true;

The probe reads the head every 10 seconds by default, and each read is an extra unfiltered backward read of $all. That cost is why it is off by default. Change the cadence with opts.SourceLagRefreshInterval, turn it on when “how far behind are we” is a question you actually ask, and leave it off on a processor you already know keeps up.

AddProcessor registers one health check for each processor. The name of the health check is the processor id. The check reports Healthy while the processor processes records. It reports Degraded while the processor starts, and while its checkpoints fail. After the processor stops or fails, the check reports Unhealthy. Running a Processor has the status table and the code that reads the report.

The processor takes an IMeterFactory.

opts.MeterFactory = meterFactory;

Under dependency injection this is supplied for you. Outside it, note that .NET ships no public IMeterFactory implementation, so a processor built by hand needs one from a host or a test double.

Metrics are built on a lifecycle seam that is public in its own right. An observer sees the same events the metrics do, which makes it the place for logging, tracing, or anything else that needs the processor’s internal transitions.

opts.Observers.Add(new MyObserver());
Hook Fires when
OnStateChanged The processor changes state
OnRecordReady A record is about to be dispatched
OnRecordCompleted A record finished
OnRecordSkipped CanProcess rejected a record
OnBatchReady A batch is about to be dispatched
OnBatchCompleted A batch finished
OnCheckpointCommitted A position was committed
OnCheckpointFailed A commit failed
OnSourceLagSampled The lag probe read the head

Every hook has a default, so implement only the ones you want.

A processor emits spans through System.Diagnostics, so any OpenTelemetry setup picks them up by subscribing to the source.

ActivitySource: Kurrent.Surge.Processors
builder.Services.AddOpenTelemetry().WithTracing(tracing => tracing
.AddSource("Kurrent.Surge.Processors")
.AddOtlpExporter());
Span Covers
processor.process_record One record through your handler
processor.process_batch One batch through your handler
process {stream} A record, named by the stream it came from
process A batch, before it is dispatched

The two processor.* spans come from the drone. The two process* spans come from the telemetry observer and carry the OpenTelemetry messaging.* conventions, with messaging.system set to kurrentdb. They nest, so a record appears under both.

Spans start only when something is listening, so an application with no tracing configured pays nothing.

A record carries $traceId and $spanId in its metadata when the producer set them. That links a consumer’s work back to whatever caused the record, across process boundaries and across time, which a span alone cannot do once the producing process is gone.

context.Record.Metadata.TryGetValue<string>("$traceId", out var traceId);
using var scope = context.Logger.BeginScope("{TraceId}", traceId);