Skip to content

Clusters and Load Balancing

A single-node connection has one address and no decisions to make. A cluster has several, and which one a call should go to changes over time, because the leader changes. Discovery is how the client answers that question continuously rather than once at startup.

Discovery answers that question a level lower than most clients do. Node selection here is a gRPC load balancing policy registered on the channel. The channel itself resolves addresses, ranks them, holds the connection, queues calls while it reconnects, and follows an election. Most clients wrap the channel instead: resolve an address, construct a GrpcChannel for it, and rebuild the whole thing when that address stops working.

Pinning a client to kurrentdb://node1:2113 addresses that node and only that node. When node1 stops being the leader, writes still go there and pay a redirect. When node1 goes away, the client goes with it, even though the other two members are healthy.

kurrentdb://node1:2113 one node, no failover
kurrentdb+discover://node1:2113,node2:2113 the cluster, followed as it changes

The hosts in a discovery string are seeds, not the membership. The client asks one of them for the cluster’s gossip, and from then on it works from what gossip reports. Listing every member makes the first contact more likely to succeed; it does not pin anything.

Gossip returns the members and their roles. The client ranks them by your nodePreference and uses the best one available.

Preference Ranks toward
leader The leader. The default, and correct for any client that writes
follower A follower, keeping read load off the leader
readOnlyReplica A read-only replica, so reads never touch a voting member
random Any available member

Ranking is a preference, not a filter. When no node of the preferred role is available the client still connects, because a degraded connection is more useful than none.

DiagramGrpcChannelServiceConfig (LoadBalancingConfigs)kurrent-node-preferenceServiceProviderResolverFactoryGossipResolverLoadBalancerFactoryKurrentDBLoadBalancerInterceptorsLeaderChangeInterceptor
Piece Type it implements What it does
GossipResolver PollingResolver Reads gossip and publishes the members, ranked
KurrentDBLoadBalancer LoadBalancer Owns the subchannel and decides when to move it
Four pickers SubchannelPicker Answer each call: connected, connecting, reconnect, failed
LeaderChangeInterceptor Interceptor Turns a not-leader response into a routing change

The channel address is kurrent:///<connection-name>, a scheme the resolver factory claims. The balancer and the resolver live in a service provider private to that one channel, so two clients in the same process never share resolver state.

The resolver publishes the full membership every pass, ordered by your nodePreference, with the rank attached to each address. Rank 0 is the routing target. The rest are not idle: they are the fallback list the transport already holds.

Diagramgossiprank memberspublish ranksone subchannelrank 0 down, try rank 1

The balancer keeps one subchannel at a time, pointed at rank 0 and carrying the whole ranked list. When rank 0 changes, because the leader moved or the preferred node went away, the balancer replaces the subchannel. Traffic then follows the new target instead of staying on a connection that still works but is no longer the one you asked for.

One subchannel is a deliberate shape, not a limit of the policy. Calls are not spread across nodes, because a client that writes wants the leader, and one that reads from followers says so with nodePreference and gets a follower. Spreading writes would only add redirects.

What a call sees while the connection moves

Section titled “What a call sees while the connection moves”

Each state has a picker, and the picker is what a call actually hits.

Channel state Picker A call is
Ready ConnectedPicker Sent
Connecting QueuePicker Queued, not failed
Idle RequestConnectionPicker Sent, and reconnection is triggered
TransientFailure FailurePicker Failed with the underlying status

A call issued while the balancer is replacing a subchannel waits for the new one rather than throwing, so a leader election is not something your code has to catch.

A resolver failure while the channel is connected is logged and ignored. Gossip being briefly unreachable is not a reason to drop a working connection.

An election is not an error, and the client does not surface it as one. When a call reaches a node that is no longer the leader, the server answers FailedPrecondition and names the new one. The interceptor hands that endpoint to the resolver, which republishes immediately with the announced leader at rank 0 and the previously published members behind it.

DiagramYour callLeaderChangeInterceptorGossipResolvernode1node2 appendFailedPrecondition, leader is node2AnnounceLeader(node2)appendsucceedsrank 0 becomes node2

No gossip query happens on that path. That is why a leader change costs a retry rather than an outage, and why nodePreference=leader is inexpensive to hold. The client is told when it is wrong instead of discovering it by timing out. Discovery resumes on the next scheduled refresh, so an announcement is a shortcut rather than a replacement for gossip.

The policy has no settings of its own. It runs on what discovery gives it, so everything you tune is on the connection string. The defaults suit a cluster on the same network as its clients. Change them when the path between them is slower or less reliable than that.

Parameter Default Raise it when
maxDiscoverAttempts 10 A single lost packet should not collapse discovery, such as over a VPN or between regions
discoveryInterval 100 ms A cluster is mid-election and backing off is better than hammering it
gossipTimeout 5 s The round trip to the cluster approaches five seconds

Tune gossipTimeout against measured latency. Tune keepAliveInterval against the idle timeout of whatever sits between the client and the cluster, which is a different question with a different answer. See the Connection String Reference for both.

Retries live in the same ServiceConfig as the balancing policy, as a gRPC method config, so a retry is enforced by the transport rather than by a loop above it. See Configuring the Client.

Node preference belongs to the client, not to the call. An application that writes to the leader and reads from followers builds two clients, each with its own connection string.

var writes = KurrentClient.Create("kurrentdb+discover://cluster:2113?nodePreference=leader");
var reads = KurrentClient.Create("kurrentdb+discover://cluster:2113?nodePreference=follower");

They share nothing internally, so the second client costs its own channels and nothing else. Reading from a follower means reading data that may lag the leader slightly, which is the trade you are making in exchange for keeping read load off it.

A single-node kurrentdb:// connection runs the identical policy with discovery switched off. The resolver publishes the seed verbatim, and the balancer, the pickers and the leader interceptor all behave the same. A direct connection still follows a leader redirect for that reason.