GH-4592: make the capability-constrained distribution paths capacity-aware

[jeremydmiller]

Closes #4592. Follows #4596.

#4297 reached one distribution method. The deployment that motivated the whole feature — a multi-database event store with thousands of async daemon agents — goes through a different one, DistributeByGroupAffinity, which ignored node load entirely. So did the blue/green path and the durability-affinity spread. As merged, the people most likely to enable capacity-aware assignment got nothing from it.

The decision this turns on

In these three methods capacity is a preference and never a filter — the opposite of DistributeEvenly, and deliberately so.

DistributeEvenly can hard-refuse an overloaded node because every node is a candidate there, so refusing some of them can only ever delay an agent. In the capability-aware paths the candidate set has already been narrowed — by declared capabilities, by the GH-3341 rescue for a partition no node declares, by the GH-4562 per-member grandfathering — and stacking a second hard constraint on top can empty it. An empty candidate set in these methods is not "the agent waits". It is a shard database that stops projecting with no running agent, no log and no self-heal until a restart: precisely the failure GH-3341 exists to prevent.

Memory pressure is transient. A stalled shard is not. So a node with headroom always wins, and an overloaded node still beats nothing.

Group affinity

Ordering aside, this path also sheds — at partition granularity, because a partition is exactly what the method refuses to split. An overloaded incumbent gives up a whole shard database only when another candidate can take the entire thing, and at most OverloadShedBatchSize partitions per evaluation.

Both conditions earn their place. Detaching with nowhere to put it would stop the database outright — the mistake #4590 fixed in the even path, applied to an indivisible unit. And a node hosting twenty databases shedding all twenty at once would hand the cluster a connection-pool stampede plus twenty rounds of projection catch-up, to relieve pressure that is re-sampled every heartbeat anyway.

The affinity path is the weakest of the three, on purpose

Worth stating plainly rather than letting it read as finished. In DistributeEvenlyWithAffinity, capacity decides the order the capable nodes fill in, not how many agents each may hold:

• The even ceiling still applies, so two agents over two nodes is one apiece however loaded either is.
• The explicit Database affinity is per agent family, not per database: 73% of shard databases have durability and projections on different nodes #3785 co-location preference still outranks load.

Overriding either would trade the memory problem for the connection-pool problem those mechanisms exist to solve. Making that path actually cap an overloaded node needs a separate decision about which of the two costs wins, and I did not want to make it silently inside this change.

Tests

15 new, in two labelled categories:

• 7 discriminate — they fail against the pre-change code. Headroom preference in each path, the incumbent shed, and the shed budget being both bounded and configurable.
• 8 are invariant guards — never park a group, never split one to relieve pressure, keep the Event-subscription agents left unassigned after node scale-down (shards silently stop projecting) #3341 undeclared-partition rescue working under cluster-wide pressure, keep the even ceiling, keep the affinity preference, and "no node advertising anything ⇒ byte-identical placement".

The guards matter as much as the discriminating ones here: most of the ways this change could go wrong are ways it could remove a placement, and a test that only proves the new preference works would not catch any of them.

Two of my own expectations were wrong and the code was right — the affinity remainder correctly caps each node at its even share regardless of load, and I had asserted both agents would pile onto the idle node. Corrected rather than worked around.

Verification

• dotnet build wolverine.slnx -c Release -f net9.0 — 0 warnings, 0 errors
• CoreTests — 3077 passed, 0 failed
• MartenTests.MultiTenancy — 56 passed (real Postgres)
• PolecatTests.Distribution — 9 passed (real SQL Server)

Still open from the #4297 review: #4593 (the other message stores — a multi-database deployment on anything but PostgreSQL still advertises nothing, so this change is inert there), #4594 (docs), JasperFx/CritterWatch#1343, JasperFx/ai-skills#233.

🤖 Generated with Claude Code