[doc] Performance benchmark #1594

Activity

1. the-glu

   added a parent issue on Jul 7, 2026

   • [doc] Performance boundaries #1362
2. wing-utm-sharing-airspace commented on Jul 8, 2026

   

   wing-utm-sharing-airspace

   on Jul 8, 2026

   Contributor

   More actions

   Could these tests also be run with 7 USSs / 3 nodes each? That's currently our largest pool. Would be curious to see how performance is affected. Thanks!
3. the-glu commented on Jul 9, 2026

   

   the-glu

   on Jul 9, 2026

   MemberAuthor

   More actions

   Could these tests also be run with 7 USSs / 3 nodes each? That's currently our largest pool. Would be curious to see how performance is affected. Thanks!

   Yes, I will do as soon as possible. I will also try to find time to make a graph of number of nodes vs performance to see the impact.
4. utmverse commented on Jul 14, 2026

   

   utmverse

   on Jul 14, 2026

   Last edited by utmverse

   More actions

   Re: Running the #1594 scenarios at 7 USS × 3 nodes

   We reproduced the #1594 scenarios on our test environment.

   Environment

   • 7 USS, 1 DSS per USS, HAProxy across all 7 DSS instances
   • Datastore: the CockroachDB variants ran on a shared 21-node CockroachDB cluster (7 USS × 3 nodes); the raft variant replaces CockroachDB and ran as a 7-node raft cluster (one node per USS). NUM_NODES=3 governs only the CockroachDB node count.
   • netem: 16.5 ms ± 7.5 ms inter-USS, 600 µs intra-USS
   • datastore_max_open_conns=10
   • FlightsInSub.py with the same parameters
   • Database cleared between each run

   Variants tested

   The global SCD lock and the time-based notification index are off by default; each variant differs only in which flag (or datastore) is enabled:

   Variant	Global lock	Time-based index	Datastore
   Baseline (v0.23.0-rc2)	off	off	CockroachDB
   Time-based	off	on	CockroachDB
   Global-lock (busy-cell probe)	on	off	CockroachDB
   Raft (mcuoorb/dss:raft-1587)	off	— (not in build)	raft

   The Global-lock column in the results below is that probe (lock explicitly on); every other variant — including raft — has the lock off.

   Busy S2 cell

   In this current test environment, raft is the only variant that successfully handles the busy cell:

   • Sustains ~25 QPS (peaks around 42 QPS in shorter bursts)
   • p99 latency: sub-second to 1.4 s
   • ~0% 5xx
   • Achieved with no mitigations — raft runs with the global lock off and no time-based index (that flag isn't in the raft build), per the variant table above.

   Under contention, every CockroachDB variant serialized to the 10 s server timeout at roughly 2 QPS. The increasing 409 responses are expected OVN conflicts, not request failures.

   Busy S2 cell — 300 s runs (QPS / p99 / 5xx)

   Target: ≥2 QPS, p99 <5 s, 5xx <1%

   Users	Baseline	Time-based	Global-lock	Raft
   1	1.6 / 1.6 s / 0%	1.9 / 1.5 s / 0%	1.7 / 1.4 s / 0%	6.4 / 0.14 s / 0%
   4	1.9 / 10 s / 1.6%	0.8 / 10 s / 3.4%	2.0 / 2.7 s / 0%	18.7 / 0.39 s / 0.1%
   8	2.1 / 10 s / 15%	0.9 / 10 s / 63%	1.1 / 8.2 s / 0%	24.9 / 0.96 s / 0.1%
   16	2.8 / 10 s / 39%	1.6 / 10 s / 89%	1.6 / 10 s / 80%	22.7 / 1.4 s / 0%

   Observations

   • Time-based is worst on the busy cell — degrades to 63–89% 5xx by 8–16 users, matching earlier reports that it handles this case worse than older releases.
   • Global lock trades throughput for stability — 0% 5xx and bounded latency up to ~8 users, but no throughput gain (~2 QPS), then collapses (80% 5xx at 16 users).
   • Raft is the only variant that scales under contention — throughput climbs with load while every CockroachDB variant flatlines at ~2 QPS.

   Multi-site (first-pass, 90 s runs)

   Part 1 (116 sites) — QPS / p99 / 5xx

   Users	Baseline	Time-based	Raft
   20	17 / 1.8 s / 0%	11 / 10 s / 4%	35 / 1.1 s / 0%
   40	5 / 10 s / 37%	14 / 10 s / 10%	31 / 2.3 s / 0%
   80	9 / 10 s / 37%	12 / 10 s / 27%	27 / 3.6 s / 0%

   Part 2 (11 clusters) — QPS / p99 / 5xx

   Users	Baseline	Time-based	Raft
   20	12 / 10 s / 2%	17 / 10 s / 3%	26 / 1.1 s / 0%
   40	13 / 10 s / 8%	34 / 2.7 s / 0%	24 / 2.4 s / 0%
   80	14 / 10 s / 26%	20 / 10 s / 16%	23 / 4.7 s / 0%

   Observations

   • Raft is the most consistent performer, maintaining p99 <5 s with ~0% 5xx throughout the load ramp.
   • Time-based gives the best CockroachDB throughput — its top result was 34 QPS (Part 2, 40 users) with p99 2.7 s and 0% errors, better than baseline.

   Caveats

   • The 10 s p99 values are censored by the default --server_timeout (i.e. "timed out," not a true percentile).
   • Multi-site tests used a single Locust process, single load balancer, and 90 s duration — treat these as comparative measurements, not absolute capacity numbers. Busy-cell testing used hardened 300 s runs.
   • Raft is a datastore replacement: the performance improvement is significant, but durability and failover characteristics still need production validation.

   Takeaway

   • The busy-cell bottleneck persists at 7 USS / 3 nodes — scaling the deployment larger doesn't fix it.
   • Raft largely resolves the busy-cell bottleneck without requiring feature flags or mitigations.
   • Among CockroachDB-based approaches, time-based indexing provides the best improvement for aggregate multi-site throughput, but it does not improve the busy-cell contention problem.