bug: wildcard watch on apiexport virtual workspace fails with 'Too large resource version' #4497

Activity

1. kcp-ci-bot

   added

   kind/bugCategorizes issue or PR as related to a bug.Categorizes issue or PR as related to a bug.

   on Oct 26, 2026
2. kcp-ci-bot

   added this to kcpon Oct 26, 2026
3. github-project-automation

   moved this to New in kcpon Oct 26, 2026
4. neolit123 commented on Oct 30, 2026

   

   neolit123

   on Oct 30, 2026

   Contributor

   More actions

   i could not reproduce this locally on the current main branch of kcp, @2mind
   do you have a reproducer script, that e2e builds kcp, runs it and produces the bug?
5. neolit123

   added

   priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.Lowest priority. Possibly useful, but not yet enough support to actually get it done.

   on Oct 30, 2026
6. 2mind commented on Nov 4, 2026

   

   2mind

   on Nov 4, 2026

   Author

   More actions

   Update from our side — we found the root cause in our deployment and can explain why this is hard to reproduce:

   Why we couldn't reproduce it with load. We tried extensively: local single-shard and 2-replica setups, sustained multi-tenant churn (up to ~900 rev/s), 190 workspaces with APIBindings, deletion waves, and rolling restarts of all shards under load — hundreds of wildcard LIST→watch probes, zero TooLargeResourceVersion. The trigger is not write throughput; on a healthy cacher the 3s blockTimeout window is unreachable.

   What actually broke for us. Our production kcp (fork base e006560fc76a, v0.33.1) had WatchList structurally disabled:

   1. The apiserver's etcdfeature.DefaultFeatureSupportChecker only enables RequestWatchProgress for etcd ≥ 3.5.13 (endpointSupportsRequestWatchProgress, hard-coded version gate due to Etcd not respond to progress notification on newly created watch RPC until there is a event in a watch etcd-io/etcd#17507). We ran etcd v3.5.12 — the check silently returned "unsupported" even with the etcd flag set.
   2. experimental-watch-progress-notify-interval was not set on etcd.
   3. The shard ran without the server-side WatchList feature gate — with the gate off, CheckClient is never even called (see store.New).

   Result: every client (sync agents included) fell back to LIST+watch. The wildcard LIST returns the etcd RV, the watch connects to a replica cacher that is behind, waits up to blockTimeout, and returns TooLargeResourceVersion → client-go relists → gets a fresh etcd RV again → repeats. The lag was persistent in production (post-restart + large accumulated dataset), which is why restarting the agent (landing on a healthy replica / fresh cacher) temporarily fixed it while shard restarts did not.

   What fixed it for us (all three required, verified live — sendInitialEvents on the wildcard VW now returns the k8s.io/initial-events-end bookmark):

   • etcd ≥ 3.5.13 (we run 3.5.21),
   • experimental-watch-progress-notify-interval=1s on etcd,
   • WatchList=true feature gate on the shard, and clients migrated to WatchList.

   After the migration: churn ~145 ops/s across 100+ logical clusters + rolling restart of all shards → 0 TooLarge, agents keep syncing.

   What we think is still worth fixing upstream (our issue was filed against the fork's behavior, but the semantics are upstream kcp design):

   • A watch established with an RV sourced from an etcd-backed LIST should not surface TooLargeResourceVersion to the client as a terminal-ish error; returning a retryable 410 Gone (or making wildcard LIST cache-consistent) would turn the infinite relist loop into a bounded retry.
   • Worth documenting the etcd ≥ 3.5.13 requirement for WatchList — the failure mode (silent "RequestWatchProgress is disabled" despite etcd accepting the flag) is expensive to diagnose; ours took a full root-cause pass through the fork source.

   Happy to share our repro/measure scripts (storm + wildcard-watch victim probe) if useful.