Repository navigation
Feature: Warmup for controllers #3220
Description
Tasks
- [v0.21] Design: 📖 Add a design for supporting warm replicas. #3121
- [v0.21] Initial implementation: ✨ Implement warm replica support for controllers #3192
- Further improvements:
- Metrics: 📖 Add a design for supporting warm replicas. #3121 (comment)
- [TBD] util func for health/readiness checks: 📖 Add a design for supporting warm replicas. #3121 (comment)
- Deprioritization of events that came in before the replica became active
- (Check proposal if there is more)
Activity
sbueringer commentedon May 16, 2025
@godwinpang @alvaroaleman
Let me know if I should add additional tasks to this umbrella issue
joey-yi-zhao commentedon Oct 16, 2026
@godwinpang @alvaroaleman @sbueringer
Follow-up on two points from the design review (#3121) that aren't in the final design doc or tracked here
1. Standby queue growth and replay on takeover. As discussed in #3121, a warm replica keeps the key of every object it has seen, including objects that were deleted while it was on standby, because nothing takes items out of the queue before leadership. With churn and generated names, the queue keeps growing for as long as the replica is on standby, not only during warmup. In the review this was treated as a memory question. The other cost shows up at takeover, when every stale key is reconciled once. With a cached client each one is a local NotFound. A reconciler that reads from a remote API, or a source without a local store, pays one remote call per stale key, so a takeover after a long standby period turns into a large request burst against that API.
The "Operational Considerations" section currently only mentions a depth spike during warmup. It would help to describe the trade-off. A cold start lists the live objects and reconciles each of them once. A warm replica skips the wait for the list, but at takeover it reconciles the live objects plus every key deleted during standby. When the reconciler reads from a remote API, or leader changes are rare so standby periods are long, that replay can cost more than the cold start that warmup avoids.
2. Lower priority for adds made before leadership. In the priority queue discussion on #3121, @alvaroaleman suggested a wrapper that enqueues all adds at low priority until the replica is leader. That would also limit the impact of point 1: after takeover, fresh events would be handled ahead of the stale backlog. I couldn't find an issue or PR for it, and I'd like to work on it.
I'd go with that wrapper. Warmup would hand the sources a wrapper around the priority queue. Before leadership, the wrapper sets
LowPriorityon every add, including adds where a handler asked for a higher priority, as @sbueringer suggested.Startswitches the wrapper to pass-through before the workers start. Other queue types are not affected. I'd skip the alternative of lowering everything already queued at election, because the wrapper is less racy.Does this approach sound good to you? I can also add the doc note from point 1 in the same PR, or in a separate one.