Repository navigation
Dynamic controller scaling #2576
Description
Currently we allow to specify a fixed number of nodes for each controller.
After attending the talk at KubeCon on how to scale Cluster API to 2k clusters (link tba), it's be good to allow controller runtime to spin up and down workers dynamically based on objects in the queue, and on the 90th percentile of the overall duration of the reconciler.
Activity
sqbi1024 commentedon Nov 8, 2023
Can you describe it in more detail?
halfcrazy commentedon Nov 13, 2023
I think there are two tasks
troy0820 commentedon Nov 13, 2023
/kind feature
timebertt commentedon Nov 27, 2023
I'm skeptical whether changing the number of workers during runtime is a good idea.
I always considered the number of workers to be the "size" of a controller – somewhat related to resource requests/limits.
Increasing the number of workers without increasing its requests/limits might cause the process to be throttled, i.e., it might not help in increasing the controller's capacity/throughput.
Instead, I suggest looking into horizontally scaling controllers including some form of sharding.
I explored the idea in this project: https://github.com/timebertt/kubernetes-controller-sharding
vincepri commentedon Nov 27, 2023
Like any other change we usually propose, this would be opt in.
Controller Runtime is focused on a single controller scenario acting as a leader for the time being; but this is probably good to document outside of this project.
k8s-triage-robot commentedon Feb 25, 2024
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
vincepri commentedon Feb 26, 2024
/lifecycle frozen
shubhM13 commentedon Mar 4, 2025
Hi @vincepri,
We’re seeing similar issues with Spark Operator—event spikes overwhelm the controller, causing high latencies and timeouts for our time-sensitive batch workloads.
Are you proposing dynamically adjusting MaxConcurrentReconciles based on queue depth and reconciliation latency, or modifying controller thread scaling more broadly? Would love to understand the approach and potentially contribute in this area
sbueringer commentedon Mar 4, 2025
I would just use #2374
shubhM13 commentedon Apr 20, 2025
Thanks @sbueringer - to use that feature, is that just a boolean set, while initializing the controller ? or we also are expected to define priority levels, and handle priority assignments to events ?
sbueringer commentedon May 8, 2025
@shubhM13 you only have to enable the feature as described in the issue description here: #2374
jlandowner commentedon Jun 26, 2025
Hi, I am looking for a solution for controller scaling, which reconciling 2k+ CRD objects and I knew timebertt/kubernetes-controller-sharding in KubeCon Japan.
I am wondering if the feature can be implemented in controller-runtime and I reached this issue.
For example, it's just an idea but if controller runtime has options for sharding mode and if it can be enabled automatically when it detects the multiple number of controller in leader election lock logic, I think this will help both controller developers and users because we can simply configure HorizontalPodAutoscaler for the controller.
I have not deepdived and I didn't know the philosophy and high hurdle or not, but is there a possibility to consider it in the future?