Repository navigation
bug: workspace deletion stalls and finishes slowly #4072
Description
If you have a symmetric tree of 10000 workspaces, it takes roughly 1 hour to delete all of them when using kubectl delete on the top-level workspace. Deletion does finish successfully, but stalls significantly after a couple of minutes.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
SimonTheLeg commentedon May 7, 2026
Here are the logs for such a stalling deletion kcp-logs.gz.txt. To unpack them simply
And here is the corresponding time-graph
SimonTheLeg commentedon May 29, 2026
Tried with the latest version on main branch and the issue still persists. It takes over an hour to delete 10000 ws in some cases.
SimonTheLeg commentedon Jun 8, 2026
alright so I had a detailed look and spend some time looking into ways to optimize this, but tl;dr I don't think any of them are worth the hassle for now:
Reduce initial build-up by increasing worker go-routines
The controller we use for processing deletion requests, is currently using two go-routines:
kcp/pkg/server/controllers.go
Line 557 in ca38e68
So if we queue up 10,000 items to two workers, and then they have to do 2 calls to the api-server (GET logicalcluster + DELETE logicalcluster), which take on average ~24ms, then it adds up to 2 minutes of initial stall. It is indeed a real stall, because deletion needs to fully propagate through to the lowest level of the tree first before any logicalcluster can be deleted successfully.
If we really wanted to, we can change this to 5 or 10, the savings though would be (assuming api-server is not too busy) only 1 to 1/2 minutes at 10000 workspaces, since this is not really CPU, but rather network heavy
Reducing Wait period between levels
A parent cannot be deleted if it still has children. For these cases, we use an estimate of 15 seconds
kcp/pkg/reconciler/core/logicalclusterdeletion/deletion/logicalcluster_resource_deletor.go
Line 323 in 29e6c6f
which then gets turned into a requeue interval of 8 seconds
kcp/pkg/reconciler/core/logicalclusterdeletion/logicalcluster_deletion_controller.go
Lines 217 to 218 in 8ef4002
So what that means is that each of the parents checks every 8 seconds if it still has any children.
Reducing this to 4s, would save us only 36 seconds in our 10000 workspaces example, so that is pretty underwhelming. Additionally we would also be polling more aggressively on all lc deletions, which might result in additional overhead, specifically if we don't expect it to be gone in 4 seconds (which is realistic on smaller installations)
Eliminating Wait period between levels, by reconciling parents on child deletion
So what we could do is have an informer on the children's deletion, which re-queues the parent. This would completely eliminate any waits between different levels.
However this would fire an event on every single deletion and especially for parents with a lot of children we enque lots of events. For example if we delete a parent with 100 children then we'll fire 100 events before we finally have the event that leads to the parents deletion. So optimistic waiting is far more efficient
Resource Discovery on every Deletion
One of the most expensive calls is the the full resource discovery we do on every deletion (including re-queues). We do need this for every single LogicalCluster as they can have different apis between them + I guess apis can change while an lc is in deletion, thus we cannot simply cache them.
kcp/pkg/server/controllers.go
Lines 471 to 478 in ca38e68
Conclusion
While it would be really nice to speed up the loadtest, I don't think any of the optimizations are worth it at the moment. Anyone who is deleting 10000 workspaces in a single run, most likely is fine with the current deletion time of 31 minutes.