Add blog for stale controller mitigation by michaelasp · Pull Request #54792 · kubernetes/website

michaelasp · 2026-03-06T17:17:53Z

Description

Issue

This blog goes over the new stale controller mitigations we have added in 1.36 and explains the feature gates and changes done.

netlify · 2026-03-06T17:28:28Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`445a2e6`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-main-staging/deploys/69ab0c45f6219a00087b125e
😎 Deploy Preview	https://deploy-preview-54792--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify · 2026-03-06T17:38:38Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`372be5d`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-main-staging/deploys/69d44785bd216c000702abcc
😎 Deploy Preview	https://deploy-preview-54792--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

chadmcrowell · 2026-03-09T22:21:05Z

Hi @michaelasp 👋 v1.36 Communications team here,

@lukasmetzner as author of #54693, I'd like you to be a writing buddy for @michaelasp on this PR.

Please:

Review this PR, paying attention to the guidelines and review hints
Update your own PR based on any best practices you identify that should be applied
Remember to be compassionate with your fellow article author

lukasmetzner · 2026-03-31T07:09:16Z

Hey @michaelasp, I am your assigned writing buddy. The blog article looks good to me apart from the open thread above, where I agree on adding a basic example.

alvaroaleman · 2026-04-01T15:43:15Z

+An informer author can use the `ConsistencyStore` to track the latest resource version of the objects that the informer cares about.
+It provides 3 main functions:
+
+1. `consistencyStore.WroteAt(owningObj, UID, groupResource, resourceVersion)`: This function is called by the controller when it writes to the API server for an object.


I find this pseudo code confusing, because the parameter names are sometimes a description like owningObject and sometypes the type like UID, I think maybe just using go code to have both a parameter name and a type would be better

I just dropped in the interface we provide here, I think that should work.

serathius · 2026-04-01T18:30:05Z

/cc @jpbetz @liggitt

lmktfy

Thanks.

avoid using "we"
for official post-repease Comms I do recommend aligning to the style guide
I think the feature gate names might be wrong in one place

lmktfy · 2026-04-05T10:30:08Z

+### client-go improvements
+
+In client-go, we have added the `AtomicFIFO` feature on top of the existing RealFIFO queue implementation. Doing so allows for
+the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures


Suggested change

the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures

the queue to atomically handle operations that batch events such as the list portion of the `ListWatch()` operation. This ensures

I recommend the parents to imply it's a function you can call. Have I got that right? The actual verb is watch.

It's based on the list->watch pattern that informers do that has recently been updated to streaming lists https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists

It doesn't really matter which you use. I updated this section a bit to make this more clear.

lmktfy · 2026-04-05T10:43:41Z

+
+### Metrics
+
+We have added the following alpha metrics to kube-controller-manager:


Suggested change

We have added the following alpha metrics to kube-controller-manager:

Nit: could hyperlink to the relevant entry in the metrics reference docs.

Those are auto generated right? How should I best list this? I don't see a specific way to link any better than https://kubernetes.io/docs/reference/instrumentation/metrics/#list-of-alpha-kubernetes-metrics

Added that link for now since it matches the alpha metrics line.

michaelasp · 2026-04-06T16:03:34Z

for official post-repease Comms I do recommend aligning to the style guide

Let me look through that along with your review and update this, thanks for the pass!

michaelasp

Thanks for the review, this should address most of your comments. Looking through the style guide to see if anything else was missed.

michaelasp · 2026-04-06T19:52:31Z

+### client-go improvements
+
+In client-go, we have added the `AtomicFIFO` feature on top of the existing RealFIFO queue implementation. Doing so allows for
+the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures


It's based on the list->watch pattern that informers do that has recently been updated to streaming lists https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists

It doesn't really matter which you use. I updated this section a bit to make this more clear.

michaelasp · 2026-04-06T20:29:05Z

+
+### Metrics
+
+We have added the following alpha metrics to kube-controller-manager:


Those are auto generated right? How should I best list this? I don't see a specific way to link any better than https://kubernetes.io/docs/reference/instrumentation/metrics/#list-of-alpha-kubernetes-metrics

Added that link for now since it matches the alpha metrics line.

lmktfy

Ok; this looks good to me. We can fix the filename after we merge it.

/lgtm
/approve

k8s-ci-robot · 2026-04-17T13:45:46Z

LGTM label has been added.

Details

Git tree hash: da29803b0595879edc8e94f2d3c7f419ecd2e4ee

k8s-ci-robot · 2026-04-17T13:45:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lmktfy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~content/en/blog/OWNERS~~ [lmktfy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the area/blog Issues or PRs related to the Kubernetes Blog subproject label Mar 6, 2026

k8s-ci-robot requested review from Gauravpadam and lmktfy March 6, 2026 17:18

k8s-ci-robot added language/en Issues or PRs related to English language cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 6, 2026

michaelasp force-pushed the staleBlog branch from 445a2e6 to ecdd781 Compare March 6, 2026 17:20

michaelasp mentioned this pull request Mar 6, 2026

Stale Controller Mitigation kubernetes/enhancements#5647

Open

18 of 21 tasks

michaelasp force-pushed the staleBlog branch from ecdd781 to 625dcc8 Compare March 6, 2026 17:26

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 6, 2026

michaelasp changed the title ~~Add blog for stale controller mitigation~~ WIP: Add blog for stale controller mitigation Mar 6, 2026

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 6, 2026

chadmcrowell mentioned this pull request Mar 9, 2026

feature-blog(ccm): new metric route_sync_total #54693

Merged

michaelasp force-pushed the staleBlog branch from 625dcc8 to 4deecbf Compare March 26, 2026 21:46

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 26, 2026

michaelasp force-pushed the staleBlog branch from 4deecbf to efc7637 Compare March 26, 2026 21:47

michaelasp changed the title ~~WIP: Add blog for stale controller mitigation~~ Add blog for stale controller mitigation Mar 30, 2026

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 30, 2026

michaelasp commented Mar 30, 2026

View reviewed changes

Comment thread content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Resolved

serathius reviewed Mar 30, 2026

View reviewed changes

Comment thread content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Resolved

sbueringer mentioned this pull request Mar 31, 2026

✨ Read your own write client design kubernetes-sigs/controller-runtime#3473

Merged

michaelasp force-pushed the staleBlog branch from efc7637 to fddd23f Compare March 31, 2026 18:00

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 31, 2026

alvaroaleman reviewed Apr 1, 2026

View reviewed changes

michaelasp force-pushed the staleBlog branch from fddd23f to 169694c Compare April 1, 2026 17:27

michaelasp requested a review from alvaroaleman April 1, 2026 17:27

k8s-ci-robot requested review from jpbetz and liggitt April 1, 2026 18:30

lmktfy reviewed Apr 5, 2026

View reviewed changes

michaelasp commented Apr 6, 2026

View reviewed changes

Add blog for stale controller mitigation

372be5d

michaelasp force-pushed the staleBlog branch from 169694c to 372be5d Compare April 6, 2026 23:53

michaelasp mentioned this pull request Apr 8, 2026

KEP-5647: Stale informer docs #52901

Merged

lmktfy reviewed Apr 17, 2026

View reviewed changes

k8s-ci-robot assigned lmktfy Apr 17, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 17, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 17, 2026

k8s-ci-robot merged commit 5c514d5 into kubernetes:main Apr 17, 2026
6 checks passed

	the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures
	the queue to atomically handle operations that batch events such as the list portion of the `ListWatch()` operation. This ensures


		### Metrics

		We have added the following alpha metrics to kube-controller-manager:

Conversation

michaelasp commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue

Uh oh!

netlify Bot commented Mar 6, 2026

✅ Pull request preview available for checking

Uh oh!

netlify Bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Pull request preview available for checking

Uh oh!

chadmcrowell commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

lukasmetzner commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

serathius commented Apr 1, 2026

Uh oh!

lmktfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelasp commented Apr 6, 2026

Uh oh!

michaelasp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lmktfy left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Apr 17, 2026

Uh oh!

k8s-ci-robot commented Apr 17, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

michaelasp commented Mar 6, 2026 •

edited

Loading

netlify Bot commented Mar 6, 2026 •

edited

Loading