Skip to content

🐛 controller/metrics: drop MetricsAll from the default GoCollector - #3507

Closed
SAY-5 wants to merge 1 commit into
kubernetes-sigs:mainfrom
SAY-5:fix/gocollector-default-3505
Closed

SAY-5 wants to merge 1 commit into
kubernetes-sigs:mainfrom
SAY-5:fix/gocollector-default-3505

Conversation

@SAY-5

@SAY-5 SAY-5 commented Apr 22, 2026

Copy link
Copy Markdown

Summary

Fixes #3505.

#3070 switched the default GoCollector to

collectors.NewGoCollector(collectors.WithGoCollectorRuntimeMetrics(collectors.MetricsAll))

which exposes every runtime/metrics path. This ballooned Prometheus cardinality ~5x: unchanged daemonset controllers bumping v0.19.7 → v0.23.1 went from ~31 go_* series per pod to ~142, enough to trip per-project metric quotas. The opt-out was reverted on release-0.19 via #3147 and on release-0.20 via #3148 after users hit the wall, but the revert never got forward-ported to release-0.21+. Every release from v0.21 onward still carries the regression.

Newly exposed families (per pod, all DaemonSet-amplified): go_godebug_non_default_behavior_* (~40 series), go_gc_heap_allocs_by_size_bytes_bucket, go_gc_heap_frees_by_size_bytes_bucket, go_sched_latencies_seconds_bucket, go_gc_pauses_seconds_bucket + 3 other pause histograms, go_cpu_classes_*, go_memory_classes_*, go_gc_cycles_*.

Fix

Drop the WithGoCollectorRuntimeMetrics(MetricsAll) option so the default GoCollector reverts to the conservative runtime set. Operators who want the extra runtime metrics can still register their own GoCollector with MetricsAll on ctrlmetrics.Registry themselves — the Registry is already exported.

Test plan

  • go build ./pkg/internal/controller/metrics/... clean
  • go vet ./pkg/internal/controller/metrics/... clean
  • Manual: scrape :8443/metrics on a bare controller built against this revision and confirm go_* series are back to ~30/pod (vs ~140 on v0.21+ today). go_godebug_non_default_behavior_*, go_gc_heap_*_by_size_bytes_bucket, go_sched_latencies_seconds_bucket, and the extra pause histograms should all be absent.

The envtest integration suite in pkg/manager/internal/integration/ requires $KUBEBUILDER_ASSETS / /usr/local/kubebuilder/bin/etcd which is not present in a bare clone, so that suite errors on both master and this branch — unrelated to the change.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Apr 22, 2026
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: SAY-5
Once this PR has been reviewed and has the lgtm label, please assign joelanford for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Welcome @SAY-5!

It looks like this is your first PR to kubernetes-sigs/controller-runtime 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-runtime has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Hi @SAY-5. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 22, 2026
The PR that switched the default GoCollector to
collectors.WithGoCollectorRuntimeMetrics(collectors.MetricsAll)
exposes every runtime/metrics path and ballooned Prometheus
cardinality roughly 5x (31 -> 142 go_* series per pod on an
unchanged daemonset controller bumping v0.19.7 -> v0.23.1). That
change was reverted on release-0.19 and release-0.20 after users
hit per-project metric quotas, but the revert never got
forward-ported to 0.21+, so every release from v0.21 onward
carries the cardinality regression.

Restore the conservative default. Operators who want the extra
runtime metrics can still register their own GoCollector with
MetricsAll on ctrlmetrics.Registry themselves.

Refs issue 3505.

Signed-off-by: SAY-5 <SAY-5@users.noreply.github.com>
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Apr 23, 2026
@SAY-5 SAY-5 changed the title controller/metrics: drop MetricsAll from the default GoCollector 🐛 controller/metrics: drop MetricsAll from the default GoCollector Apr 23, 2026
@alvaroaleman

Copy link
Copy Markdown
Member

/hold
#3505 (comment)

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 23, 2026
@alvaroaleman

Copy link
Copy Markdown
Member

Also, undisclosed AI usage

@SAY-5

SAY-5 commented Apr 23, 2026

Copy link
Copy Markdown
Author

Thanks @alvaroaleman — withdrawing per your comment on #3505. Understood the revert was about the breaking change and operators can deregister/filter on their side; no default change needed. Appreciate the quick turnaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GoCollector.MetricsAll re-registered by default on v0.21+ inflates Go runtime metric cardinality

3 participants