Repository navigation
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: SAY-5 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @SAY-5! |
|
Hi @SAY-5. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
The PR that switched the default GoCollector to collectors.WithGoCollectorRuntimeMetrics(collectors.MetricsAll) exposes every runtime/metrics path and ballooned Prometheus cardinality roughly 5x (31 -> 142 go_* series per pod on an unchanged daemonset controller bumping v0.19.7 -> v0.23.1). That change was reverted on release-0.19 and release-0.20 after users hit per-project metric quotas, but the revert never got forward-ported to 0.21+, so every release from v0.21 onward carries the cardinality regression. Restore the conservative default. Operators who want the extra runtime metrics can still register their own GoCollector with MetricsAll on ctrlmetrics.Registry themselves. Refs issue 3505. Signed-off-by: SAY-5 <SAY-5@users.noreply.github.com>
|
/hold |
|
Also, undisclosed AI usage |
|
Thanks @alvaroaleman — withdrawing per your comment on #3505. Understood the revert was about the breaking change and operators can deregister/filter on their side; no default change needed. Appreciate the quick turnaround. |
Summary
Fixes #3505.
#3070 switched the default
GoCollectortowhich exposes every
runtime/metricspath. This ballooned Prometheus cardinality ~5x: unchanged daemonset controllers bumping v0.19.7 → v0.23.1 went from ~31go_*series per pod to ~142, enough to trip per-project metric quotas. The opt-out was reverted onrelease-0.19via #3147 and onrelease-0.20via #3148 after users hit the wall, but the revert never got forward-ported torelease-0.21+. Every release from v0.21 onward still carries the regression.Newly exposed families (per pod, all DaemonSet-amplified):
go_godebug_non_default_behavior_*(~40 series),go_gc_heap_allocs_by_size_bytes_bucket,go_gc_heap_frees_by_size_bytes_bucket,go_sched_latencies_seconds_bucket,go_gc_pauses_seconds_bucket + 3 other pause histograms,go_cpu_classes_*,go_memory_classes_*,go_gc_cycles_*.Fix
Drop the
WithGoCollectorRuntimeMetrics(MetricsAll)option so the defaultGoCollectorreverts to the conservative runtime set. Operators who want the extra runtime metrics can still register their ownGoCollectorwithMetricsAllonctrlmetrics.Registrythemselves — the Registry is already exported.Test plan
go build ./pkg/internal/controller/metrics/...cleango vet ./pkg/internal/controller/metrics/...clean:8443/metricson a bare controller built against this revision and confirmgo_*series are back to ~30/pod (vs ~140 on v0.21+ today).go_godebug_non_default_behavior_*,go_gc_heap_*_by_size_bytes_bucket,go_sched_latencies_seconds_bucket, and the extra pause histograms should all be absent.The envtest integration suite in
pkg/manager/internal/integration/requires$KUBEBUILDER_ASSETS//usr/local/kubebuilder/bin/etcdwhich is not present in a bare clone, so that suite errors on both master and this branch — unrelated to the change.