Repository navigation
controller histogram buckets configuration #258
Description
Metrics are created for the controller's workqueue, using Prometheus's default buckets. Unfortunately, the default buckets are poorly chosen for event processing.
The default buckets are tailored to broadly measure the response time (in seconds) of a network service.
This can easily result in metrics that are extremely coarse. In a controller I was working on today, every single reconcile was faster than the smallest bucket:
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.005"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.01"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.025"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.05"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.1"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.25"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="0.5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="1"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="2.5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="5"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="10"} 31
controller_runtime_reconcile_time_seconds_bucket{controller="application",le="+Inf"} 31
It is possible to change the default buckets by modifying DefBuckets in init, as my init will be called after the DefBuckets variable has been initialized, but before the controllermetrics package init. But this is a very heavy handed brush, changing the defaults of all histograms.
I propose two paths forward:
- Allow the user to pass their own metrics to the controller, with the current collectors used if the none are provided.
- Alternatively, move the current package outside of the internal package, This will allow users to call Unregister and then assign a replacement.
My preference would be the first option, as the change to the system is would be more easily understood.
Activity
mengqiy commentedon Dec 17, 2018
The 1st option makes sense to me.
DirectXMan12 commentedon Dec 21, 2018
I think we can probably choose some better buckets, too. This smae discussion is being had right now in the main k/k repo ;-).
DirectXMan12 commentedon Dec 27, 2018
It seems to me like exposing the exact mechanics of how we observe metrics is exposing an implementation detail as public API (at least, that's my gut feeling), particularly since the only way to actually change the bucket definition would be to expose the individual metrics at a API level. That would mean that we'd be tied to a particular set of metrics with a particular interface for measuring them (and it would mean that only a major revision could change/remove metrics, but it's unclear if that's a bad thing). If we ever wanted to switch to, say, contextual metrics measurement, we couldn't without a major version rev, even if we didn't actually change the metrics exposed.
My instinct here is to fix our buckets, and wait for more evidence that a "one-size-fits-all" solution won't work here, especially since we have little evidence that our currently buckets are correct for many cases. The Kubernetes SLO for request latency against the API server is
1s, IIRC, and that's on the high end. This bucket selection seems to be poor in that case. k/k is discussing the same thing right now in the switch from summaries to buckets.mengqiy commentedon Dec 27, 2018
Ref: kubernetes/kubernetes#63750 and kubernetes/kubernetes#67476 are the upstream issue and PR.
fejta-bot commentedon Apr 27, 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot commentedon May 27, 2019
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with
/remove-lifecycle rotten.Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
DirectXMan12 commentedon Jun 4, 2019
/remove-lifecycle stale
looks like kube has fixed this upstream, so we can follow suit
/good-first-issue
k8s-ci-robot commentedon Jun 4, 2019
@DirectXMan12:
This request has been marked as suitable for new contributors.
Please ensure the request meets the requirements listed here.
If this request no longer meets these requirements, the label can be removed
by commenting with the
/remove-good-first-issuecommand.Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Merge pull request #258 from pwittrock/master
vincepri commentedon Feb 20, 2020
/priority awaiting-more-evidence
fejta-bot commentedon May 20, 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
irajdeep commentedon Nov 23, 2020
Hi @DirectXMan12
We also have a use-case where we need more time duration buckets (> 10secs) for
controller_runtime_reconcile_time_seconds_buckethistogram.Should we add the buckets to controller-runtime metrics in accordance with the upstream changes done in this PR: kubernetes/kubernetes#73638. ?
Specify buckets for "controller_runtime_reconcile_time_seconds" histo…