Support OpenTelemetry for metrics on top of Prometheus

Using the Prometheus library to collect metrics works fine mostly, but has some limitations: #258 wants to change the way metrics are aggregated, and #297 wants to add additional handlers to the manager's HTTP endpoint.

Maybe this is a far-out idea, but I wonder if switching to OpenCensus for measurement instead of Prometheus client at this early stage would be a good idea. Tl;dr: OpenCensus is a collection of libraries in multiple languages that facilitates the measurement and aggregation of metrics in-process and is agnostic to the export format used. It doesn't replace Prometheus the service, it just replaces Prometheus the Go library. OpenCensus can export to Prometheus servers, so this is strictly an in-process change.

The OpenCensus Go library is similar to the Prometheus client, but separates the collection of metrics from their aggregation and export. This theoretically allows libraries to be instrumented without dictating how users will aggregate metrics (solving #258) and export metrics (solving #297), though default solutions can be provided for both (likely the same as today's default bucketing and Prometheus HTTP exporter).

Here's an example from knative/pkg of defining measures and views (aggregations): https://github.com/knative/pkg/blob/53b1235c2a85e1309825bc467b3bd54243c879e6/controller/stats_reporter.go. The view is defined separate from the measure, giving the library user the ability to define their own views with library-defined metrics.

And an example of exporting metrics to either stackdriver or prometheus: https://github.com/knative/pkg/blob/225d11cc1a40c0549701fb037d0eba48ee87dfe4/metrics/exporter.go. The user of the library can export views in whatever format they wish, independent of the measures and views that are defined.

It additionally has support for exporting traces, which IMO would be a useful debugging tool and a good use for the context arguments in the client interface (mentioned in #265). Threading the trace id into that context would give the controller author a nice overview of the entire reconcile, with spans for each request, cached or not.

/kind feature

I do kind-of like this idea. Want to put together a PoC?

/priority backlog

/good-first-issue

Working on this now!
/assign

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@ncdc

@ncdc I unfortunately have not gotten around to it yet, but would still love to help out. Don't want to block anyone else who is already getting started though 👍

No worries, just checking so folks don't step on your toes! Thanks for the update.

@bboreham

@bboreham This group is interested in getting OpenTracing in Controller Runtime

cc @DirectXMan12

@vincepri

@vincepri presume you meant OpenTelemetry.
I opened a PR to make it easier to see what I've done: #1211

A few comments from experience in Knative:

If you're just starting to add instrumentation, try to use the same data in the context for both (structured) logging, tracing, and metrics. This has three benefits:

It reduces the amount of boilerplate you need to record the information, and increases the chance that you'll get your labels right.
It reduces the amount of "stuff" you have to carry around in the Context, which is a linked list.
It provides a consistent interface for observability (e.g. Record(ctx, measurement), Info(ctx, "message"), Start(ctx, "spanname")),

Design your Resource schema ahead of time. For many applications, the application acts a single tenant and only a single Resource is needed. Kubernetes controllers are not the typical application; I'd suggest defaulting to a separate Resource for each Kubernetes Resource, on the theory that k8s Resources probably represent an "entity which produces telemetry". In some cases for short-lived resources (for example, a TaskRun in Tekton), it may make sense to associate the resource with the parent (i.e. Task in Tekton).

Note that using Metrics and Traces in a multi-resource way requires passing provider options to NewAccumulator/NewTracerProvider... you probably want to cache these (on the context?), rather than creating a new one every time.

@dashpole

We've got some work done for us in that regard from @dashpole's work on the apiserver tracing KEP, so we should consider that as well

Removing good first issue label, as it is not quite clear if we want this and if so, how it would look.
/remove-good-first-issue

/retitle Support OpenTelemetry for metrics on top of Prometheus

We will definitely not replace the support of prometheus metrics or change its usage by default as that would be a huge breaking change. What we could do is to support opentelemetry in addition to that, but it would likely require a lot of work, as we would need an abstraction over the two of them.

as we would need an abstraction over the two of them.

Hey, I'm one of the maintainers of the OpenTelemetry-Go client libraries, and also a TL for k8s SIG-instrumentation. My recommendation to start would be to continue using the prometheus client library for instrumentation, and add the Prometheus -> OpenTelemetry in-process bridge when the user has configured OpenTelemetry export: go.opentelemetry.io/contrib/bridges/prometheus.

For configuration, I would recommend allowing users to configure OpenTelemetry using the new configuration file format: https://pkg.go.dev/go.opentelemetry.io/contrib/config. It isn't stable yet, but should be made stable in the near future if you are willing to wait a few months, and will save you a lot of time trying to define your own config knobs. This also has the benefit of supporting tracing configuration, if tracing is something you plan to add in the future.

Oh nice, the bridge is a good pointer. We already export the metrics registry so I think that means people can already get the metrics into otel by using it?

Yes, thats correct. Maybe I just need to write a blog post or something :)

package main

import (
	"context"
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"go.opentelemetry.io/contrib/bridges/prometheus"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/sdk/metric"
	"sigs.k8s.io/controller-runtime/pkg/metrics"
)

func main() {
	controllerRuntimeRegistry := metrics.Registry

	bridge := prometheus.NewMetricProducer(
		prometheus.WithGatherer(controllerRuntimeRegistry), 
	)

	otlpExporter, _ := otlpmetrichttp.New(
		context.Background(),
		otlpmetrichttp.WithEndpoint("otel-collector:4318"), 
		otlpmetrichttp.WithInsecure(),
	)

	reader := metric.NewPeriodicReader(
		otlpExporter,
		metric.WithProducer(bridge), 
	)

	provider := metric.NewMeterProvider(metric.WithReader(reader))
	defer provider.Shutdown(context.Background())

	// ...
}

Navigation Menu

Repository navigation

Support OpenTelemetry for metrics on top of Prometheus #305

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support OpenTelemetry for metrics on top of Prometheus #305

Description

Activity

DirectXMan12 commented on Jan 29, 2019

DirectXMan12 commented on Jan 29, 2019

DirectXMan12 commented on Mar 6, 2019

grantr commented on Mar 6, 2019

fejta-bot commented on Jun 4, 2019

DirectXMan12 commented on Jun 5, 2019

fejta-bot commented on Sep 3, 2019

15 remaining items

hasheddan commented on Sep 24, 2020

ncdc commented on Sep 24, 2020

vincepri commented on Oct 8, 2020

bboreham commented on Oct 9, 2020

evankanderson commented on Oct 12, 2020

DirectXMan12 commented on Oct 13, 2020

alvaroaleman commented on Oct 8, 2021

alvaroaleman commented on Feb 17, 2025

dashpole commented on Feb 19, 2025

alvaroaleman commented on Feb 19, 2025

dashpole commented on Feb 20, 2025

halfcrazy commented on Apr 1, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions