Description
Background
In attempting to leverage the auto-instrumentation mechanism with a vendored distribution , I have run into a few issues. I believe with a few small tweaks this could be much polished and remove the need for workarounds.
Setup
Auto instrumentation has two ways of being invoked, one using the command line wrapper opentelemetry-instrument and the other explicitly invoking it in code. Some of the issues identified later in this document force the latter approach.
Placing the following at the top of airflow_local_settings.py achieves autoinstrumentation:
from opentelemetry.instrumentation import auto_instrumentation
auto_instrumentation.initialize()
from airflow._shared.observability.traces import OverrideableRandomIdGenerator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import get_tracer_provider
if isinstance((tracer_provider := get_tracer_provider()), TracerProvider):
tracer_provider.id_generator = OverrideableRandomIdGenerator()Issue 1: OverrideableRandomIdGenerator
Change Difficulty: Easy
This class was introduced in #63839. My understanding is this a key piece in the task span propagation strategy, but also looks alot like an anti-pattern: creating a span, just to throw it away, and re-use it's IDs later. This is not a blocker. See my comment requesting a more detailed explanation #63839 (comment).
The workaround is to explicitly override any Distro's ID Generator, e.g.
tracer_provider.id_generator = OverrideableRandomIdGenerator()
This class is marked private.
OpenTelemetry offers an environment variable to choose your ID generator, so long as the class is a defined entry point.
Could it be exposed using a Python Entry Point so that others could enforce it via ENV VAR such that I wouldn't have to reference it directly?
Issue 2: Hardcoded Propagation logic
Change Difficulty: Easy
OpenTelemetry vendors may choose to leverage TraceState to propagate implementation-specific details, see definition in SpanContext.
Coupled with the fact that TraceState is immutable, vendors may also supply custom propagators (not just because of a different carrier format):
Please note, since SpanContext is immutable, it is not possible to update SpanContext with a new TraceState. Such changes then make sense only right before SpanContext propagation or telemetry data exporting. In both cases, Propagators and SpanExporters may create a modified TraceState copy before serializing it to the wire.
The current strategy at the Airflow boundary was introduced in #69633. See callout to hardcoding the propagator.
Suggest using the global extract/inject functions instead of instantiating a specific propagator, e.g. TraceContextTextMapPropagator().extract(carrier) becomes just extract(carrier).
This issue also holds true for Baggage.
Workarounds may exist for this depending on the situation.
Issue 3: DAG spans and cross-task propagation
Change Difficulty: Moderate
A incongruency exists between the DAG / trace waterfall and task dependency.
How do we intend to offer OpenTelemetry vendors a way to propagate information across tasks via context_carriers?
Distros may leverage TraceState or Baggage.
The task span is created from the dag run context and not from the previous task context
Because the dag_run.context_carrier is not exposed at the task level, there is no way OpenTelemetry native way to inject or propagate info to downstream tasks.
Ideally a task should be able to inject information into a context_carrier and have it be extracted for dependent tasks.
Workarounds using XCom exist, and maybe this is the correct place for it seeing as it is the intended cross-task communication medium.
See my comment regarding the resolution of context_carrier vs parent_context_carrier.
Use case/motivation
There's been significant progress in integration OpenTelemetry to Airflow. As it stands right now, OpenTelemetry configuration is baked into the airflow.cfg configuration flow. However, OpenTelemetry also provides a way to auto-instrument Python applications as well as for vendors to ship their own bundled set of configurations and key SDK components, see: OpenTelemetry distribution, OpenTelemetry autoinstrumentation.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
Background
In attempting to leverage the auto-instrumentation mechanism with a vendored distribution , I have run into a few issues. I believe with a few small tweaks this could be much polished and remove the need for workarounds.
Setup
Auto instrumentation has two ways of being invoked, one using the command line wrapper
opentelemetry-instrumentand the other explicitly invoking it in code. Some of the issues identified later in this document force the latter approach.Placing the following at the top of
airflow_local_settings.pyachieves autoinstrumentation:Issue 1: OverrideableRandomIdGenerator
Change Difficulty: Easy
This class was introduced in #63839. My understanding is this a key piece in the task span propagation strategy, but also looks alot like an anti-pattern: creating a span, just to throw it away, and re-use it's IDs later. This is not a blocker. See my comment requesting a more detailed explanation #63839 (comment).
The workaround is to explicitly override any Distro's ID Generator, e.g.
This class is marked private.
OpenTelemetry offers an environment variable to choose your ID generator, so long as the class is a defined entry point.
Could it be exposed using a Python Entry Point so that others could enforce it via ENV VAR such that I wouldn't have to reference it directly?
Issue 2: Hardcoded Propagation logic
Change Difficulty: Easy
OpenTelemetry vendors may choose to leverage TraceState to propagate implementation-specific details, see definition in SpanContext.
Coupled with the fact that TraceState is immutable, vendors may also supply custom propagators (not just because of a different carrier format):
The current strategy at the Airflow boundary was introduced in #69633. See callout to hardcoding the propagator.
Suggest using the global extract/inject functions instead of instantiating a specific propagator, e.g.
TraceContextTextMapPropagator().extract(carrier)becomes justextract(carrier).This issue also holds true for Baggage.
Workarounds may exist for this depending on the situation.
Issue 3: DAG spans and cross-task propagation
Change Difficulty: Moderate
A incongruency exists between the DAG / trace waterfall and task dependency.
How do we intend to offer OpenTelemetry vendors a way to propagate information across tasks via context_carriers?
Distros may leverage TraceState or Baggage.
The task span is created from the dag run context and not from the previous task context
airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Line 568 in 83edc67
Because the dag_run.context_carrier is not exposed at the task level, there is no way OpenTelemetry native way to inject or propagate info to downstream tasks.
Ideally a task should be able to inject information into a context_carrier and have it be extracted for dependent tasks.
Workarounds using XCom exist, and maybe this is the correct place for it seeing as it is the intended cross-task communication medium.
See my comment regarding the resolution of context_carrier vs parent_context_carrier.
Use case/motivation
There's been significant progress in integration OpenTelemetry to Airflow. As it stands right now, OpenTelemetry configuration is baked into the
airflow.cfgconfiguration flow. However, OpenTelemetry also provides a way to auto-instrument Python applications as well as for vendors to ship their own bundled set of configurations and key SDK components, see: OpenTelemetry distribution, OpenTelemetry autoinstrumentation.Related issues
No response
Are you willing to submit a PR?
Code of Conduct