Introduce parent task spans and nest worker and trigger spans under them by dstandish · Pull Request #63839 · apache/airflow

dstandish · 2026-03-17T18:06:23Z

This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

example output

dstandish · 2026-03-17T18:10:43Z

@xBis7 @ashb @nickstenning

xBis7 · 2026-03-18T13:18:49Z

I changed the otel_test_dag.py to make task1 deferrable.

-@task
-def task1():
-    logger.info("starting task1")
+class Task1(BaseOperator):
+    def execute(self, context):
+        self.defer(
+            trigger=TimeDeltaTrigger(timedelta(minutes=1)),
+            method_name="execute_complete",
+        )
+
+    def execute_complete(self, context, event=None):
+        logger.info("starting task1")

-    with tracer.start_as_current_span("sub_span1") as s1:
-        s1.set_attribute("attr1", "val1")
-
-    logger.info("task1 finished.")
+        with tracer.start_as_current_span("sub_span1") as s1:
+            s1.set_attribute("attr1", "val1")
+        logger.info("task1 finished.")


with DAG(
    "otel_test_dag",
    default_args=args,
    schedule=None,
    catchup=False,
) as dag:
-    task1()
+    Task1(task_id="task1")

This is the result

The span for the task1 doesn't appear as a child of the dag_run span. It looks like that's the case in your screenshot as well.

dstandish · 2026-03-18T14:14:50Z

The span for the task1 doesn't appear as a child of the dag_run span. It looks like that's the case in your screenshot as well.

oh interesting you are right thanks for pointing that out

this stuff is so tricky

xBis7

@dstandish Thanks for the PR! I posted a few questions.

dstandish · 2026-03-18T14:43:34Z

The span for the task1 doesn't appear as a child of the dag_run span. It looks like that's the case in your screenshot as well.

ok @xBis7 i think i fixed it!

i updated the screen shot in the description. confirm when you have a chance!

xBis7 · 2026-03-18T15:06:59Z

ok @xBis7 i think i fixed it!

i updated the screen shot in the description. confirm when you have a chance!

@dstandish I tested it locally and now it looks good!

This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

Move all local opentelemetry and observability imports to the top of each test file, fix TestMakeTaskCarrier fixture to also patch the shared lib tracer so new_task_run_carrier(None) works correctly, and fix assertion indentation in test_emit_dagrun_span_with_none_or_empty_carrier.

test_insert_mapping_includes_context_carrier and test_clear_task_instances_resets_context_carrier only patched airflow.models.taskinstance.tracer, but new_task_run_carrier uses the tracer from airflow._shared.observability.traces. Without patching that, older OTel versions (LowestDeps CI) produce an invalid span and an empty carrier. Apply the same fix already used in TestMakeTaskCarrier.

…hem (#63839) This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

…hem (apache#63839) This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

…hem (#63839) This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time. (cherry picked from commit 0dc4d33)

…hem (apache#63839) This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

…hem (apache#63839)

The execution API's _emit_task_span() was setting the "airflow.task_instance.id" span attribute to the raw UUID object. The OTEL SDK only accepts str/bytes/int/float/bool attribute values (or homogeneous sequences thereof), so it dropped the attribute and logged a warning per task state transition: Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types The fix is to stringify the UUID, matching the convention already used in the same module (e.g. bind_contextvars(ti_id=str(...))). The bug was introduced in apache#63839 and ships in 3.2.0 / 3.2.1. Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com>

* Stringify task_instance.id when setting OTEL span attribute The execution API's _emit_task_span() was setting the "airflow.task_instance.id" span attribute to the raw UUID object. The OTEL SDK only accepts str/bytes/int/float/bool attribute values (or homogeneous sequences thereof), so it dropped the attribute and logged a warning per task state transition: Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types The fix is to stringify the UUID, matching the convention already used in the same module (e.g. bind_contextvars(ti_id=str(...))). The bug was introduced in #63839 and ships in 3.2.0 / 3.2.1. Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> * Add newsfragment for OTEL UUID attribute fix Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> * Drop newsfragment per review feedback Reviewer (henry3260) noted the newsfragment isn't needed for this fix. --------- Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com>

…te (#66053) (#66645) * Stringify task_instance.id when setting OTEL span attribute The execution API's _emit_task_span() was setting the "airflow.task_instance.id" span attribute to the raw UUID object. The OTEL SDK only accepts str/bytes/int/float/bool attribute values (or homogeneous sequences thereof), so it dropped the attribute and logged a warning per task state transition: Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types The fix is to stringify the UUID, matching the convention already used in the same module (e.g. bind_contextvars(ti_id=str(...))). The bug was introduced in #63839 and ships in 3.2.0 / 3.2.1. * Add newsfragment for OTEL UUID attribute fix * Drop newsfragment per review feedback Reviewer (henry3260) noted the newsfragment isn't needed for this fix. --------- (cherry picked from commit 8310049) Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> Co-authored-by: Chris Premus <63758245+c-premus@users.noreply.github.com>

…6053) * Stringify task_instance.id when setting OTEL span attribute The execution API's _emit_task_span() was setting the "airflow.task_instance.id" span attribute to the raw UUID object. The OTEL SDK only accepts str/bytes/int/float/bool attribute values (or homogeneous sequences thereof), so it dropped the attribute and logged a warning per task state transition: Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types The fix is to stringify the UUID, matching the convention already used in the same module (e.g. bind_contextvars(ti_id=str(...))). The bug was introduced in apache#63839 and ships in 3.2.0 / 3.2.1. Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> * Add newsfragment for OTEL UUID attribute fix Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> * Drop newsfragment per review feedback Reviewer (henry3260) noted the newsfragment isn't needed for this fix. --------- Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com>

…te (#66053) (#66645) * Stringify task_instance.id when setting OTEL span attribute The execution API's _emit_task_span() was setting the "airflow.task_instance.id" span attribute to the raw UUID object. The OTEL SDK only accepts str/bytes/int/float/bool attribute values (or homogeneous sequences thereof), so it dropped the attribute and logged a warning per task state transition: Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types The fix is to stringify the UUID, matching the convention already used in the same module (e.g. bind_contextvars(ti_id=str(...))). The bug was introduced in #63839 and ships in 3.2.0 / 3.2.1. * Add newsfragment for OTEL UUID attribute fix * Drop newsfragment per review feedback Reviewer (henry3260) noted the newsfragment isn't needed for this fix. --------- (cherry picked from commit 8310049) Signed-off-by: Chris <63758245+c-premus@users.noreply.github.com> Co-authored-by: Chris Premus <63758245+c-premus@users.noreply.github.com>

dstandish requested review from XD-DENG, amoghrajesh, ashb, bugraoz93, choo121600, dheerajturaga, ephraimbuddy, hussein-awala, jason810496, kaxil, o-nikolas, pierrejeambrun, rawwar and shubhamraj-git as code owners March 17, 2026 18:06

boring-cyborg Bot added area:API Airflow's REST/HTTP API area:Executors-core LocalExecutor & SequentialExecutor area:task-sdk area:Triggerer labels Mar 17, 2026

dstandish force-pushed the parent-task-span branch from 1e8cecf to 89ef70a Compare March 17, 2026 18:09

dstandish force-pushed the parent-task-span branch from 89ef70a to ffde871 Compare March 17, 2026 22:05

xBis7 reviewed Mar 18, 2026

View reviewed changes

dstandish added 4 commits March 18, 2026 12:11

Introduce parent task spans and nest worker and trigger spans under them

1e6f518

This lets us tie together the worker and trigger phases of task execution. Also lets us see the delta between task queued time and task start time.

curren tstate

1546540

current state

edc298e

--wip-- [skip ci]

065882b

dstandish added 2 commits March 23, 2026 09:48

handle ti-less trigger better

2370601

Merge branch 'main' into parent-task-span

4021a4b

dstandish requested a review from potiuk as a code owner March 23, 2026 16:55

dstandish added 2 commits March 23, 2026 09:59

be more cautious about querying dr

25d90c8

move dr up

bc0c1fe

jedcunningham reviewed Mar 23, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py Outdated

jedcunningham reviewed Mar 23, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/jobs/triggerer_job_runner.py

Comment thread airflow-core/src/airflow/models/taskmap.py

dstandish added 2 commits March 23, 2026 11:48

mapped task span name

463efb4

mapped task span name

69dd399

jedcunningham approved these changes Mar 23, 2026

View reviewed changes

dstandish added 2 commits March 23, 2026 11:55

Fix mapped task span name assertion to use [map_index] format

570f013

dstandish merged commit 0dc4d33 into apache:main Mar 24, 2026
143 of 144 checks passed

dstandish deleted the parent-task-span branch March 24, 2026 01:56

xBis7 added a commit to xBis7/airflow that referenced this pull request Apr 17, 2026

Introduce parent task spans and nest worker and trigger spans under t…

c0b90d2

…hem (apache#63839)

This was referenced Apr 28, 2026

OTEL warning "Invalid type UUID for attribute 'airflow.task_instance.id'" floods api-server logs on task state updates #66052

Closed

Stringify task_instance.id when setting OTEL span attribute #66053

Merged

github-actions Bot mentioned this pull request May 10, 2026

[v3-2-test] Stringify task_instance.id when setting OTEL span attribute (#66053) #66645

Merged

tailg8nj mentioned this pull request Jul 24, 2026

Support OpenTelemetry Auto-Instrumentation, Distros & Configurators #70388

Open

2 tasks done

Navigation Menu

Repository navigation

Introduce parent task spans and nest worker and trigger spans under them - #63839

Introduce parent task spans and nest worker and trigger spans under them#63839
dstandish merged 38 commits into
apache:mainfrom
astronomer:parent-task-span

dstandish commented Mar 17, 2026 •

edited

Loading

Uh oh!

dstandish commented Mar 17, 2026

Uh oh!

xBis7 commented Mar 18, 2026 •

edited

Loading

Uh oh!

dstandish commented Mar 18, 2026

Uh oh!

xBis7 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dstandish commented Mar 18, 2026

Uh oh!

xBis7 commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dstandish commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstandish commented Mar 17, 2026

Uh oh!

xBis7 commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstandish commented Mar 18, 2026

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dstandish commented Mar 18, 2026

Uh oh!

xBis7 commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dstandish commented Mar 17, 2026 •

edited

Loading

xBis7 commented Mar 18, 2026 •

edited

Loading