Skip to content

OTEL warning "Invalid type UUID for attribute 'airflow.task_instance.id'" floods api-server logs on task state updates #66052

Description

@c-premus

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.1

What happened and how to reproduce it?

With native OpenTelemetry tracing enabled, every task state PATCH on the execution API logs a warning from the OTEL SDK and silently drops the airflow.task_instance.id span attribute:

[warning  ] Invalid type UUID for attribute 'airflow.task_instance.id' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types [opentelemetry.attributes] correlation_id=019dd5ed-763f-738e-baf1-a81a8e1bec32 loc=__init__.py:119 ti_id=019dd5ed-6f3f-76dc-9340-9bc6a82df7ae

The warning fires once per PATCH /execution/task-instances/{id}/state call. In an idle deployment running a few small DAGs we see ~70+ warnings per hour.

Root cause: _emit_task_span() in airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py (line 472 in 3.2.1) passes the raw uuid.UUID value:

span.set_attributes(
    {
        ...
        "airflow.task_instance.id": ti.id,   # uuid.UUID — rejected by OTEL
    }
)

Per the OTEL spec, attribute values must be bool / str / bytes / int / float (or a homogeneous sequence). The OTEL Python SDK's _clean_attribute() therefore drops the value and emits the warning. The same file already uses bind_contextvars(ti_id=str(task_instance_id)) higher up (line 119), so the stringification convention is already established elsewhere in the module.

Introduced in #63839 ("Introduce parent task spans and nest worker and trigger spans under them", merged 2026-03-24); first shipped in 3.2.0, still present on main.

Repro steps:

  1. Run Airflow 3.2.0 or 3.2.1 with [traces] otel_on = True and an OTLP endpoint (e.g. Grafana Alloy).
  2. Trigger any DAG.
  3. docker logs <api-server> | grep "Invalid type UUID" — one warning per task state transition.

What you think should happen instead?

The span attribute should be set successfully (and surface the TI UUID in Tempo / your trace backend). The fix is a one-character change:

-                "airflow.task_instance.id": ti.id,
+                "airflow.task_instance.id": str(ti.id),

Operating System

Ubuntu 24.04 (host); base image apache/airflow:3.2.1-python3.11

Deployment

Other Docker-based deployment

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

Minimal — apt installs (wget, curl, jq, bzip2, gosu), UID/GID alignment, and pip install of standard providers (fab, standard, postgres, http, celery, docker). No changes to OTEL or execution-API code.

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. added
    kind:bugThis is a clearly a bug
    needs-triagelabel for new issues that we didn't triage yet
    on Apr 28, 2026
  2. boring-cyborg commented on Apr 28, 2026

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

  3. c-premus commented on Apr 28, 2026

    @c-premus
    ContributorAuthor

    PR up: #66053

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:APIAirflow's REST/HTTP APIarea:monitoringkind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions