Airflow 3.2.0: structured task logging increases log size by 1.89× and high-volume task runtime by 2.64× #71721

Under which category would you file this issue?

Task SDK

Apache Airflow version

3.2.0

What happened and how to reproduce it?

What happened

I ran the same DAG and task workload on Apache Airflow 2.9.2 and 3.2.0.
The task emitted exactly the same payload:

• 100,001 payload lines
• 37.9 bytes of useful message content per line

Despite identical logical output, Airflow 3.2.0 produced substantially larger
task logs and took significantly longer to complete.

Per-line record comparison

The following lines contain the same logical payload message.

Airflow 2.9.2:

[2026-08-16T13:54:02.571+0000 +0000] MainProcess MainThread {subprocess.py:93} INFO - Line No 100000: Airflow 2 Log Size Test

Airflow 3.2.0:

{"timestamp":"2026-08-16T17:43:20.929261Z","level":"info","event":"Line No 100000: Airflow 3 Log Size Test","logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook","filename":"subprocess.py","lineno":99}

The Airflow 3.2.0 record repeats a fully qualified logger identity for every line:

"logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"

The useful event message is unchanged, but the Airflow 3.2.0 representation adds structured fields for timestamp, level, logger, filename, and line number, as well as JSON syntax.

The Airflow 2.9.2 representation uses a compact source reference:

{subprocess.py:93}

The expanded logger identity is a significant source of repeated per-line metadata. It contributes to both storage amplification and the per-line
processing required to serialize and write task log records.

We are not claiming that the logger field alone explains the complete regression. It is one directly observable contributor to the broader structured-record overhead.

Reproduction

The task emits 100,001 deterministic progress messages. The DAG, task code, payload, worker resources, executor, logging backend, and deployment settings were kept constant between the two tests. Only the Airflow version changed.

The task is representative of high-volume logging workloads.

Results

──────────────────────┬───────────────┐
│ Metric │ AF 2.9.2 │ AF 3.2.0 │ Delta │
──────────────────────┼───────────────┤
│ Task log file size │ 12,493,072 B (11.91 MiB) │ 23,671,803 B (22.58 MiB) │ 1.89x │
──────────────────────┼───────────────┤
│ Payload lines emitted │ 100,001 │ 100,001 │ identical │
──────────────────────┼───────────────┤
│ Avg bytes per payload line │ 124.89 B │ 235.89 B │ +111.0 B/line │
──────────────────────┼───────────────┤
│ Useful message content per line │ 37.9 B │ 37.9 B │ identical │
──────────────────────┼───────────────┤
│ Signal ratio (content ÷ line) │ 30.3% │ 16.1% │ −14.2 pp │
──────────────────────┼───────────────┤
│ gzip -6 size │ 378,417 B │ 732,143 B │ 1.94x │
──────────────────────┼───────────────┤
│ Payload emit window │ 16.33 s │ 42.53 s* │ 2.60x │
──────────────────────┼───────────────┤
│ Sustained log throughput │ 6,123 lines/s │ 2,351 lines/s* │ −62% │
──────────────────────┼───────────────┤
│ Total task wall time │ 16.86 s │ 44.43 s* │ 2.64x │
──────────────────────┼───────────────┤
│ Framework (non-payload) lines │ 21 │ 259 │ 12.3x │
──────────────────────┼───────────────┤
│ Framework (non-payload) bytes │ 3,930 B │ 82,551 B │ 21.0x │
──────────────────────┴───────────────┘

Impact

The Airflow 3.2.0 task log is approximately 89.5% larger in bytes and remains 1.94x larger after gzip compression.

The task takes approximately 163.5% longer overall, and sustained logging throughput falls by approximately 62%.

The impact affects more than remote storage:

• Increased local and remote log size.
• Increased upload and transfer volume.
• Higher CPU and serialization overhead.
• Slower access to completed task logs in the Airflow UI.
• Slower scrolling and log rendering for large logs.
• Potentially increased risk of incomplete or failed UI rendering.

What you think should happen instead?

Expected behavior

For identical task output, Airflow 3.x should not introduce a significant
runtime and storage regression solely because task logs are represented using
structured records.

At minimum, users should have a supported deployment-level option to choose a
compact persisted task-log representation without modifying operators or DAGs.

Requested behavior

Please consider providing an opt-in compact or legacy task-log mode that:

1. Preserves the Airflow 3.x task-process/supervisor protocol.
2. Converts structured events to compact plaintext before persistence/upload,
   if JSON is required internally.
3. Preserves timestamps, levels, tracebacks, multiline messages, ordering, and
   task-attempt boundaries.
4. Works with worker pods and supported executors.
5. Remains compatible with the Airflow UI, API, local handlers, and remote
   logging backends.
6. Does not require modifications to operators or provider code.

Possible configuration concepts:

[logging]
task_log_format = legacy

or:

[logging]
structured_task_logs = false

The exact setting name is not important; the required capability is an
explicit, documented way to control the persisted task-log representation.

Related reports

• Airflow Task log format seems to ignore configuration #50274: task log format appears to ignore configuration
• Airflow Logging Configs are ignored / not used in Airflow 3.x #53442: Airflow 3.x logging configuration behavior
• Discussion Airflow 3.0.2 configure_logging from airflow.sdk.log ignores logging_config_class #53006: Task SDK logs sent as JSON to the supervisor
• Viewing large task logs get slower to scroll as they get larger. #55173: large task logs become slower to scroll
• Large logs not rendering in UI #58226: large logs not rendering in the UI
• Large logs don't appear in Airflow 3.1.2 #57896: large logs do not appear in Airflow 3.1.2
• Airflow logs support tail (how supervisor does it) #54717: Airflow log tail support
• Unable to see logs in the web UI when the job is running #45516: unable to see logs in the web UI

Operating System

Amazon Linux 2023, x86_64

Deployment

Other

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

apache-airflow 3.2.0
apache-airflow-core 3.2.0
apache-airflow-providers-amazon 9.23.0
apache-airflow-providers-apache-hive 9.4.0
apache-airflow-providers-cncf-kubernetes 10.14.0
apache-airflow-providers-common-compat 1.14.1
apache-airflow-providers-common-io 1.7.1
apache-airflow-providers-common-sql 1.33.0
apache-airflow-providers-fab 3.5.0
apache-airflow-providers-git 0.2.4
apache-airflow-providers-http 6.0.0
apache-airflow-providers-postgres 6.6.1
apache-airflow-providers-smtp 2.4.3
apache-airflow-providers-standard 1.12.1
apache-airflow-task-sdk 1.2.0

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct