CeleryExecutor: no way to forward task logs to worker stdout (subprocess_logs_to_stdout hard-coded False)

[nosami]

Apache Airflow version

3.2.0 (also present on main)

What happened

When running the CeleryExecutor, task subprocess stdout/stderr is not forwarded to the Celery worker's own stdout, so it never reaches a container-level log collector (e.g. Kubernetes/AKS ContainerLogV2, Loki, etc.). The LocalExecutor and the KubernetesExecutor per-task pod both forward task logs to stdout, so migrating a deployment from KubernetesExecutor to CeleryExecutor silently drops task-level logs from any stdout-based collector. Task logs still reach the remote/base task-log handler (files/blob) and the UI — only the container-stdout stream is affected.

This is a companion asymmetry to the JSON-rendering gap fixed in #68912 / #68916, but it is a separate flag and is not addressed by that fix.

What you think should happen instead

The CeleryExecutor worker should have a supported way (config option, ideally [celery]/[logging]) to also emit task logs to the worker stdout — i.e. parity with LocalExecutor / the Kubernetes pod path — instead of the value being hard-coded per call site.

Root cause

Whether task logs are forwarded to the worker's stdout logger is controlled by the keyword-only subprocess_logs_to_stdout argument of supervise(), which defaults to False:

• Default False: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L479 and https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2012
• The stdout logger is only added to the log forwarder's target_loggers when the flag is True: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L569-L572 and https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1505-L1507

The call sites are inconsistent:

• LocalExecutor passes subprocess_logs_to_stdout=True: https://github.com/apache/airflow/blob/3.2.0/airflow-core/src/airflow/executors/local_executor.py#L144-L152
• Kubernetes pod path (execute_workload) passes subprocess_logs_to_stdout=True: https://github.com/apache/airflow/blob/3.2.0/task-sdk/src/airflow/sdk/execution_time/execute_workload.py#L66-L77
• CeleryExecutor calls supervise(...) without the argument, so it defaults to False: https://github.com/apache/airflow/blob/providers-celery/3.17.1/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py#L202-L210

There is currently no [celery]/[logging] config option to toggle this for the Celery worker, so operators must monkeypatch supervise's default to restore the KubernetesExecutor behaviour.

How to reproduce

1. Run a deployment with [core] executor = CeleryExecutor (Airflow 3.2.0, apache-airflow-providers-celery==3.17.1).
2. Run any task that writes to stdout/stderr using a subprocess (e.g. a BashOperator echo, or an ssh subprocess).
3. Observe the Celery worker container stdout (e.g. kubectl logs <worker-pod> / docker logs): the task's stdout/stderr lines are absent (they appear in the task log file / UI only).
4. Repeat with LocalExecutor or the KubernetesExecutor pod path: the same lines do appear on the process stdout.

Proposed solution

Add a config option honored by the Celery worker's task execution path — consistent with the [celery] json_logs / [logging] json_logs fallback pattern added in #68916 — and pass it through to supervise(...):

# providers/celery/.../executors/celery_executor_utils.py, in execute_workload()
subprocess_logs_to_stdout = conf.getboolean("celery", "logs_to_stdout", fallback=None)
if subprocess_logs_to_stdout is None:
    subprocess_logs_to_stdout = conf.getboolean("logging", "worker_logs_to_stdout", fallback=False)
supervise(
    ...,
    subprocess_logs_to_stdout=subprocess_logs_to_stdout,
)

(Exact config names open to maintainer preference; the key point is Celery parity + a supported toggle rather than a hard-coded default.)

Are you willing to submit a PR?

Happy to test and open a PR if maintainers agree on the config surface.

Anything else

Related: #68912 / #68916 fixed the rendering (json_output) half of this Kubernetes→Celery asymmetry; this issue tracks the forwarding (subprocess_logs_to_stdout) half, which remains hard-coded.