Repository navigation
@task.kubernetes fails on non-root images due to /airflow/xcom directory creation #56545
Description
Apache Airflow version
3.1.0
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
The @task.kubernetes decorator fails when used with non-root container images such as the official apache/airflow image.
After PR #28942, the initialization phase for Kubernetes-decorated tasks creates the directory /airflow/xcom (see providers/cncf/kubernetes/decorators/kubernetes.py#L98).
However, in the default Airflow image the container runs as the airflow user, (see HERE) which does not have write permissions to /. As a result, the attempt to create /airflow/xcom fails with a permissions error, and the Kubernetes task cannot start.
{"timestamp":"2025-10-10T13:10:06.622360Z","level":"info","event":"[base] + mkdir -p /airflow/xcom","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-10T13:10:06.622493Z","level":"info","event":"[base] mkdir: cannot create directory ‘/airflow’: Permission denied\n"
What you think should happen instead?
The Kubernetes-decorated task should run successfully, even when the image runs as a non-root user.
How to reproduce
-
Use the official Apache Airflow 3 image, e.g.:
image: apache/airflow:3.1.0
-
Define a DAG:
from airflow.decorators import dag, task from datetime import datetime @dag(start_date=datetime(2024, 1, 1), schedule=None) def test_dag(): @task.kubernetes(image="apache/airflow:3.1.0") def example_task(): print("Hello from Airflow task") example_task() test_dag()
-
Trigger the DAG.
Operating System
Kubernetes (Airflow charts)
Versions of Apache Airflow Providers
Latest
Deployment
Official Apache Airflow Helm Chart
Deployment details
Airflow version: 3.1.0
Executor: @task.kubernetes)
Base image: apache/airflow:3.1.0
Kubernetes version: (v1.32)
Anything else?
The /airflow/xcom directory is currently created unconditionally during initialization, assuming root permissions.
Possible approaches to resolve this:
- Create the directory under a writable path such as /tmp/airflow/xcom, or
- Make the XCom path configurable for Kubernetes-decorated tasks, or
- Check for existence and create the directory only if it’s writable by the current user.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
hussein-awala commentedon Oct 11, 2025
The change you mention was released in
5.2.1more than 2 years and half ago, I'm surprised no one faced/reported the issue before. This would make sense if we have the folder/airflowcreated in the Docker image and we removed it recently, but according the blame history, it is not the case:Could you try reproduce the issue in the K8S integration tests? That way we can figure out where and how we should fix it.
hussein-awala commentedon Oct 11, 2025
This should be the fastest and easiest solution to implement (it's full b/c), feel free to open a PR.
jhgoebbert commentedon Oct 11, 2025
You can see from the tests of the kubernetes provider that only images with an entrypoint as root are used:
So the integration tests cannot show this issue.
I have added the links for each image to its layer-details to show the root-style of the image
and the link to the tests where it is used in the kubernetes-provider below.
image="ubuntu:16.04"
image="alpine"
image="bash:5.2"
image="perl:5.34.0"
image: "gcr.io/spark-operator/spark:v2.4.4"
jhgoebbert commentedon Oct 11, 2025
Could it be that it is intended that non-root images have no support?
I might miss something here - but I do not see a reason why 🤔
hussein-awala commentedon Oct 11, 2025
In fact, it’s recommended to use rootless mode, which means this is a bug.
jhgoebbert commentedon Oct 13, 2025
@hussein-awala Is there any additional information I can give or can you remove the
pending-responsetag?15 remaining items
github-actions commentedon Nov 16, 2025
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
Prab-27 commentedon Nov 16, 2025
I sincerely apologize for the delay.
I assure you the PR will be raised very soon.
Prab-27 commentedon Dec 3, 2025
Still in progress ...
Fix XCom directory creation logic in Kubernetes decorator and add tes…
aaron-y-chen commentedon Dec 12, 2025
Hi, my team encountered this issue as well. Since we needed a fix, I've submitted a PR #59347 to address it. Hope this helps!
Prab-27 commentedon Dec 12, 2025
Sorry, I’m still testing with Kubernetes and learning about it, so I couldn’t fix this one.
I won’t make such mistakes again
aaron-y-chen commentedon Dec 12, 2025
Hi @Prab-27, please don't apologize! K8s has a steep learning curve and we are all learning here. Taking on an issue is a brave first step, and it is definitely not a mistake. Keep exploring and happy coding! 💪
potiuk commentedon Dec 13, 2025
Very much so !
Fix XCom directory creation logic in Kubernetes decorator (#56545) (#…
Fix XCom directory creation logic in Kubernetes decorator (apache#56545…
Fix XCom directory creation logic in Kubernetes decorator (apache#56545…
Fix XCom directory creation logic in Kubernetes decorator (apache#56545…