Sometimes sporadically some kubernetes executor task pods fail with the following error:
{"timestamp":"2025-12-19T10:51:28.704195Z","level":"info","event":"Connecting to server:","server":"http://airflowtest-api-server.airflow.svc.cluster.local:8080/execution/","logger":"__main__","filename":"execute_workload.py","lineno":64}
{"timestamp":"2025-12-19T10:51:29.055323Z","level":"info","event":"Secrets backends loaded for worker","count":2,"backend_classes":["VaultBackend","EnvironmentVariablesBackend"],"logger":"supervisor","filename":"supervisor.py","lineno":1931}
{"timestamp":"2025-12-19T10:51:34.119133Z","level":"info","event":"Process exited","pid":13,"exit_code":-9,"signal_sent":"SIGKILL","logger":"supervisor","filename":"supervisor.py","lineno":709}
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 125, in <module>
main()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 121, in main
execute_workload(workload)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 66, in execute_workload
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1940, in supervise
process = ActivitySubprocess.start(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 954, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 965, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 215, in start
resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 885, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
hook(response)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 186, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 176, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
Correlation-id=019b363c-6fb7-7b53-ba74-df8e23546e12
The signature should not expire for when the task is still running.
I have set my AIRFLOW__API_AUTH__JWT_EXPIRATION_TIME along with session_lifetime_minutes to "43200" or 12 hours to control the sign out of logged in users after this time. Also i am explicitly setting a apiSecretKeySecretName and jwtSecretName with appropriate k8 secrets.
This does not happen for all dag runs but when scheduling multiple dag runs with multiple tasks. Also the dag fails long before 12 hours set for the jwt expiration time.
Apache Airflow version
Other Airflow 3 version (please specify below)
If "Other Airflow 3 version" selected, which one?
3.1.3
What happened?
Sometimes sporadically some kubernetes executor task pods fail with the following error:
What you think should happen instead?
The signature should not expire for when the task is still running.
How to reproduce
I have set my AIRFLOW__API_AUTH__JWT_EXPIRATION_TIME along with session_lifetime_minutes to "43200" or 12 hours to control the sign out of logged in users after this time. Also i am explicitly setting a apiSecretKeySecretName and jwtSecretName with appropriate k8 secrets.
This does not happen for all dag runs but when scheduling multiple dag runs with multiple tasks. Also the dag fails long before 12 hours set for the jwt expiration time.
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct