ExecuteTask activity token can expire before the task starts running

[jnstockley]

Apache Airflow version

3.0.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Longer running tasks will occasionally fail because of an invalid auth token that has expired.

2025-07-23 01:27:58,629 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0967-7bc4-b053-e6663b1a45ec/state "HTTP/1.1 204 No Content"
2025-07-23 01:27:58 [info     ] Task finished                  [supervisor] duration=1076.4064389939886 exit_code=0 final_state=success
2025-07-23 01:27:58 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['SecretsManagerBackend', 'EnvironmentVariablesBackend'] count=2
2025-07-23 01:27:58,919 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0955-74d4-8db6-2aea4332ab1b/run "HTTP/1.1 403 Forbidden"
2025-07-23 01:27:58 [warning  ] Server error                   [airflow.sdk.api.client] detail=None
2025-07-23 01:27:58 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=1258981 signal_sent=SIGKILL
2025-07-23 01:27:58,929 {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
    _execute_work(log, workload)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
    supervise(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1683, in supervise
    process = ActivitySubprocess.start(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 851, in start
    proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 862, in _on_child_started
    ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 149, in start
    resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 707, in request
    return super().request(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
    hook(response)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 120, in raise_on_4xx_5xx_with_note
    return get_json_error(response) or response.raise_for_status()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 110, in get_json_error
    raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
Correlation-id=019835f7-a470-78d0-b268-b388c323926f

What you think should happen instead?

The task instance should get a new token and continue

How to reproduce

Create with multiple tasks, and different connections in the tasks where the first task runs for 30+ minutes. Store connections in AWS Secrets Manager.

Operating System

Debain

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.10.0
apache-airflow-providers-common-compat==1.7.2
apache-airflow-providers-common-io==1.6.1
apache-airflow-providers-common-sql==1.27.3
apache-airflow-providers-fab==2.3.0
apache-airflow-providers-google==16.1.0
apache-airflow-providers-http==5.3.2
apache-airflow-providers-microsoft-mssql==4.3.1
apache-airflow-providers-openai==1.6.1
apache-airflow-providers-oracle==4.1.2
apache-airflow-providers-postgres==6.2.1
apache-airflow-providers-sftp==5.3.2
apache-airflow-providers-smtp==2.1.1
apache-airflow-providers-ssh==4.1.1
apache-airflow-providers-standard==1.4.1

Deployment

Other Docker-based deployment

Deployment details

When running in production, we use AWS ECS tasks, I have only see the issue happen here, but also haven't run the longer running tasks locally.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[gopidesupavan]

did you tried using this config ? https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#jwt-expiration-time

[jnstockley]

@gopidesupavan No, but the default is 1 day, so I don't see why increasing it would make a difference, plus just increasing that wouldn't fix the underlying issue.

[potiuk]

@gopidesupavan No, but the default is 1 day, so I don't see why increasing it would make a difference, plus just increasing that wouldn't fix the underlying issue.

I think it's an interesting new feature to conider. For now extending the time is likely a good solution - taking into account that there is no token refresh functionality.

[potiuk]

@gopidesupavan No, but the default is 1 day, so I don't see why increasing it would make a difference, plus just increasing that wouldn't fix the underlying issue.

I think it's an interesting new feature to conider. For now extending the time is likely a good solution - taking into account that there is no token refresh functionality.

And I think if you see a problem with expiring faster than the timeout, this might be connected with time desynchronized on your machines. Strongly recommend using ntpd or similar - often such problems are result of time drift.

[phanikumv]

Since this is a new feature, lets consider moving this to 3.1.0. Do you all agree?