Skip to content

EksPodOperator deferrable=true pause, restart and then wait until completion #55368

Closed
Closed
EksPodOperator deferrable=true pause, restart and then wait until completion#55368
Labels
area:providersgood first issuekind:bugThis is a clearly a bugpending-responseprovider:amazonAWS/Amazon - related issues

Description

@mhaure-touze
mhaure-touze
opened on Sep 8, 2025

Apache Airflow Provider(s)

cncf-kubernetes, amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-cncf-kubernetes==10.7.0

Apache Airflow version

2.10.3

Operating System

amazon linux

Deployment

Amazon (AWS) MWAA

Deployment details

EksPodOperator which launch a pod on a EKS cluster v1.32

What happened

  1. the operator launch the pod
  2. the triggerer pause the task as "DEFERRED"
  3. the triggerer send a "running" event
  4. the pod launch trigger_reentry method
  5. some how the task wait for pod completion
  6. task stay alive until a hearbeat timeout kill it
ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal
*** Reading remote log from Cloudwatch log_group: airflow-data-eng-mwaa-env-Task log_stream: dag_id=wait/run_id=manual__2025-09-08T12_14_15.298019+00_00/task_id=wait/attempt=1.log
*** Could not read served logs: HTTPConnectionPool(host='ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal', port=8793): Max retries exceeded with url: /log/dag_id=wait/run_id=manual__2025-09-08T12_14_15.298019+00_00/task_id=wait/attempt=1.log (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f35b57ac110>, 'Connection to ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal timed out. (connect timeout=30)'))
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_eks_role'
[Invalid date] {baseoperator.py:416} WARNING - EksPodOperator.execute cannot be called outside TaskInstance!
[Invalid date] {pod.py:1280} INFO - Building pod wait-y8q3mrf8 with labels: {'dag_id': 'wait', 'task_id': 'wait', 'run_id': 'manual__2025-09-08T121415.2980190000-9557e53e5', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[Invalid date] {taskinstance.py:288} INFO - Pausing task as DEFERRED. dag_id=wait, task_id=wait, run_id=manual__2025-09-08T12:14:15.298019+00:00, execution_date=20250908T121415, start_date=20250908T121422
[Invalid date] {taskinstance.py:340} ▶ Post task execution logs
[Invalid date] {pod.py:146} INFO - Checking pod 'wait-y8q3mrf8' in namespace 'namespace'.
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {pod.py:221} INFO - Still waiting for pod to start. The pod state is Pending
[Invalid date] {triggerer_job_runner.py:631} INFO - Trigger wait/manual__2025-09-08T12:14:15.298019+00:00/wait/-1/1 (ID 22) fired: TriggerEvent<{'status': 'running', 'last_log_time': None, 'namespace': 'namespace', 'name': 'wait-y8q3mrf8', 'eks_cluster_name': 'cluster'}>
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_eks_role'
[Invalid date] {pod.py:910} INFO - Resuming logs read from time None
[Invalid date] {pod_manager.py:535} INFO - [base] sleeping...
[Invalid date] {pod_manager.py:535} INFO - [base] sleeping...
[Invalid date] {pod_manager.py:535} INFO - [base] sleeping...
...
[Invalid date] {pod_manager.py:535} INFO - [base] sleeping...
[Invalid date] {pod_manager.py:535} INFO - [base] sleeping...
[Invalid date] {pod_manager.py:554} INFO - [base] sleeping...
[Invalid date] {taskinstance.py:288} INFO - Pausing task as DEFERRED. dag_id=wait, task_id=wait, run_id=manual__2025-09-08T12:14:15.298019+00:00, execution_date=20250908T121415, start_date=20250908T121422
[Invalid date] {taskinstance.py:340} ▶ Post task execution logs
[Invalid date] {pod.py:146} INFO - Checking pod 'wait-y8q3mrf8' in namespace 'namespace'.
[Invalid date] {kube_config.py:515} WARNING - Config not found: /usr/local/airflow/.kube/config
[Invalid date] {kube_config.py:515} WARNING - Config not found: /usr/local/airflow/.kube/config
[Invalid date] {kube_config.py:515} WARNING - Config not found: /usr/local/airflow/.kube/config
[Invalid date] {triggerer_job_runner.py:631} INFO - Trigger wait/manual__2025-09-08T12:14:15.298019+00:00/wait/-1/1 (ID 23) fired: TriggerEvent<{'name': 'wait-y8q3mrf8', 'namespace': 'namespace', 'status': 'error', 'message': 'Invalid kube-config file. Expected key current-context in kube-config', 'stack_trace': 'Traceback (most recent call last):\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 148, in run\n    state = await self._wait_for_pod_start()\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 213, in _wait_for_pod_start\n    pod = await self._get_pod()\n          ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped\n    return await copy(fn, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__\n    do = await self.iter(retry_state=retry_state)\n         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter\n    result = await action(retry_state)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner\n    return call(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 418, in exc_check\n    raise retry_exc.reraise()\n          ^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 185, in reraise\n    raise self.last_attempt.result()\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result\n    return self.__get_result()\n           ^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\n    raise self._exception\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__\n    result = await fn(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 276, in _get_pod\n    pod = await self.hook.get_pod(name=self.pod_name, namespace=self.pod_namespace)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 851, in get_pod\n    async with self.get_conn() as connection:\n  File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__\n    return await anext(self.gen)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 838, in get_conn\n    kube_client = await self._load_config() or async_client.ApiClient()\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 808, in _load_config\n    await async_config.load_kube_config(\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 603, in load_kube_config\n    loader = _get_kube_config_loader_for_yaml_file(\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 567, in _get_kube_config_loader_for_yaml_file\n    return KubeConfigLoader(\n           ^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 150, in __init__\n    self.set_active_context(active_context)\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 161, in set_active_context\n    context_name = self._config[\'current-context\']\n                   ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 448, in __getitem__\n    raise ConfigException(\nkubernetes_asyncio.config.config_exception.ConfigException: Invalid kube-config file. Expected key current-context in kube-config\n', 'eks_cluster_name': 'cluster
'}>
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_eks_role'
[Invalid date] {pod.py:895} ERROR - Trigger emitted an error event, failing the task: Invalid kube-config file. Expected key current-context in kube-config
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
...
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
[Invalid date] {pod.py:982} INFO - [base] logs: sleeping...
[Invalid date] {pod_manager.py:713} INFO - Pod wait-y8q3mrf8 has phase Running
[Invalid date] {pod_manager.py:713} INFO - Pod wait-y8q3mrf8 has phase Running
[Invalid date] {pod_manager.py:713} INFO - Pod wait-y8q3mrf8 has phase Running
...

Invalid kube-config file. Expected key current-context in kube-config', 'stack_trace': 'Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 148, in run
    state = await self._wait_for_pod_start()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 213, in _wait_for_pod_start
    pod = await self._get_pod()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 276, in _get_pod
    pod = await self.hook.get_pod(name=self.pod_name, namespace=self.pod_namespace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 851, in get_pod
    async with self.get_conn() as connection:
  File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 838, in get_conn
    kube_client = await self._load_config() or async_client.ApiClient()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 808, in _load_config
    await async_config.load_kube_config(
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 603, in load_kube_config
    loader = _get_kube_config_loader_for_yaml_file(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 567, in _get_kube_config_loader_for_yaml_file
    return KubeConfigLoader(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 150, in __init__
    self.set_active_context(active_context)
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 161, in set_active_context
    context_name = self._config[\'current-context\']
                   ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes_asyncio/config/kube_config.py", line 448, in __getitem__
    raise ConfigException(
kubernetes_asyncio.config.config_exception.ConfigException: Invalid kube-config file. Expected key current-context in kube-config

The trigger event

What you think should happen instead

I am expecting the task to alternate between a running and deferred state until pod completion/failure

  • Operator mode is deferrable=true
  • logging_interval is set to 600 seconds

How to reproduce

import datetime

from airflow.decorators import dag
from airflow.providers.amazon.aws.operators.eks import EksPodOperator

@dag(
    dag_id="wait",
    start_date=datetime.datetime(2025, 8, 4),
    schedule=None,
    catchup=False,
)
def wait() -> None:
    EksPodOperator(
            task_id="wait",
            aws_conn_id="aws_eks_role",
            cluster_name="cluster,
            deferrable=True,
            namespace="namespace",
            region="eu-west-1",
            pod_name="wait",
            cmds=["/bin/sh", "-c"],
            arguments=["while true; do echo 'sleeping...'; sleep 2; done"],
            image="alpine:3.22.1",
            on_finish_action="delete_pod",
            poll_interval=60,
            logging_interval=600,
        )
wait()

Anything else

I think the error come from the KubernetesPodOperator.trigger_reentry method but I observed it using the EksPodOperator

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. mhaure-touze
    added
    kind:bugThis is a clearly a bug
    area:providers
    needs-triagelabel for new issues that we didn't triage yet
    on Sep 8, 2025

  2. boring-cyborg commented on Sep 8, 2025

    @boring-cyborg
    boring-cyborgBot
    on Sep 8, 2025 – with boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

  3. mhaure-touze
    changed the title [-]EksPodOperator in deferrable mode pause, restart and then wait until completion[/-] [+]EksPodOperator deferrable=true , restart and then wait until completion[/+] on Sep 8, 2025
  4. mhaure-touze
    changed the title [-]EksPodOperator deferrable=true , restart and then wait until completion[/-] [+]EksPodOperator deferrable=true pause, restart and then wait until completion[/+] on Sep 8, 2025
  5. dosubot
    added
    provider:amazonAWS/Amazon - related issues
    provider:cncf-kubernetesKubernetes (k8s) provider related issues
    on Sep 8, 2025

  6. eladkal commented on May 26, 2026

    @eladkal
    eladkal
    on May 26, 2026
    Contributor

    cc @o-nikolas if someone on the team has time to look into this

  7. eladkal
    added
    good first issue
    and removed
    needs-triagelabel for new issues that we didn't triage yet
    provider:cncf-kubernetesKubernetes (k8s) provider related issues
    on May 26, 2026

  8. o-nikolas commented on May 28, 2026

    @o-nikolas
    o-nikolas
    on May 28, 2026
    Contributor

    This is quite an old bug report, sorry we missed it, @mhaure-touze can you try with the latest provider version? A lot has changed in both the EKS and K8s code since September 2025.

  9. darshil929 commented on Jun 13, 2026

    @darshil929
    darshil929
    on Jun 13, 2026
    Contributor

    working on this

  10. VivinAT04 commented on Jun 17, 2026

    @VivinAT04
    VivinAT04
    on Jun 17, 2026

    Hi, I'd like to investigate this issue as well.

    I'll first try to reproduce it against the latest provider versions as suggested above and determine whether the problem still exists. If I can reproduce it and identify a fix, I'll open a PR with my findings.

  11. eladkal
    added
    pending-response
    on Jun 17, 2026

  12. darshil929 commented on Jun 21, 2026

    @darshil929
    darshil929
    on Jun 21, 2026
    Contributor

    Hi @o-nikolas

    Thanks for the pointer. I took your suggestion and dug into this against current main rather than the version it was originally reported on, and I think it's largely been resolved since.

    The relevant change looks like #66705 ("Re-defer task when Kubernetes pod is not completed"). It reworks KubernetesPodOperator.trigger_reentry so that when the trigger returns an error event but the pod is still running / pending / waiting, the operator re-defers and keeps monitoring instead of falling through and waiting for completion. That matches the pause → restart → wait-until-completion behavior described here.

    A few things that point to it being fixed:

    • Grepping the operator on 10.7.0 (the version reported here) for the re-defer logic turns up nothing:
      $ git show providers-cncf-kubernetes/10.7.0:providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py | grep -n "_redefer_count\|MAX_REDEFER_ATTEMPTS"

    No matches. On main the same grep shows it living in trigger_reentry:

      276:    MAX_REDEFER_ATTEMPTS = 3
  1026:        redefer_count = event.get("_redefer_count", 0)
  1028:        if event["status"] == "error" and pod_is_not_done and redefer_count < self.MAX_REDEFER_ATTEMPTS:
  1039:            self.trigger_kwargs["_redefer_count"] = redefer_count + 1
    Image

    One thing worth flagging: in the original logs the re-entry is actually kicked off by a ConfigException: Invalid kube-config file. Expected key current-context in kube-config. #66705 stops that from immediately failing the task, but I don't think it addresses why the kube-config ends up invalid on the triggerer, that part may be closer to #61736.

    @mhaure-touze would you be able to retest on cncf-kubernetes>=10.17.1 and check whether the deferrable behavior looks correct now? If the kube-config error is gone as well, this can probably be closed. if it's still there, it might be worth tracking separately.

  13. o-nikolas commented on Jun 23, 2026

    @o-nikolas
    o-nikolas
    on Jun 23, 2026
    Contributor

    Thanks for the deep dive @darshil929

    As for the credentials expiring or being non-existent. I think this PR I put out a little while ago should fix that: #64187

    What do you think?

  14. darshil929 commented on Jun 23, 2026

    @darshil929
    darshil929
    on Jun 23, 2026
    Contributor

    @o-nikolas yep, #64187 looks like the right fix for the credential/kube-config half, regenerating creds and rebuilding the kubeconfig in EksPodTrigger before it delegates to KubernetesPodTrigger will clear that Expected key current-context error directly.

    So between that and #66705, both halves should be sorted from cncf-kubernetes 10.17.1 and amazon 9.25.0 onward which are both newer than the 10.7.0 / 9.12.0 in the report.

  15. o-nikolas
    closed this as completedon Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:providersgood first issuekind:bugThis is a clearly a bugpending-responseprovider:amazonAWS/Amazon - related issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions