Add EKS operator for commands in existing Pods

[AlejandroMorgante]

Add EksPodExecOperator to execute commands in existing EKS Pods without managing their lifecycle. It reuses KubernetesPodExecOperator for execution and AWS credentials for authentication.

Existing EKS operators retain compatibility with older Kubernetes providers; the new operator requires cncf-kubernetes>=10.22.0. Like EksPodOperator, it supports a configurable, templatable kubernetes_conn_id.

Includes documentation, unit tests, and an EKS system test. The shell-portability fix was merged separately in #73690.

Validation of the current revision:

• EKS operator unit tests: 73 passed, including the missing pod_exec import regression test.
• Project structure checks: 10 passed, 1 xfailed (expected).
• Focused mypy: no issues found.
• Pre-commit and manual static checks passed.

Earlier live AWS validation completed the full system test: create the EKS cluster and nodegroup, create an externally managed Pod, execute and validate the command output, delete the Pod, and tear down the AWS resources (1 passed in 15:37). This live test was not rerun for the import compatibility change.

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5), Codex (GPT-6)

Generated-by: Codex (GPT-5), Codex (GPT-6) following the guidelines

[AlejandroMorgante]

I also validated the operator with a real end-to-end run against an existing Pod in an EKS cluster. The task connected to the running Pod, executed the command, streamed its logs, and completed successfully in approximately 20 seconds. The Pod remained running and its lifecycle was not managed by Airflow.

This confirms the intended use case without provisioning a new Pod for each command.

[AlejandroMorgante]

@SameerMesiah97, could you please review this when you have a chance? This is the follow-up to #71244, adding the EKS integration for executing commands in existing Pods. Thank you!

[SameerMesiah97]

There is one potential issue with the new operator. EksPodExecOperator inherits from KubernetesPodExecOperator, which was introduced in PR #72144 but has not yet been included in a released cncf-kubernetes provider version. This will raise the minimum cncf-kubernetes version considerably from 7.20.0, but since it is an optional dependency, the impact should be limited to users installing the Amazon provider’s cncf.kubernetes extra or otherwise using the EKS/Kubernetes integration.

The implementation itself looks relatively clean. I have left a few comments.

Edit: alternatively, we could explore conditional imports i.e try/excepct ImportError then implement the EksPodExecOperator if KubernetesPodExecOperator is successfully imported or maybe putting this new operator into its own module and version gate it there. However, this is atypical for providers so I can't approve that without the input of the AWS maintainers.

[AlejandroMorgante]

Thanks for raising this. I agree that the Amazon release must depend on the first cncf-kubernetes release containing KubernetesPodExecOperator, but this is the exact cross-provider scenario covered by Airflow's documented # use next version mechanism, which is already present in this PR. The Release Manager will update the optional dependency and coordinate both releases, so I would prefer this documented mechanism over conditional imports unless the AWS maintainers specifically prefer that approach.

Sometimes, when you add a new feature to a common distribution, you might add a feature to it or change the API in the way that other packages can use it. This is especially true for common packages such as apache-airflow-providers-common-compat, but can happen for other packages (for example apache-airflow-providers-apache-beam is used by apache-airflow-providers-google to use Apache Beam hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package remember that the feature you just add will only be released in the FUTURE release of such common package and you cannot add >==x.y.z dependency to it where x.y.z is the version you are going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares both packages together. Let us repeat - such changes in versions between different airflow package should NOT be added to the dependencies manually by the contributor. They should exclusively be added by the Release Manager. when preparing the release of both packages together. We have a custom mechanism to support such additions, where it is contributor's responsibility to mark dependency with a special comment - simply communicating with the Release Manager that such dependency should be updated to the next version when the release is prepared. If you see such a need to use newly added feature and using it at the same time in a different distribution - make sure to add this comment in the line where dependency you want to use the new feature from is defined

[SameerMesiah97]

Thanks for raising this. I agree that the Amazon release must depend on the first cncf-kubernetes release containing KubernetesPodExecOperator, but this is the exact cross-provider scenario covered by Airflow's documented # use next version mechanism, which is already present in this PR. The Release Manager will update the optional dependency and coordinate both releases, so I would prefer this documented mechanism over conditional imports unless the AWS maintainers specifically prefer that approach.

Sometimes, when you add a new feature to a common distribution, you might add a feature to it or change the API in the way that other packages can use it. This is especially true for common packages such as apache-airflow-providers-common-compat, but can happen for other packages (for example apache-airflow-providers-apache-beam is used by apache-airflow-providers-google to use Apache Beam hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package remember that the feature you just add will only be released in the FUTURE release of such common package and you cannot add >==x.y.z dependency to it where x.y.z is the version you are going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares both packages together. Let us repeat - such changes in versions between different airflow package should NOT be added to the dependencies manually by the contributor. They should exclusively be added by the Release Manager. when preparing the release of both packages together. We have a custom mechanism to support such additions, where it is contributor's responsibility to mark dependency with a special comment - simply communicating with the Release Manager that such dependency should be updated to the next version when the release is prepared. If you see such a need to use newly added feature and using it at the same time in a different distribution - make sure to add this comment in the line where dependency you want to use the new feature from is defined

I think I may have been a bit too presumptuous in my previous comment. Since the cncf-provider is an optional dependency, it means that only those users who use the EKS modules will be affected. Regular AWS provider users can simply ignore the cncf-provider. It is still a large bump in the minimum version for a dependency but given that EKS is more niche, it might be acceptable.

Lets see what the AWS maintainers think.

[AlejandroMorgante]

@vincbeck WDYT?

[vincbeck]

@seanghaeli @ramitkataria

[o-nikolas]

I'm not sure I'm in favour of bumping the minimum cncf version to bleeding edge. If users don't install with the amazon extra (apache-airflow-providers-amazon[cncf.kubernetes]) and instead are installing with (apache-airflow-providers-amazon, which won't bump their cncf version) or if they for some reason cannot upgrade cncf to the bleeding edge then all the operators in eks.py will fail to load due to the import failure of the cncf Excec operator.

You can actually see another different, but similar, workaround we put in to avoid bumping this floor previously:

airflow/providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py

Lines 55 to 61 in 503e807

	try:
	from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
	except ImportError:
	# preserve backward compatibility for older versions of cncf.kubernetes provider, remove this when minimum cncf.kubernetes provider is 10.0
	from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import ( # type: ignore[no-redef]
	KubernetesPodOperator,
	)

I think @SameerMesiah97 instincts were right here, we should at least for a few releases have some error handling if that import fails.

@vincbeck @ferruzzi thoughts?

Putting a request for changes for now to avoid merging this until we're decided.

[vincbeck]

I'm not sure I'm in favour of bumping the minimum cncf version to bleeding edge. If users don't install with the amazon extra (apache-airflow-providers-amazon[cncf.kubernetes]) and instead are installing with (apache-airflow-providers-amazon, which won't bump their cncf version) or if they for some reason cannot upgrade cncf to the bleeding edge then all the operators in eks.py will fail to load due to the import failure of the cncf Excec operator.

You can actually see another different, but similar, workaround we put in to avoid bumping this floor previously:

airflow/providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py

Lines 55 to 61 in 503e807

	try:
	from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
	except ImportError:
	# preserve backward compatibility for older versions of cncf.kubernetes provider, remove this when minimum cncf.kubernetes provider is 10.0
	from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import ( # type: ignore[no-redef]
	KubernetesPodOperator,
	)

I think @SameerMesiah97 instincts were right here, we should at least for a few releases have some error handling if that import fails.

@vincbeck @ferruzzi thoughts?

Putting a request for changes for now to avoid merging this until we're decided.

I agree with Niko hee

[ramitkataria]

+1 to Niko's comments as well. Added a couple other comments

[ramitkataria]

Does kubernetes_conn_id=None actually skip the connection lookup here? KubernetesHook.__init__ does self.conn_id = conn_id or kubernetes_conn_id, so I think None falls back to kubernetes_default. If that connection has kube_config or cluster_context set, get_conn would fail against the generated kubeconfig, and since the parameter is hardcoded there is no way to work around it. Would it make sense to either override hook to pass conn_id=None explicitly, or keep kubernetes_conn_id user-settable like EksPodOperator does?

[AlejandroMorgante]

Done, we applied the same strategy used by EksPodOperator: kubernetes_conn_id is now configurable and templatable.

[AlejandroMorgante]

I based this on the existing backward-compatible import fallback in the EKS module

airflow/providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py

Lines 55 to 61 in 503e807

	try:
	from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
	except ImportError:
	# preserve backward compatibility for older versions of cncf.kubernetes provider, remove this when minimum cncf.kubernetes provider is 10.0
	from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import ( # type: ignore[no-redef]
	KubernetesPodOperator,
	)

The difference is that KubernetesPodExecOperator has no equivalent in older versions of the cncf-kubernetes provider. This fallback keeps existing EKS operators importable and only raises an error when someone tries to instantiate EksPodExecOperator without the required provider version.