Skip to content

Fix KubernetesPodOperator dropping init container logs when deferrable=True - #72511

Open
KafkaOtto wants to merge 12 commits into
apache:mainfrom
KafkaOtto:kpo-init-container-logs-deferrable
Open

KafkaOtto wants to merge 12 commits into
apache:mainfrom
KafkaOtto:kpo-init-container-logs-deferrable

Conversation

@KafkaOtto

Copy link
Copy Markdown

Summary

init_container_logs on KubernetesPodOperator only worked when deferrable=False. The sync execution path (await_init_containers_completion) fetched and streamed init container logs, but KubernetesPodTrigger (used when deferrable=True) had no awareness of init containers at all — the parameter was silently accepted and ignored, and no init container logs ever surfaced for deferred tasks.

This PR:

  • Threads init_container_logs from the operator through invoke_defer_method into KubernetesPodTrigger, and serializes it so it survives triggerer restarts.
  • Adds fetch_requested_init_container_logs / _await_init_container_start / _stream_init_container_logs_until_completion to AsyncPodManager, streaming each requested init container's logs in spec.initContainers order once the pod leaves Pending, mirroring the sync PodManager's behaviour.
  • Extracts the shared container-name reconciliation logic (_reconcile_requested_log_containers) into a module-level reconcile_requested_log_containers function reused by both PodManager and AsyncPodManager.
  • Fixes a related bug found while implementing this: completion detection uses get_container_status() rather than the existing container_is_terminated() helper, since the latter only inspects pod.status.container_statuses and never matches init containers (which live in init_container_statuses), which would have caused the new polling loop to hang.

closes: #72504

Test plan

  • Added unit tests for AsyncPodManager.fetch_requested_init_container_logs and its helpers, including a regression test guarding the container_is_terminated pitfall above.
  • Added a KubernetesPodTrigger._wait_for_pod_start test confirming init container logs are (and aren't, when unset) fetched.
  • Added an operator test confirming init_container_logs reaches the trigger via invoke_defer_method.
  • uv run ruff format / ruff check --fix clean on all changed files.
  • mypy clean on all changed source files.
  • Full pytest run of the three affected test files: all pass except two pre-existing, unrelated flaky tests (since_seconds timezone-sensitive assertions that also fail on main without this change).

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5)

Generated-by: Claude Code (Sonnet 5) following the guidelines

…e=True

init_container_logs was only wired into the sync execution path
(await_init_containers_completion); the async KubernetesPodTrigger had no
awareness of init containers at all, so setting init_container_logs together
with deferrable=True silently produced no init container logs.
@boring-cyborg

boring-cyborg Bot commented Sep 4, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

…s/utils/pod_manager.py


handle the case when users create a pod without init_containers

Co-authored-by: Aaron Chen <nailo2c@gmail.com>
Covers the pod.spec.init_containers or [] guard, requested in review.

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, it seems this PR is causing the EKS part to fail, could you help fix it?

breeze run pytest \
	providers/amazon/tests/unit/amazon/aws/triggers/test_eks.py::TestEksPodTrigger::test_serialize_roundtrip \
	-xvs


# console output
FAILED

================================================================ FAILURES ================================================================
_______________________________________________ TestEksPodTrigger.test_serialize_roundtrip _______________________________________________
providers/amazon/tests/unit/amazon/aws/triggers/test_eks.py:407: in test_serialize_roundtrip
    trigger2 = EksPodTrigger(**kwargs)
E   TypeError: EksPodTrigger.__init__() got an unexpected keyword argument 'init_container_logs'
======================================================== short test summary info =========================================================
FAILED providers/amazon/tests/unit/amazon/aws/triggers/test_eks.py::TestEksPodTrigger::test_serialize_roundtrip - TypeError: EksPodTrigger.__init__() got an unexpected keyword argument 'init_container_logs'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================== 1 failed, 1 warning in 16.14s ======================================================
Error 1 returned
None

zhinuan.guo added 2 commits September 10, 2026 09:11
…d-trip

KubernetesPodTrigger.serialize() now includes init_container_logs, but
EksPodTrigger's __init__ didn't accept that kwarg, so reconstructing the
trigger from its own serialized kwargs raised a TypeError.
…le' into kpo-init-container-logs-deferrable
Same root cause as the EksPodTrigger fix: GKEStartPodTrigger overrides
serialize() with its own explicit dict instead of delegating to the base
class, and GKEStartPodOperator.invoke_defer_method never passed
init_container_logs when constructing it, so init container logs were
silently ignored for GKE the same way they were for base KubernetesPodOperator
and EksPodOperator before this PR.
@KafkaOtto
KafkaOtto requested a review from shahar1 as a code owner September 10, 2026 07:24
@KafkaOtto

KafkaOtto commented Sep 17, 2026 •

Copy link
Copy Markdown
Author

Hi @aaron-y-chen, thanks for catching this.

Both issues should be fixed now:

  • 95f46956 fixes EksPodTrigger — it overrides serialize() with its own
    explicit dict instead of delegating to the base class, so
    init_container_logs was dropped on the serialize round-trip.
  • 5fd5f4e8 fixes the same root cause in GKEStartPodTrigger /
    GKEStartPodOperator.invoke_defer_method, which had the identical bug.

test_eks.py::TestEksPodTrigger::test_serialize_roundtrip and the rest of
the amazon-provider suite are green on the latest commit. I also replied to
the test-coverage thread on pod_manager.py:1293 — let me know if there's
anything else you'd like covered.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesPodOperator: init_container_logs is silently ignored when deferrable=True

2 participants