Skip to content

Discover executable bundles by footer, not by execute bit - #74227

Merged
jason810496 merged 3 commits into
mainfrom
fix/coordinator/executable-bundle-exec-bit
Oct 6, 2026
Merged

jason810496 merged 3 commits into
mainfrom
fix/coordinator/executable-bundle-exec-bit

Conversation

@jason810496

@jason810496 jason810496 commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Unblocks serving a Go executable bundle from an object-store Dag bundle, needed for the native Go Dag work in the lang-SDK e2e stack (related: #73847).

Why

ExecutableCoordinator only scanned files that already had the execute bit set. An object-store Dag bundle such as S3DagBundle has no concept of file permissions, so a Go bundle synced through one always lost the execute bit a build produced, and was never found.

The only workaround was a side-channel init container that re-stages the bundle with a forced chmod outside the normal Dag-bundle path.

How

  • The scanner now walks every regular file in the configured Dag bundle, instead of filtering by the execute bit first.
  • Bundle identity still comes entirely from the existing footer-magic and binary_sha256 checks, so non-bundle files are ignored exactly as before.
  • A file that matches the requested dag_id is marked executable, mirroring its read bits, right before it is launched.
  • Updated the three places that documented the old limit: the executable-bundle spec, the Go SDK guide, and the Go SDK README.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines

An object-store Dag bundle such as S3DagBundle has no concept of file
permissions, so a Go bundle synced through one always loses the execute
bit a build produced. ExecutableCoordinator required that bit to scan a
candidate at all, so those bundles were unreachable, with no path short
of a side-channel init container to restore it.

The scanner now considers every regular file and relies on the existing
footer-magic and binary_sha256 checks to identify a bundle. A file that
matches the requested dag_id is marked executable, mirroring its read
bits, before it is launched.

Co-Authored-By: Claude <noreply@anthropic.com>
@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:go-sdk area:task-sdk kind:documentation labels Oct 5, 2026
@jason810496 jason810496 self-assigned this Oct 5, 2026

@guan404ming guan404ming left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice, just left some comments!

Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py Outdated
jason810496 and others added 2 commits October 6, 2026 01:53
…as skipped

An already-executable bundle whose mode is not exactly the 'read implies
execute' pattern this function produces (0o744, say) was rewritten anyway,
which could fail on a mount where chmod itself is refused even though the
file already runs fine. Check os.access first and return early.

Mark executable only after the candidate has fully passed schema
validation, not before, so a bundle that gets rejected is never touched.
A chmod failure is now recorded in the same 'rejected' list the metadata
checks already use, so it surfaces in the raised error instead of being
indistinguishable from no match at all.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review, they're all valid concerns, I just addressed them in the latest commits.

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@jason810496
jason810496 merged commit 6b6b1fa into main Oct 6, 2026
108 checks passed
@jason810496
jason810496 deleted the fix/coordinator/executable-bundle-exec-bit branch October 6, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:go-sdk area:task-sdk kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants