Fix Beam async hook launching pipelines through a shell - #72510
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
|
this must wait for #66952 |
dff9c46 to
145d86c
Compare
145d86c to
db09881
Compare
The test passes on main too, because shlex.quote plus sh already keeps an argument with spaces together, and it spawns a real interpreter. test_run_beam_command_async_uses_exec_with_argv already covers the switch to create_subprocess_exec. Generated-by: Claude Opus 5
db09881 to
bfba8a0
Compare
asyncio's Process.returncode is typed int | None, so assigning it to the int inferred from the OSError branch failed mypy-providers. Generated-by: Claude Opus 5
potiuk
left a comment
There was a problem hiding this comment.
Thanks for waiting on #66952, and for the fix. I've rebased the branch onto current main (no conflicts) and pushed small follow-up commits on top of yours:
- Dropped
test_run_beam_command_async_preserves_arguments_with_spaces. It also passes onmainon Linux/macOS, becauseshlex.quoteplusshalready keeps"hello world"together, so it doesn't exercise the change, and it starts a real interpreter.test_run_beam_command_async_uses_exec_with_argvalready covers the behaviour. - Fixed a mypy error in
_beam_version(dcda56fba2):asyncio'sProcess.returncodeis typedint | None, soreturncodeneeded that annotation; CI'smypy-providerscaught it once the branch was on currentmain.
The change itself looks right to me. create_subprocess_exec(*cmd, cwd=working_directory) passes argv through unchanged, which matches what the sync run_beam_command already does with Popen(cmd, shell=False). The environment is inherited in both cases, and the logged command line is unchanged. A missing interpreter in run_beam_command_async now raises FileNotFoundError, which matches the sync hook, and the triggers already turn any exception into an error event.
Merging once CI is green.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
BeamAsyncHookstarted pipeline processes by joining the argv list with POSIXshlexquoting and running the result through a shell. The sync hook already usessubprocess.Popenwithshell=Falseand the original list. The async path could therefore split interpreter or pipeline paths that contain spaces, especially on Windows where POSIX quoting is not what the shell expects.This runs
_beam_versionandrun_beam_command_asyncwithasyncio.create_subprocess_execso each argument stays one argv entry. Missing interpreters still raiseAirflowException. Tests assert exec is used (not shell) and that an argument containing spaces survives a live process.Was generative AI tooling used to co-author this PR?
Generated-by: Grok following the guidelines
pythonverification ofrun_beam_command_asyncwith a spaced argument, exec-vs-shell construction, and_beam_versionerror wrapping. Full provider suite via CI:providers/apache/beam/tests/unit/apache/beam/hooks/test_beam.py.