Repository navigation
Fix Task SDK dry-run mode crashing on task instance start - #73373
Eason09053360 wants to merge 1 commit into
Conversation
dry_run=True is the documented way to debug a task locally without an API server, but it has been unusable since partition_key landed on DagRun: the hand-written fake response drifted from the generated model, so every start() call ends in a ValidationError and the supervisor kills the child process it has already forked. The drift survived eighteen months because the only test compared the fake payload against a hand-copied duplicate of itself and never parsed it through TIRunContext. Building the fixture from the models instead turns the next missing required field into a mypy error.
|
Hello @Eason09053360 - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 33 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
Why
dry_run=Trueis the Task SDK's documented way to run or debug a task locally without an APIserver. It has been broken for a long time:
noop_handler's hand-writtendag_runpayload hasnot changed since the Task SDK became its own distribution, while the generated
DagRunmodelgained five required fields (
data_interval_start,data_interval_end,end_date,state,partition_key).task_instances.start()raises aValidationErroron the first call, and thesupervisor then kills the child process it has already forked.
What
noop_handlerserves aTIRunContextbuilt from the generated models instead of a literal dict,so the next required field added to
DagRunfailsmypy-task-sdkat the definition rather thansurfacing at runtime under
dry_run.TestClient::test_dry_runis an existing test this PR rewrites, so it is worth saying why: itcompared the payload against a hand-copied duplicate of itself and fetched it with
client.get(),never reaching
TIRunContext.model_validate_json(). That is how the drift shipped — the testpasses on
mainwith the bug live. It now goes throughtask_instances.start()and failswithout the fix.
Also drops a duplicate
datetimeimport in theTYPE_CHECKINGblock thatruffflags (TC004)once
datetimeis used at runtime.How to test
uv run --project task-sdk pytest task-sdk/tests/task_sdk/api/test_client.py -k dry_runWas generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines