Repository navigation
Skip scheduling tasks when their Dag cannot be deserialized - #75263
vatsrahul1001 merged 3 commits into
Conversation
Keep one Dag's deserialization failure from stopping scheduler progress for unrelated Dags.
|
please review this and merge this pr after the check passes |
|
Hi maintainers, the Could someone please confirm whether this is expected or if the check is stuck? Thanks! |
luc-pimentel
left a comment
luc-pimentel
left a comment
There was a problem hiding this comment.
Thanks for picking up the issue I opened. The change mirrors the guard #74499 added for queued runs, so it fits the existing pattern. I ran the issue's reproduction against this PR's current head (after the merge of main from October 22) in Breeze on Python 3.11 with SQLite: a plugin timetable that stops deserializing, two runs of a Dag with max_active_tis_per_dag=1 so the second run's task waits in scheduled, every process stopped, then airflow scheduler --num-runs 8 alone.
- Timetable broken: the scheduler finishes its loops, logs "Failed to load Dag probe_bad_long for task instance ..., skipping it" once per loop, and the task stays scheduled. On main this exits with the
DeserializationErrortraceback from_task_concurrency_allows_execution. - Restart with the timetable still broken: same, no crash. On main it crashes again.
- Timetable fixed again: the task is queued.
It seems like this fixes what I reported.
Drafted-by: Claude Code 2.1.310 (Claude Fable 5.1); reviewed by @luc-pimentel before posting
amoghrajesh
left a comment
amoghrajesh
left a comment
There was a problem hiding this comment.
Thanks for working on this @abhijeets25012-tech. Have some feedback
Isolate deserialization failures to the affected DAG run instead of starving the whole DAG, and let DB errors propagate so the scheduler does not silently skip task instances.
amoghrajesh
left a comment
amoghrajesh
left a comment
There was a problem hiding this comment.
LGTM +1. @ephraimbuddy?
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker e3f5e6f v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker e3f5e6f v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
test_find_executable_task_instances_skips_dag_that_cannot_be_deserialized fails on v3-4-test independently of this backport. apache#75745 brought the guard over from apache#75263 without the starved set that apache#75263 relies on, because that set came with apache#70307, which is not on this branch. The guard then returns False without narrowing the next query, so a cycle whose batch is filled by the bad Dag ends with nothing queued. Co-Authored-By: Claude <noreply@anthropic.com>
…5795) (#75802) * [v3-4-test] Load the serialized Dag in ti_run only for stub tasks (#75795) Starting any task deserialized its whole Dag so the route could find out whether the task was a stub and read its arg_bindings. Denormalize the serialized is_stub flag onto task_instance, which ti_run already fetches and locks, and load the Dag only when it is set. * Mark a Go or Java TriggerDagRun task a stub, as TypeScript does * Inline the table name in the is_stub migration * Starve a Dag the scheduler cannot deserialize test_find_executable_task_instances_skips_dag_that_cannot_be_deserialized fails on v3-4-test independently of this backport. #75745 brought the guard over from #75263 without the starved set that #75263 relies on, because that set came with #70307, which is not on this branch. The guard then returns False without narrowing the next query, so a cycle whose batch is filled by the bad Dag ends with nothing queued. ---------
When the scheduler checks task concurrency, it loads the Dag for each scheduled task instance. If a Dag cannot be deserialized, that exception can stop scheduling and cause the scheduler to crash again on restart.
This change catches and logs non-database errors during that load, defers the affected task for the current selection pass, and allows the scheduler to continue processing other Dags. Database errors still propagate so the existing transaction retry behavior is preserved.
Adds a regression test confirming that the affected task remains scheduled while a healthy Dag’s task is selected.
closes: #75254
Testing
airflow-core/tests/unit/jobs/test_scheduler_job.py::TestSchedulerJob::test_find_executable_task_instances_skips_dag_that_cannot_be_deserializedairflow-core/tests/unit/jobs/test_scheduler_job.py::TestSchedulerJob::test_start_queued_dagruns_does_not_crash_on_dag_deserialization_errorprek run --files airflow-core/src/airflow/jobs/scheduler_job_runner.py airflow-core/tests/unit/jobs/test_scheduler_job.pyWas generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-6) following the guidelines