Repository navigation
Skip FK-referenced dag_version rows during db clean - #68339
Merged
ephraimbuddy merged 1 commit intoJun 14, 2026
Merged
ephraimbuddy merged 1 commit into
ephraimbuddy merged 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves airflow db clean behavior for dag_version by preventing deletion attempts of rows that are still referenced by task_instance.dag_version_id (an ON DELETE RESTRICT FK), allowing cleanup to make progress by pruning only orphaned, older DAG versions.
Changes:
- Add a generic
skip_if_referenced(andreferenced_pk_column) option to db cleanup table configuration, implemented as a correlatedNOT EXISTSfilter in_build_query(). - Apply this option to
dag_versionto skip versions still referenced bytask_instance.dag_version_id. - Add a unit regression test to ensure pinned
dag_versionrows are skipped while orphaned old versions are deleted.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
airflow-core/src/airflow/utils/db_cleanup.py |
Adds skip_if_referenced filtering to the cleanup query builder and configures dag_version to use it. |
airflow-core/tests/unit/utils/test_db_cleanup.py |
Adds a regression test verifying dag_version cleanup skips FK-pinned versions but prunes orphaned old ones. |
ephraimbuddy
force-pushed
the
fix-db-clean-dag-version-fk-pinned-66177
branch
2 times, most recently
from
June 10, 2026 17:04
5a53021 to
bec56dc
Compare
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history. Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned. related: apache#66177
ephraimbuddy
force-pushed
the
fix-db-clean-dag-version-fk-pinned-66177
branch
from
June 11, 2026 11:48
bec56dc to
88f8fee
Compare
1 task done
potiuk
approved these changes
Jun 14, 2026
1 of 2 tasks
1 of 2 tasks
pgagnon
pushed a commit
to pgagnon/airflow
that referenced
this pull request
Jun 15, 2026
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history. Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned. related: apache#66177
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Jun 16, 2026
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history. Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned. related: apache#66177
dingo4dev
pushed a commit
to dingo4dev/airflow
that referenced
this pull request
Jun 16, 2026
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history. Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned. related: apache#66177
RulerChen
pushed a commit
to RulerChen/airflow
that referenced
this pull request
Jun 16, 2026
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history. Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned. related: apache#66177
ephraimbuddy
added a commit
that referenced
this pull request
Jul 14, 2026
* Fix airflow db clean hang on MySQL when delete fails (#66177) * Rename newsfragment to match PR number The check-newsfragment-pr-number CI job requires the newsfragment filename to match the PR number, not the linked issue number. * Remove newsfragment per review * Suppress rollback errors so the original delete error propagates * Explain why the archive table drop reuses the session connection * Force the FK-failure path in backend regression tests via config copy Keeps the hang-regression tests meaningful once dag_version cleanup learns to skip FK-pinned rows (#68339). * Guard archive-table cleanup so it can't mask the original delete error When _do_delete unwinds from a failed DELETE, dropping the archive table in the finally block could itself raise (e.g. a dead connection on MySQL). Python makes a finally-raised error the top-level exception, so that cleanup error would replace the original delete failure -- and an OperationalError from the drop could then be downgraded to a warning by _suppress_with_logging, silently masking the real IntegrityError. Guard the cleanup so it only propagates on the success path; on the failure path it is logged and the original error survives. * Update airflow-core/tests/unit/utils/test_db_cleanup.py Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> * Retry CI after provider compat startup failure The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal. --------- Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> Co-authored-by: Ramachandra Nalam <nalamvamsi13@gmail.com>
joshuabvarghese
pushed a commit
to joshuabvarghese/airflow
that referenced
this pull request
Jul 16, 2026
* Fix airflow db clean hang on MySQL when delete fails (apache#66177) * Rename newsfragment to match PR number The check-newsfragment-pr-number CI job requires the newsfragment filename to match the PR number, not the linked issue number. * Remove newsfragment per review * Suppress rollback errors so the original delete error propagates * Explain why the archive table drop reuses the session connection * Force the FK-failure path in backend regression tests via config copy Keeps the hang-regression tests meaningful once dag_version cleanup learns to skip FK-pinned rows (apache#68339). * Guard archive-table cleanup so it can't mask the original delete error When _do_delete unwinds from a failed DELETE, dropping the archive table in the finally block could itself raise (e.g. a dead connection on MySQL). Python makes a finally-raised error the top-level exception, so that cleanup error would replace the original delete failure -- and an OperationalError from the drop could then be downgraded to a warning by _suppress_with_logging, silently masking the real IntegrityError. Guard the cleanup so it only propagates on the success path; on the failure path it is logged and the original error survives. * Update airflow-core/tests/unit/utils/test_db_cleanup.py Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> * Retry CI after provider compat startup failure The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal. --------- Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> Co-authored-by: Ramachandra Nalam <nalamvamsi13@gmail.com>
1 task done
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
airflow db clean on the dag_version table selected old, non-latest versions for deletion regardless of whether they were still referenced. Because task_instance.dag_version_id is ON DELETE RESTRICT, deleting a version still referenced by a task instance fails the foreign key, so the command could not prune dag_version at all for any DAG with history.
Add a generic skip_if_referenced option to the cleanup table config that excludes rows still referenced by a given (table, fk_column) via a correlated NOT EXISTS, and apply it to dag_version for task_instance.dag_version_id. Cleanup now prunes only orphaned older versions and makes progress as task instances age out and are cleaned.
related: #66177
Was generative AI tooling used to co-author this PR?
Generated-by: claude opus 4.8 following the guidelines