Fix airflow db clean hang on MySQL when delete fails by Vamsi-klu · Pull Request #66296 · apache/airflow

Vamsi-klu · 2026-05-03T04:29:12Z

Fixes a MySQL hang in airflow db clean --skip-archive when a delete fails on a foreign-key constraint.

On the failure path, _do_delete() now rolls back the session before dropping the temporary archive table, and the drop uses the session's active connection instead of checking out a new pooled connection. This prevents metadata-lock contention from leaving the cleanup command stuck indefinitely.

Also adds:

unit coverage for rollback-before-drop ordering
a MySQL backend regression test for the deadlock case
a Postgres backend regression test for the failure path

Validation:

prek run --from-ref main --stage pre-commit
prek run --from-ref main --stage manual
targeted MySQL and Postgres regression tests in Breeze
full airflow-core/tests/unit/utils/test_db_cleanup.py in Breeze (57 passed, 1 skipped)
breeze ci selective-check --commit-ref HEAD indicates the shared test utility change expands the CI matrix beyond what was practical to run locally

closes: #66177

Was generative AI tooling used to co-author this PR?

Yes — Codex; Claude Code (Fable 5)

Generated-by: Codex; Claude Code (Fable 5) following the guidelines

Important

🛠️ Maintainer triage note for @Vamsi-klu · by @potiuk · 2026-07-02 17:46 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed (see the Pull Request quality criteria):

This PR has merge conflicts with main. Please rebase onto the latest main and resolve them.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

csrinivas2130 · 2026-05-05T22:26:31Z

@Vamsi-klu can you close your PR, I have made the fix 4 days ago and waiting for the #PR-66249 approval for merge and close this issue.

potiuk · 2026-05-24T18:03:56Z

@Vamsi-klu A few things need addressing before review — see our Pull Request quality criteria.

❌ CI fails: Low dep tests: providers / All-prov:LowestDeps:14:3.10:airbyte...apprise (and possibly other checks — see the Checks tab for the full list).

No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

ephraimbuddy

I think this will solve the hang but the cleanup would still raise on every dag with run history since dagversion is fk-pinned to task_instance.

The newsfragments is not necessary.

aaron-y-chen

Hi, thanks for the fix! While testing this, I found a small edge case where the cleanup path can still mask the original delete error. Would you mind fixing it in this PR as well? :)

The check-newsfragment-pr-number CI job requires the newsfragment filename to match the PR number, not the linked issue number.

Keeps the hang-regression tests meaningful once dag_version cleanup learns to skip FK-pinned rows (apache#68339).

When _do_delete unwinds from a failed DELETE, dropping the archive table in the finally block could itself raise (e.g. a dead connection on MySQL). Python makes a finally-raised error the top-level exception, so that cleanup error would replace the original delete failure -- and an OperationalError from the drop could then be downgraded to a warning by _suppress_with_logging, silently masking the real IntegrityError. Guard the cleanup so it only propagates on the success path; on the failure path it is logged and the original error survives.

ephraimbuddy

We shouldn't have PR numbers on tests doc strings

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>

The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal.

* Fix airflow db clean hang on MySQL when delete fails (apache#66177) * Rename newsfragment to match PR number The check-newsfragment-pr-number CI job requires the newsfragment filename to match the PR number, not the linked issue number. * Remove newsfragment per review * Suppress rollback errors so the original delete error propagates * Explain why the archive table drop reuses the session connection * Force the FK-failure path in backend regression tests via config copy Keeps the hang-regression tests meaningful once dag_version cleanup learns to skip FK-pinned rows (apache#68339). * Guard archive-table cleanup so it can't mask the original delete error When _do_delete unwinds from a failed DELETE, dropping the archive table in the finally block could itself raise (e.g. a dead connection on MySQL). Python makes a finally-raised error the top-level exception, so that cleanup error would replace the original delete failure -- and an OperationalError from the drop could then be downgraded to a warning by _suppress_with_logging, silently masking the real IntegrityError. Guard the cleanup so it only propagates on the success path; on the failure path it is logged and the original error survives. * Update airflow-core/tests/unit/utils/test_db_cleanup.py Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> * Retry CI after provider compat startup failure The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal. --------- Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> Co-authored-by: Ramachandra Nalam <nalamvamsi13@gmail.com>

potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label May 26, 2026

This comment was marked as spam.

Sign in to view

ephraimbuddy reviewed Jun 10, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/utils/db_cleanup.py Outdated

Comment thread airflow-core/src/airflow/utils/db_cleanup.py Outdated

This comment was marked as spam.

Sign in to view

potiuk marked this pull request as draft July 2, 2026 17:47

ephraimbuddy approved these changes Jul 3, 2026

View reviewed changes

aaron-y-chen reviewed Jul 6, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/utils/db_cleanup.py Outdated

Vamsi-klu added 7 commits July 6, 2026 03:52

Fix airflow db clean hang on MySQL when delete fails (apache#66177)

6f9dc25

Rename newsfragment to match PR number

14311cf

The check-newsfragment-pr-number CI job requires the newsfragment filename to match the PR number, not the linked issue number.

Remove newsfragment per review

053b50f

Suppress rollback errors so the original delete error propagates

8d6f67c

Explain why the archive table drop reuses the session connection

eba2b73

Force the FK-failure path in backend regression tests via config copy

ce1cfe1

Keeps the hang-regression tests meaningful once dag_version cleanup learns to skip FK-pinned rows (apache#68339).

This comment was marked as spam.

Sign in to view

ephraimbuddy approved these changes Jul 13, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/utils/test_db_cleanup.py Outdated

Update airflow-core/tests/unit/utils/test_db_cleanup.py

7b35686

Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>

This comment was marked as spam.

Sign in to view

Retry CI after provider compat startup failure

247d707

The previous run failed because the Mongo provider test fixture could not connect to its temporary mongod container. No source change is needed; this commit retriggers CI so the PR can use a fresh signal.

ephraimbuddy merged commit d6cd459 into apache:main Jul 14, 2026
150 checks passed

vatsrahul1001 mentioned this pull request Oct 28, 2026

Status of testing of Apache Airflow 3.4.0rc1 #75865

Open

Navigation Menu

Repository navigation

Fix airflow db clean hang on MySQL when delete fails - #66296

Fix airflow db clean hang on MySQL when delete fails#66296
ephraimbuddy merged 9 commits into
apache:mainfrom
Vamsi-klu:fix-db-clean-mysql-hang-66177

Vamsi-klu commented May 3, 2026 •

edited by potiuk

Loading

Uh oh!

csrinivas2130 commented May 5, 2026 •

edited

Loading

Uh oh!

potiuk commented May 24, 2026

Uh oh!

This comment was marked as spam.

ephraimbuddy left a comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

aaron-y-chen left a comment

Uh oh!

Uh oh!

This comment was marked as spam.

ephraimbuddy left a comment

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Vamsi-klu commented May 3, 2026 • edited by potiuk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

csrinivas2130 commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented May 24, 2026

Uh oh!

This comment was marked as spam.

ephraimbuddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

aaron-y-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as spam.

ephraimbuddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Vamsi-klu commented May 3, 2026 •

edited by potiuk

Loading

csrinivas2130 commented May 5, 2026 •

edited

Loading