[safaehar]

airflow db clean --skip-archive currently copies every selected batch into an archive table before deleting the source rows and dropping the archive table. On large metadata database cleanups, this extra copy generates substantial WAL on top of the delete and can cause sharp disk-usage spikes. It also requires schema CREATE/DROP privileges even though the archive is not preserved.

This change gives --skip-archive a direct-delete path:

• PostgreSQL and SQLite delete primary keys selected by the existing limited cleanup query.
• MySQL uses a derived primary-key selection with EXISTS, avoiding MySQL's unsupported LIMIT in an IN subquery.
• Each batch remains a separate transaction and retains the current selection, keep-last, composite-primary-key, rollback, and progress-output behavior.
• Cleanup without --skip-archive continues to create and retain archive tables.

The CLI help and cleanup documentation now state that --skip-archive deletes directly without creating archive tables.

PostgreSQL WAL comparison

On a disposable PostgreSQL 13 database, deleting 1,000,000 rows from the same two-column table and measuring with pg_wal_lsn_diff(pg_current_wal_lsn(), :start_lsn) produced:

• archive path (CREATE TABLE AS, joined DELETE, DROP): 172,750,560 bytes
• direct-delete path: 100,445,792 bytes

The direct path generated approximately 42% less WAL in this comparison.

Tests

• uv run --project airflow-core pytest airflow-core/tests/unit/utils/test_db_cleanup.py --with-db-init -xq (SQLite: 100 passed, 3 backend-specific skipped)
• Focused MySQL 8.0 run in the published Airflow CI image covering composite XCom keys without batching and with batch_size=2 (2 passed)
• Focused PostgreSQL run covering skip-archive batching, keep-last, composite XCom keys, archive regression, and failure handling (11 passed)
• prek run --from-ref upstream/main --stage pre-commit

---

Was generative AI tooling used to co-author this PR?

• Yes (Codex)

Generated-by: Codex following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.

[SameerMesiah97]

Has the new direct-delete path been tested against MySQL, both with and without batching? We should verify that the derived-table query works in both cases, including composite primary keys.

Also, I believe the CI failure should be fixed by PR #74122

[ferruzzi]

Nice approach. Skipping the archive entirely instead of create-then-drop also removes the need for that finally and the workaround from #66177.

There's going to be a conflict with #73568 which adds some warnings int he same block you are editing, would you consider putting this in draft until that one merges and them rebasing on top of it? #73417 is also in the same block so there's a lot of churn in there right now.

Edit: actually, looking closer, this is nearly identical to the change I asked for in #73417 when I asked them to split theirs into two PRs. I'd prefer if we merged #73568, then this one, then (assuming mallesh splits the chart changes out) the chart portion of #73417

[safaehar]

Has the new direct-delete path been tested against MySQL, both with and without batching? We should verify that the derived-table query works in both cases, including composite primary keys.

Also, I believe the CI failure should be fixed by PR #74122

@SameerMesiah97
Thanks, this is now covered. I parameterized the composite-primary-key XCom regression test to run both without batching (batch_size=None) and with batching (batch_size=2). Both cases pass against MySQL 8.0 using Airflow’s published CI image, exercising the derived-table EXISTS delete in both modes.

I also rebased onto the latest main. The fresh CI run is in progress

[hussein-awala]

Could we keep the transaction boundary in the existing batch orchestration rather than committing inside _delete_directly()?

The existing cleanup loop already commits batches, so this is not a new behavioral regression, but the new helper could remain responsible only for executing the DELETE while its caller preserves the commit-per-batch behavior.

[safaehar]

adressed !

[ferruzzi]

#73568 merged. Ping me when you get the CI passing on this one and I'll give it a proper review. Thanks for your patience!

[safaehar]

#73568 merged. Ping me when you get the CI passing on this one and I'll give it a proper review. Thanks for your patience!

@ferruzzi CI passed, thanks!

[ferruzzi]

As written, this continue is right for the fix but will cause another because it makes the whole skip_if_referenced block around L613 unreachable.

One fix would be to have _delete_directly return the number of rows deleted and compare that against a count of the selection, warning when it deleted fewer than it found. deleted == 0 won't do, since L597-600 rejects it for staying silent on a partial skip. You can throw the warning at L604 into a helper that both paths can call.

[safaehar]

the continue is intentional here. The direct DELETE embeds limited_query, including its skip_if_referenced guard, so the condition is reevaluated within the DELETE itself. Unlike the archive path, there is no separately copied row that can remain both archived and live. Adding a count would introduce another query per batch and a new count/delete race. The regression test covers both whole and partial direct skips. I’ll add a comment to make this distinction clearer.

[ferruzzi]

It would likely be more efficient to build the table once and pass it into this method, no? Maybe I'm missing a reason to redo it each pass like this?

[safaehar]

Agreed. the source table does not change between batches, so reflecting it each time is unnecessary. I’ll cache the reflected table in _do_delete() and pass it to _delete_directly(), while keeping the initialization lazy so an empty cleanup does not perform reflection