Repository navigation
Add --asset-ids filtering to db clean - #74498
Conversation
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
There was a problem hiding this comment.
🟢 Approval recommended
The implementation is scoped, preserves existing cleanup semantics, and has thorough regression coverage.
0 open findings
What changed in this PR
Adds asset-specific retention filtering to airflow db clean.
Changes:
- Adds validated
--asset-idsCLI filtering forasset_event. - Applies request-local asset predicates while preserving existing cleanup behavior.
- Adds comprehensive tests and usage documentation.
| File | Description |
|---|---|
airflow-core/src/airflow/cli/cli_config.py |
Defines the new CLI option. |
airflow-core/src/airflow/cli/commands/db_command.py |
Forwards parsed asset IDs. |
airflow-core/src/airflow/utils/db_cleanup.py |
Validates and applies asset filtering. |
airflow-core/tests/unit/cli/commands/test_db_command.py |
Tests parsing, forwarding, and validation. |
airflow-core/tests/unit/utils/test_db_cleanup.py |
Tests filtering, archival, cascades, and isolation. |
airflow-core/docs/howto/usage-cli.rst |
Documents asset-specific cleanup. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Eason09053360
left a comment
There was a problem hiding this comment.
Thanks for the patch! An old event of a that is still queued for a Dag scheduled on a & b gets deleted along with its asset_dag_run_queue row, so that Dag no longer runs when b arrives. #74495 refuses this case by default (409). Should --asset-ids skip queued events too? Otherwise LGTM.
I don’t think so. The REST API endpoint is more than just a cleanup utility, unlike what we’re doing here. In this PR, I’m simply adding a filter to limit the cleanup to a specific set of Assets, without changing the existing cleanup behavior. The new endpoint introduced in the other PR, on the other hand, is an entirely new feature. However, if you think the additional filter would be useful, feel free to open a feature request or submit a PR to implement it. |
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data. (cherry picked from commit b9ae12a) Co-authored-by: Hussein Awala <hussein@awala.fr>
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data. (cherry picked from commit b9ae12a) Co-authored-by: Hussein Awala <hussein@awala.fr>
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data. (cherry picked from commit b9ae12a) Co-authored-by: Hussein Awala <hussein@awala.fr>
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data. (cherry picked from commit b9ae12a) Co-authored-by: Hussein Awala <hussein@awala.fr>
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data. (cherry picked from commit b9ae12a) Co-authored-by: Hussein Awala <hussein@awala.fr>
Summary
Add
--asset-idstoairflow db clean, following the existing Dag-ID filtering pattern:airflow db clean \ --tables asset_event \ --asset-ids 1,2 \ --clean-before-timestamp '2024-01-01T00:00:00+00:00' \ --dry-runAdministrators can apply different event-retention policies to individual assets without cleaning every asset's history.
--tables asset_eventselection. Omitted, mixed, or unrelated table selections fail before cleanup.The diff is limited to CLI option/forwarding, request-local cleanup configuration, focused tests, and a short documentation example. It adds no new deletion engine, REST behavior, migrations, dependencies, or queue-deletion policy.
Validation
main.git range-diffconfirmed the patch was unchanged by the rebase.Broader verification note
The database suite selected by
breeze verifypassed. The post-rebase non-database run reported 4,813 passed, 11 skipped, 2 xfailed, and 2 failures in the unchangedtest_the_runtime_inherits_only_its_standard_streams[fork/spawn]language-SDK tests.Both failures were reproduced on a clean worktree at the original upstream base (
86ebf1c8a2) with the matching Breeze non-database harness. They report an unexpected/usr/bin/dashfile descriptor. No unrelated test or runtime code was changed to hide these baseline/local-environment failures. MySQL was not exercised.Was generative AI tooling used to co-author this PR?
Generated-by: Pi 0.84.3 following the guidelines