Skip to content

Add --asset-ids filtering to db clean - #74498

Merged
vatsrahul1001 merged 2 commits into
apache:mainfrom
hussein-awala:db-clean-asset-ids
Oct 24, 2026
Merged

vatsrahul1001 merged 2 commits into
apache:mainfrom
hussein-awala:db-clean-asset-ids

Conversation

@hussein-awala

@hussein-awala hussein-awala commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Add --asset-ids to airflow db clean, following the existing Dag-ID filtering pattern:

airflow db clean \
  --tables asset_event \
  --asset-ids 1,2 \
  --clean-before-timestamp '2024-01-01T00:00:00+00:00' \
  --dry-run

Administrators can apply different event-retention policies to individual assets without cleaning every asset's history.

  • Accept a comma-separated list of positive integer IDs, with whitespace trimming and validation.
  • Require an explicit --tables asset_event selection. Omitted, mixed, or unrelated table selections fail before cleanup.
  • Apply the IDs as a request-local predicate through the existing cleanup query. Timestamp and existing Dag filters still apply.
  • Include the selected IDs in confirmation and dry-run output.
  • Preserve current archive, skip-archive, batch, transaction, and automatic cascade behavior. An explicit empty programmatic selection matches no events; omitting the option leaves existing behavior unchanged.

The diff is limited to CLI option/forwarding, request-local cleanup configuration, focused tests, and a short documentation example. It adds no new deletion engine, REST behavior, migrations, dependencies, or queue-deletion policy.

Validation

  • 26 new cases, all passing on SQLite and PostgreSQL.
  • Full DB command/cleanup test files: 256 passed, 3 skipped on SQLite; 258 passed, 1 skipped on PostgreSQL. Skips are existing backend-specific cases. The SQLite suite also passed after rebasing onto current main.
  • Coverage includes parsing/forwarding, invalid and empty inputs, fail-closed table selection, asset/cutoff scoping, Dag-filter intersections, dry runs, archive/skip-archive, batching, FK cascades, preserved parents, and request-local filter state.
  • Regular and manual prek checks passed, including core mypy; documentation build and spelling checks passed.
  • Independent review found no actionable defects. git range-diff confirmed the patch was unchanged by the rebase.

Broader verification note

The database suite selected by breeze verify passed. The post-rebase non-database run reported 4,813 passed, 11 skipped, 2 xfailed, and 2 failures in the unchanged test_the_runtime_inherits_only_its_standard_streams[fork/spawn] language-SDK tests.

Both failures were reproduced on a clean worktree at the original upstream base (86ebf1c8a2) with the matching Breeze non-database harness. They report an unexpected /usr/bin/dash file descriptor. No unrelated test or runtime code was changed to hide these baseline/local-environment failures. MySQL was not exercised.


Was generative AI tooling used to co-author this PR?
  • Yes — Pi 0.84.3

Generated-by: Pi 0.84.3 following the guidelines

Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation is scoped, preserves existing cleanup semantics, and has thorough regression coverage.

0 open findings

What changed in this PR

Adds asset-specific retention filtering to airflow db clean.

Changes:

  • Adds validated --asset-ids CLI filtering for asset_event.
  • Applies request-local asset predicates while preserving existing cleanup behavior.
  • Adds comprehensive tests and usage documentation.
File Description
airflow-core/​src/​airflow/​cli/​cli_config.py Defines the new CLI option.
airflow-core/​src/​airflow/​cli/​commands/​db_command.py Forwards parsed asset IDs.
airflow-core/​src/​airflow/​utils/​db_cleanup.py Validates and applies asset filtering.
airflow-core/​tests/​unit/​cli/​commands/​test_db_command.py Tests parsing, forwarding, and validation.
airflow-core/​tests/​unit/​utils/​test_db_cleanup.py Tests filtering, archival, cascades, and isolation.
airflow-core/​docs/​howto/​usage-cli.rst Documents asset-specific cleanup.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@hussein-awala
hussein-awala marked this pull request as ready for review October 9, 2026 14:44
@hussein-awala hussein-awala added type:new-feature Changelog: New Features ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. labels Oct 9, 2026
@hussein-awala hussein-awala added this to the Airflow 3.4.0 milestone Oct 9, 2026

@Eason09053360 Eason09053360 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! An old event of a that is still queued for a Dag scheduled on a & b gets deleted along with its asset_dag_run_queue row, so that Dag no longer runs when b arrives. #74495 refuses this case by default (409). Should --asset-ids skip queued events too? Otherwise LGTM.

@hussein-awala

Copy link
Copy Markdown
Member Author

Should --asset-ids skip queued events too?

I don’t think so. The REST API endpoint is more than just a cleanup utility, unlike what we’re doing here.

In this PR, I’m simply adding a filter to limit the cleanup to a specific set of Assets, without changing the existing cleanup behavior. The new endpoint introduced in the other PR, on the other hand, is an entirely new feature.

However, if you think the additional filter would be useful, feel free to open a feature request or submit a PR to implement it.

@vatsrahul1001
vatsrahul1001 enabled auto-merge (squash) October 24, 2026 17:57
@vatsrahul1001
vatsrahul1001 merged commit b9ae12a into apache:main Oct 24, 2026
68 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 24, 2026
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
(cherry picked from commit b9ae12a)

Co-authored-by: Hussein Awala <hussein@awala.fr>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 24, 2026
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
(cherry picked from commit b9ae12a)

Co-authored-by: Hussein Awala <hussein@awala.fr>
vatsrahul1001 pushed a commit that referenced this pull request Oct 26, 2026
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
(cherry picked from commit b9ae12a)

Co-authored-by: Hussein Awala <hussein@awala.fr>
vatsrahul1001 pushed a commit that referenced this pull request Oct 26, 2026
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
(cherry picked from commit b9ae12a)

Co-authored-by: Hussein Awala <hussein@awala.fr>
vatsrahul1001 pushed a commit that referenced this pull request Oct 26, 2026
Administrators need to retain event history independently for different assets without cleaning every asset in the table. Restricting the new filter to an explicit asset_event selection avoids unintentionally cleaning unrelated data.
(cherry picked from commit b9ae12a)

Co-authored-by: Hussein Awala <hussein@awala.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:CLI backport-to-v3-4-test Backport to v3-4-test kind:documentation ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants