Skip to content

Add an Index on trigger_id to callback table to speed up trigger cleanup lookups - #75228

Open
shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:fix-callback-trigger-id-index
Open

shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:fix-callback-trigger-id-index

Conversation

@shivaam

@shivaam shivaam commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Trigger cleanup checks callback.trigger_id before deleting an unreferenced trigger. Unlike the
equivalent task-instance and asset-watcher columns, this callback column has no explicit Airflow
index on PostgreSQL or SQLite, so those checks can scan the callback table.

This is a follow-up to #70961, where the missing callback.trigger_id index was left as separate
work.

This adds a non-unique idx_callback_trigger_id index. PostgreSQL and SQLite use a partial index
containing only non-null trigger references. MySQL uses a full index because it does not support
equivalent partial indexes; InnoDB already creates an index for the foreign key, so the migration
standardizes its name without adding a duplicate index.

The column remains nullable and non-unique. Callback lifecycle and trigger-cleanup SQL are
unchanged.

In a local PostgreSQL benchmark with one million callback rows and one percent non-null references,
the cleanup query improved from 12.1 seconds without an index to about 7.9 milliseconds with either
indexed shape. The partial index was about 29 times smaller than the full index (240 KiB versus 6.8
MiB).

Testing performed:

  • Upgrade, downgrade, and re-upgrade tests on SQLite, PostgreSQL 14, and MySQL 8.0
  • SQLite cleanup-query plan regression test
  • Migration stairway, schema synchronization, migration-head, and pending-upgrade checks
  • Complete changed-file prek suite

Trade-offs:

  • Maintaining the index adds the normal write cost when callback trigger references are inserted
    or changed.
  • The migration uses ordinary CREATE INDEX, consistent with existing Airflow migrations, so it
    may temporarily block conflicting writes while the index is created.

related: #68243


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5) and Claude Code (Opus 5.5)

Generated-by: Codex (GPT-5) and Claude Code (Opus 5.5) following the guidelines

@shivaam
shivaam marked this pull request as ready for review October 20, 2026 16:43
@shivaam shivaam changed the title Speed up trigger cleanup callback lookups Add an Index on trigger_id to callback table to speed up trigger cleanup lookups Oct 20, 2026
@eladkal eladkal added this to the Airflow 3.4.0 milestone Oct 21, 2026
@eladkal eladkal added the backport-to-v3-4-test Backport to v3-4-test label Oct 21, 2026
@eladkal
eladkal requested a review from kaxil October 21, 2026 07:36
@kaxil

kaxil commented Oct 21, 2026

Copy link
Copy Markdown
Member

Also paging experts on this @dstandish @seanmuth

@eladkal eladkal modified the milestones: Airflow 3.4.1, Airflow 3.5.0 Oct 28, 2026
@eladkal eladkal removed the backport-to-v3-4-test Backport to v3-4-test label Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants