Skip to content

Remove the task_map table, folding its length into xcom.mapped_length - #73005

Merged
ashb merged 1 commit into
apache:mainfrom
astronomer:worktree-drop-task-map
Sep 24, 2026
Merged

ashb merged 1 commit into
apache:mainfrom
astronomer:worktree-drop-task-map

Conversation

@ashb

@ashb ashb commented Sep 11, 2026

Copy link
Copy Markdown
Member

The task_map table was created just to store the number of results (length) that a task upstream of a mapped task created. This is then used by the scheduler to know how many mapped tasks to expand to.

The keys column has never been written to, it was probably to let people iterate over tasks that returned dictionaries, but well, that turned out to not be required.

This is a pre-cursor tidy up to make the PRs for task loops (AIP-111) and after that dynamic sub-graphs (AIP-113) simpler.

The wire protocol for both public API and Exec API remain unchanged, this storage on the DB table was purely a server-side storage decision.

Comment thread airflow-core/src/airflow/models/xcom.py
Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/xcoms.py
Comment thread devel-common/src/tests_common/test_utils/mapping.py Outdated
@ashb
ashb force-pushed the worktree-drop-task-map branch from 2018749 to ef67497 Compare September 14, 2026 12:27
@ashb
ashb requested a review from kaxil September 14, 2026 12:37
@ashb
ashb force-pushed the worktree-drop-task-map branch 2 times, most recently from c8c2bf1 to 547e2c4 Compare September 14, 2026 14:50

@kaxil kaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Nothing tests the backfill though, and it's the one spot where an existing DagRun's expansion length either survives the upgrade or quietly disappears. airflow-core/tests/unit/migrations/ already has seven (test_0131 is the preceding 3.4.0 data migration), and test_0117 shows the pattern, load the module by path and run the real statements against an isolated table. Worth seeding a TI with a return_value row plus a second key, where only return_value should pick up the length, and a return_value row with no task_map row, which should stay NULL through the EXISTS.

You'll also want a rebase before this goes in: main picked up 0134_3_4_0_add_dag_draining_state.py with down_revision = "f8c2a1d94e03", the same parent as this one, so post-merge there'd be two alembic heads and db migrate stops. Filenames differ, so git won't flag it as a conflict.

@ashb
ashb force-pushed the worktree-drop-task-map branch from 547e2c4 to bb507b6 Compare September 22, 2026 13:35
@ashb ashb added the area:core label Sep 22, 2026
@ashb
ashb force-pushed the worktree-drop-task-map branch from bb507b6 to 0fd38a2 Compare September 23, 2026 16:40
The task_map table was created (by me and TP) years ago to store the
number of results (length) that a task upstream of a mapped task
created. This is then used by the scheduler to know how many mapped
tasks to expand to.

The `keys` column has never been read from (and only got written to in
2.x, but not in any 3.x), it was probably to let people iterate over
tasks that returned dictionaries, but well, that turned out to not be
required.

This is a pre-cursor tidy up to make the PRs for task loops (AIP-111) and
after that dynamic sub-graphs (AIP-113) simpler.

The wire protocol for both public API and Exec API remain unchanged, this
storage on the DB table was purely a server-side storage decision.
@ashb
ashb force-pushed the worktree-drop-task-map branch from 0fd38a2 to 0fb08ca Compare September 23, 2026 20:26
@ashb
ashb merged commit 3deb27e into apache:main Sep 24, 2026
302 of 304 checks passed
@ashb
ashb deleted the worktree-drop-task-map branch September 24, 2026 11:48
dabla added a commit to dabla/airflow that referenced this pull request Sep 24, 2026
…removal

Main folded the task_map table into xcom.mapped_length (apache#73005). The runtime
batch size was read from that table, so the scheduler-side lookup now reads
the mapped_length of the size task's return_value row, the same query main
uses for a plain task's mapped length, and the tests seed and expand through
main's helpers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rjgoyln added a commit to rjgoyln/airflow that referenced this pull request Sep 27, 2026
The task_map table is gone from main (apache#73005) and an unmapped upstream's
length now lives on its map_index -1 XCom row, where a NULL is as
unresolved as a missing row used to be.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core area:db-migrations PRs with DB migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants