Repository navigation
Resolve loop iterations through family and path columns on dynamic regions - #74892
Merged
Merged
Conversation
…gions First off: why is this needed, wasn't loops landed and working? "Yes, but." Part of the goal with the data-modelling changes for task loops (AIP-111) is to support dynamic graphs (AIP-113) and other dynamic things, such as choosing to allow loops inside loops (currently a parse error), with no future migrations or data modelling changes needed. This change lets us enable these features without expensive recursive queries in TIDep evaulation etc. And why now? By getting this change in before 3.4.0 enters the RC stage, while we can still change the migration "at zero-cost". And adding it as a migration in 3.5 would involve a (possibly costly) migration. So the meat of the change: Finding the task instances that sit inside one iteration of a loop currently means finding every region of the loop's fork chain (a fork is when a loop is cleared part way through and some iterations run a-fresh), plus everything nested under that iteration, however deep. The previous lookup resolved those region ids with two recursive CTEs and then passed them to the task_instance query as literal lists. The nested list holds every region under the iteration, whichever task is being read, so once something under an iteration can fan out again (nested loops, or dynamic graphs inside a loop) its size grows with the product of those widths. Every lookup hits this path: the scheduler's trigger-rule check of a loop task repeats it on each pass until the task is ready, and every XCom read from inside a loop pays it on the API server. Measured with 5,000 regions under one iteration, a lookup that returns one row took about 30 ms on PostgreSQL and MySQL, almost all of it building, sending and parsing the list; with 50,000 it took about 550 ms. Two columns on dynamic_region lets us get the same result but without recursion. `family_root_id` is the id of the first region of a fork chain, so a chain is one equality column comparisson. `path` contains, for each ancestor from the top, its family root and the index the region sits at inside it, so everything under one iteration of a loop shares a byte prefix and is one range of an index. The lookup becomes a UNION ALL of two joins from task_instance to dynamic_region, each driven by its own index; a single OR across the two legs gives the planner no such choice. Its cost no longer depends on how many regions sit under the iteration. The hook that fills the columns rejects a fork whose parent differs from its source's, because both the family equality and the prefix range rely on every member of a chain sharing one position. Region inserts are rare next to task instance reads and writes, so the hook and the two extra indexes cost comparitavely little. The columns are NOT NULL so that raw inserts which skip the ORM fail loudly instead of writing rows the lookup would silently miss. They are added to the migration that creates dynamic_region, which has only shipped in a beta, so no backfill is needed. A 3.4.0b1 database already ran that migration without them so users in that situation will need to downgrade to 3.3 and upgrade again, or reset the database. Nesting is capped at ten ancestors, which sets the 200-byte path width on MySQL. The cap is a chosen bound rather than an index limit, and authoring already rejects nested loops, so current real depth is far lower. In order to allow more easily resizing this column on MySQL in the future if we want or need to, the length needs to be over 255 bytes (as InnoDB stores VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen 300, allowing 15 levels, such that we can change it in future with a meta-data only storage change. Regardless of max length of column, all DBs only use as much storage as needed (+ size counter).
boring-cyborg
Bot
added
area:API
kaxil
reviewed
Oct 16, 2026
kaxil
approved these changes
Oct 16, 2026
Contributor
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 6a925a8 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Contributor
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 18, 2026
…n dynamic regions (#74892) (#74963) First off: why is this needed, wasn't loops landed and working? "Yes, but." Part of the goal with the data-modelling changes for task loops (AIP-111) is to support dynamic graphs (AIP-113) and other dynamic things, such as choosing to allow loops inside loops (currently a parse error), with no future migrations or data modelling changes needed. This change lets us enable these features without expensive recursive queries in TIDep evaulation etc. And why now? By getting this change in before 3.4.0 enters the RC stage, while we can still change the migration "at zero-cost". And adding it as a migration in 3.5 would involve a (possibly costly) migration. So the meat of the change: Finding the task instances that sit inside one iteration of a loop currently means finding every region of the loop's fork chain (a fork is when a loop is cleared part way through and some iterations run a-fresh), plus everything nested under that iteration, however deep. The previous lookup resolved those region ids with two recursive CTEs and then passed them to the task_instance query as literal lists. The nested list holds every region under the iteration, whichever task is being read, so once something under an iteration can fan out again (nested loops, or dynamic graphs inside a loop) its size grows with the product of those widths. Every lookup hits this path: the scheduler's trigger-rule check of a loop task repeats it on each pass until the task is ready, and every XCom read from inside a loop pays it on the API server. Measured with 5,000 regions under one iteration, a lookup that returns one row took about 30 ms on PostgreSQL and MySQL, almost all of it building, sending and parsing the list; with 50,000 it took about 550 ms. Two columns on dynamic_region lets us get the same result but without recursion. `family_root_id` is the id of the first region of a fork chain, so a chain is one equality column comparisson. `path` contains, for each ancestor from the top, its family root and the index the region sits at inside it, so everything under one iteration of a loop shares a byte prefix and is one range of an index. The lookup becomes a UNION ALL of two joins from task_instance to dynamic_region, each driven by its own index; a single OR across the two legs gives the planner no such choice. Its cost no longer depends on how many regions sit under the iteration. The hook that fills the columns rejects a fork whose parent differs from its source's, because both the family equality and the prefix range rely on every member of a chain sharing one position. Region inserts are rare next to task instance reads and writes, so the hook and the two extra indexes cost comparitavely little. The columns are NOT NULL so that raw inserts which skip the ORM fail loudly instead of writing rows the lookup would silently miss. They are added to the migration that creates dynamic_region, which has only shipped in a beta, so no backfill is needed. A 3.4.0b1 database already ran that migration without them so users in that situation will need to downgrade to 3.3 and upgrade again, or reset the database. Nesting is capped at ten ancestors, which sets the 200-byte path width on MySQL. The cap is a chosen bound rather than an index limit, and authoring already rejects nested loops, so current real depth is far lower. In order to allow more easily resizing this column on MySQL in the future if we want or need to, the length needs to be over 255 bytes (as InnoDB stores VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen 300, allowing 15 levels, such that we can change it in future with a meta-data only storage change. Regardless of max length of column, all DBs only use as much storage as needed (+ size counter). (cherry picked from commit 6a925a8) Co-authored-by: Ash Berlin-Taylor <ash@astronomer.io>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 20, 2026
…n dynamic regions (#74892) (#74963) First off: why is this needed, wasn't loops landed and working? "Yes, but." Part of the goal with the data-modelling changes for task loops (AIP-111) is to support dynamic graphs (AIP-113) and other dynamic things, such as choosing to allow loops inside loops (currently a parse error), with no future migrations or data modelling changes needed. This change lets us enable these features without expensive recursive queries in TIDep evaulation etc. And why now? By getting this change in before 3.4.0 enters the RC stage, while we can still change the migration "at zero-cost". And adding it as a migration in 3.5 would involve a (possibly costly) migration. So the meat of the change: Finding the task instances that sit inside one iteration of a loop currently means finding every region of the loop's fork chain (a fork is when a loop is cleared part way through and some iterations run a-fresh), plus everything nested under that iteration, however deep. The previous lookup resolved those region ids with two recursive CTEs and then passed them to the task_instance query as literal lists. The nested list holds every region under the iteration, whichever task is being read, so once something under an iteration can fan out again (nested loops, or dynamic graphs inside a loop) its size grows with the product of those widths. Every lookup hits this path: the scheduler's trigger-rule check of a loop task repeats it on each pass until the task is ready, and every XCom read from inside a loop pays it on the API server. Measured with 5,000 regions under one iteration, a lookup that returns one row took about 30 ms on PostgreSQL and MySQL, almost all of it building, sending and parsing the list; with 50,000 it took about 550 ms. Two columns on dynamic_region lets us get the same result but without recursion. `family_root_id` is the id of the first region of a fork chain, so a chain is one equality column comparisson. `path` contains, for each ancestor from the top, its family root and the index the region sits at inside it, so everything under one iteration of a loop shares a byte prefix and is one range of an index. The lookup becomes a UNION ALL of two joins from task_instance to dynamic_region, each driven by its own index; a single OR across the two legs gives the planner no such choice. Its cost no longer depends on how many regions sit under the iteration. The hook that fills the columns rejects a fork whose parent differs from its source's, because both the family equality and the prefix range rely on every member of a chain sharing one position. Region inserts are rare next to task instance reads and writes, so the hook and the two extra indexes cost comparitavely little. The columns are NOT NULL so that raw inserts which skip the ORM fail loudly instead of writing rows the lookup would silently miss. They are added to the migration that creates dynamic_region, which has only shipped in a beta, so no backfill is needed. A 3.4.0b1 database already ran that migration without them so users in that situation will need to downgrade to 3.3 and upgrade again, or reset the database. Nesting is capped at ten ancestors, which sets the 200-byte path width on MySQL. The cap is a chosen bound rather than an index limit, and authoring already rejects nested loops, so current real depth is far lower. In order to allow more easily resizing this column on MySQL in the future if we want or need to, the length needs to be over 255 bytes (as InnoDB stores VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen 300, allowing 15 levels, such that we can change it in future with a meta-data only storage change. Regardless of max length of column, all DBs only use as much storage as needed (+ size counter). (cherry picked from commit 6a925a8) Co-authored-by: Ash Berlin-Taylor <ash@astronomer.io>
SameerMesiah97
pushed a commit
to SameerMesiah97/airflow-oss-sm97
that referenced
this pull request
Oct 27, 2026
…gions (apache#74892) First off: why is this needed, wasn't loops landed and working? "Yes, but." Part of the goal with the data-modelling changes for task loops (AIP-111) is to support dynamic graphs (AIP-113) and other dynamic things, such as choosing to allow loops inside loops (currently a parse error), with no future migrations or data modelling changes needed. This change lets us enable these features without expensive recursive queries in TIDep evaulation etc. And why now? By getting this change in before 3.4.0 enters the RC stage, while we can still change the migration "at zero-cost". And adding it as a migration in 3.5 would involve a (possibly costly) migration. So the meat of the change: Finding the task instances that sit inside one iteration of a loop currently means finding every region of the loop's fork chain (a fork is when a loop is cleared part way through and some iterations run a-fresh), plus everything nested under that iteration, however deep. The previous lookup resolved those region ids with two recursive CTEs and then passed them to the task_instance query as literal lists. The nested list holds every region under the iteration, whichever task is being read, so once something under an iteration can fan out again (nested loops, or dynamic graphs inside a loop) its size grows with the product of those widths. Every lookup hits this path: the scheduler's trigger-rule check of a loop task repeats it on each pass until the task is ready, and every XCom read from inside a loop pays it on the API server. Measured with 5,000 regions under one iteration, a lookup that returns one row took about 30 ms on PostgreSQL and MySQL, almost all of it building, sending and parsing the list; with 50,000 it took about 550 ms. Two columns on dynamic_region lets us get the same result but without recursion. `family_root_id` is the id of the first region of a fork chain, so a chain is one equality column comparisson. `path` contains, for each ancestor from the top, its family root and the index the region sits at inside it, so everything under one iteration of a loop shares a byte prefix and is one range of an index. The lookup becomes a UNION ALL of two joins from task_instance to dynamic_region, each driven by its own index; a single OR across the two legs gives the planner no such choice. Its cost no longer depends on how many regions sit under the iteration. The hook that fills the columns rejects a fork whose parent differs from its source's, because both the family equality and the prefix range rely on every member of a chain sharing one position. Region inserts are rare next to task instance reads and writes, so the hook and the two extra indexes cost comparitavely little. The columns are NOT NULL so that raw inserts which skip the ORM fail loudly instead of writing rows the lookup would silently miss. They are added to the migration that creates dynamic_region, which has only shipped in a beta, so no backfill is needed. A 3.4.0b1 database already ran that migration without them so users in that situation will need to downgrade to 3.3 and upgrade again, or reset the database. Nesting is capped at ten ancestors, which sets the 200-byte path width on MySQL. The cap is a chosen bound rather than an index limit, and authoring already rejects nested loops, so current real depth is far lower. In order to allow more easily resizing this column on MySQL in the future if we want or need to, the length needs to be over 255 bytes (as InnoDB stores VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen 300, allowing 15 levels, such that we can change it in future with a meta-data only storage change. Regardless of max length of column, all DBs only use as much storage as needed (+ size counter).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First off: why is this needed, wasn't loops landed and working? "Yes, but."
Part of the goal with the data-modelling changes for task loops (AIP-111)
is to support dynamic graphs (AIP-113) and other dynamic things, such as
choosing to allow loops inside loops (currently a parse error), with no future
migrations or data modelling changes needed. This change lets us enable these
features without expensive recursive queries in TIDep evaulation etc. And why
now? By getting this change in before 3.4.0 enters the RC stage, while we can
still change the migration "at zero-cost". And adding it as a migration in 3.5
would involve a (possibly costly) migration.
So the meat of the change:
Finding the task instances that sit inside one iteration of a loop currently
means finding every region of the loop's fork chain (a fork is when a loop is
cleared part way through and some iterations run a-fresh), plus everything
nested under that iteration, however deep. The previous lookup resolved those
region ids with two recursive CTEs and then passed them to the task_instance
query as literal lists. The nested list holds every region under the
iteration, whichever task is being read, so once something under an iteration
can fan out again (nested loops, or dynamic graphs inside a loop) its size
grows with the product of those widths. Every lookup hits this path: the
scheduler's trigger-rule check of a loop task repeats it on each pass until
the task is ready, and every XCom read from inside a loop pays it on the API
server. Measured with 5,000 regions under one iteration, a lookup that returns
one row took about 30 ms on PostgreSQL and MySQL, almost all of it building,
sending and parsing the list; with 50,000 it took about 550 ms.
Two columns on dynamic_region lets us get the same result but without
recursion.
family_root_idis the id of the first region of a fork chain, soa chain is one equality column comparisson.
pathcontains, for each ancestorfrom the top, its family root and the index the region sits at inside it, so
everything under one iteration of a loop shares a byte prefix and is one range
of an index. The lookup becomes a UNION ALL of two joins from task_instance
to dynamic_region, each driven by its own index; a single OR across the two
legs gives the planner no such choice. Its cost no longer depends on how many
regions sit under the iteration. The hook that fills the columns rejects a
fork whose parent differs from its source's, because both the family equality
and the prefix range rely on every member of a chain sharing one position.
Region inserts are rare next to task instance reads and writes, so the hook
and the two extra indexes cost comparitavely little.
The columns are NOT NULL so that raw inserts which skip the ORM fail loudly
instead of writing rows the lookup would silently miss. They are added to the
migration that creates dynamic_region, which has only shipped in a beta, so no
backfill is needed. A 3.4.0b1 database already ran that migration without them
so users in that situation will need to downgrade to 3.3 and upgrade again,
or reset the database. Nesting is capped at ten ancestors, which sets the
200-byte path width on MySQL. The cap is a chosen bound rather than an index
limit, and authoring already rejects nested loops, so current real depth is far lower.
In order to allow more easily resizing this column on MySQL in the future if
we want or need to, the length needs to be over 255 bytes (as InnoDB stores
VARBINARY with shorter values with a 1 byte length prefix.) So we have chosen
300, allowing 15 levels, such that we can change it in future with a meta-data
only storage change. Regardless of max length of column, all DBs only use as
much storage as needed (+ size counter).
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.