Repository navigation
Conversation
Task-group lookups can decompress and deserialize serialized Dags and initialize operator links. This work must stay off the API event loop when database reads use an async session.
The async states route ran ORM entity hydration for every matching task instance and its eagerly joined Dag run on the API event loop, stalling other requests on large mapped runs. Only four columns are consumed. Per-developer agent instruction files must never be committed.
Large async state responses still hold the event loop while processing all rows at once. Follow the review recommendation to keep it available during large requests.
zach-overflow
force-pushed
the
zach.gottesman/async-get-ti-states
branch
2 times, most recently
from
October 5, 2026 13:05
be60f19 to
66be871
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stream task-state results in partitions of 500 to reduce event-loop stalls when the async
/task-instances/statesendpoint handles large runs. Build the response by unpacking rows and assigning states directly, and skip the Dag-wide query when a task group is requested without explicit task IDs.The streamed result is closed before fetching task-group states, including on cancellation or failure. Response fields, mapped-task keys, and the existing task/task-group selection semantics are preserved.
Implements the follow-up proposed in the review comment on apache/airflow#73966. This PR targets
DataDog:zach.gottesman/async-get-ti-states.Validation:
ruffandruff-formatprek hooks passed. Tests, benchmarks, and full Breeze hooks were not run for this PR, as requested. Performance measurements in the linked comment predate this PR; streaming uses a server-side cursor inside the request transaction, and PgBouncer compatibility was not verified.Was generative AI tooling used to co-author this PR?
Generated-by: Codex (GPT-6) following the guidelines