Skip to content

Fix N+1 API calls issue in DAG Tasks tab - #59814

Closed
Arunodoy18 wants to merge 1 commit into
apache:mainfrom
Arunodoy18:fix/n-plus-1-query-tasks-tab
Closed

Arunodoy18 wants to merge 1 commit into
apache:mainfrom
Arunodoy18:fix/n-plus-1-query-tasks-tab

Conversation

@Arunodoy18

Copy link
Copy Markdown

This change addresses a critical performance issue where the Tasks tab in the DAG details view triggers N individual API calls for each task to fetch recent task instances, causing severe performance degradation and timeouts for DAGs with 200+ tasks.

Problem:

  • Each TaskCard component independently called the API to fetch its task instances: /api/v2/dags/{dag_id}/dagRuns/~/taskInstances?task_id={task_id}
  • For a DAG with 200 tasks, this resulted in 200+ sequential API calls
  • Backend experienced SQLAlchemy timeouts due to excessive query load
  • UI became unresponsive and unusable for large DAGs

Solution:

  • Modified Tasks.tsx to batch-fetch all task instances for all tasks in a single API call using the existing batch endpoint: POST /api/v2/dags//dagRuns//taskInstances/list
  • Task instances are grouped by task_id and passed as props to TaskCard
  • Eliminated N+1 query pattern, reducing 200+ calls to just 1 call
  • Maintained existing functionality including auto-refresh for pending tasks

Changes:

  • Tasks.tsx: Added batch query using TaskInstanceService.getTaskInstancesBatch() with grouping logic to distribute instances to cards
  • TaskCard.tsx: Modified to accept taskInstances as prop instead of fetching independently

Performance Impact:

  • Reduces API calls from O(N) to O(1) where N is number of tasks
  • For 200 tasks: 200 calls 1 call (99.5% reduction)
  • Eliminates backend timeout issues
  • Significantly improves UI responsiveness for large DAGs

Fixes: #59598

This change addresses a critical performance issue where the Tasks tab
in the DAG details view triggers N individual API calls for each task
to fetch recent task instances, causing severe performance degradation
and timeouts for DAGs with 200+ tasks.

Problem:
- Each TaskCard component independently called the API to fetch its
  task instances: /api/v2/dags/{dag_id}/dagRuns/~/taskInstances?task_id={task_id}
- For a DAG with 200 tasks, this resulted in 200+ sequential API calls
- Backend experienced SQLAlchemy timeouts due to excessive query load
- UI became unresponsive and unusable for large DAGs

Solution:
- Modified Tasks.tsx to batch-fetch all task instances for all tasks
  in a single API call using the existing batch endpoint:
  POST /api/v2/dags/~/dagRuns/~/taskInstances/list
- Task instances are grouped by task_id and passed as props to TaskCard
- Eliminated N+1 query pattern, reducing 200+ calls to just 1 call
- Maintained existing functionality including auto-refresh for pending tasks

Changes:
- Tasks.tsx: Added batch query using TaskInstanceService.getTaskInstancesBatch()
  with grouping logic to distribute instances to cards
- TaskCard.tsx: Modified to accept taskInstances as prop instead of
  fetching independently

Performance Impact:
- Reduces API calls from O(N) to O(1) where N is number of tasks
- For 200 tasks: 200 calls  1 call (99.5% reduction)
- Eliminates backend timeout issues
- Significantly improves UI responsiveness for large DAGs

Fixes: #[issue_number]
@boring-cyborg boring-cyborg Bot added the area:UI Related to UI/UX. For Frontend Developers. label Dec 26, 2025
@Arunodoy18

This comment was marked as spam.

@Shally-Katariya Shally-Katariya mentioned this pull request Dec 27, 2025
1 of 2 tasks
@Shally-Katariya

Copy link
Copy Markdown
Contributor

I went through the changes in Tasks.tsx and TaskCard.tsx.

Moving task instance fetching to the parent component and passing grouped data to TaskCard makes the data flow much cleaner and clearly avoids the N+1 API calls.

I'm continuing to read through the changes to understand edge cases for large DAGs. Happy to help with testing or follow-ups if needed.

@Shally-Katariya

Copy link
Copy Markdown
Contributor

I tested this locally using Breeze on the Tasks tab for example_branch_labels.

On the Tasks view, I no longer see per-task taskInstances API calls.
Instead, task data is loaded without N+1 requests, and task cards render correctly.

The refactor where Tasks.tsx batch-fetches task instances and passes them down to TaskCard works as expected.
This looks like a solid performance improvement 👍

dagRunId: "~",
requestBody: {
dag_ids: [dagId],
order_by: "-run_after",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_after is not a valid sorting field on TaskInstance. On loading the tasks tab in dag details I can see below error.

400 Bad Request

Ordering with 'run_after' is disallowed or the attribute does not exist on the model

@Shally-Katariya Shally-Katariya Dec 28, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this .

You’re right — run_after is not a valid ordering field for TaskInstance, and using it results in a 400 error on the Tasks tab.

I verified this locally as well. We should switch to a supported ordering field (or remove explicit ordering if not required).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please test your changes before creating PRs. If you're using AI tools to help with development (which is fine!), you must review and test the generated code thoroughly. The volume of untested AI-generated PRs creates significant review burden for maintainers.

To test this properly:

  1. Run the UI locally, check the contributing guide on how if unsure!
  2. Navigate to a DAG with 200+ tasks
  3. Open the Tasks tab and verify it loads without errors
  4. Check the browser console and network tab for API calls

Would you like to revise this PR with a working implementation?

@potiuk

potiuk commented Dec 29, 2025

Copy link
Copy Markdown
Member

Please test your changes before creating PRs. If you're using AI tools to help with development (which is fine!), you must review and test the generated code thoroughly. The volume of untested AI-generated PRs creates significant review burden for maintainers.

+10000

@pierrejeambrun pierrejeambrun changed the title Fix N+1 query issue in DAG Tasks tab Fix N+1 API calls issue in DAG Tasks tab Jan 5, 2026
@kaxil kaxil closed this Jan 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:UI Related to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UI - N+1 Calls in the Tasks Tab

5 participants