Repository navigation
Load connection type metadata on-demand to improve Add Connection page load time #61815
Description
Apache Airflow version
main (development)
If "Other Airflow 3 version" selected, which one?
No response
What happened?
The Connection Add page takes ~10 seconds to load on first access.
GET /ui/connections/hook_metaimports all Hook classes and their form definitions, which loads every provider and its dependencies.
With 100+ connection types, this causes significant delay before users can interact with the form.
I understand the slowdown is more noticeable because the main branch has all providers installed by default.
Nevertheless, in my opinion, it would be good if this could be improved.
Screen.Recording.2026-02-12.at.9.34.40.PM.mov
What you think should happen instead?
It's not a critical issue, but for better user experience, the Add Connection form should be usable within about 1 second. To achieve this, connection types should load first, then metadata only when the user selects a type.
| Step | Current | Proposed |
|------|---------|----------|
| Add button click | Load all types + metadata | Load types only |
| Connection type dropdown | — | Populated from types API |
| User selects type | — | Load metadata for that type only |
| Form fields | — | Rendered from detail API |
Proposed API changes:
1. GET /ui/connections/connection_types (new, lightweight)
Returns only connection type identifiers
2. GET /ui/connections/hook_meta/{connection_type} (new, on-demand)
Returns full metadata (standard_fields, extra_fields) for the selected type only
3. GET /ui/connections/hook_meta (existing)
Kept unchanged for backward compatibility
Adopted approach: Use YAML-based metadata
Instead of adding new APIs, we use the connection metadata already migrated to YAML by @amoghrajesh :
hook_metareads from provider.yaml / get_provider_info.py – no Hook class importsProvidersManager._load_ui_metadata()loads connection-types from provider.yamlhook-nameadded to provider.yamlconnection-typesfor display in the dropdownget_provider_info()(from get_provider_info.py) provides metadata without importing Hooks
- Connection form dropdown – shows
hook-namewhen present, otherwiseconnection_type - No new endpoints – existing
GET /ui/connections/hook_metais kept and made faster by avoiding Hook imports
### Implementation Plan
1. GET /ui/connections/connection_types
- PR: TBD
2. GET /ui/connections/hook_meta/{connection_type}
- PR: #63650
3. Split hooks + Update ConnectionForm to use new APIs
- PR: TBD
How to reproduce
- Install Airflow with many providers
- Go to Connections page.
- Click Add button.
- This reproduces when the cache is cold (first load or after restart): the page takes ~10 seconds before the connection type dropdown is usable.
Operating System
macOS
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else?
If the committers agree with this approach, I would like to implement it.
If you approve, I would appreciate it if you could assign this issue to me.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
yuseok89 commentedon Feb 12, 2026
If the committers agree with this approach, I would like to implement it.
If you approve, I would appreciate it if you could assign this issue to me.
jroachgolf84 commentedon Feb 12, 2026
@yuseok89, feel free to open up a PR for this issue!
choo121600 commentedon Feb 13, 2026
Thanks for raising this! I agree the initial load time is noticeable.
Just for context, #60410 recently introduced YAML-based discovery for connection form metadata
Since I think this change is closely related to this issue, it might be helpful to align with @amoghrajesh to ensure the approach is consistent.
cc @amoghrajesh
jason810496 commentedon Feb 13, 2026
Here is more context for the long-term goal: #60404
The idea is to read the metadata from
provider.yamlinstead of importing viaget_ui_field_behaviour.yuseok89 commentedon Feb 13, 2026
@choo121600 @jason810496
Thanks for the context and pointers.
I'll look into the code to see how the YAML-based approach works.
@amoghrajesh
Would splitting the API(connection_types + on-demand detail per type) align with the YAML-first direction?
jason810496 commentedon Feb 13, 2026
Sorry that I didn't look into the proposed "API(connection_types + on-demand detail per type)" approach carefully in my last reply.
From my perspective, the static yaml based approach might already be fast enough
Since we only need the
provider.yamlfor each provider instead of importing hooks for each provider, which should be less than 1 seconds for those file I/O operation. Maybe we can wait until the yaml base approach finishes and then see whether do we need the additional separate routes for the UX purpose.Arunodoy18 commentedon Feb 14, 2026
yuseok89 commentedon Feb 15, 2026
@jason810496
Thanks again for the context.
I'd like to double-check my understanding.
As I understand it, the framework for reading from YAML is complete, and most of the YAML migration still appears to be needed.
I'd like to contribute to this migration.
Is anyone already working on it?
I'd prefer to avoid overlapping or conflicting work.
cc @amoghrajesh
jason810496 commentedon Feb 15, 2026
I’m not sure whether Amogh has started yet.
amoghrajesh commentedon Feb 16, 2026
Hello @yuseok89, I will be primarily working on it. As @jason810496 mentioned it will be significantly faster than the current approach.
I have plans to start on this later this week, I also have some proposals for the more "long term" goal of this which would allow no providers to be installed on Airflow API server at all. That will take some time and is dependent on another parallel effort.
I can loop you in onto the effort if there is some room for collaboration.
yuseok89 commentedon Feb 16, 2026
@amoghrajesh
Thanks for the update!
I'd really like to be part of this.
Please do loop me in when you're ready.
I'm happy to help with any tasks you have in mind.
I'll leave this ticket open for now.
It might be good to see if additional work is needed after the YAML-related work is done.
potiuk commentedon Mar 2, 2026
@yuseok89 We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
yuseok89 commentedon Mar 31, 2026
@amoghrajesh
After your YAML migration I re-measured load time and the Add Connection flow was still noticeably slow on a cold cache.
I traced it and found that the Add Connection page (via GET /ui/connections/hook_meta) was not actually going through the YAML.
This PR(#63826) wires the endpoint up to use the migrated YAML metadata so we avoid that work.
Could you take a look when you have time?