Repository navigation
Proposal: Runtime CRUD API and UI support for DAG Bundles #61645
Description
Description
Airflow 3 introduced DAG Bundles configured via [dag_processor].dag_bundle_config_list.
Currently bundle definitions are static and only loaded from configuration at process startup.
This makes it difficult to manage bundles dynamically in environments where DAG sources are discovered or change over time.
It would be helpful to support:
- runtime CRUD of bundle definitions via the REST API (e.g. /api/v1/dag_bundles)
- persistence in the metadata DB
- automatic reload by DAG Processor(s)
- visibility and management from the Airflow UI (list, inspect, enable/disable, create/delete)
This would make bundles first-class operational resources instead of static configuration.
Use case/motivation
I run Airflow with a GitHub OAuth/GitHub App that has access to many organizations and repositories (private + public).
Each git repository may contain DAGs under a workflows/ directory.
Repositories are added/removed over time and should be discovered automatically.
Expected behavior:
- dynamically detect accessible repositories
- create one bundle per repository
- refresh them automatically
- avoid hard-coded configuration
- avoid restarting Airflow when repositories change
Security and operational goals:
- per-repository isolation (one repo = one bundle)
- independent bundle versioning
- no long-lived per-repo credentials
- zero-downtime onboarding of new repositories
Today this requires regenerating config and restarting components, or aggregating multiple repositories into a single bundle, which reduces isolation, weakens security boundaries between repositories, and increases operational overhead.
A dynamic bundle registry and API would address this cleanly.
Related issues
- Automatic DagBundle Configuration Loading #59799
- Proposal: Improve
dag_bundle_config_listConfiguration in Airflow Helm Chart #60630
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Feb 8, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
SakshamSinghal20 commentedon Feb 8, 2026
can i work on this?
shivaam commentedon Apr 9, 2026
Hi, I am considering to work on this given there is enough demand. I'm thinking of a phased approach:
Phase 1 (ready to submit): Read-only API + refresh trigger
GET /api/v2/dagBundles— list all bundles (paginated, sortable)GET /api/v2/dagBundles/{name}— get single bundle detailsPATCH /api/v2/dagBundles/{name}— enable/disablePUT /api/v2/dagBundles/{name}/refresh— trigger bundle refresh (async, fire-and-forget). This inserts aDagPriorityParsingRequestand the DAG Processor picks it up on its next loop iteration — same pattern as the existingparseDagFileendpoint. Synchronous refresh would require architectural changes to the DAG Processing Manager.No DB migration needed — this exposes the existing
dag_bundletable through a proper API surface.Phase 2 (pending design feedback): Full CRUD
classpath,kwargs,sourcecolumns todag_bundlePOST /dagBundlesandDELETE /dagBundles/{name}endpointsPhase 2 has security implications (classpath storage = code execution) and would benefit from maintainer input on the design. Happy to start a dev mailing list thread if needed.
Would Phase 1 be welcome as an initial PR?
potiuk commentedon Apr 9, 2026
I think that deserves a devlist discussion to start with.
While I understand the need to manage such bundle configuration, I do not think personally this is necessary good idea to do it via REST API. If you look at our (updated wiht 3.2.0) security model https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html - this approach introduce a compleletly new ,security realm for Airflow. It's been vastly updated now and refreshed for Airflow 3.2 and you are welcome to take a deep look there and understand some basica assumptions of security principles and isolation between roles, components and workflows that we have in Airflow. This should be a good and deep reading before starting discussion on devlist - because this will be the first question you will get asked when you raise such proposal in devlist.
So Phase 2 in your case is not even nearly enough - and it should be rather Phase 0. We need to know what kind of security realm we are proposing, what kind of roles it creates for Airflow and whether we want to take some risks in Airlfow for moving part of the currently delegated security responsibility (to deployment and Deployment Manager) - to be handled by Airflow software.
Currrently bundle definition, access to Dags and configuration of Airflow - including where Dags are retrieved from, what is the bundle definition etc. is strictly limited to "Deployment Manager" role. Deployment Manager uses Deployment-specific mechanisms to set the configuration and manage it - and it's not exposed or managed for security from Airflow "software" point of view - it is left entirely to Deployment Manager and all kinds of security mechanisms that are specific to deployment type chosen by Deployment Manager.
By adding Dag Bundle configuration, we are extending the model with a new Role - someone who can manage part of Airflow configuration - boundle definition - via API and api -server.
This has far-reaching implication. For example currently api-server can be run in an environment where airflow configuration - including bundle definition is read only. And this is a built-in security feature - it makes it simply impossible, even if api-server security is broken, to permanently change configuration of Airflow remotely via REST API .
So while I am not 100% against such feature - it has to be first discussed in the devlist and it should be - eventually voted on - likely with a complete AIP describing security implications, new roles proposed, scope and security boundaries of such change.
samraj2k commentedon Jul 17, 2026
Thanks @potiuk, we have a similar use case, and I agree that writable bundle configuration through the Airflow API would introduce a new security realm.
We’ are proposing following change
This way, we wont we creating any new write APIs. The logic to fetch the bundles (on which we do reload) resides outside of Airflow, so users can fetch the bundles either from external source like S3, any API etc
This also solves for: #59799 by being generic.
Let me know how this sounds and the next steps. Would a dev-list [DISCUSS] thread/AIP be the appropriate next step, or PRs can be raised directly. I believe the hook plus DAG processor reload address the security concern you have mentioned.
potiuk commentedon Jul 19, 2026
we already have many abstractions, I am not sure how you want to provide custom dynamic manager and why it would be better than managing airflow configuration dynamically - but yes any change like htat would require and AIP and devlist discussion.
samraj2k commentedon Jul 24, 2026
Hey @potiuk , I have started a discuss thread on mail, please check mail:
https://lists.apache.org/thread/860cv5t71v5mnq59b50fk5891lv5nh26
Let me know about the proposal, thanks