Repository navigation
Migrate to FastAPI 0.137 (lazy router inclusion) — scope decision, target 3.4.0 #69065
Description
Summary
FastAPI 0.137.0 reworked the router
layer into lazy router inclusion: include_router() no longer leaves a flat list of materialized
APIRoutes in router.routes — included routers can appear as _IncludedRouter wrappers, and a
router's dependencies are now frozen into each route at include time.
We currently cap fastapi<0.137.0 (#68562). #68826 lifts the cap (cadwyn 7.1.0 now requires
fastapi>=0.137.1) and fixes the execution-API part — but the bump is really a broader migration
that touches security-relevant route introspection, so we want the FastAPI / token-scope / security
owners to confirm the direction before we commit.
Related: #68826 (WIP), #68562 (the cap), #68578 (added the cap).
Why this makes our code better — 0.137 enables the cleaner pattern
Today the execution API injects the OTel trace-context dependency by mutating route.dependencies on a
shared, module-global router after assembly (_inject_trace_context_dep) — with an idempotent
strip-and-re-add, because that global router is processed more than once per process
(cached_app + InProcessExecutionAPI) and we have to de-dupe our own prior injection.
0.137's lazy inclusion breaks that today — but that's the point: the strip-and-re-add only ever
existed to work around the old mutable-shared-router model. 0.137 freezing a router's dependencies
into each route at include time is exactly the contract we want: it lets us declare the
trace-context dependency once, at build time, in a fresh router per app, and it stays put. No
shared mutable dependencies after routes are built, no post-hoc patching, no strip-and-re-add, and
the test fixture that snapshot/restored the mutated global disappears. The migration removes a
workaround rather than adding one.
The hard part — token scope & security boundaries
Several security guards introspect router.routes expecting flat APIRoutes and break under lazy
inclusion:
execution_api/test_token_scope_boundaries.py— asserts each route's allowed token types
({execution}/{workload}); iteratesexecution_api_router.routes.core_api/.../test_routes.py::test_no_auth_routes— asserts which routes do/don't require 401/403;
fails withAttributeError: '_IncludedRouter' object has no attribute 'path'.
These are the checks that guarantee a route doesn't silently lose auth or change token scope. They
must be migrated to traverse the new lazy structure and assert the same invariants — not silenced
to go green. This needs the token/security owners' eyes, and likely a rethink of how the boundaries
are introspected and tested.
Proposed approach
One migration PR (re-scope #68826 → "Migrate to FastAPI 0.137"):
- Bump
cadwyn>=7.1.0+fastapi>=0.137.1, drop the cap. - A single shared
iter_materialized_routes()helper, routed through every introspection site
(both security guards + execution token-scope) — one place to get the lazy→materialized traversal
right, reviewed once. - The build-time router factory for the trace-context dependency (already in Restore FastAPI 0.137 support in the Task Execution API #68826).
- Confirm api-server / dag-processor startup on 0.137 end-to-end (not just unit tests).
- A negative test proving the auth guards still bite (a route stripped of auth must fail
no_auth_routes).
Splitting this doesn't work cleanly — the introspection fixes can't be tested without 0.137 active, and
the cap can't lift without the fixes — so it's one coherent migration.
Timing
Not a 3.3.0 change — too security-sensitive and too broad to rush into the imminent release.
Targeting 3.4.0; please do not merge #68826 until this is agreed and properly scoped.
Who should weigh in
cc @vincbeck (auth manager / security), @ashb · @kaxil · @amoghrajesh (execution_api/ CODEOWNERS —
token scope + the trace-context dep), @pierrejeambrun · @bugraoz93 (core-API FastAPI / no_auth_routes
guard), @ferruzzi (token-scope-boundaries test) — does the direction (build-time factory + a shared
route-materialization helper for the security introspection) look right, and should we proceed with the
single-PR migration for 3.4.0? The call is yours.
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
Activity
potiuk commentedon Jun 26, 2026
BTW. I did review the change and even if overview is Claude prepared, it reflects what I learned about the change - in case people asked.
ferruzzi commentedon Jun 26, 2026
To start with, I'm absolutely not going to claim any expertise in the token scope domain. Ash caught a bug that I missed and existing unit tests didn't cover so I covered it based on his suggestion. So I have some small experience, but I'm definitely not The Token Scope Guy 😅 .
That said, based on what I learned while fixing that, I like the direction this is going and I'd be +1 for moving forward in 3.4.0 with a single PR migration. Updating
test_token_scope_boundaries.pyshould be trivial._all_route_policies()currently iterates overexecution_api_router.routescheckingisinstance(route, APIRoute). A sharediter_materialized_routes()helper should be a drop-in replacement for the iterator and the test's actual policy assertions in theNON_DEFAULT_TOKEN_POLICYdict stay untouched.The build-time factory looks like the right way forward and as long as the proposed new iterator includes all of the token types, it should be a straight substitution on this end.
Revanth14 commentedon Jun 27, 2026
Thanks for opening this. My 2 cents here
I'm +1 on handling this as one coherent 3.4.0 migration rather than rushing it into 3.3.0.
One extra thing worth considering: FastAPI 0.137.2 added
iter_route_contexts()for advanced use cases that previously inspectedrouter.routes- exactly the category our failing guards fall into. That makes this cheaper than it first looks: rather than a minor-version jump, we'd only bump the lower bound from>=0.137.1to>=0.137.2and use FastAPI's helper as the shared introspection primitive. The fallback is to keep>=0.137.1and add a small local compatibility helper with equivalent behavior, but given the patch-level cost, I'd lean toward the upstream primitive.I also agree the token-scope tests should not be weakened. They should keep asserting the same policy, just over the effective route tree instead of the old flat
router.routeslist.potiuk commentedon Jun 27, 2026
Oh nice find! I looked only at the 0.137.1 - but indeed, that's then far easier change than it seems - the only thing, I think it needs is to remove our hacks that were added before (the one @ferruzzi spoke about) as they should not be needed any more.
Maybe you can attempt to do it in your PR @Revanth14 -> and that should give others a chance to see how it would look like (and it will likely need some serious manual testing of the auth cases that are related to those route manipulations of ours around OTEL/ token scope ?
And yes - it's TOTALLY fine to have
>=0.137.2:)pierrejeambrun commentedon Jun 29, 2026
Thanks for the summary and investigation work on this.
This makes sense to me.
ferruzzi commentedon Jun 29, 2026
Another thought, and it's entirely possible this maybe scope creep and should be ignored, but I'll drop it here in case the migration ends up making this relevant:
The sole reason the token scope tests have
isinstance(route, APIRoute)is to make MyPy happy. Technically, therouter.routesiterator I used in those tests is typed aslist[BaseRoute]so it can return a couple other types which Airflow never implemented (WebSocketRoute, Mount, Host, and I feel like there is another I'm forgetting at the moment).If the new official helper changes that return type, it is possible that it may require changes elsewhere. Or if we end up making our own helper then maybe we can scope it down to just the APIRoute type we actually use.
bugraoz93 commentedon Jun 29, 2026
Great summary Jarek, thanks! Sounds good to me! Version 3.4.0 especially better so that we can create some test cases and execute those with more confidance.
Sweep deps, polars 2.0, pydantic 2.14 et image Airflow 3.3.2 (#33)
zmievsa commentedon Oct 20, 2026
@potiuk JFYI if you need any support from Cadwyn side to ease this migration or others, I am here to help