Skip to content

Fix ORM models failing to configure in processes that skip database setup - #74642

Merged
kaxil merged 1 commit into
apache:mainfrom
astronomer:models-configure-without-db
Oct 14, 2026
Merged

kaxil merged 1 commit into
apache:mainfrom
astronomer:models-configure-without-db

Conversation

@kaxil

@kaxil kaxil commented Oct 13, 2026

Copy link
Copy Markdown
Member

Models name each other in string relationships, such as Team.dag_bundles = relationship("DagBundleModel", ...), and SQLAlchemy can only resolve a name once its module has been imported. Airflow covers this with a before_configured hook that imports every model before mappers are configured, but the hook was registered inside configure_orm().

settings.initialize() skips configure_orm() in virtualenv-mode subprocesses (PYTHON_OPERATORS_VIRTUAL_ENV_MODE), in a task process re-executed for run_as_user (_AIRFLOW__REEXECUTED_PROCESS) and in non-DB unit tests, and airflow_local_settings is imported before it runs. Those processes only work if they happen to have imported the right model modules already. When they have not, building the first model fails:

sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper[Team(team)], expression 'DagBundleModel'
failed to locate a name ('DagBundleModel').

Non-DB tests started hitting this when #74496 stopped import airflow from loading the task runner, which had been importing dagbundle along the way. That PR added a test-only copy of the hook to the pytest plugin.

This registers the hook in airflow/models/base.py, next to the registry, so it is in place before the first model class is defined, in every process. The copy in the pytest plugin goes away. The new tests build a model in a process that skips configure_orm(), and configure mappers from airflow_local_settings while Airflow is being imported. Both fail on main.

Why not fix Team.dag_bundles instead. 37 other string relationships also name a model whose module the declaring module does not import. On main before #74496, in a process without the hook, importing any one of 59 modules (the model modules plus entry points such as the API app and the Dag processor) and then configuring mappers failed for 28 of them. Fixing each relationship in place runs into circular imports between the model modules. Importing every model before configuration covers all of them, including relationships added later.

SQLAlchemy 2.0 only has before_configured on the Mapper class (2.1 adds a per-registry event). So, as before, the hook fires for whichever registry configures first, FAB's included, and once=True keeps it to one call per process. The cost is importing every model the first time a process configures mappers, which processes that run configure_orm() already pay.

test_orm_event_handlers.py is removed. It checked that the event.listen line raised no SQLAlchemy deprecation warning, and with that line gone from setup_event_handlers it passed whatever the code did.

…etup

Models name each other in string relationships, which only resolve once
every model module is imported. The before_configured hook that imports
them all was registered in configure_orm(), so a process that skips it
(virtualenv-mode subprocesses, a task re-executed for run_as_user,
non-DB tests) or configures mappers before it runs (local settings)
only worked if it had already imported the right model modules. If it
had not, building a model failed, for example with Team.dag_bundles
unable to find DagBundleModel.

Register the hook in airflow.models.base, before any model class is
defined, and drop the test-only copy from the pytest plugin. Remove the
orm_event_handlers test: it guarded the line that moved, and now passes
whatever the code does.
@kaxil
kaxil requested review from XD-DENG and ashb as code owners October 13, 2026 10:38
@kaxil
kaxil merged commit d79c0b1 into apache:main Oct 14, 2026
276 of 278 checks passed
@kaxil
kaxil deleted the models-configure-without-db branch October 14, 2026 10:14
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.4.0 milestone Oct 18, 2026
vatsrahul1001 added a commit that referenced this pull request Oct 18, 2026
…p database setup (#74642) (#74971)

Models name each other in string relationships, which only resolve once
every model module is imported. The before_configured hook that imports
them all was registered in configure_orm(), so a process that skips it
(virtualenv-mode subprocesses, a task re-executed for run_as_user,
non-DB tests) or configures mappers before it runs (local settings)
only worked if it had already imported the right model modules. If it
had not, building a model failed, for example with Team.dag_bundles
unable to find DagBundleModel.

Register the hook in airflow.models.base, before any model class is
defined, and drop the test-only copy from the pytest plugin. Remove the
orm_event_handlers test: it guarded the line that moved, and now passes
whatever the code does.
(cherry picked from commit d79c0b1)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request Oct 20, 2026
…p database setup (#74642) (#74971)

Models name each other in string relationships, which only resolve once
every model module is imported. The before_configured hook that imports
them all was registered in configure_orm(), so a process that skips it
(virtualenv-mode subprocesses, a task re-executed for run_as_user,
non-DB tests) or configures mappers before it runs (local settings)
only worked if it had already imported the right model modules. If it
had not, building a model failed, for example with Team.dag_bundles
unable to find DagBundleModel.

Register the hook in airflow.models.base, before any model class is
defined, and drop the test-only copy from the pytest plugin. Remove the
orm_event_handlers test: it guarded the line that moved, and now passes
whatever the code does.
(cherry picked from commit d79c0b1)

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants