Repository navigation
Fix airflow-core tests that silently required the full provider set - #71637
Open
coleheflin wants to merge 4 commits into
Open
coleheflin wants to merge 4 commits into
coleheflin wants to merge 4 commits into
Conversation
coleheflin
force-pushed
the
move-core-provider-tests
branch
from
August 17, 2026 23:48
94846a3 to
e7cb86a
Compare
coleheflin
marked this pull request as ready for review
August 17, 2026 23:49
Contributor
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
1 task done
coleheflin
force-pushed
the
move-core-provider-tests
branch
from
August 20, 2026 17:03
e7cb86a to
b3640f9
Compare
shahar1
reviewed
Sep 13, 2026
shahar1
left a comment
shahar1
left a comment
Contributor
There was a problem hiding this comment.
Please resolve conflicts
A scoped `uv sync --project airflow-core` installs only the six providers airflow-core's own dev group declares (amazon, celery, cncf-kubernetes, fab, git, ftp), not the ~100+ providers a full workspace sync brings in. Several airflow-core tests assumed the full set was always present and failed rather than skipped when it wasn't, so a minimal core-only environment couldn't run the test suite clean. devel-common's tests_common also imported airflow_shared.secrets_masker and airflow_shared.module_loading directly without declaring those shared distributions as dependencies, relying on them being installed as a side effect of a full workspace sync.
importlib.util.find_spec("airflow.providers.google") imports the
parent airflow package as a side effect, which broke shared/observability's
own isolated test suite (it doesn't install airflow-core, so importing
airflow.configuration failed on a missing jsonschema dependency).
Checking installed distribution metadata instead avoids the import.
…pendency The previous fix added apache-airflow-shared-module-loading and apache-airflow-shared-secrets-masker as devel-common dependencies, but those distributions are marked "Private: Do Not Upload" and never published — they only resolve via the local workspace. CI's provider compat-testing jobs (Compat 3.1.8/3.2.2/3.3.1) deliberately uninstall all such local editable packages before installing a real released apache-airflow wheel, so the dependency broke virtually every provider test in those jobs with the same ModuleNotFoundError this PR set out to fix. airflow-core already vendors this exact code into its own installed package (airflow._shared.*, baked in at build time — see its sdist force-include config) specifically so it's available regardless of how airflow-core was installed. Importing from there instead — the same pattern already used elsewhere in this codebase, including tests_common's own pytest_plugin.py — needs no extra dependency and survives the compat-testing environment.
The TODO already acknowledged this should go away eventually, but had no tracking issue. Filed one covering the remaining migration scope (dropping the 6 hardcoded providers once tests no longer need them unconditionally), since it's a larger, separate piece of work from the fixes in this PR.
coleheflin
force-pushed
the
move-core-provider-tests
branch
from
September 14, 2026 02:47
b3640f9 to
49bf1d0
Compare
Contributor
Author
|
Rebased onto the latest |
Contributor
|
This pull request has been automatically marked as stale because the author has not responded to a request for more information. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
github-actions
Bot
added
the
stale
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
tests_commonfiles (test_utils/logs.py,api_fastapi.py,providers.py) that importedairflow_shared.secrets_masker/module_loading— a private, never-published distribution only resolvable in a full local workspace checkout — instead of the vendoredairflow._shared.*copy airflow-core bakes into its own installed package (same pattern already used elsewhere, e.g. this file's ownpytest_plugin.py).skip_unless_all_providers_installedmarker (plus a couple of targetedpytest.importorskips) to 11 tests intest_providers_manager.py/test_configuration.pythat assumed the full ~100+ provider set is always installed.airflow-core/pyproject.toml(about eventually dropping its own hardcoded provider dev-dependencies) to a new tracking issue, Drop airflow-core's own dev-group dependency on 6 providers #71641, since it had none.Why
addresses (partially): #60770
uv sync --project airflow-core(a scoped, core-only sync) succeeds, but the test suite didn't run clean: collection crashed withModuleNotFoundError: No module named 'airflow_shared'(it only worked before as a side effect of an unrelated full workspace sync), and 11 more tests failed on full-provider-set assumptions. Fixing both getspytest airflow-core/tests/unit -m "not db_test"passing cleanly in that scoped environment.(An earlier version fixed the import crash by adding those private packages as explicit
devel-commondependencies. That broke theCompat 3.1.8/3.2.2/3.3.1CI jobs instead — they deliberately strip local editable packages before installing a releasedapache-airflowwheel, so the new dependency reintroduced the same error there. Using the vendoredairflow._shared.*path avoids the dependency entirely and survives that environment too.)This only partially addresses #60770. Its title ("move remaining provider tests from airflow-core") implies a bigger migration:
airflow-core's own dev group still hardcodes 6 providers (amazon,celery,cncf-kubernetes,fab,git,ftp) so tests depending on them still pass — nothing was moved or removed here. Untangling which tests genuinely need which of those 6, and moving vs. guarding each, is a much larger multi-PR effort, so it's tracked separately in #71641 rather than folded into this PR. Leaving #60770 open for a maintainer to decide whether this is enough to close it.Testing performed
uv sync --project airflow-core+pytest; after the fix, collection is clean (0 errors) and the non-db unit suite passes fully (0 failed, was 11 failed) — with no new dependency added.uv syncand reran the same files to confirm all tests still pass for real (marker correctly no-ops when all providers are present).shared/observabilityvenv, and a compat-style env with airflow-core but no standalone shared packages) to verify the intermediate approach failed and the final one passes.Compatjobs pass (including the two that broke on the intermediate approach),Shared observability testspasses, no failures anywhere.main;prek run --from-ref upstream/main --stage pre-commit(ruff, mypy) passes.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5) following the guidelines