Skip to content

Warn at provider discovery about an unresolvable connection type - #72904

Merged
kaxil merged 2 commits into
apache:mainfrom
astronomer:warn-unreachable-connection-types
Sep 16, 2026
Merged

kaxil merged 2 commits into
apache:mainfrom
astronomer:warn-unreachable-connection-types

Conversation

@kaxil

@kaxil kaxil commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

Summary

A hook registers under the connection-type string verbatim. A connection read from a URI, from JSON, or rebuilt from the secrets cache presents the decoded form of that string: get_uri() lowercases the type and encodes _ as -, because RFC 3986 forbids _ in a URI scheme, and reading a connection back decodes it again.

Where the declaration and the decoded name differ, the hook is unreachable for every such connection. A connection created through the UI, the REST API or connections add --conn-type keeps the declared spelling and does resolve, which is what lets the mismatch go unnoticed until a connection is served from somewhere else. That is #72316.

Provider discovery now says so once, naming the package that declared it.

The case that never fails

Where two providers declare the two spellings of one name, a connection for either resolves whichever hook holds the decoded name:

stored directly        conn_type='foo-bar'  -> provider_a.HookA
after a URI round trip conn_type='foo_bar'  -> provider_b.HookB

Neither call raises. The connection is handed the other provider's hook, and because nothing fails there is no error anywhere that could carry the explanation.

Why discovery rather than somewhere else

The authoring schema validates the providers in this repository alone, so a third-party distribution shipping get_provider_info() never passes through it. The runtime provider_info.schema.json is deliberately lenient so installs of already-released providers keep working, which is why #72854 left it alone and #71104 declined the same tightening.

Connection.get_hook() is reached only after a lookup has already failed, so it cannot reach the collision above, which resolves.

Discovery sees every declared type together with the package that declared it, and it already warns there when one connection type is registered twice. This is that check against the decoded names rather than the literal ones.

What it logs

A declared connection type is read back under a different name, so a connection read from a
URI or from JSON cannot reach its hook.
    connection_types=['dummy-vendor'] read_back_as='dummy_vendor'
    packages=['apache-airflow-providers-dummy']

Several declared connection types are read back under one name, so a connection for any of
them resolves whichever hook holds that name.
    connection_types=['shared-name', 'shared_name'] read_back_as='shared_name'
    packages=['apache-airflow-providers-one', 'apache-airflow-providers-two']

A declared connection type cannot be carried in a connection URI, so a connection read from
a URI or from JSON has no connection type at all.
    connection_types=['dummy vendor'] packages=['apache-airflow-providers-dummy']

Design rationale

The decoded name is asked of the decoder, Connection._normalize_conn_type applied to the parsed scheme, rather than restated as a separator swap. Restating it misses what that decoder actually does: postgresql is aliased to postgres, so a provider declaring postgresql is unreachable while a hand-rolled rule calls it fine, and a type that cannot be a URI scheme at all, such as one containing a space, is lost rather than re-spelled and needs its own message. A test asserts the derived name against a real get_uri/from_uri round trip, so the two cannot drift apart.

The import is inside the function because airflow.sdk.definitions.connection imports this module at module level, following the commented precedent already in this file.

Note on log style

These warnings pass structlog keyword arguments rather than the %s positional arguments used by the neighbouring warnings in this file. A positional argument is interpolated into the message and then dropped, so the connection type and the package name would exist only inside the rendered string. As keyword arguments they stay separate fields in the JSON event, which is what makes them queryable and what makes one field schema across the three messages worth having.

Related

#72855 improves what Connection.get_hook() says when a lookup fails. It is independent of this change and touches different files; this one covers the case that does not fail.

A hook registers under the connection-type string verbatim, while a
connection read from a URI, from JSON, or rebuilt from the secrets cache
presents the decoded form of it: get_uri() lowercases the type and
encodes '_' as '-', because RFC 3986 forbids '_' in a URI scheme, and
reading a connection back decodes it again. Where the declaration and the
decoded name differ, the hook is unreachable for those connections. A
connection created through the UI, the REST API or the CLI keeps the
declared spelling and does resolve, which is what lets the mismatch go
unnoticed until a connection is served from somewhere else.

Where two providers declare two spellings of one name it is worse than
unreachable. A connection for either resolves whichever hook holds the
decoded name, so it can be given the other provider's hook, and because
that resolves rather than failing there is no error anywhere to carry the
explanation.

Discovery is the only place either can be reported. The authoring schema
validates the providers in this repository alone, so a third-party
distribution never passes through it, and Connection.get_hook() is
reached only once a lookup has already failed, which the collision never
does. Discovery already warns there about a connection type registered
twice, so this is that check against the decoded names.

The decoded name comes from Connection._normalize_conn_type applied to
the parsed scheme, rather than from restating the separator rule, so it
follows the aliases that decoder applies and reports a type a URI cannot
carry at all. A test asserts it against a real get_uri/from_uri round
trip so the two cannot drift.

The existing inconsistency test declares 'wrong-connection-type', which
is read back under another name in its own right, so it now asserts both
warnings rather than a single one.
@kaxil
kaxil marked this pull request as ready for review September 10, 2026 22:16
@kaxil
kaxil force-pushed the warn-unreachable-connection-types branch from c5616ba to 3c1254f Compare September 10, 2026 22:16
@kaxil
kaxil requested a review from vatsrahul1001 September 12, 2026 22:57

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.
No concern from manually review in the first place.
Here're some minor nits pointed out by agents that makes sense to me.

Comment thread task-sdk/src/airflow/sdk/providers_manager_runtime.py Outdated
Comment thread task-sdk/src/airflow/sdk/providers_manager_runtime.py
Comment thread task-sdk/src/airflow/sdk/providers_manager_runtime.py
Comment thread task-sdk/tests/task_sdk/test_providers_manager_runtime.py
Comment thread task-sdk/tests/task_sdk/test_providers_manager_runtime.py Outdated
Comment thread task-sdk/tests/task_sdk/test_providers_manager_runtime.py Outdated
All three warnings now carry connection_types/packages, so a query keyed on
those fields matches the single-declaration case too. The single-group skip
states its invariant directly rather than leaving it to be derived from the
keys being distinct.

In the tests, the entry check in test_warning_logs_generated now pins one
known pattern per entry, the two-provider case asserts the package/spelling
pairing rather than sorting it away, and the round-trip test's Connection
import moves to the top of the file. There is no cycle there; the one in the
production method is a real cycle and stays.

Also narrows the docstring's claim about the CLI: connections add --conn-uri
builds Connection(uri=...) and --conn-json goes through from_json, both of
which decode the type, so only --conn-type keeps the declared spelling.
@kaxil
kaxil requested a review from jason810496 September 14, 2026 19:44

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@kaxil
kaxil merged commit be95313 into apache:main Sep 16, 2026
107 checks passed
@kaxil
kaxil deleted the warn-unreachable-connection-types branch September 16, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants