Repository navigation
Validate provider metadata list entries instead of ignoring them - #71104
Merged
Merged
Conversation
Lee-W
force-pushed
the
queues-schema-items
branch
2 times, most recently
from
August 5, 2026 08:10
0af7ff4 to
a509680
Compare
Lee-W
marked this pull request as ready for review
August 5, 2026 08:34
Lee-W
force-pushed
the
queues-schema-items
branch
from
August 5, 2026 09:30
a509680 to
fc354e4
Compare
kaxil
reviewed
Aug 5, 2026
The items blocks for queues, plugins and task-decorators listed their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead. Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field. Raised in review of apache#70190.
The runtime schema is documented as deliberately looser than the development one, and its validate() call is unwrapped in provider discovery, so a single installed provider emitting an outdated entry shape would abort discovery for every provider in that process instead of costing just that entry a logged warning. The structural descriptions stay behind so the schema no longer documents a shape it never actually enforced.
Requiring the right keys does not stop a leftover one sitting beside them, so a stale `path:` next to a correct `class-name:` still validated — exactly the confusion this branch set out to remove. Every sibling item block in this authoring schema already closes itself off this way.
Lee-W
force-pushed
the
queues-schema-items
branch
from
August 7, 2026 15:05
fc354e4 to
da294aa
Compare
potiuk
approved these changes
Aug 9, 2026
dabla
pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
…che#71104) * Validate provider metadata list entries instead of ignoring them The items blocks for queues, plugins and task-decorators listed their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead. Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field. Raised in review of apache#70190. * Keep runtime provider metadata validation permissive The runtime schema is documented as deliberately looser than the development one, and its validate() call is unwrapped in provider discovery, so a single installed provider emitting an outdated entry shape would abort discovery for every provider in that process instead of costing just that entry a logged warning. The structural descriptions stay behind so the schema no longer documents a shape it never actually enforced. * Reject unknown keys in provider.yaml decorator and plugin entries Requiring the right keys does not stop a leftover one sitting beside them, so a stale `path:` next to a correct `class-name:` still validated — exactly the confusion this branch set out to remove. Every sibling item block in this authoring schema already closes itself off this way.
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 10, 2026
The runtime schema (provider_info.schema.json) is validated against the get_provider_info() output of every installed provider, third-party included, and that validate() call is unwrapped in provider discovery. A pattern there turns one non-conforming provider into a failure for all provider discovery in the process, and every released apache-airflow-providers-common-ai wheel declares the hyphenated types, so installing a released provider against new core would fail to import Airflow at all. PR apache#71104 declined the same tightening for the same reason. The authoring gate is fully delivered by provider.yaml.schema.json, which the provider-yaml pre-commit validates for every in-tree provider. Tightens that authoring pattern to ^[a-z][a-z0-9_]*$ so it also rejects a leading digit, which is not a valid URI scheme under RFC 3986 and which Python 3.10 and 3.14 parse differently. Corrects the three connection-doc notes, which claimed every stored connection needed updating. URI and JSON values decode '-' back to '_' and resolve unchanged, so only types stored verbatim (metadata DB rows and object-form imports) need a change; those now carry the procedure. Also fixes the hyphenated connection-type example in the new-provider guide, and pins the resolution tests to the exact hook class with a per-parametrization conn_id.
This was referenced Sep 10, 2026
kaxil
added a commit
that referenced
this pull request
Sep 10, 2026
…#72853) * Fix PydanticAI vendor connections not resolving from secrets backends The three vendor connection types declared their `conn_type` with hyphens (`pydanticai-azure`, `pydanticai-bedrock`, `pydanticai-vertex`), but a hook is registered under the literal `connection-type` from `provider.yaml` while `Connection.from_uri` and `Connection.from_json` rewrite `-` to `_` before the lookup. The registered key was therefore unreachable from every secrets backend, including the `AIRFLOW_CONN_*` environment variables enabled by default, and any task using one of these connections failed with: AirflowException: Unknown hook type "pydanticai_azure" Only a connection read straight out of the metadata DB kept its hyphen and resolved, which is why this went unnoticed. Renames the three types to underscores, matching every other multi-word hook in the provider tree (`azure_data_factory`, `google_cloud_platform`, `spark_connect`). `default_conn_name` values were already underscored and are unchanged. * Reject hyphenated connection-type in the provider schema and document the rename Adds `"pattern": "^[a-z0-9_]+$"` to the connection-type property in both provider schemas, so a hyphenated type fails schema validation at authoring time instead of at a user's first task run. All 141 connection-type values across the provider tree already satisfy it. Also parametrizes the round-trip guard so it reports every offending type rather than aborting at the first, and notes the rename in the three connection docs for users whose connections are already stored with the old type. * Keep the connection-type gate in the authoring schema only The runtime schema (provider_info.schema.json) is validated against the get_provider_info() output of every installed provider, third-party included, and that validate() call is unwrapped in provider discovery. A pattern there turns one non-conforming provider into a failure for all provider discovery in the process, and every released apache-airflow-providers-common-ai wheel declares the hyphenated types, so installing a released provider against new core would fail to import Airflow at all. PR #71104 declined the same tightening for the same reason. The authoring gate is fully delivered by provider.yaml.schema.json, which the provider-yaml pre-commit validates for every in-tree provider. Tightens that authoring pattern to ^[a-z][a-z0-9_]*$ so it also rejects a leading digit, which is not a valid URI scheme under RFC 3986 and which Python 3.10 and 3.14 parse differently. Corrects the three connection-doc notes, which claimed every stored connection needed updating. URI and JSON values decode '-' back to '_' and resolve unchanged, so only types stored verbatim (metadata DB rows and object-form imports) need a change; those now carry the procedure. Also fixes the hyphenated connection-type example in the new-provider guide, and pins the resolution tests to the exact hook class with a per-parametrization conn_id. * Anchor the connection-type pattern past a trailing newline A schema pattern is matched with re.search, and Python's '$' matches before a trailing newline as well as at the end of the string, so ^[a-z][a-z0-9_]*$ accepted a connection type carrying one. A YAML block scalar produces exactly that, so 'connection-type: |' followed by an indented name would have registered a hook under a name no connection can reach. A hyphen was already rejected either way, since '-' is outside the character class and the newline allowance excuses only a newline, so this does not reopen the case the pattern was added for. It closes the one value that could still slip past it. Adds the first test for this pattern, covering the spellings that cannot be resolved: a hyphen, a block scalar's trailing newline, uppercase, a leading digit, whitespace and an embedded newline.
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
…che#71104) * Validate provider metadata list entries instead of ignoring them The items blocks for queues, plugins and task-decorators listed their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead. Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field. Raised in review of apache#70190. * Keep runtime provider metadata validation permissive The runtime schema is documented as deliberately looser than the development one, and its validate() call is unwrapped in provider discovery, so a single installed provider emitting an outdated entry shape would abort discovery for every provider in that process instead of costing just that entry a logged warning. The structural descriptions stay behind so the schema no longer documents a shape it never actually enforced. * Reject unknown keys in provider.yaml decorator and plugin entries Requiring the right keys does not stop a leftover one sitting beside them, so a stale `path:` next to a correct `class-name:` still validated — exactly the confusion this branch set out to remove. Every sibling item block in this authoring schema already closes itself off this way.
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
…apache#72853) * Fix PydanticAI vendor connections not resolving from secrets backends The three vendor connection types declared their `conn_type` with hyphens (`pydanticai-azure`, `pydanticai-bedrock`, `pydanticai-vertex`), but a hook is registered under the literal `connection-type` from `provider.yaml` while `Connection.from_uri` and `Connection.from_json` rewrite `-` to `_` before the lookup. The registered key was therefore unreachable from every secrets backend, including the `AIRFLOW_CONN_*` environment variables enabled by default, and any task using one of these connections failed with: AirflowException: Unknown hook type "pydanticai_azure" Only a connection read straight out of the metadata DB kept its hyphen and resolved, which is why this went unnoticed. Renames the three types to underscores, matching every other multi-word hook in the provider tree (`azure_data_factory`, `google_cloud_platform`, `spark_connect`). `default_conn_name` values were already underscored and are unchanged. * Reject hyphenated connection-type in the provider schema and document the rename Adds `"pattern": "^[a-z0-9_]+$"` to the connection-type property in both provider schemas, so a hyphenated type fails schema validation at authoring time instead of at a user's first task run. All 141 connection-type values across the provider tree already satisfy it. Also parametrizes the round-trip guard so it reports every offending type rather than aborting at the first, and notes the rename in the three connection docs for users whose connections are already stored with the old type. * Keep the connection-type gate in the authoring schema only The runtime schema (provider_info.schema.json) is validated against the get_provider_info() output of every installed provider, third-party included, and that validate() call is unwrapped in provider discovery. A pattern there turns one non-conforming provider into a failure for all provider discovery in the process, and every released apache-airflow-providers-common-ai wheel declares the hyphenated types, so installing a released provider against new core would fail to import Airflow at all. PR apache#71104 declined the same tightening for the same reason. The authoring gate is fully delivered by provider.yaml.schema.json, which the provider-yaml pre-commit validates for every in-tree provider. Tightens that authoring pattern to ^[a-z][a-z0-9_]*$ so it also rejects a leading digit, which is not a valid URI scheme under RFC 3986 and which Python 3.10 and 3.14 parse differently. Corrects the three connection-doc notes, which claimed every stored connection needed updating. URI and JSON values decode '-' back to '_' and resolve unchanged, so only types stored verbatim (metadata DB rows and object-form imports) need a change; those now carry the procedure. Also fixes the hyphenated connection-type example in the new-provider guide, and pins the resolution tests to the exact hook class with a per-parametrization conn_id. * Anchor the connection-type pattern past a trailing newline A schema pattern is matched with re.search, and Python's '$' matches before a trailing newline as well as at the end of the string, so ^[a-z][a-z0-9_]*$ accepted a connection type carrying one. A YAML block scalar produces exactly that, so 'connection-type: |' followed by an indented name would have registered a hook under a name no connection can reach. A hyphen was already rejected either way, since '-' is outside the character class and the newline allowance excuses only a newline, so this does not reopen the case the pattern was added for. It closes the one value that could still slip past it. Adds the first test for this pattern, covering the spellings that cannot be resolved: a hyphen, a block scalar's trailing newline, uppercase, a leading digit, whitespace and an embedded newline.
xvega
pushed a commit
to xvega/airflow
that referenced
this pull request
Sep 13, 2026
…apache#72853) * Fix PydanticAI vendor connections not resolving from secrets backends The three vendor connection types declared their `conn_type` with hyphens (`pydanticai-azure`, `pydanticai-bedrock`, `pydanticai-vertex`), but a hook is registered under the literal `connection-type` from `provider.yaml` while `Connection.from_uri` and `Connection.from_json` rewrite `-` to `_` before the lookup. The registered key was therefore unreachable from every secrets backend, including the `AIRFLOW_CONN_*` environment variables enabled by default, and any task using one of these connections failed with: AirflowException: Unknown hook type "pydanticai_azure" Only a connection read straight out of the metadata DB kept its hyphen and resolved, which is why this went unnoticed. Renames the three types to underscores, matching every other multi-word hook in the provider tree (`azure_data_factory`, `google_cloud_platform`, `spark_connect`). `default_conn_name` values were already underscored and are unchanged. * Reject hyphenated connection-type in the provider schema and document the rename Adds `"pattern": "^[a-z0-9_]+$"` to the connection-type property in both provider schemas, so a hyphenated type fails schema validation at authoring time instead of at a user's first task run. All 141 connection-type values across the provider tree already satisfy it. Also parametrizes the round-trip guard so it reports every offending type rather than aborting at the first, and notes the rename in the three connection docs for users whose connections are already stored with the old type. * Keep the connection-type gate in the authoring schema only The runtime schema (provider_info.schema.json) is validated against the get_provider_info() output of every installed provider, third-party included, and that validate() call is unwrapped in provider discovery. A pattern there turns one non-conforming provider into a failure for all provider discovery in the process, and every released apache-airflow-providers-common-ai wheel declares the hyphenated types, so installing a released provider against new core would fail to import Airflow at all. PR apache#71104 declined the same tightening for the same reason. The authoring gate is fully delivered by provider.yaml.schema.json, which the provider-yaml pre-commit validates for every in-tree provider. Tightens that authoring pattern to ^[a-z][a-z0-9_]*$ so it also rejects a leading digit, which is not a valid URI scheme under RFC 3986 and which Python 3.10 and 3.14 parse differently. Corrects the three connection-doc notes, which claimed every stored connection needed updating. URI and JSON values decode '-' back to '_' and resolve unchanged, so only types stored verbatim (metadata DB rows and object-form imports) need a change; those now carry the procedure. Also fixes the hyphenated connection-type example in the new-provider guide, and pins the resolution tests to the exact hook class with a per-parametrization conn_id. * Anchor the connection-type pattern past a trailing newline A schema pattern is matched with re.search, and Python's '$' matches before a trailing newline as well as at the end of the string, so ^[a-z][a-z0-9_]*$ accepted a connection type carrying one. A YAML block scalar produces exactly that, so 'connection-type: |' followed by an indented name would have registered a hook under a name no connection can reach. A hyphen was already rejected either way, since '-' is outside the character class and the newline allowance excuses only a newline, so this does not reopen the case the pattern was added for. It closes the one value that could still slip past it. Adds the first test for this pattern, covering the spellings that cannot be resolved: a hyphen, a block scalar's trailing newline, uppercase, a leading digit, whitespace and an embedded newline.
dlactin
pushed a commit
to dlactin/airflow
that referenced
this pull request
Sep 14, 2026
…apache#72853) * Fix PydanticAI vendor connections not resolving from secrets backends The three vendor connection types declared their `conn_type` with hyphens (`pydanticai-azure`, `pydanticai-bedrock`, `pydanticai-vertex`), but a hook is registered under the literal `connection-type` from `provider.yaml` while `Connection.from_uri` and `Connection.from_json` rewrite `-` to `_` before the lookup. The registered key was therefore unreachable from every secrets backend, including the `AIRFLOW_CONN_*` environment variables enabled by default, and any task using one of these connections failed with: AirflowException: Unknown hook type "pydanticai_azure" Only a connection read straight out of the metadata DB kept its hyphen and resolved, which is why this went unnoticed. Renames the three types to underscores, matching every other multi-word hook in the provider tree (`azure_data_factory`, `google_cloud_platform`, `spark_connect`). `default_conn_name` values were already underscored and are unchanged. * Reject hyphenated connection-type in the provider schema and document the rename Adds `"pattern": "^[a-z0-9_]+$"` to the connection-type property in both provider schemas, so a hyphenated type fails schema validation at authoring time instead of at a user's first task run. All 141 connection-type values across the provider tree already satisfy it. Also parametrizes the round-trip guard so it reports every offending type rather than aborting at the first, and notes the rename in the three connection docs for users whose connections are already stored with the old type. * Keep the connection-type gate in the authoring schema only The runtime schema (provider_info.schema.json) is validated against the get_provider_info() output of every installed provider, third-party included, and that validate() call is unwrapped in provider discovery. A pattern there turns one non-conforming provider into a failure for all provider discovery in the process, and every released apache-airflow-providers-common-ai wheel declares the hyphenated types, so installing a released provider against new core would fail to import Airflow at all. PR apache#71104 declined the same tightening for the same reason. The authoring gate is fully delivered by provider.yaml.schema.json, which the provider-yaml pre-commit validates for every in-tree provider. Tightens that authoring pattern to ^[a-z][a-z0-9_]*$ so it also rejects a leading digit, which is not a valid URI scheme under RFC 3986 and which Python 3.10 and 3.14 parse differently. Corrects the three connection-doc notes, which claimed every stored connection needed updating. URI and JSON values decode '-' back to '_' and resolve unchanged, so only types stored verbatim (metadata DB rows and object-form imports) need a change; those now carry the procedure. Also fixes the hyphenated connection-type example in the new-provider guide, and pins the resolution tests to the exact hook class with a per-parametrization conn_id. * Anchor the connection-type pattern past a trailing newline A schema pattern is matched with re.search, and Python's '$' matches before a trailing newline as well as at the end of the string, so ^[a-z][a-z0-9_]*$ accepted a connection type carrying one. A YAML block scalar produces exactly that, so 'connection-type: |' followed by an indented name would have registered a hook under a name no connection can reach. A hyphen was already rejected either way, since '-' is outside the character class and the newline allowance excuses only a newline, so this does not reopen the case the pattern was added for. It closes the one value that could still slip past it. Adds the first test for this pattern, covering the spellings that cannot be resolved: a hyphen, a block scalar's trailing newline, uppercase, a leading digit, whitespace and an embedded newline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#70190 (comment)
The items blocks for
queues, plugins and task-decoratorslisted their field names directly under items, where they are not JSON Schema keywords, so the blocks were inert and any shape passed. A provider author following what those blocks appeared to document — a dict for queues, or a path key for a decorator — got no error and was quietly dropped by the consumers instead.What
Pinning each section to the shape the provider.yaml files and generated payloads actually use turns that into a failure at authoring time. The custom-provider howto named the same nonexistent path field.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.