Repository navigation
Link registry modules to the guide section that documents them - #71477
Conversation
38c6381 to
b708b7a
Compare
b708b7a to
e84eaec
Compare
a83ec1d to
78c01a0
Compare
78c01a0 to
72aeea4
Compare
|
Round-1 items all landed, thanks. Four follow-ups and two nits, none blocking.
Two nits. |
|
The
|
72aeea4 to
850b7d2
Compare
e11a7c9 to
d1bb360
Compare
|
Thanks, this looks good to me. A few non-blocking things, fine as a follow-up or in this PR, your call:
|
A module card offers only generated API reference, which lists arguments but never says how the thing is meant to be used. The prose guides say it, and they already mark which class each section is about by titling that section with the class name -- so the pointer exists in the docs and just wasn't being read. Resolving it from the guides rather than from a declared list is deliberate: a hand-maintained class-to-guide table goes stale silently every time a guide is split or renamed, and a link that lands on the wrong section is worse than no link at all. A class documented only in prose gets no link. Both extraction paths resolve it, since a superseded version's page is rendered only from its own per-version metadata.
The catalog carries task-flow decorators as modules of their own, named `@task.agent` and `@task.llm_file_analysis`, and a guide documents such a decorator in the same section as the operator it wraps -- titled ``AgentOperator`` & ``@task.agent``. Only the leading literal was read, so the decorator half of those sections never got a link although the title named it. A title's leading run of inline literals is now collected, one anchor shared by every name in the run. The run stops at the first thing that is neither a name nor a "&", "," or "/" separator, so a prose title that happens to mention a literal still produces nothing.
Both readers handed every `.rst` under a provider's docs directory to the anchor collector. `_api/` is gitignored autoapi output, so the working-tree reader saw those pages on any tree where the docs had been built while the git-tag reader never can -- the anchor set a provider ended up with depended on whether a build had run where the extractor happened to execute. `_`-prefixed paths also sort ahead of lowercase ones, so they won the first-page tie-break against the guide that actually documents the class. `is_guide_page` now gates both readers on the same rule: nothing under a `_`-prefixed path segment at any depth, and neither `changelog.rst` nor `commits.rst` -- real pages, but release notes rather than how-to guides, whose headings can be inline-literal-formatted by coincidence. Gating both readers on one predicate is what keeps the two paths from drifting apart again. The filter is not a cost fix: at `providers-amazon/9.17.0` it takes the file count from 102 to 98 and leaves the wall time where it was (~1.9s), because the cost is one `git show` subprocess per file rather than the bytes read. Batching those reads is left to a change of its own.
Covering another provider's modules is a matter of that provider's section titles leading with an inline literal -- the extractor needs no change for it. Worth saying next to the convention, along with the fact that a matching title still needs a same-named module in the catalog before a link appears.
The earlier field-count fix counted `guide_url` as a fixed thirteenth field. `attach_guide_urls` skips a class no guide documents, so those entries carry twelve keys, and a bare count cannot say which one a reader will get. Naming the field alongside its condition does, and matches what `make_entry` already says a few lines below.
`validate_modules_catalog` returns the dict it was handed rather than a `model_dump()`, so asserting on that dict said nothing about `ModuleContract`: the absent-field test could not fail at all, and the one named for a round trip performed none. Both now read the attribute off `ModuleContract.model_validate`, where the default and the parsed value actually live, and the second test is named for what it checks.
"Nothing declares that link" was wrong: `provider.yaml` declares `how-to-guide` for integrations and transfers, and `check_doc_files` set-compares it against the operator, sensor and transfer guide paths, so a stale entry fails CI rather than rotting. What that declaration cannot do is name a section inside a page, and it never covers the toolset, hook and decorator pages this reads -- which is the gap the title convention fills, and the reason worth recording in a file future readers take instructions from.
The row holds three links now that a Guide link sits between Docs and Source, and it was a flex row with no wrap, so the third one is pushed out of the card at narrow viewports or a raised root font size.
Filtering to guide pages ahead of the read, as the previous change did, took
`providers-amazon/9.36.0` from 111 files to 107 -- the cost was never the bytes
read, it was one `git show` subprocess per file, and `--all-versions` pays it
once per provider-version. `git_cat_file_batch` hands the survivors to a single
`git cat-file --batch` on stdin: 107 subprocesses and ~1.9s become one and
~0.05s, for a dict that compares equal to the loop's, key for key and value for
value.
The batch protocol has to be parsed on bytes, because its header counts bytes
and decoding before slicing would drift on multi-byte content. Content is then
decoded as UTF-8 explicitly rather than following the process locale, and a
decode failure is left to propagate: `.rst` is Sphinx-convention UTF-8, and bad
data should fail loudly here the way a missing object already does.
`git_ls_tree` now runs with `core.quotePath=false`. It C-quoted a non-ASCII path
into a string that fails the `.endswith(".rst")` test the working-tree reader
passes, so the two readers disagreed about the same file. No such path exists
under `providers/` today -- this closes the divergence rather than fixing an
observed break.
Following the process locale for ls-tree while cat-file --batch decoded explicit UTF-8 meant a non-UTF-8 locale could break the round-trip of paths between the two. The subprocess mocks now carry autospec like the rest of the file.
The common.ai docs reorganization retitled its dedicated pages to lead with prose and end with the class name, so most of its modules lost their Guide link, and a subsection that happened to open with a name won over the page actually dedicated to it (HookToolset landed on the security guidelines). Older release tags keep the leading-literal titles, so both shapes are read.
@task.llm_file_analysis, @task.llm_sql, @task.llm_branch and @task.llm_schema_compare were documented only under an untitled "TaskFlow Decorator" subsection, so the registry found no section naming them. Each page's title now names its decorator alongside its operator, as the LLMOperator and AgentOperator pages already do.
d1bb360 to
cac4877
Compare
Guide links point at /stable, which serves released docs, so the anchors now come from the docs at the provider's release tag when it exists, instead of a working tree that may be ahead of it.
|
3 participants
A module card offers only generated API reference, which lists arguments but never says how the thing is meant to be used. The prose guides say it, and they already mark which class each section is about by titling that section with the class name -- so the pointer exists in the docs and just wasn't being read.
Resolving it from the guides rather than from a declared list is deliberate: a hand-maintained class-to-guide table goes stale silently every time a guide is split or renamed, and a link that lands on the wrong section is worse than no link at all. A class documented only in prose gets no link.
Both extraction paths resolve it, since a superseded version's page is rendered only from its own per-version metadata.
Was generative AI tooling used to co-author this PR?
Generated-by: [Claude] following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.