feat(docs): Publish the documentation as an MkDocs site - #2193
Merged
Hzfengsy merged 5 commits intoJul 29, 2026
Merged
Conversation
📝 WalkthroughWalkthroughThe pull request adds a bilingual MkDocs documentation site, expands documentation navigation and entry pages, rewrites repository links during builds, validates navigation coverage, and automates strict builds and deployment through pre-commit and GitHub Actions. ChangesDocumentation site
Estimated code review effort: 3 (Moderate) | ~30 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69eba34636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # Matches inline markdown links and images: `[label](target)` / ``, | ||
| # with an optional `"title"` after the target. Group 3 is the target. | ||
| _LINK = re.compile(r'(!?)\[([^\]]*)\]\(([^)\s]+)((?:\s+"[^"]*")?)\)') |
There was a problem hiding this comment.
When documentation uses reference-style links, this regex never rewrites their definitions; for example, docs/en/dev/passes/36-auto_derive_task_dependencies.md defines [pass-source] as ../../../../src/... on line 159, as does its Chinese mirror. MkDocs therefore sees an out-of-docs target, emits the configured not_found warning, and mkdocs build --strict turns that warning into a failure, so the newly added build and deployment workflow cannot complete until reference definitions are rewritten too.
Useful? React with 👍 / 👎.
|
|
||
| deploy: | ||
| # Only publish from main. PRs stop after `build`. | ||
| if: github.event_name != 'pull_request' |
There was a problem hiding this comment.
When an authorized user dispatches this workflow from any non-main branch, github.event_name is workflow_dispatch, so this condition runs the deploy job and force-pushes that branch's rendered documentation to the production gh-pages branch. This contradicts the preceding “Only publish from main” constraint and can replace the public site with unmerged content; include a github.ref == 'refs/heads/main' restriction.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/docs.yml:
- Around line 90-92: Update the deploy job’s `if` condition so publishing only
runs when the workflow is not a pull request and the GitHub ref is the main
branch. Preserve the existing build behavior and prevent manually dispatched
runs from feature branches from deploying.
In `@mkdocs.yml`:
- Around line 34-41: Update the existing validation.links configuration in
mkdocs.yml to add anchors: warn alongside the other link checks, ensuring broken
in-site fragment links are treated as warnings and cause the strict docs build
to fail.
In `@scripts/mkdocs_hooks.py`:
- Around line 54-66: Update _FENCE and _code_spans to retain the opening fence’s
delimiter character and length, and only close a span when a later fence uses
the same character, has at least the opening length, and has no non-whitespace
suffix. Preserve unmatched-opening behavior through the end of the markdown and
reset the tracked opening fence after closing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a90de24e-d918-4eba-95f0-02e510a5eba9
⛔ Files ignored due to path filters (1)
docs/assets/pypto-arch.pngis excluded by!**/*.png
📒 Files selected for processing (32)
.github/workflows/docs.yml.pre-commit-config.yamlREADME.mdREADME.zh-CN.mddocs/en/dev/backend/index.mddocs/en/dev/codegen/index.mddocs/en/dev/debug/index.mddocs/en/dev/index.mddocs/en/dev/ir/00-overview.mddocs/en/dev/ir/index.mddocs/en/dev/language/index.mddocs/en/dev/passes/index.mddocs/en/index.mddocs/en/reference/index.mddocs/en/reference/pto-isa/index.mddocs/en/user/index.mddocs/zh-cn/dev/backend/index.mddocs/zh-cn/dev/codegen/index.mddocs/zh-cn/dev/debug/index.mddocs/zh-cn/dev/index.mddocs/zh-cn/dev/ir/00-overview.mddocs/zh-cn/dev/ir/index.mddocs/zh-cn/dev/language/index.mddocs/zh-cn/dev/passes/index.mddocs/zh-cn/index.mddocs/zh-cn/reference/index.mddocs/zh-cn/reference/pto-isa/index.mddocs/zh-cn/user/index.mdmkdocs.ymlpyproject.tomlscripts/mkdocs_hooks.pytests/lint/check_docs_nav.py
Comment on lines
+90
to
+92
| deploy: | ||
| # Only publish from main. PRs stop after `build`. | ||
| if: github.event_name != 'pull_request' |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Restrict publishing to main.
Line 92 also permits workflow_dispatch, so a manually dispatched run from a feature branch can force-push that branch’s site to gh-pages. Gate deployment on the main ref.
Proposed fix
- if: github.event_name != 'pull_request'
+ if: github.ref == 'refs/heads/main'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| deploy: | |
| # Only publish from main. PRs stop after `build`. | |
| if: github.event_name != 'pull_request' | |
| deploy: | |
| # Only publish from main. PRs stop after `build`. | |
| if: github.ref == 'refs/heads/main' |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/docs.yml around lines 90 - 92, Update the deploy job’s
`if` condition so publishing only runs when the workflow is not a pull request
and the GitHub ref is the main branch. Preserve the existing build behavior and
prevent manually dispatched runs from feature branches from deploying.
Comment on lines
+34
to
+41
| validation: | ||
| links: | ||
| not_found: warn | ||
| absolute_links: warn | ||
| unrecognized_links: warn | ||
| nav: | ||
| omitted_files: warn | ||
| not_found: warn |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repo files matching mkdocs/config:'
git ls-files | grep -E '(^|/)mkdocs\.ya?ml$|requirements.*\.txt$|pyproject\.toml$' || true
if [ -f mkdocs.yml ]; then
printf '\n--- mkdocs.yml relevant section ---\n'
nl -ba mkdocs.yml | sed -n '1,80p'
fi
printf '\n--- dependency/context references ---\n'
for f in requirements.txt requirements-dev.txt requirements-base.txt pyproject.toml; do
if [ -f "$f" ]; then
echo "### $f"
rg -n 'mkdocs|strict|plugin|navigation|validation' "$f" || true
fi
done
printf '\n--- lockfile pin candidates ---\n'
for f in Pipfile.lock poetry.lock .github/workflows/*.yml .github/workflows/*.yaml poetry.lock; do
if [ -f "$f" ]; then
echo "### $f"
rg -n 'mkdocs|version|uses:' "$f" || true
fi
doneRepository: hw-native-sys/pypto
Length of output: 294
🌐 Web query:
MkDocs 1.6 configuration validation.links.anchors default info strict mode invalid anchor warning
💡 Result:
In MkDocs 1.6, a new feature for anchor validation was introduced to detect broken links to headings within documents [1][2][3]. Configuration and Behavior: - The setting validation.links.anchors controls the strictness of these checks [1][4]. - The default value is info [1][5][4]. - When set to warn, MkDocs will output warning messages for missing anchors during the build process [2][3]. - If you run mkdocs build --strict, setting this option to warn (or higher) will cause the build to fail if any invalid anchors are detected [2][6]. - To disable these checks entirely, you can set the option to ignore [2][3][6]. Configuration Example: To enable warnings for missing anchors, add the following to your mkdocs.yml [2][3]: validation: links: anchors: warn Technical Context: - This validation detects Markdown-authored anchors and standard heading-generated anchors [2][3]. - Anchors generated by external plugins or extensions may be detected incorrectly if the plugin does not insert them as etree elements during the appropriate phase of processing (e.g., if they use raw HTML or incorrect treeprocessor priorities) [3][7][8]. - If you encounter false positives, it is recommended to ensure your anchors match the format generated by Python-Markdown and that any extensions you use are compatible with this validation mechanism [7][9]. For maximal strictness (a configuration recommended for many sites), you can set multiple validation options to warn [1][4]: validation: omitted_files: warn absolute_links: warn # Or 'relative_to_docs' unrecognized_links: warn anchors: warn
Citations:
- 1: https://www.mkdocs.org/user-guide/configuration/
- 2: https://github.com/mkdocs/mkdocs/blob/master/docs/about/release-notes.md
- 3: https://newreleases.io/project/github/mkdocs/mkdocs/release/1.6.0
- 4: https://github.com/mkdocs/mkdocs/blob/master/docs/user-guide/configuration.md
- 5: https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/defaults.py
- 6: https://deepwiki.com/mkdocs/mkdocs/3.3-page-processing
- 7: Anchor validation false positives after upgrading to 1.6 mkdocs/mkdocs#3690
- 8: Implement validation of anchors in links to other pages mkdocs/mkdocs#3463
- 9: Anchor validation and special characters. mkdocs/mkdocs#3704
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- mkdocs.yml lines 1-120 ---\n'
awk '{printf "%6d %s\n", NR, $0}' mkdocs.yml | sed -n '1,120p'
printf '\n--- pyproject mkdocs config ---\n'
awk '{printf "%6d %s\n", NR, $0}' pyproject.toml | sed -n '1,200p'
printf '\n--- workflow references to mkdocs build / strict (if workflows exist) ---\n'
python3 - <<'PY'
from pathlib import Path
for p in Path('.github/workflows').glob('*'):
if p.is_file():
txt = p.read_text(errors='ignore')
if 'mkdocs build' in txt or '--strict' in txt or 'uses:' in txt:
print(f'### {p}')
lines = txt.splitlines()
for i,l in enumerate(lines,1):
if 'mkdocs build' in l or '--strict' in l or 'uses:' in l:
lo=max(1,i-2); hi=min(len(lines),i+2)
for j in range(lo,hi+1):
print(f'{j:4}: {lines[j-1]}')
PYRepository: hw-native-sys/pypto
Length of output: 248
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- mkdocs.yml lines 1-120 ---'
awk '{printf "%6d %s\n", NR, $0}' mkdocs.yml | sed -n '1,120p'
printf '%s\n' ''
printf '%s\n' '--- pyproject mkdocs config ---'
awk '{printf "%6d %s\n", NR, $0}' pyproject.toml | sed -n '1,200p'
printf '%s\n' ''
printf '%s\n' '--- workflow references to mkdocs build / strict (if workflows exist) ---'
python3 - <<'PY'
from pathlib import Path
root = Path('.github/workflows')
if not root.exists():
print('no .github/workflows directory')
else:
for p in root.glob('*'):
if p.is_file():
txt = p.read_text(errors='ignore')
if 'mkdocs build' in txt or '--strict' in txt or 'uses:' in txt:
print(f'### {p}')
lines = txt.splitlines()
for i,l in enumerate(lines,1):
if 'mkdocs build' in l or '--strict' in l or 'uses:' in l:
lo=max(1,i-2); hi=min(len(lines),i+2)
for j in range(lo,hi+1):
print(f'{j:4}: {lines[j-1]}')
PY
printf '%s\n' ''
printf '%s\n' '--- dependency/config references ---'
rg -n 'mkdocs|strict|plugin|navigation|validation|python =|dependencies|tool\.uv|requires-python' pyproject.toml requirements*.txt .github/workflows || trueRepository: hw-native-sys/pypto
Length of output: 26689
Make broken heading links fail CI.
mkdocs build --strict is the docs job, but MkDocs 1.6 defaults validation.links.anchors to info, so untargeted #fragment links are not treated as warnings. Add validation.links.anchors: warn under the existing link checks so broken in-site anchor navigation fails the build.
Proposed fix
validation:
links:
not_found: warn
+ anchors: warn
absolute_links: warn
unrecognized_links: warn📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| validation: | |
| links: | |
| not_found: warn | |
| absolute_links: warn | |
| unrecognized_links: warn | |
| nav: | |
| omitted_files: warn | |
| not_found: warn | |
| validation: | |
| links: | |
| not_found: warn | |
| anchors: warn | |
| absolute_links: warn | |
| unrecognized_links: warn | |
| nav: | |
| omitted_files: warn | |
| not_found: warn |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@mkdocs.yml` around lines 34 - 41, Update the existing validation.links
configuration in mkdocs.yml to add anchors: warn alongside the other link
checks, ensuring broken in-site fragment links are treated as warnings and cause
the strict docs build to fail.
Comment on lines
+54
to
+66
| def _code_spans(markdown: str) -> list[tuple[int, int]]: | ||
| """Return (start, end) offsets of fenced code blocks, so they can be skipped.""" | ||
| spans: list[tuple[int, int]] = [] | ||
| open_at: int | None = None | ||
| for match in _FENCE.finditer(markdown): | ||
| if open_at is None: | ||
| open_at = match.start() | ||
| else: | ||
| spans.append((open_at, match.end())) | ||
| open_at = None | ||
| if open_at is not None: | ||
| spans.append((open_at, len(markdown))) | ||
| return spans |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Track the opening fence before skipping code spans.
Lines 58-63 treat every fence-looking line as the next close. A ```` block containing or a block containing ~~~ closes early, allowing links in code examples to be rewritten. Track the opening delimiter character and length; only close on the same delimiter with at least the opening length and no info-string suffix.
Proposed fix
-_FENCE = re.compile(r"^(\s*)(```+|~~~+)", re.MULTILINE)
+_FENCE = re.compile(r"^( {0,3})(`{3,}|~{3,})([^\n]*)$", re.MULTILINE)
def _code_spans(markdown: str) -> list[tuple[int, int]]:
spans: list[tuple[int, int]] = []
open_at: int | None = None
+ opening_fence: str | None = None
for match in _FENCE.finditer(markdown):
+ fence = match.group(2)
+ suffix = match.group(3)
if open_at is None:
open_at = match.start()
- else:
+ opening_fence = fence
+ elif (
+ opening_fence is not None
+ and fence[0] == opening_fence[0]
+ and len(fence) >= len(opening_fence)
+ and not suffix.strip()
+ ):
spans.append((open_at, match.end()))
open_at = None
+ opening_fence = None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _code_spans(markdown: str) -> list[tuple[int, int]]: | |
| """Return (start, end) offsets of fenced code blocks, so they can be skipped.""" | |
| spans: list[tuple[int, int]] = [] | |
| open_at: int | None = None | |
| for match in _FENCE.finditer(markdown): | |
| if open_at is None: | |
| open_at = match.start() | |
| else: | |
| spans.append((open_at, match.end())) | |
| open_at = None | |
| if open_at is not None: | |
| spans.append((open_at, len(markdown))) | |
| return spans | |
| def _code_spans(markdown: str) -> list[tuple[int, int]]: | |
| """Return (start, end) offsets of fenced code blocks, so they can be skipped.""" | |
| spans: list[tuple[int, int]] = [] | |
| open_at: int | None = None | |
| opening_fence: str | None = None | |
| for match in _FENCE.finditer(markdown): | |
| fence = match.group(2) | |
| suffix = match.group(3) | |
| if open_at is None: | |
| open_at = match.start() | |
| opening_fence = fence | |
| elif ( | |
| opening_fence is not None | |
| and fence[0] == opening_fence[0] | |
| and len(fence) >= len(opening_fence) | |
| and not suffix.strip() | |
| ): | |
| spans.append((open_at, match.end())) | |
| open_at = None | |
| opening_fence = None | |
| if open_at is not None: | |
| spans.append((open_at, len(markdown))) | |
| return spans |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/mkdocs_hooks.py` around lines 54 - 66, Update _FENCE and _code_spans
to retain the opening fence’s delimiter character and length, and only close a
span when a later fence uses the same character, has at least the opening
length, and has no non-whitespace suffix. Preserve unmatched-opening behavior
through the end of the markdown and reset the tracked opening fence after
closing.
Scaffolding for the user manual plan (issue hw-native-sys#2120): the existing 162 markdown files go online as-is, so every later documentation batch is reviewable as rendered pages and `--strict` guards cross-references from day one. - `mkdocs.yml`: Material theme with four tabs (User Manual / Reference / Developer / an external Runtime link), `mkdocs-static-i18n` in folder mode over the existing `docs/en` + `docs/zh-cn` layout, and `mkdocstrings` configured up front so the generated `pl.*` reference has a fixed docstring contract to land against. - `.github/workflows/docs.yml`: pushes to main build and deploy to `gh-pages`; pull requests build only. The job installs the `docs` extra's dependencies directly rather than via `pip install -e ".[docs]"`, which would trigger a full C++ build the site does not need -- mkdocstrings reads docstrings statically. - `scripts/mkdocs_hooks.py`: 87 links in the existing docs point at source files outside `docs/` (`python/`, `include/`, `examples/`, `.claude/rules/`). MkDocs cannot resolve those and `--strict` fails on each one, so the hook rewrites them to GitHub blob URLs at build time. The markdown keeps repo-relative links and stays clickable when browsing the repository. - `tests/lint/check_docs_nav.py`: fails when a page is missing from the nav -- which would leave it reachable only by direct URL -- or when a nav entry points at a deleted file. Wired into pre-commit beside the existing en/zh parity check. - 22 `index.md` landing pages, one per directory in both locales, each listing its pages with a one-line description of the target. - `docs/images/` -> `docs/assets/`, the shared-asset location the i18n folder mode expects. The site is a build artifact and is never committed; the markdown under `docs/` remains the single source of truth and stays readable on GitHub.
The runtime (hw-native-sys/simpler) publishes its site from the same MkDocs Material + mkdocstrings stack, and four of its choices are better than what this PR started with. - Publish through `actions/upload-pages-artifact` + `actions/deploy-pages` instead of `mkdocs gh-deploy`. No `gh-pages` branch to carry, and the repository setting becomes Settings -> Pages -> Source = "GitHub Actions". - Move the toolchain to `docs/requirements.txt` so CI installs it with a plain `pip install -r`, replacing the pyproject `docs` extra that the workflow had to parse back out with `tomllib` to avoid triggering a full C++ build. - Point rewritten repo links at `DOCS_REF` (`main` on a push, the commit SHA on a pull request) rather than a hard-coded `blob/main`. A source file added in a PR does not exist on main yet, so its link used to 404 in the PR's own build. - Stop rewriting images and emit `tree/` rather than `blob/` for directory targets. Neither case occurs today, but a `blob/` URL renders as an HTML page, so an image rewritten that way would silently break instead of failing `--strict`. Also drops the shebang from `check_docs_nav.py`: `check_headers.py` requires the copyright block on line 1, and only flagged it once the file became tracked.
The Pages build type on hw-native-sys/pypto is already "GitHub Actions" and the `github-pages` environment already restricts deployments to `main`, so no manual setup is outstanding. Note it where the next reader would otherwise re-derive it -- and where it explains why publishing must not go through a `gh-pages` branch, which that build type ignores entirely.
`mkdocs build --strict` aborted on the config:
Plugin 'i18n' option 'languages': Sub-option 'locale': Language code values
must be either ISO-639-1 lower case or represented with their
territory/region/country codes, received 'zh-cn'
mkdocs-static-i18n validates each locale against
(^[a-z]{2}(-[A-Za-z]{4})?(-[A-Z]{2})?$)|(^[a-z]{2}_[A-Z]{2}$)
so a lower-case territory is rejected. `zh-CN` and `zh_CN` pass that check but
fail later: the plugin assigns the locale straight to `theme.language`, and
Material's partials/language.html does a hard
`{% import "partials/languages/" ~ language ~ ".html" %}`. Material ships
zh.html, zh-Hant.html and zh-TW.html but no zh-CN.html, and the en.html fallback
beside that import only covers missing keys, not a missing file -- so anything
but `zh` raises TemplateNotFound at render time.
In folder mode the locale is the directory name, so the directory has to move
with it. Nothing about the content changes; `docs/en` remains authoritative and
the trees stay mirrored page for page.
Updated alongside the 92 moved files: the en/zh parity and english-only lint
scripts, the pre-commit exclude, `.claude/rules/documentation.md` and
`pass-doc-ordering.md`, the add-op and code-review skills, AGENTS.md, and both
READMEs. Also corrects an mkdocs.yml comment left stale by the switch to
actions/deploy-pages, which still claimed CI publishes to a `gh-pages` branch.
lyfne123
force-pushed
the
docs/mkdocs-site-scaffolding
branch
from
July 29, 2026 03:30
df26bdf to
b6791bd
Compare
`mkdocs build --strict` failed on the one link form the hook did not handle:
WARNING - Doc file 'en/dev/passes/36-auto_derive_task_dependencies.md'
contains a link '../../../../src/ir/transforms/auto_derive_task_dependencies_pass.cpp',
but the target is not found among documentation files.
The target is declared as a reference definition on its own line --
`[pass-source]: ../../../../src/...` -- not as an inline `[text](url)`. MkDocs
resolves both the same way, so a definition escaping `docs/` fails `--strict`
exactly like an inline link, but the hook's regex only matched the inline form.
Adds `_REF_DEF` and factors the shared resolve-and-rewrite step into
`escaped_url()`, so both forms go through one code path. Two definitions in the
docs are affected (the en and zh copies of the same page); the two that stay
inside `docs/` are left for MkDocs to validate.
Hzfengsy
approved these changes
Jul 29, 2026
7 tasks
georgebisbas
added a commit
to georgebisbas/pypto
that referenced
this pull request
Jul 29, 2026
…iew findings Rebases the restructured PR hw-native-sys#2162 chapters onto main's MkDocs site scaffold (hw-native-sys#2193) and the docs/zh-cn -> docs/zh locale rename, adds the new distributed/ and performance/ chapters to mkdocs.yml nav and the top-level user manual index, and retargets cross-references that still pointed at the deleted flat 04/05/06 files. Also fixes markdownlint failures introduced by the restructure commit, and addresses the outstanding technical-accuracy review comments: - distributed/00-model.md: the Quickstart HelloAllReduce example now matches the working test_l3_allreduce.py reference (adds the missing Orchestration dispatch layer, fixes a rank mismatch from pl.slice, drops a redundant store, and uses pld.world_size() for the signal window). - distributed/02-primitives.md: window/alloc_window_buffer moved out of the pld.system.* table into their actual pld.tensor.* namespace. - distributed/01-collectives.md and tensor_ops.py (6 docstrings): corrects the host-builtin signal shape (rank-1 or rank-2, not only 1-D) and replaces the InCore-hand-rolled-vs-HOST-builtin dichotomy with the real three-way split (hand-rolled / InCore composite / HOST builtin), since ring mode is InCore-composite-only. - distributed/03-execution.md, performance/00-methodology.md: drops fabricated PYPTO_BENCH* defaults that belong to pypto-lib, not this repo. - distributed/04-debugging.md: removes a non-functional `cmake -D...` example for compile-time macros that no CMakeLists.txt in the repo reads. - performance/02-distributed.md: corrects Ring Sizing to reflect that RunConfig must be passed to every dispatch, not just prepare(), and states the real power-of-2 size constraints. - 02-operation_reference.md: drops a stale branch reference and an unverified dtype restriction from the AllReduce capability matrix. - bench.py: fixes a malformed RST table (column marker narrower than its widest cell). All changes mirrored in docs/zh/user/.
YunjiQin
pushed a commit
that referenced
this pull request
Jul 30, 2026
…-level CommRemoteOffset helper (#2168) > **Rebased onto `main` (`a55399d4`); #2135 is closed, so its 5 toolchain commits (`update PTOAS to v0.51` … `fix(runtime): Keep the _ptoas_binary seam`) now land here.** The PTOAS pin moved on to **v0.54** in the process. ## What 1. Pin PTOAS **v0.54**. 2. Reland the `InsertCommFence` pass (reverts #2138). 3. Collapse the `system.cacheinvalid` codegen down to a single path. 4. Restore the module-level `@CommRemoteOffset_<dtype>` helper that #2161 inlined. ## 1. PTOAS v0.54 `toolchain/versions.env` is the single source of truth — every workflow reads it through the `toolchain` lead job, so the bump needs no workflow change. Both digests were verified by downloading the release assets and hashing them locally: | Arch | sha256 | | ---- | ------ | | aarch64 | `011e980dbc46c796e31a1b213051d943ba8eb4c67d356ae6bda2148fe512964a` | | x86_64 | `4e3acb9623384c18fe264610525777210095f2ba24f6c52b9823bf1cb81d7a99` | The release tarball layout changed at v0.51 (`<root>/ptoas` went from an executable launcher script to a Python package directory). `python/pypto/backend/_ptoas_locate.py` probes each candidate for being an executable file rather than keying off the release version, so it is version-agnostic and needed no change for v0.52 or v0.54. ## 2. Reland InsertCommFence (reverts #2138) #2138 backed out #2076 because ptoas could not lower the publish-side region `cacheinvalid`: | ptoas | `pto.cmo.cacheinvalid <partition_tensor_view> single_cache_line` | | ----- | --------------------------------------------------------------- | | 0.50 | parsed, but **no call emitted** — the marker never reached the device | | 0.51 | emitted `PTOAS__DCCI_SINGLE_CACHE_LINE(<GlobalTensor>)`, whose body casts to `__gm__ void*` — a conversion `GlobalTensor` does not have, so every kernel carrying the op failed to compile | hw-native-sys/PTOAS#1001 fixes this with a `GlobalTensor` overload that takes the address via `tensor.data()`, shipped in v0.52 and carried by the v0.54 pin. Verified against the real 0.52 binary — the emitted C++ now carries both overloads and binds the object one: ```cpp static AICORE inline void PTOAS__DCCI_SINGLE_CACHE_LINE( pto::GlobalTensor<Element, Shape, Stride, TensorLayout> &tensor) { dcci((__gm__ void*)tensor.data(), cache_line_t::SINGLE_CACHE_LINE); } ``` This is a plain revert of #2138 except for three deliberate deviations: - `toolchain/versions.env` stays on this branch's pin (now v0.54); the revert's restore of the v0.50 pin is dropped. - The pass doc is renumbered **43 → 44**. #2141 landed `LegalizeTileCast` at slot 14 and shifted everything below it, so `classify_iter_arg_carry` now owns 43 and `InsertCommFence` — still dead last in the pipeline — takes 44. - The revert's CommRemoteOffset inlining is **not** relanded, because #2161 landed the same inlining on `main` independently. See §4 — this PR takes that emission the other way. `44-insert_comm_fence.md` is wired into the mkdocs nav, `passes/index.md` and `00-pass_manager.md` (en + zh). The docs became an MkDocs site in #2193, and `mkdocs build --strict` fails on a page absent from the nav; `docs/zh-cn/` was also renamed to `docs/zh/` there, which the rebase picked up. ## 3. Route every `cacheinvalid` region through `partition_view` The scalar-write branch of `system.cacheinvalid` codegen emitted a bare pointer operand. Measured against ptoas 0.52: | operand | result | | ------- | ------ | | `!pto.ptr`, no type annotation ← **what we emitted** | parse error: `expected ':'` | | `!pto.ptr`, with type annotation | lowering: `addptr must feed make_tensor_view, ...` | | `!pto.tensor_view<1xf32>` | compiles | | `!pto.partition_tensor_view<1x1xf32>` | compiles | So that branch has never produced working code. It is reachable from the DSL (`pl.system.cacheinvalid(t, [1, 1], off)`) and, with this reland, from `InsertCommFence` too — `MakeCacheInvalid` uses the target's full shape, so any published tensor that is itself 1x1 lands there. The special case only existed because the region path was broken on ptoas <= 0.51. Now that it lowers correctly, a 1x1 `partition_view` is right: verified on the real binary, a `[1, 1]` region at offsets `[0, 8]` over a `[16, 16]` f32 tensor emits `GlobalTensor<float, Shape<1,1,1,1,1>, Stride<16,16,16,16,1>>` at `v1 + 8`. End-to-end, pypto's own generated `.pto` for that case now compiles where it previously hit the parse error. The branch is therefore deleted — one construction, one emit site. `GetFlatOffsetSSA` and `GetTensorBasePtr` remain in use elsewhere, so nothing is orphaned. **This supersedes #2137**, which fixes the same bug by routing the scalar case through `tensor_view<1xT>` and converging the two branches on a shared emit. Both work on 0.52; this one removes the branch entirely. Closing one of the two is a call for the authors — see "Open questions". `test_cacheinvalid_scalar_write_emits_ptr` asserted the broken form (`"partition_tensor_view" not in cmo_line`) and could not survive the fix as written. It is folded into a parametrized `test_cacheinvalid_region_emits_partition_view` covering both sizes; the dynamic-offset test now asserts the partition-view operand instead of `pto.addptr`. ## 4. Restore the module-level CommRemoteOffset helper #2161 ("Complete arbitrary-length allreduce support") inlined the distributed peer-address calculation at every remote-op call site and deleted the per-dtype `@CommRemoteOffset_<dtype>` helper. Its stated reason was that ptoas' `pto-memory-consistency` pass rejected a `func.call` to a callee holding the CommContext `pto.load_scalar` reads. **ptoas no longer performs that check**, so the helper form is viable again, and this PR restores it: - Op lowering registers the dtype via `PTOCodegen::RegisterCommRemoteOffsetHelper` and emits one `func.call @CommRemoteOffset_<dtype>(ctx, peer) -> index`. - `EmitCommRemoteOffsetHelpers` flushes one `func.func private` body per registered dtype at module end; MLIR resolves the forward references whole-module. - Sharing the CommContext field reads and the byte→element division across call sites keeps the emitted kernels smaller than the inlined form. `pto.addptr` and `pto.make_tensor_view` stay at the call site, as they did both before and after #2161 — PTOAS verifies per-function that `addptr` feeds `make_tensor_view`, and a tensor view cannot cross a func boundary because its lowered memref is strided. Returning the element offset is the only shape that satisfies both constraints. **Everything else from #2161 is preserved**: ragged-tail handling, DN / column-vector stride derivation, the tightened valid-shape inference, and the `allow_physical_tail_padding` attr on `pld.tile.remote_load`. Docs (en/zh), op descriptions, DSL docstrings, binding comments and the codegen unit-test assertions move back to the helper wording in lockstep. ## 5. Document that ptoas does not gate the markers The pass doc claimed that removing the wait-side `cacheinvalid all` from the ring-allreduce `.pto` "is rejected". That was measured on 0.50 and no longer holds. Re-run on 0.52 against the same `ring_step` kernel from `tests/st/distributed/collectives/test_l3_allreduce_ring.py`: | variant | ptoas 0.52 | | ------- | ---------- | | unmodified | accepted | | wait-side `cacheinvalid all` removed | accepted | | every `cacheinvalid` **and** `system.fence` removed | accepted | All three compile with no diagnostic; the instructions are simply absent from the generated C++. The doc now qualifies the original claim with the version it was measured on and adds a section stating there is no compile-time gate — a missing marker is a data race, not a build error — with an explicit warning against reading a green test run as evidence a marker is unnecessary. ## Testing - [x] Full unit suite on the rebased tree: `pytest tests/ut/ -n auto` → **8116 passed, 2 skipped** - [x] One pre-existing failure unrelated to this branch: `test_benchmark.py::test_benchmark_l3_ignores_prepare_setup_groups` — the test double's `prepare()` predates the `persistent=` kwarg added by #2163; `bench.py` and that test file are byte-identical to `main` here - [x] `tests/ut/ir/transforms/test_insert_comm_fence.py` + `tests/ut/codegen/distributed/` → 95 passed - [x] `ruff check` / `ruff format --check` / `clang-format --dry-run --Werror` clean on the diff - [x] Docs updated en + zh; new pass doc wired into the nav and both indices - [x] The ptoas claims in §2/§3/§5 were measured against the real v0.52 binary, not inferred - [ ] **`dist-system-tests` not run** — needs a 2-device host. This is the meaningful acceptance gate: those 36 cases are what #2138 backed the pass out for, and they are also what would exercise the restored `func.call` form from §4 on device. ## Open questions 1. **Overlap with #2137.** Same bug, same file, two fixes. One should close. 2. **The pass's benefit is inferred, not observed.** Under the 0.50 pin the publish-side region `cacheinvalid` emitted no call at all (hw-native-sys/PTOAS#995), so it never reached the device — and the distributed suite was green throughout. Combined with §5 (ptoas does not check for the markers), no existing test demonstrates what this pass fixes. Confirming it needs a case built to expose the race — large transfers, multiple ranks, repeated runs. Relates to #2076, #2138, #2161, hw-native-sys/PTOAS#995, hw-native-sys/PTOAS#1001
Hzfengsy
pushed a commit
that referenced
this pull request
Jul 30, 2026
…amming model (#2198) ## Summary Batch **B1** of the user manual plan (#2120), on top of the site scaffolding from #2193. Adds the three onboarding pages the manual was missing and turns the landing page into a real entry point — in both locales. | Page | What it covers | | ---- | -------------- | | `01-installation.md` (new) | Prerequisites, install modes, build options, where compile output goes, what each level of use actually requires, a tour of `examples/` | | `02-quickstart.md` (new) | **Tensor-level** kernels with `@pl.jit` — no manual data movement — operator chains, loops, splitting work across functions, compiling, reading the IR | | `03-programming-model.md` (new) | Tensor / Tile / Block levels, control plane vs execution plane, the pass pipeline, the memory hierarchy, the execution model | | `index.md` (rewritten) | Reading paths per goal, a capability-to-page table, and where the not-yet-written chapters' material currently lives | Two editorial decisions shaped the result: - **Everything is `@pl.jit`.** `examples/` is written entirely in that idiom; a reader who followed a `@pl.function` quickstart and then opened `examples/kernels/` would find a different language. - **The quickstart stays at tensor level.** `out = pl.add(a, b)` inside `with pl.at(...)`, with no `pl.load` / `pl.store` anywhere. Deciding what sits on chip and when is not a prerequisite for a first compile; `03-programming-model.md` introduces tile level and says explicitly why you would descend to it. ## Corrections found by executing the examples Every complete example was run against the current source. That caught seven inaccuracies — four in the material being migrated, three in my own drafts: | Claim | Reality | | ----- | ------- | | `output_dir = ir.compile(...)`, printed as a path | `ir.compile()` returns a `CompiledProgram`; the directory is `compiled.output_dir` (a `pathlib.Path`) | | `ir.compile()` takes 7 parameters | **15**. `dump_passes` also accepts a three-level `PassDumpLevel`, not just `bool` | | `pl.__all__` has 223 symbols (per the plan) | 226 | | `pl.muls`, `pl.assign` at the unqualified `pl.*` level | Neither exists there | | `compile(skip_ptoas=True)` skips ptoas | **Silently ignored.** `compile(*args, **kwargs)` binds the *kernel's* parameters. The artefact directory proved it: full `kernels/aiv/*.cpp` and a populated `ptoas/`. And the flag is unnecessary — `@pl.jit` calls `_ptoas_available()` and sets `skip_ptoas` itself | | `compile_for_test()` verifies a kernel | It stops before code generation, so it passes on codegen-stage failures. My first draft used it to "verify" examples and got a false green | | "44 passes" | A pass dump shows 45. Now phrased so it cannot drift | The `compile(skip_ptoas=True)` one is the interesting failure: it was in my own draft, it *appeared* to work on a machine that has ptoas, and only inspecting the output directory revealed the flag had done nothing. Constraints the examples surfaced, now documented: tile operations directly in a `@pl.jit` body fail with *"Misplaced tensor op ... should be inside InCore block"*; a module-level annotation alias fails because the parser reads annotations as source text; reassigning an `Out` parameter with a different dtype is rejected. ## What happens to `00-getting_started.md` Its introductory half moved into `02-quickstart.md`. The remaining ~60% is device-execution material whose destinations (`execution/01-run.md`, `performance/00-methodology.md`, `distributed/03-execution.md`) do not exist until B4–B6. Rather than duplicate or delete it, the file is retitled **"Running on Device"** and carries a table naming each section's destination. Content stays reachable and is maintained in exactly one place; only the address is temporary. ## Chinese Mirrors ship in the same commits — the `check-docs-en-zh-parity` gate makes English-only batches fail CI, so the plan's original "Chinese last" batch was dropped and folded into every batch. One deliberate asymmetry: Chinese cross-references link to pages, not in-page anchors. Python-Markdown's default slugify strips non-ASCII, so a `#中文标题` fragment resolves to nothing — this is why the build logs ~20 INFO-level broken anchors in existing `dev/` pages. Fixing it repo-wide wants a unicode-aware slugify (`toc: slugify: !!python/name:pymdownx.slugs.slugify`) and belongs in its own change. ## Testing | Check | Result | | ----- | ------ | | Examples executed via full `compile()` | 4 quickstart + 2 programming-model examples build end to end; installation-page snippet verified separately | | `check_docs_nav.py` | 95 nav entries cover all 95 pages | | `check_docs_en_zh_parity.py` | 95 paired paths | | Link audit (inline, images, reference definitions) | 93 rewritten, 778 internal links + 2 images resolve, 0 broken | | `markdownlint-cli2` | 0 errors | | `check_headers.py` / `check_english_only.py` | pass | | Page length vs the 500-line cap | New pages under. `01-language_guide.md` remains over at 704 — B2 splits it into `language/` | `mkdocs build --strict` runs on this PR via the docs workflow. **On the device jobs:** `system-tests-direct` and `dist-system-tests` have each failed once and passed once across the pushes on this branch, alternating, on a documentation-only diff. Both failures are pre-existing nondeterministic ones (`test_mat_scratch_dbc[ptoas-a2a3]` golden drift; the `tests/st/distributed` directory-level races). Neither is reachable from a markdown change. ## Follow-ups - `01-language_guide.md` still teaches the `@pl.function` / `@pl.program` class form while the quickstart teaches `@pl.jit`. B2 splits that page into `language/` and should reconcile the two idioms there. - The unicode-aware slugify fix, so Chinese in-page anchors work. ## Related Issues Part of #2120
Hzfengsy
pushed a commit
that referenced
this pull request
Aug 6, 2026
Second batch of the user manual (see #2120). Adds the two reference chapters the manual was missing, retires the page they replace, and adds a lint gate so the `pl.*` surface cannot drift out of the docs unnoticed. Follows #2193 (site scaffolding) and #2198 (skeleton). ## What lands **`docs/*/user/language/` — 7 pages ×2 languages.** One page per part of the language, each following the manual's fixed structure (Concept → Quickstart → Mechanics → Edge Cases → See Also): | Page | Covers | | ---- | ------ | | `00-types.md` | dtypes, `Tensor`/`Tile`/`Scalar`/`Array`, layouts, dynamic shapes, parameter directions | | `01-functions.md` | the `@pl.jit` family, `@pl.function`/`@pl.program`, cross-function calls | | `02-control-flow.md` | `range`/`parallel`/`unroll`/`pipeline`/`while_`, loop carries, `yield_`, SSA | | `03-memory.md` | the six on-chip spaces, `load`/`store`/`move`, valid shape and padding | | `04-scopes-and-tasks.md` | `at`/`cluster`/`spmd`/`split_aiv`, runtime scopes, `submit`, `deps=`, `predicate=` | | `05-directives.md` | `static_print`/`static_assert`, `dump_tag`/`dumps=`, `pl.array`, subscript sugar | **`docs/*/user/ops/` — 3 pages ×2 languages.** Which namespace an operator belongs to (`pl.*` vs `pl.tensor.*` vs `pl.tile.*`), how unified dispatch picks one, and the catalog. **Migration.** `02-operation_reference.md` is deleted — superseded by `ops/`. `01-language_guide.md` is reduced to its one section not covered by `language/` ("Compiling a Program"), which B6 will absorb into the execution chapter. 12 inbound links across both languages repointed. **`tests/lint/check_docs_symbol_coverage.py`** (wired into pre-commit). Reads `pypto.language.__all__` **statically** via `ast` — pre-commit has no built extension, so it must not import pypto — and fails when an exported symbol appears nowhere in `docs/en/user/**`. Symbols belonging to unwritten chapters sit in a `DEFERRED` list, each tagged with the batch that reclaims it; a `DEFERRED` entry that *has* since been documented also fails, so the list cannot rot. It caught 13 genuine omissions while this batch was being written. Coverage now: 209 of 222 documentable symbols, 13 deferred to B5/B6. ## Every complete example was compiled Each fenced example that defines a full `@pl.jit` entry is driven through a real `ir.compile()` — not `lower()`, which stops before code generation. **9 complete examples, 9 compiled, 0 failures.** Fragments stay fragments and are labelled as such. This is new for the manual, and it immediately paid for itself. Three examples did not compile: - a 1-D tensor example rejected by codegen (`tile.store tile valid_shape must be 2D`); - a `@pl.jit.host` example calling a function it never defined; - a control-flow Quickstart using `pl.create_tensor` inside an `.incore` body. The third was inherited from `01-language_guide.md` — **the old examples were never compile-verified, which is how it survived.** Chasing the last one surfaced two compiler defects, filed separately as #2228 (tile-typed loop carry emits `.pto` that ptoas rejects) and #2229 (unified `pl.add` dispatching to `tensor.add` for a Tile). ## Rebased onto `lower()` #2230 landed mid-batch and removed `compile_for_test()`. `language/01-functions.md` (EN+ZH) is synced: the IR-inspection guidance now distinguishes `lower()` (post-pass `ir.Program`) from `compiled.program.as_python()` (specialized, pre-pass), and the "verify with a full compile" pitfall no longer names a deleted API. Each new claim was executed rather than read off the docstring — `lower(*args)`, argument-less `lower()`, and the `AttributeError` on `JITFunction.as_python()`. ## Checks - Example compilation: 9/9, against this branch's build (import origin confirmed — an editable-install finder had silently redirected the first run to another worktree). - `check_docs_symbol_coverage.py`: exit 0. - Link/anchor sweep over 212 pages: zero problems under `user/`. The 6 reported are pre-existing `dev/passes/` headings containing `&`/`—`, where the slug approximation differs from python-markdown. - `check_docs_en_zh_parity` / `check_docs_nav` / `check_english_only`: pass. - `mkdocs build --strict` was not run locally (PyPI is unreachable from this host); the Docs workflow is the check for it.
lyfne123
pushed a commit
that referenced
this pull request
Sep 3, 2026
> **Phase 4 of [#2066](#2066 — dispatch-predicate phased rollout. Phase 1 ([#2067](#2067)) shipped the call form, Phase 3 ([#2092](#2092)) the `pl.spmd` scope form. This is the `pl.at` cut Phase 3 explicitly deferred. Pure frontend on the current runtime — no submodule bump. Rebased onto `main` (309 commits) with review feedback addressed; see the rebase notes in the comment below. ## What this does Extends `predicate=` to `with pl.at(level=pl.Level.CORE_GROUP, ...)`: ```python with pl.at(level=pl.Level.CORE_GROUP) as g_tid: # producer of rc rc = pl.store(pl.load(rc, [0, 0], [128, 128]), [0, 0], rc) with pl.at(level=pl.Level.CORE_GROUP, # producer is a dep deps=[g_tid], predicate=(rc[0, 0] > 0)) as tid: out = pl.store(pl.load(x, [0, 0], [128, 128]), [0, 0], out) ``` Same expression, validation, lowering and runtime ABI as the two existing forms. The predicate rides on `InCoreScopeStmt::attrs_` from parse until `OutlineIncoreScopes` moves it onto `Submit::predicate_`. ## Why it really was "parser + dsl_api only" Phase 3's prediction held. Every scope-attr walker — `IRVisitor::VisitScopeAttrs`, `IRMutator::MutateScopeAttrs`, `ConvertToSSA::SubstScopeAttrs`, DCE `FindLiveRoots`, `structural_hash`, `NoNestedCall::ShouldVisitScopeAttr` — and `ScopeOutliner` itself are already scope-kind agnostic. `pl.at` InCore scopes are outlined at pass 8, a strict *subset* of the passes the `pl.spmd` predicate (pass 9) already survives, so there is no new pass exposure. The only C++ additions are the printer hookup and two backstop asserts. ## Placement restrictions A predicate only has a runtime carrier where the scope becomes an **independently submitted L0 task**. Three placements would silently drop it, so all three are rejected at parse time: | Placement | Why it would be lost | | --- | --- | | `level != CORE_GROUP` | Builds a `HierarchyScopeStmt`, outlined into an `Opaque` level/role function that orchestration codegen never dispatches as a task. | | nested in `pl.cluster()` / `pl.spmd()` / another **CORE_GROUP** `pl.at` | The inner dispatch is folded into the enclosing Group / Spmd wrapper (or the enclosing kernel body); `BuildSpmdCallDispatchPlan` / `BuildAivOnlyGroupDispatchPlan` / `BuildMixedGroupDispatchPlan` all pass the **outer** wrapper call to `EmitPredicateHint`. | | in a `Group` / `Spmd` / InCore-family function body | `OutlineIncoreScopes` skips those function types, so **no `Submit` is built at all** — verified: the printed program after pass 9 still shows `with pl.at(..., predicate=(...)):` and contains no `pl.submit(`. | Nesting inside a **non**-CORE_GROUP `pl.at` is deliberately **allowed**. A Hierarchy scope is not a task wrapper: the inner CORE_GROUP scope is still outlined into its own `Submit` (whether the enclosing Hierarchy scope is itself outlined, in an `Opaque` host, or survives in place, in an `Orchestration` one), and the emitted orchestration C++ carries the same `set_predicate` as the un-nested form. Pinned by `test_at_predicate_accepted_inside_a_hierarchy_scope`. Mirrors `_reject_spmd_submit_only_kwargs_in_cluster`. For hand-built / deserialized IR the restrictions are re-asserted in C++ (same pattern as the existing `UnwrapNestedSpmd` guard): - `ScopeOutliner::OutlineScope` rejects a predicated Hierarchy scope it outlines, and a new program-wide `AssertNoHierarchyDispatchPredicate` runs in `OutlineHierarchyScopes` **before** its `Opaque`-only function skip — that skip is exactly what would otherwise hide such a scope from every `ScopeOutliner`. - A new `AssertNoInnerDispatchPredicate` in `OutlineClusterScopes` walks each Group / Spmd body for a `Submit::predicate_`, sweeping the pass's **final** function list rather than only the freshly outlined wrappers, so a wrapper the input program already declared is covered too. ## Scope-producer tracking `_record_scope_producer` (added for `pl.spmd` in Phase 3) now also runs on `with pl.at(...) as tid:`, so the Phase-2 producer∈`deps=` contract genuinely applies to tensors a `pl.at` body writes. `pl.at` accepts `deps=` on every form, so the remediation hint always offers `deps=[tid]`. (On current `main` `pl.spmd` does too, so this is no longer a difference between the two surfaces.) ## Outliner bug found and fixed (also affects the merged `pl.spmd` form) `ScopeOutliner` read `kAttrPredicate` raw, bypassing its own `store_target_renames_` substitution — a target-kind scope skips the base `MutateScopeAttrs` walk, since `VisitScopeKind` goes straight to `OutlineScope`. A `pl.at` body typically writes its tensor with `pl.store`, so after SSA the attr names a **scope-local post-store alias**. Outlining exports that store target under a fresh call-site Var and drops the alias, leaving the emitted `Submit` referencing a Var with no definition in the parent function: ```python ret__tmp_v0_1 = pl.submit(main_incore_1, ..., predicate=(0 < pl.cast(pl.tensor.read(rc__ssa_v1__FREE_VAR, [1, 2]), pl.INDEX))) # ^^^^^^^^^^^^^^^^^^^^^ dangling ``` `UseAfterDefCheck` catches it, so it fails loudly rather than miscompiling — but it made the feature unusable for the most natural `pl.at` spelling. The operand is now resolved through the same rename map the synthesised call's args use, i.e. to *the value current as of this scope*. This was **latent for `pl.spmd` too**: a `pl.spmd` predicate over a tensor written by an earlier sibling `pl.at` scope hits the identical path. The `pl.spmd` STs did not surface it because their producer scopes bind via a call result, not a store. ## Testing - **Parser** (`test_submit_predicate.py`) — attr landing, canonical attr order, print → reparse `assert_structural_equal`, no-`as tid` form, the level rejection, the three nesting rejections, the three host-function-type rejections, the *accepted* Hierarchy nesting, shared shape validation, the scope-producer contract (accepts with `deps=`, rejects without), and the hint wording. - **Transforms** (`test_submit_passes.py`) — SSA versions the operand inside the InCore scope attr; `test_outlining_rebinds_a_store_produced_predicate_operand` pins the fix above (it reproduces the `__FREE_VAR` output without it) and asserts the operand is the `rc` result the gate `Submit` exports, not the pre-store input; DCE keeps a Var used only in the predicate index. - **Codegen** (`test_predicate_codegen.py`, +9 tests) — the `L0TaskPredicate` block with `&ext_rc`, operator/operand-order mapping, `set_predicate` before `rt_submit_*`, exactly one predicated task. Plus a **`windowize=True`** case: `windowize` is `pl.at`-only, making this the only DSL path that reaches the `kAttrPredicate` entry in `WindowExternalization`'s view-synthesised key filter, which until now was written defensively and unreachable from the frontend. Verified the rewrite genuinely fires (the emitted args switch to `ext_x.view(...)`). - **On-board a2a3 ST** — `tests/st/runtime/scheduling/test_predicated_dispatch.py` gains a `pl.at` variant with the identical task chain and expected values. 6/6 passed when last run on board, before the rebase; the CI device jobs cover the current head (gate=0 skips, gate=1 dispatches; the consumer unlocks in both cases, so inline-retire still settles fanout). - **Regression** — full UT suite **11036 passed, 8 skipped, 1 xfailed**; `tests/lint/clang_tidy.py --strict-version` clean over all four changed C++ files; `pre-commit` clean. ## Docs `docs/en/dev/language/02-manual_dependencies.md` + `docs/zh/...` — the stale "It is not accepted on `pl.at(...)`" note is replaced with the surface and a `pl.at` subsection covering the placement rules, including the allowed Hierarchy nesting. (Both files moved on `main` since this PR opened: #2193 split `00-python_syntax.md`, and `docs/zh-cn/` was renamed to `docs/zh/`.) The `pl.at ... as tid` table row is updated to match how `pl.spmd`'s row documents the kwarg. Refs #2066
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Site scaffolding for the user manual plan (#2120). The existing 162 markdown files go online as-is, so every later documentation batch is reviewable as rendered pages and
--strictguards cross-references from day one.The stack and the publishing path match the runtime's docs job (
hw-native-sys/simpler): MkDocs Material +mkdocstrings[python], deps indocs/requirements.txt, published throughactions/deploy-pages.mkdocs.yml— Material theme with four tabs (User Manual / Reference / Developer / an external Runtime link to https://hw-native-sys.github.io/simpler/),mkdocs-static-i18nin folder mode overdocs/en+docs/zh, andmkdocstringsconfigured up front so the generatedpl.*reference has a fixed docstring contract to land against..github/workflows/docs.yml— pushes tomainbuild and publish; pull requests build only. The job installsdocs/requirements.txtrather than the project, which would run scikit-build-core and compile the C++ extension the site does not need (mkdocstrings readspython/pyptostatically through griffe).scripts/mkdocs_hooks.py— 93 links in the docs point at source files outsidedocs/(python/,include/,examples/,.claude/rules/). MkDocs cannot resolve those and--strictfails on each one, so the hook rewrites them to GitHub URLs at the ref being built (DOCS_REF:mainon a push, the commit SHA on a PR). Covers both inline links and reference-style definitions. The markdown keeps repo-relative links and stays clickable when browsing the repository.tests/lint/check_docs_nav.py— fails when a page is missing from the nav (it would be reachable only by direct URL) or a nav entry points at a deleted file. Wired into pre-commit beside the existing en/zh parity check.index.mdlanding pages, one per directory in both locales, each listing its pages with a one-line description of the target.docs/images/→docs/assets/, the shared-asset location the i18n folder mode expects.The site is a build artifact and is never committed; the markdown under
docs/remains the single source of truth and stays readable on GitHub.docs/zh-cn→docs/zh(the one content-tree change)mkdocs build --strictrejected the old directory name, and neither obvious alternative works either. mkdocs-static-i18n validates every locale againstso
zh-cnfails on the lower-case territory.zh-CNandzh_CNpass that check but fail later: the plugin assigns the locale straight totheme.language, and Material'spartials/language.htmldoes a hard{% import "partials/languages/" ~ language ~ ".html" %}. Material shipszh.html,zh-Hant.htmlandzh-TW.htmlbut nozh-CN.html, and theen.htmlfallback beside that import only covers missing keys, not a missing file — so anything butzhraisesTemplateNotFoundat render time.In folder mode the locale is the directory name, so the directory moved with it. No content changed;
docs/enremains authoritative and the trees stay mirrored page for page. Updated alongside the 92 moved files: the en/zh parity and english-only lint scripts, the pre-commit exclude,.claude/rules/documentation.mdandpass-doc-ordering.md, the add-op and code-review skills,AGENTS.md, and both READMEs.No repository setup is outstanding
Pages is already configured on this repository, and that configuration is what dictates the publishing path:
workflow— i.e. Source = "GitHub Actions"github-pagesenvironmentmainnull— nothing published yetThat build type is why publishing goes through
actions/deploy-pages: under it agh-pagesbranch is ignored entirely, so themkdocs gh-deployapproach this PR originally used would have pushed a branch that never got served. Merging is all that is required for the first deploy.Testing
mkdocs build --strictis green in this PR's own Docs run — the workflow triggers onpull_request, so the build is verified here rather than only after merge. It built both locales in 10.2s and translated 12 nav elements.Locally, alongside it:
tests/lint/check_docs_nav.pydocs/entests/lint/check_docs_en_zh_parity.pymarkdownlint-cli2check_headers.py/check_english_only.pyruff check/ruff format/pyrightFollow-ups
dev/pages that predate this PR. They do not fail--strictbecausevalidation.anchorsdefaults toinfo. Worth fixing and then promoting towarnin a separate change.mike(version selector) is deliberately absent — it should track PyPTO's release cadence rather than an arbitrary latest/dev split, so it lands with that process ([Docs] Docstring-driven API reference for the frontend DSL (pl.*) — full audit + auto-generated, always-in-sync reference #2120 §8.2).Related Issues
Part of #2120