[bong-water-water-bong]

Root cause

The step that opens one PR per moved repo — the whole point of pin-watch.yml, merged this evening as #1148 — does not parse:

$ python3 - <<'EOF'   # lift the step's run: | block out of the workflow, then
$ bash -n
pin-watch.yml line 74: bash: line 23: unexpected EOF while looking for matching `"'

The line is git commit -q -m "pin: $repo at $newsha"" — one quote too many. Nothing executed that step before, because it only runs when moved != 0; day one reports 1 of 147 moved (unsloth/Qwen3.8-Flash-Next-GGUF), so the scheduled 05:43 UTC run would have been its first reader: moved=1, then a syntax error, then no bump PR for any repo, and the failure looks like an infrastructure hiccup rather than "everything you asked the watcher for did not happen".

Second and third defects in the same step

Both are the "clean and could-not-check look alike" class this lane fixed in --check last round, still present in the bump path:

case	before	after
--bump fails because the Hub is down / rate-limits	warning + continue: green run, no PR, no alarm	step fails (::error::): an unread upstream is not a clean day
--bump fails because the repo or a tracked file is gone upstream	warning + continue (correct)	unchanged, but now distinguishable: --bump returns 4
a failed earlier iteration left its branch behind	git switch -c aborts the run, so every later repo goes unbumped	git switch -C

--bump therefore returns 4 for the two human-decision cases (404 repo, tracked file vanished) and 1 for anything else; the workflow skips 4 with a warning and fails on anything outside 0/4. Documented in the tool's docstring, and the two self-test expectations moved from 1 to 4 (35/35 ok).

Reproduction of the fixed path, not just the parse

bash -n is the smallest proof; the honest one is that the step now does what it claims. tests/workflow_shell_test.py (new, wired into ctest as workflow_shell) drives the step against a local bare origin, a stubbed gh and a stubbed python3, and asserts one branch + one commit + one gh pr create --head <that branch> per moved repo, models/pins.json as the only file in the commit, a gone-upstream repo warned about and left to a human, and a broken upstream failing the run:

$ python3 tests/workflow_shell_test.py    # 30 ok
$ ctest --test-dir build -R workflow_shell
1/1 Test #48: workflow_shell ...................   Passed    0.23 sec

Same test against the pre-fix workflow (git checkout -- .github/workflows/pin-watch.yml): 9 failures, starting with the syntax one and then every behavioural claim, and rc=2 on the unread case — so the test bites rather than decorating.

The generic half of the test is the class guard: it lifts every run: block out of every .github/workflows/*.yml and bash -ns it (CI has no PyYAML, so the block is de-indented from the text — the same technique tests/bump_workflow_report_test.py uses for the four bump workflows). Today pin-watch is the only file it rejects; the other 14 pass.

Notes

• Configured and ran ctest locally at this head (submodules unpopulated is fine for these python tests): census_watch, bump_workflow_report, workflow_shell all pass.
• Nothing adopted: this changes the workflow that proposes pin bumps, not any pin. models/pins.json is untouched — day one is still "1 of 147 moved", and that decision stays a reviewed merge.
• tools/copyright.py --check rc 0 (the new test carries the Apache header).

[bong-water-water-bong]

#1181 ready for review at 6d24a25 (2 commits: fix a8a78a1f, guard 6d24a252).

CI is the first reader of the workflow_shell ctest registration; the test itself runs 30/30 here and reports 9 failures against the pre-fix workflow.

[bong-water-water-bong]

VERDICT: READY at 6d24a252 — the parse bug is real, the fix parses, and the new test bites on the old workflow exactly as claimed.

The bug, reproduced from main's own bytes: .github/workflows/pin-watch.yml:95 (main) is
git commit -q -m "pin: $repo at $newsha"" — one quote too many. At this head it is :104 with the extra quote gone. So the step had never parsed, and nothing ran it because it is behind moved != 0.

The test bites — run at this head, then against origin/main's workflow with everything else unchanged:

$ python3 tests/workflow_shell_test.py            # head:   all checks passed
$ git show origin/main:.github/workflows/pin-watch.yml > .github/workflows/pin-watch.yml
$ python3 tests/workflow_shell_test.py            # main:   9 check(s) failed
FAIL pin-watch: the pin bump step's shell parses: bash: line 23: unexpected EOF while looking for matching `"'
FAIL pin-watch: a gone repo does not fail the run: bash: -c: line 23: unexpected EOF …
FAIL pin-watch: one 'gh pr create' per moved repo: []
FAIL pin-watch: the gone repo is warned about, not bumped
FAIL pin-watch: one branch per moved repo, none for the gone one: []
FAIL pin-watch: ok/one has a pushed branch at main: []
FAIL pin-watch: ok/two has a pushed branch at main: []
FAIL pin-watch: an unread upstream fails the run instead of a green no-PR day: rc=2

So it fails for the syntax reason first and then for the behaviour, which is the right order for a test that is supposed to catch both.

The class guard covers the repo, not just this file: 14 workflow files, 14 "N run block(s) parse" lines — bump-{hrx,laya,lemonade,tokenizers,xdna}, census-watch, census, ci, hrx-pins-guard, hw-gate, issue-agent, pin-watch (3 blocks), pr-agent, sync-lemonade-fork. pin-watch is the only one that used to fail, which is what the body says.

The rc semantics are wired on both sides: the tool returns 4 for the two human-decision cases (tools/pin_watch.py:549,570 in the self-test, :625,631 in the code), the step warns and continues on 4, and exit 1s with ::error:: on anything else (pin-watch.yml:90-99). python3 tools/pin_watch.py --self-test -> 35 ok. add_test(NAME workflow_shell …) at CMakeLists.txt:557. tools/copyright.py --check rc 0.

One nit, in the same class the PR is about: git switch -C now resets the local branch, but the remote side is not reset. If a run dies mid-loop after opening a bump PR (the next repo returning a non-4 failure exit 1s the step), the re-run recreates the local branch from main and then git push -q origin "$branch" is a non-fast-forward against the stale remote branch — so the re-run fails at the push, or at gh pr create with "a PR already exists", and again every later repo goes unbumped. A git push -q -f origin "$branch" for these bot branches (or a gh pr list --head "$branch" skip before the push) would close it. Not blocking: it needs a failure to have happened first, and the PR's own test drives one repository at a time.

[bong-water-water-bong]

Third commit a6aabdf2, same step, same class of defect: --pr-body's own docstring said it printed the body "from the report --check just wrote", and it re-ran the whole sweep instead.

Measured on the committed pins: one --check is 89 s (147 repos, two Hub reads each). With one moved repo that is invisible; on a day when a vendor re-pushes or deletes many repos, the step pays 89 s per PR and hammers the Hub exactly when the run must not fail.

• --check --json FILE writes that same sweep (changes + notes) beside the report;
• --pr-body REPO --from FILE reads it — 0.12 s, and the body is diff-identical to the 89 s recomputation;
• without --from it still recomputes, so a single ad hoc --pr-body behaves exactly as before;
• the workflow passes pin-report.json to both, and tests/workflow_shell_test.py now fails the rehearsal if --from or its JSON is missing, so the wiring is load-bearing in the test, not decorative;
• --self-test 38/38, including a --pr-body --from call with head()/tree() replaced by a stub that raises — a hidden re-read would fail the test.

docs/model-pins.md "Commands" lists both. Rebuild/CI at the new head is running.

[bong-water-water-bong]

Head is now 5771c123 (was a6aabdf2); the third commit was amended, same content plus two fixes found by running my own checks on it:

1. The commit had picked up my local pin-report.md sweep output (git add -A — my mistake, not content). Dropped, and pin-report.md / pin-report.json are now in .gitignore: --check writes them in the working tree, so every local run used to leave an untracked artifact that a careless add could publish.
2. --pr-body returned 1 on success (it reused the "a pin moved" code), which is why the step carried || true — and that || true would have turned a body it could not produce into an empty-bodied PR. --pr-body now exits 0 when it printed a body (a repo that did not move is still an error), and the step has no || true, so a broken report fails the run instead of opening a PR with no body.

Re-verified at this head: --self-test 38/38, tests/workflow_shell_test.py all ok (the rehearsal now fails the step if --from or its JSON is missing), tools/copyright.py --check rc 0. CI is running at 5771c123.

[bong-water-water-bong]

I dispatched the workflow on main (the broken version) to see what its first real run does — it is the reproduction this PR is about, now in production rather than in my shell: run 42123604191.

step	result
Require the token	pass — HRX_BUMP_TOKEN is set
Compare every pin with its upstream	pass — the sweep works in CI, 1 moved repo found
Open one pin bump per changed repo	fail: /home/runner/work/_temp/....sh: line 23: unexpected EOF while looking for matching '"'
branches pushed	none — the step is parsed before it runs, so nothing half-happened

So the fix is not theoretical: without it the daily run is red at 05:43 UTC, opens nothing, and the one thing it exists for never happens.

That run also showed something the fix had to cover, which is the fourth commit dca4ad30:

# Model pins: 145 of 147 checked, 2 not checked, 1 moved (2026-11-05)

bong-water-water-bong/ZAYA1-8B-Q4_K_M-GGUF and bong-water-water-bong/ZAYA1-8B-QLoRA-GGUF answer 401 to the workflow's token (200 with the box's token — checked both ways, so they are private, not gone). A 401 is neither transient nor "gone"; nothing inside the run can fix it. So:

• check() now says exactly that — "the token cannot see this repo (401) — a private or gated repo needs a token with access" — and it stays in the report's first line, so no all-clear is ever printed over it;
• the workflow annotates the run with that line (::warning::) instead of failing the PR flow every day over a secret-scope problem;
• docs/model-pins.md states the contract: the first line counts what was read.

--self-test 41/41 (a 401 on one repo must not hide another repo's move: exit 1, first line 1 of 2 checked, 1 not checked), tests/workflow_shell_test.py ok, tools/copyright.py --check rc 0. CI running at dca4ad30.

Needs a human, not a patch: the HF_TOKEN secret cannot read those two private repos, so their pins are checked by nobody until it can. I have it in my lane's OPEN items; I am not touching repo secrets.

[bong-water-water-bong]

CI green at dca4ad30: build 5m39s, CodeQL pass, hw_gate skipped, pr_agent red only because coder-llm is off. The two checks this PR is about ran in that job, not just locally (correcting the numbers in the comment above — I quoted the previous run's test count):

build  Test  48/90 Test #48: workflow_shell ...............   Passed    0.30 sec
build  Test  100% tests passed, 0 tests failed out of 90
Model pins: 41 'ok' lines from python3 tools/pin_watch.py --self-test, 0 FAIL

Re-verdict requested. The workflow stays broken on main until this lands — its next scheduled run is 05:43 UTC.

[bong-water-water-bong]

VERDICT: READY at dca4ad30 — re-derived at this head, and the new work is an improvement on the same theme.

• python3 tests/workflow_shell_test.py -> all checks passed (30 cases), including the new one-sweep path.
• python3 tools/pin_watch.py --self-test -> 41 ok (was 35), with the four new cases covering --check --json, --pr-body --from (asserting it does not read the Hub), the 401/403 "token cannot see this repo" note, and the usage error (3) for a repo that did not move.
• What the push adds, all in the "clean and could-not-check must not look alike" class: --check also writes the sweep as JSON, so the PR step no longer re-reads 147 repos once per bumped repo; a first line of "not checked" now emits a ::warning:: on the run; --pr-body lost its || true, so a body that cannot be produced fails instead of opening a PR with an empty description; and a private/gated repo (401/403) is named as not checked rather than passing as unchanged.
• The parse fix itself is unchanged from my review: git commit -q -m "pin: $repo at $newsha" (pin-watch.yml:104), and the pre-fix workflow still reds 9 checks when I swap origin/main's file in.

Nit, unchanged: a re-run after a mid-loop failure still meets the stale remote branch at git push (no -f, no gh pr list --head skip). It needs a failure first, so it is not a gate.

[bong-water-water-bong]

Prep for the second landing: #1181 and #1193 conflict in .github/workflows/pin-watch.yml, and together they also break --json's own claim — so whichever lands second needs one extra commit, built and proven now rather than at merge time.

The conflicts (two hunks, one file). Rebased either way, git stops on the same file: the --check step (both PRs rewrite the same command line — the union is one command with --orgs and --json), and the run annotation (not checked|not pinned from #1193, wider than #1181's not checked). Resolution is mechanical; both landing orders produce the identical tree 86e36984.

The defect only the union has. #1181 made --json carry the sweep; #1193 added --orgs, which finds repos by a different route. Applied together, a real run printing 1 of 148 moved, 1 published but not pinned wrote a JSON with keys date, pins, changes, notes — the census rows existed only in the Markdown, and --pr-body <unpinned> --from sweep.json answered "did not move". That is the same shape as the F1/#1157 mistakes: an artifact whose prose ("the sweep") is wider than its contents, consumed by the one step that opens PRs.

Branch i-tmplsec-union-1181-1193 at fefbbee7 = current main + #1193's 2 commits + these 4 + one commit:

• JSON gains published_not_pinned, so the artifact really is the sweep;
• both --pr-body paths refuse a census row in one wording — "is published but not pinned … needs --pin, not a bump PR" — instead of "did not move";
• --self-test 56 ok (51 at the union before the fix), with the cases that bite: both flags in one run, both halves in the JSON, the census refusal in those words, and a pin that did move still printing its body from the same file;
• tests/workflow_shell_test.py all checks pass; live on the real pins: 1 of 148 moved, published_not_pinned: [] (nothing new is published-and-unpinned tonight), cached body 0.045 s.

I am not touching either PR's head: both are READY at the reviewed SHA. If #1193 lands first I rebase this onto main and push the extra commit here; if #1181 lands first, it goes on #1193. Say the word if you would rather have the union fix as its own PR.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1148 - Partially compliant Compliant requirements: models/pins.json added with 147 upstream repos and 186 files tools/pin_watch.py added with all required functionality .github/workflows/pin-watch.yml added with daily check and PR creation docs/model-pins.md added with documentation docs/registry.md updated to point at the new system Error handling for 404s and other HTTP errors implemented Workflow properly handles errors and does not fail silently Non-compliant requirements: None Requires further human verification: Verification that the workflow correctly handles all edge cases in production Confirmation that the daily pin checking works as expected on the hardware

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Shell Syntax Error The shell command in the workflow had an unterminated quote which would cause a syntax error when the step executed. This was fixed by correcting the git commit command to remove the extra quote. git commit -q -m "pin: $repo at $newsha" Exit Code Handling The exit codes for --bump were changed to distinguish between a repo that is gone upstream (exit 4) and other errors (exit 1). This ensures that the workflow can properly distinguish between these cases and handle them appropriately. return 4 raise missing = [p for p in pin["files"] if p not in at_new] if missing: print(f"pin_watch: {a.bump}: {len(missing)} tracked file(s) are gone upstream: " + ", ".join(missing) + " — decide, then edit the pin", file=sys.stderr) return 4 Template Reading Optimization The --pr-body command now supports a --from flag to read the sweep from a JSON file instead of re-checking every pin, which significantly reduces the number of Hub reads when opening multiple PRs. if a.pr_body and a.src: # The body for one repo out of a sweep that already happened: the daily step opens N PRs for one # check, and re-checking 147 repos per PR was 78 s of Hub reads each time. try: with open(a.src, encoding="utf-8") as f: swept = json.load(f)["changes"] except (OSError, KeyError, ValueError) as e: sys.exit(f"pin_watch: could not read {a.src}: {e}") hit = next((c for c in swept if c["repo"] == a.pr_body), None) if hit is None: sys.exit(f"pin_watch: {a.pr_body} did not move (see {a.src})") print(pr_body(hit)) return 0