[scuttletoby]

Fork tests can run its Linux jobs on this repository's own machines (the fleet). Each run picks per job, and the repository variable CI_RUNNERS controls it: auto (also unset), fleet or github. Until a fleet host runs the dispatcher, auto behaves as today: everything is GitHub-hosted, with a note that says why.

Runbook, host setup, owner steps, a real-runner check and the full failure table: .github/fork-fleet.md.

Design: no runner waits for work

The first version had a standing self-hosted runner and a job-started hook that refused untrusted jobs. Security lens s1 showed that this hook is a pre-job step: when it fails, later if: always() / failure() steps still run. A stranger's pull request that edits the workflow to name the fleet label could then run code on the host, and rewrite the runner's .env to remove the hook. The fix removes the cause: an untrusted job never finds a runner at all.

1. Route is now the last step of changes (no extra job). It calls no API and reads no secret. In auto, a pool counts only when its dispatcher's heartbeat (a repository variable, at most 180 s old) reports a free slot. For each fleet job it picks two labels: the pool's label, and a label made for this run (<pool label>-<job key>-<16 random hex>). Untrusted runs go GitHub-hosted.

2. The dispatcher (fork-fleet.py dispatch) is the root of the design. It runs as root on the fleet host, from a root-owned copy of main's script and config. It starts a one-job (JIT) runner only when all of these hold:

   • the queued job asks for exactly that label pair;
   • GitHub's own record of the run passes the trust rule, with each pull request read from pulls/<n>;
   • the job's attempt is the run's attempt it read;
   • no other queued job, of any run, could take that runner.

   The launcher (fork-fleet-launch.sh) runs that runner from a fresh hard-link copy of a root-owned, unconfigured runner template. It runs as a user of its own, in a transient systemd unit (whole process group killed on stop, 1 h at most). Its environment comes from a root-only file that is deleted once the runner has started: the JIT config, the run and attempt it was made for, and the hook path. A job that names a fleet label by itself, from a changed workflow or a fork's pull request, stays queued with no runner.

3. The hook (fork-fleet-hook.sh → gate, root-owned) is the second line. If the job is not the run and attempt its runner was made for, or if it breaks the trust rule, the hook kills the job's Runner.Worker (SIGKILL). The job then fails with no later step run, always() ones included. A gate that crashes refuses the job.

4. Bounded waits. fleet_watch (a job in each run that sent a job to the fleet) cancels the run with an error naming the job when a fleet job waits 10 minutes (fleet mode: 2 hours). The stateless sweep (every 10 min) re-runs such a run once, GitHub-hosted, in auto/github. In fleet mode it re-runs nothing, and it cancels with an error any trusted run whose fleet job waited 2 hours. A re-run of failed jobs has no watch of its own, so this bounds it. The sweep never touches an untrusted run.

Why this design: the hook alone cannot be made safe, because a failed pre-job step does not stop the job. Killing the worker does stop it, but it still depends on the hook's environment. With a standing runner, a job could rewrite that environment for the next job. With one-job runners made only for a trusted run's own labels, untrusted code needs both a runner it can never get and a gate it cannot change. The PC pool needs no standing connection either: its runners start on demand.

Runner source this relies on (actions/runner main, read 2026-11-12)

• src/Runner.Worker/JobExtension.cs ~303-315: the hook path is read from the worker's process environment, and the hook is a pre-job step with condition always(). Container setup is added after it, so the hook runs first.
• src/Runner.Worker/JobHookProvider.cs:74: the hook's environment starts empty and gets the process environment plus GITHUB_*/RUNNER_*, not the workflow's env:. So FORK_FLEET_RUN/FORK_FLEET_ATTEMPT from the root-only file can be trusted.
• src/Runner.Listener/JobDispatcher.cs:1179-1219: when the worker dies, the listener marks the job failed, and no later step runs.
• src/Runner.Listener/CommandSettings.cs:112-138: ACTIONS_RUNNER_INPUT_JITCONFIG is read from the environment and then deleted, so the JIT config is never in argv.
• src/Runner.Listener/Program.cs:172-194: the runner loads .env from its own directory and lets it override its environment. That is why the launcher refuses a template with .env, .path, .runner or .credentials, and uses each copy for one job only.

Also fixed (lens minors)

• s2-1, re-runs: runs-on also checks vars.CI_RUNNERS. A re-run follows the setting as it is now (github → hosted, auto → hosted for a re-run of failed jobs, fleet → the run's labels while it is still fleet). The watch reads its own attempt, and the sweep reads the latest attempt of every run.
• s2-2: route is a step of changes; the route job is gone.
• s2-3: a truncated or garbled reply (http.client.IncompleteRead, ValueError and the like) is a warning and never turns a run red, in watch, sweep and dispatch (route calls no API).
• s2-4 / s1: a runner-writable .env can no longer turn off the hook (see the design).
• s1, sweep trust: the sweep uses the dispatcher's rule on GitHub's record and leaves a stranger's run alone.
• s1, no silent 24 h: fleet_watch and the sweep (above).
• s1, a person's cancel stands: now tested, and removing it turns a test red.
• Doc: in fleet mode, shards that only print a skip run GitHub-hosted.

Found while testing, and fixed:

• the launcher's JIT config check was a bash regex error that refused every config;
• WORK_ROOT must be 0711 so the runner user can reach its copy;
• the launcher's EXIT trap named a local, so every start exited 1 under set -u and left the environment file behind;
• the dispatcher could make a runner for attempt 1 while the owner's re-run had moved the job to attempt 2, and the gate would then kill a trusted job. It now skips such a job for one pass;
• the dispatcher's lock directory under /run is created after a boot.

Owner steps (none done here: no settings, variables, secrets, runners or repositories were changed)

# Setting (optional; unset = auto):
gh variable set CI_RUNNERS -R scuttletoby/t3code --body auto
# The dispatcher's token: fine-grained PAT, resource owner scuttletoby, only scuttletoby/t3code, repository
# permissions Administration: Read and write, Actions: Read-only, Variables: Read and write,
# Pull requests: Read-only. It is stored on the fleet host only (root, 0600), not as a repository secret.
# The first design's route token is no longer used:
gh secret delete FLEET_RUNNERS_TOKEN -R scuttletoby/t3code   # if it was set
# Approval for outside contributors must stay all_external_contributors:
gh api repos/scuttletoby/t3code/actions/permissions/fork-pr-contributor-approval --jq .approval_policy

Host setup on VM106 is in .github/fork-fleet.md as paste-ready root commands:

• a dedicated runner user: no sudo, no docker group, no linger, no cron or at;
• a root-owned runner template, checked by sha256;
• the four files from a pinned main commit;
• /etc/fork-fleet/launch.conf and the token;
• the dispatcher unit with --pool vm.

The pc pool stays off until a launch-pc ssh wrapper exists. Whether the PC joins is an owner decision.

Real-runner check before the fleet takes jobs: the steps are in the runbook. They need a throwaway private repository under the owner's account (the owner's call; none was created) and check:

• a refused job's always() step never runs;
• the environment file is gone after the start;
• the runner user cannot write the template;
• a bare fleet label finds no runner;
• the owner's re-run gets a runner for attempt 2.

Failure table: what happens when this runs again, stops half-way, or meets existing/old/empty state?

The full table (about 50 rows) is in the runbook. All state lives in GitHub (runs, jobs, runner registrations, heartbeat variables) and in systemd on the host. The dispatcher keeps only pacing in memory.

Part	Situation	What happens
Route	CI_RUNNERS misspelled	changes fails with a clear message: red. Unset or empty means auto.
Route	No, old or garbled heartbeat	That pool counts no slot. auto: GitHub-hosted with a note. fleet: queue on vm.
Route	Untrusted run, release run, run from before this PR	GitHub-hosted.
Route	Re-run of failed jobs / of all jobs	Follows the setting as it is now (see s2-1). For all jobs, route runs again: fleet only if the owner re-ran it.
Dispatcher	Crash after generate-jitconfig, before start	An offline registration with no unit: the next pass deletes it, and the pass after starts the job again (at most 3 tries per job).
Dispatcher	Crash after start	The next pass sees the unit and starts nothing twice.
Dispatcher	Launcher killed half-way through the copy	No unit: the registration is deleted, prune removes the half copy and any environment file, and the job is started again.
Dispatcher	Restart or reboot	It reads everything again. The lock directory is made again. A running job fails as a lost runner (auto: the sweep re-runs it hosted), and a queued job gets a new runner.
Dispatcher	Two dispatchers	The second fails on the lock.
Dispatcher	The job was cancelled before its runner took it	The runner is stopped after 3 passes, or at once if another queued job could take it.
Dispatcher	Re-run between reading the run and its jobs	Skipped for one pass, then a runner is made for the new attempt.
Dispatcher	A stranger's queued job could take the runner	No launch. The trusted job falls back (auto: hosted after 10 min; fleet: an error at 2 h).
Dispatcher	Down	Heartbeats go stale. auto: hosted. fleet: the watch reports at 2 h.
Launcher	Template configured, a symlink or writable by others; conf or hook writable by others; bad input; a name that ran already	It refuses before it starts anything.
Launcher	systemd-run fails	The environment file is removed on exit, and the copy goes at the next prune.
Hook	Wrong run or attempt, or an untrusted run	Runner.Worker is killed: no step runs, always() included.
Hook	Broken installed config, or a runner the dispatcher did not start	Refused (fails closed).
Hook	No Runner.Worker above the hook (a renamed worker)	It can only exit 1, and says so. The real-runner check covers this.
Watch	Truncated or garbled reply	A warning, then it tries again. Past the bound it stops watching, never red. The sweep is the fallback.
Sweep	Stops after the cancel, before the re-run	The next sweep re-runs it (the job waited 10 min and never started).
Sweep	Runs again on its own re-run	That run is untrusted (github-actions), so it is left alone: one re-run at most.
Sweep	A person cancels; branch moved; a real test failure	Stands; not re-run; stays red.
Old state	Old copies or environment files	prune removes them on every pass.
Old state	First design's token or a standing runner	Unused; delete them. The fork has no runners today.
Old state	Installed copy older than main	The old rule applies until re-installed. /opt/fork-fleet/SOURCE names the commit.
Empty state	No heartbeat variables yet	The dispatcher creates them on its first pass.

Tests

.github/scripts/fork-fleet.test.py has 104 tests, run by the workflows job:

• config;
• the trust rule from the environment and from GitHub's record;
• the gate;
• route in every mode;
• watch;
• sweep;
• the dispatcher against a fake GitHub and a fake host;
• the launcher, run for real against fake systemd-run/systemctl;
• the wiring of every workflow against the config.

GateProcess models the s1 blocker. A process named Runner.Worker runs the installed hook, then an always() step that writes a marker. A refused job leaves no marker, in these cases:

• a fork's pull request;
• an old runner seeing a re-run;
• a runner the dispatcher did not start;
• a crashing gate.

A mutant gate that only exits 1 does leave the marker.

Mutation check: 15 mutations, each red, then green once restored (details in the fix-round comment). They cover:

• a person's cancel;
• the SIGKILL;
• the run and attempt check, in the gate, the dispatcher and runs-on;
• the sweep's and the dispatcher's trust rule;
• the reply errors;
• the launcher's input and template checks, and its three bugs;
• both fleet-mode sweep cases.

Locally:

• actionlint is clean on fork-*.yml;
• node .github/scripts/fork-ci-tests.cjs passes, and so do fork-contracts.test.py and fork-release-workflow.test.py;
• fork-tests-drift.cjs matches the record;
• bash -n passes on both new scripts;
• gitleaks finds nothing in the diff;
• shellcheck is not installed here, so it did not run.

General vs project

• General: fork-fleet.py (route, watch, sweep, dispatcher, gate), fork-fleet-launch.sh and the hook. They fit any repository and any pools.
• Project: everything in .github/fork-fleet.json: the repository, actors, branches, pools (labels, slots, heartbeat variables, launch commands), jobs (shards, gates, memory) and the time limits.

Follow-up

Plan row 9 is not in this PR: the config path as an input, a hosted: false option, at most one heavy job per run on the PC, a runner pool by draft state, and the SkyRPG fleet requirements as tests. This PR is already about 2,300 changed lines, so row 9 will be a stacked follow-up PR on this branch.

Open for the owner

• Merge, then the host setup and the real-runner check, in that order. Nothing runs on the fleet before the dispatcher does.
• Upstream-sync runs and pull requests by anyone but scuttletoby (bots included) always run GitHub-hosted. Allowing them would be a change to trusted.
• fleet mode runs up to 16 Linux jobs two at a time on vm. A slow full run can come near the 2-hour bound. auto is meant as the everyday mode.
• The sweep adds 144 short runs a day to the Actions list. They cost no minutes, because the repository is public.

🤖 Generated with Claude Code

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2) · read at: f2d6e15
covered: the whole diff against main 8b0ad2c (fork-fleet.py route/gate/capacity/sweep, fork-fleet.json, fork-fleet.md, fork-fleet.yml, every fork-tests.yml change, fork-ci-tests.txt), plus what it calls into: fork-release.yml's workflow_call into fork-tests.yml (it passes no secrets), the changes job outputs, fork-ci-result's needs, and GitHub's docs on job hooks and default variables.
checks:

• Outside-fork PR hard rule: holds. route sends any untrusted run GitHub-hosted. The only real line is the job-started hook, because a fork PR controls both its workflow file and the fork-fleet.py that route runs. The hook gets the default variables, which a workflow's env: cannot override (GitHub docs: "the assignment is ignored"), plus GitHub's own event file. A stranger's fork PR fails on GITHUB_ACTOR. The owner re-running it keeps actor = stranger and head.repo = the fork, so it is refused. A maintainer push to a fork branch has actor = owner but head.repo = the fork: refused. A workflow_call from a release or another workflow gets the caller's GITHUB_WORKFLOW_REF: refused. pull_request_target cannot be added by a PR (it reads the base file). A bare "self-hosted" label lands on vm106 and is refused. A non-zero hook exit fails the job before any step (docs), and the gate fails closed on a crash or a missing config. FLEET_RUNNERS_TOKEN reaches only the hosted route job. A fork PR gets no secrets, and a release workflow_call passes none. Fleet jobs read no secrets (tested).
• Sweep acting on others' runs: it can only cancel a stranger's own run that hard-codes a fleet label. Re-running one is blocked by current(): head_branch is the fork's branch name, so the base branch sha never matches. A re-run keeps the original actor's privileges anyway. No way to act on the owner's runs from outside.
• auto fallback: no token, an API error, a re-run, a busy or offline runner, a memory shortage and a lost runner all end hosted or get re-run once. The "lost runner = no failed step" signal relies on GitHub's real step conclusions, which can only be checked on a live runner.
• fleet never silently hosted: correct for trusted runs. The one exception is test_server shards whose gate is false: they run hosted, but they are no-op shards, so this is harmless and arguably right.
• General and upstream: fork-fleet.py holds no repository or project facts (all in fork-fleet.json). Only fork-* files change; ci.yml and release.yml are untouched.
• State question: unset variable = auto (tested). A typo = red route (tested; a typo also reds the release workflow_call's tests, which is consistent). A pre-PR run re-run uses its old workflow, so hosted. A sweep that stops half-way is finished by the next one (stateless, tested). Two sweeps queue in one concurrency group, and attempt 2 is ignored. A runner missing from fork-fleet.json fails capacity, which the sweep rescues. A stale installed gate still checks the trust rule only, so it stays safe. A gate with no config fails closed (tested with -I). The setting changing between attempts: see finding 1.
• Tests: scratch clone at the head, 51/51 OK. I broke 33 guards by hand (each actor, head/base repo, opener, push branch/repo, workflow ref, repository, other events, no event, gate crash-open, route trust skip, auto re-run, busy/offline idle, fleet leftovers, typo=auto, capacity unknown/timeout, the sweep's attempt, branch-moved, real-failure, fleet-mode, stuck-threshold and hosted-job guards, the runs-on attempt check, macOS to fleet, a secret in a routed job, sudo without the self-hosted check, the capacity step removed, a wrong shard key). 31 went red. Two stayed green: the gate using {} for an unreadable event (equivalent: {} is refused anyway), and the "a person's cancel stands" rule at fork-fleet.py:350 (no test has a fleet job failing and then a person cancelling; a test gap only). Clone deleted.
  finding: minor .github/workflows/fork-tests.yml:280 (and the 6 other routed runs-on lines) + .github/scripts/fork-fleet.py:384. Attempt 1 ran with CI_RUNNERS=fleet, then vm106 goes down; the owner sets CI_RUNNERS=github (or auto) and clicks "Re-run failed jobs". route is not re-run, and its attempt-1 output mode=='fleet' still wins in runs-on, so the jobs queue on fleet-vm106-t3code again despite the setting being github. The sweep skips run_attempt != 1, so nothing cancels them: they wait up to GitHub's 24 h limit. The failure table's "switching fleet -> github: the sweep cancels and re-runs" holds only for attempt 1. Fix: also require the current setting in the expression, e.g. (... || (needs.route.outputs.mode == 'fleet' && vars.CI_RUNNERS == 'fleet')), and add a wiring test. (high)
  finding: minor .github/workflows/fork-tests.yml:153 The route job is a new GitHub-hosted job hop on every run's critical path, even in github mode (one more runner start, ~15-30 s, before any Linux job). Its logic needs only the changes outputs and a sparse checkout, so it could be a step in the changes job, which saves a queue and start-up on every run. The owner puts CI speed first. (medium)
  finding: minor .github/scripts/fork-fleet.py:195 TROUBLE leaves out http.client.HTTPException (IncompleteRead, BadStatusLine are not OSError). A truncated runner-list response during route crashes it, so the run is red in auto, which goes against "auto never makes a run red". Add http.client.HTTPException to TROUBLE. (medium)
  finding: minor .github/fork-fleet.md (job-started hook section) The hook script is root-owned, but the line that enables it (ACTIONS_RUNNER_HOOK_JOB_STARTED in the runner directory's .env) is writable by the runner user, and vm106 is not ephemeral. So one compromised trusted job (for example a bad upstream npm dependency in the owner's own PR) can drop the hook for every later job, including strangers' fork PRs that name the label. Defence in depth: make .env root-owned (or set the variable in the systemd unit), or make vm106 JIT/ephemeral like the PC. (medium)

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1) · read at: f2d6e15
covered: the whole diff against main (fork-fleet.py route/gate/capacity/sweep, fork-fleet.json, fork-fleet.yml, the fork-tests.yml route job + routed runs-on + capacity step + no-sudo apt branches, fork-fleet.md runbook and hook install, the 51 tests). It also covers what this calls into: actions/runner main (JobExtension.cs, JobHookProvider.cs, ScriptHandler.cs, GitHubContext.cs, StepsRunner.cs, JobRunner.cs), read to see when the job-started hook runs and what env it gets.
checks: ran fork-fleet.test.py in a scratch clone at the head: 51 OK. Then I broke guards by hand, one at a time. Each of these turned the tests red: dropping the actor or triggering_actor check, the head-repo check, the PR-author check, the push-branch check, the workflow check, letting other events through, the gate allowing an unreadable event, the gate crashing open, route skipping trust, route ignoring the attempt in auto, the sweep re-running attempt 2+, the sweep ignoring a moved branch, the sweep re-running real failures, an unknown runner passing capacity, capacity skipped everywhere, lint's runs-on without the attempt guard, a secret+sudo in a routed job, and an extra job hard-coding a fleet label. Survived (no test failed): removing the "cancelled by a person stands" return in rescue_reason. Making capacity never time out hangs the suite instead of failing it.
Hook env: the runner builds the hook's env from an empty dict plus the github context, so workflow/job env: cannot spoof GITHUB_ACTOR, GITHUB_TRIGGERING_ACTOR, GITHUB_EVENT_NAME, GITHUB_REF or GITHUB_EVENT_PATH. Actor vs triggering_actor vs head repo vs PR author is right: a stranger's re-run, a fork PR the owner edited, and a same-repo-head PR a stranger opened are all refused. Route reads only env (no title/branch interpolation). Fork PRs get no FLEET_RUNNERS_TOKEN. No upstream file is touched. The script has no project names (all facts in fork-fleet.json).
State question: variable unset -> auto (tested). Typo -> route red. A failed route -> routed jobs fall to ubuntu-24.04 and fork-ci-result is red. Re-run failed jobs in auto -> hosted (attempt guard tested). Re-run of a pre-PR run -> the old workflow file, hosted. Two sweeps -> concurrency group, and attempt != 1 is skipped. A sweep that stops half-way -> the next one finds the cancelled run (the not-finished-cancelling path is tested). Runner lost mid-job -> no-failed-step rescue. Runner name not in fork-fleet.json -> capacity fails at once: re-run hosted in auto, red in fleet. Branch deleted -> 404 -> no re-run. Fleet mode goes hosted only for untrusted runs and for test_server shards whose gate is false (they only print a skip; the doc says "every job that runs goes to the fleet").
finding: blocker .github/fork-fleet.md:105-124 + fork-fleet.py:157 gate. A refused job-started hook does NOT stop the job. In actions/runner (JobExtension.cs ~303-312) the hook is just a pre-job step with condition always(). StepsRunner then evaluates each later step's own if:, so a step with if: always() or if: failure() still runs after the gate exits 1. Inputs: an outside contributor (any account that does not need fork-PR approval, or an approval the owner gives without spotting a workflow edit) opens a PR whose fork-tests.yml adds a job with runs-on: [self-hosted] or [fleet-vm106-t3code] / [fleet-pc-t3code] and a step with if: always(). Result: the gate prints "may not run on a fleet runner" and the step runs the stranger's code on vm106, or on the owner's PC as the runner user. On the always-on vm106 it can rewrite the runner directory's .env (the hook line lives there, owned by the runner user), stay behind (it can drop RUNNER_TRACKING_ID to escape orphan cleanup), and read later trusted jobs' tokens. This repo already shows the effect: test_server's "Detect transfer budget report" step (fork-tests.yml:807-809, if: always()) would also run after a refusal. So the failure-table row "the gate refuses it: red, and nothing runs on the fleet" is wrong. No test can catch this, because it is runner behaviour. Fix direction: on refusal, the hook must abort the worker itself (for example, kill its Runner.Worker parent so no later step is scheduled), proven with a livelike run where an if: always() step does not run. Also make the fleet runners one-job/ephemeral, keep the hook and .env out of the runner user's reach, and as an owner step set "Require approval for all external contributors" for fork PR workflows. (high: read in runner source; not run on a real runner)
finding: minor .github/scripts/fork-fleet.py:383-417 sweep. The sweep cancels and re-runs runs without applying the trust rule: it selects only by a fleet label on a job. Inputs: a stranger's fork PR hard-codes a fleet label, and its job queues 10 min (pc offline or vm106 busy). The sweep cancels it, which is harmless, then checks current() against the base repo's branch of the same name as the fork branch, so a re-run is unlikely but not ruled out. Then GitHub re-runs a stranger's run with the repo's token, without approval. Filter runs by event, head repository and actor (the refused() rule from run fields) before acting. (medium)
finding: minor .github/workflows/fork-fleet.yml:16-19 auto fallback. In auto, the only bound on a fleet job queued for a runner that went away is this cron sweep. GitHub delays or drops scheduled runs under load, and disables schedules in public repos after 60 days without activity. Inputs: the sweep is disabled or broken, and route picked a runner that then went offline. Result: the job sits queued up to GitHub's 24 h limit, silently (timeout-minutes does not count queue time). Consider an in-run bound, or have the sweep's own failure page the owner. (medium)
finding: minor .github/scripts/fork-fleet.test.py rescue_reason. No test pins "a run a person cancelled while its fleet job ran stands": removing that return at fork-fleet.py:351 leaves all 51 green. Inputs: a fleet job fails with no failed step, then a person cancels the run. Result: the sweep re-runs it against the person's cancel. (high)

[scuttletoby]

not a lens: fix round r1

New head: eaf13202f9062257474076ec0b3ac89378398370. Before it comes ce8b61eb4, which merges fork main (919556a1a, #12: outside PRs run nothing) into this branch. For about a minute, ce8b61eb4 alone was the head, because a commit failed before a push. The PR stays a draft. The PR body and .github/fork-fleet.md have the new design, the owner steps and the failure table.

Per finding

Finding	Result
s1 blocker: a failed job-started hook still lets always()/failure() steps run, and a runner-writable .env can remove the hook	Fixed at the root. No standing runner exists. A root dispatcher makes a one-job JIT runner only for a queued job that asks for exactly a pool label plus the run's own random label, of a run whose GitHub record passes the trust rule, when no other queued job could take that runner. The runner runs from a fresh copy of a root-owned template, as a user of its own; its environment comes from a root-only file, removed after the start. The hook is the second line, and it now SIGKILLs Runner.Worker. Proved by GateProcess.test_the_hook_ends_a_refused_job_before_its_always_steps: a refused job leaves no always() marker, and a mutant gate that only exits 1 leaves one.
s2-1: a re-run keeps the first attempt's routing	Fixed. runs-on also checks vars.CI_RUNNERS. The watch reads its own attempt, and the sweep reads the latest attempt of every run. The dispatcher makes each runner for one run and attempt, and the gate checks both.
s2-2: fold route into an existing job	Done: Route is the last step of changes.
s2-3: a truncated or invalid reply must not turn auto red	Fixed. TROUBLE includes http.client.HTTPException (IncompleteRead), ValueError, KeyError and TypeError. Tested for watch, sweep and dispatch. Route calls no API now.
s2-4 / s1: runner-writable .env	Covered by the root-owned config. The launcher refuses a template with .env, .path, .runner or .credentials, and uses each copy once.
s1: the sweep must use the trust rule	Fixed. run_refused reads GitHub's record of the run plus pulls/<n>. A stranger's run is never cancelled or re-run.
s1: auto must not hang on one cron	Fixed. fleet_watch in each fleet run cancels with an error after 10 min (fleet: 2 h). The sweep is the fallback, and in fleet mode it now cancels, with an error, a run past 2 h, because a re-run of failed jobs has no watch.
s1: test for "a person's cancel stands"	Added: Sweep.test_a_persons_cancel_stands.
Doc: skip-only shards in fleet mode	The doc now says they run GitHub-hosted.
Plan row 9	Not here. The diff is already about 2,300 lines, so it will be a stacked follow-up PR.

Found while testing, and fixed:

• the launcher's JIT check was a bash regex error that refused every config;
• WORK_ROOT had to be 0711 so the runner user can reach its copy;
• the launcher's EXIT trap named a local, so every start exited 1 under set -u and left the environment file behind;
• the dispatcher could make a runner for attempt 1 after the owner's re-run had moved the job to attempt 2, and the gate would then kill a trusted job. It now waits a pass;
• the lock directory under /run is created after a boot.

Red → green evidence

Each mutation was applied to the real file, the suite was run (red), the file was restored, and the suite was run again (green):

ok  a person's cancel stands (s1 minor, fork-fleet.py rescue_reason): mutated -> FAILED (failures=1); restored -> OK
ok  the hook kills the worker (s1 blocker, stop_job): mutated -> FAILED (failures=2); restored -> OK
ok  the gate checks the run its runner was made for: mutated -> FAILED (failures=3); restored -> OK
ok  the sweep applies the trust rule (s1 minor): mutated -> FAILED (failures=6); restored -> OK
ok  the dispatcher applies the trust rule (blocker root): mutated -> FAILED (failures=4); restored -> OK
ok  a garbled reply is never red (s2 minor 3): mutated -> FAILED (errors=2); restored -> OK
ok  a re-run follows the setting (s2 minor 1, runs-on): mutated -> FAILED (failures=3); restored -> OK
ok  the launcher takes only its three input lines: mutated -> FAILED (failures=2); restored -> OK
ok  the launcher refuses a configured template (.env): mutated -> FAILED (failures=5); restored -> OK
ok  launcher bug 1: the JIT check was a bash regex error: mutated -> FAILED (failures=2); restored -> OK
ok  launcher bug 2: WORK_ROOT must let RUNNER_USER through (0711): mutated -> FAILED (failures=1); restored -> OK
ok  launcher bug 3: the EXIT trap named a local: mutated -> FAILED (failures=2); restored -> OK
ok  the dispatcher makes a runner only for the attempt it read: mutated -> FAILED (failures=1); restored -> OK
ok  fleet mode: the sweep reports a long wait (s1 minor: no silent 24 h): mutated -> FAILED (failures=1); restored -> OK
ok  fleet mode: the sweep cancels a run past fleet_stuck_seconds: mutated -> FAILED (failures=1); restored -> OK

Suites at the new head, all green:

• fork-fleet.test.py: 104 tests, OK;
• fork-contracts.test.py and fork-release-workflow.test.py;
• node .github/scripts/fork-ci-tests.cjs (rc 0);
• fork-tests-drift.cjs: matches the record;
• actionlint on fork-*.yml;
• bash -n on both new scripts;
• gitleaks on the staged diff.

shellcheck is not installed here, so it did not run.

Not proved here: a real runner. The runbook's real-runner check needs a throwaway private repository under the owner's account, and none was created. It runs on VM106 next to the real install and checks:

• a refused job's always() step never runs;
• the environment file is gone after the start;
• the runner user cannot write the template;
• a bare label finds no runner;
• a re-run gets a runner for attempt 2.

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1-r2) · read at: eaf1320
covered: the fix diff f2d6e15..eaf1320 minus the merge of main 919556a (git diff 919556a eaf1320 over the 10 changed files). I read in full: fork-fleet.py (route, watch, sweep, dispatch, Host, gate, worker_above, stop_job, refused, run_refused), fork-fleet-launch.sh, fork-fleet-hook.sh, fork-fleet.json, fork-fleet.yml, every fork-tests.yml change (Route step, fleet_watch, 7 routed runs-on lines, no-sudo branches, fork-ci-result), fork-release.yml's permissions, the runbook .github/fork-fleet.md (host steps, owner steps, real-runner check, failure table), and the GateProcess/Dispatch/Launcher/Wiring tests. I also read what this reaches: .github/actions/setup-ci-deps (the ~/.vite-plus cache) and fork-release.yml's build job, which uses it.
checks:

• Hard rule (stranger never runs on the fleet): holds by design. The dispatcher starts a runner only for a job that asks for exactly a pool label plus a well-formed nonce label (fleet_job), and only for a run that GitHub's own record says is trusted (run_refused: actor, triggering_actor, event, workflow path, head repo = run head repo = base, and PR author from pulls/; a fork run lists no PRs, so it is refused). A fork PR whose workflow names a fleet label, or copies a run's public nonce label, finds no runner, or it blocks the launch through the rival check. If it wins the race for a fresh runner, the gate refuses it on made_for (run id/attempt from the root-only env file) before any step. Before the hook, only the job message and action downloads touch the host; no stranger code runs. Re-runs by others, workflow_run, labels and comments are all refused by the same rule. fork: outside PRs run nothing, one checkout without actions/checkout, contract tests #12 and all_external_contributors come on top.
• GitHub input reaching root: none reaches a shell, path, unit name or file. The unit or dir name is the pool label from config plus int(job id). Labels are regex-checked before generate-jitconfig. The JIT config is checked against [A-Za-z0-9+/=] in both Python and bash, and run and attempt are int()ed. The launcher accepts only its three stdin keys, reads launch.conf as data and never sources it, and names reach systemctl only after name_ok. prune runs rm -rf --one-file-system only on top-level names that root created in a 0711 root dir. Job names and labels go only to the journal.
• Hook kill: stop_job SIGKILLs the nearest Runner.Worker ancestor. hook.sh execs python -I, so the worker is the direct parent. A crashing gate is killed too. The GateProcess model shows that no always() marker is written, and the mutant that only exits 1 is caught. This is runner behaviour; I did not prove it on a real runner (the runbook check 2 covers it). One residual case: if the installed fork-fleet.py cannot even start (a syntax error, or GNU install replacing it as a hook starts), python exits 2 without the kill. Only a runner made for a trusted run can hit this.
• Escape between jobs: the template is safe. The copies hard-link root-owned go-w files, the subdirs stay root's, and the launcher refuses .env/.runner/.credentials. The hook path comes from the root-only env file, so a job cannot drop it (this fixes S-2 fork: follow each stable upstream release with the patch series #4's hook part). The unit has KillMode=control-group, RuntimeMaxSec, NoNewPrivileges and PrivateTmp; with no linger, cron or at, nothing outlives the unit. Jobs are not separated from each other: see finding 1.
• Races: the dispatcher's attempt race is fixed (it skips a job whose run_attempt differs). A second queued job that could take the runner is a rival, so nothing starts. The case left open is GitHub handing an idle leftover runner the next attempt's job: see finding 2.
• Round-1 findings: S-1 blocker fixed at the root (no standing runner, dispatcher trust, gate kill). S-1 sweep trust fixed (run_refused, 7 tests red when removed). S-1 cron-only bound fixed (fleet_watch). S-1 "a person's cancel stands" test added. S-2 fork: the server and CLI update from the desktop app's repository #1 re-run routing fixed in runs-on (vars.CI_RUNNERS checked, wiring test). S-2 fork: release workflow for v<upstream>-fork.N on GitHub-hosted runners #2: route is now a step of changes. S-2 fork: CI check that refuses database schema and migration changes #3: TROUBLE has HTTPException, ValueError, KeyError and TypeError. S-2 fork: follow each stable upstream release with the patch series #4: the hook part is fixed, but the "one compromised trusted job reaches later jobs" part is not: finding 1.
• auto never red or stuck: an unclear heartbeat goes hosted. A job queued 10 min means watch cancels, then the sweep re-runs hosted (github-actions is untrusted). A gate-killed or lost runner has no failed step, so it is re-run hosted. The watch has API trouble handling and gives up quietly after bound+2 min. The only lasting red is when both watch and sweep are dead. fleet never silently hosted: untrusted runs and skip-only test_server shards go hosted with a notice (documented). A re-run of failed jobs after a switch auto->fleet goes hosted (attempt differs), which is right.
• State question: unset variable = auto. A typo makes the Route step red. Dispatcher restarted mid-launch: an offline registration with no unit is deleted, prune removes the half copy and the env file, and it retries (MAX_TRIES 3, in memory, so it resets on restart; harmless). After a reboot, /run is recreated (lock dir and RUN_DIR), offline registrations are deleted, and copies are pruned. A leftover unit is counted in active and its name is skipped. A leftover runner user is fine; useradd fails on a re-run (finding 5). A missing, world-readable or empty token file refuses at start, Restart=always retries every 30 s, and auto goes hosted. An expired token is a 401 (HTTPError is in TROUBLE), so each pass warns, heartbeats go stale, auto goes hosted, and fleet errors after 2 h. Two dispatchers: flock on /run/fork-fleet/dispatch.lock, but finding 3 covers two pools. A run from before this PR, re-run, uses its own workflow: hosted. Empty history: no heartbeat, so hosted. The sweep is stateless and serialised.
• Runbook as root: steps 3-5 are safe and idempotent. The token goes in with umask 077 and read -s, launch.conf is chmod 600, and the template is sha256-checked and extracted with --no-same-owner + go-w. Steps 1-2 are not idempotent or atomic: finding 5.
• Tests (scratch clone at the head, GH_TOKEN/GITHUB_TOKEN unset): fork-fleet.test.py 104 OK; fork-contracts and fork-release-workflow tests pass. actionlint 1.7.12 on fork-tests/fork-fleet/fork-release: clean. shellcheck on launch.sh and hook.sh: only SC2064 at launch.sh:107, which is intended (expand now). I broke 41 guards by hand, one at a time, and 40 went red: fleet_job label count, rival check, dispatcher trust, PR head vs run head, the no-PR fork run, gate made_for, made_for attempt, the kill, the crash path, the attempt-race skip, the at-once idle stop, MAX_TRIES, token perms, lock, JIT check, deleting a live registration, GENERIC in could_take, started_mb in fits, route trust, sweep trust, and in the launcher: conf/hook/template ownership, the template .env refusal, root RUNNER_USER, name_ok in start and stop, dir reuse, the trap, the hook line, RuntimeMaxSec, KillMode, PrivateTmp/NNP, the copy and RUN_DIR modes, extra input keys, prune of live units, hook -I, runs-on ignoring github, and the watch's cancel. The one that stayed green is finding 4. The clone is clean afterwards and was deleted.
  finding: major .github/scripts/fork-fleet-launch.sh:117 (--uid "$RUNNER_USER") + .github/fork-fleet.md:196-198 + fork-fleet.json slots 2. Every one-job runner runs as the same user fork-runner, with one persistent $HOME, and two run at once. Inputs: the owner's own unmerged PR pulls in a bad dependency (a postinstall), and that is a trusted run, so it lands on the fleet. (a) While a push-to-main job runs in the other slot, the PR job reads /proc//environ of that job's processes (same uid; Yama does not limit environ reads). It gets ACTIONS_RUNTIME_TOKEN, which can write main-scope Actions cache, and also the JIT config of a runner that is starting: run.sh, run-helper and the Listener's initial environ keep ACTIONS_RUNNER_INPUT_JITCONFIG, because the runner's delete changes only its .NET view. With that config it can take the main job onto its own runner. (b) Without any concurrency, it plants files under ~/.vite-plus, which setup-ci-deps:36-41 caches by os/arch/version key only. A later main job on a key miss saves the planted tree to main scope, and fork-release.yml's Linux build job (signing secrets, setup-ci-deps at :196) restores that key and runs vp from it. GitHub-hosted runners keep PR scope away from main scope; this design removes that wall. WORK_ROOT 0711 and the 0700 copies do not separate jobs of one uid. The stranger rule is not broken. Fix: one uid per unit (systemd DynamicUser=yes, or a pool of users with exactly one unit per user at a time); HOME (and XDG/tool caches) inside the copy so prune deletes it; drop the "home persists" row, or mark it as a deliberate risk the owner accepts. (medium: same-uid environ access is standard Linux; the cache chain comes from reading setup-ci-deps and fork-release, not from a run)
  finding: minor .github/scripts/fork-fleet.py:672-679 + the routed runs-on lines in fork-tests.yml. Inputs: fleet mode; the owner cancels a run while a fleet job's runner is still coming up; then, within about a minute, clicks "Re-run failed jobs". The attempt-2 job keeps attempt 1's labels (route is not re-run, so it has the same nonce). The attempt-1 runner, now online and idle, matches it, and GitHub assigns it within seconds, before the dispatcher's next 20 s pass can stop it. The gate sees FORK_FLEET_ATTEMPT 1 != 2 and kills the job, so the run is red. fork-fleet.md:431 ("stopped at once if another queued job could take it") overstates this. Fix: put github.run_attempt into the job's own label (for example, build the nonce label in runs-on as -a, and match the attempt in fleet_job), so that a runner of one attempt can never match another attempt. (medium)
  finding: minor .github/scripts/fork-fleet.py:653 (units for every pool) + :789. One dispatcher serving several pools (the default with no --pool, and the json says pc's dispatcher runs on the always-on host): a HostError from one pool (pc: ssh to a PC that is off) aborts the whole pass. The vm pool then gets no launches and no heartbeat, so auto goes all hosted and fleet waits 2 h, then errors. The alternative, a second dispatcher with --pool pc, fails on the shared default --lock /run/fork-fleet/dispatch.lock. test_a_host_that_cannot_answer_stops_the_pass pins today's behaviour. This is not live while the runbook runs --pool vm only. Fix: catch HostError per pool (skip that pool, beacon free=0) or default the lock per pool, before pc joins. (high)
  finding: minor .github/scripts/fork-fleet.py:743. Test gap: dropping "- pending[...]" from the heartbeat's free count leaves all 104 tests green. Inputs: trusted fleet jobs are queued and waiting (no slot or memory). Result: the heartbeat still reports free slots, auto routes more jobs to a full pool, and they wait 10 min, get cancelled and re-run. Add a heartbeat test with pending jobs. (high)
  finding: minor .github/fork-fleet.md:215-232. Host steps 1-2 run again: useradd fails on an existing fork-runner, and the cron/at.deny line is appended each time. Step 2's template refresh (rm -rf then mv) races a dispatcher in the middle of cp -al: you get a half copy and a failed runner, which is retried. Fix: id fork-runner || useradd ..., grep -qx before appending, and stop the dispatcher around step 2 (or use an atomic rename -T swap). (high)

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2-r2) · read at: eaf1320
covered: the fix round's own changes (git diff 919556a..eaf1320, which leaves out what the #12 merge brought). Read in full: fork-fleet.py (route, watch, sweep, dispatch, gate), fork-fleet-launch.sh, fork-fleet-hook.sh, fork-fleet.json, fork-fleet.yml, every fork-tests.yml change (Route step, fleet_watch, the 7 routed runs-on lines, the no-sudo apt branches, fork-ci-result), the fork-release.yml permissions, the runbook .github/fork-fleet.md (host steps, owner steps, real-runner check, failure table) and the GateProcess/Launcher/Dispatch/Wiring tests. Round-1 S-1 and S-2 findings checked one by one.
checks:

• Hard rule (strangers never reach the fleet): holds. The dispatcher starts a runner only for a job asking for exactly {pool label, pool-label-key-16hex}, of a run whose GitHub record passes refused() (fork PR runs list no pull request: refused; each PR is read from pulls/, head must equal the run's head repo, opener/actor/triggering actor must be trusted). A bare fleet label, self-hosted, a group-only runs-on or a copied nonce label from the public jobs API is either not matched or counted as a rival, so no launch. A stranger's job that queues after the launch and races onto the runner is killed by the gate (made_for fails: wrong run). A stranger opening a PR from the owner's own branch: actor/opener check refuses it, and the owner's run on that branch is refused too (fail-closed, DoS only). workflow_call/release, workflow_run, schedule, sweep re-runs (github-actions) and bots: refused. Re-runs by others: strangers cannot re-run here.
• Root attack surface: no GitHub-controlled string reaches a shell, path, unit name or file as root. The unit/runner name is label + int(job id), runner names from the API must fullmatch label-digits, generate-jitconfig labels are only the validated pair, the JIT config is base64-checked in Python and again in the launcher, run/attempt are ints, launch.conf is parsed (never sourced). Job names only reach the journal. prune runs rm -rf --one-file-system as root on runner-written trees; GNU rm does not follow symlinks, so it stays inside. The token file (root 0600 in a 0700 dir), the template (root-owned, hard links the runner user cannot write) and root (no sudo, NoNewPrivileges, PrivateTmp, KillMode=control-group, RuntimeMaxSec) are out of a job's reach. Not out of reach: the other runner and later jobs (finding 1).
• Gate kill: the hook execs python under Runner.Worker, and worker_above finds it by comm. SIGKILL ends the worker before any step, always() included. GateProcess shows it, and the mutant that only exits 1 leaves the marker. Not proven on a real runner, which is what the runbook's real-runner check is for. Do it before the fleet takes jobs.
• Round-1 findings: S-1 blocker fixed at the root (no standing runner; trust is decided from GitHub's record, outside any job). S-2 fork: the server and CLI update from the desktop app's repository #1 (re-run routing) fixed (vars.CI_RUNNERS in runs-on, mutation red). fork: release workflow for v<upstream>-fork.N on GitHub-hosted runners #2 done (Route is a step of changes). fork: CI check that refuses database schema and migration changes #3 fixed (http.client.HTTPException in TROUBLE). fork: follow each stable upstream release with the patch series #4 (.env writable) only partly fixed (finding 1). S-1 sweep trust, auto bound (fleet_watch) and the person's-cancel test: fixed and tested.
• auto: no heartbeat, a stale or garbled one, a dispatcher that is down, a job that never gets a runner (rival, MAX_TRIES, memory) all end GitHub-hosted, or cancelled at 10 min by the watch and re-run hosted by the sweep. Never red for good. fleet: a trusted job is never hosted silently: route crash = red, refused/blocked = 2 h error by the watch or the sweep; skip-only shards and untrusted runs hosted, as documented.
• State question: variable unset = auto, typo = red (tested). Dispatcher restarted mid-launch: offline registration deleted, half copy and env file pruned, relaunched. Reboot: /run lock dir and RUN_DIR re-created, copies pruned, lost job rescued in auto. A leftover unit or registration not matching label-digits is ignored (the first design's standing runner must be deleted by hand, as the doc says). Token missing or wrong mode = refuses to start; expired = every pass warns and the heartbeat goes stale = hosted. Two dispatchers = second fails on flock. A run from before this PR re-run = its old workflow, hosted. The re-run race between reading the run and its jobs is handled (tested). Runbook as root: safe, nothing prints a secret, and every step re-runs harmlessly (useradd just fails, cron.deny gains a duplicate line).
• Tests in a scratch clone at the head with GH_TOKEN/GITHUB_TOKEN unset: fork-fleet.test.py 104 OK (5.5 s). Broke 40 guards by hand, one at a time, file restored after each: 39 red. Red: dispatch trust skip, exact label pair, rival check, attempt check, gate made_for, gate refused, no SIGKILL, fork PR with no pulls, PR head compare, could_take GENERIC/case, idle stop at once, sweep trust, route trust, push branch, triggering actor, PR opener, token mode, flock mode, gate crash-open, pending count, MAX_TRIES, person's cancel, watch fleet bound, launcher (.env allowed, template find check, RUN_DIR 0700, EXIT trap, NoNewPrivileges, PrivateTmp, KillMode, RuntimeMaxSec, uid 0 allowed, JIT charset, dir-exists, conf owner, prune deleting live copies), runs-on (vars fleet check, github check). Green: worker_above matching comm.startswith('Runner') (equivalent: the nearest ancestor is still Runner.Worker).
• actionlint 1.7.12 on fork-*.yml: clean. shellcheck 0.10.0 on both scripts: one SC2064 at fork-fleet-launch.sh:107, deliberate (expanded now on purpose); add a disable comment if shellcheck ever gates. Clone deleted.
  finding: minor .github/scripts/fork-fleet-launch.sh:109,117 + .github/fork-fleet.md:74-75,196-198,215 All runners run as one uid (fork-runner), and the copy's top directory is that uid's (chown + 0700), so "a user of its own" is not true across runners. Inputs: vm has 2 slots; a trusted job runs a malicious dependency (an owner dependency-bump or upstream-sync PR) while the other slot's runner is starting. The job can open WORK_ROOT/fleet-vm-t3code- by name (0711 hides only the listing; job ids are public). There it can write .env (the listener reads it on start, and again after a self-update restart), swap run.sh/bin, or read /proc//environ, where the JIT config stays in the initial env block after the runner unsets it. Each of these runs that runner without the gate, or as itself. $HOME (rustup, caches) and /dev/shm also carry the compromise into every later fleet job, pushes to main included. The dispatcher still keeps strangers out (a stranger would also need to race onto that runner), so this is defence in depth. It is round-1 S-2 fork: follow each stable upstream release with the patch series #4 again: one job can still change another job's gate. Fix: one uid per runner (a fixed fork-runner-1..N, one per slot and never two units at once, or DynamicUser with the copy as its state directory), a per-job HOME (e.g. HOME=/home in the env file, or ProtectHome=tmpfs), and a real-runner check step where one job tries to write another runner's copy. (medium-high)
  finding: minor .github/scripts/fork-fleet.py:672-679 (+ gate made_for at :236-242) Fleet mode, a runner left over from a cancelled attempt. Inputs: CI_RUNNERS=fleet. The owner cancels a run while the runner made for job X (attempt 1) is up but has not taken it, then clicks "Re-run failed jobs" within about a minute. Attempt 2's job has the same two labels (in fleet mode runs-on keeps attempt 1's labels), and GitHub can give it to the idle attempt-1 runner before a pass stops it (idle_passes x 20 s, or the next pass at best). The gate then kills a trusted job: red, and fleet mode never rescues. Fix: stop the unit and delete the registration as soon as the runner's own job (its id is in the name) is completed or cancelled. Read actions/jobs/ instead of waiting for idle passes. (medium: depends on GitHub's assignment timing)
  finding: minor .github/fork-fleet.md:193-195 + 323-392 Runner self-update in the hard-linked copy is assumed, not checked. Inputs: the template falls one actions/runner release behind (every few weeks). Each JIT runner then updates itself before its job, inside a copy whose bin/, externals/ and top-level files are root-owned hard links. The real-runner check uses a fresh template, so it never runs this path. If the updater cannot rename or overwrite those files, every fleet job loses its runner: auto pays 10 min plus a full hosted re-run on every run, and fleet goes red, until someone refreshes the template. Even when the update works, every job pays for a runner download. Fix: add a step to the real-runner check with a template one release behind, and have the dispatcher report free=0 (or warn) when the template is older than the newest release. (low-medium)