Add the shared tool catalogue, from team run be60 - #110
Merged
Merged
Conversation
added 17 commits
September 23, 2026 16:18
Piece 1 of the tool registry plan. A reusable tool is a manifest, a script and its cases, kept on this machine under <State>/tools. ToolRegistry.Root() is the only place that location is decided (D1). What is here: - Models (src/Loadout.Models/Tools): ToolRecord, ToolVersion, ToolCase, ToolUsage - ownership, version, capabilities, dependencies, compatibility, test status, usage and lifecycle state. - ToolRegistry: drafts, verify, promote, set-active, deprecate, usage and an audit trail. A promoted version is written once; only a known-good version whose fingerprint still matches can be active, so a failed or tampered refinement never replaces the last known-good one. Deprecation needs a replacement or a reason. - ToolHarness: runs every case in a fresh temp directory and refuses a set of cases missing any of success, failure, edge or invalid-input. Running goes through RemedyCeiling with kind tool-test and asks a person by default (D2); the fingerprint is RemedyCeiling.Fingerprint, reused. - ToolPromotion: the manifest must name purpose, inputs, outputs, error behaviour, examples and origin; a draft is gated on its own cases and on every case of the active known-good version. - ToolCompatibility: a removed required input, a new required input with no default, or a changed exit meaning is a break, allowed only when declared with a major bump and a migration. - ToolGenericity: refuses project detail (absolute paths, repository URLs, e-mail addresses, GUIDs, known project and team names) and reports secrets by type, never by value. - ToolOverlap: duplicate by normalised-script fingerprint, overlap by a 0.6 score over description words and script shingles. - Stop rule: two stand-downs with no new signal and a tool is left alone. Verification: the Tool filter passes 56 of 56 and the full suite passes 3158 of 3180, with 22 skipped. Every one of the 19 new tests was mutation-checked: 20 compilable mutations, each failed its named test and was then reverted. Reverting meant a matching edit, checked against a staged copy of every file, not git checkout. Generic_text_passes had survived every plausible mutation, because its input contained nothing that came close to matching any check. It now contains the near-miss "alpha-shopfront", and dropping the known-name regex's trailing boundary fails it. Left out on purpose: a registry lock (write-once is a Directory.Exists check, which is not atomic), the nightly overlap pass, and any CLI or MCP surface.
The review of 9c5e7b2 found four places where the registry believed a file that anything with Bash can rewrite, and no lock around its writes. - Verify is no longer forgeable. Promote read status and fingerprints from the draft's own manifest.yaml, so a draft written by hand as "verified", or a rejected draft edited to say so, was promoted. A passing verify now writes verified/<name>/<version>.json under Root(), outside the drafts; a rejected one deletes it. Promote compares the draft's current script and cases fingerprints with that record and refuses without it. - A secret is never echoed. ToolGenericity quoted each match, and the repository URL pattern captured user-info, so a token in a clone URL came back in the refusal, twice (once as a URL, once read as an e-mail address). Once any secret pattern matches, no value is quoted; URL user-info is stripped before a URL is quoted; a value that itself matches a secret pattern is not quoted; and an address no longer starts after ':' or '/'. - Drafts are confined to Root()/drafts, after the path is resolved. Verify wrote manifest.yaml into whatever path it was given. The test fixture now writes its drafts there too. - Standing checks a version's files against the fingerprints recorded in the promote audit entry ("script <fp> cases <fp>"), not against the manifest beside them, so rewriting the script and the manifest's fingerprint together reads as tampered. - registry.lock, as the plan's section 1 says: an exclusive FileStream under Root(), waited for with back-off up to LockWait (10s), held across Promote and every read-modify-write of a head (usage, set active, deprecate, retire) and around verify's writes. A writer that cannot get it is refused with a sentence saying so. Before it, twelve promotions at once failed four of twelve. Minor items: - Promote writes into a dot-named staging directory beside the target and moves it into place, deleting it on failure, so a failed copy leaves no half-written versions/<v>/. Not covered by a test: forcing a copy to fail after the draft has been read needs a seam that does not exist. - ToolHarness refuses any argument with a '..' segment, which covers '{tmp}/..', '{tmp}\..' and relative '../'. - A missing tool still returns ExitCode.ProjectNotFound (3): there is no better code, and adding one changes the public exit-code contract. - Correction to 9c5e7b2's message: promotion requires eight manifest fields, not six - purpose, inputs, outputs, dependencies, constraints, error behaviour, examples and origin, as ToolPromotion.Missing and the theory in ToolRegistryTests check. Tests, each seen failing before the fix and failing again under a compilable mutation of it, restored from a copy: ToolRegistrySafetyTests (8) - A_draft_claiming_verified_without_a_verify_ record_is_refused, A_rejected_draft_edited_to_verified_is_refused, A_token_inside_a_url_never_reaches_the_refusal_text, A_password_in_a_ url_is_not_quoted_even_when_no_secret_pattern_knows_it, A_draft_outside_ the_drafts_directory_is_refused, Rewriting_script_and_manifest_ fingerprint_together_reads_as_tampered, A_second_writer_is_refused_while_ the_lock_is_held, Promotions_at_once_lose_no_known_good_version; and ToolHarnessTests.An_argument_climbing_out_of_tmp_is_refused_without_ running (4 cases). dotnet build Loadout.slnx: 0 Warning(s), 0 Error(s). Tool filter: 68 passed of 68. Full suite: 3170 passed, 0 failed, 22 skipped, of 3192.
A tool called drafts, inbox or verified would share a directory with the catalogue's working files: a tool named drafts would put its promoted versions where agents are allowed to write, and one named verified would sit among the proofs promotion trusts. Verify and promote now both refuse those names.
Standing read the LAST "promote" line in the audit log for a version. The log is append-only and a version is written once, so any later promote line for the same version is somebody vouching for files the gate never saw: edit versions/1.0's script, append a promote line carrying the new fingerprint, and the version read as known-good. It now reads the first. 'versions' joins drafts, inbox and verified as a name no tool may take, since a tool called that would share its directory name with every tool's own version store. The registry's remarks now state the precondition the whole design rests on: a node may write only Root()/drafts and Root()/inbox. Tests: Appending_a_forged_promote_entry_leaves_the_version_tampered and a "versions" row in A_tool_named_after_the_catalogues_own_directories_is_refused both failed before the change and pass after.
Piece 2 of the tool registry: the catalogue becomes reachable from where agents and people work, without any of it deciding what may run. - 'loadout tools' search, show, submit, used, audit, verify, promote, deprecate, retire and trust, with --json and --dry-run. - Four MCP tools making the same calls: loadout_tools_search, _show, _submit and _used. Promote, trust and deprecate are deliberately absent: promotion is the gate's and trust is a person's. - Every brief carries a fixed pointer to the catalogue and never lists tools, so a growing catalogue costs no brief anything. - A remediator is offered active, known-good tool versions through the same RemedyCeiling ruling remedies use, named tool:<name>@<version> and matched by the versioned script file name. Agreement lives in MachineTeams.TrustedTools, fingerprinting the script ScriptOf reads from disk, so a record in the catalogue claiming trust decides nothing and a changed script is asked about again. Tests cover the brief pointer and its fixed length, the --json shapes and that --dry-run changes nothing, trust by fingerprint, offering only to the remediator and never a candidate, secret screening on MCP submit, and a guard that no node can reach the catalogue outside drafts and inbox. Each new test was mutation-checked against a compiling mutation and failed. docs/commands.md gains the rows for 'loadout tools' and the four MCP tools.
Verify already refuses a tool named after the catalogue's own directories, so promote failed for want of a verify record whether or not its reserved-name check was there. The theory now asserts promote's own refusal text. Deleting the check at ToolRegistry.cs:574 fails all four cases; restored afterwards.
The content comes from 9a7d57f on teams-20260923-1216-be60-implementer-3, which was cut from main rather than from the registry pieces. It is applied here on top of piece 2 unchanged: the 'loadout tools' subcommands the roles name (search, show, submit, used, audit, verify, deprecate, retire) all exist in ToolCommands.cs, so SpecialistCommandTests, which failed on 9a7d57f alone, now passes. Neither role can promote or trust a tool. Each allowed subcommand is named individually and promote and trust are denied as well; the refiner alone may deprecate and retire. tool-works is role.project-lead delegating to creator and refiner, autonomous, $3 and 30 turns per node, outward: ask, stop_when goal_met, budget_spent, no_progress_2_rounds. Specialist counts go 101 -> 103, roles 23 -> 25. Mutation checks: writing the creator's allow list as the plan's blanket 'Bash(loadout tools:*)' without the promote deny fails Creator_and_refiner_roles_cannot_promote_or_trust; a $5 budget fails tool_works_team_validates. Both restored from 9a7d57f. Suite: 3202 passed, 0 failed, 22 skipped.
…when runs finish
ToolNominator reads finished, non-tool-works runs (reports through
RunDocuments.In, Kind == "report") and every team's remedy shelf, and
files a candidate when: (1) one remedy, by fingerprint or overlap >= 0.6,
sits on two or more team shelves; (2) a remedy revised at least once passed
in two or more runs; (3) one command shape, arguments replaced, passed in
runs of two or more teams; (4) a lesson names a command that passed. It
never nominates what an active tool already covers at overlap >= 0.6, and
files each nomination once, keyed in the audit log.
It writes through a new IToolRegistry.Nominate, which is not on the CLI or
MCP. It skips the genericity check, because a nomination quotes the project
it came from so the Creator can find it. It keeps the secret screen and the
audit record ("nominate").
ScheduleService.Events gains run-finished. RunFinished is a pure rule over
RunSummary: the run must be finished, from a team that is neither tool-works
nor the schedule's own, and ordinally after the watermark kept in
LastCommit, with LastRun more than an hour ago. The first look records a
baseline and does not fire. A run held back by the hour is not written off,
because the watermark moves only when the schedule fires. The daemon's
ReadyAsync applies it, and SeenAsync skips these schedules so it cannot
overwrite the watermark with a git head.
TeamRunner gives role.tool-creator and role.tool-refiner nodes
<State>/tools/drafts and inbox, and nothing else under <State>/tools. The
new tool-works test runs a creator node, so the drafts-and-inbox branch of
the catalogue guard now actually runs. Before, it launched only a lead.
Mutation checks: each new test failed under a compilable mutation of its
own guard, except the tool-works schedule rule. Its test's schedule belonged
to tool-works, so the own-team check hid the mutation. The schedule now
belongs to another team; that fix was not re-mutated.
…scripts directly 'loadout tools verify --dry-run' and 'promote --dry-run' reported that they would act on a draft the real run could not even read. Both now call IToolRegistry.CheckDraft first, which is ReadDraft's answer without the side effects, and fail with InvalidArguments exactly as the real run would. A contract test runs both against a draft that does not exist and expects a non-zero exit, "no readable manifest.yaml", and no "would be". The tool-creator and tool-refiner roles no longer allow 'pwsh -NoProfile -File'. A draft runs only through 'loadout tools verify', which goes through the same gate as every other script; a direct pwsh call went round it. RoleLibraryTests pins that for both roles: restoring the refiner's allow fails the role.tool-refiner case. The installing guide's doctor sample now counts 103 specialists, the number the library loads since the two tool roles were added.
…tarts ToolNominator existed but nothing called it. ToolNominationPass runs it over the run journal, the remedy shelves and every lesson topic of every registered project, and the daemon calls it just before starting a schedule that fires on run-finished or starts tool-works, so the inbox the Creator reads already holds what finished work nominated. A project whose memory cannot be read is passed over rather than failing the pass. The daemon prints how many it filed when there were any. Nothing else calls the pass: a nomination nobody reads until the next tool-works run can wait until then to be filed. ToolNominationPassTests cover both triggering schedules and the ones that do not; making Precedes return false fails the two triggering cases. DaemonControlTests and DaemonStartsWorkTests pass the new constructor argument.
…t rule-2 match The review of b215176 returned it with five findings. Each is fixed here with a test that pins it. (a) run-finished kept the newest run id as its watermark, and a run id is ordered by when the run began. A long run that began first and finished after a shorter one already seen sorted below the watermark and never fired. The watermark is now the latest finish time, in round-trip form; a watermark that does not parse as a time (one an older build wrote) is treated as a first look and re-baselined. Test: A_run_that_finishes_after_a_later_starting_run_still_fires. (b) A hit an active tool already covers was dropped, where plan section 6 says it goes to the Refiner as a usage hint. The nominator now files it through ToolRegistry.Nominate naming the tool, and Nominate files a submission that names a tool as an idea about that tool rather than as a candidate. Test: A_covered_hit_becomes_a_refiner_hint. (c) For rules 3 and 4 there is no script, so coverage compared prose with script shingles and could almost never reach the overlap threshold. A command is now covered where a tool's examples run the same shape, or its script or capabilities name the command. Test: A_command_an_active_tool_wraps_is_not_nominated. (d) Rule 2 matched evidence by file-name substring from any team, so another team's clear-cache.ps1 counted and fix.ps1 matched prefix.ps1. It now matches the whole file name, in the owning team's runs only. Tests: Another_teams_same_named_script_does_not_count, fix_ps1_does_not_match_prefix_ps1. (e) Rule 1's key listed every shelf in the cluster, so a third team shelving the same remedy changed the key and filed the cluster again. The key is now the scripts' fingerprints, with the other fingerprints kept as alternative keys checked before filing. Test: A_third_team_joining_a_cluster_is_not_filed_again.
The review of dd24997 returned it with three findings. (1) The daemon awaited the nomination pass with nothing around it, so one folder the pass could not read threw out of FireAsync and the scheduled run never started. Starting a schedule's team is now TeamDaemonCommand.StartAsync, which catches IOException, UnauthorizedAccessException and RegexMatchTimeoutException from the pass, writes one line to the daemon's output ("nominated nothing before <schedule>: ...") and starts the run anyway. The daemon takes the pass through a new IToolNominationPass, which is the seam the test needed. Inside the nominator, one unreadable folder now skips only itself: the enumeration of team work directories, each team's shelf, and each run's summary and documents are guarded separately. RunJournal.Summarise lists the run folder too (NodePermissions.Pending), which is where the test's first failure came from, so it is inside the per-run guard. (2) The 'passed over' branch in ToolNominationPass tested topics.Failed, which MemoryService.ListAsync never returns: it leaves out a scope it cannot read. It is replaced by a catch around each project's memory read that logs "passed over the lessons of <slug>" and carries on. BeforeAsync takes an optional log callback for these lines, which the daemon supplies. (3) Tests: - An_unreadable_run_folder_does_not_stop_the_pass denies listing on one of two finished runs (an ACL deny on Windows, mode --x on Unix) and expects the other still to be nominated. Before the change it failed with UnauthorizedAccessException from RunJournal.Summarise. - The_daemon_still_starts_the_run_when_the_nomination_pass_throws uses a pass that throws UnauthorizedAccessException and expects "team run" to be asked and the log line to be written. With UnauthorizedAccessException taken out of the catch it failed with that exception.
… tool GlobalToolLifecycleTests drives a lesson from team A into a global tool that team B uses, over the catalogue's own code. Scripted: team A's two runs are written as the runner writes them (a journal and a remediator's report), and its remedy is placed on its shelf directly. The Creator's turns are the calls its commands make - tools search, submit, verify, promote - not an agent session. Team B's run is not driven through TeamRunner: its node is a brief rendered by TeamRunner.Render and a permission policy from ToolOffer.For, the two calls the runner makes when briefing a remediator. A person's answers (agreeing to a harness run, trusting a version) are written into the records the CLI writes. Every harness run goes through a stub launcher, so nothing needs pwsh. Real: the daemon's nomination pass, the genericity and overlap screens, the verify hold and regression gate, write-once promotion, the trust ruling, and usage. Three tests: the lesson crossing teams with no project detail under versions/; a failed refinement leaving 1.0 active and known-good; and each way a project can ride along being refused. For the last, a case carrying an absolute path is refused by the harness (ToolHarness.RunAsync) before promote, so that draft now expects verify's gate to fail naming the path; the other three still pass verify and are refused at promote. Mutation checks, each undone before the suite ran: - ToolGenericity.Check returning nothing for any text (a plain inversion of IsNullOrWhiteSpace does not compile under warnings-as-errors) fails tests 1 and 3. - regressions.Count <= 1 in ToolPromotion.GateAsync fails test 2. Suite: 3227 passed, 0 failed, 22 skipped.
Four findings from the reviewer's return of d080f42. Rule 1 now records every member fingerprint of a cluster it files, not only the lowest. A cluster {A,B} filed under A, which then loses A and gains C, was filed again as {B,C}; the audit note now carries every key and the filed check reads them all. A_cluster_that_loses_its_filed_key_is_not_filed_again failed before this and passes after. A command with no script is covered by a tool only where the tool's script runs it: its words, whole, at the start of a line or after a pipe, separator or bracket. The substring check read "# Make sure the cache path exists" as running make and hid a rule 3 nomination. A_one_word_command_is_not_covered_by_a_script_mentioning_it_in_prose failed before this and passes after. A_third_team_joining_a_cluster_is_not_filed_again now gives gamma a near copy whose fingerprint sorts lowest, so the cluster's key moves and only the Also check can recognise it. Replacing that check with the primary key alone fails the test; reverted. An_unreadable_run_folder_does_not_stop_the_pass takes a permission away from itself, which uid 0 cannot lose on Unix, so it now skips there through a DeniableFact attribute beside the other platform attributes. Not exercised as root on this Windows machine.
The from-case draft in Nothing_project_specific_reaches_the_registry uses an
absolute path, which the harness refuses at verify, so the genericity check
over case arguments (ToolGenericity.cs:96) was never what stopped it. A fifth
draft, from-case-name, passes the harness with {tmp}/alpha/cache and must be
refused at promote with "carries a project".
Mutation-checked: deleting ToolGenericity.cs:96 fails the test with
"Expected promoted.Failed to be True because from-case-name carries a
project, but found False."
implementer-1's branch at d6b247f, accepted by the run's reviewer and verified by its verifier on 0.44.0: 0 warnings, 3229 passed, 0 failed at the time. One conflict, in RoleLibraryTests.cs: main added the theory replaying commands the roles must be allowed, the branch added the tool-creator and tool-refiner promotion test, at the same place. Both are kept. Merged onto main as it now stands, the suite is 3308 passed, 0 failed, 22 skipped, with 0 warnings.
Both added a test at the same place in RoleLibraryTests.cs, and both are kept. 3314 passed, 0 failed, 22 skipped, 0 warnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The work of team run
20260923-1216-be60, whose goal was a global Tool Creator and Tool Refiner shared by every team. This is implementer-1's branch atd6b247f(15 commits), merged onto today's main.What it adds
src/Loadout.Core/Tools.loadout tools:search,show,submit,used,audit,verify,promote,deprecate,retireandtrust.loadout_tools_search,show,submitandused. Promote, trust and deprecate are deliberately not exposed over MCP.role.tool-creatorandrole.tool-refinerroles, and atool-worksteam. Neither role may promote a tool or trust one.Evidence
d0d5a9fandd6b247f, and its verifier verifiedd6b247f. The verifier ran it itself: 0 warnings, and the full suite at the time passed 3229 with 0 failures.RoleLibraryTests.cs. Main and the branch each added a test in the same place, and both are kept.What the run says is not finished
23 of its 26 criteria are met. Three are only partly met:
GlobalToolLifecycleTests). It needs a livetool-worksrun once this is merged and installed.Caveats the lead recorded as open tasks
tools trustnorteam remedy trustchecks that a person is running it. The roles deny both commands to agents, but the commands don't check for a person themselves. The lead ranked this the most important follow-up.ToolNominator.cs:165:Runs()treats(or;inside a comment as the start of a command.