Finish the shared tool catalogue, and let a person agree to a draft's test run - #113
Merged
Merged
Conversation
added 8 commits
September 24, 2026 19:33
Three new rules for ToolNominator, each filed through IToolRegistry.Nominate with the existing filed-key dedup, secret screen and active-tool hinting: - Rule 5, workflow: the longest ordered run of at least three passing command shapes (rule 3's shapes) that runs of two different teams share. Back-to-back repeats of one command collapse to a step, so a retry is not a workflow. Report documents are now read in the order they were written. - Rule 6, prompt: a paragraph of 30 or more words, folded for whitespace and case, found in the briefs or reports of runs of two teams. Filed as a candidate with capability 'prompt'. It carries the paragraph as first written, not folded, because a lower-cased credential gets past the secret screen; the mutation check showed exactly that. Only the run ids are recorded as origin, never the task around the paragraph. - Rule 7, integration: an MCP server (from mcp__<server>__ tool names) or HTTP host used by nodes of two teams. RunDocuments lists only briefs, reports, policies, questions and answers, so the source is both report evidence refs and the node streams (stream-*.jsonl) that sit beside them. Loadout's own server and loopback are left out, because every node uses them. tool-creator.md now covers the three kinds. A prompt tool is drafted as a script that prints the text with its placeholders filled from named inputs, because the harness runs scripts and checks output; a bare text file could not be run or given cases.
The review of 3217b25 found four ways the new workflow, prompt and integration rules nominated the wrong thing or did the work twice. Each has a test that fails without its fix. - Workflows: back-to-back repeats of one step (a retried build) made two workflows out of one. Steps are now collapsed before they are compared. Back_to_back_repeated_steps_collapse_into_one. - Prompts: the boilerplate the runner writes into every lead's brief was counted as a prompt two teams had both written. It is now excluded, and a shared paragraph naming a project path is kept out of the inbox. Runner_boilerplate_in_two_teams_lead_briefs_is_not_a_prompt, A_shared_paragraph_naming_a_project_path_does_not_reach_the_inbox. - Cost: every pass built a new ToolNominator, so a finished run's streams were read again on every schedule. ToolNominationPass now keeps one for its lifetime. A_second_pass_does_not_read_a_finished_runs_streams_again. - Integrations: an existing tool covered an MCP nomination if its script contained the server's name anywhere, so a script clearing a github runner's cache covered "mcp github". Coverage now needs the script to call mcp__<name>__, or to name an HTTP host whole. An_unrelated_script_mentioning_github_does_not_cover_an_mcp_github_nomination. Mutation checks: the MCP pattern replaced by a bare \bgithub\b fails the github test; 'at == 0' changed to 'at >= 0' in Collapsed fails the collapse test. Full suite: 3344 passed, 0 failed, 22 skipped.
ToolNominator keeps what it read of each finished run, by run id, so a long-lived pass does not re-read the same streams on every schedule. Nothing ever removed an entry, so a run that was forgotten, or pushed beyond Depth by newer runs, stayed in memory for as long as the daemon lived. Finished() now drops every cached run whose id was not among those _journal.List(Depth) returned this time. A_run_that_leaves_the_listing_is_evicted_from_the_cache reads two finished runs, deletes one folder, finds again, and expects one entry left. It failed before the change (expected 1, found 2). It observes the cache through a new internal Cached count. ToolNominationPass keeps one nominator for its life, and nothing pinned that. A_second_pass_does_not_read_a_finished_runs_streams_again rewrites a run's report between passes: a fresh pass loses the nomination, and the same pass still returns it, because its nominator does not read the run again. Mutation checks: replacing TryRemove with TryGetValue in the eviction loop failed the eviction test; building a new ToolNominator on every ToolNominationPass.BeforeAsync failed the pass test. Both were restored from copies. Correction to 78b6c8f: its message says "Steps are now collapsed before they are compared." They already were: collapsing came in 3217b25, and 78b6c8f added only its test, Back_to_back_repeated_steps_collapse_into_one. Suite: Passed 3346, Failed 0, Skipped 22, Total 3368.
Run 20260923-1216-be60's lead was refused loadout_tools_search with "Nothing in the role.project-lead role allows" it, although the brief every node is given tells it to search the catalogue before building. No role's front matter named the launcher's own catalogue tools, the same gap MemberTools closed for progress and task_declare. TeamRunner.AllowedFor now adds search and show for every role, and submit and used for every role except reviewer and verifier: those judge one piece of work and write nothing, so a lesson they find goes in their report for the lead to submit. The list goes to both the agent's --allowed-tools and the node policy the permission answerer reads, and a role's deny list still wins. Every_member_role_may_search_the_tool_catalogue iterates the real role files through NodePermissions.Decide.
… the Refiner The Refiner judged correctness and generality through the gate, but had nothing to measure usability, maintainability, performance or continued relevance with. ToolHealth computes all four per active tool from what the registry already holds: - performance: each case's time, now recorded at verify in the manifest's tests.seconds, with the median and the worst; - usability: failed and workaround rates over the last 20 uses, and distinct teams per use; - maintainability: non-blank script lines, inputs, dependencies, known-good versions and promotions in the last 90 days; - relevance: days since last use (or since promotion when never used), and a newer active tool that overlaps it. NeedsRefining now returns true when a threshold is crossed, even after two stand-downs: a rate of 0.25 or more over at least 8 uses, a case slower than 30 s, or 60 days idle while active. Each value is a named constant on ToolHealth with its reason. Exposed as 'loadout tools health [name] --json'; the Refiner role may run it and is told to read it first and state its gain against those figures.
61f8050 let any crossed health threshold bring a tool back to the Refiner however many times it had stood down. An idle tool crosses "idle" on every pass for ever, so it would be looked at on every pass and the two-stand-down stop rule (criterion 15) never applied to it. A stand-down now records, by name, the thresholds the tool had crossed when it was looked at (failed, workaround, slow, idle), in a new optional `crossed` field on the audit entry. After two stand-downs with no signal between them, NeedsRefining is true only if a threshold is crossed now that the last stand-down did not record. ToolHealth carries the same crossings by name in a new Thresholds list, because the sentences in Crossed carry numbers (days idle, rates) that move. Only a new name counts, not a bigger number. The rates and case times change only through uses and promotions, and the failed or workaround uses and the promotions start the count again on their own. Idleness grows every day whether or not anything has changed. A deliberate change in what a test expects: NeedsRefining_turns_true_on_a_crossed_threshold_even_after_stand_downs asserted the uncapped behaviour. It is now A_crossing_the_stand_downs_already_saw_does_not_override_them, the same setup asserting false. New: An_unchanged_crossing_after_two_stand_downs_is_not_refined_again (an idle tool, a day more idle on its third pass) and A_new_crossing_after_stand_downs_is_refined (it stood down twice while healthy, then went idle). ToolStoreFixture.Registry takes an optional clock. Also gives ChurnWindowDays the reason its neighbouring thresholds have.
A draft cannot be promoted until its cases have run, and they run only when this machine has agreed to that script and those cases: a TrustedRemedy named tool-test:<name>@<version> carrying their fingerprint. Nothing a person could type wrote one. tools trust takes only promoted versions, team remedy trust only remedies in a team's directory, and without an entry RemedyCeiling.Decide asks whatever team-remediation says. The first live tool-works run (20260924-1750-3b1c) drafted git-tree-clean and was held for a person who had no way to answer. GlobalToolLifecycleTests passed throughout, because ToolStoreFixture.Agreed writes the agreement itself. tools verify <draft> --agree shows the script and the cases it would run, with the fingerprint, and asks. On yes it records the agreement in the machine's configuration and verifies with ToolHarness.Answered: the person's answer settles a rule of ask for this run, never stays never, and the fingerprint still has to match, so changing the script or a case afterwards is held again. IToolRegistry.Agreement gives the name and fingerprint from the same read of the draft that VerifyAsync decides on. --agree is refused wherever the command cannot ask (piped input or output, --json, --non-interactive). That is a team node's shell: the creator and refiner roles may run tools verify, and without this they could agree to their own scripts. Nothing marks a node's process as a node today, so the terminal is the only evidence of a person this command has. A draft named relative to the catalogue's drafts is now found there rather than against the current directory, as the help always said; a path from the current directory is still taken when nothing under drafts has that name, and climbing out of drafts is still refused. A held verify also says how a person can answer. Tests: ToolAgreementTests (five, through the registry rather than the fixture), a contract test that --agree over pipes is refused and records nothing, and a safety test for the drafts-relative name. Mutation-checked: dropping the terminal guard, letting an answer overrule never, fingerprinting without the cases, and dropping the drafts-relative lookup each fail a test.
In system-watch run 20260924-1853-2a26 an investigator found git-tree-clean in the shared catalogue, ran it, and met every criterion, but never called loadout_tools_used, although the goal said to record the use and every brief carries the catalogue pointer that says so. The tool still read "Used 0 times" afterwards. A pointer in the brief was not enough; the role now says it as a rule. Unrecorded use is not harmless: the Refiner's health measure and retirement both read the usage records, so a tool that is used but never recorded looks idle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The rest of the shared tool catalogue from team run 20260923-1216-be60, and the one thing its first live trial found missing: a way for a person to agree to a draft's test run.
What it does
loadout_tools_searchand_showfor every member role;_submitand_usedfor every role except reviewer and verifier.loadout tools health: case times, recent failures and workarounds, size and churn, and whether anyone still uses a tool. A crossed threshold brings a tool back to the Refiner, but only when it is new since the last stand-down.loadout tools verify <draft> --agree. Shows the script and every case it would run, with the fingerprint of both, asks, records the agreement in the machine's configuration and runs the cases. It is refused anywhere it cannot ask (piped input or output,--json,--non-interactive), which is what a team node's shell looks like; the creator and refiner roles may runtools verifyand must not be able to agree to their own scripts. An answer settles a rule ofaskfor that run,neverstaysnever, and changing the script or a case afterwards is held again.loadout_tools_used.Why
--agreewas neededUntil now nothing a person could type wrote the agreement a draft's harness run needs (a
TrustedRemedynamedtool-test:<name>@<version>with its fingerprint):tools trusttakes only promoted versions andteam remedy trustonly remedies in a team's directory. Every verify on a real machine was held.GlobalToolLifecycleTestspassed throughout becauseToolStoreFixture.Agreedwrites that agreement itself.Evidence
A live trial on this machine, with real agents:
git-tree-cleanfrom one of them (inbox/2026-09-24-53fb0737), stood the other six down with reasons, and was held for a person.--agree(every case passed) and promoted asgit-tree-clean@1.0, owner tool-works.Tests:
ToolAgreementTests(five, through the registry rather than the fixture), a contract test that--agreeover pipes is refused and records nothing, and a safety test for the drafts-relative name.never, fingerprinting without the cases, and dropping the drafts-relative lookup each fail a test.Limits
tests.ranAtas0001-01-01. Tracked separately.--agreerelies on the terminal instead;tools trustandteam remedy truststill have no such guard, which is the open task on write scope and trust.