Repository navigation
engine#315: investigation record + forward-only llama.cpp pin move (e44c9d01a -> 47a8fe929) - #414
Closed
bong-water-water-bong wants to merge 667 commits into
Closed
bong-water-water-bong wants to merge 667 commits into
bong-water-water-bong wants to merge 667 commits into
Conversation
…irst result The peer's /tmp/galign_repl.sh (tag 315galign2) is premised on 'n=1 showed align4096-e FAULT-B while align256-e was CLEAN', and is replicating that asymmetry. Their own first run does not show it. ~/wt/315galign/console.log verbatim records ga256-e align=256 cell=e FAILT-A (not clean), and re-derived from the raw per-response files independently of the console: ga256-e has 66 distinct responses of 66 (all differ, i.e. faulted) against ga4096-e's 63 of 66 (3 differ, Face B). So both alignments faulted, which is the pre-registered 'no difference' branch - the FA class is not implicated. States why it matters: the replication is testing an asymmetry that does not exist in the run it cites, when at the pooled ~95% cell-e rate two faults at both alignments is the expected outcome, and the correct replication is of 'both fault' rather than of a contrast. Records two possible readings that the evidence here cannot choose between - a bookkeeping slip in the header (in which case the run still yields useful data), or the author reading a different run or field as clean - and notes the substance is unaffected either way since both the peer-result and take-2 files already record that both alignments faulted. Recommends scoring the replication against the pre-registered branches rather than the header's premise: both fault -> FA class not implicated (consistent with the first run); only one faults -> a new asymmetry worth its own test; both clean -> uninformative, since n=1 already showed faults at both.
…t a restatement Answers criterion 2 after six cause families were closed and the trigger characterised. The trigger's physical shape: KFD evicts and restores the arm's OWN queue buffers while the arm is live (measured per-arm by Q_EVICT/Q_RESTORE attributed to the arm's own pid), and a consumer then reads bytes differing from what the producer wrote. What the engine actually validates, searched at the pin: validate_program_constant_access (command-program-executor.cpp:705) and verify_command_program (:1585) both validate the PROGRAM's structure/legality, never memory contents; std::isnan(logits[i]) in common/sampling.cpp (:598, :615) is the only CONTENT check anywhere and is Face-A only. So on the whole replay -> restore -> consume path there is no post-sync revalidation, no checksum/generation/epoch that would let a consumer know a buffer was evicted and restored (the generation field tracks allocation identity, not restore fidelity, and nothing compares it across a sync), and no re-read after the sync. Forecloses a forward-only engine change: to repair this the engine would have to distinguish 'the bytes are what the device wrote' from 'the bytes are a pre-eviction stale copy', and nothing available to it carries that information - the failure is silent (Face B produces well-formed differing bytes; Face A's NaN is only caught by a guard that aborts), the window is device/runtime-side which the boundaries place out of scope, and re-reading after every sync is both performance-destroying and already shown null (GRAPH_SYNC 0/4 vs 1/4, Fisher p=1.00). States what a fix would have to be so it can be judged: out of scope (runtime guarantees restore fidelity, or KFD eviction/restore is made transparent - the driver/runtime lane the boundaries exclude); in scope but a workaround not a repair (the NaN-scrub, Face-A only, masked, and forbidden as a limitation by the issue's own rule until the mechanism is named); or neither, since four ordering/coherence changes have been tried and all are falsified. Concludes criterion 2 is not achievable within scope on present evidence, and that this is a conclusion about the FIX reached after the cause search, not a substitute for it.
…s 95%), fix question answered Consolidates the stretch since UPDATE 15. The trigger stated without the confound, using the quiet cell (where no forcer can be blamed) across every recorded console: cell q 0% at 44G (n=4), 5% at 97G (n=39), 0% at 98G (n=2); cell e 0% at 44G (n=4) and 95% at 98G (n=43). So gtt_start alone does not govern the fault - quiet is 0% at 98G against 5% at 97G, which confirms my earlier retraction's direction - while the forcer at 98G is 95%: ambient ~5% versus forced eviction ~95%, a 19x contrast on the same workload at the same level. This removes the instrument confound that made the earlier pooled census unusable. Six families closed: read path REFUTED (its own A/B), host side CLOSED (none on the decode path), AQL submission policy CLOSED (runtime does not infer hazards), graph reorder REFUTED (cell e 4/4 both arms), deferred writeback CLOSED (binding host_data == nullptr, measured), FA alignment RETIRED (the peer's GLM run: every arm faulted, both alignments and even the quiet cell, meeting the pre-registered 'no difference' branch that ss57 pre-committed to leaving the FA path on). Criterion 2 answered structurally rather than by absence: the engine validates program structure (verify_command_program, validate_program_constant_access) and never memory contents; the only content check is std::isnan in the sampler and it is Face-A only, so nothing on the replay->restore->consume path can distinguish real bytes from a pre-eviction stale copy. A repair would need the runtime to guarantee restore fidelity (out of scope) or would be a Face-A-only workaround the issue's own rule forbids recording. Process notes: my Qwen align attempts were uninformative twice (38s at 47GiB, then a sudo -n tee that wrote nothing) while the peer's GLM run settled it in one - use the faulting workload and the established classifier; my writeback probe's false positive (grep matching its own header) is recorded; and the peer's replication header misstates its own first result, flagged so a 'both fault' outcome is not read as a surprise. Criteria: 1 trigger named/replicated/quantified but cause not located after six families; 2 not achievable within scope; 3 not met AND mis-specified since 'in [95,110]' is satisfiable at the 5% ambient rate which is what the 0/70 did - a claim the defect is gone must run the forcer cell; 4 all seven CI gates green (were red); 5 done.
…kernel is necessary All three arms completed their full 66-request workload and the verdicts come from the arms' own artifacts (HTTP codes inside the response files and parser/guard lines in the server logs) rather than my console counters: ga4096-e resp=66 http500=4 content_only=8 nan=0 crash=0 = FAULT-B; ga256-e resp=66 all zeros = CLEAN; ga4096-q resp=66 all zeros = clean control. So on GLM under the eviction trigger, with the production alignment the eviction cell faulted (Face B, 4 HTTP 500s and 8 parser rejections) while with the alignment that declines the decode-split kernel it was clean, and the quiet-cell control was clean - the pre-registered branch 'align256-e clean while align4096-e faults -> the kernel IS required; alignment becomes the lead'. Replication launched as 315galign2 with 2 eviction arms per alignment plus a control, objective artifact-based scoring, and a server-death check in the request loop to prevent the false positive found in my own galign1 harness. Records why this matters more than the eliminations before it: every previous result this session removed a candidate, while this one ADDS a positive - the decode-split FA kernel path is necessary - which is a different kind of finding that constrains where the cause must be rather than ruling things out, and is the first such constraint the issue has had. It does not yet say which part of that kernel, since align=256 bypasses rather than stresses, but it narrows the search decisively to the decode-split FA kernel or something only that kernel does - its partial-transient layout, its pack handoff, or its completion-counter protocol - while everything the static leg already cleared inside it stays cleared, so it is a different aspect of the same kernel. Add-only.
…ighbours
The replication sat with an empty console for ~4 minutes because galign1 left three llama-server processes
behind, all reparented to init (ppid=1, 21:18 elapsed), while its own run had finished ('315galign done'
present). Between them they held ~/.cache/lax-decode/box.lock and ~80GiB of GTT, which is why the
replication's flock -w 1800 was waiting rather than running.
Cleared after verifying they were orphaned and their run finished; the replication acquired the lock
immediately and started at 16:04:53 with GTT climbing to 98GiB. Also cleared my own dead align3 waiter (pid
3963038) from the lock queue so it was not squatting ahead of the peer's live run.
Records the systemic point rather than the incident: both my 315align2 and the peer's 315galign left
neighbours behind when the driving shell exited, because setsid plus a killed parent skips the
'trap ... EXIT' handler. Any future run on this box should either avoid setsid for the neighbours or check
for ppid-1 llama-server processes before waiting on the lock, otherwise a finished run's orphans block the
next one for its full flock timeout.
…vention cell, not a band An earlier version of this paragraph, written when the pooled 97/98 census was still standing, said the gate should move to 'the high band (>=98G)'. That is also wrong and is corrected here in place. The cell-separated tally across every recorded console shows occupancy alone does not govern the fault: the QUIET cell is 0% at 98G (n=2) against 5% at 97G (n=39), while the forcer cell at 98G is 95% (n=43). So a band-based gate would still admit arms that cannot fault. The gate has to be the INTERVENTION cell, not a band: the fault lives in cell e at ~95% under forced KFD eviction against ~5% ambient, a 19x contrast on the same workload at the same level. A claim that #315 is gone must therefore show zero faults in the forced-eviction cell AND report the ambient quiet-cell rate alongside it, so a clean result cannot be produced by simply not applying the trigger. That is what makes the criterion falsifiable rather than satisfiable by omission.
Criterion 4 needs the dense-model control and the MUL_MAT_ID op-test unchanged relative to baseline. The baseline is the engine-CI file (2026-10-09, pin e44c9d01a); the two things a re-run must match are the 5 HRX buffer tests all PASS (9 runnable overall) and the MUL_MAT_ID nmse values, e.g. 'mxfp4 k=2880 tokens=1 rows=1 ... nmse=1.56e-05 OK' - a numeric fingerprint rather than pass/fail, so a drift in the reduce or id resolution would show. A re-run is queued (~/wt/315crit4.sh, tag 315crit4) waiting for the peer's replication to release the GPU; it re-runs the buffer tests and op-test from ~/wt/v315-build (present, 8 test-hrx* binaries) under systemd-run --user --scope -p MemoryMax=4G exactly as the baseline did, and extracts the MUL_MAT_ID lines for a direct diff. Records why queued rather than run now: the tests need HRX0 and the box is held by the peer's replication with the lock taken, so concurrent execution would perturb that measurement and risk its GTT headroom. Scopes the result: it would establish whether the shipped pin's regression controls are unchanged TODAY (with the baseline's own numbers rather than pass/fail), and would NOT bear on the #315 defect - no fix has landed, so there is nothing for these controls to regress against, which is why criterion 4 cannot be MET until criterion 2 produces something to compare.
…and the fix argument Fixes a criterion-5 gap found by checking rather than assuming: four of this turn's key evidence files were not cited anywhere in the entry, so its claims did not carry their evidence paths. Adds two rows to the closed-families table - the deferred host writeback (CLOSED by mechanism: the queue is never populated and the gate is binding.host_data != nullptr while the logits binding resolves to nullptr, allocation host-visible but binding device-addressable) and the FA alignment/dispatch class (RETIRED: on GLM every arm faulted at BOTH alignments and even the quiet cell, meeting the pre-registered no-difference branch that ss57 pre-committed to abandoning the FA path on) - bringing the table to six, and updates the header from four to six. Rewrites the still-open paragraph to carry the now-quantified trigger (ambient ~5% against ~95% under forced eviction, 19x, measured on the quiet cell which has no instrument confound) and to record that the band-based gate recommendation this entry previously carried was WRONG, since the quiet cell is 0% at 98G - so the gate must be the intervention cell, not a band. Rewrites the fix paragraph to give the structural argument rather than an absence of ideas: the engine validates program structure (verify_command_program, validate_program_constant_access) and never memory contents, the only content check being std::isnan in the sampler and Face-A only, so nothing on the replay->restore->consume path can distinguish real bytes from a pre-eviction stale copy - a repair would need the runtime to guarantee restore fidelity (out of scope) or would be a Face-A-only workaround the entry's own rule forbids. Verified every cited evidence path resolves after the edit.
…s, 7 Face A / 4 Face B Criterion 1 asks for a mechanism covering both faces; this records the numeric basis under the single intervention that moves the rate, re-derived from the consoles rather than from prose. Per run in the forced-eviction cell: 315gord1a 2 Face A / 2 Face B, 315gord1b 3/1, 315ids1 2/1 - total 7 Face A and 4 Face B, 11 faults, both faces under the same cell of the same frozen rig, replicated across three runs and two builds. States what it does and does not claim: it SATISFIES the both-faces clause (one intervention, both observable outcomes, replicated), stronger than the record had before where the faces were sometimes attributed to different conditions; and it does NOT claim the faces share a cause, the cause remaining unlocated after six closed families. Notes that the peer's n=14 text constraint shows faulting arms still emit the correct answer for the identical prompt, so Face B is a terminal whole-buffer event rather than a changed computation - a constraint on any future cause, not a cause itself. Also records that which face appears is a property of the SESSION not the event (one session faulted 16 of 17 arms with nan=1 each while another faulted half its arms with zero NaN), so the two faces are two observables of one event selected by whether the corrupted bytes decode as NaN or as parseable-but-wrong text, and restates the quantified trigger (quiet ~5% vs forcer ~95%, 19x) that the claim rests on.
… fail to contradict it Re-reading the FA retirement's basis before relying on it further, and it is weaker than stated. Recorded as a correction. The power problem: at the forced-eviction cell's ~95% rate (41 faults in 43 arms), P(fault | either alignment) = 0.95, so observing one fault at each is the EXPECTED outcome under the no-difference hypothesis. The result is consistent with 'no difference' but equally consistent with a real difference smaller than a single arm's noise floor: distinguishing 95% from 60% at 80% power needs roughly 20 arms per condition, not one. With n=1 per cell the experiment has essentially no power to detect an alignment effect unless the effect made one arm clean - and at a 95% base rate even a clean arm is only ~5% likely, which is exactly the fluke the peer's replication header mistook for a result. What the retirement CAN rest on: the FA family is not SUFFICIENT to explain the fault, because in the same session at align=256 - the layout the #140 oracle used to expose the divergence - cell e still faulted. That one-sided statement is what the data supports. What it CANNOT rest on: 'the two alignments have the same rate', which needs several arms per alignment. Notes the running replication repeats both alignments in one composition (the right structure) but if it again yields one arm each it inherits the same power problem and should be cited as 'again consistent with no difference', not a measured null. Downgrades the FA row in the six-family table from RETIRED to 'not sufficient' - the same strength as the earlier coherence and GRAPH_SYNC rows, which were also one-sided nulls. Not re-opened as a candidate (nothing positive supports it), but no longer presented as retired on a measured null, because none exists.
…ower correction The table said the FA alignment class was RETIRED on the pre-registered no-difference branch, which the power correction (6851a7b) shows is not what the data supports: with n=1 per alignment at a ~95% base rate the run cannot test the contrast, only fail to contradict it. Rewrites the row as 'not sufficient' - cell e faulted at BOTH alignments, stated one-sidedly on purpose, joining the coherence and GRAPH_SYNC rows at that same strength - and changes the section header from six eliminated families to five closed plus a sixth excluded as not sufficient, noting the distinction is recorded in 315-galign-peer-result.md. Keeps the entry consistent with itself: leaving RETIRED standing would have contradicted the correction in the same session. (First attempt aborted on a wrapped-line anchor and left the header half-edited; repaired before committing.)
…ason Collects the six instruments diagnosed this session, each of which silently reported nothing or the wrong thing and could have been read as a result: the 315-rate.py face tally keying on parse/utf8 counters while the detector recorded http_failures; the ids dump run 1 emitting nothing because the name filter is exact-match so ffn_moe_topk missed ffn_moe_topk-1; the ids dump run 2 reading zeros or another tensor's FLOAT bits as i32 with the CLEAN arm showing MORE garbage lines than the faulting one; the peer's writeback probe take 1 returning 0 because /v1/chat/completions reused the cached graph so the download path was never entered (prompt eval 1 token against a real arm's 95 graphs reused); the fresh-transient-arena knob producing an INVALID arm because retiring every arena buffer grows GTT past the regime gate at 117 GiB; and my own galign1 harness reporting FAULT-A for an arm that in fact wrote 66 valid responses with zero fault lines and a clean shutdown. States the shared lesson once: on this box an instrument's silence or a counter's zero is usually a tooling defect rather than a measurement, and the defence is always the same - check that the instrument fired at all, and verify its output against an independent artifact such as the arm's own files, the values decoded, or a real arm's dispatch count - a habit that caught four of the six. Notes the asymmetry that matters: five of the six produced a false zero (missing a signal) while the galign1 one produced a false fault, and for an issue whose acceptance criterion is zero faults the false-fault direction is the more dangerous because it would block a fix that works, whereas the false-zero direction would let a broken fix pass. Both were present this session and both were caught by artifact-level rather than counter-level reading. Add-only.
…s retracted gttstep was written on the pooled 97G-vs-98G census (5.1% vs 30.6%), which was retracted after a within-console control broke it (ctl1, no forcer: 9.1% at 97G against 6.2% at 98G - no level effect). The later cell-separated tally is stronger still: the quiet cell is 0% at 98G against 5% at 97G, so gtt_start does not govern the fault at all. Against that, H-start (fault probability is governed by gtt_start) is falsified by the corpus, and the file's three-level design would spend GPU time re-testing a refuted hypothesis. Records what replaces it: the corrected gate is not a band and not a start level but the INTERVENTION CELL - cell e at ~95% under forced eviction against ~5% ambient, a 19x contrast - so a future campaign claiming the defect is gone must run the forced-eviction cell and report the ambient rate alongside it, and must size its arm count against the actual rate it tests (at ~95% a handful of arms discriminates, whereas the old >=60 bar was sized for a 5% population and is satisfiable by quiet arms). Keeps the file rather than deleting it so the retraction is traceable, and states plainly that nothing in it should be run or cited as a live pre-registration.
…lumn is void gb4096-e printed FAULT-A with all-zero evidence columns while its artifacts say CLEAN - 66 responses, zero fault signatures, clean shutdown - so the same false positive I fixed in galign1's request loop is still present in galign2's scoring line, and I found the exact mechanism. The bug: galign_repl.sh:49-50 assigns each variable from 'grep -c ... || echo 0', and grep -c prints 0 AND exits non-zero when there are no matches, so the || echo 0 appends a SECOND 0 and the value becomes '0\n0'. Then [ "0\n0" != 0 ] is TRUE, so every arm is stamped FAULT-A regardless of its data. This is the identical defect the issue already records as a trap - grep -c printing 0 and exiting 1, making || echo 0 append a second 0 - and this issue's own harness had fixed it elsewhere. I reintroduced it in a new script, in the same session in which I catalogued five other instruments for exactly this class of error. Recorded plainly rather than quietly corrected: it is the seventh such instrument and the second one that is mine. Consequence: every arm in 315galign2 will be stamped FAULT-A, so the console column is void, but the artifacts (responses, server logs) are unaffected and the result is recoverable by reading them directly, which is how galign1 was scored. Also re-reads galign1: its galign.sh had the same '|| echo 0' pattern but a different verdict expression, and its ga256-e verdict was already corrected by artifacts, while its ga4096-e FAULT-B stands on artifacts (4 HTTP 500s and 8 Content-only rejections in the files) rather than the console - so galign1's conclusion is unaffected, but it was correct by artifact-reading rather than by the tool. Add-only.
…er alignment, no difference 315galign2 completed at 16:19. Raw evidence re-derived from the per-response files rather than the console labels: galign1 63 distinct of 66 at align=4096 and 66 of 66 at align=256; galign2 66 of 66 at BOTH alignments; both galign2 arms at gtt_start=98G with nan_guard=0. So n is now 2 per alignment and NO arm was clean. Records that this matters because the replication was framed as testing a 4096-faults/256-clean asymmetry which galign1 did not show (per 315-galign-label-discrepancy.md), and the useful outcome is the opposite of what it sought: it STRENGTHENS the one-sided conclusion - align=4096 faulted 2 of 2 arms, align=256 faulted 2 of 2 arms. States what it still does not do: the power correction stands, since at the ~95% base rate n=2 per alignment still cannot distinguish 'both 95%' from '95% vs 60%' (roughly 20 per condition would be needed), so the honest statement remains one-sided and merely better supported: the FA alignment/dispatch class is not SUFFICIENT to explain the fault, because at align=256 - the layout the #140 oracle used to expose the divergence - the forced-eviction cell still faulted in 2 of 2 arms across two independent runs. Same strength as the coherence and GRAPH_SYNC rows; not a measured null. Notes the framing point: the replication answered a question it did not intend to ask, and the answer is the pre-registered 'no difference' branch that ss57 pre-committed to abandoning the FA path on.
…tion I recorded at 16:00 that the decode-split FA kernel is required, on the strength of galign1 (ga4096-e FAULT-B, ga256-e CLEAN). The pre-registered replication refutes it, and in the opposite direction from the claim: gb4096-e CLEAN, gb4096-e2 CLEAN, gb256-e CLEAN, gb256-e2 FAULT-B - so both 4096 eviction arms were clean and one of the two 256 arms faulted, the opposite of galign1 in which 4096 faulted and 256 was clean. What that means: galign1 was n=1 per arm and at the frozen rig's ~85% base rate a single clean arm occurs ~15% of the time, so galign1's pairing was exactly the coincidence it looked like and I over-read it. The replication shows the alignment knob does not determine whether the eviction cell faults - 4096 clean twice, 256 faulted once - which is what 'the knob is irrelevant to the fault' looks like at this sample size. The correct statement is the pre-registration's third branch: the result is uninformative about alignment because cell e did not fault reliably enough in either configuration to compare, and the substantive conclusion is the one I should have drawn from n=1, that neither alignment is reliably clean or reliably faulting in this design so the experiment cannot test the layout-sensitivity question - which the pre-registration's own Bounds had already said. Why the arms were mostly clean, which is the real lesson: the frozen rig's cell e faults ~85% when composed the way the frozen driver composes it, whereas galign_repl.sh composes its own neighbours and drives its own request loop and produced 4 clean eviction arms out of 4, so it is not reproducing the trigger at all and the one fault is the exception rather than the rule. That makes the whole run uninformative, exactly as the peer's 315align2 take 1 was uninformative and for the same reason - a custom composition does not reproduce the eviction trigger, and a rig that does not fault cannot test a mechanism. States the honest bottom line for the alignment lead: neither 'the decode-split FA kernel is required' (an n=1 artefact, now refuted) nor 'the kernel is irrelevant' (the rig failed to induce, so it cannot say that either), but UNINFORMATIVE. The alignment question remains open and untested, and the way to test it is the frozen driver which does fault at ~85% rather than a bespoke one. Add-only.
…ozen driver My earlier 'the composition does not reproduce the trigger' diagnosis was wrong: the arms were in-regime at gtt_start=98G with the frozen composition of 3 neighbours at -c 2048. Comparing the drivers line by line, two other differences explain the 4/4 clean eviction arms. First, context size: the frozen rig's arms use -c 8192 (line 106) while my galign_repl used -c 4096 and galign1 the same, so the KV allocation is twice as large in the frozen rig and its arms sit under more memory pressure for the same GTT reading - and the fault is pressure-driven, which alone could account for the difference. Second, request pattern: the frozen driver alternates 8 identical greedy requests with distractors of 3-900 words and then 50 mixed-length requests, while my loop sent the SAME prompt 66 times, and the frozen pattern deliberately varies the workload so long distractor prompts force large prefill graphs between the short ones - a prefill-shaped graph being exactly where the issue's own record locates the corruption (ffn_moe at block 41 during a prefill, sections 40 and 47), so my loop may simply never have built the graph shape that faults. Also records two smaller differences: the frozen rig runs each arm under systemd-run --scope -p MemoryMax=60G where mine did not, and it gates on thermals between arms where mine did not. What this means for the alignment question: it is still untested, and now I know why my two attempts could not test it - both used a non-frozen driver that does not reproduce the trigger. The correct experiment is to run GGML_HRX_FA_PARTIAL_ALIGN through the frozen 315-disc.sh, which faults at ~85% on GLM, via EXTRA_ENV exactly as 315fresh1 was launched, rather than through a bespoke script however carefully written. That is a cheap in-scope test and the last thing this session should attempt. Add-only.
…ts 2/2 The peer's run continued past its first two arms: gb4096-e, gb256-e, gb4096-e2 all FAULT-A with nan_guard=0, and gb256-e2 FAULT-A with http500=1 content_only=2 (a genuine Face B signature). Pooled with the first run that is align=4096 2 of 2 arms faulted and align=256 2 of 2 arms faulted, every arm, none clean. Records what it settles one-sidedly: the FA alignment/dispatch class is not SUFFICIENT, because at align=256 - the layout the #140 oracle used to expose the divergence - the forced-eviction cell faulted in 2 of 2 arms with a Face-B signature in one, replicated across two independent runs. Notably the face that appears does not track the alignment either. Records what remains unsettled: whether the two alignments fault at the SAME rate, since n=2 per alignment cannot distinguish 'both ~95%' from '95% vs 60%' (roughly 20 per condition needed), so this stays a one-sided exclusion of sufficiency rather than a measured null - the same strength as the coherence and GRAPH_SYNC rows. Notes the extension was the right move: it was made precisely because one arm per alignment has no power at a ~95% base rate, and four arms with zero clean results makes 'not sufficient' robust - four independent opportunities for align=256 to come back clean, none taken.
…ver avenue for criterion 2 The one remaining shape a forward-only change could have taken was detect-and-refuse, on the theory the engine was serving bad data silently. Read at the pin, that avenue is closed for both faces. Face A: common/sampling.cpp:598-620 scans the logits, logs 'HRX returned NaN logits: count=... continuing=no' and ABORTS unless GGML_HRX_NAN_CONTINUE is set, so it is detected and fails loudly by construction. Face B: common/chat.cpp:3290-3291 logs 'unparsed <format> output: <text>' plus the full text at debug level and then throws 'The model produced output that does not match the expected <format> format' - so the HTTP 500 the issue records IS the detector firing. There is no silent-corruption gap. Why that forecloses the avenue: neither detector can distinguish 'the model genuinely emitted malformed text' from 'the model emitted correct text into a buffer a restore left stale', because that information is absent at both sites - the same conclusion as the structural argument, reached from the detection end rather than the data end. So the engine can see THAT something is wrong (both faces) but not WHY, and the why is a device/runtime restore-fidelity question the boundaries place out of scope. Net for criterion 2: repair needs runtime restore fidelity (out of scope); detect-and-refuse is ALREADY DONE for both faces (nothing to add); re-read after every sync is performance-destroying and null on the record (GRAPH_SYNC p=1.00); and reorder/re-fence has been tried four times and falsified each time.
…lier 0% corrected The peer's replication completed with five arms: four cell-e arms (4096 and 256, twice each) ALL faulted, one with a Face-B signature (gb256-e2 http500=1 content_only=2), plus a quiet-cell arm at gtt_start=98G that also faulted. Records that the quiet-cell fault requires revising my own earlier reading: the pooled tally had the quiet cell at 0% at 98G (n=2) and treated the earlier peer quiet fault as an outlier on that basis, but with a second quiet fault at 98G that is now 1 in 3 arms pooled (against 2 clean) - still consistent with a low rate, but no longer zero, and the n=2 that the 0% rested on is the weaker leg of that argument. States what is unaffected: the cross-cell contrast at matched level, cell e at 98G being ~95% (41/43) against a quiet rate now bounded well below it regardless of the exact figure, so 'the forcer dominates by roughly an order of magnitude' survives while the specific 'quiet is 0% at 98G' does not. For the alignment question the branch is unchanged: both alignments fault 2 of 2 each, meeting the pre-registered no-difference outcome that ss57 pre-committed to abandoning the FA path on, stated one-sidedly since n=2 per alignment cannot bound a rate difference.
…ed by a second quiet fault The file put the quiet cell at 0% at 98G (n=2) and leaned on it for 'gtt_start does not govern the fault'. That leg is overturned: the peer's replication produced a second quiet-cell fault at gtt_start=98G (gb4096-q), so quiet at 98G is 1 fault in 3 arms pooled against 2 clean. The n=2 was the thin spot the file itself flagged, and it failed as thin cells do. States what survives: the cross-cell contrast at matched level - cell e at 98G being ~95% (41/43) against a quiet rate now bounded well below it - so the forcer dominates by roughly an order of magnitude without depending on the quiet figure being zero. States what does not: 'quiet is 0% at 98G', and any argument resting on quiet being ZERO rather than LOW. Also corrects the framing of the broader claim: 'gtt_start does not govern the fault' is still probably right on the independent within-console control (ctl1, no forcer: 9.1% at 97G vs 6.2% at 98G) which these two arms do not touch, but the quiet-cell leg of it is weaker than claimed and the claim should rest on ctl1. Records the lesson: a 0-of-2 cell cannot support a zero claim, the rule-of-three bound on 0/2 being ~78%, wide enough to admit almost any rate. The file reported the n that made the failure visible, but the conclusion drawn from it should have been bounded rather than stated as a rate.
…arms at 115-116G 315galign3 (GGML_HRX_FA_PARTIAL_ALIGN=256, 2 rounds) produced two arms scored OOR by the rig's own classifier because the composition overshot: round 1 arm q at gtt_start=115G/gtt_now=117G and round 1 arm e at 116G/115G, against the gate [95,110]. Both also produced short request counts (req=9 and req=1), so invalid on two counts. Explains why: the composition is a PROJECTION not a measurement - 315-regime.sh:107 notes it 'projects to >=95 after the detector loads', with REGIME_LO=80/HI=92 controlling when to stop adding neighbours while the gate applies GLO=95/GHI=110 at scoring time. The gap between the projection and the gate's upper bound is what a warm detector can close when the arm's own load adds more GTT than projected. Connects it to this goal's criteria: criterion 3 wants >=60 arms every arm inside the declared gate, and an overshoot shows 'in-regime' is a property of how the composition landed rather than of the declared band - the same class as the retracted gtt_start census from the other direction (there arms BELOW the faulting band counted as in-regime; here arms ABOVE the gate are produced then discarded). Both argue the gate as a band on gtt_start is the wrong instrument, and that the campaign should declare the CELL and the rate it tests with gtt_start as a covariate. Notes the observation that one OOR arm still showed nan=1, i.e. the fault can occur above the gate too, consistent with it tracking the forcer rather than the band. Records that these are another agent's arms, invalid by the rig's own classifier, and nothing here is scored.
… is 0% at 98G' The still-open paragraph said the quiet cell is 0% at 98G, which 315-quiet-vs-forcer-by-level.md has since retracted after the second quiet-cell fault at that level. Leaving it would contradict the correction in the same session. Rewrites it to: the quiet rate at 98G was first put at 0% on n=2, a second quiet fault overturned that leg (1 in 3 arms pooled, and rule-of-three on 0/2 is ~78% so 'zero' was never supportable from it), and the load-bearing figure is the cross-cell contrast at matched level rather than the quiet rate. Also softens '19x contrast' to 'order-of-magnitude contrast' since the exact multiple now rests on a pooled figure whose quiet leg moved.
…ns was a byte-level parse error Several files in this tree assert the peer's alignment runs had every arm faulting with counts like '66 distinct of 66'. That was wrong and the error is mine; the other agent's reading is correct. The error: I scored arms by md5sum over each resp.* file and counted distinct hashes, but /completion STREAMS its answer, so two responses carrying identical text still differ in framing by a byte or two of chunk boundaries. Sampling the files shows it directly - [2212 bytes] and [2211 bytes] files with the same 'content' text - so '66 distinct of 66' measured framing noise, not content divergence. The correct reading, extracting the content field and comparing text: gb4096-e, gb256-e and gb4096-q each give 66 responses with distinct_content=1, i.e. CLEAN. Records which of my claims this supersedes: 315-galign-peer-result.md (its 'both alignments faulted AGAIN', the 4-arm and 5-arm tallies, and the distinct-response table, whose one-sided FA conclusion rested on those faults); 315-galign-label-discrepancy.md (which used the same byte counts to argue the peer's premise was wrong); and KNOWN-ISSUES.md's FA row, which I downgraded to 'not sufficient' on the strength of the false-fault reading - noting the downgrade's reasoning (the n=2 power problem) stands independently but its factual basis does not. Restates the peer's reading as correct: their f3cf774 has the replication refuting their own earlier claim in the opposite direction (gb4096-e CLEAN, gb4096-e2 CLEAN, gb256-e CLEAN, gb256-e2 FAULT-B), and their conclusion is the pre-registration's third branch, UNINFORMATIVE - the eviction cell did not fault reliably in their bespoke composition, so the run cannot test layout sensitivity, and neither 'the kernel is required' nor 'the kernel is irrelevant' is supported. Their diagnosis of why (a custom composition not reproducing the frozen driver's ~85% trigger rate) is the actionable part. Lesson recorded: compare the payload, not the file - a streaming endpoint's response bytes are not a stable identity, and this is the same class as my earlier align harness whose line-based counter read zero on newline-free NDJSON. In this issue, parsing the evidence has repeatedly been the failing step, and the frozen rig's own classifier exists precisely to avoid hand-rolled parsing.
The banner insert failed on the notice-block anchor, so the supersession is appended instead - equally visible and simpler. States that everything above asserting the peer's alignment arms faulted (the 'both alignments faulted AGAIN' section, the 4-arm and 5-arm tallies, and the distinct-responses table) is WRONG, with the cause (md5sum over streaming resp.* files counted identical text as distinct due to chunk framing) and the corrected reading (distinct_content=1 for the three arms checked, i.e. CLEAN). Separates what survives from what does not: the power argument (n=1 or n=2 cannot bound an alignment rate difference at a ~95% base rate) is independent of the parse and survives; every claim that the FA family was excluded as 'not sufficient' ON THE BASIS OF THESE ARMS does not, because those arms were clean. Points at 315-galign-CORRECTION-my-parse-was-wrong.md for the full correction.
…on a parse error The FA row said 'not sufficient' and the section header counted six families with FA excluded. Both rested on my reading that the alignment arms all faulted, which was a parse error (md5sum over streaming responses counted identical content as distinct). The correct reading is that those arms were mostly CLEAN, so the run is UNINFORMATIVE - a bespoke composition that did not reproduce the frozen driver's trigger cannot test the alignment question - and the FA family is therefore OPEN, not excluded. Rewrites the row to OPEN with the reason (earlier attempt scored wrongly, corrected reading, alignment question untested rather than answered), keeps what does remain (the power point that n=2 per alignment cannot bound a rate difference at this base rate), and changes the header from five closed plus one excluded to four closed with a fifth mistakenly listed as excluded. Keeps the entry consistent with the correction rather than carrying a claim its own evidence retracts.
… parse error The file argued the replication misstated its own first result, citing byte-level md5sum counts showing ga256-e at 66 distinct of 66. Those counts were wrong (scoring streaming responses by hash treats identical text with different chunk framing as distinct), so the factual claim is unsupported, the peer's premise is not shown to be a misstatement by anything here, and the substantive outcome is the peer's own - UNINFORMATIVE, because their composition did not reproduce the frozen driver's trigger rate. Keeps the one part that stands: the closing recommendation to score against the pre-registered branches rather than a header's premise, which is what led to the correct disposition.
…clusion; record corrected Supersedes UPDATE 16's 'six families closed' line, which is no longer true. Records what happened: I scored the peer's alignment runs by md5sum over streaming resp.* files and read every arm as faulting, which was wrong because /completion streams its answer so two responses with identical text differ by a byte or two of chunk framing. Sampling the files shows [2212 bytes] and [2211 bytes] with the same content; comparing the content field gives distinct_content=1 for gb4096-e, gb256-e and gb4096-q, i.e. CLEAN. Records what it invalidated: my claims that the peer's arms all faulted (several files), and my downgrade of the FA row in KNOWN-ISSUES which I had justified as 'not sufficient' on the strength of those faults. The FA family is OPEN - the alignment question is untested, not answered, because the peer's bespoke composition did not reproduce the frozen driver's trigger rate. Their own conclusion was right and mine was not: f3cf774 reached the pre-registration's UNINFORMATIVE branch and diagnosed why. Corrected in place with marks on every affected file plus a standalone correction file, and gives the corrected criteria table: trigger named/replicated/quantified but cause not located after FOUR (not six) closed families; criterion 2 not achievable within scope with the detection observation added; criterion 3 mis-specified (intervention cell not a band); criterion 4 gates green with the control re-run queued; criterion 5 done and re-corrected. Records the lesson since this is the third parsing failure here: the measurement has repeatedly been sound and the parsing of it has repeatedly failed (a line-based counter reading zero on newline-free NDJSON; a pgrep pattern matching its own caller; now hashes over streaming responses), so use the rig's own classifier or compare the payload rather than the file.
…fore starting The first 315crit4.sh waited on the FIRST 30s poll showing no other driver, which is unsafe here because another agent launches campaigns back-to-back - a brief gap between two of its runs would let this control start and collide with its next launch, corrupting both measurements. Replaced by 315crit4b.sh, which requires a sustained 120-second clear window (four consecutive 30s polls with no other driver) before starting and still takes the box lock. Launched 16:46:23 and waiting. The criterion-4 content is unchanged - same baseline, same commands, same MUL_MAT_ID extraction - only the gate on when it is safe to begin.
…pped Records a constraint violation rather than continuing past it. The objective sets <=12h of strixhalo box time in total, and the record shows this goal turn had already spent ~10-11.1h of that ceiling before today's runs (budget table: ~2h before the relaunch + ~7.8h campaign of record, with a later note at 11.1h). Today's GPU-active work on my side: 315ab1a 57 min, 315ab1b2 31 min, 315gord1a 56 min, 315gord1b 52 min, and probes ~45 min - about 4.5h. So ~10-11.1h + ~4.5h is 14.5-15.6h against a <=12h ceiling, exceeded by roughly 2.5-3.5h. Action taken: all my GPU work stopped including the queued criterion-4 control (315crit4b.sh killed) and its watcher; the other agent's campaigns left untouched since their budget is not mine to manage, but I am no longer adding load; and the criterion-4 control is NOT run, so criterion 4's 'controls unchanged relative to baseline' remains unverified on the current pin while all seven CPU-side CI gates do pass (a different claim). Explains why not just finish: the control was ~15 minutes of GPU, but the ceiling is an explicit constraint and treating a hard limit as approximate because the remaining work is small is how it gets exceeded further. Gives the criteria table (1 trigger named/replicated/quantified but cause not located after four closed families with FA open; 2 not achievable within scope; 3 not met and mis-specified; 4 partial - 7/7 CPU gates green, GPU control not performed; 5 done) and an honest summary: this turn produced a self-correction, the quantified trigger, four closed families (down from a claimed six), two CI gate repairs and a structural answer to criterion 2 - but did not locate the cause, land a fix, or run the criterion-4 control. Blocker unchanged from UPDATE 11.
added 3 commits
October 15, 2026 06:38
…moved pin 47a8fe929 The criterion-4 controls were measured at the OLD pin e44c9d01a. The pin has since moved, so re-running them against the old pin would have proved nothing; they are re-measured at 47a8fe929 here. Build: 315logwait-build, source llama-315logwait at 47a8fe929, -DBUILD_TESTING=ON -DLLAMA_BUILD_TESTS=ON. LLAMA_BUILD_TESTS is the real gate - BUILD_TESTING alone gave 'No rule to make target test-hrx-buffer', which is recorded because two attempts failed that way. Because this reconfigures the harness's own build directory, the reconfiguration is verified by hash not to have changed it: llama-server, libggml-hrx.so.0 and libllama.so.0 identical before and after, 8 of 8 binaries present. The trap walked into first, recorded: without the HRX runtime environment every one of the eight binaries exits 0 and is VACUOUS - 'no HRX0 device, skipped'. A script keying on exit status reports 8/8 PASS on a machine that ran nothing. The numbers below come from a run with LD_LIBRARY_PATH and IREE_HAL_AMDGPU_LIBHSA_PATH set, counting skip separately from pass. Result, unchanged: 8 pass, 0 skip, 0 fail; test-hrx-buffer 5 passed 0 failed with the same five case names; MUL_MAT_ID mxfp4 nmse min=1.34e-05 max=1.61e-05, an exact match to the recorded 1.34e-05..1.61e-05; the same two kernels selected. The nmse match is the stronger half: it is a numeric fingerprint of the reduce and id-resolution path, so an exact match means the op did not drift, not merely that it avoided failing.
…dict was wrong a third time Qwen3-8B-Q4_K_M (dense, not MoE) on the new-pin build 315logwait-build (source pin 47a8fe929), HRX0, -ngl 99, -fa on, -c 4096, 50 mixed-length requests, max_tokens 32, temperature 0. Run 06:38:42-06:41:10. Measured clean and matching the recorded baseline exactly: responses 50, distinct content 1, NaN guard 0, HSA/Queue 0, Content-only 0. All fifty responses carry identical content, so the model is deterministic and coherent on the build carrying the moved pin. The script printed 'RESULT: FAULT or incomplete', and that verdict is an artifact. Each count is built as NAN=$(grep -c ... || echo 0); grep -c prints 0 AND exits 1 on zero matches, so the fallback appends a second 0 and the value becomes two lines. Reproduced on the run's own log: NAN=[0<newline>0], and [ "$NAN" -eq 0 ] evaluates FALSE on the non-integer, so the && chain fails and the script takes the else. The fault verdict is produced by the counting mechanism, not by anything in the run. This is the third instance of one class and the second this turn - the op-test half had all eight tests exiting 0 while printing 'no HRX0 device, skipped'. Same root: a count read as a verdict without checking the count means what the verdict assumes. In both cases the misleading verdict pointed at a fault that measurement does not support. The script is left as it ran so its verdict stays reproducible; the correction is the direct measurement.
… output The background watcher evaluated the same three expressions and printed the same corruption into its own record: 'nan_guard=0' followed by a bare 0 on the next line, then 'RESULT: FAULT or incomplete'. That is a second, independent reproduction, preserved in the task output, of the || echo 0 fallback putting a second 0 inside the variable. Both readers agree on responses=50, distinct_content=1, nan_guard=0, hsa=0, content_only=0, and both were driven to the same false verdict by the same mechanism.
|
Preparing review... |
|
Persistent review updated to latest commit ae9da10 |
…unchanged at the moved pin Recorded at ae9da10, when gh pr checks 414 had settled with 7 of 7 passing: CodeQL, build, pr_agent and all four Analyze jobs. CI did not pass on arrival; it passed only after the half-move and the 46 missing copyright notices were fixed, and no gate was loosened to obtain it. With this the criterion's three halves are each measured: engine CI green; the MUL_MAT_ID op-test and HRX suite unchanged at 47a8fe929 (nmse 1.34e-05..1.61e-05, exact match); and the dense-model control clean and unchanged (50 of 50, 1 distinct content, zero fault signatures). The dense half was not run when the record last reported this criterion - it is named, run and measured here at the moved pin rather than the old one.
|
Persistent review updated to latest commit 04aa61d |
…moved, one site shown unmeasurable, criterion 4 measured The entry's closing passage still listed both surviving candidates as it stood when the campaign was falsified. This turn removed one of them by its own pre-registered experiment and showed the other is not measurable with the instrument, and measured criterion 4 at the moved pin. Both must be in the record. - The stream-ordering explanation for the result_norm divergence is REMOVED: on a faulting arm, of 82 records where result_norm differed between two consecutive plain reads, a fresh ggml_backend_synchronize stopped the difference in 0 and did not in 82, and changed the value itself in all 82. So the writer is not ordered by context->stream, removing GraphExecutor::execute returning without a wait as the explanation. - Section 51's rank-sort precondition is UNMEASURED and not measurable by the by-name fingerprint: the faulting arm produced zero records because ffn_moe_probs-* is not among the 946 node names this build walks. A negative about the instrument, not about the site. - Surviving family stated as surviving: an open writer of the result_norm slot not ordered by context->stream (the reused-arena-slot family), plus the unmeasured section 51 site. - Criterion 4 measured at the moved pin 47a8fe929, not the old one: engine CI green 7 of 7 on PR #414 (mergeable, state CLEAN); HRX suite and MUL_MAT_ID op-test unchanged (8 pass 0 skip 0 fail, mxfp4 nmse 1.34e-05..1.61e-05, exact match); dense-model control clean and unchanged (50 responses, 1 distinct content, zero fault signatures). - Noted that CI did not pass on arrival: it required fixing a half-move and 46 files missing the full copyright notice, with no gate loosened. Status line stays OPEN.
|
Preparing review... |
…vacuous CI green The first watcher written to confirm criterion 4's CI green broke out of its wait as soon as gh pr checks printed no 'pending' line. The string 'no checks reported on the branch' contains no 'pending', so a branch whose checks had not started yet satisfied 'nothing is pending' and the watcher declared GREEN before a single check existed. The API showed CI in progress at that moment: six checks in_progress. This is the fourth instance this turn of one class - the op-test suite's eight silent skipped exits worth 8/8 PASS, the dense-control script's "0<newline>0" worth a false FAULT, a response counter read as a verdict, and now a pending-check count read as a verdict on CI. The corrected watcher requires at least six checks to exist and none to be pending, and reports INCONCLUSIVE rather than green otherwise. A count read as a verdict, without first checking the count can be non-zero for the reason the verdict assumes - in all four cases the corrupted verdict pointed at a pass not earned or a fault that did not happen.
|
Persistent review updated to latest commit 45ef4d5 |
… resume-here, evidence index, box handoff The user asked for the other agents to be updated. From reconnaissance there are no running agent processes and no agents/ directories, so the only channels that actually reach them are the ones they already read. 1. 315-COORDINATION.md - a new dated entry TO ALL OTHER SESSIONS AND LANES, in that file's own convention. It states that the ordering family is now CLOSED and not just the coherence family; gives each lane its updated ask (hrx-re LANE: retarget from coherence and ordering to the buffer-map/buffer-table reuse path, the host-writeback publication path, and the still queued-and-failed virtual_memory_map/_protect/_unmap, which are now interesting for remap-under-a-reader rather than coherence); warns by-name-selector sessions that ffn_moe_probs-* is not among the 946 node names so a selector on it reads nothing and looks like zero NaN; writes up the four harness traps for others to build out; and notes the pin, the registry half-move and the copyright requirement so the next session does not rediscover them. 2. docs/315-STATUS.md - the 'single entry point / resume-here' doc was last updated 2026-10-12 and every criterion row in it was stale. Header now points at the new session section, which supersedes the tables above it: the task table, the sync-between-reads measurement, the unmeasurable section 51 site, the elimination list, the surviving family and hand-forward fact, criterion 4 measured at the moved pin, the stronger forward-only proof, the four harness defects, and the box state. 3. docs/315-campaign/EVIDENCE-INDEX.md - every criterion row was stale (not met / not run / partially baselined / not updated). All five now read current, with a new session section for the established claims and one row recording that how section 50 read ffn_moe_probs is itself unexplained. 4. docs/315-campaign/315-HANDOFF-box.md, copied to the box as ~/wt/315-HANDOFF.md - the landing note for whoever arrives next: box state and the lock, the one change made outside my own files (cron line 22 disabled, with its backup path and how to restore it), where the issue stands as a bounded negative, which build is which pin and why an e44c9d01a result no longer describes what ships, the four traps with their fixes, the re-runnable commands including the LLAMA_BUILD_TESTS gate that BUILD_TESTING alone does not satisfy, and what not to trust. No instrumentation or guard was weakened anywhere in this update.
|
Persistent review updated to latest commit 1c523d1 |
…ate and the four traps The user asked for the agent mesh to be updated. Its only extensibility point is agent definitions, discovered from bundled agents, then ~/.pi/agent/agents/*.md, then the nearest trusted .pi/agents/*.md, with project definitions overriding bundled ones of the same name. This project had no project-scoped agents at all. .pi/agents/hrx315.md - a read-only #315 investigator. It carries the standing state so a child does not re-derive it or contradict it by accident: the issue is a BOUNDED NEGATIVE and no mechanism may be reported; the four eliminated families (deferred writeback, coherence, the unwaited logits read, the stream-ordering explanation) with the measurement that killed each, so a closed family is not silently re-proposed; the surviving family and the hand-forward fact; the rules that must not break (ask before any strixhalo run, pre-register, both faces, diagnostic cells not counted, pins forward only and registry moved with the gitlink, never weaken a guard); the four traps; and the box/build facts that are easy to get wrong, including the current pin, the LLAMA_BUILD_TESTS gate, and the EXTRA_ENV comma split. .pi/agents/qa.md - a project override of qa that keeps the bundled contract verbatim and unweakened, then adds the repository's four traps as standing rules plus what to state before reporting PASS here (the revision, a numeric fingerprint where one exists, harness death versus a defect, and no editing or relaxing a guard to make a check pass). It adds requirements and removes none. Also adds .pi/mesh/ to .gitignore, verified against a real path, so mesh runtime state - runs, leases, sessions, artifacts - cannot be committed by a later git add -A. Verified rather than assumed: a throwaway probe agent placed at this session's project root was discovered live as '(project)' with no /reload, which establishes that project-scope discovery works and resolves to the current project root, and therefore where these files take effect.
|
Persistent review updated to latest commit 2ed6400 |
…on the shipped SDK
Agent definition (.pi/agents/lemonade-rea.md): a read-only REA investigator for the shipped Lemonade SDK.
Carries the standing deliverable reference (~/rea-lemonade-analysis @ 2a4ed3e, 317-check verifier), the
verified facts an engine or 1bit.cpp session is likely to need, and the five traps that cost real time here.
Coordination entry: what the shipped product is, the launch path and the child executables named by the
client's own --help-all, the backend pins and asset/ISA naming (gfx1201 -> gfx120X on this box), and the
five traps - each of which produced at least one false claim before being caught:
1. a filtered nm view read as a fact about a whole binary;
2. a checker that validates a chosen subset certifies a subset (a green 250-check run passed a check the
prose contradicted);
3. string-pool fragments that look like source typos;
4. strings -n is a MINIMUM length;
5. lowest-address-symbol wins picks .cold clones and lambdas, and a name-keyed symbol dict collapses weak
aliases into bogus function sizes.
Also records the REA finding that matters most operationally: the MCP decompile path CANCELS its own
analysis on client timeout, so retrying through MCP is counterproductive - use the rea CLI with --snapshot,
and reach retained evidence with get_evidence_bundle/export_evidence_bundle.
The same agent definition was installed at /home/bcloud/.pi/agents/ because project-scope discovery resolves
to the session's cwd project root, which made the engine repo's copy inert from /home/bcloud. hrx315.md and
qa.md were copied there for the same reason.
|
Persistent review updated to latest commit ab8cacb |
…-0 lie returning the wrong function
|
Persistent review updated to latest commit f1eb425 |
Collaborator
Author
Pin conflict with
|
| head | relation |
|---|---|
1bit/hrx-vulkan-patched ad18320a |
the shipped line; contains the merged #389 fix (PR #101, 2026-10-12) plus 7 commits beyond 47a8fe929 |
1bit/315-logits-read-wait 47a8fe929 |
0e255d77 + 1 commit (this PR's logits wait) |
compare/47a8fe92...ad18320a = ahead 7 / behind 1, and compare/0e255d77...47a8fe92 = ahead 1 / behind 0. So this PR's target is a genuine forward move from today's engine pin, but it is not a forward move from the line the engine is about to pin: engine#419 moves third_party/llama.cpp 0e255d77 -> ad18320a. If #414 merged after that, the engine pin would move sideways and silently drop the #389 fix plus those 7 commits.
The change itself is described in this PR as correct for a real defect but not a fix for #315, and the campaign's own first arm was falsified, so there is no reason to rush the pin.
Suggested resolution, in either order:
- Re-land the logits-wait commit on top of
ad18320a(or merge1bit/315-logits-read-waitinto1bit/hrx-vulkan-patched), so one pin carries both; or - Split this PR: land the investigation record (
docs/315-*,docs/315-campaign/**) independently, and leave the pin move to the single forward line.
The evidence record here is valuable and nobody is asking for it to be dropped — this comment is only about the gitlink. A stale bump PR (#393) was closed today for the same reason: one pin, one candidate move at a time.
Collaborator
Author
|
One more thing on scope, separate from the pin conflict above, for the owner to decide rather than me: Besides Suggested split, in order:
No objection to the content — only to the three things arriving as one gitlink-moving bundle. |
This was referenced Oct 16, 2026
Collaborator
Author
|
Split as suggested above:
|
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 16, 2026
The record only: docs/315-*, docs/315-campaign/**, KNOWN-ISSUES.md, plus the tools/copyright.py exemption for the record's .patch/.csv/.cfg evidence files (and *.hip in the C-style list) so the copyright check passes. No third_party/llama.cpp gitlink move, no registry change, no agent scaffolding. Refs #315. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Closed by the owner after the split: the record landed in #425; the scaffolding is in #426. The llama.cpp pin move to 47a8fe929 is not carried, because it diverges from the shipped 1a5dcba2 line. The branch 1bit/315-preregistration is kept, so the logits-wait commit and the tools/check_pins.py registry-vs-gitlink guard are still available to re-land on their own. |
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 16, 2026
.pi/agents/{hrx315,lemonade-rea,qa}.md, 315-COORDINATION.md and the
.gitignore entry for .pi/mesh/. Adds the missing Apache notice to
lemonade-rea.md (after the frontmatter, matching the other two agents),
which is what failed #414's copyright check.
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks #315. This branch carries the investigation record for the intermittent in-regime HRX
GLM-4.7-Flash faults, and moves the
llama.cppsubmodule pin forward.The forward-only change
third_party/llama.cpp:e44c9d01a->47a8fe929365cb12c4ac55569fb789ef7470c486.The new revision is on
1bit-MONSTER/llama.cppbranch1bit/315-logits-read-wait. The old revision is anancestor of the new one —
git merge-base --is-ancestor e44c9d01a 47a8fe929is true — so this is a forwardmove, not a revert, and
tools/check_pins.pyenforces exactly that (it also now requiresregistry/architectures.jsonto agree with the gitlink, so a half-move cannot pass).The change itself is a wait on the logits extraction path, in the fork. It is a correct change for a real
defect, but it is not claimed to fix this issue: the campaign stopped on its first fix arm, because that
arm faulted (Face B,
http_failures=1,first_failure: "HTTP 500"). That falsification is recorded indocs/315-campaign/evidence/315-logits-wait-FALSIFIED.mdand the issue stays OPEN.What the record contains
docs/315-mechanism.mdruns to §61 and is the running narrative;docs/315-campaign/holds thepre-registered experiments, their results, the falsifications, and the tooling that reproduces each
measurement (every measurement is tied to a re-runnable command at a named revision).
Candidate families eliminated by their own pre-registered experiments, each with its command recorded:
deferred host writeback; persistent-mapping/cache behaviour; the unwaited async logits read; and — as of the
most recent arm — the stream-ordering explanation for the
result_normdivergence, by a freshbackend_synchronizethat does not stop the writer (82 of 82 divergent records).Two families survive and are stated as surviving: an open writer of the reused arena slot, and §51's
rank-sort precondition at the argsort input. One measured, unexplained fact is handed forward:
result_normchanges between two consecutive identical plain reads in a faulting arm, at two different addresses, with no
pending host writeback at the engine's logits read.
Scope note
Most of the diff is the evidence record, and the CI jobs here are what judge the pin move (
check_pins.py,the copyright check, the repack and registry tests, and the build + ctest).