From 77b4681ba500ae0c30314b4bd309b2afc5076567 Mon Sep 17 00:00:00 2001 From: alice Date: Tue, 11 Aug 2026 06:46:19 +0000 Subject: [PATCH] =?UTF-8?q?vms-0425:=20rejoin=20=E2=80=94=20sub-question?= =?UTF-8?q?=20settled=20+=20CM=20JOIN=20gate=20bisected=20(spec=20=C2=A74(?= =?UTF-8?q?O.24))?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4(O.23) relocated the frontier to "reclaim succeeds, CM JOIN does not run" and left one open sub-question: its reclaim was shown with a blatantly-future incarnation pin (1-OCT-2026, for SDA day-resolution visibility). Does a REALISTIC near-now incarnation — what OVMX presents normally — also reclaim, or does the reclaim need a coarse delta (which would move the frontier to "make the member see a new incarnation")? Sub-question SETTLED (same-boot single-factor SDA bracket, vaxlab-1): a realistic return (0425Rr, OVXR40, default live incarnation) reclaims the member's CSB (8796AE80 -> 879DF540, fresh CSB at status_rcvd, non-member) IDENTICALLY to a future-pinned return (0425Fr, OVXF40, 1-OCT-2026: 879DEF00 -> 879DF2C0). Both stall at 02040000 status_rcvd and never reach member,selected; XITDONE=0. Only the SDA-displayed date differs. The reclaim needs no coarse delta — the frontier does NOT relocate; it stays at the CM JOIN transition. §4(O.22)'s "never reclaims" was a SHOW CLUSTER-string artifact. CM JOIN gate BISECTED (Rf-vs-Rr CM-frame wire diff, same identity/pod): on the first join OVMX sends op 0x02 -> coordinator RELAYS (op 0x12) + COMMITS (op 0x03) -> joined. On the return OVMX STILL sends op 0x02 (correctly addressed) but the coordinator sends no op 0x12 relay and no op 0x03 commit; it runs a member<->member cat-0x06 reconcile that excludes OVMX, then a transition ABORT. This bisects the frontier's either/or: NOT an OVMX op02 omission (OVMX drives it), NOT a silent member wait (the coordinator actively runs a different transition). Per Davis p.7-38 the coordinator IGNORES the join request when its admission tests fail and "no state transition occurs"; a first join passes them, a return does not. Relocated frontier: SDA on the COORDINATOR (the member OVMX sends op 0x02 to), across departure+return, to see why it declines an op02-driven ADD for a reclaimed identity it admits when fresh (residual CLUB nodemap / quorum-vote accounting?). Ships (no wire change, guard 8 — first-join path byte-unchanged): - READMITMAP verdict refinement: JOIN-ABANDONED (joiner_cfg2_sent && cm_responses ==0) split out of RECLAIMED-NOJOIN/NO-ENGAGE, naming the member OVMX actually DROVE the join request to (the coordinator) distinctly from one it only reached. Checked before the open latch (OVMX's own VMS$VAXcluster VC ends CONNSTUCK on the return, so the latch under-reports). Log-only, kill-switch OVMX_NO_READMITMAP unchanged. Live pass-post: scsd-0425Rr2.log reads coordinator JOIN-ABANDONED, non-coordinator NO-ENGAGE, SUMMARY join_abandoned=1 (old classifier read both NO-ENGAGE). - fail-pre/pass-post unit case in tests/vmsscs/test_scsd_wire.c. - docs/clean-room/tools/cmdiff.py: CM-frame (category/opcode) decoder. - spec §4(O.24) + design record docs/design-rejoin-cm-state-map.md §8. XITDONE did NOT flip (0->0, as expected): this SETTLES the sub-question and BISECTS the gate; it does not fix rejoin. VAXcluster Principles pp. 7-24/7-25/7-37/7-38/7-39 (page cites only, Rule 8). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/clean-room/tools/cmdiff.py | 60 +++++++++++++++++ docs/cluster-protocol-spec.md | 70 ++++++++++++++++++++ docs/design-rejoin-cm-state-map.md | 100 +++++++++++++++++++++++++++++ src/vmsscs/scsd.c | 51 +++++++++++++-- tests/vmsscs/test_scsd_wire.c | 42 +++++++++++- 5 files changed, 315 insertions(+), 8 deletions(-) create mode 100644 docs/clean-room/tools/cmdiff.py diff --git a/docs/clean-room/tools/cmdiff.py b/docs/clean-room/tools/cmdiff.py new file mode 100644 index 000000000..c528f33b1 --- /dev/null +++ b/docs/clean-room/tools/cmdiff.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +# vms-0425: decode the CM (membership/config) dialogue on the VMS$VAXcluster +# connection, per member, so a first-join capture can be diffed against a +# return capture. Offsets from scs_member_parse (scs_member.c): category=body[8] +# =frame[80], opcode=frame[81], conids frame[64:72], SYSAP name frame[76:92]. +import os, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from pcap import frames, is6007 + +VAX1 = bytes.fromhex('aa0004000104') +def le16(b,o): return b[o]|(b[o+1]<<8) +def le32(b,o): return b[o]|(b[o+1]<<8)|(b[o+2]<<16)|(b[o+3]<<24) +def name_at(p,o): return p[o:o+16].split(b'\x00')[0].decode('latin1','replace') + +CAT={0x01:'CONFIG',0x02:'DLM',0x04:'ACK',0x06:'MEMBSHIP', + 0x81:'CONFIG.r',0x82:'DLM.r',0x84:'ACK.r',0x86:'MEMBSHIP.r'} +OP ={0x14:'MODEL',0x01:'PARAMS',0x02:'op02-CONFIG',0x03:'op03-COMMIT', + 0x05:'op05-LOCKRB',0x06:'op06-MEMB',0x09:'op09-XITION',0x0a:'op0a-GO', + 0x12:'op12-RELAY',0x0d:'op0d-DEPART',0x04:'op04-ABORT',0x08:'op08',0x00:'-'} + +path=sys.argv[1] +fr=[(t,p) for (t,p) in frames(path) if is6007(p) and len(p)>=82] +if not fr: sys.exit('no 6007 frames') +t0=fr[0][0] +# detect MACs +macs={} +for t,p in fr: + macs[p[6:12]]=macs.get(p[6:12],0)+1 +others=[m for m in macs if m!=VAX1] +# OVMX = the non-DEC-ish, but just label: VAX1, and the rest by hex +def who(mac): + if mac==VAX1: return 'VAX1' + if mac[:3]==bytes.fromhex('08002b'): return 'VAX2' + return 'OVMX' + +# map (conid-pair) -> sysap from connect frames (msgtype 0x5b/0x4b carrying a name) +SYSAPS=[b'VMS$VAXcluster',b'SCS$DIRECTORY',b'MSCP$DISK',b'MSCP$TAPE',b'SCS$DIR_LOOKUP',b'VMS$DISK_CL_DRVR'] +conid_sysap={} +for t,p in fr: + nm=p[76:92].split(b'\x00')[0] + if nm in SYSAPS: + rc=le32(p,64); lc=le32(p,68) + conid_sysap[(rc,lc)]=nm.decode() + conid_sysap[(lc,rc)]=nm.decode() + +print(f"# {path}") +print(f"# MACs: " + ", ".join(f"{who(m)}={m.hex()}({macs[m]})" for m in macs)) +print(" t src>dst mt cat op rconid lconid sysap") +for t,p in fr: + mt=p[30] + if mt not in (0x4b,0x5b): continue + cat=p[80]; op=p[81] + # only CM-category frames (config/membership/ack and their responses) + if (cat & 0x7f) not in (0x01,0x04,0x06): continue + src=who(p[6:12]); dst=who(p[0:6]) + rc=le32(p,64); lc=le32(p,68) + sysap=conid_sysap.get((rc,lc),'?') + # focus on the cluster membership connection + print("%8.3f %-4s>%-4s %02x %-9s %-13s %08x %08x %s"%( + t-t0, src, dst, mt, CAT.get(cat,'%02x'%cat), OP.get(op,'%02x'%op), rc, lc, sysap)) diff --git a/docs/cluster-protocol-spec.md b/docs/cluster-protocol-spec.md index 5443a10da..cd75252d7 100644 --- a/docs/cluster-protocol-spec.md +++ b/docs/cluster-protocol-spec.md @@ -5632,6 +5632,76 @@ scsd-944B1f.log`/`scsd-944B2r.log` + `d94-944B1f/944B2r.pcap`; the wire incarnat (`readmit_verdict_of`, `READMIT_RECLAIMED_NOJOIN`); unit `tests/vmsscs/test_scsd_wire.c`; design record `docs/design-rejoin-cm-state-map.md` §7; *VAXcluster Principles* pp. 7-23/7-24/7-25/7-29/7-30/7-37/7-39. +#### 4(O.24) The §4(O.23) sub-question is SETTLED and the CM JOIN gate is BISECTED on the wire: a REALISTIC near-now incarnation reclaims the CSB IDENTICALLY to the future pin (so the frontier stays at the CM JOIN transition, NOT at "make the member see a new incarnation"); and a same-pod first-join-vs-return op02 diff proves OVMX DOES drive op 0x02 to the coordinator on the return — the coordinator receives it and starts NO ADD transition (no op 0x12 relay, no op 0x03 commit), running a member↔member reconcile + ABORT instead, so the gate is the COORDINATOR ignoring an op02-DRIVEN join for a returning identity (Davis p. 7-38), NOT an OVMX op02 omission (GROUNDED, `vms-0425`, same-boot single-factor SDA bracket + Rf/Rr CM-frame wire diff on `vaxlab-1` + *VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39, 2026-08-11) + +**Frame.** §4(O.23) relocated the frontier to "reclaim succeeds, CM JOIN does not run" and left ONE +open sub-question: its reclaim was shown with a **blatantly future** incarnation pin (1-OCT-2026, +for SDA day-resolution visibility). Does a **realistic near-now** incarnation — what OVMX presents +normally — ALSO reclaim, or is the reclaim an artifact of a coarse delta (which would move the +frontier to "make the member see a new incarnation")? `vms-0425` settles it, then bisects the JOIN +gate on the wire. + +**Sub-question SETTLED (same boot, single factor).** Two first-join→return pairs on ONE +freshly-booted `vaxlab-1`, two brand-new identities, first-join op02 form constant, `OVMX_JOIN_SEQ=1`, +VAX1 in SDA per arm — the pairs differ in **exactly one factor**, the return's incarnation: + +| arm | identity | return incarnation | member's CSB for it (SDA) | `XITDONE` | +|---|---|---|---|---| +| `0425Rf` | `OVXR40`/1975 | — (first join) | built `8796AE80` → `member,selected` | **1** | +| `0425Rr` | `OVXR40` | default live (near-now) | `8796AE80` DEALLOCATED → NEW `879DF540`, `02040000 status_rcvd` (non-member) | **0** | +| `0425Ff` | `OVXF40`/1976 | — (first join) | built `879DEF00` → `member,selected` | **1** | +| `0425Fr` | `OVXF40` | `1-OCT-2026` | `879DEF00` DEALLOCATED → NEW `879DF2C0`, Incarnation `11-AUG → 1-OCT`, `02040000 status_rcvd` (non-member) | **0** | + +The realistic return (`Rr`) and the future-pinned return (`Fr`) produce **byte-identical CSB +outcomes**: both DEALLOCATE the residual CSB and REBUILD a fresh one at a new address (Davis +pp. 7-24/7-25), both reach `02040000 status_rcvd` — the exact intermediate state a SUCCESSFUL first +join passes through — and both **stall there, never reaching `member,selected`** (`XITDONE=0`). Only +the SDA-displayed Incarnation *date* differs. So a realistic incarnation reclaims exactly as the +future pin does; §4(O.22)'s "never reclaims" was a `SHOW CLUSTER`-string artifact. **The frontier +does NOT relocate — it stays at the CM JOIN transition.** + +**CM JOIN gate BISECTED (Rf-vs-Rr CM-frame wire diff, same identity/pod).** Decoding the +`VMS$VAXcluster` CM frames (category `body[8]`, opcode `body[9]`; OVMX `b2df5fa41233`, coordinator +VAX2 `08002b45161a`): on the FIRST join OVMX sends op 0x02 → the coordinator RELAYS it (op 0x12) to +the other member and COMMITS (op 0x03) to OVMX → op 0x05/op 0x06 → joined. On the RETURN **OVMX +still sends op 0x02** (its own log: `SCSD-I-CMCONFIG2 … sent DEFERRED op 0x02 … to node 2 … expect +its op 0x03 COMMIT`), correctly addressed — but the coordinator sends **no op 0x12 relay and no op +0x03 commit**; it runs a member↔member cat-0x06 MEMBERSHIP reconcile that EXCLUDES OVMX, then a +transition ABORT, and OVMX's `VMS$VAXcluster` VCs end CONNSTUCK (`still-open-at-exit=2`). This +**bisects the either/or**: it is NOT that OVMX omits op 0x02 (it drives it), and NOT that the member +waits silently (it actively runs a different transition). Per Davis p. 7-38 the coordinator, on a +join request, runs admission/quorum tests and **"ignores the request … and no state transition +occurs"** when they fail; a FIRST join passes them, a RETURN (against a member that just reclaimed +OVMX's CSB) does not. **The gate is the coordinator ignoring an op02-driven ADD for a returning +identity** — one step in from §4(O.23)'s "CM JOIN does not run". + +**Relocated frontier.** The next isolation is **SDA on the COORDINATOR** (the member OVMX sends op +0x02 to), across departure+return, to see why it declines an op02-driven ADD for a reclaimed +identity it admits when fresh — i.e. whether its CLUB nodemap / quorum-vote accounting still carries +residual state for the departed OVMX so its p. 7-38 tests fail. A separate OVMX-side defect surfaced +(not the gate): on the return OVMX re-drives its add-member burst with `remote_conid=0x00000000` and +leaves its `VMS$VAXcluster` VCs CONNSTUCK. + +**Non-claims / what ships (no wire change, guard 8).** (1) rejoin is NOT closed; this SETTLES the +sub-question and BISECTS the gate. (2) The READMITMAP verdict gains `JOIN-ABANDONED` +(`joiner_cfg2_sent && cm_responses==0`), split out of `RECLAIMED-NOJOIN`/`NO-ENGAGE`, naming the +member OVMX actually DROVE the join request to (the coordinator) distinctly from a member it only +reached — checked before the open latch because OVMX's own VC ends CONNSTUCK on the return (so the +latch under-reports; the live `0425Rr` read `NO-ENGAGE` under the old latch-first classifier, masking +that op 0x02 had been driven). Log-only, kill-switch `OVMX_NO_READMITMAP` unchanged, fail-pre/pass-post +case in `tests/vmsscs/test_scsd_wire.c`, byte-unchanged on the wire. + +**Evidence** (host, tank volume): `/data/training/vax/cluster/work/0425{Rf,Rr,Ff,Fr}.csb` (SDA CSB +timelines) with `.status` (`Rf`/`Ff` `XITDONE=1`; `Rr`/`Fr` `XITDONE=0`); +`/lab/k8s-labs/vaxlab-1/logs/scsd-0425{Rf,Rr,Ff,Fr}.log` + `d94-0425{Rf,Rr,Ff,Fr}.pcap` (identities +`OVXR40`/1975, `OVXF40`/1976 on the wire); CM-frame decoder `docs/clean-room/tools/cmdiff.py`; code `src/vmsscs/scsd.c` +(`readmit_verdict_of`, `READMIT_JOIN_ABANDONED`); unit `tests/vmsscs/test_scsd_wire.c`; design record +`docs/design-rejoin-cm-state-map.md` §8. A live re-run with the refined daemon (`0425Rr2`, +`scsd-0425Rr2.log`) reads the new verdict on the wire: coordinator `08:00:2b:45:16:1a` +`verdict=JOIN-ABANDONED`, the non-coordinator `NO-ENGAGE`, SUMMARY `join_abandoned=1 no_engage=1` +(the old latch-first classifier read both `NO-ENGAGE`, masking the op02-driven fact). +*VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39. + ## 5. Summary of unknown/inferred fields (RE gaps) For visibility, every field NOT marked GROUNDED above: diff --git a/docs/design-rejoin-cm-state-map.md b/docs/design-rejoin-cm-state-map.md index 47fec8b10..674f8fa91 100644 --- a/docs/design-rejoin-cm-state-map.md +++ b/docs/design-rejoin-cm-state-map.md @@ -470,3 +470,103 @@ timelines above), `944B1f.status` (`XITDONE=1`) / `944B2r.status` (`XITDONE=0`), `/lab/k8s-labs/vaxlab-1/logs/scsd-944B1f.log`/`scsd-944B2r.log` + `d94-944B1f/944B2r.pcap`. Wire `[66:74]` incarnation distinct per run (arm A `0x00bc0cd311bc6c62`, arm B′ `0x00bc0cd380815b38`). *VAXcluster Principles* pp. 7-23/7-24/7-25/7-29/7-30/7-37/7-39. + + +## 8. Sub-question settled, and the CM JOIN gate bisected on the wire — OVMX DRIVES op 0x02, the coordinator ABANDONS it (`vms-0425`, spec §4(O.24)) + +§7 relocated the frontier to "reclaim succeeds, CM JOIN does not run" and left ONE open +sub-question: §7's reclaim was demonstrated with a **blatantly future** incarnation pin +(1-OCT-2026, chosen for SDA day-resolution visibility). Does a **realistic near-now** +incarnation — what OVMX presents in normal operation — ALSO trigger the reclaim, or is the +reclaim an artifact of a coarse incarnation delta (in which case the true frontier would be +"make the member see a new incarnation", not the JOIN transition)? `vms-0425` settles it with a +same-boot single-factor bracket, then bisects the JOIN gate itself on the wire. + +### 8.1 Sub-question SETTLED — a realistic incarnation reclaims identically to the future pin (GROUNDED, `vaxlab-1`, clean CN_2, 2026-08-11) + +Two first-join→return pairs on ONE freshly-booted pod, two brand-new identities, first-join op02 +form held constant, `OVMX_JOIN_SEQ=1`, hard `timeout` per op, VAX1 parked in SDA sampling each +identity's CSB (`csbwatch.sh`). The two pairs differ in **exactly one factor**: the incarnation +the RETURN presents. + +| pair | arm | identity | return incarnation | member's CSB for the identity (SDA) | `XITDONE` | +|---|---|---|---|---|---| +| R (realistic) | `0425Rf` | `OVXR40`/1975 | — (first join) | CSB built `8796AE80`, advances to `member,selected` (02060002) | **1** | +| | `0425Rr` | `OVXR40` | **default live (near-now)** | old CSB `8796AE80` DEALLOCATED → **NEW CSB `879DF540`**, `1-JAN-2001` ref-time (a fresh CSB), Flags `02040000 status_rcvd` (**non-member**) | **0** | +| F (future) | `0425Ff` | `OVXF40`/1976 | — (first join) | CSB built `879DEF00`, advances to `member,selected` | **1** | +| | `0425Fr` | `OVXF40` | **`1-OCT-2026`** (`OVMX_INCARNATION_TIME=52975296000000000`) | old CSB `879DEF00` DEALLOCATED → **NEW CSB `879DF2C0`**, Incarnation visibly `11-AUG-2026 → 1-OCT-2026`, Flags `02040000 status_rcvd` (**non-member**) | **0** | + +**The flip that is NOT there.** The realistic return (`0425Rr`) and the future-pinned return +(`0425Fr`) produce **byte-identical CSB outcomes**: the member DEALLOCATES the residual CSB and +REBUILDS a fresh one at a new address (a genuine reclaim, Davis pp. 7-24/7-25), which reaches +`02040000 status_rcvd` — the **exact** intermediate state a SUCCESSFUL first join passes through — +and then **stalls there, never advancing to `member,selected`**. The only observable difference is +the SDA-displayed Incarnation *date* (`11-AUG` vs `1-OCT`); the reclaim, the stall, and `XITDONE=0` +are the same. **A realistic incarnation reclaims exactly as the future pin does — the reclaim needs +no coarse delta.** §6.4's "never reclaims" was a `SHOW CLUSTER`-string artifact (SDA sees the +address change the console strings could not). The sub-question is closed: **the frontier does NOT +relocate; it stays at the CM JOIN transition.** + +### 8.2 The CM JOIN gate bisected — OVMX DRIVES op 0x02; the coordinator does not START the ADD + +Both the SUCCESS first-join (`0425Rf`) and the return (`0425Rr`) were captured (pcap, in-pod +`br0`). Decoding the CM (config/membership) frames on the `VMS$VAXcluster` connection — category +`body[8]`, opcode `body[9]`, per `scs_member_parse` — with the SAME identity on the SAME pod +(OVMX `b2df5fa41233`, VAX1 `aa0004000104`, VAX2 `08002b45161a`, VAX2 = the coordinator OVMX +selects) gives the discriminating diff: + +| step | first join `0425Rf` (`XITDONE=1`) | return `0425Rr` (`XITDONE=0`) | +|---|---|---| +| OVMX → coordinator op 0x02 (join request) | **sent** (t≈29.57s) | **sent** (t≈31.47s) — *OVMX drives op 0x02 on the return too* | +| coordinator → other member op 0x12 RELAY | **sent** (relays the new member) | **absent** | +| coordinator → OVMX op 0x03 COMMIT | **sent** → OVMX 0x81-responds (`cm_responses++`) | **absent** (`cm_responses=0`) | +| what the coordinator does instead | op 0x05 lock-rebuild, op 0x06 membership burst → OVMX joins | a member↔member **cat-0x06 MEMBERSHIP** reconcile with the OTHER member that EXCLUDES OVMX, then a transition **ABORT** (op 0x04) | +| OVMX's own `VMS$VAXcluster` VCs at exit | OPEN | **CONNSTUCK in DISC SENT** (`still-open-at-exit=2`) | + +**This bisects the either/or the frontier posed.** It is **not** that OVMX omits op 0x02 — OVMX's +own log confirms it drives the deferred op 0x02 in REJOIN form to the coordinator +(`SCSD-I-CMCONFIG2 … sent DEFERRED op 0x02 … to node 2 … expect its 0x04 ack then its op 0x03 +COMMIT`) and correctly addresses it. And the member is **not** merely "waiting silently" — the +coordinator RECEIVES op 0x02 and actively runs a *different* transition (member↔member reconcile + +abort), starting **no** ADD for OVMX. Per Davis p. 7-38 the coordinator, on receiving a join +request, runs admission/quorum tests and **"ignores the request … and no state transition occurs"** +when they are not satisfied; the FIRST join satisfies them, the RETURN (against a member that has +just reclaimed OVMX's CSB) does not. The gate is the **coordinator's decision to ignore an +op02-driven ADD for a returning identity** — one step further in than §7's "CM JOIN does not run". + +Note also an OVMX-side connection-hygiene defect surfaced by the same capture (not the gate, but +real): on the return OVMX's `VMS$VAXcluster` VCs end **CONNSTUCK** in `DISC SENT` and OVMX re-drives +its add-member burst with `remote_conid=0x00000000` (`SCSD-I-CMREADMIT … remote=0x00000000`). The +correctly-addressed op 0x02 (with the member's live conid) still drew no commit, so this is not the +gate — but it is a distinct bug worth its own item. + +### 8.3 The relocated frontier, and what `vms-0425` ships (no wire change) + +**Relocated frontier.** The next isolation is **SDA on the COORDINATOR** (the member OVMX sends +op 0x02 to — VAX2 here), across the departure and the return, to see WHY it declines to start an +ADD for a reclaimed identity that it admits when fresh: does its CLUB nodemap / quorum-vote +accounting still carry residual state for the departed OVMX (so its p. 7-38 admission tests fail), +even though the per-identity CSB was reclaimed? That is the named, testable next step — and it is a +COORDINATOR-side observation, a different oracle vantage than §4–§7's member-CSB SDA. + +**What `vms-0425` ships (no wire change, guard 8 — the working first-join path is byte-unchanged).** +(1) This record (§8) and the spec §4(O.24) note. (2) The sub-question bracket above (realistic vs +future incarnation, same boot, single factor) closing §7's open sub-question. (3) A READMITMAP +**verdict refinement**: `JOIN-ABANDONED` (`joiner_cfg2_sent` && `cm_responses==0`) split out of +`RECLAIMED-NOJOIN`/`NO-ENGAGE`, so the member OVMX actually DROVE the join request to (the +coordinator) is named distinctly from a member it only reached — surfacing "OVMX did its part, the +coordinator abandoned" as one log line for the next isolation. It is checked BEFORE the open latch +because OVMX's own VC ends CONNSTUCK on the return (so `vaxcluster_open_reached` under-reports, but +the op02-was-driven fact does not — the live `0425Rr` READMITMAP read `NO-ENGAGE` under the old +latch-first classifier, masking that OVMX had driven op 0x02). Log-only, kill-switch +`OVMX_NO_READMITMAP` unchanged, with a fail-pre/pass-post case in `tests/vmsscs/test_scsd_wire.c`. +Still a non-admission (never `ADMITTED`; INV-6). + +Evidence (host, tank volume): CSB timelines `/data/training/vax/cluster/work/0425{Rf,Rr,Ff,Fr}.csb` +with `.status` (`0425Rf`/`0425Ff` `XITDONE=1`; `0425Rr`/`0425Fr` `XITDONE=0`); daemon logs +`/lab/k8s-labs/vaxlab-1/logs/scsd-0425{Rf,Rr,Ff,Fr}.log` and pcaps `d94-0425{Rf,Rr,Ff,Fr}.pcap` +(identities `OVXR40`/1975, `OVXF40`/1976 on the wire); CM-frame decoder `docs/clean-room/tools/cmdiff.py`; code +`src/vmsscs/scsd.c` (`readmit_verdict_of`, `READMIT_JOIN_ABANDONED`); unit +`tests/vmsscs/test_scsd_wire.c`. Live pass-post with the refined daemon: `scsd-0425Rr2.log` reads +coordinator `verdict=JOIN-ABANDONED`, non-coordinator `NO-ENGAGE`, SUMMARY `join_abandoned=1 +no_engage=1` (the old classifier read both `NO-ENGAGE`). *VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39. diff --git a/src/vmsscs/scsd.c b/src/vmsscs/scsd.c index b570e0915..427690156 100644 --- a/src/vmsscs/scsd.c +++ b/src/vmsscs/scsd.c @@ -11403,6 +11403,20 @@ enum readmit_verdict { * the CM JOIN transition. It is a non-admission, but a * DIFFERENT one from NO-ENGAGE: the reclaim + incarnation path * worked; the failure is downstream at the CM JOIN. */ + READMIT_JOIN_ABANDONED, /* vms-0425 (§4(O.24)): OVMX DROVE its op 0x02 join request to + * THIS peer (joiner_cfg2_sent) and the peer returned ZERO CM + * responses -- no op 0x03 COMMIT. The wire diff of a same-pod + * first-join (Rf) vs return (Rr) for one identity showed the + * coordinator RELAYS (op 0x12) + COMMITS (op 0x03) a FIRST join + * but, for a RETURNING identity whose CSB it has just reclaimed, + * receives the identical op 0x02 and starts NO ADD transition -- + * no op 0x12 relay to the other member, no op 0x03 to OVMX (Davis + * p. 7-38: the coordinator IGNORES the join request when its + * admission tests are not satisfied, and "no state transition + * occurs"). This is SHARPER than RECLAIMED_NOJOIN: it is the + * verdict for the member OVMX actually SENT the join request to + * (the coordinator), naming that the frontier is the coordinator + * abandoning an op02-DRIVEN join -- not an OVMX op02 omission. */ READMIT_ENGAGED_NC, /* member sent CM responses but membership not latched */ READMIT_ADMITTED, /* member ran the JOIN handshake and OVMX latched membership */ }; @@ -11430,6 +11444,20 @@ static enum readmit_verdict readmit_verdict_of(const struct peer_state *ps) if (ps->cm_responses > 0) { return READMIT_ENGAGED_NC; } + /* vms-0425 (spec §4(O.24)): OVMX drove its op 0x02 join request to THIS peer + * (the deferred config/topology went out -- joiner_cfg2_sent) and got ZERO CM + * responses back. OVMX sends op 0x02 only to the coordinator it selects + * (Davis p. 7-37/38 "highest protocol level"), so this is the coordinator, and + * the wire diff (Rf op 0x02 -> op 0x12 relay + op 0x03 commit; Rr the SAME op + * 0x02 -> no relay, no commit) proves the coordinator ABANDONS the driven join + * for a returning identity. Name it distinctly from RECLAIMED_NOJOIN (a member + * we only REACHED) and NO_ENGAGE: here OVMX did its part -- it drove the join + * request -- and the coordinator did not commit. Checked before the open latch + * because OVMX's own VMS$VAXcluster VC often ends CONNSTUCK (DISC SENT) on the + * return, so the latch under-reports; the op02-was-driven fact does not. */ + if (ps->joiner_cfg2_sent) { + return READMIT_JOIN_ABANDONED; + } /* vms-944 (spec §4(O.23)): SYSAP re-opened (latch set) but ZERO CM responses. * §4(O.22) read this exact shape (bracket `4dBr`) as a "residual/stale * reached-OPEN" and folded it into NO-ENGAGE. The `944B2r` SDA bracket @@ -11451,6 +11479,7 @@ static const char *readmit_verdict_name(enum readmit_verdict v) case READMIT_NO_CHANNEL: return "NO-CHANNEL(never reached transport)"; case READMIT_NO_ENGAGE: return "NO-ENGAGE(member sent 0 CM responses, SYSAP never re-opened -- returning-id non-admission, spec 4(O.21))"; case READMIT_RECLAIMED_NOJOIN: return "RECLAIMED-NOJOIN(SYSAP re-opened + member reclaimed the CSB & read our incarnation, but ran 0 CM JOIN -- relocated frontier, spec 4(O.23))"; + case READMIT_JOIN_ABANDONED: return "JOIN-ABANDONED(OVMX drove op 0x02 to the coordinator, coordinator ran 0 CM JOIN -- no op 0x12 relay, no op 0x03 commit; it IGNORES the returning-id join request, Davis p.7-38 -- relocated frontier, spec 4(O.24))"; case READMIT_ENGAGED_NC: return "ENGAGED-NOT-LATCHED(CM responses seen, membership not OPEN)"; case READMIT_ADMITTED: return "ADMITTED(member ran JOIN handshake, membership OPEN)"; default: return "?"; @@ -11718,6 +11747,7 @@ static void scsd_exit_summary(struct scsd_rx *rx, FILE *out) if (getenv("OVMX_NO_READMITMAP") == NULL) { unsigned long long incn = (unsigned long long)ovmx_incarnation_time(); int reached = 0, admitted = 0, engaged = 0, no_engage = 0, reclaimed_nojoin = 0; + int join_abandoned = 0; for (int i = 0; i < OVMX_MAX_PEERS; i++) { if (rx->peers[i].pb == NULL) { continue; @@ -11733,6 +11763,8 @@ static void scsd_exit_summary(struct scsd_rx *rx, FILE *out) engaged++; } else if (v == READMIT_RECLAIMED_NOJOIN) { reclaimed_nojoin++; + } else if (v == READMIT_JOIN_ABANDONED) { + join_abandoned++; } else if (v == READMIT_NO_ENGAGE) { no_engage++; } @@ -11749,13 +11781,22 @@ static void scsd_exit_summary(struct scsd_rx *rx, FILE *out) fprintf(out, " READMITMAP-SUMMARY incarnation_presented=0x%016llx%s" " members_reached=%d admitted=%d engaged=%d" - " reclaimed_nojoin=%d no_engage=%d --" - " %s (spec 4(O.21)/4(O.23), docs/design-rejoin-cm-state-map.md)\n", + " reclaimed_nojoin=%d join_abandoned=%d no_engage=%d --" + " %s (spec 4(O.21)/4(O.23)/4(O.24), docs/design-rejoin-cm-state-map.md)\n", incn, incn ? "(live)" : "(frozen/template)", - reached, admitted, engaged, reclaimed_nojoin, no_engage, + reached, admitted, engaged, reclaimed_nojoin, join_abandoned, no_engage, (reached > 0 && admitted == 0 - && (no_engage + reclaimed_nojoin) == reached) - ? (reclaimed_nojoin > 0 + && (no_engage + reclaimed_nojoin + join_abandoned) == reached) + ? (join_abandoned > 0 + ? "RETURNING-IDENTITY NON-ADMISSION (RELOCATED FRONTIER, spec" + " 4(O.24)): OVMX DROVE its op 0x02 join request to the" + " coordinator and the coordinator ran ZERO CM JOIN -- no op" + " 0x12 relay, no op 0x03 commit. It is NOT an op02 omission" + " (OVMX did its part); the coordinator IGNORES/ABANDONS the" + " returning identity's join request (Davis p.7-38). Next" + " isolation: SDA on the COORDINATOR's CLUB/nodemap to see why" + " it declines an op02-driven ADD for a reclaimed identity" + : reclaimed_nojoin > 0 ? "RETURNING-IDENTITY NON-ADMISSION (RELOCATED FRONTIER, spec" " 4(O.23)): member(s) RECLAIMED the residual CSB and re-opened" " SCS -- the reclaim + incarnation-read path works -- but ran" diff --git a/tests/vmsscs/test_scsd_wire.c b/tests/vmsscs/test_scsd_wire.c index 7987b2b19..308de775d 100644 --- a/tests/vmsscs/test_scsd_wire.c +++ b/tests/vmsscs/test_scsd_wire.c @@ -10243,12 +10243,48 @@ static void test_readmit_verdict_classifies_the_rejoin_frontier(void) "RECLAIMED-NOJOIN must never read ADMITTED (INV-6: no success from a" " member that ran 0 CM JOIN handshakes)"); - /* And the genuine never-re-opened non-admission stays NO-ENGAGE. */ + /* vms-0425 (spec §4(O.24)): the SHARPEST non-admission verdict -- OVMX DROVE + * its op 0x02 join request to THIS peer (joiner_cfg2_sent=1) and got ZERO CM + * responses back. The same-pod first-join(Rf)-vs-return(Rr) wire diff for one + * identity (OVXR40, vaxlab-1) showed the coordinator RELAYS (op 0x12) + COMMITS + * (op 0x03) the FIRST join but, for the RETURNING identity whose CSB it just + * reclaimed, receives the SAME op 0x02 and starts NO ADD transition -- no relay, + * no commit (Davis p.7-38: the coordinator IGNORES the join request when its + * admission tests are not satisfied). This is the member OVMX actually sent the + * join request to (the coordinator); JOIN-ABANDONED names that the frontier is + * the coordinator abandoning an op02-DRIVEN join -- NOT an OVMX op02 omission. + * + * Checked BEFORE the open latch on purpose: on the return OVMX's own + * VMS$VAXcluster VC often ends CONNSTUCK (DISC SENT) so vaxcluster_open_reached + * under-reports, but the op02-was-driven fact does not. Fail-pre/pass-post gate: + * before vms-0425 this shape read RECLAIMED-NOJOIN (open latched) or NO-ENGAGE + * (latch missed, the actual Rr shape). */ + ps->cm_responses = 0; + ps->joiner_cfg2_sent = 1; + ps->vaxcluster_open_reached = 1; + CHECK(readmit_verdict_of(ps) == READMIT_JOIN_ABANDONED, + "OVMX drove op 0x02 to the coordinator (joiner_cfg2_sent) with 0 CM" + " responses back must read JOIN-ABANDONED (spec 4(O.24)), got %s", + readmit_verdict_name(readmit_verdict_of(ps))); + /* The real Rr shape: op02 driven, but OVMX's VC ended CONNSTUCK so the open + * latch never fired. Still JOIN-ABANDONED -- the driven-op02 fact governs. */ + ps->vaxcluster_open_reached = 0; + CHECK(readmit_verdict_of(ps) == READMIT_JOIN_ABANDONED, + "op 0x02 driven + 0 CM responses must read JOIN-ABANDONED even when the" + " open latch missed (VC CONNSTUCK), got %s", + readmit_verdict_name(readmit_verdict_of(ps))); + CHECK(readmit_verdict_of(ps) != READMIT_ADMITTED, + "JOIN-ABANDONED must never read ADMITTED (INV-6)"); + + /* And the genuine never-re-opened, op02-never-driven non-admission stays + * NO-ENGAGE (reset joiner_cfg2_sent: a member we only REACHED, never drove the + * join request to -- e.g. the non-coordinator). */ ps->cm_responses = 0; + ps->joiner_cfg2_sent = 0; ps->vaxcluster_open_reached = 0; CHECK(readmit_verdict_of(ps) == READMIT_NO_ENGAGE, - "transport up but SYSAP never re-opened and 0 CM responses must read" - " NO-ENGAGE (spec 4(O.21)), got %s", + "transport up but SYSAP never re-opened, 0 CM responses, op02 never driven" + " must read NO-ENGAGE (spec 4(O.21)), got %s", readmit_verdict_name(readmit_verdict_of(ps))); }