Skip to content

vms-0425: rejoin — sub-question settled + CM JOIN gate bisected (spec §4(O.24)) #314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
baron-3dl merged 1 commit into from
Aug 11, 2026
Merged

vms-0425: rejoin — sub-question settled + CM JOIN gate bisected (spec §4(O.24)) #314

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/clean-room/tools/cmdiff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
# vms-0425: decode the CM (membership/config) dialogue on the VMS$VAXcluster
# connection, per member, so a first-join capture can be diffed against a
# return capture. Offsets from scs_member_parse (scs_member.c): category=body[8]
# =frame[80], opcode=frame[81], conids frame[64:72], SYSAP name frame[76:92].
import os, sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from pcap import frames, is6007

VAX1 = bytes.fromhex('aa0004000104')
def le16(b,o): return b[o]|(b[o+1]<<8)
def le32(b,o): return b[o]|(b[o+1]<<8)|(b[o+2]<<16)|(b[o+3]<<24)
def name_at(p,o): return p[o:o+16].split(b'\x00')[0].decode('latin1','replace')

CAT={0x01:'CONFIG',0x02:'DLM',0x04:'ACK',0x06:'MEMBSHIP',
0x81:'CONFIG.r',0x82:'DLM.r',0x84:'ACK.r',0x86:'MEMBSHIP.r'}
OP ={0x14:'MODEL',0x01:'PARAMS',0x02:'op02-CONFIG',0x03:'op03-COMMIT',
0x05:'op05-LOCKRB',0x06:'op06-MEMB',0x09:'op09-XITION',0x0a:'op0a-GO',
0x12:'op12-RELAY',0x0d:'op0d-DEPART',0x04:'op04-ABORT',0x08:'op08',0x00:'-'}

path=sys.argv[1]
fr=[(t,p) for (t,p) in frames(path) if is6007(p) and len(p)>=82]
if not fr: sys.exit('no 6007 frames')
t0=fr[0][0]
# detect MACs
macs={}
for t,p in fr:
macs[p[6:12]]=macs.get(p[6:12],0)+1
others=[m for m in macs if m!=VAX1]
# OVMX = the non-DEC-ish, but just label: VAX1, and the rest by hex
def who(mac):
if mac==VAX1: return 'VAX1'
if mac[:3]==bytes.fromhex('08002b'): return 'VAX2'
return 'OVMX'

# map (conid-pair) -> sysap from connect frames (msgtype 0x5b/0x4b carrying a name)
SYSAPS=[b'VMS$VAXcluster',b'SCS$DIRECTORY',b'MSCP$DISK',b'MSCP$TAPE',b'SCS$DIR_LOOKUP',b'VMS$DISK_CL_DRVR']
conid_sysap={}
for t,p in fr:
nm=p[76:92].split(b'\x00')[0]
if nm in SYSAPS:
rc=le32(p,64); lc=le32(p,68)
conid_sysap[(rc,lc)]=nm.decode()
conid_sysap[(lc,rc)]=nm.decode()

print(f"# {path}")
print(f"# MACs: " + ", ".join(f"{who(m)}={m.hex()}({macs[m]})" for m in macs))
print(" t src>dst mt cat op rconid lconid sysap")
for t,p in fr:
mt=p[30]
if mt not in (0x4b,0x5b): continue
cat=p[80]; op=p[81]
# only CM-category frames (config/membership/ack and their responses)
if (cat & 0x7f) not in (0x01,0x04,0x06): continue
src=who(p[6:12]); dst=who(p[0:6])
rc=le32(p,64); lc=le32(p,68)
sysap=conid_sysap.get((rc,lc),'?')
# focus on the cluster membership connection
print("%8.3f %-4s>%-4s %02x %-9s %-13s %08x %08x %s"%(
t-t0, src, dst, mt, CAT.get(cat,'%02x'%cat), OP.get(op,'%02x'%op), rc, lc, sysap))
70 changes: 70 additions & 0 deletions docs/cluster-protocol-spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5632,6 +5632,76 @@ scsd-944B1f.log`/`scsd-944B2r.log` + `d94-944B1f/944B2r.pcap`; the wire incarnat
(`readmit_verdict_of`, `READMIT_RECLAIMED_NOJOIN`); unit `tests/vmsscs/test_scsd_wire.c`; design
record `docs/design-rejoin-cm-state-map.md` §7; *VAXcluster Principles* pp. 7-23/7-24/7-25/7-29/7-30/7-37/7-39.

#### 4(O.24) The §4(O.23) sub-question is SETTLED and the CM JOIN gate is BISECTED on the wire: a REALISTIC near-now incarnation reclaims the CSB IDENTICALLY to the future pin (so the frontier stays at the CM JOIN transition, NOT at "make the member see a new incarnation"); and a same-pod first-join-vs-return op02 diff proves OVMX DOES drive op 0x02 to the coordinator on the return — the coordinator receives it and starts NO ADD transition (no op 0x12 relay, no op 0x03 commit), running a member↔member reconcile + ABORT instead, so the gate is the COORDINATOR ignoring an op02-DRIVEN join for a returning identity (Davis p. 7-38), NOT an OVMX op02 omission (GROUNDED, `vms-0425`, same-boot single-factor SDA bracket + Rf/Rr CM-frame wire diff on `vaxlab-1` + *VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39, 2026-08-11)

**Frame.** §4(O.23) relocated the frontier to "reclaim succeeds, CM JOIN does not run" and left ONE
open sub-question: its reclaim was shown with a **blatantly future** incarnation pin (1-OCT-2026,
for SDA day-resolution visibility). Does a **realistic near-now** incarnation — what OVMX presents
normally — ALSO reclaim, or is the reclaim an artifact of a coarse delta (which would move the
frontier to "make the member see a new incarnation")? `vms-0425` settles it, then bisects the JOIN
gate on the wire.

**Sub-question SETTLED (same boot, single factor).** Two first-join→return pairs on ONE
freshly-booted `vaxlab-1`, two brand-new identities, first-join op02 form constant, `OVMX_JOIN_SEQ=1`,
VAX1 in SDA per arm — the pairs differ in **exactly one factor**, the return's incarnation:

| arm | identity | return incarnation | member's CSB for it (SDA) | `XITDONE` |
|---|---|---|---|---|
| `0425Rf` | `OVXR40`/1975 | — (first join) | built `8796AE80` → `member,selected` | **1** |
| `0425Rr` | `OVXR40` | default live (near-now) | `8796AE80` DEALLOCATED → NEW `879DF540`, `02040000 status_rcvd` (non-member) | **0** |
| `0425Ff` | `OVXF40`/1976 | — (first join) | built `879DEF00` → `member,selected` | **1** |
| `0425Fr` | `OVXF40` | `1-OCT-2026` | `879DEF00` DEALLOCATED → NEW `879DF2C0`, Incarnation `11-AUG → 1-OCT`, `02040000 status_rcvd` (non-member) | **0** |

The realistic return (`Rr`) and the future-pinned return (`Fr`) produce **byte-identical CSB
outcomes**: both DEALLOCATE the residual CSB and REBUILD a fresh one at a new address (Davis
pp. 7-24/7-25), both reach `02040000 status_rcvd` — the exact intermediate state a SUCCESSFUL first
join passes through — and both **stall there, never reaching `member,selected`** (`XITDONE=0`). Only
the SDA-displayed Incarnation *date* differs. So a realistic incarnation reclaims exactly as the
future pin does; §4(O.22)'s "never reclaims" was a `SHOW CLUSTER`-string artifact. **The frontier
does NOT relocate — it stays at the CM JOIN transition.**

**CM JOIN gate BISECTED (Rf-vs-Rr CM-frame wire diff, same identity/pod).** Decoding the
`VMS$VAXcluster` CM frames (category `body[8]`, opcode `body[9]`; OVMX `b2df5fa41233`, coordinator
VAX2 `08002b45161a`): on the FIRST join OVMX sends op 0x02 → the coordinator RELAYS it (op 0x12) to
the other member and COMMITS (op 0x03) to OVMX → op 0x05/op 0x06 → joined. On the RETURN **OVMX
still sends op 0x02** (its own log: `SCSD-I-CMCONFIG2 … sent DEFERRED op 0x02 … to node 2 … expect
its op 0x03 COMMIT`), correctly addressed — but the coordinator sends **no op 0x12 relay and no op
0x03 commit**; it runs a member↔member cat-0x06 MEMBERSHIP reconcile that EXCLUDES OVMX, then a
transition ABORT, and OVMX's `VMS$VAXcluster` VCs end CONNSTUCK (`still-open-at-exit=2`). This
**bisects the either/or**: it is NOT that OVMX omits op 0x02 (it drives it), and NOT that the member
waits silently (it actively runs a different transition). Per Davis p. 7-38 the coordinator, on a
join request, runs admission/quorum tests and **"ignores the request … and no state transition
occurs"** when they fail; a FIRST join passes them, a RETURN (against a member that just reclaimed
OVMX's CSB) does not. **The gate is the coordinator ignoring an op02-driven ADD for a returning
identity** — one step in from §4(O.23)'s "CM JOIN does not run".

**Relocated frontier.** The next isolation is **SDA on the COORDINATOR** (the member OVMX sends op
0x02 to), across departure+return, to see why it declines an op02-driven ADD for a reclaimed
identity it admits when fresh — i.e. whether its CLUB nodemap / quorum-vote accounting still carries
residual state for the departed OVMX so its p. 7-38 tests fail. A separate OVMX-side defect surfaced
(not the gate): on the return OVMX re-drives its add-member burst with `remote_conid=0x00000000` and
leaves its `VMS$VAXcluster` VCs CONNSTUCK.

**Non-claims / what ships (no wire change, guard 8).** (1) rejoin is NOT closed; this SETTLES the
sub-question and BISECTS the gate. (2) The READMITMAP verdict gains `JOIN-ABANDONED`
(`joiner_cfg2_sent && cm_responses==0`), split out of `RECLAIMED-NOJOIN`/`NO-ENGAGE`, naming the
member OVMX actually DROVE the join request to (the coordinator) distinctly from a member it only
reached — checked before the open latch because OVMX's own VC ends CONNSTUCK on the return (so the
latch under-reports; the live `0425Rr` read `NO-ENGAGE` under the old latch-first classifier, masking
that op 0x02 had been driven). Log-only, kill-switch `OVMX_NO_READMITMAP` unchanged, fail-pre/pass-post
case in `tests/vmsscs/test_scsd_wire.c`, byte-unchanged on the wire.

**Evidence** (host, tank volume): `/data/training/vax/cluster/work/0425{Rf,Rr,Ff,Fr}.csb` (SDA CSB
timelines) with `.status` (`Rf`/`Ff` `XITDONE=1`; `Rr`/`Fr` `XITDONE=0`);
`/lab/k8s-labs/vaxlab-1/logs/scsd-0425{Rf,Rr,Ff,Fr}.log` + `d94-0425{Rf,Rr,Ff,Fr}.pcap` (identities
`OVXR40`/1975, `OVXF40`/1976 on the wire); CM-frame decoder `docs/clean-room/tools/cmdiff.py`; code `src/vmsscs/scsd.c`
(`readmit_verdict_of`, `READMIT_JOIN_ABANDONED`); unit `tests/vmsscs/test_scsd_wire.c`; design record
`docs/design-rejoin-cm-state-map.md` §8. A live re-run with the refined daemon (`0425Rr2`,
`scsd-0425Rr2.log`) reads the new verdict on the wire: coordinator `08:00:2b:45:16:1a`
`verdict=JOIN-ABANDONED`, the non-coordinator `NO-ENGAGE`, SUMMARY `join_abandoned=1 no_engage=1`
(the old latch-first classifier read both `NO-ENGAGE`, masking the op02-driven fact).
*VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39.

## 5. Summary of unknown/inferred fields (RE gaps)

For visibility, every field NOT marked GROUNDED above:
Expand Down
100 changes: 100 additions & 0 deletions docs/design-rejoin-cm-state-map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -470,3 +470,103 @@ timelines above), `944B1f.status` (`XITDONE=1`) / `944B2r.status` (`XITDONE=0`),
`/lab/k8s-labs/vaxlab-1/logs/scsd-944B1f.log`/`scsd-944B2r.log` + `d94-944B1f/944B2r.pcap`. Wire
`[66:74]` incarnation distinct per run (arm A `0x00bc0cd311bc6c62`, arm B′ `0x00bc0cd380815b38`).
*VAXcluster Principles* pp. 7-23/7-24/7-25/7-29/7-30/7-37/7-39.


## 8. Sub-question settled, and the CM JOIN gate bisected on the wire — OVMX DRIVES op 0x02, the coordinator ABANDONS it (`vms-0425`, spec §4(O.24))

§7 relocated the frontier to "reclaim succeeds, CM JOIN does not run" and left ONE open
sub-question: §7's reclaim was demonstrated with a **blatantly future** incarnation pin
(1-OCT-2026, chosen for SDA day-resolution visibility). Does a **realistic near-now**
incarnation — what OVMX presents in normal operation — ALSO trigger the reclaim, or is the
reclaim an artifact of a coarse incarnation delta (in which case the true frontier would be
"make the member see a new incarnation", not the JOIN transition)? `vms-0425` settles it with a
same-boot single-factor bracket, then bisects the JOIN gate itself on the wire.

### 8.1 Sub-question SETTLED — a realistic incarnation reclaims identically to the future pin (GROUNDED, `vaxlab-1`, clean CN_2, 2026-08-11)

Two first-join→return pairs on ONE freshly-booted pod, two brand-new identities, first-join op02
form held constant, `OVMX_JOIN_SEQ=1`, hard `timeout` per op, VAX1 parked in SDA sampling each
identity's CSB (`csbwatch.sh`). The two pairs differ in **exactly one factor**: the incarnation
the RETURN presents.

| pair | arm | identity | return incarnation | member's CSB for the identity (SDA) | `XITDONE` |
|---|---|---|---|---|---|
| R (realistic) | `0425Rf` | `OVXR40`/1975 | — (first join) | CSB built `8796AE80`, advances to `member,selected` (02060002) | **1** |
| | `0425Rr` | `OVXR40` | **default live (near-now)** | old CSB `8796AE80` DEALLOCATED → **NEW CSB `879DF540`**, `1-JAN-2001` ref-time (a fresh CSB), Flags `02040000 status_rcvd` (**non-member**) | **0** |
| F (future) | `0425Ff` | `OVXF40`/1976 | — (first join) | CSB built `879DEF00`, advances to `member,selected` | **1** |
| | `0425Fr` | `OVXF40` | **`1-OCT-2026`** (`OVMX_INCARNATION_TIME=52975296000000000`) | old CSB `879DEF00` DEALLOCATED → **NEW CSB `879DF2C0`**, Incarnation visibly `11-AUG-2026 → 1-OCT-2026`, Flags `02040000 status_rcvd` (**non-member**) | **0** |

**The flip that is NOT there.** The realistic return (`0425Rr`) and the future-pinned return
(`0425Fr`) produce **byte-identical CSB outcomes**: the member DEALLOCATES the residual CSB and
REBUILDS a fresh one at a new address (a genuine reclaim, Davis pp. 7-24/7-25), which reaches
`02040000 status_rcvd` — the **exact** intermediate state a SUCCESSFUL first join passes through —
and then **stalls there, never advancing to `member,selected`**. The only observable difference is
the SDA-displayed Incarnation *date* (`11-AUG` vs `1-OCT`); the reclaim, the stall, and `XITDONE=0`
are the same. **A realistic incarnation reclaims exactly as the future pin does — the reclaim needs
no coarse delta.** §6.4's "never reclaims" was a `SHOW CLUSTER`-string artifact (SDA sees the
address change the console strings could not). The sub-question is closed: **the frontier does NOT
relocate; it stays at the CM JOIN transition.**

### 8.2 The CM JOIN gate bisected — OVMX DRIVES op 0x02; the coordinator does not START the ADD

Both the SUCCESS first-join (`0425Rf`) and the return (`0425Rr`) were captured (pcap, in-pod
`br0`). Decoding the CM (config/membership) frames on the `VMS$VAXcluster` connection — category
`body[8]`, opcode `body[9]`, per `scs_member_parse` — with the SAME identity on the SAME pod
(OVMX `b2df5fa41233`, VAX1 `aa0004000104`, VAX2 `08002b45161a`, VAX2 = the coordinator OVMX
selects) gives the discriminating diff:

| step | first join `0425Rf` (`XITDONE=1`) | return `0425Rr` (`XITDONE=0`) |
|---|---|---|
| OVMX → coordinator op 0x02 (join request) | **sent** (t≈29.57s) | **sent** (t≈31.47s) — *OVMX drives op 0x02 on the return too* |
| coordinator → other member op 0x12 RELAY | **sent** (relays the new member) | **absent** |
| coordinator → OVMX op 0x03 COMMIT | **sent** → OVMX 0x81-responds (`cm_responses++`) | **absent** (`cm_responses=0`) |
| what the coordinator does instead | op 0x05 lock-rebuild, op 0x06 membership burst → OVMX joins | a member↔member **cat-0x06 MEMBERSHIP** reconcile with the OTHER member that EXCLUDES OVMX, then a transition **ABORT** (op 0x04) |
| OVMX's own `VMS$VAXcluster` VCs at exit | OPEN | **CONNSTUCK in DISC SENT** (`still-open-at-exit=2`) |

**This bisects the either/or the frontier posed.** It is **not** that OVMX omits op 0x02 — OVMX's
own log confirms it drives the deferred op 0x02 in REJOIN form to the coordinator
(`SCSD-I-CMCONFIG2 … sent DEFERRED op 0x02 … to node 2 … expect its 0x04 ack then its op 0x03
COMMIT`) and correctly addresses it. And the member is **not** merely "waiting silently" — the
coordinator RECEIVES op 0x02 and actively runs a *different* transition (member↔member reconcile +
abort), starting **no** ADD for OVMX. Per Davis p. 7-38 the coordinator, on receiving a join
request, runs admission/quorum tests and **"ignores the request … and no state transition occurs"**
when they are not satisfied; the FIRST join satisfies them, the RETURN (against a member that has
just reclaimed OVMX's CSB) does not. The gate is the **coordinator's decision to ignore an
op02-driven ADD for a returning identity** — one step further in than §7's "CM JOIN does not run".

Note also an OVMX-side connection-hygiene defect surfaced by the same capture (not the gate, but
real): on the return OVMX's `VMS$VAXcluster` VCs end **CONNSTUCK** in `DISC SENT` and OVMX re-drives
its add-member burst with `remote_conid=0x00000000` (`SCSD-I-CMREADMIT … remote=0x00000000`). The
correctly-addressed op 0x02 (with the member's live conid) still drew no commit, so this is not the
gate — but it is a distinct bug worth its own item.

### 8.3 The relocated frontier, and what `vms-0425` ships (no wire change)

**Relocated frontier.** The next isolation is **SDA on the COORDINATOR** (the member OVMX sends
op 0x02 to — VAX2 here), across the departure and the return, to see WHY it declines to start an
ADD for a reclaimed identity that it admits when fresh: does its CLUB nodemap / quorum-vote
accounting still carry residual state for the departed OVMX (so its p. 7-38 admission tests fail),
even though the per-identity CSB was reclaimed? That is the named, testable next step — and it is a
COORDINATOR-side observation, a different oracle vantage than §4–§7's member-CSB SDA.

**What `vms-0425` ships (no wire change, guard 8 — the working first-join path is byte-unchanged).**
(1) This record (§8) and the spec §4(O.24) note. (2) The sub-question bracket above (realistic vs
future incarnation, same boot, single factor) closing §7's open sub-question. (3) A READMITMAP
**verdict refinement**: `JOIN-ABANDONED` (`joiner_cfg2_sent` && `cm_responses==0`) split out of
`RECLAIMED-NOJOIN`/`NO-ENGAGE`, so the member OVMX actually DROVE the join request to (the
coordinator) is named distinctly from a member it only reached — surfacing "OVMX did its part, the
coordinator abandoned" as one log line for the next isolation. It is checked BEFORE the open latch
because OVMX's own VC ends CONNSTUCK on the return (so `vaxcluster_open_reached` under-reports, but
the op02-was-driven fact does not — the live `0425Rr` READMITMAP read `NO-ENGAGE` under the old
latch-first classifier, masking that OVMX had driven op 0x02). Log-only, kill-switch
`OVMX_NO_READMITMAP` unchanged, with a fail-pre/pass-post case in `tests/vmsscs/test_scsd_wire.c`.
Still a non-admission (never `ADMITTED`; INV-6).

Evidence (host, tank volume): CSB timelines `/data/training/vax/cluster/work/0425{Rf,Rr,Ff,Fr}.csb`
with `.status` (`0425Rf`/`0425Ff` `XITDONE=1`; `0425Rr`/`0425Fr` `XITDONE=0`); daemon logs
`/lab/k8s-labs/vaxlab-1/logs/scsd-0425{Rf,Rr,Ff,Fr}.log` and pcaps `d94-0425{Rf,Rr,Ff,Fr}.pcap`
(identities `OVXR40`/1975, `OVXF40`/1976 on the wire); CM-frame decoder `docs/clean-room/tools/cmdiff.py`; code
`src/vmsscs/scsd.c` (`readmit_verdict_of`, `READMIT_JOIN_ABANDONED`); unit
`tests/vmsscs/test_scsd_wire.c`. Live pass-post with the refined daemon: `scsd-0425Rr2.log` reads
coordinator `verdict=JOIN-ABANDONED`, non-coordinator `NO-ENGAGE`, SUMMARY `join_abandoned=1
no_engage=1` (the old classifier read both `NO-ENGAGE`). *VAXcluster Principles* pp. 7-24/7-25/7-37/7-38/7-39.
Loading
Loading