vms-f6b: SYMMETRIC CN=2 — OVMX founds a cluster + admits a joiner to full membership (op-05 adopt, commit-at-GO) - #1119
Merged
Merged
Conversation
THE RIG. Two minimal Linux guests, each loading the REAL executive (vms.ko)
and issuing STARTUP.EXE's own two cluster ioctls -- VMS_IOCTL_SYSGEN_LOAD then
VMS_IOCTL_CLUSTER_START -- on one L2 segment carried by a QEMU socket netdev
inside a single pod's netns (no bridge, no NET_ADMIN, no NET_RAW; only
/dev/kvm). There is NO userspace SCS daemon: CLUSTER_START opens the LAN port
INSIDE the executive and every 0x6007 frame on the segment is the executive's.
The guest-side `cluster_node` issues ioctls and READS STATE BACK; it never
touches the wire, never parses a frame and never computes a CSID, a member
count or a quorum. Every printed value is a field read from the executive one
line earlier through three independent projections (CLUSTER_DIAG_CSB's CLUB,
CLUSTER_GETSYI, CLUSTER_MEMBER_GET), and the verdict prints whether they AGREE
rather than picking one (INV-6). A value the executive has not learned prints
"-", never 0.
THE LIGHT PATH WORKS. No distro boot, no ODS-2 system disk, no STARTUP.EXE:
insmod + REGISTER + SYSGEN_LOAD + CLUSTER_START is enough to bring PEA0:, SCS
and the connection manager up on the guest's virtio NIC.
WHAT IT MEASURED -- GENESIS HOLDS AT RUNTIME, on a real executive:
RIG-A-FINAL role=founder member=1 state=MEMBER csid=0x00010001 cn=1
quorum=1 cevotes=1 epoch=1 projections=agree
Node A (VOTES=1, EXPECTED_VOTES=1, VAXCLUSTER=2) spent its whole RECNXINTERVAL
discovery window hearing nobody and formed generation 1 -- "%CNXMAN, this node
has quorum by its own votes: forming an OpenVMS Cluster" -> "this node is a
member of the cluster" -> "completed VAXcluster state transition" -- and its
executive reports MEMBER with the minted CSID and epoch 1.
NEGATIVE CONTROL (same image, same pod, VOTES=0 on BOTH nodes): neither node
founds, neither reaches MEMBER, neither holds a CSID. The control is stronger
than "CN != 2" deliberately, because genesis is what this rung actually
measures. Both transcripts are committed under tests/qemu/captures/.
WHAT IT DID NOT REACH, HONESTLY: CN=2. Node B stays JOINING with no CSID, so
the rig FAILS its own verdict and says so. The join now runs deep -- both
nodes open the VMS$VAXcluster VC and EMIT their cat-0x01 op-0x02 membership
request (RIG-*-JOINREC kind=2 cat=0x01 op=0x02) -- but neither node's
coordinator is ever driven by the peer's request, so admission times out on
both sides. Beyond that sits a second, structural wall: OVMX has no op-0x06
MEMBERSHIP builder (its payload map is not grounded -- Rule 8), and op-0x06 is
the ONLY path by which a joiner learns the generation it needs for its own
CSID, which phase 2 needs to set the local CSB's MEMBER flag. Both are named
in the report, not worked around here.
TWO REAL DEFECTS THE RIG FOUND, AND FIXED:
1. vms_lan_rx_thunk() linearized a SHARED skb -- src/kernel/
exec_kbackend_linux.h. A packet_type handler is not the exclusive owner of
its skb: with anything else listening on the same NIC it arrives via
deliver_skb() with an extra reference, and pskb_expand_head() asserts
BUG_ON(skb_shared(skb)). MEASURED: with a passive AF_PACKET capture bound
to the same interface BOTH guests took a kernel panic ("kernel BUG at
net/core/skbuff.c:2138 ... vms_lan_rx_thunk [vms]") on the FIRST 0x6007
frame -- i.e. tcpdump on a clustered OVMX node crashed that node. Fixed
with skb_share_check(), the contract every in-tree ptype handler observes.
Proven by the same rig: 150 s clean with the capture bound, and an 81 KB
pcap per node where there used to be a panic.
2. cnxman_join_drive() downgraded a MEMBER to JOINING -- src/kernel-core/
vms_cnxman.c. The beat drives a join whenever a system is present and the
join FSM is idle, INCLUDING on a node that is already a member (either
side may open the VMS$VAXcluster connection, E67). The unconditional
`cl->state = VMS_CLUSTER_JOINING` therefore un-asserted a membership the
executive genuinely held: the local CSB still carried MEMBER and the CLUB
still carried the CSID, while the one cell SYI$_CLUSTER_MEMBER and SHOW
CLUSTER read said JOINING. MEASURED: node A founded the cluster, was
committed MEMBER by phase 2, and reported member=0 the instant node B was
powered on -- a cluster's founder and coordinator describing itself as not
a member of it. Only phase 2 may write MEMBER and only a real transition
may take it away; a join ATTEMPT is neither.
TWO RIG-LEVEL FINDINGS RECORDED IN THE CODE:
- CLUSTER_CREDITS is load-bearing. With it unset the port advertises a grant
of 0 at abs 95, the peer honours "no credit, no message" (p. 2-43) and the
circuit opens with NOTHING able to travel on it: measured as both VCs OPEN,
credits_send=0, zero sequenced frames either way. The executive's behaviour
was exactly right (it counts vc_credits_absent and refuses); the rig was
misconfigured. It now loads 32, VMS's own default.
- QEMU's socket,mcast= netdev loops a guest's own group multicast back to it,
and a node booted ALONE formed a channel with ITSELF off its own HELLO
("%PEA0, channel verified" with no peer on the segment). The rig uses a
point-to-point socket instead. That the port accepts a HELLO bearing its
own SCSSYSTEMID is a separate robustness observation, recorded not patched.
TESTS RUN. Full host ctest 247/247 pass, including a new source scan in
test_cnxman_glue.c pinning fix 2 (the guarded write present, the unguarded one
gone). cluster_core_includes_gate and cluster_wire_safety_gate pass. The Linux
vms.ko builds clean (it is what the rig boots) and the whole executive module
cross-compiles + relocatable-links for elf32-vax against real NetBSD/vax
kernel headers, ILP32 width-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN
…r can
admit a joiner, and the joiner reaches MEMBER on a wire-learned CSID
DELIVERABLE 1 -- docs/design-op06-membership-builder.md. Every field of the
cat-0x01 op-0x06 record, grounded or honestly omitted, re-decoded from the
captures by a script that uses NO project code: primary
tests/lab/captures/op06-join-20260903.pcap (255 real op-0x06 frames, VAX1
coordinator) cross-referenced against cn3-achieved-20260905.pcap (254 frames,
VAX2 coordinator, a DIFFERENT cluster and epoch -- and the burst OVMX itself
consumed to reach CN=3), with scs-tier0 grounding SCSSYSTEMID inside the
cluster-LOGICAL LAN address (aa:00:04:00:01:04 -> 0x0401 = 1025).
The whole 24-byte CM header is byte-identical across all 509 frames: category
0x01, opcode 0x06, role 0x20 (COMMIT), class 0x02 (ADD), body[18:20] zero --
and body[12:16] the sender's real transition epoch (6 in one capture, 5 in the
other). Form A's CSID at body[24:28] carries ONLY genuine CSIDs, 24/24 in each
capture, zero false positives.
THE :715 QUESTION, ANSWERED BY MEASUREMENT. That note forbids a zero-filled
op-0x06 on the premise that the burst carries a membership LIST, so zeros would
assert an empty cluster. E30 falsified that premise and the captures refute it:
23 of 255 real op-0x06 frames -- and 23 of 254 in the second capture -- carry an
ENTIRELY ZERO body[24:132], interleaved through the burst (first at position 5
of 255, then every ~11 frames), in joins that SUCCEEDED. A zero payload is a
shape the reference emits as a matter of course. The prohibition still stands
for the two opcodes that note also names, and neither gains a builder here.
DELIVERABLE 2 -- vms_cm_membership_build(). ONE frame per admission (the
reference's 254-frame burst is E78's crash vector), carrying the coordinator's
OWN real CSID read from club->local_csid at the form-A offset -- the offset the
receive path tries FIRST, so builder and reader share one form, enforced by a
round-trip test. A value failing the SAME shape test the reader applies is
VMS_CODEC_E_RANGE and no frame is built. Seven nonzero bytes in the whole 132,
asserted as such; the countdown, the incarnation and the sub-record body are
left zero and COUNTED (membership_fields_omitted).
MEASURED ON TWO REAL EXECUTIVES (rig on the k3s worker, KVM; every value read
back through CLUSTER_DIAG_CSB / _GETSYI / _MEMBER_GET):
RIG-A-FINAL role=founder member=1 MEMBER csid=0x00010001 cn=2 epoch=2
RIG-B-FINAL role=member-no-coordinator member=1 MEMBER csid=0x00010002 cn=1 epoch=2
node A: %CNXMAN, proposing addition of a system to the cluster
%CNXMAN, system 0000000000000402 was added to the cluster
node B: %CNXMAN, this node is a member of the cluster
Node B did not receive that CSID -- it read a GENERATION out of node A's
op-0x06 and computed 1<<16 | (1026 & 0x3ff) from its own SYSGEN state, and
phase 2 committed its membership off the nodemap bit node A really asserted.
TWO EXECUTIVE DEFECTS THE RIG FOUND ON THE WAY, both blocking any admission:
1. THE JOINER'S ADVERT HANDLER SWALLOWED THE op-0x02 MEMBERSHIP REQUEST.
join_h_peer_advert() consumed cat-0x01 op-0x02 as a "peer advert", and the
router offers a body to the join FSM FIRST -- so the coordinator's ONE
selection edge ([IDLE][RX_TR_REQUEST]; being asked is what MAKES a node the
coordinator, book pp. 7-37/7-38) was never reached. Measured: both nodes put
their op-0x02 on the wire for a whole run and neither ever proposed
anything. It now returns NOT_MINE for that one opcode -- op-0x01 is handled
there and op-0x14 has no other owner, so neither becomes an unroutable
frame.
2. Two silent refusal paths in the coordinator now say what they did once.
A transition that stalls in COMMIT used to leave no trace at all: the
counters are projected through no ioctl and the executive has no console
log of its own.
CSID ASSIGNMENT IS PROVISIONAL PENDING ORACLE (vms-3a7c), and is GATED rather
than kill-switched. coord_csid_unambiguous(): admit a subject only when the
round-robin CSV slot and the joiner's own SCSSYSTEMID & 0x3ff -- the two
candidate rules -- name it IDENTICALLY. Under that gate the CSID OVMX asserts is
correct whichever way the oracle settles, which is the only "safe toward a real
VAX" available before the answer exists. A disagreement is a named refusal with
nothing stamped and nothing emitted. The note lays out both candidate rules,
what the captures ground (every real CSID is consistent with BOTH -- the lab's
SCSSYSTEMIDs are consecutive from 1025, so no capture can separate them) and the
single lab run that would.
THREE RIG MODES, all three RUN:
proof A founds, admits B; both MEMBER; A cn=2, B cn=1 (see below) -- the
rig FAILS its own strict verdict and says precisely what held.
ambig node B's SCSSYSTEMID 1030 makes the two rules disagree: node A refuses
("cannot be assigned unambiguously"), sends NO op-0x06, and node B
stays role=none member=0 csid=- JOINING. HELD. This is the INV-6
control on the joiner -- same code, same wire, one SYSGEN digit apart.
negctl VOTES=0 on both: nobody founds, nobody reaches MEMBER, nobody holds a
CSID. HELD.
THE REMAINING GAP, STATED NOT PAPERED OVER (note sec 8). Node B counts 1 member,
not 2: phase 2 can only match a CSB to a nodemap bit if that CSB's CSID is
known, and no grounded wire field associates a PEER's SCSSYSTEMID with its CSID.
B's own executive says so -- "%CNXMAN, committed member count differs from the
transition nodemap". B cannot take the CSID off the op-0x06 it just read,
because E30 and the captures are explicit that a burst carries the sender's own
CSID OR another member's: in cn3 the sender is VAX2 (1026) and its form-A record
carries VAX1's 0x00010001. Attributing it to the sender is a fabrication the
capture refutes, so it is NOT done. Two candidate closures that would work
inside an OVMX-coordinated cluster but assert the unresolved vms-3a7c rule about
a system this node was never told about are written down and ESCALATED, not
implemented; the clean answer is one lab capture, which would settle vms-3a7c in
the same run.
TESTS RUN. Host ctest 247/247. New: test_membership_build (builder bytes,
refusal, round-trip through this codec's own reader, every-nonzero-byte count),
test_membership_record_is_built_from_real_state (the emitted frame's grounded
bytes and its zeros, against real CLUB state) and
test_ambiguous_csid_assignment_refuses. test_cnxman_coord's bed now carries REAL
SCSSYSTEMIDs consistent with its own CSID constants (1025/1026/1027/1028) rather
than LAN-address-shaped 48-bit values for which slot != sysid & 0x3ff -- a
fixture made faithful, with every assertion unchanged. cluster_core_includes_gate
and cluster_wire_safety_gate pass. Linux vms.ko builds (it is what the rig
boots) and the whole executive module cross-compiles + relocatable-links for
elf32-vax against real NetBSD/vax headers, ILP32 width-clean.
NO REAL-VAX EMISSION. The rig is all-OVMX and emits toward no VAX. Nothing here
is claimed interop-verified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN
…d why
The hypothesis was attractive: cn3-achieved-20260905.pcap admits OVMXJ1, whose
SCSSYSTEMID 1986 is non-consecutive (1986 & 0x3ff = 962 = 0x3C2 vs next free CSV
slot 3), so if the coordinator's assignment for it were visible the two rival
rules would predict 0x00010003 vs 0x000103C2 and vms-3a7c would be over from
existing data. Decoded. It is not visible, for three independent reasons, each
measured:
1. NO PRE-JOIN BASELINE. cn3 contains ZERO cat-0x01 op-0x06 frames before
OVMXJ1 first speaks (t+15.247s). All 254 are VAX2 -> OVMXJ1 and begin at
t+18.933s BECAUSE of the join. All three CSIDs (0x00010001/2/3) first appear
within ONE MILLISECOND of each other in the burst's opening frames, so
"a third CSID appears when the new member arrives" is unobservable -- every
CSID appears then. The temporal discriminator has nothing to discriminate
against.
2. 0x00010003 IS NOT OVMXJ1's, AND THE SIBLING CAPTURE PROVES IT.
op06-join-20260903.pcap has only TWO stations on the wire -- 1025 and 1026,
no third VAX at all -- and its burst nonetheless asserts 0x00010003 x46 at
the grounded form-B offset, the IDENTICAL multiplicity cn3 shows. That
record therefore belongs to a member of the coordinator's own table that is
ABSENT from the segment (p. 7-25's retained CSB), and cn3 re-asserts the
same foreign record. The elimination argument ("three stations, three
CSIDs, so the third is the joiner's") fails on this.
3. A BURST NEVER CARRIES ITS RECIPIENT'S OWN CSID -- E30 from a second angle:
VAX1 -> VAX2 : 0x00010001 x47, 0x00010003 x46 (0x00010002 ABSENT)
VAX2 -> OVMXJ1 : 0x00010001 x47, 0x00010002 x23, 0x00010003 x46
(OVMXJ1's ABSENT)
So OVMXJ1's assigned CSID is not in cn3 at all, under either rule.
The negative check is inconclusive, not confirmatory: 0x000103C2 occurs 0 times
in all 11,478 SCA frames of cn3 -- but since the burst structurally omits the
recipient's own CSID, that absence is exactly what BOTH rules predict.
(A bare value-scan for 0x00010003 across all frames finds 302 hits, but 242 sit
at abs 82 = body[10], the uninitialised-residue span, and the first is 6.8s
BEFORE OVMXJ1 exists -- which is its own warning against attributing by value.)
WHAT THIS BUYS ANYWAY: the required lab capture is now sharply specified rather
than merely named. It must (a) capture the coordinator's burst toward an
ALREADY-PRESENT member, never toward the joiner -- point 3 is why -- and (b)
include a PRE-ADMISSION baseline, which cn3 lacks entirely. With the new node's
SCSSYSTEMID & 0x3ff chosen NOT to equal the next free CSV slot, that one run
settles vms-3a7c AND grounds the {SCSSYSTEMID -> CSID} association sec 8 needs
for symmetric CN=2.
Note-only; no code changed. The provisional round-robin rule and the sec 5
ambiguity gate stand exactly as they were.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN
… of the
same nodemap decode (ROUND-ROBIN CSV SLOT, settled)
Decode only; no code changed. Two captures re-decoded with a script that uses
no project code.
=== 1. The cn3 note's premise was scoped wrong, and the error mattered ===
cn3-achieved-20260905.md says "no cat 0x01 op=0x0c (barrier release) frame ever
appears". Whole-capture census: cn3 contains 01/0b x13 and **01/0c x12**. They
belong to a SECOND transition the note does not mention -- the epoch-6
class-0x03 REMOVE at t+1584.23s, VAX1 coordinating and VAX2 participating after
OVMXJ1 departed -- which walks the canonical barrier in full: 01/0b step N ->
81/0b ack -> 01/0c release N, N = 1..12, step index in body[16:20], no gaps.
op06-join shows the same 12-step walk for a class-0x02 ADD (frames 450-1107).
The SCOPED claim -- nothing toward OVMXJ1 -- stands, and is corrected in place.
=== 2. What actually commits the membership: PHASE 2 AT THE GO ===
The cn3 epoch-5 ADD really did stall its barrier: nodemap {1,2,3} -> M=3 -> 24
op-0b expected, exactly ONE observed (OVMXJ1's step 1, acked at frame 856, never
released). VAX1 sent no op-0b for epoch 5, and that is not a capture blind spot
-- the whole epoch-6 barrier between VAX1 and VAX2 is in the same file.
The membership committed anyway: VAX1 took the GO at frame 850 (t+18.9821s) and
counted CN_3 from then, sustained to t+600s, then ran a clean epoch-6 REMOVE 26
minutes later. That is p. 7-42 exactly -- Phase 2 (nodemap into the CSBs,
quorum, the count, the CLUSTER flag) runs at the GO; the 12-step barrier is the
lock-rebuild synchronisation that FOLLOWS the commit.
VERDICT: OVMX neither over- nor under-models this. barrier_h_go() already calls
barrier_commit_phase2() ("the count commits HERE ... p. 7-42") and
phase2_commit_local_membership() sets cl->state = MEMBER there. Only the join
FSM's OWN promotion waits for op-0c #12 (E79's deliberate choice) -- a narrower
thing than "am I a member", worth a decision but not a silent change.
NO CODE FIX WARRANTED from this decode.
E85's hypothesis that the barrier gates on cat-02 op-0d rebuild records is
REFUTED: cn3 has zero 02/0d frames in the entire capture, yet its epoch-6
barrier walked all 12 steps. Why VAX1 skipped the epoch-5 barrier is genuinely
AMBIGUOUS on the wire; the run that would settle it is a 3-VAX ADD with no OVMX
present (24 steps => OVMX implicated; early stop => M>=3 ADDs differ).
=== 3. vms-3a7c IS SETTLED -- by the same nodemap, by causality ===
The answer was never in the op-06 burst; it is in the transition-open NODEMAP,
whose bits ARE the CSV slots:
epoch 5 frame 273 VAX2->VAX1 op-09 ADD body[55]=0x0e = {1,2,3}
epoch 5 frame 834 VAX2->OVMXJ1 op-09 ADD body[55]=0x0e = {1,2,3}
epoch 6 frame 11347 VAX1->VAX2 op-08 REMOVE body[55]=0x06 = {1,2}
Three stations ever on that segment (1025, 1026, 1986). Bit 3 is present in
EXACTLY the transition that admits OVMXJ1 and absent in EXACTLY the one that
removes it; no other station enters or leaves. So slot 3 is OVMXJ1's -- and
OVMXJ1's SCSSYSTEMID is 1986, whose & 0x3ff is 962, a bit index the nodemap byte
cannot even express.
=> real VMS assigns the ROUND-ROBIN CSV SLOT, not SCSSYSTEMID & 0x3ff.
Corroborated independently in op06-join (different cluster incarnation, only
1025 and 1026 on the wire): its ADD open carries nodemap 0x0a = {1,3}, and both
bits must be real members (VAX1 coordinates, VAX2 walks the 12 steps), so one of
1025/1026 holds a slot neither 1025&0x3ff=1 nor 1026&0x3ff=2 can produce.
CONSEQUENCES:
* OVMX's COORDINATOR (coord_next_slot round-robin) was already right -- no
change, and the vms-3a7c ambiguity gate can eventually be retired rather
than widened.
* OVMX's JOINER self-derive is now PROVEN WRONG: for OVMXJ1 it computes
0x000103C2 while the cluster assigned 0x00010003. With low word 962,
phase2_csb_in_nodemap() finds the slot inexpressible, answers "unknown", and
OVMX can never select itself into the map -- which is exactly why OVMX's own
executive could not have read MEMBER in the CN=3 run even while VAX1's
SHOW CLUSTER counted it. A real, grounded defect.
* But the decode does NOT say HOW a joiner should learn its slot (the nodemap
names slots, not systems). ESCALATED, not implemented.
=== 4. Two of my own earlier claims were WRONG; corrected in the note ===
(a) "0x00010003 belongs to a system absent from the segment" -- FALSE; in
op06-join slot 3 is held by one of the two VAXes that are present.
(b) "A burst never carries its recipient's own CSID" -- FALSE, and it was the
load-bearing step of the old sec 8 argument. op06-join's burst does carry
it. cn3's burst lacks OVMXJ1's record for a TEMPORAL reason: frames
245-833 precede the op-09 (834) that admitted it.
Host ctest still 247/247 (nothing built changed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN
… assigned
CSID, and SYMMETRIC CN=2 lands
RIG-A-FINAL role=founder member=1 state=MEMBER csid=0x00010001 cn=2 epoch=2 projections=agree
RIG-B-FINAL role=joiner member=1 state=MEMBER csid=0x00010002 cn=2 epoch=2 projections=agree
GENESIS 2-NODE PROOF PASSED
Node B now COUNTS TWO. Its own CSB table names both systems (1026 ->
0x00010002 self, 1025 -> 0x00010001 peer, both MEMBER) and its $GETSYI reads
member=1 nodes=2 -- three projections that agree, all read back out of the
executive (INV-6).
GROUNDING FIRST. The claimed op-0x05 layout was verified against every real
op-0x05 frame in both repository captures BEFORE anything was built on it --
5 in cn3-achieved-20260905.pcap and 3 in op06-join-20260903.pcap, 8/8 agreeing:
body[16:20] the constant 0x00000220, body[20:24] SCSSYSTEMID, body[28:36] boot
time, body[36:40] assigned CSID, body[40:42] index == (CSID & 0xffff) - 1.
The pairings independently re-confirm the round-robin rule (cn3: 1025->slot 1,
1026->slot 2, 1986->slot 3 although 1986 & 0x3ff = 962; op06-join: 1026->slot 3
although 1026 & 0x3ff = 2), and the distribution is on the wire too: cn3 frames
230-233 carry the FULL set to the joiner, frame 235 carries only the DELTA to
the present member.
(a) COORDINATOR. vms_cm_membership_rec_build/_parse -- one shared form, one
shared self-consistency predicate applied by builder and parser alike, so
what OVMX emits is by construction what OVMX accepts. Only grounded fields;
body[42:132] (uninterpreted stale buffer in the reference -- leftover
strings and VAX kernel pointers) is emitted ZERO and COUNTED, never
reproduced (Rule 8). coord_send_membership_set() distributes them the
reference's way. EXECUTIVE-BACKED: every field is read off the CSB the
record is about -- the SCSSYSTEMID the port learned, the CSID
coord_assign_slot() actually stamped, that CSID's own slot, that member's
real incarnation. A member this node holds no complete identity for gets NO
record (counted); a record that does not hold together is refused by the
codec and never becomes a frame.
(b) JOINER. join_adopt_membership_rec() adopts the CSID from the record naming
its own SCSSYSTEMID -- re-adopting on every admission, caching nothing
(p. 7-25: a rejoining system gets a NEW CSID; the oracle measured one
SCSSYSTEMID taking slot 4 then slot 5). Records about OTHER members are
filed on the block this CLUB already holds for that SCSSYSTEMID -- which is
precisely what lets a joiner COUNT the cluster -- and a record about a
system it holds no block for is counted and dropped, never invented.
The generation << 16 | (SCSSYSTEMID & 0x3ff) SELF-DERIVE IS GONE: it gave
OVMXJ1 low word 962 where the cluster had assigned slot 3, so
phase2_csb_in_nodemap() could never match it and OVMX could never select
itself into a cluster that had really admitted it.
(c) COMMIT AT THE GO (vms-9c99). The barrier gains phase2_commits, moved by
barrier_commit_phase2(), and the join FSM promotes on that -- the same
criterion membership and the coordinator already use. Grounded: cn3 shows a
real VAX committing an ADD and counting its joiner from the GO for 600 s
with no on-wire op-0x0c to that joiner at all, so the old op-0x0c #12
trigger was an interop hang.
ALSO REMOVED: the interim vms-3a7c "ambiguity gate" I added earlier, which
admitted a system only when the round-robin slot and SCSSYSTEMID & 0x3ff
agreed. With the rule settled and the joiner adopting rather than deriving it
is unnecessary -- and it was actively WRONG: the oracle's own behaviour
(1986 -> slot 3) is exactly a case it would have refused. op-0x05 is renamed
from VMS_CM_OP_LOCKRB to VMS_CM_OP_MEMBREC so no reader can hold the old
"lock/resource rebuild" meaning.
(d) MIRRORS: none required -- no new TU, no new ioctl, nothing crossing the shr
vector or vms_lock_nb.h. Verified.
CONTROLS, both run on the same image:
noderive node B's SCSSYSTEMID is 1030 (low ten bits = 6) and its executive
reports CSID 0x00010002 -- CSV slot 2, the slot the coordinator
ASSIGNED. A value it could not have computed, so the identity was
demonstrably adopted off the wire. HELD.
negctl VOTES=0 on both: neither founds, neither reaches MEMBER, neither
holds a CSID. HELD.
NOT CLAIMED, recorded rather than hidden: node B's quorum arithmetic is still
empty (quorum=0 cevotes=0) -- it counts both members but has not learned node
A's VOTES, which ride op-0x01 PARAMS, not the membership record. And node A
logs two "unroutable VMS$VAXcluster frame" lines: node B's 0x81/0x05 echoes,
which A's coordinator owns no edge for and says so rather than inventing one.
TESTS. Host ctest 250/250 (serial; opcom_record_body_gate times out only under
-j parallel load and passes standalone in 9.9 s -- pre-existing, unrelated).
New: test_membership_rec (builder/parser round-trip on the captures' own
vectors, the refusals, body[42:132] proven zero),
test_membership_records_are_projected_from_the_csbs (full-set/delta
distribution, and every emitted field compared against the CSB it is about),
test_slot_is_assigned_not_derivable, and four joiner cases -- adopt-when-named,
not-adopted-when-about-another (but filed on that peer), re-adopt-on-rejoin,
unusable-record-answered-not-adopted. The op-0x06 self-derive assertions were
re-pointed at the corrected behaviour, and the E79 promotion test now pins the
GO. cluster_core_includes_gate and cluster_wire_safety_gate pass; elf32-vax
cross-compiles + relocatable-links ILP32-clean; the Linux vms.ko is what the
rig boots.
Lab torn down (ovmx-genesis-rig + fetch pod deleted); vaxlab-* untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN
baron-3dl
force-pushed
the
work/vms-f6b-dlm-rig
branch
from
September 10, 2026 22:30
1b3a82b to
ce910e0
Compare
baron-3dl
changed the title
baron-3dl
marked this pull request as ready for review
September 10, 2026 22:31
baron-3dl
added a commit
that referenced
this pull request
Sep 10, 2026
…c CN=2); correct the CSID rule (#1131) * compat: connection-manager$genesis + $membership → verified (symmetric CN=2 on real /dev/vms); correct the CSID rule The symmetric-CN=2 proof (#1119) earns two verified rows for the OVMX^n founding + admission capability, and corrects claims the proof falsified: - ADD connection-manager$genesis (verified/real): a node founds a cluster from nothing when quorum is met by its own votes (authentic VMS formation, not the #1052-removed phantom default), proven at runtime on real /dev/vms + votes-gate negctl. - ADD connection-manager$membership (verified/real): the coordinator admits a joiner via op-05 membership records (executive-backed); the joiner ADOPTS its assigned CSID → symmetric CN=2, node B counts 2, real state both nodes; noderive + negctl controls. - CORRECT $join + the summary: the CSID low-word is a coordinator-assigned round-robin CSV slot (sysid 1030 → slot 4 then 5), carried in op-0x05, ADOPTED by the joiner — NOT the self-derived generation<<16|sysid&0x3ff (deleted bug), and NOT op-06 (that is the lock-rebuild push). Grounded on a real-VMS own-lab capture (E30 resolved). The "never reached MEMBER / op-06 not pinned" caveats in the summary, $executive-membership and $boot-join are resolved (a node reaches MEMBER now, OVMX^n). Scoped to exactly what is proven: cluster-dlm rows STAY implemented until vms-1ee; connection-manager$real-vax-join STAYS absent (OVMX^n, not a mixed/real-VAX cluster); quorum-on-a-joiner NOT claimed (op-01 PARAMS follow-on). Surface regenerated in lockstep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN * compat: fix dangling verified_against paths — fully-qualify the host test citations (vms-ffc class) The genesis/membership rows cited bare filenames (test_codec_cm.c, test_cnxman_join.c, test_cnxman_genesis_negctl.c) which resolved to the wrong dir (tests/vmsscs/). All three exist at tests/cluster/host/. Fully-qualified so the public /compat/ ships no dangling path. Substantively unchanged — the tests + the real /dev/vms CN=2 proof back verified as before. Surface regenerated in lockstep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
baron-3dl
added a commit
that referenced
this pull request
Sep 12, 2026
… learned Node B of the live 2-node cluster reached MEMBER, counted both systems and reported CEVOTES/QUORUM of ZERO for a whole run (#1119) -- with node A's real advertised VOTES sitting in B's own CSB table the entire time. ROOT CAUSE, three facts that only bite together: - cnxman_quorum_recompute() had exactly ONE caller, cnxman_try_genesis(): FOUNDER-only. - p. 7-42 task 2 (phase2_commit_quorum) copies the PROPOSED cells to the effective ones. The proposed cells are the COORDINATOR's arithmetic; an admitted node never ran one, so on the joiner the copy moves its own pre-cluster zeroes. (That refusal to invent is correct -- the bug is that nothing ELSE ever computed the figures.) - join_h_peer_advert() learns a peer's VOTES from its op-0x01 PARAMS into that peer's own CSB, and returned CONSUMED without recomputing. THE FIX is one guarded trigger, cnxman_quorum_member_recompute(), with two callers -- the transition COMMIT and each PARAMS record learned in between (design SS3.7: "recomputed on transitions", and the CLUB "tracks CEVOTES/QUORUM from every member's advertised VOTES ... from day one so $GETSYI reports truth"). It applies the SAME p. 7-6 walk the founder applies, over this node's own CSB table. INV-6: it runs only when cl->state is VMS_CLUSTER_MEMBER (which only a real Phase 2 commit of a real MEMBER flag sets) AND this node's own local CSB is in the selected, params-learned set; the walk itself skips any CSB that never received a PARAMS record. An unheard peer's VOTES stay UNKNOWN, never a zero; a node that is not a member publishes nothing. EXPECTED_VOTES has no isolated op-0x01 offset, so it is passed through as the CSB already holds it. R1 (host): 3 new join-FSM cases (before-admission computes nothing; votes learned BEFORE the GO are in the figures Phase 2 commits; votes learned or CHANGED after it move them, and CEVOTES never decreases) + the recompute gate's own unit test. Cluster host+sim suite 74/74, full host suite 262/262. R3/R4 (both substrates): vms.ko builds clean; the NetBSD elf32-vax module cross-compile + relocatable link is ILP32 width-clean. LIVE 2-node (k3s-worker, KVM, exit 0, never-crash), every field read back from /dev/vms -- tests/lab/captures/vms-d0d-quorum-2node-20260912/: before RIG-B-FINAL role=joiner member=1 cn=2 quorum=0 cevotes=0 after RIG-B-FINAL role=joiner member=1 cn=2 quorum=1 cevotes=1 qlost=0 RIG-B-GETSYI member=1 nodes=2 votes=0 quorum=1 The rig now also prints the SUMMANDS (RIG-*-CSBQ: each CSB's votes + votes_valid, and the CLUB's quorum_lost), so the figure is attributable: while still JOINING, B already shows the peer CSB votes=1 votes_valid=1 and publishes NO quorum; at the commit it shows 0 (its own) + 1 (A's, learned) = CEVOTES 1, QUORUM 1, and qlost=0 -- which only A's learned, OPEN vote can hold, since B's own VOTES are 0. The negctl run (VOTES=0 on both) still reports 0/0 with no member: the gate does not fabricate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014icBz4a5wS2nFyD7nhUXnd
baron-3dl
added a commit
that referenced
this pull request
Sep 12, 2026
… learned (#1201) Node B of the live 2-node cluster reached MEMBER, counted both systems and reported CEVOTES/QUORUM of ZERO for a whole run (#1119) -- with node A's real advertised VOTES sitting in B's own CSB table the entire time. ROOT CAUSE, three facts that only bite together: - cnxman_quorum_recompute() had exactly ONE caller, cnxman_try_genesis(): FOUNDER-only. - p. 7-42 task 2 (phase2_commit_quorum) copies the PROPOSED cells to the effective ones. The proposed cells are the COORDINATOR's arithmetic; an admitted node never ran one, so on the joiner the copy moves its own pre-cluster zeroes. (That refusal to invent is correct -- the bug is that nothing ELSE ever computed the figures.) - join_h_peer_advert() learns a peer's VOTES from its op-0x01 PARAMS into that peer's own CSB, and returned CONSUMED without recomputing. THE FIX is one guarded trigger, cnxman_quorum_member_recompute(), with two callers -- the transition COMMIT and each PARAMS record learned in between (design SS3.7: "recomputed on transitions", and the CLUB "tracks CEVOTES/QUORUM from every member's advertised VOTES ... from day one so $GETSYI reports truth"). It applies the SAME p. 7-6 walk the founder applies, over this node's own CSB table. INV-6: it runs only when cl->state is VMS_CLUSTER_MEMBER (which only a real Phase 2 commit of a real MEMBER flag sets) AND this node's own local CSB is in the selected, params-learned set; the walk itself skips any CSB that never received a PARAMS record. An unheard peer's VOTES stay UNKNOWN, never a zero; a node that is not a member publishes nothing. EXPECTED_VOTES has no isolated op-0x01 offset, so it is passed through as the CSB already holds it. R1 (host): 3 new join-FSM cases (before-admission computes nothing; votes learned BEFORE the GO are in the figures Phase 2 commits; votes learned or CHANGED after it move them, and CEVOTES never decreases) + the recompute gate's own unit test. Cluster host+sim suite 74/74, full host suite 262/262. R3/R4 (both substrates): vms.ko builds clean; the NetBSD elf32-vax module cross-compile + relocatable link is ILP32 width-clean. LIVE 2-node (k3s-worker, KVM, exit 0, never-crash), every field read back from /dev/vms -- tests/lab/captures/vms-d0d-quorum-2node-20260912/: before RIG-B-FINAL role=joiner member=1 cn=2 quorum=0 cevotes=0 after RIG-B-FINAL role=joiner member=1 cn=2 quorum=1 cevotes=1 qlost=0 RIG-B-GETSYI member=1 nodes=2 votes=0 quorum=1 The rig now also prints the SUMMANDS (RIG-*-CSBQ: each CSB's votes + votes_valid, and the CLUB's quorum_lost), so the figure is attributable: while still JOINING, B already shows the peer CSB votes=1 votes_valid=1 and publishes NO quorum; at the commit it shows 0 (its own) + 1 (A's, learned) = CEVOTES 1, QUORUM 1, and qlost=0 -- which only A's learned, OPEN vote can hold, since B's own VOTES are 0. The negctl run (VOTES=0 on both) still reports 0/0 with no member: the gate does not fabricate. Claude-Session: https://claude.ai/code/session_014icBz4a5wS2nFyD7nhUXnd Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What — SYMMETRIC CN=2: OVMX founds a cluster AND admits a joiner that reaches full membership
The genesis→admission stack (rd vms-f6b / vms-77c / vms-3a7c / vms-9c99), proven at runtime on two real /dev/vms executives:
Node A founds (VOTES=1, quorum by own votes); node B joins (VOTES=0), the coordinator assigns it a CSV slot, B adopts the assigned CSID, and BOTH count CN=2 — read from real executive state on both (three agreeing projections). B's own CSB table lists both members;
$GETSYI member=1 nodes=2 csid=0x00010002.The rule + mechanism — grounded on real OpenVMS VAX (own-lab, torn down)
A fresh 3-node real OpenVMS VAX 7.3 cluster (isolated pod, torn down after; shared oracle nodes never touched), 3× admission, real executive readout (OPCOM + F$CSID/F$GETSYI, causally grounded — byte-grep rejected as unsound):
coord_next_slotis authentic.[20:24]=SCSSYSTEMID,[36:40]=assigned CSID (hi=incarnation, lo=slot),[40:42]=0-based CSV index;[42:132]=uninterpreted (emitted zero+counted, never the VAX's bytes). Verified against all 8 real op-05 frames before building.The fix
vms_cm_membership_rec_build/_parse(one shared form + consistency predicate for builder and parser);coord_send_membership_set()sends the full set to the joiner, the delta to present members. Executive-backed: every field read off the CSB the record is about (the sysid the port learned, the CSIDcoord_assign_slotstamped, its slot, the member's real incarnation). Incomplete identity → no record (counted).sysid&0x3ff) deleted — it computed the inexpressible slot 962, the root cause of the CN=3 asymmetry and the earlier "node B counts 1".barrier_commit_phase2(p.7-42, the same criterion membership + the coordinator use), NOT waiting for op-0c#12. A real VAX commits an ADD with no on-wire op-0c to the joiner, so waiting hung the joiner = interop regression (conductor-ruled).VMS_CM_OP_LOCKRB→VMS_CM_OP_MEMBREC. No new TU/ioctl/shr-vector/lock_nb.h mirror needed (verified).Anti-LARP controls (both HELD)
csid=0x00010002(the ASSIGNED slot 2), a value it could not compute ⇒ adopted, not derived.Honest, not hidden
quorum=0 cevotes=0) — it counts both members but hasn't learned A's VOTES (ride op-01 PARAMS); a follow-on, not needed for the DLM proof (which addresses members by CSID).Tests
Host ctest 250/250 (serial; one
-j-only timeout is pre-existing/unrelated, passes standalone). Both cluster gates + elf32-vax ILP32-clean. New: codec round-trip + refusals + zeroed-stale-region; coordinator full-set/delta with every field compared to its CSB; slot-not-derivable; four joiner cases (adopt / not-ours-filed / re-adopt-on-rejoin / unusable-answered-not-adopted). Rig captures included.Register row for the runtime-proven capability is a follow-up (INV-0, coordinated with the conductor). Unblocks vms-1ee (the DLM cross-node proof).
🤖 Generated with Claude Code
https://claude.ai/code/session_01GKRFgtQkJPTJg7Gg5QyLcN