vms-d00: fix 64-bit width overflow in MMK.EXE early path (objtree tree head)

[baron-3dl]

Summary

MMK.EXE SIGSEGV'd in the QEMU guest before sp_open (post-parse / pre-drive), nondeterministically, and only with a real executive (on the host the same path reached sp_open cleanly). This blocked spine #4's capstone drive (vms-b23). Prior work (vms-b23 #462) characterized it via a SIGSEGV handler but lacked a real backtrace — this PR gets one and fixes the root cause.

Root cause — a 64-bit pointer-width bug (not the stack/foreign-command overflow the SIGSEGV handler suggested)

tests/corpus/tier3-mmk/objects.c declared the LIB$*_TREE root cell as static unsigned int objtree — 4 bytes. On the VAX a longword is a pointer, so stock MadGoat MMK is correct; on a 64-bit OVMX target that cell is too small. lib$insert_tree / lib$lookup_tree take the head by reference and dereference it as a full 8-byte pointer, so they:

• over-READ 8 bytes of the 4-byte global (the adjacent global's 4 bytes become the high half of a bogus root pointer), and
• on insert, over-WRITE it, truncating the stored root node address.

The reconstructed garbage pointer is later dereferenced → SIGSEGV. Whether it faults depends on address-space layout — which is exactly why it was nondeterministic and "executive-dependent" (the executive changes the heap layout). On the host the reconstructed pointer happened not to fault, so MMK reached sp_open.

Real backtrace (the deliverable prior work lacked)

Pinned with ASan on the host mmk_native ELF, run with the identical input the guest capstone uses (VMS_FOREIGN_CMD="/DESCRIPTION=OVMXB23.MMS OVMXB23.OUT"):

ERROR: AddressSanitizer: global-buffer-overflow ... READ of size 8
  #0 lib$lookup_tree   src/libvms/rtl/lib_tree.c:108
  #1 Find_Object       tests/corpus/tier3-mmk/objects.c:102
  #2 make_objrefs      tests/corpus/tier3-mmk/parse_descrip.c:1135
  #3 parse_store       tests/corpus/tier3-mmk/parse_descrip.c:1051
  #4 act_prs           tests/libvms/mmk_parse_tables.c:85
  ... lib$table_parse -> parse_descrip -> Read_Description
  #9 main              tests/corpus/tier3-mmk/mmk.c:705
0 bytes after global variable 'objtree' (size 4)

This lands exactly where the item said the crash was — post-parse, pre-drive, in Read_Description's object-tree build.

Fix

Declare objtree pointer-width (void *), matching symbols.c's apply_sort() void *tree (already correct) and the LIB$ manual's quadword tree head on 64-bit architectures. One-line change + an inline OVMX tag. Node links were already pointer-width (struct OBJECT *flink/blink), so only the head cell was wrong.

Clean-room (Rule 8): objects.c is stock MadGoat freeware; this is an OVMX 64-bit portability fix, tagged inline. No cross-image symbols touched → no .vec changes.

Proof

Check	Result
ASan before fix (exact guest input)	global-buffer-overflow at lib_tree.c:108 (above)
ASan after fix (exact guest input)	clean; MMK proceeds through the object tree and reaches the drive (sp_open)
toolchain-mmk-parse host ctest	PASS — no host regression
build-static (musl) mmk_native	builds clean (this is the exact static ELF staged into the guest)

Why ASan is the right proof for this bug class: the fault was nondeterministic precisely because it's deterministic memory corruption whose fault depends on address layout. ASan's redzone detects the corruption on every run regardless of whether it would fault, so before/after ASan on the identical guest input is a stronger verdict than N QEMU boots that could each pass by luck with the bug still present.

Guest / /dev/vms status and next blocker

MMK now reaches sp_open (the drive). The live guest QEMU exercise belongs to vms-b23's capstone (test_syssvc_mmk_drive, deferred at 0be3591c): that test staged this same static MMK.EXE and drove the full build. This fix unblocks it past this crash — but the full drive then hits the separate completion-hang blocker vms-95c, so the capstone can only go green once vms-95c is also fixed. Answering the item's question directly: MMK reaches sp_open; the next blocker is vms-95c.

Fixes the crash tracked by vms-d00. Do not merge without CI review.

🤖 Generated with Claude Code