emrg: Windows TUI Unicode input via ReadConsoleInputW

[argszero]

Summary

Fixes Windows TUI Chinese (IME) input on v0.2.11 (host-verified rant 2026-08-07T21:35:47). #546 (ENABLE_VIRTUAL_TERMINAL_INPUT) only translated function/arrow keys to ANSI sequences — arrows work now, but IME-confirmed characters still arrive through os.read as console input code-page bytes (GBK/CP936 on Chinese systems), which the UTF-8-assuming input chain garbles.

Root cause

ENABLE_VIRTUAL_TERMINAL_INPUT changes how conhost delivers function keys (ANSI CSI sequences) but not how character keystrokes are encoded — those still come through the byte stream in the console input code page. The only reliable way to get IME-confirmed Unicode is the wide-char API ReadConsoleInputW (UTF-16 KEY_EVENT_RECORD.UnicodeChar).

Changes

• emrg/client/python_tui/win32.py
  • Added _KEY_EVENT_RECORD / _INPUT_RECORD ctypes structs matching the Win32 layout (bKeyDown, wRepeatCount, wVirtualKeyCode, wVirtualScanCode, uChar union, dwControlKeyState).
  • read_console_unicode(fd) — reads INPUT_RECORDs via ReadConsoleInputW; key-down events with UnicodeChar != 0 (ASCII / Ctrl chars / IME-confirmed CJK) are UTF-8 encoded; UnicodeChar == 0 (function/arrow keys) falls back to the existing _LEGACY_SCAN_TO_ANSI scan-code table; key-up events dropped (avoids duplicate characters).
  • flush_console_input(fd) — drops stale byte-stream residue when entering raw mode.
  • Guarded msvcrt import + ctypes.windll access so the module (and its pure translation helper) imports cleanly on POSIX — CI is ubuntu-only and tests import it.
• emrg/client/app.py — _win_stdin_loop (Windows stdin thread) now uses read_console_unicode / flush_console_input instead of os.read; non-blocking reads sleep 5 ms (_win_stdin_stop.wait) to avoid busy-polling. POSIX path untouched.
• tests/test_win32_input.py — 17 pure-logic tests of _key_event_to_bytes (CJK positive, arrows/Home/End/PgUp/PgDn via scan codes, key-up drops, unknown scan codes, Ctrl chars), passable on POSIX like test_input_parser.py.
• Doc-count sync (README.md / Agent.md: 548 → 565, per emrg: sync test counts to 484 + guard test against doc drift (recurs #426/#430/#510) #511 guard).

Design notes

• The emrg: fix Windows TUI CJK input + legacy arrow keys + /rant project visibility #546 VT-input byte path is retained as fallback for pre-Win10-1607 consoles that reject the VT flag.
• No pywin32 dependency — plain ctypes, keeps the packaged runtime lean.
• POSIX (macOS/Linux) regression-free; 565 tests pass on ubuntu.

[argszero]

❌ Needs fix: INPUT_RECORD ctypes layout is not platform-stable

The KEY_EVENT_RECORD.uChar field uses wintypes.WCHAR (= c_wchar, 2 bytes on Windows but 4 bytes on POSIX). The struct layout is coincidentally correct on Windows (KEY_EVENT_RECORD 16B / INPUT_RECORD 20B) but inflated on POSIX (32B / 40B, Event at offset 8 instead of 2) — verified with ctypes.sizeof on this machine.

Impact:

1. The struct cannot be validated or simulated on POSIX — the rant's acceptance item 'win32 read_console_unicode 单测（模拟 INPUT_RECORD 中文/功能键）' is impossible with the current layout (a simulated record written at the wrong offsets would not exercise the real loop).
2. Any future code path reading INPUT_RECORDs from a mocked buffer on a non-Windows build would corrupt data silently.

Fix: use ctypes.c_ushort for uChar (explicit 16-bit, identical size/alignment to the WCHAR union member on Windows, and matches the Win32 ABI on every platform), convert to chr() at the call site, and add a struct-layout assertion test (sizeof == 16/20, Event offset 2) plus simulated-read tests for the record loop (CJK char, key-up drop, MOUSE_EVENT skip, arrow scan-code translation).

[argszero]

Fix pushed as dda46ec: KEY_EVENT_RECORD now uses fixed-width ctypes (c_int/c_uint/c_ushort) — the root cause was broader than WCHAR: wintypes.BOOL/DWORD are c_long/c_ulong, 4B on Windows but 8B on LP64 POSIX, inflating the struct to 32/40B vs the Win32 ABI 16/20B. Added layout assertions (sizeof 16/20, Event offset 4) and simulated INPUT_RECORD loop tests (CJK char, arrow scan code, key-up drop, MOUSE_EVENT skip) that exercise the real read loop via a monkeypatched ReadConsoleInputW. 572 tests pass on POSIX.

[argszero]

✅ LGTM — cycle 20260807-215157 (2nd angle: fixed-layout correctness + test discriminative power)

Verified on head dda46ec:

• KEY_EVENT_RECORD now uses fixed-width ctypes (c_int/c_uint/c_ushort): sizeof == 16, INPUT_RECORD == 20, Event at offset 4 — exact Win32 ABI on every platform (wintypes.BOOL/DWORD would be 8B on LP64 POSIX).
• Layout assertion tests pin the ABI: reverting uChar to wintypes.WCHAR would fail the 16/20 size assertions (discriminating).
• Simulated record-loop tests exercise the real n_read traversal through a monkeypatched ReadConsoleInputW: CJK char → UTF-8, arrow scan-code 0x48 → ESC [ A, key-up dropped, MOUSE_EVENT skipped (positive + negative states).
• read_console_unicode returns b"" when no events; app.py _win_stdin_loop sleeps 5ms on empty (no busy-poll) and flushes stale bytes on raw-mode entry; POSIX path untouched.
• 572 tests pass locally; CI test workflow green (31184503135).

[argszero]

✅ LGTM — cycle 20260807-215524 (3rd angle: regression surface vs existing input paths)

Verified on head dda46ec:

• events.py is untouched — the existing _LEGACY_SCAN_TO_ANSI / normalize_legacy_scan_codes path is fully preserved; the new _key_event_to_bytes only READS the shared scan-code dict (no mutation, no conflict).
• Runtime mutual exclusion: the win32 branch of _win_stdin_loop switches from os.read to read_console_unicode; POSIX _stdin_reader + loop.add_reader path is byte-identical (no cross-platform regression).
• ReadConsoleInputW blocks until a record arrives (matching the old blocking os.read design); the 5ms _win_stdin_stop.wait only covers the empty-return case — no busy-poll.
• Downstream chain unchanged: scan codes map to ANSI CSI which InputParser already handles; the legacy 0xE0 byte path remains as pre-VT fallback.
• Non-blocking note: wRepeatCount is not applied (held-key auto-repeat may be less smooth than OS-native typematic) — acceptable for v1, could batch repeats later.
• 572 tests pass locally (incl. test_input_parser.py interaction); CI green (31184503135).

[argszero]

✅ LGTM — cycle 20260807-215819 (4th/final angle: mergeability + end-to-end Windows behavior)

Final gate on head dda46ec:

• MERGEABLE, clean diff vs master (+323/−16 over 5 files), no conflict markers, no unrelated changes; doc counts consistent (README + Agent.md = 572).
• End-to-end on Windows: ReadConsoleInputW reads UTF-16 KEY_EVENT_RECORDs → key-down UnicodeChar != 0 encoded UTF-8 → stdin_queue → InputParser (UTF-8-native) → CJK renders correctly. Function keys (UnicodeChar == 0) → _LEGACY_SCAN_TO_ANSI → ANSI CSI → same parser path. POSIX: byte-identical _stdin_reader/add_reader.
• Two prior ✅ from distinct cycles on this head (215157 fixed-layout correctness + discriminative tests; 215524 regression surface); the sole ❌ was on the pre-fix head and is resolved by dda46ec.
• 572 tests pass locally; CI test workflow green (31184503135). Ready to merge.