Skip to content

ops(gate): the chaos legs — api-down and engine-reconnect (STORY-447, gh-#777) - #792

Merged
genwave-radio merged 3 commits into
mainfrom
ops/chaos-legs-447
Sep 16, 2026
Merged

genwave-radio merged 3 commits into
mainfrom
ops/chaos-legs-447

Conversation

@genwave-radio

Copy link
Copy Markdown
Collaborator

PR-7 of the gh-#777 epic (STORY-447, SPEC F178.8). The gate gains --chaos: two scenarios that break the running stack on purpose and measure how the stream and the stack come back.

💥 What

  • API-down (T496), inside the capture leg while ffmpeg is still recording: docker compose stop api, sleep GATE_OUTAGE_SECS (90), start api, then poll for a non-safe track_id within GATE_RECOVERY_SECS (120). The capture is measured once and attributed: silence during a chaos run fails the chaos leg api-down silence (the loudness check is skipped, a silent stretch masks it), silence without chaos fails the capture leg as before. A recovery timeout fails api-down recovery.
  • Engine-reconnect (T497), after api-down passes: docker compose restart engine, wait for api health then first on-air within GATE_RECONNECT_SECS (60, gap counted from the restart command), then a fixed 60 s capture-reconnect.wav measured for silence only. Failures: engine-reconnect on-air / capture / measure / silence.
  • --chaos requires --capture (exit 2, like --capture requires --fresh). The three knobs validate like GATE_POLL_SECS.
  • Report: api-down outage / recovery, engine-reconnect on-air / silence events in the md; json twins under legs.chaos.measurements (nulls when a scenario never ran).
  • Story447_ChaosLegs.cs: 17 facts green; the harness stub answers stop api / start api / restart engine with markers and can be told to stay dark after a stop or a restart.

🔌 Real runs against v5.8.3 (dev box, dev station stopped)

30 s outage (GATE_OUTAGE_SECS=30 tools/gate/stack_gate.sh --tag v5.8.3 --fresh --capture --chaos): exit 0, wall 497 s.

measurement value
capture secs 360
silence events 0
integrated -21.0 LUFS
booth_log 15
api-down outage 30 s
api-down recovery 0 s
engine-reconnect on-air 6 s
engine-reconnect silence events 0

Gate defaults, 90 s outage (T498): exit 1, chaos | ran/failed | api-down silence, silence events 2, recovery 0 s. Twice, reproducibly. The engine log puts the silence on the timeline: the main queue drains 63 s into the outage, the safe branch plays its single prefetched track, then the 7 s gap, then mksafe blank until the api answers /internal/safe-track again (about 23 s of silence). Filed as #791. So the gate did its job: F178.8(a) "mksafe/safe loop holds" is not true of v5.8.3 once the queue drains, and T498 stays open until #791 is decided (fix the product, tolerate the designed gap in the gate overlay, or amend the spec).

🔍 Review rounds worth knowing

  • T496 passed with notes (comment fixes: early-return on a failed stop/start, the loudness-masking rule, why an aborted background ffmpeg is left to -t and the down -v).
  • T497 passed with a note on my brief, not the code: the engine does not re-ask the api for the next item on its own, the api's feeder pushes the next track_id over a fresh TCP client. The header comment now says so.

📝 For later tasks

✅ Gate

Full solution, Category!=Integration: 9 projects, 0 failed (Host 3008 passed / 110 skipped, Architecture 150 / 10). bash -n and shellcheck clean.

…#777)

With capture running (F178.8a): stop/start api around GATE_OUTAGE_SECS, recovery polled through the engine metadata against GATE_RECOVERY_SECS, silence during the outage attributed to the chaos leg (not capture) under --chaos, chaos report block in md+json, GATE_OUTAGE_SECS/GATE_RECOVERY_SECS knob validation. Story447 api-down facts green.
After the api-down scenario passes, --chaos restarts the engine container,
waits for api health then first on-air within GATE_RECONNECT_SECS (60),
records a fixed 60 s capture-reconnect.wav and measures it: a timeout
fails the leg "engine-reconnect on-air", a recording failure
"engine-reconnect capture", a measure error "engine-reconnect measure",
any silence event "engine-reconnect silence" (loudness is not judged).
The report gains engine-reconnect on-air seconds and silence events in
the md and json; the new knob is validated like GATE_POLL_SECS.
Story447 facts for the engine scenario are un-skipped; the harness stub
passes GATE_RECONNECT_SECS=4 and answers the restart marker.
@genwave-radio
genwave-radio merged commit 4070991 into main Sep 16, 2026
11 checks passed
@genwave-radio
genwave-radio deleted the ops/chaos-legs-447 branch September 16, 2026 20:20
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant