Skip to content

fix(setup-ai): harden AI bootstrap against real-world failure modes - #13

Merged
msitarzewski merged 1 commit into
mainfrom
fix/setup-ai-hardening
Sep 12, 2026
Merged

fix(setup-ai): harden AI bootstrap against real-world failure modes#13
msitarzewski merged 1 commit into
mainfrom
fix/setup-ai-hardening

Conversation

@msitarzewski

Copy link
Copy Markdown
Owner

Follow-up to #12. That PR delivered the bootstrap script we needed; this hardens the paths where it breaks on a real network with a real .env.

Fixes

Interrupted model downloads were cached as success. The 1.5 GB model was written straight to its final path, so a dropped transfer left a truncated file that every later run accepted. Because server/lib/capabilities.js:58 detects the model with a bare existsSync, the UI would un-gate Transcribe over a corrupt model — the exact failure the v0.3.2 capability gating exists to prevent. Reproduced with a curl stub that dies mid-transfer:

RUN 1: curl: (18) transfer closed with bytes remaining to read
       -> models/ggml-medium.bin left on disk, 11 bytes
RUN 2 (network healthy): OK: Whisper model already present
                         OK: AI pipeline setup complete   # over an 11-byte model

Downloads now land in models/ggml-medium.bin.part, are size-checked, and only then renamed. Also dropped curl --silent — a multi-minute silent download reads as a hung script.

.env was sourced as shell. Under set -euo pipefail, both of these abort the script:

JWT_SECRET=k7$Rm2pQ9wZx        -> .env: line 1: Rm2pQ9wZx: unbound variable   (exit 1)
STATION_NAME=Pirate Radio One  -> .env: line 1: Radio: command not found      (exit 127)

A value containing ; or $(...) would execute. Our own .env carries JWT_SECRET, ICECAST_*_PASSWORD, and COTURN_PASSWORD, any of which can hold a $. Only the three LLM_* keys are read now, via sed.

whisper.cpp was a tracked gitlink that .gitignore could not touch. #12 added whisper.cpp/ to .gitignore while the path remained a 160000 entry with no .gitmodules backing — and git never ignores a tracked path:

git check-ignore -v whisper.cpp  -> exit 1 (NOT ignored)
git ls-files -s whisper.cpp      -> 160000 1fe009ca...

Meanwhile setup-ai.sh clones --depth 1 of the default branch, which will not equal the pinned SHA — so anyone who ran the script got a permanently dirty tree showing modified: whisper.cpp (new commits). Untracked it, which is what the setup script already assumed and what .gitignore was reaching for.

Missing ffmpeg aborted the whole bootstrap, though neither ffmpeg nor ffprobe is needed to clone whisper.cpp, build it, or fetch the model. Now warns and continues.

Also removed a dead already-built check that printed OK: whisper.cpp already built twice.

Testing

Kept the fake-toolchain harness from #12 — it is a good pattern — and added a regression case per bug. Both new cases were verified to fail against the pre-fix script, so they are testing something real:

vs pre-fix setup-ai.sh:  [2] FAIL: truncated model left on disk
                         [3] FAIL: setup-ai.sh choked on a realistic .env
vs this branch:          4 checks, all PASS

Full suite 18/18 (9 signaling + 9 rooms), server running as CI starts it.

tests/test-setup-ai.sh is now wired into ci.yml. It was only reachable via run-pre-validation.sh and an npm script, so it would have rotted.

Note

npm test alone reports 0/9 on a clean main — the tests need the signaling server already listening on 6736, which ci.yml starts but the npm script does not. Not addressed here, but the npm script is a misleading front door.

Thanks to @avezou for the original script — the CMake layout fix in it caught a live bug in the server auto-build path.

🤖 Generated with Claude Code

Follow-up to #12. The bootstrap script works on a good network with a
simple .env; these are the paths where it didn't.

Interrupted model downloads were cached as success. The 1.5 GB model was
written straight to its final path, so a dropped transfer left a truncated
file that every later run accepted as valid. Worse, capabilities.js detects
the model with a bare existsSync, so the UI un-gated Transcribe over a
corrupt model -- exactly the failure v0.3.2 gating exists to prevent.
Downloads now land in models/ggml-medium.bin.part, are size-checked against
a floor, and only then renamed into place. Also dropped curl --silent; a
multi-minute silent download reads as a hung script.

.env was sourced as shell. Under set -euo pipefail a JWT_SECRET containing
$ aborted the script with "unbound variable" and a value with spaces ran as
a command. Only the three LLM_* keys are read now, via sed, leaving every
other secret untouched.

whisper.cpp was a tracked 160000 gitlink with no .gitmodules backing, so the
new .gitignore rule could never apply to it -- git does not ignore tracked
paths. Anyone who ran setup-ai.sh got a permanently dirty working tree
showing "modified: whisper.cpp (new commits)". Untracked it, which is what
the setup script already assumed.

Missing ffmpeg aborted the entire bootstrap, though neither ffmpeg nor
ffprobe is needed to clone whisper.cpp, build it, or fetch the model. Now
warns and continues.

Tests: kept the fake-toolchain harness from #12 and added a regression case
per bug. Both new cases were verified to fail against the pre-fix script.
Wired tests/test-setup-ai.sh into CI, where it was previously unreachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@msitarzewski
msitarzewski merged commit f7f226b into main Sep 12, 2026
4 of 6 checks passed
@msitarzewski
msitarzewski deleted the fix/setup-ai-hardening branch September 12, 2026 03:45
msitarzewski added a commit that referenced this pull request Sep 12, 2026
…ubmodule steps

Three latent breakages found sweeping the repo.

setup.sh could never run. PR #14 changed it to `npm ci`, but package-lock.json
is gitignored (.gitignore:6) and no lockfile is tracked, so a fresh clone died
immediately:

    npm error code EUSAGE
    npm error The `npm ci` command can only install with an existing
    package-lock.json

Reverted to `npm install --omit=dev`, which works either way. Worth deciding
separately whether this repo should commit its lockfiles — an application
normally should, for reproducible installs; ignoring them is a library
convention.

Four env vars the server reads were undocumented in .env.example: PORT,
ALLOWED_ORIGINS, ICECAST_MOUNT, ICECAST_USER. The consequential one is
ALLOWED_ORIGINS — when unset or empty EVERY origin is permitted
(server/server.js:53), which is fine for local development and wrong for a
public deployment. All four now documented with their real defaults.

Docs still instructed `git submodule update --init` for whisper.cpp. PR #13
untracked that gitlink, so the command silently does nothing and leaves users
with no whisper.cpp and no error. Corrected across the memory bank to point at
./setup-ai.sh. The 2026-05 task log keeps the old wording on purpose — it is a
record of what was true then, not instructions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant