Repository navigation
Indexing worker can freeze the whole OS on binary-heavy folders, and always dies with a 0-byte log — needs resource guards and early log flush #1132
Description
Environment: v0.9.0, Windows amd64 (ui variant), 32GB RAM. mem.init prints budget_mb=11429.
Incident: index_repository --mode moderate over a 29GB non-git folder (~65k files: CCTV footage frames, a 535MB .mp4, .zip archives, YOLO .pt/.onnx weights, plus ~50 real .py files) hard-froze Windows twice (both times requiring a physical power-off, confirmed via Kernel-Power 41 events). The second freeze happened on a fresh boot with nothing else running — the index run was the only workload.
The diagnostic gap: every crashed/killed worker leaves its log at 0 bytes — not a single line is ever flushed (logs/.worker-*.log). I now have 4 separate incidents with this pattern: the two OS freezes above, plus two contained worker crashes in cross-repo mode (reported separately). The budget_mb printed at init evidently didn't constrain the worker before the OS became unresponsive.
Workaround that confirms the cause: adding a .cbmignore that excludes the footage dirs + media extensions makes the exact same folder index fine in seconds (1.3k nodes, worker RSS never visibly elevated).
Asks:
- Enforce the memory budget in the worker itself — abort with a structured error well before the OS starts thrashing.
- Skip obvious binary/media files (extension + magic bytes) before reading them into the parser pipeline.
- Flush a log header (pid, repo, mode) synchronously at worker start, and the current file path per file — so postmortems are possible. Today a dead worker is forensically invisible.
Happy to run any diagnostic build against the same folder.
Activity
DeusData commentedon Jul 16, 2026
Thanks for the careful incident report and for isolating the binary-heavy directories with
.cbmignore. A workload that can make the host OS unresponsive is a high-priority 0.9.1-rc stability issue. Please do not rerun the unsafe corpus until resource containment is in place. The RED work should use bounded synthetic binary/media fixtures and separately cover early binary rejection, worker memory-budget enforcement, and a synchronously flushed startup/current-file diagnostic so a killed worker cannot leave a zero-byte log.DeusData commentedon Aug 14, 2026
@junk151516 — two hard power-offs with Kernel-Power 41 events, the second on a fresh boot where our indexer was the only workload, is a serious thing to have put you through. I am sorry.
Your
.cbmignoreexperiment is what makes the report conclusive rather than suggestive: same folder, media directories excluded, indexes in seconds. So thebudget_mbwe print at init is plainly not constraining the worker before the machine starts thrashing. All three of your asks are the right ones — enforce the budget inside the worker, reject binary and media files before the parser reads them, and flush a log header synchronously so a dead worker is not forensically invisible. That zero-byte log keeps surfacing in other reports too, which is its own indictment.Status honestly: not fixed yet. Tonight's v0.10.5 targets the issues that make cbm impossible to operate, and "can take the whole machine down" sits at the top of that class. Please keep the ignore file in place until it lands.
fix(index): make a crashed worker's log survive and name the run
DeusData commentedon Sep 26, 2026
Thank you, @junk151516! Your
.cbmignoreexperiment is what made this conclusive. Here's what has landed:A clean stop when a Windows worker hits its memory cap is still in progress, and we'll verify it with synthetic fixtures. Please keep your ignore file in place; there's no need to rerun the folder that froze your machine.
DeusData commentedon Oct 19, 2026
Thank you for the binary-heavy folder isolation. The last update still left clean Windows memory-cap termination pending; better logs and parser filtering alone do not establish that containment. I have no new verified result, and there is still no need to rerun the corpus that froze your machine.