Repository navigation
fix(collector-runtime): stop failed spool writes leaving temp files - #491
Merged
Merged
Conversation
A failed write to the blob spool (`LocalDeviceBlobSpool`, which keeps artifact bytes on local disk under their sha256) can leave its temp file in the spool's tmp/ directory. Nothing reads that file as an artifact, because the rename into place never runs, but it occupies disk space. No production code calls sweepTemp(), the method meant to reclaim leftover temp files, so a leftover stays in place. put(), the spool's write method, writes incoming bytes to a temp file and renames it to its sha256 path only after the whole body has been written. If reading the content throws partway, Node's pipeline() helper rejects as soon as that happens. It does not wait for the file's open, which Node runs asynchronously on its thread pool. put()'s catch block then deletes the temp file synchronously. When the open has not created the file yet, the delete finds nothing, and the file appears afterwards with nothing to remove it. The catch block now waits for the write stream's 'close' event before deleting. A file stream emits 'close' only after a pending open has settled and its file descriptor is closed, so the delete runs after the file has been created or the open has failed. The process sees this failure, so it removes the file now rather than leaving it for sweepTemp(). The success path is unchanged. The failure path now waits for one extra open and close before put() rejects. The regression test delays the file's open (by mocking fs.open, Node's file-open function) until the failure has reached put()'s catch block, then checks that the spool's tmp directory is empty. Without the fix it fails in 10 of 10 runs, leaving the temp file. With the fix it passes in 10 of 10. The existing test "leaves no addressable entry when the source fails mid-write" exercises the same race without delaying the open. It failed in 12 of 200 runs before the fix and in 0 of 200 after. With eight CPU-bound processes running alongside, it failed in 16 of 200 before the fix and in 0 of 200 after. Not verified: how often production writes take this path, how large a leftover file can get, and whether code outside the collector-runtime package depends on how quickly put() rejects. The one caller inside the package, captureBlobArtifact, which spools bytes and then queues an upload, propagates the error unchanged. The added latency on the failure path is not measured. Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
tnunamak
added a commit
that referenced
this pull request
Oct 19, 2026
…551) * fix(collector-runtime): stop failed spool writes leaving temp files Reapplies #491 (reverts 3fa11ee), now that its collector-runtime patch release is approved. A failed write in LocalDeviceBlobSpool.put() left its temp file behind; the race also made local-device-blob-spool.test.ts fail intermittently in package-verify on main. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> * chore(collector-runtime): regenerate artifact metadata for the spool fix Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> --------- Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak
added a commit
that referenced
this pull request
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LocalDeviceBlobSpool.put()can leave a temp file in the spool'stmp/directory when reading the content fails partway.pipeline()rejects as soon as the source throws, without waiting for the write stream's asynchronousopen(), so the catch block's synchronous unlink can run before the file exists; the open then creates it and nothing removes it. Nothing in production callssweepTemp(), so a leftover stays. This is what madelocal-device-blob-spool.test.ts"leaves no addressable entry when the source fails mid-write" fail intermittently in the full package run.The catch block now waits for the stream's
closeevent before unlinking. A file stream closes only after a pending open has settled, so the unlink runs after the file exists or the open has failed. The success path is unchanged; the failure path waits for one more open and close beforeput()rejects.A new regression test delays the file open (by mocking
fs.open) until the failure has reached the catch block, then assertstmp/is empty. It fails 10 of 10 runs on the old code and passes 10 of 10 with the fix. The existing flaky test failed 12 of 200 runs before the fix and 0 of 200 after; with eight CPU-bound processes alongside, 16 of 200 before and 0 of 200 after. On this head the spool test file passes 12/12; with the oldlocal-device-blob-spool.tsrestored, the new test fails (11/12). The package's fullnpm testshows only the known stale artifact-receipt failure.test:reclaim-racepasses 6/6.Not changed: temp files left by earlier failures are still only removed by
sweepTemp(), which nothing calls; wiring it into startup or the drain is a separate decision. A hungopen()would now hold the failure path, as it already holds the success path; not tested.Assisted-by: AI