Skip to content

fix(collector-runtime): stop failed spool writes leaving temp files - #491

Merged
tnunamak merged 1 commit into
testeng/integrationfrom
testeng/dca-blob-spool-temp
Oct 18, 2026
Merged

tnunamak merged 1 commit into
testeng/integrationfrom
testeng/dca-blob-spool-temp

Conversation

@tnunamak

Copy link
Copy Markdown
Contributor

LocalDeviceBlobSpool.put() can leave a temp file in the spool's tmp/ directory when reading the content fails partway. pipeline() rejects as soon as the source throws, without waiting for the write stream's asynchronous open(), so the catch block's synchronous unlink can run before the file exists; the open then creates it and nothing removes it. Nothing in production calls sweepTemp(), so a leftover stays. This is what made local-device-blob-spool.test.ts "leaves no addressable entry when the source fails mid-write" fail intermittently in the full package run.

The catch block now waits for the stream's close event before unlinking. A file stream closes only after a pending open has settled, so the unlink runs after the file exists or the open has failed. The success path is unchanged; the failure path waits for one more open and close before put() rejects.

A new regression test delays the file open (by mocking fs.open) until the failure has reached the catch block, then asserts tmp/ is empty. It fails 10 of 10 runs on the old code and passes 10 of 10 with the fix. The existing flaky test failed 12 of 200 runs before the fix and 0 of 200 after; with eight CPU-bound processes alongside, 16 of 200 before and 0 of 200 after. On this head the spool test file passes 12/12; with the old local-device-blob-spool.ts restored, the new test fails (11/12). The package's full npm test shows only the known stale artifact-receipt failure. test:reclaim-race passes 6/6.

Not changed: temp files left by earlier failures are still only removed by sweepTemp(), which nothing calls; wiring it into startup or the drain is a separate decision. A hung open() would now hold the failure path, as it already holds the success path; not tested.

Assisted-by: AI

A failed write to the blob spool (`LocalDeviceBlobSpool`, which keeps
artifact bytes on local disk under their sha256) can leave its temp
file in the spool's tmp/ directory. Nothing reads that file as an
artifact, because the rename into place never runs, but it occupies
disk space. No production code calls sweepTemp(), the method meant
to reclaim leftover temp files, so a leftover stays in place.

put(), the spool's write method, writes incoming bytes to a temp
file and renames it to its sha256 path only after the whole body
has been written. If reading the content throws partway, Node's
pipeline() helper rejects as soon as that happens. It does not wait
for the file's open, which Node runs asynchronously on its thread
pool. put()'s catch block then deletes the temp file synchronously.
When the open has not created the file yet, the delete finds nothing,
and the file appears afterwards with nothing to remove it.

The catch block now waits for the write stream's 'close' event before
deleting. A file stream emits 'close' only after a pending open has
settled and its file descriptor is closed, so the delete runs after
the file has been created or the open has failed. The process sees
this failure, so it removes the file now rather than leaving it for
sweepTemp(). The success path is unchanged. The failure path now
waits for one extra open and close before put() rejects.

The regression test delays the file's open (by mocking fs.open,
Node's file-open function) until the failure has reached put()'s
catch block, then checks that the spool's tmp directory is empty.
Without the fix it fails in 10 of 10 runs, leaving the temp file.
With the fix it passes in 10 of 10. The existing test "leaves no
addressable entry when the source fails mid-write" exercises the
same race without delaying the open. It failed in 12 of 200 runs
before the fix and in 0 of 200 after. With eight CPU-bound processes
running alongside, it failed in 16 of 200 before the fix and in 0
of 200 after.

Not verified: how often production writes take this path, how large a
leftover file can get, and whether code outside the collector-runtime
package depends on how quickly put() rejects. The one caller inside
the package, captureBlobArtifact, which spools bytes and then queues
an upload, propagates the error unchanged. The added latency on the
failure path is not measured.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI
@tnunamak
tnunamak merged commit 7ec0963 into testeng/integration Oct 18, 2026
@tnunamak
tnunamak deleted the testeng/dca-blob-spool-temp branch October 18, 2026 05:52
tnunamak added a commit that referenced this pull request Oct 18, 2026
… files (#491)"

This reverts commit 7ec0963.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 18, 2026
… files (#491)"

This reverts commit 7ec0963.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 18, 2026
… files (#491)"

This reverts commit 7ec0963.

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 18, 2026
… files (#491)"

This reverts commit 7ec0963.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 19, 2026
… files (#491)"

This reverts commit 7ec0963.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 19, 2026
…p files (#491)"

Batch 1 (#494) reached main with #491 reverted, pending a release decision.
Merging main back into testeng/integration brought that revert here; this
restores #491 on the integration branch.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 19, 2026
…p files (#491)"

Batch 2 (#544) reached main with #491 reverted, pending a release decision.
Merging main back into testeng/integration brought that revert here; this
restores #491 on the integration branch.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 19, 2026
…551)

* fix(collector-runtime): stop failed spool writes leaving temp files

Reapplies #491 (reverts 3fa11ee), now that its collector-runtime patch
release is approved. A failed write in LocalDeviceBlobSpool.put() left its
temp file behind; the race also made local-device-blob-spool.test.ts fail
intermittently in package-verify on main.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* chore(collector-runtime): regenerate artifact metadata for the spool fix

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

---------

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
tnunamak added a commit that referenced this pull request Oct 20, 2026
Brings in #551 (the #491 spool fix, now released on main), #416, #413 and the
dependency bumps #541 and #453, so batch PRs no longer need to revert #491.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant