Skip to content

bug(storage): context-offload single-flight test is flaky when the conflicting open wins the race #5801

Description

@sunrioa

What happened

single-flights one limit-bound writer and snapshots admitted inputs in packages/storage/src/__tests__/context-offload-store.test.ts occasionally fails with an unhandledRejection:

The input did not match the regular expression /different limits/u. Input:

'StorageRootAuthorityError: Expected an active interactive write storage root lease'

The test starts three concurrent openInteractiveContextOffloadStoreForWrite calls on one lease, two with the same limits and a third with different limits, and expects the first call to become the writer so that only the third is rejected with "different limits". Each call, however, first awaits assertStorageRootLease (context-offload-store.ts:126), which checks the root identity on disk, and only afterwards looks up or registers the in-flight opening (:132, :170). Those checks can finish in any order.

When the call with different limits finishes first, it becomes the writer and the two same-limit calls are rejected with "different limits" instead. The test body then throws, the fixture closes the owner, and the conflicting opener fails its second lease check (:154), which produces the lease error above. Because conflictingOpening (test line 76) is only awaited at line 86, after the Promise.all at line 84, its rejection is reported as unhandled, which is why the report shows only the lease error.

I'd expect the test not to depend on the order in which the three lease checks complete.

How to reproduce

It is intermittent. After npm ci && npm run build && npm run build:test, running the whole file repeatedly on Node 24 failed once in 60 runs on my machine:

cd packages/storage
for i in $(seq 60); do node --test --test-reporter=tap dist/__tests__/context-offload-store.test.js | grep '^# fail'; done

Running only this test with --test-name-pattern did not fail in 60 runs, so the timing left by the preceding test seems to matter.

The race itself shows up more often when the three opens are replayed directly. Saved as packages/storage/race.mjs and run with node race.mjs after npm run build, the script below had the conflicting call win 4–10 times in 200 on Node 24.19.0 (three runs) and 2–3 times in 200 on Node 22.23.1 (two runs), for example:

{ 'ok | ok | different-limits': 196, 'different-limits | different-limits | ok': 4 }
race.mjs
// Run from packages/storage after `npm run build`.
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { openInteractiveContextOffloadStoreForWrite } from './dist/context-offload-store.js';
import { resolveStorageRoot, tryAcquireInteractiveRootOwner } from './dist/root-authority.js';

const limits = (physical = 64) => ({
  ownerMaxBytes: { read_image_snapshot: 64, tool_result_archive: 64 },
  sessionLogicalBytes: 64,
  workspacePhysicalBytes: physical,
});
const outcome = (p) =>
  p.then(() => 'ok', (e) => (/different limits/u.test(e.message) ? 'different-limits' : e.message));

const tally = {};
for (let i = 0; i < 200; i++) {
  const root = await mkdtemp(join(tmpdir(), 'offload-race-'));
  const owner = await tryAcquireInteractiveRootOwner(
    await resolveStorageRoot({ path: root, kind: 'interactive' }),
  );
  // Same order as the test: two opens with the same limits, then one with different limits.
  const results = await Promise.all(
    [limits(), limits(), limits(63)].map((l) =>
      outcome(openInteractiveContextOffloadStoreForWrite(owner.lease, { limits: l })),
    ),
  );
  const key = results.join(' | ');
  tally[key] = (tally[key] ?? 0) + 1;
  await owner.close();
  await rm(root, { recursive: true, force: true });
}
console.log(tally);

Environment

  • Commit: 2f3220552 (main). I reproduced it in a worktree on top of that commit; context-offload-store.ts, sqlite-context-offload-store.ts, root-authority.ts and the test file are identical to main there.
  • OS: macOS 26.5.2 (Darwin 25.5.0), arm64
  • Surface: @maka/storage unit tests
  • Node.js: 24.19.0 (CI uses Node 24) and 22.23.1

Logs, screenshots, or additional context

Trimmed TAP output from a failing run:

not ok 2 - single-flights one limit-bound writer and snapshots admitted inputs
  failureType: 'unhandledRejection'
  error: |-
    The input did not match the regular expression /different limits/u. Input:
    'StorageRootAuthorityError: Expected an active interactive write storage root lease'
  stack: |-
    invalidLease (.../dist/root-authority.js:580:12)
    requireLease (.../dist/root-authority.js:575:15)
    assertStorageRootLease (.../dist/root-authority.js:399:20)
    .../dist/context-offload-store.js:87:19
    async openInteractiveContextOffloadStoreForWrite (.../dist/context-offload-store.js:106:16)

I checked the 40 most recent failed CI workflow runs. The storage test output appears in 5 of them and this test passed in all 5, so I have no evidence yet that it has failed in CI.

Possible directions, for whoever picks this up:

  • Test only: issue the conflicting open once the first call is known to be in flight, or stop assuming which call wins.
  • Implementation: reserve the in-flight slot before the async lease check, so the first caller always becomes the writer, which seems to be what "single-flights" intends.
  • Either way, awaiting conflictingOpening together with the other two opens would let a mismatch surface as an ordinary assertion failure instead of an unhandled rejection.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions