Repository navigation
[Bug][CI]: Windows fixture teardown exhausts the remove retry budget (EBUSY/EPERM) in two existing tests #7515
Description
Client or integration
Other
Area
Platform (Windows / macOS / Linux)
Summary
Two existing test files failed on hosted Windows shards in one afternoon. Both failures came from fixture teardown, after every assertion in the case had passed. In both, the shared retrying remover (removeTreeWithRetry / removeTestTempTree) gave up on the case's own temp directory:
tests/codex-integration/main-account-hard-lock-fallback-routing.test.ts:52,configured combo aliases, policy profiles and account namespaces keep their admission:EBUSY: resource busy or locked, rm '...\\tmp\\ocx-main-lock-fallback-1XFyU0'after 15.2 s.tests/server/server-management-auth.test.ts:126(viatests/helpers/fixture-teardown.ts:32drainAndRemoveFixtureRoots),data and management environment tokens authorize only their own planes:EPERM: operation not permitted, rm '...\\tmp\\ocx-management-auth-P0CAUG'after 21.7 s.
Unlike #7000, both already go through the retrying helper. The long durations suggest the helper used up its whole retry budget while something in the process still held a handle under the directory. In the first case, pendingBackgroundWork() reported nothing still running. Neither PR in the reproduction touches these tests or the code under them in a way that opens files: one changes only scripts/ci/windows-sensitive-diff.sh and its selector test, and the other adds fallback policy with no new file I/O, timers or watchers.
Expected: teardown either waits for the code under test to release its handles, or the test closes whatever it opened, such as a server, a log stream or a lock, before the remove. A Windows shard should not fail on a teardown race.
Reproduction
- Run https://github.com/lidge-jun/opencodex/actions/runs/39720859021 (
ci.yml lane=all, headc31b08f43, PR ci(selector): flag process-signal code as Windows-sensitive #7503), jobwindows 2/9, batch 44/52:server-management-auth.test.tsteardown throwsEPERM. - Run https://github.com/lidge-jun/opencodex/actions/runs/39717538971 (
ci.yml lane=all, headb39ec1d01, PR feat(fallback): OCX-11 fallback off per scope, with home_unreachable on Children #7482), jobwindows 1/9, batch 21/55:main-account-hard-lock-fallback-routing.test.tsteardown throwsEBUSY. - Both files pass on macOS and Ubuntu, and
devlane=all39720820141 (bf4eb12) passed both on Windows. The race is intermittent and needs Windows file-handle semantics.
Version
dev bf4eb12 (2.91.0 line)
Operating system
GitHub-hosted windows-latest
Provider and model
Not applicable (test fixtures).
Logs or error output
error: EBUSY: resource busy or locked, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\opencodex-test-RkZ7Ge\tmp\ocx-main-lock-fallback-1XFyU0'
at removeTreeWithRetry (tests\helpers\remove-tree.ts:41:11)
at <anonymous> (tests\codex-integration\main-account-hard-lock-fallback-routing.test.ts:52:3)
(fail) applyMainHardLockFallbackOverride > configured combo aliases, policy profiles and account namespaces keep their admission [15162.99ms]
EPERM: operation not permitted, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\opencodex-test-4qavmb\tmp\ocx-management-auth-P0CAUG'
at removeTestTempTree (scripts\test-temp.ts:199:7)
at removeTreeWithRetry (tests\helpers\remove-tree.ts:34:5)
at drainAndRemoveFixtureRoots (tests\helpers\fixture-teardown.ts:46:33)
at async <anonymous> (tests\server\server-management-auth.test.ts:126:9)
(fail) management and data-plane credential separation > data and management environment tokens authorize only their own planes [21656.46ms]Screenshots and supporting files
No response
Redacted configuration
No response
Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Oct 22, 2026
Automated translation bookkeeping — detected language: English.
lidge-jun commentedon Oct 22, 2026
Third case, same class:
tests/codex-integration/codex-auth-context.test.ts:119,guardian and quota consumers back off and retry a stale credential refresh without reauth,EPERM: operation not permitted, rm '...\tmp\ocx-auth-ctx-Q7vEvp'after 24.9 s. Run https://github.com/lidge-jun/opencodex/actions/runs/39734497680 (lane=all, head14c3a6f8e, #7482 updated with devfc034c84c), jobwindows 7/9. The helper's diagnostic printed the directory's ACLs and attributes forconfig-mutation.sqlite,.opencodex-native-main.claim.sqlite,codex-accounts.jsonandconfig.json, then[test-temp] child-process probe did not complete; leaving the EPERM in place.The SQLite files inside the directory suggest a database handle that is still open at teardown; the config mutation lock and the native-main claim are the candidates.Ingwannu commentedon Oct 22, 2026
I independently retrieved the cited attempt-1 Windows job logs and confirmed the two teardown failures:
server-management-auth.test.tsexhausts removal with EPERM, andmain-account-hard-lock-fallback-routing.test.tswith EBUSY. These are cleanup failures, not evidence that the new selector/fallback behavior is wrong. Later reruns and a green dev control do not identify the handle owner or establish a fix.Current dev already drains startup/config/ACL work and closes the history index in
drainAndRemoveFixtureRoots; the hard-lock fixture still has its direct remover after releasing the spend home. That difference is a lead, not proof that adding the shared helper alone fixes both failures.pendingBackgroundWork()being empty is also not proof that all file/SQLite/ACL-child handles are closed.Keeping this open for a Windows reproduction with the fixture's failing child path and its remaining handle owner captured before final removal. A targeted fix should settle that owner (and exercise the actual release boundary), rather than enlarge retries, suppress EPERM/EBUSY, or claim Linux/mock runs establish Windows behavior. No service or production directory was touched here.
test: treat an acquiring, retrying or closing native-main owner as open
lidge-jun commentedon Oct 22, 2026
Update from the maintainer lane: the first case here (
main-account-hard-lock-fallback-routing.test.ts, EBUSY after 15.2 s) only reproduced on #7482's branch. Its cause was in that PR: classifying a routing-profile hard-lock hop ran the policy evaluator, which openedrouting-history.sqliteunder OPENCODEX_HOME and held it during teardown. #7482 was fixed atd7e4df6db9to classify the hop from config only, with a regression test that asserts the test home stays untouched, and merged todevas1025acd4b1(shipped in 2.91.0). Its exact-headlane=allrun 39786507816 passed all nine Windows shards.The remaining teardown work (including
server-management-auth.test.ts) is in #7528. This issue stays open until that lands ondev.test: settle Windows ACL work and owner locks before fixture removal (#…