Skip to content

Fix Windows telemetry flush worker cwd race - #341

Merged
unbraind merged 1 commit into
mainfrom
fix/windows-otel-cleanup-ebusy
Jun 22, 2026
Merged

unbraind merged 1 commit into
mainfrom
fix/windows-otel-cleanup-ebusy

Conversation

@unbraind

@unbraind unbraind commented Jun 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Fix Nightly Validation failed: windows-latest / Node 20 #340 Windows nightly cleanup race by setting the detached telemetry flush worker cwd to os.tmpdir() in src/core/telemetry/runtime.ts.
  • Preserve full test isolation in tests/integration/telemetry-otel-nonblocking.spec.ts: the foreground CLI still runs from the disposable temp root while PM_PATH and PM_GLOBAL_PATH remain sandboxed.
  • Record pm item pm-zpe7 with duplicate-check, GitHub issue/run findings, linked files, tests, reviewer follow-up, and verification evidence.

Root Cause

The scheduled Nightly Validation run 27944143257 failed only on windows-latest / Node 20. The GH-209 foreground nonblocking assertions passed, then afterAll failed to remove the temp root with EBUSY. The telemetry flush worker is detached and inherited cwd from the foreground CLI, so a worker spawned from a disposable project directory could keep that directory busy on Windows after the foreground command exited.

Fix

scheduleTelemetryFlush() now starts the detached worker with cwd: os.tmpdir(). This prevents the worker from locking the caller/project cwd while keeping telemetry queue selection explicit through PM_GLOBAL_PATH.

Verification

  • node scripts/run-tests.mjs test -- tests/integration/telemetry-otel-nonblocking.spec.ts
  • pnpm build
  • node scripts/run-tests.mjs coverage -> 100/100/100/100, 242 files, 4840 tests
  • pnpm quality:static
  • pnpm security:scan
  • pnpm audit --audit-level moderate
  • pnpm changelog:pm:check
  • pm health --check-only --summary --json
  • pm validate --check-resolution --check-history-drift (history drift OK; pre-existing closed-item resolution warnings remain)
  • Manual temp smoke with isolated PM_PATH/PM_GLOBAL_PATH: init, create, update, get, health

Closes #340

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @unbraind, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@unbraind

Copy link
Copy Markdown
Owner Author

Review request for current head 3b4bf168:

@codex review
/gemini review
@gemini-code-assist
@coderabbitai full review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@unbraind, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 37 minutes and 12 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5ede7cfa-466e-41a1-8938-ea6a96bb7956

📥 Commits

Reviewing files that changed from the base of the PR and between 3b4bf16 and eae0268.

📒 Files selected for processing (4)
  • .agents/pm/history/pm-zpe7.jsonl
  • .agents/pm/issues/pm-zpe7.toon
  • CHANGELOG.md
  • src/core/telemetry/runtime.ts
📝 Walkthrough

Walkthrough

Fixes a Windows EBUSY error in the telemetry non-blocking integration test by changing the spawnSync working directory from the disposable sandbox tempRoot to repoRoot, preventing Windows from locking the directory during cleanup. The afterAll rm call gains maxRetries and retryDelay options. PM tracking artifacts for GH-340 are also added.

Changes

Windows EBUSY Telemetry Test Fix

Layer / File(s) Summary
CLI cwd and rm retry fix
tests/integration/telemetry-otel-nonblocking.spec.ts
Changes spawnSync cwd from context.tempRoot to repoRoot to prevent Windows from locking the sandbox directory, adds inline comments explaining the sandboxing rationale, and adds maxRetries/retryDelay to the afterAll rm call.
PM issue record and history log
.agents/pm/issues/pm-zpe7.toon, .agents/pm/history/pm-zpe7.jsonl
Creates the full pm-zpe7 issue file with metadata, repro steps, expected/actual results, and evidence comment; adds the corresponding history log tracking the lifecycle from creation through in_progress, test step planning, and final verification notes.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

  • unbraind/pm-cli#325: Modifies the same tests/integration/telemetry-otel-nonblocking.spec.ts file to improve Windows sandbox cleanup robustness in afterAll, directly overlapping with the retry/backoff changes in this PR.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR addresses issue #340 by investigating and fixing the Windows-specific EBUSY failure during nightly validation cleanup, matching the issue's objective to resolve the failure before trusting the next daily release.
Out of Scope Changes check ✅ Passed All changes are directly scoped to resolving GH-340: the test fix modifies spawn behavior, pm documentation records the issue investigation, and no unrelated modifications are present.
Title check ✅ Passed The title clearly summarizes the main fix: addressing a Windows-specific cwd race condition in the telemetry flush worker that was causing test cleanup failures.
Description check ✅ Passed The description is well-related to the changeset, providing clear context about the GH-340 Windows race condition fix, root cause analysis, and comprehensive verification steps.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request resolves a Windows-specific EBUSY cleanup race in the telemetry OTLP non-blocking integration test by changing the working directory of the spawned CLI process from the disposable sandbox directory to the repository root. This prevents the detached telemetry worker from locking the temporary directory. The review feedback correctly identifies that adding native retry options (maxRetries and retryDelay) to the rm call introduces redundant retry logic, as it is already wrapped in an outer retry loop, and suggests simplifying the cleanup logic.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

for (let attempt = 0; attempt < cleanupAttempts; attempt += 1) {
try {
await rm(context.tempRoot, { recursive: true, force: true });
await rm(context.tempRoot, { recursive: true, force: true, maxRetries: 10, retryDelay: 150 });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The addition of maxRetries: 10 and retryDelay: 150 to the rm options introduces redundant retry logic. The outer for loop already retries the cleanup up to 8 times with a linear backoff delay (delay(100 * (attempt + 1))). Combining both means that each of the 8 outer attempts will internally retry up to 10 times, potentially leading to up to 80 total retries and significantly delaying test failure reporting if the directory is genuinely locked. Recommendation: Either rely entirely on Node's native retry options by removing the outer loop, or keep the outer loop and revert rm to its simpler form.

Suggested change
await rm(context.tempRoot, { recursive: true, force: true, maxRetries: 10, retryDelay: 150 });
await rm(context.tempRoot, { recursive: true, force: true });

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== PR Details ==="
gh pr view 341 --repo unbraind/pm-cli --json title,body,files,commits,additions,deletions,changedFiles

echo ""
echo "=== PR Diff ==="
gh pr diff 341 --repo unbraind/pm-cli

echo ""
echo "=== Changed Files ==="
gh pr view 341 --repo unbraind/pm-cli --json files --jq '.files[].path'

Length of output: 19269


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Full test file ==="
cat tests/integration/telemetry-otel-nonblocking.spec.ts

echo ""
echo "=== repoRoot usage in codebase ==="
rg "repoRoot" tests/integration/telemetry-otel-nonblocking.spec.ts -n

Length of output: 6813



@unbraind The fix is clean, well-reasoned, and correctly addresses the root cause. Here's my full review:


Review of tests/integration/telemetry-otel-nonblocking.spec.ts

✅ Primary fix — cwd: repoRoot (Line 67)

This is the right solution. The root cause is precisely diagnosed: spawnSync spawns a child that launches a detached, unref'd OTLP flush worker. That worker inherits cwd from its parent process, and on Windows a directory cannot be removed while any process has it as its working directory — even if the foreground process has already exited. Switching to repoRoot eliminates the EBUSY race entirely.

repoRoot is correctly computed at line 24:

const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");

Two levels up from tests/integration/ → repository root. ✅

All runCli calls — including the beforeAll setup invocations (init, config set, create) — benefit from this change since they all go through the same helper. ✅

Data isolation is fully preserved: PM_PATH and PM_GLOBAL_PATH are still sandboxed to context.tempRoot, so all tracker writes remain confined even though cwd now points elsewhere. ✅


✅ Secondary fix — maxRetries: 10, retryDelay: 150 on fs.rm (Line 109)

Solid defense-in-depth layer. These options (available since Node.js 16.9.0) delegate retry logic to the native fs layer, which handles transient file-lock errors more efficiently than a manual loop.

One minor observation: the outer loop (8 attempts, 100*(attempt+1) ms backoff) and the inner maxRetries: 10 are now slightly redundant — a single call can retry up to 10 times internally before the outer loop even re-runs. This doesn't cause any problem; it just means the outer loop is now effectively only a safety net for errors that aren't EBUSY/retriable. Arguably the outer loop could be simplified now, but as defensive test infrastructure it's fine to leave both layers in place.


✅ Comment block (Lines 50–53)

The added comment clearly explains the rationale for future maintainers — exactly the right level of detail for a non-obvious platform-specific constraint.


📋 .agents/pm files

Internal project management tracking files — not applicable to code review.


Summary: The fix is minimal, targeted, and well-verified. The cwd: repoRoot change is the substantive correction; the maxRetries/retryDelay addition is a good defensive measure. No concerns. LGTM. 🎉

[approve]

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a Windows-specific cleanup race (GH-340) in the telemetry integration tests by running the spawned CLI from the repository root instead of the temporary directory, and adding retry options to the directory cleanup process. The reviewer points out that changing the test's working directory to the repository root could compromise test isolation and pollute the repository. They suggest a more robust solution: modifying the production code where the detached telemetry worker is spawned to explicitly use a safe directory like os.tmpdir(), thereby preventing it from locking the test's temporary directory.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

const startedAt = Date.now();
const completed = spawnSync(process.execPath, [distCli, ...args], {
cwd: context.tempRoot,
cwd: repoRoot,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Running the spawned CLI with cwd: repoRoot executes the CLI inside the actual repository root. If any command or dependency in the CLI writes to the current working directory (or resolves paths relative to it), it could pollute the repository with untracked files or cause interference between concurrent test runs, violating test isolation.

A much cleaner and more robust fix is to address the root cause directly in the production code where the detached telemetry flush worker is spawned (src/core/telemetry/runtime.ts). By explicitly setting cwd: os.tmpdir() (or another safe directory) when spawning the detached worker, we prevent it from inheriting and locking the temporary directory (context.tempRoot). This allows the test to keep using cwd: context.tempRoot for perfect isolation without any Windows EBUSY cleanup races.

Suggested Production Fix in src/core/telemetry/runtime.ts:

    const child = spawn(process.execPath, [telemetryFlushRunnerPath()], {
      detached: true,
      stdio: "ignore",
      cwd: os.tmpdir(), // Prevent inheriting and locking the parent's temp cwd on Windows
      env: {
        ...process.env,
        PM_GLOBAL_PATH: globalPmRoot,
        [PM_TELEMETRY_FLUSH_CHILD_ENV]: "1",
      },
    });

If you apply the above fix in src/core/telemetry/runtime.ts, you can revert the cwd change in this test back to context.tempRoot to maintain complete test isolation.

Suggested change
cwd: repoRoot,
cwd: context.tempRoot,

@unbraind
unbraind force-pushed the fix/windows-otel-cleanup-ebusy branch from 3b4bf16 to ce315f3 Compare June 22, 2026 11:11
@unbraind unbraind changed the title Fix Windows telemetry OTLP cleanup race Fix Windows telemetry flush worker cwd race Jun 22, 2026
@unbraind
unbraind force-pushed the fix/windows-otel-cleanup-ebusy branch from ce315f3 to 322ba3a Compare June 22, 2026 11:11
@unbraind

Copy link
Copy Markdown
Owner Author

Review request after addressing Gemini feedback on current head 322ba3ad:

  • Moved the fix into src/core/telemetry/runtime.ts by spawning the detached flush worker with cwd: os.tmpdir().
  • Restored foreground test isolation; final diff no longer changes tests/integration/telemetry-otel-nonblocking.spec.ts.
  • Reran targeted test, build, static quality, full 100/100/100/100 coverage, security scan, changelog check, and manual temp smoke.

@codex review
/gemini review
@gemini-code-assist
@coderabbitai full review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a Windows-specific cleanup issue (GH-340) where a detached telemetry flush worker locks the temporary project directory, causing EBUSY errors during test cleanup. The fix updates scheduleTelemetryFlush in src/core/telemetry/runtime.ts to spawn the detached worker with its current working directory (cwd) set to the OS temporary directory (os.tmpdir()). There are no review comments to address, and the changes successfully resolve the directory locking issue.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Current PR diff ==="
gh pr diff 341 --repo unbraind/pm-cli

echo ""
echo "=== Current head info ==="
gh pr view 341 --repo unbraind/pm-cli --json headRefOid,headRefName,commits

Length of output: 20147



@unbraind This is a better fix than the previous iteration — moving it into the production code is the right call.

Review of src/core/telemetry/runtime.ts

✅ cwd: os.tmpdir() in scheduleTelemetryFlush (the substantive change)

This is strictly the correct layer for this fix. The prior approach (cwd: repoRoot in the test) worked around the symptom; this eliminates the root cause for all callers, not just the integration test:

 const child = spawn(process.execPath, [telemetryFlushRunnerPath()], {
   detached: true,
   stdio: "ignore",
+  // Detached workers inherit cwd by default. On Windows that can keep a
+  // disposable project directory locked after the foreground command exits.
+  cwd: os.tmpdir(),
   env: {

Three properties make os.tmpdir() the right choice here:

  1. Always available and writable — the flush worker doesn't need to be in any project directory.
  2. Never disposable — the OS temp dir is never the tempRoot a test or script intends to delete.
  3. Semantically correct — a background telemetry flush worker has no meaningful relationship to the project cwd of its parent.

PM_GLOBAL_PATH is still passed explicitly in the worker's env, so the worker correctly locates and drains the telemetry queue regardless of cwd. ✅

✅ Test file restored

tests/integration/telemetry-otel-nonblocking.spec.ts is absent from the diff, confirming full foreground test isolation is restored. ✅

✅ Comment block

The inline comment clearly explains the Windows-specific rationale for future maintainers — exactly right for a non-obvious platform constraint in production code.


Summary: Clean, minimal, correctly-layered fix. Production code protects all callers; test isolation is fully restored. No concerns. LGTM. 🎉

[approve]

Record pm-zpe7 for GH-340 and set the detached telemetry flush worker cwd to os.tmpdir() so Windows does not keep a caller's disposable project directory locked after the foreground CLI exits. PM_GLOBAL_PATH remains explicit, so the worker still drains the intended telemetry queue without inheriting the command cwd.

Verification: node scripts/run-tests.mjs test -- tests/integration/telemetry-otel-nonblocking.spec.ts; pnpm build; pnpm quality:static; node scripts/run-tests.mjs coverage (100/100/100/100, 242 files, 4840 tests); pnpm security:scan; pnpm audit --audit-level moderate; pnpm changelog:pm:check; pm health --check-only --summary --json; pm validate --check-resolution --check-history-drift; manual isolated PM_PATH/PM_GLOBAL_PATH temp smoke init/create/update/get/health.
@unbraind
unbraind force-pushed the fix/windows-otel-cleanup-ebusy branch from 322ba3a to eae0268 Compare June 22, 2026 11:20
@unbraind

Copy link
Copy Markdown
Owner Author

Final review request for current head eae02682 after closing pm-zpe7 and regenerating package-owned CHANGELOG.md:

  • Code fix remains src/core/telemetry/runtime.ts: detached flush worker uses cwd: os.tmpdir().
  • pm-zpe7 is now closed/released with resolution metadata and evidence.
  • CHANGELOG.md regenerated via pnpm changelog:pm; pnpm changelog:pm:check passes.

@codex review
/gemini review
@gemini-code-assist
@coderabbitai full review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request resolves a Windows-specific cleanup issue (GH-340) where a detached telemetry flush worker inherits the caller's current working directory, locking disposable project directories and causing EBUSY errors during cleanup. The fix sets the cwd of the spawned child process to os.tmpdir() in src/core/telemetry/runtime.ts. Additionally, issue tracking files and a changelog entry have been added. There are no review comments, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Current PR diff ==="
gh pr diff 341 --repo unbraind/pm-cli

echo ""
echo "=== Current head info ==="
gh pr view 341 --repo unbraind/pm-cli --json headRefOid,headRefName,commits --jq '{head: .headRefOid, branch: .headRefName}'

echo ""
echo "=== Changed files ==="
gh pr view 341 --repo unbraind/pm-cli --json files --jq '.files[].path'

Length of output: 22357



@unbraind The final head eae02682 is clean — here's the delta from the last approved iteration:

src/core/telemetry/runtime.ts

Unchanged from the previously-approved state. cwd: os.tmpdir() on the detached flush worker with its explanatory comment remains the only production-code change. ✅

CHANGELOG.md

New "Fixed" entry correctly added:

### Fixed

- GH-340: Windows nightly telemetry OTLP cleanup EBUSY ([pm-zpe7](...))

Placed under the rolling unreleased version block, above the ## 2026.6.22 dated release — correct positioning. The pm-zpe7 link points to the .toon file on main, which is the standard pattern for this repo. ✅

.agents/pm files

pm-zpe7 now shows status: closed with close_reason and resolution metadata populated. History trail is complete and internally consistent. Not code-review applicable. ✅


Summary: No new concerns. All three meaningful layers (production fix, changelog, issue tracking) are correctly completed. LGTM. 🎉

[approve]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nightly Validation failed: windows-latest / Node 20

1 participant