Providers: bound the Cursor skill scan and keep a partial list instead of losing it on timeout #229

Summary

For Cursor, BiBCode finds skills by walking skill folders on disk. The walk has no limits: it goes into node_modules and other dependency folders inside a skill, walks a symlinked root again for every name that points at it, and has no depth, entry or byte budget. A large tree under any skill root can keep it walking until the 15 s discovery timeout. Then the whole catalog is lost: the composer's / menu shows no skills and offers Retry, even though most skills were already found.

T3 Code fixed the same three problems in its Cursor skill scan (pingdotgg/t3code#19806, #18664). This issue makes BiBCode's scan bounded and returns a partial list marked incomplete instead of an error.

Current behavior (verified at a0ea849ef92b)

Roots

• skill_roots (apps/server/src/provider/cursor/capabilities.rs:110-138) builds 4 roots (.cursor/skills, .agents/skills, .claude/skills, .codex/skills) for the workspace, each ancestor up to the Git root, and the user home.
• discover_workspace_capabilities_with_home (:30-71) walks them in that order (:46-58) and dedupes skills by lowercase name, so the first root wins (project before user).

The walk (skill_files, :188-262)

• Depth-first with a pending stack.
• let mut visited = HashSet::new(); at :191 is created per call, i.e. per root. Each folder is canonicalized (:193) and skipped if already visited (:207). Cycles inside one root are caught; this is what follows_user_skill_links_without_looping_or_leaking_another_home (:308) tests.
• But a root that is a symlink to another root (e.g. ~/.agents/skills → ~/.claude/skills), or a project opened at the home directory, is walked again from scratch. Dedupe by name hides the duplicates, but the time is paid twice.
• Every child is checked with fs::metadata (:234-237), which follows symlinks. Every directory is pushed, including the children of a folder that already has a SKILL.md. So a skill's tests/node_modules, .venv, target or .git is walked in full.
• There is no folder skip list, no depth limit, no entry count, and no byte budget.
• Errors become issues strings ("Cannot read skills in …", "Incomplete skills in …"), and the partial catalog is kept.

The timeout

• The Cursor branch of discover_capabilities (apps/server/src/production/provider_inventory/capabilities.rs:27-38) wraps the whole scan in timeout(CURSOR_DISCOVERY_TIMEOUT, …). That is 15 s (apps/server/src/production/provider_inventory.rs:39).
• On timeout it returns Err("Skill discovery timed out. Retry to reload the catalog."), and everything found so far is dropped. Per docs/architecture/providers.md:100-106, a failed discovery raises ServerProviderCapabilitiesError, and the composer offers Retry with an empty catalog.
• Catalogs that report issues are never cached (providers.md:56-62), so a partial catalog is re-read on the next request.

Docs – docs/architecture/providers.md:83-87: "Its traversal follows nested and symlinked directories, guards canonical paths against cycles, and deduplicates names with project precedence."

Existing tests (capabilities.rs)

• :276 user_catalog_survives_an_empty_repo_and_combines_with_project_skills. It includes .codex/skills/.system/system-user, so dot-folders must keep being walked.
• :308 follows_user_skill_links_without_looping_or_leaking_another_home.
• :337 includes_worktree_ancestors_but_excludes_sibling_projects.

Not measured. I haven't timed a BiBCode scan over a real node_modules tree, or checked how large a tree reaches 15 s on each OS. I traced the timeout dropping the catalog in the code, but haven't reproduced it.

Proposed change

1. Skip dependency and build folders. Don't stat or descend into entries named exactly node_modules, .git, target, .venv, venv or __pycache__. Keep walking other dot-folders (.system must still be found). A SKILL.md inside a skipped folder is not a skill. Cursor's own loader skips node_modules, per T3 #19806.
2. One visited set for the whole discovery. Create visited once in discover_workspace_capabilities_with_home and pass it to every skill_files call. A root whose canonical path was already walked, or that leads into an already-walked tree, costs nothing. Root order and project-first precedence stay as they are.
3. A shared scan budget that returns a partial list. One budget across all roots:
   • max depth below a root, e.g. 10;
   • max entries read, e.g. 10,000;
   • skip a SKILL.md over a size cap, e.g. 1 MiB (we only stat it today, so this matters only if we start reading it).
     When the budget runs out, stop the walk and keep the skills found so far. Add one issue such as "Skill list is incomplete: stopped after 10,000 entries under ", and mark the result incomplete (as an issues entry, or a new incomplete flag if the composer should show it differently; see Open questions).
4. The timeout keeps what was found. The budget should normally end the walk well before 15 s. If the timeout still fires (a slow network folder), return the skills found so far with an "incomplete" issue instead of Err. For example, collect into a shared buffer the timeout branch can read, or check a deadline inside the walk.
5. Update providers.md:83-87 to describe the skip list, the shared visited set and the budget.

T3 Code reference (their claims, not measured by us)

• pingdotgg/t3code#19806, "installed node_modules no longer exhaust skill discovery". It skips entries named exactly node_modules (packages/provider-cursor/src/server/skills.ts:206-207). They report that 10,001 entries under review/tests/node_modules made the probe fail with scan-budget-exhausted, and that it returns all skills after the fix. They still walk dot-directories.
• pingdotgg/t3code#18664, "a symlinked skills root is no longer scanned twice". It keeps a set of the real paths of scanned roots (skills.ts:139-143). They report a home where ~/.agents/skills → ~/.claude/skills with 6,000 files charged 12,006 entries and showed "No skills found"; discovery took about 4m45s in that setup.
• T3's limits: MAX_SKILL_DEPTH = 10, MAX_SKILL_SCAN_ENTRIES = 10_000, MAX_SKILL_SCAN_BYTES = 8 MB, MAX_SKILL_BYTES = 1 MB. Difference: T3 still fails the whole probe when the entry budget runs out (#19806 leaves that open for a maintainer decision). This issue asks for a partial list instead.

Acceptance criteria

• Rust tests in apps/server/src/provider/cursor/capabilities.rs, with temp dirs and a passed-in home only (hermetic, never the real HOME):
  • new: a skill with 10,001 entries under review/tests/node_modules, plus a sibling skill and a skill in a later root, returns all three skills with no issues. It fails with only the skip list reverted (or takes the budget path).
  • new: a SKILL.md inside node_modules (and inside .git) is not listed.
  • new (unix): ~/.agents/skills symlinked to ~/.claude/skills walks that tree once (count entries through a test hook or a counter on the scan result), and each skill keeps the scope of the first root that reached it.
  • new: a tree over the entry budget returns the skills found before the budget ran out plus exactly one "incomplete" issue, and it returns in well under 15 s.
  • new: a tree deeper than the depth limit lists skills down to the limit and reports it as incomplete.
  • the three existing tests (:276, :308, :337) still pass; .codex/skills/.system/… is still found.
• A test in apps/server/src/production/provider_inventory/capabilities.rs (or apps/server/tests/) where the scan is slower than the timeout (with a test-only shorter timeout) returns a partial catalog with an incomplete issue, not Err.
• If an incomplete flag is added to the capabilities result: vp run check:contracts with regenerated fixtures, rpcRustParity.test.ts passing, and a composer test (happy-dom, from apps/web) showing the notice.
• Existing behavior unchanged: project skills win over user skills with the same name; broken symlinks are still reported once; Codex, Claude and OpenCode discovery are untouched.
• Desktop e2e: vp run test:ui:desktop, including apps/desktop/e2e/specs/composer-native-triggers.e2e.ts.
• Gates: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, vp check (zero warnings), vp run typecheck, vp run check:contracts, Release Smoke.
• UI.md and vercel-react-best-practices reviews are recorded if the composer changes; otherwise state that there's no UI change. docs/testing/ runbooks (e.g. docs/testing/reports/2026-10-01-provider-skill-discovery.md and the skills section of cross-platform-validation.md) are reviewed and updated or stated "remain accurate".
• Profiler coverage line in the delivery report. Skill paths and names stay out of profiler records; issue strings keep their current shape.

Living docs to update

• docs/architecture/providers.md:83-87 (Cursor traversal) and :100-106 (partial catalogs).

Not in scope

• Reading SKILL.md front matter or changing how skills are named.
• Codex, Claude and OpenCode discovery; they use the providers' own catalogs.
• Caching partial catalogs (see Open questions).

Related

• T3 issue pingdotgg/t3code#17215, the report behind both T3 PRs.

Open questions

• Partial result: is an issues entry enough, or should the composer show an explicit "Skill list is incomplete" notice (a contract field)?
• Partial catalogs aren't cached today (providers.md:56-62), so a home over budget would be re-walked up to the budget on every request. Should a budget-limited result be cached like a complete one?
• Skip list: just node_modules (as T3 and Cursor do), or also .git, target, .venv, venv, __pycache__?
• Separate from Cursor: Claude's Launch arguments setting (launchArgs, packages/contracts/src/settings.ts:372-385, default in apps/server/src/server_settings/mod.rs:17) is only validated as a string (apps/server/src/production/control.rs:2223); I found no use of it in the server's launch or discovery code. If it's really not applied, --add-dir skills from launch arguments (T3 #20167) can't be discovered either. Needs confirming; it would be its own issue.