Best-effort indexing-coverage signal: mark files/regions that were not fully indexed

Best-effort indexing-coverage signal

Framing (read first): this is a best-effort attempt to mark code that was not fully indexed into the graph. A flag means "we know this file/region is incomplete — prefer grep here." The absence of a flag is NOT a guarantee of completeness — it means no gap was detected, not that none exists. We never present this as a completeness proof.

Why

Today the graph can silently drop constructs and nothing surfaces it. Concrete case: cbm_path_within_root (an #ifdef-split-brace function) was entirely absent from the graph while its callers stayed linked (#960, #961). An agent searching the graph has no way to know it might be reading an incomplete picture and should fall back to grep.

What we can detect (best-effort, in decreasing confidence)

Class	Example	Signal
Purposely ignored	gitignore / excluded dirs / `.cbmignore` / vendored	already in the index response `excluded` — surface as `not_indexed_by_design` (a design note, kept separate from gaps)
File-level skip	unreadable / oversized / parse-timeout / worker crash	already in `skipped[]` (#785)
Parse-partial	`#ifdef`-split brace, truncated file, unparseable syntax	NEW — `ts_node_has_error(root)` + ERROR/MISSING line ranges. Verified: tree-sitter reports `has_error=TRUE` for exactly this case.
Extractor gap	tree-sitter parsed a `function_definition`, extractor didn't emit a node	NEW — per-file reconciliation of tree-sitter recognized-construct count vs. emitted node count (analog of the existing `dump_verify`). Per-language, only where calibrated to zero false positives.
Clean-but-wrong misparse	valid tree, mis-categorized def, count matches	Undetectable — the honest residual; part of why this is best-effort.

Deliverables

Signal A (first increment): capture has_error + ERROR line-ranges per file → mark the File node (a present-when-incomplete marker, so absence never implies "complete") → surface in the index_repository response as parse_partial[]. Reproduce-first test with the #ifdef fixture (RED before, GREEN after). This alone would have caught cbm_path_within_root.
Signal B (follow-up): parse-vs-extract count reconciliation, per calibrated language.
Agent surfacing: the File-node marker so search/trace/snippet results carry "this file is partial" inline; and an index_coverage view (extend index_status) with grep-fallback guidance.
Docs: state the best-effort guarantee explicitly wherever the signal is exposed.

Non-goals / honesty guards

No claim of exhaustive completeness. Absence of a flag ≠ complete.
Signal B must not false-positive — uncalibrated languages get Signal A only.
not_indexed_by_design (ignored files) stays a separate, clearly-labelled list — never mixed into "gaps."

Relates to #961 (the extractor limitation this makes visible).

feat(index): missed graph in UI + read-time coverage hook + recovery …

feat(ui): missed skeleton beside the code galaxy with click-to-focus

test(smoke): matcher guard tracks Read inclusion for the coverage note

feat(mcp): fold coverage into index_status; coverage note on get_code…

feat(ui): report-an-edge-case callout when the missed skeleton is sel…

fix(ui): serve the upstream issues URL from the backend

Shipped via PR #968 (merge commit 97c6750, 8 commits a4c17bd…3eb294f, milestone 0.9.1-rc):

Capture: cbm_extract_file flags files whose parse tree contains ERROR/MISSING regions, with 1-based line ranges (top-most regions, capped at 64). Definite recovery is subtracted first — a region fully re-extracted as definitions is not a miss, so e.g. a recovered def broken(: no longer produces a false flag, while the #ifdef-split case stays flagged (its first-branch function is genuinely lost).
Report: index_repository returns parse_partial (distinct from skipped[] — these files WERE indexed); the per-run logfile lists both classes.
Persist: separate index_coverage table written in the same persist step as all other post-dump SQL writes; full runs replace, incremental runs merge (fixed file → flag clears, deleted file → pruned). Coverage never mixes into the graph tables.
Query: index_status carries the coverage report (no dedicated tool); query_graph(graph="missed") queries the miss graph — the file structure of only the not-fully-indexed files, materialized under a shadow project.
Agent surfaces: the PreToolUse augmenter (matcher now Grep|Glob|Read) injects a coverage note when a flagged file is read; get_code_snippet responses from flagged files carry a correlated coverage_note. Both non-blocking, both anchored to a concrete file — no uncorrelated noise.
UI: the missed graph renders as a white skeleton satellite beside the code galaxy (click to focus, click empty space to return, toggle in the sidebar); selecting it opens a report-an-edge-case callout with a prefilled upstream issue link and a copy-paste agent prompt.

Everything is deliberately best-effort: a flag means constructs in the listed ranges MAY be missing (prefer grep there); absence of a flag is NOT a completeness guarantee.

Validation on this repo's own index: 44 files flagged; the two large regions each contained exactly one genuinely lost function (cbm_path_within_root, handle_process_kill — both the in-function #ifdef _WIN32 split tracked in #961).

Remaining scope, keeping this issue open: the deliberately-ignored files listing ("purposely not indexed" — gitignore/excludes/mode-skips as a by-design class, exhaustive and deterministic), planned as a follow-up PR on the same milestone.

feat(index): best-effort parse-coverage signal for partially parsed f…

feat(index): persist coverage in separate table + queryable miss graph

The final piece is merged: PR #974 (merge commit d6bbb0a) adds the BY-DESIGN class — deliberately-ignored files/dirs (gitignore/.cbmignore/skip-lists) are reported as "purposely not indexed" with per-file reasons, strictly separated from the failure signals (never in skipped/parse_partial, never in the missed graph, never triggering the UI callout). Per-file entries cap at 2000 with the uncapped total always reported; excluded subtrees stay exhaustive.

With that, the full coverage taxonomy from this issue is shipped across index_repository / index_status / query_graph(graph="missed") / the read hook / get_code_snippet / the UI skeleton + callout:

parse_partial — indexed, constructs in flagged ranges MAY be missing (best-effort, recovery-subtracted)
skipped — not indexed at all (oversized/read/parse failure)
not_indexed — deliberately excluded, by design (deterministic)

Closing. Residual known limitation (documented in the tool notes): a file that parses cleanly but extracts wrongly is undetectable by this signal — absence of a flag is NOT a completeness guarantee. The concrete extractor gaps this signal exposed are tracked in #961.

perf(coverage): find a line by table lookup, not by rescanning the fi…

fix(coverage): stop reporting a duplicate range and a line past EOF (D…

Merge pull request #1941 from CaptainMittens/fix/parse-coverage-hones…

fix(coverage): stop reporting a duplicate range and a line past EOF (D…

ci(coverage): fail a PR when this repo's own parse-coverage report go…

ci(coverage): fix three ways the coverage gate could pass without che…

ci(coverage): raise the parse-partial ceiling to 59, for a gap main a…

ci(coverage): compare the parse-coverage gate against the merge base (D…

Merge pull request #1971 from CaptainMittens/fix/coverage-range-dupli…

fix(coverage): narrow parse-error ranges with the preprocessed parse (D…

fix(coverage): stop the report hiding what it dropped, and name whole…

test(coverage): pin the truncation marker, the partial ceiling and th…

fix(coverage): name the whole-file number for what it is, a range end (…

perf(coverage): find a line by table lookup, not by rescanning the fi…

fix(coverage): stop reporting a duplicate range and a line past EOF (D…

Navigation Menu

Repository navigation

Best-effort indexing-coverage signal: mark files/regions that were not fully indexed #963