Skip to content

fix(graph): incremental scan populates graph tables — closes #25 #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Wolfvin merged 4 commits into from
Jun 28, 2026
Merged

fix(graph): incremental scan populates graph tables — closes #25 #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4cf4959
feat(graph): incremental_graph_update function
Jun 28, 2026
18dad52
feat(scan): incremental path populates graph
Jun 28, 2026
5de5154
test(graph): incremental update tests
Jun 28, 2026
a51b143
docs(graph): CHANGELOG #25
Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,100 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [8.2.0] — Unreleased

### Incremental Graph Update (issue #25)

Previously, `scan --incremental` updated only the flat backend registry
and skipped graph population entirely. As a result, `graph_nodes` and
`graph_edges` became stale after any incremental scan — `trace --use-graph`
returned outdated callers/callees, and the recommended post-edit workflow
(`scan --incremental`) silently broke the graph backend that #8 introduced.

This fix adds a slice-level update path: only the changed files' nodes
and edges are deleted and re-inserted from the flat registry, then
`refine_call_edges` (from #13) is re-invoked to rebuild IMPORTS edges
and re-refine CALLS edges. The full scan path is unchanged — it still
calls `populate_graph_tables` for a bulk rebuild.

### Added (issue #25)

- **`scripts/graph_model.py:incremental_graph_update(workspace, db_path, changed_files)`**
— New function. Performs a slice-level graph update:
1. Normalize `changed_files` (absolute paths) to workspace-relative
paths. Empty input is a no-op (returns zero counts) so the
no-changes path in `scan --incremental` is safe.
2. Identify `graph_nodes` rows whose `file` is in the changed set
(the stale node ids that must be replaced).
3. Delete `graph_edges` rows that touch any changed file:
- edges whose `file` (originating file) is in the changed set
(covers CALLS edges from changed files + IMPORTS edges whose
importer changed), AND
- edges whose `source_id` or `target_id` references a stale node
id from step 2 (covers cross-file edges from an unchanged file
into a changed file — the target may have been renamed/moved).
4. Delete the stale `graph_nodes` rows themselves.
5. Re-read the flat backend registry (already updated by
`merge_backend_data` in the scan pipeline) and INSERT only the
nodes whose `file` is in the changed set, plus CALLS edges that
touch the changed set (either endpoint's file is in the set).
6. Call `refine_call_edges(workspace, db_path)` so IMPORTS edges
and import-aware CALLS-edge refinement are rebuilt for the
affected slice. `refine_call_edges` is idempotent (it clears
and rebuilds the `import_registry` table and IMPORTS edges from
scratch each call), so invoking it here is safe regardless of
whether a previous scan already ran it.

Returns: `{nodes, edges, edges_refined, edges_unresolved}` where
`nodes`/`edges` are the TOTAL graph row counts after the update
(not the delta) so the return shape matches `populate_graph_tables`.

- **`tests/test_graph_incremental.py`** — 19 tests across 9 classes
covering: no-op cases, equivalence with full populate, idempotency,
slice isolation, file modification reflection (rename/add/remove),
stale edge dropping (no orphan edges), return-value shape, end-to-end
via `cmd_scan(incremental=True)` (graph field present in BOTH full
and incremental scan output with matching counts), and a performance
assertion (<200ms for 5 changed files; issue spec targets <100ms).

### Changed (issue #25)

- **`scripts/commands/scan.py`** — Incremental path now calls
`incremental_graph_update(workspace, db_path, changed_files)` instead
of skipping graph population. The full-scan path is unchanged (still
calls `populate_graph_tables` + `refine_call_edges`). Scan output now
ALWAYS includes a `graph` field with the actual final state
(`{nodes, edges}`) of `graph_nodes` + `graph_edges`, regardless of
scan mode — previously the incremental path emitted no `graph` field
at all. The `type_resolution` field is populated by the incremental
path too (from `incremental_graph_update`'s return value).

### Non-Breaking (issue #25)

- Full-scan behavior is unchanged — `populate_graph_tables` is still
called for a clean bulk rebuild on every full scan.
- The incremental path is best-effort: any failure inside
`incremental_graph_update` is logged at WARNING level and swallowed,
so the flat registry remains the source of truth and the scan still
succeeds (matches the existing full-scan error-handling contract).
- The function is idempotent — running twice with the same
`changed_files` yields the same final graph state.
- The `graph` field added to the incremental-scan output is additive;
no existing field is removed or renamed. Consumers who previously
special-cased the missing `graph` field on incremental scans see a
populated `{nodes, edges}` shape identical to the full-scan output.
- On the `clean_app` fixture: full scan reports
`graph: {nodes: 31, edges: 134}` (CALLS + IMPORTS, after refine).
Subsequent incremental scan with no changes reports the same counts.
Incremental scan after renaming `format_text` → `format_text_renamed`
in `src/utils.py` updates the graph (renamed node present, old name
gone from that file) and reports updated edge counts.

### Migration Notes for Engine Authors (issue #25)

Engines that read `graph_nodes` / `graph_edges` after an incremental
scan no longer need to fall back to the flat registry or trigger a
manual full scan — the graph is always in sync with the flat registry
after `cmd_scan` returns, regardless of `incremental=True/False`.

### Confidence Fields on Non-Deep Output (test fix)

Previously, the `confidence` / `confidence_distribution` fields were only
Expand Down
125 changes: 92 additions & 33 deletions scripts/commands/scan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,20 @@ def cmd_scan(workspace: str, incremental: bool = False, plugins: Optional[list]
fe_classes = existing_frontend.get("classes", [])
fe_ids = existing_frontend.get("ids", [])

# No changes detected → graph tables are unchanged from the
# previous scan. Report current graph stats so the scan
# output shape (issue #25) matches the full-scan output:
# both expose a `graph` field with `{nodes, edges}`.
graph_stats_existing = {"nodes": 0, "edges": 0}
try:
from graph_model import graph_stats as _graph_stats
from graph_model import _default_db_path as _scan_db_path2
graph_stats_existing = _graph_stats(_scan_db_path2(workspace))
except Exception:
logger.debug(
"graph_stats failed in no-changes path", exc_info=True
)

return {
"status": "ok",
"workspace": workspace,
Expand Down Expand Up @@ -172,6 +186,13 @@ def cmd_scan(workspace: str, incremental: bool = False, plugins: Optional[list]
"frameworks": config.get("frameworks", []),
"unsupported_langs": fw.get("unsupported_langs", []) if fw else [],
"lang_note": _build_lang_note(fw) if fw else None,
# Issue #25: incremental scan output includes graph stats
# so consumers can verify graph is populated without a
# separate `graph-schema` call.
"graph": {
"nodes": graph_stats_existing.get("nodes", 0),
"edges": graph_stats_existing.get("edges", 0),
},
}

# Handle deleted files: remove from mtimes cache and clean registry
Expand Down Expand Up @@ -911,40 +932,60 @@ def cmd_scan(workspace: str, incremental: bool = False, plugins: Optional[list]
}
save_backend_registry(workspace, backend_registry)

# ─── Graph Data Model Population (issue #8) ─────────────────
# After the flat backend registry is built, populate the graph_nodes +
# graph_edges tables from it in a single bulk transaction. This is
# additive and non-breaking — the flat registry remains the source of
# truth; the graph tables are a derived projection that engines can
# query for structural traversals (callers/callees/cycles/blast-radius).
# ─── Graph Data Model Population (issue #8 + #25) ────────────
# After the flat backend registry is built, populate (full scan) or
# incrementally update (incremental scan) the graph_nodes + graph_edges
# tables. The incremental path (issue #25) updates only the affected
# slice — graph stays in sync without a full re-population.
#
# In incremental mode, incremental_graph_update also runs
# refine_call_edges internally (so we skip the redundant full refine
# call below). In full-scan mode, populate_graph_tables + refine_call_edges
# run separately (existing flow from #8 + #13).
#
# Failures here MUST NOT break the scan — the graph is an optimization
# layer; engines fall back to the flat registry if it's missing.
graph_population = {"nodes": 0, "edges": 0}
try:
from graph_model import populate_graph_tables, _default_db_path
db_path = _default_db_path(workspace)
graph_population = populate_graph_tables(workspace, db_path)
except Exception:
logger.warning("Graph table population failed", exc_info=True)

# ─── Hybrid Type Resolution (issue #13) ──────────────────────
# Post-pass that enriches CALLS edges with receiver type info via
# an import-aware resolver. Also writes IMPORTS edges to graph_edges
# and populates the import_registry table. Additive — only refines
# existing edges in place; never removes or replaces them. Failures
# here MUST NOT break the scan (type resolution is an optimization
# layer; unresolved edges fall back to name-based resolution).
type_resolution = {"edges_refined": 0, "edges_unresolved": 0}
try:
from hybrid_type_resolver import refine_call_edges
from graph_model import _default_db_path as _tr_db_path
tr_stats = refine_call_edges(workspace, _tr_db_path(workspace))
type_resolution = {
"edges_refined": tr_stats.get("edges_refined", 0),
"edges_unresolved": tr_stats.get("edges_unresolved", 0),
}
except Exception:
logger.warning("Hybrid type resolution failed", exc_info=True)
if incremental and changed_files:
try:
from graph_model import (
incremental_graph_update, _default_db_path,
)
db_path = _default_db_path(workspace)
inc_result = incremental_graph_update(
workspace, db_path, changed_files
)
type_resolution = {
"edges_refined": inc_result.get("edges_refined", 0),
"edges_unresolved": inc_result.get("edges_unresolved", 0),
}
except Exception:
logger.warning("Incremental graph update failed", exc_info=True)
else:
try:
from graph_model import populate_graph_tables, _default_db_path
db_path = _default_db_path(workspace)
populate_graph_tables(workspace, db_path)
except Exception:
logger.warning("Graph table population failed", exc_info=True)

# ─── Hybrid Type Resolution (issue #13) ──────────────────
# Post-pass that enriches CALLS edges with receiver type info via
# an import-aware resolver. Also writes IMPORTS edges to graph_edges
# and populates the import_registry table. Additive — only refines
# existing edges in place; never removes or replaces them. Failures
# here MUST NOT break the scan (type resolution is an optimization
# layer; unresolved edges fall back to name-based resolution).
try:
from hybrid_type_resolver import refine_call_edges
from graph_model import _default_db_path as _tr_db_path
tr_stats = refine_call_edges(workspace, _tr_db_path(workspace))
type_resolution = {
"edges_refined": tr_stats.get("edges_refined", 0),
"edges_unresolved": tr_stats.get("edges_unresolved", 0),
}
except Exception:
logger.warning("Hybrid type resolution failed", exc_info=True)

# ─── Git-aware scan bookmark (issue #14) ─────────────────────
# After a successful scan, record the current HEAD SHA + branch so the
Expand Down Expand Up @@ -997,6 +1038,20 @@ def cmd_scan(workspace: str, incremental: bool = False, plugins: Optional[list]
logger.warning(f"Failed to load plugin rules: {e}")
plugin_rules_data = {"error": str(e)}

# ─── Final graph stats (issue #25) ───────────────────────────
# Query the actual graph state AFTER all post-scan passes (populate +
# refine for full scan; incremental_graph_update for incremental scan)
# so the reported counts reflect the true final state — CALLS + IMPORTS
# edges, after type resolution. Both full and incremental paths report
# the same shape so consumers can compare counts across scan modes.
final_graph_stats = {"nodes": 0, "edges": 0}
try:
from graph_model import graph_stats as _final_graph_stats
from graph_model import _default_db_path as _final_db_path
final_graph_stats = _final_graph_stats(_final_db_path(workspace))
except Exception:
logger.debug("final graph_stats query failed", exc_info=True)

result = {
"status": "ok",
"workspace": workspace,
Expand Down Expand Up @@ -1057,9 +1112,13 @@ def cmd_scan(workspace: str, incremental: bool = False, plugins: Optional[list]
"changed_files_count": len(changed_files) if changed_files else 0,
"unsupported_langs": fw.get("unsupported_langs", []) if fw else [],
"lang_note": _build_lang_note(fw) if fw else None,
# Issue #25: graph field reflects the actual final state of
# graph_nodes + graph_edges (CALLS + IMPORTS, after type
# resolution). Both full and incremental scans report the same
# shape so consumers can compare counts across scan modes.
"graph": {
"nodes": graph_population.get("nodes", 0),
"edges": graph_population.get("edges", 0),
"nodes": final_graph_stats.get("nodes", 0),
"edges": final_graph_stats.get("edges", 0),
},
"type_resolution": {
"edges_refined": type_resolution.get("edges_refined", 0),
Expand Down
Loading
Loading