fix(scan): prevent tree-sitter segfault on large/deeply-nested files — closes #116

[Wolfvin]

What

Fixes the CRITICAL RED TEAM issue #116: codelens scan . on the CodeLens repo itself crashed with SIGSEGV (exit 139) in js_backend_parser._parse_variable_declarator and python_parser._walk.

Root cause

tree-sitter 0.26 + tree-sitter-javascript 0.25 + tree-sitter-python 0.25 binding combination has nondeterministic SIGSEGV when walking deeply-nested ASTs. The crash occurs in node.children / node.child_by_field_name access during Python GC, and even with gc.disable() the binding's internal cleanup can invalidate Node pointers held across function boundaries.

The previous two-pass JS parser design (find declarations → then walk each body) held body_node Node references across function boundaries, which dangled when tree-sitter's internal cleanup ran between passes.

Fix — four layers of defense

1. base_parser.walk_tree: convert recursive → iterative DFS with explicit (node, depth) stack, and disable cyclic GC for the walk duration. Adds parse_tree() method that returns the Tree object (not just root node) so callers can hold an explicit reference.

2. base_parser.parse: keep self._last_tree reference so the Tree is not collected while callers hold Node references from the most recent parse.

3. js_backend_parser.extract_references: rewrite from two-pass (which held body_node Node references across passes → dangling pointers) to single-pass recursive walk that processes each declaration's body immediately. Body Node references never leave the callback frame. _find_calls_in_scope also converted from iterative walk_tree to local recursive walk for the same reason. Files > 100 lines are skipped with a warning (pragmatic workaround — tree-sitter-javascript 0.25 still crashes on large files even with all the above mitigations).

4. python_parser.extract_references: disable GC during parse+walk, add max_depth=200 guard to _walk. Files > 200 lines skipped with warning.

5. outline_engine._outline_python / _outline_javascript: route to regex fallback for files > 200 / 100 lines respectively (so codelens scan still generates outline.json without crash).

Verification

# Before (main):
$ codelens scan .
Fatal Python error: Segmentation fault
---EXIT 139---

# After (this branch):
$ codelens scan .
{ ... "status": "ok", "outline_generated": true, "sqlite_persisted": true ... }
---EXIT 0---

Stable across 3 consecutive runs. Test suite: 1183 passed, 1 pre-existing failure (#118 — test expectation mismatch, unrelated to this fix).

Trade-offs

• Files > 100 lines (JS) / 200 lines (Python) are skipped from tree-sitter parsing and fall back to regex outline. They will NOT appear in the call graph (graph_nodes / graph_edges tables). This is a known limitation until tree-sitter bindings are upgraded.
• The skip is logged at WARNING level so users can see which files were excluded.
• All other CodeLens features (dead-code, secrets, vuln-scan, query, arch-metrics, memory, snapshot) still work — they operate on the graph that is built from the smaller files.

Test plan

• codelens scan . on CodeLens repo completes without crash
• Stable across 3 consecutive runs
• Test suite: 1183 passed (same as before, no new regressions)
• Small files (<100 lines JS, <200 lines Python) still parse correctly
• CI: full test suite passes

Risk

Medium. The skip threshold is conservative — large files will be missing from the graph, which could affect downstream analysis (dead-code, arch-metrics). But this is strictly better than crashing the entire scan. When tree-sitter bindings are upgraded to a version without the segfault, the thresholds can be raised or removed.

Related issues

• Closes [RED TEAM] scan command segfaults on CodeLens repo itself — recursive walk_tree crashes (JS & Python parsers) #116 (CRITICAL — scan segfault)
• Does NOT fix [RED TEAM] test_returns_correct_counts expects 97 CALLS edges but gets 76 — pre-existing parser under-count #118 (test expectation mismatch — separate issue)
• Does NOT fix [RED TEAM] find_changed_files silently returns empty when git fails — incremental scan misses modifications on non-git workspaces #117 (find_changed_files git fallback — separate issue)

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud