Repository navigation
Indexing large SQL file crashes native parser (Tree-sitter SQL assertion / stack overflow) #668
Description
Summary
index_repository can crash the native process while indexing a large .sql file. In a full repository run I observed a Tree-sitter assertion:
Assertion failed: symbol < self->token_count, file internal/cbm/vendored/ts_runtime/src/./language.c, line 79
I then narrowed this down to a single large SQL file. I cannot attach the original file because it contains private/customer database schema and data scripts, but the failure is reproducible with that single file copied into an otherwise empty temporary repository.
Environment
- OS: Windows
- Shell: PowerShell
- Binary:
C:/Users/lichen/.local/bin/codebase-memory-mcp.exe codebase-memory-mcp --version:codebase-memory-mcp 0.8.1- MCP
initializeserverInfo reported:codebase-memory-mcp0.10.0 - Command mode:
codebase-memory-mcp cli index_repository ...
Reproduction Shape
Full repository indexing crashed while scanning a repository with many SQL scripts:
codebase-memory-mcp.exe cli index_repository '{"repo_path":"C:/path/to/repo","mode":"full"}'The full run reached SQL files under Src/Database and then aborted in the Tree-sitter runtime.
To reduce the input, I copied one SQL file into a new empty directory and indexed only that directory:
$root = "$env:TEMP/cbm-sql-repro"
New-Item -ItemType Directory -Force -Path $root | Out-Null
Copy-Item C:/path/to/CreateDB.sql "$root/CreateDB.sql" -Force
codebase-memory-mcp.exe cli index_repository "{`"repo_path`":`"$($root.Replace('\','/'))`",`"mode`":`"full`"}"The single file is about 2.6 MiB and contains T-SQL database creation/schema/data script content.
Actual Result
Single-file repro exits with native crash code -1073741571 (0xC00000FD, stack overflow on Windows). The output stops during the definitions pass:
level=info msg=mem.init budget_mb=32767 total_ram_mb=65534
level=info msg=pipeline.discover files=1 elapsed_ms=1
level=info msg=pipeline.route path=full
level=info msg=pass.start pass=structure files=1
level=info msg=pass.done pass=structure nodes=2 edges=1
level=info msg=pass.timing pass=structure elapsed_ms=0
level=info msg=pipeline.mode mode=sequential files=1
level=info msg=pkgmap.scan_repo manifests=0
level=info msg=pkgmap.scan manifests_from_files=0 manifests_from_walk=0 entries=0
level=info msg=pass.start pass=definitions files=1
The full repository run also showed the Tree-sitter assertion above:
Assertion failed: symbol < self->token_count, file internal/cbm/vendored/ts_runtime/src/./language.c, line 79
Expected Result
A malformed, huge, or unsupported SQL file should not abort the whole indexing process. Ideally the file should be skipped with an indexed error, or Tree-sitter parse failures should be isolated so index_repository can continue and report failed files.
Relevant Source Observations
From the current repository source:
.sqlmaps toCBM_LANG_SQLinsrc/discover/language.c.- SQL uses the vendored Tree-sitter SQL grammar.
internal/cbm/vendored/grammars/sql/parser.chasTOKEN_COUNT = 429andSYMBOL_COUNT = 770.- The assertion is in
internal/cbm/vendored/ts_runtime/src/language.c:79insidets_language_table_entry, which expectssymbol < self->token_count.
This looks like a Tree-sitter SQL grammar/runtime crash path triggered by large/complex T-SQL input, not a normal SQL syntax error. Normal parse errors should produce ERROR nodes rather than aborting the process.
Notes
I can help test a patched binary or try to produce a minimized/redacted SQL repro if that would be useful. For now I avoided attaching the source SQL because it is private customer project material.
Activity
DeusData commentedon Jun 29, 2026
Thanks for narrowing this down. I’ve triaged it under #390 as a stability bug. Since the original SQL file is private, we can build a synthetic large-SQL fixture and require graceful skip/degradation instead of a native parser crash.
DeusData commentedon Jul 2, 2026
Update, @georgelichen — thank you for the detailed report and the two distinct symptoms; they turned out to be two different bugs.
Stack-overflow mode (
0xC00000FD, the definitions pass): root-caused and fixed in #777 (174e56b). Reproducing under a constrained stack pinned it towalk_defs, which allocated a fixed ~160 KB frame on the C stack (awalk_defs_frame_t stack[4096]). On the older 1 MB Windows main-thread stack (raised to 8 MB only in a 2026-03 build) that single frame overflowed on a file with a large/deep definition tree. It also had a second, silent bug: it dropped every top-level definition past 4096. Both are fixed by moving that stack to a growable heap buffer. (This also refuted an earlier guess that it was the SQL grammar/scanner — extraction is otherwise fully iterative.)Assertion mode (
symbol < self->token_count, the full-repo run): different bug, not yet reproduced. This is an error-recovery state issue in the SQL grammar and we couldn't trigger it with any synthetic SQL. It's being addressed by a general crash-containment layer (in progress) that isolates any file that crashes any language soindex_repositoryskips-and-reports it and keeps going — which will protect you against this mode even before we can pin its exact input.Two asks that would help a lot:
.sqlfile that still triggers the assertion, that's the fastest path to root-causing that second mode.walk_defsheap change should resolve it regardless.)Keeping #668 open until both modes are covered.
Merge pull request #777 from DeusData/feat/668-walk-defs-heap
fix(extract): heap-allocate walk_defs frame stack — fix overflow + >4…
feat(index): report per-file indexing failures via skipped[] + logfil…
fix(lsp): cap resolve-walker recursion depth in py/go/php/kotlin (Sta…
DeusData commentedon Jul 4, 2026
This is now fixed on main (merge 9871e7e, PR #827) — and more generally than the original crash: indexing now runs under a crash/hang supervisor. A file that hard-crashes the native indexer (stack overflow, assertion, scanner memory corruption) or makes it hang is pinned via a single-threaded recovery re-run, quarantined, and reported in the response's skipped[] list (phase "crash" or "hang") — while every other file still indexes normally and the run completes with status "indexed". The original walk_defs stack overflow you hit was also root-cause-fixed separately (heap-allocated frame stack, #777), so your file should now index fully rather than being skipped. Validated on the Windows CI leg. If you need the old behavior for debugging, CBM_INDEX_SUPERVISOR=0 disables the supervisor. Could you retest on main (or the next release) with your repo? Thanks again for the detailed report — it drove a whole resilience effort.
github-actions commentedon Jul 4, 2026
Thanks for the report! To move this forward we need a bit more so we can reproduce it ourselves:
codebase-memory-mcp --versionyou're onOnce that's here we'll pick it straight back up. Heads-up: issues left
awaiting-reporterare automatically closed after a few weeks of silence, but a comment reopens the door anytime.feat(index): supervise index_repository in a crash/hang-isolating worker