Repository navigation
[v0.9.0][Windows] search_code content pipeline transcodes UTF-8 through the ANSI codepage — CJK pattern search always returns 0 matches, raw_matches show mojibake #1209
Description
Summary
On Windows (system locale zh-TW, ACP=CP950), search_code never matches CJK patterns even when the pattern verifiably exists in UTF-8 files, and raw_matches[].content returns mojibake with ? replacement characters. The tree-sitter/snippet pipeline (get_code_snippet) handles the same files' CJK content perfectly — only the grep/content pipeline is affected.
Environment
- codebase-memory-mcp 0.9.0 (standard variant, Windows amd64)
- Windows 11 Home 10.0.26200, system locale zh-TW (ANSI codepage 950)
- Repo: Python project, all source files verified UTF-8 (no BOM), indexed via a junction path (ASCII) to work around Windows: get_code_snippet returns "(source not available)" and search_code returns 0 matches for projects indexed under non-ASCII (CJK) paths — indexing itself succeeds #903
Repro
- A UTF-8 Python file contains the string
未來函數(verified: present in 28 files;filereports "UTF-8 text" for all of them). - Call
search_codewith{"pattern": "未來函數", "project": "..."}. - Result:
total_grep_matches: 0, total_results: 0.
Control A (ASCII pattern, same index): search_code for DualTrackRouter → 55 grep matches, works fine.
Control B (same file, snippet pipeline): get_code_snippet on a function whose docstring contains CJK → CJK returned byte-perfect.
Evidence of ANSI-codepage transcoding
For an ASCII pattern that lands on a line that also contains CJK, raw_matches[].content comes back like:
"content": " # ?w?w STEP 2d?G???J???T???J?w?y?{?]DualTrackRouter?^?A"
The original UTF-8 line is ── STEP 2d：載入資訊…（DualTrackRouter）…. This ?-folding pattern is the signature of routing UTF-8 bytes through WideCharToMultiByte/MultiByteToWideChar with CP_ACP (CP950 here): characters unmappable in the ANSI codepage become ?. Since the grep index/content path stores (or compares) the transcoded text, a UTF-8 CJK pattern can never match.
Expected behavior (industry baseline)
Treat file content as UTF-8 by default end-to-end (ripgrep's model: UTF-8 default, BOM sniffing for UTF-16, explicit flag for legacy encodings, U+FFFD for invalid sequences — never the OS ANSI codepage). v0.9.0 already does wide-char correctness for paths (_wfopen, wide-argv); the same treatment is needed for the content pipeline used by search_code.
Related
Adjacent Windows/CJK family: #903 (CJK paths, snippets), #973, #996, #1165 (env vars lossily decoded), #1171 (persistence crash under CJK paths). This one is distinct: it's the content/grep pipeline, not paths — the repo here is indexed under an ASCII junction precisely to avoid the path family, and snippet extraction proves the files themselves are readable.
Impact
Any non-English codebase on Windows (Chinese/Japanese/Korean comments, docstrings, identifiers in strings) silently loses search_code for native-language queries — it returns a clean "0 matches" with no error, so users assume the text simply isn't there.
Activity
DeusData commentedon Jul 22, 2026
Thanks for isolating the content path from the already-known Windows path family. This is a high-priority
0.9.1-rcWindows correctness bug under #394. The RED coverage will use UTF-8 CJK source and patterns under a non-UTF-8 ACP and require byte-correct matching and returned content without any CP_ACP round trip.anfedoro commentedon Jul 22, 2026
I found a narrower, cross-platform UTF-8 issue in the
search_coderesult path.Even when
search_codefinds the correct file and line range, thesourcefield was passed through an ASCII-only sanitizer that replaced every UTF-8 byte >= 0x80 with?. I added a regression test with Russian Markdown and fixed it in my fork.The branch is here: https://github.com/anfedoro/codebase-memory-mcp/tree/agent/fix-search-code-utf8
This does not claim to fix the Windows CP_ACP matching problem described in this issue; it preserves UTF-8 source content after a successful search. If useful, I can open a PR, or the change can be folded into the broader #1209 fix.
DeusData commentedon Jul 24, 2026
Thanks for isolating the second defect. Please do not bundle the ASCII-only output sanitizer with the Windows CP_ACP matching fix: preserving UTF-8 after a successful match is cross-platform result serialization, while the original issue is Windows pattern matching before that point. A focused issue or PR with the Russian Markdown RED fixture is the reviewable path. We will not clone or execute the external fork branch during triage.
DeusData commentedon Sep 26, 2026
Thank you for carefully separating the content path from the Windows path issues, and to @anfedoro for the cross-platform output half!
Two fixes have landed since:
?.search_codeuses on Windows, and passes the pattern through a UTF-8 file. That pipe is where the ANSI codepage produced the?characters you saw.We haven't re-run your exact zh-TW / CP950 setup on Windows ourselves yet. Could you try
未來函數on v0.11.0 and tell us whether the matches and theraw_matchescontent now come back correct? If yes, we'll close this. If not, the output will show which step still transcodes.github-actions commentedon Sep 26, 2026
Thanks for the report! To move this forward we need a bit more so we can reproduce it ourselves:
codebase-memory-mcp --versionyou're onOnce that's here we'll pick it straight back up. Heads-up: issues left
awaiting-reporterare automatically closed after a few weeks of silence, but a comment reopens the door anytime.DeusData commentedon Oct 19, 2026
Thank you for the CP950 reproduction. The outstanding check is still the zh-TW PowerShell case on v0.11.0 or newer: both whether the CJK match is returned and whether its raw content is intact. I have no new environment-specific confirmation to add.