docstring truncation splits multi-byte UTF-8 characters, producing invalid UTF-8 in nodes.properties (non-ASCII comments, e.g. Japanese) #1017

Summary

Long docstrings/comments containing non-ASCII multi-byte UTF-8 text (observed with Japanese) can get truncated mid-character during extraction, leaving an incomplete UTF-8 byte sequence embedded in the properties JSON stored in the SQLite nodes table. This produces a properties value that is syntactically valid JSON but semantically invalid UTF-8 — any consumer that decodes the SQLite TEXT column strictly as UTF-8 (e.g. Python's sqlite3 module, which is the default for anyone building tooling on top of the exported database) throws on read.

Because this only manifests with multi-byte UTF-8 characters, it never reproduces on English-only codebases — a single-byte-per-character comment can be truncated anywhere without producing invalid UTF-8. It only shows up when indexing repos with non-ASCII comments (Japanese/Chinese/etc.).

Environment

• codebase-memory-mcp version: 0.9.0 (Windows amd64, pre-built release binary)
• Repro repo: EC-CUBE/ec-cube (public, MIT-licensed Japanese e-commerce platform — real-world Japanese comments throughout)
• Indexed with mode=full

Repro steps

git clone --depth 1 https://github.com/EC-CUBE/ec-cube.git
codebase-memory-mcp cli index_repository --repo-path <path>/ec-cube --mode full --name ec-cube

Then read the nodes.properties column with any strict-UTF-8 SQLite client, e.g.:

import sqlite3
conn = sqlite3.connect("ec-cube.db")
conn.execute("SELECT id, name, file_path, properties FROM nodes WHERE label IN ('Function','Method')").fetchall()
# sqlite3.OperationalError: Could not decode to UTF-8 column 'properties' with text '...'

This one bad row aborts the entire fetchall() — not just that row — since sqlite3 decodes the whole result set as UTF-8 by default.

Observed corrupted data

Node id=5570, qualified_name="ec-cube.src.Eccube.Controller.ShoppingController.ShoppingController.index", file_path="src/Eccube/Controller/ShoppingController.php:72-151".

Raw bytes of the properties value around the break (hex):

94 9f e6 88 90 e3 82 92 e8 a1 8c e3 82 8f e3 81 9a e3 81 ab e7 22 2c 22 73 69 67 ...

Decoding what's decodable: e6 88 90=成 e3 82 92=を e8 a1 8c=行 e3 82 8f=わ e3 81 9a=ず e3 81 ab=に, then a lone e7 byte — the first byte of a 3-byte UTF-8 sequence with no continuation bytes — immediately followed by the JSON closing quote (22) and ,"signature":.... The docstring was cut exactly one byte into the next kanji character.

Scope observed

In this one EC-CUBE index run: 12 of 12,577 Function/Method nodes had this corruption (~0.1%) — small in proportion, but each one is enough to break any strict-UTF-8 batch read of the whole nodes table.

Suspected root cause (not fully confirmed — flagging both candidates)

I traced two structurally identical byte-oriented (not UTF-8-codepoint-aware) truncation loops:

• append_json_string() in src/pipeline/pass_parallel.c (~line 401)
• append_json_string() in src/pipeline/pass_definitions.c (~line 191, explicitly commented as "Twin of pass_parallel.c — keep both in sync")

Both advance for (const char *s = val; *s && p < bufsize - PP_ESC_MARGIN; s++) one raw byte at a time, with the truncation boundary checked purely in bytes — no awareness of where a multi-byte UTF-8 character starts/ends.

However, the surrounding code documents an "atomic" precheck (required = ... + pp_json_escaped_len(val) + ...; if (*pos + required + ... > bufsize) return;) specifically intended so a field is only ever emitted whole-or-not-at-all, which — if working as documented — should mean the truncation branch inside the loop shouldn't fire for a value that already passed the precheck. So this loop may not actually be the site that produced this specific corruption; it's more likely the raw comment/docstring text gets copied into a size-capped buffer earlier, during comment extraction, with the same byte-oriented (not UTF-8-boundary-aware) cut. I didn't have that extraction step in the source I was working from to pin the exact line — filing this with the strongest concrete evidence I have (the corrupted output + the byte-oriented pattern found) rather than guessing further.

Suggested fix direction

Wherever a string gets truncated to a byte budget (docstring capture, and/or the append_json_string JSON-escape loop as defense in depth), snap the cut point back to the last complete UTF-8 character boundary instead of cutting mid-sequence by raw byte count — or drop the trailing partial multi-byte sequence rather than emitting it.

Happy to share the indexed ec-cube.db (or a minimal repro file) if useful — found this while building tooling on top of the exported SQLite graph.