Copilot read_file truncation can split UTF-16 surrogate pairs and poison a chat session

[martincheck]

• Copilot Chat Extension Version: 0.61.0
• VS Code Version: 1.133.0
• OS Version: Windows
• Feature: Agent mode
• Selected model: GPT-5.6 Sol
• Logs: filtered synthetic excerpt in the attached clean-reproduction ZIP; identifiers removed

Summary

The built-in Copilot read_file tool truncates lines with JavaScript slice(0, 2000). If an astral Unicode character crosses that UTF-16 boundary, the tool result retains only the high surrogate. The malformed result is persisted in the chat session and included in subsequent requests. Both WebSocket and fallback HTTP then fail with 400 invalid_request_body, and Retry cannot recover because it resends the same poisoned context.

The source appears to be:

https://github.com/microsoft/vscode/blob/main/extensions/copilot/src/extension/tools/node/readFileTool.tsx

const MAX_LINE_LENGTH = 2000;

let contents = rawContents.split('\n').map(line => {
    if (line.length > MAX_LINE_LENGTH) {
        hadLongLines = true;
        return line.slice(0, MAX_LINE_LENGTH) + ' [truncated]';
    }
    return line;
}).join('\n');

Minimal reproduction

The string operation itself is deterministic:

const max = 2000;
const line = 'x'.repeat(max - 1) + '\u{1F6E1}' + 'tail';
const result = line.slice(0, max) + ' [truncated]';

console.log(JSON.stringify(result));
// Ends with: ...xxxxxxxx\ud83d [truncated]"

End-to-end reproduction using the attached clean-repro folder:

1. Run node .\generate-repro-fixture.mjs to create a valid UTF-8 fixture containing 1,999 ASCII characters, then U+1F6E1, then an ASCII tail.
2. Open the folder in a new VS Code window and start a fresh GPT-5.6 Sol agent chat.
3. Prompt: This is a synthetic Unicode regression test. Use only read_file to read surrogate-boundary.txt from line 1 through line 1. Do not use terminal or other tools. After reading it, reply exactly FIRST_TURN_COMPLETE.
4. Observe that the initial model step succeeds and invokes only read_file.
5. Observe that the immediate continuation fails with WebSocket invalid_json, a successful CAPI connectivity ping, and fallback HTTP 400 invalid_request_body. The requested response is never emitted.

No second user prompt is needed; the failure occurs within this first turn when the model continues after the tool result.

Actual result

The synthetic fixture is valid UTF-8 and contains no unpaired surrogates. Its U+1F6E1 occupies UTF-16 indexes 1,999 and 2,000. The fresh chat ledger is valid JSON syntax but its persisted read_file result contains exactly one unpaired U+D83D at zero-based index 1,999, immediately followed by [truncated].

Representative extension-host log sequence:

WebSocket CAPI error: [invalid_json] Invalid websocket message: failed to parse JSON value
CAPI ping successful, proceeding with chat request retry
Server error: 400 {"error":{"message":"Invalid body: failed to parse JSON value...","code":"invalid_request_body"}}
[ToolCallingLoop] Auto-retrying on error (attempt 1/3)

The attached vscode-readfile-surrogate-clean-repro-20260814.zip contains only public-safe synthetic evidence:

• README.md: reproduction instructions and exact prompt.
• generate-repro-fixture.mjs: deterministic fixture generator.
• surrogate-boundary.txt: generated valid UTF-8 fixture.
• synthetic-extension.log: filtered extension-host sequence with request IDs and UUIDs redacted.
• session-structure.json: content-blind structural scan proving read_file was the only tool and locating the lone surrogate.
• extract-sanitized-log.ps1: the exact event filter and redaction script.

The raw chat ledger and raw extension-host log are intentionally not attached.

Expected result

• read_file truncation never splits a surrogate pair.
• Persisted tool results and outbound request bodies contain valid Unicode.
• Deterministic 400 invalid_request_body errors are not retried multiple times with an unchanged body.
• If legacy persisted state contains an unpaired surrogate, Copilot either repairs it with U+FFFD or identifies the bad session and offers a clear recovery path.

Proposed root fix

Before slicing at MAX_LINE_LENGTH, move the end index back by one when the boundary falls between a high and low surrogate. Add a unit test whose emoji begins at index 1,999.

This preserves the existing 2,000 UTF-16-unit cap while ensuring the result is well formed.

Fix validation

Validated against the exact @vscode/copilot-api 0.5.2 lock:

• the focused regression test fails before the production fix and all 35 focused tests pass after it;
• typecheck and compile pass;
• the full Copilot unit suite passes (8,915 tests, with 129 skipped);
• the patched extension completes the synthetic GPT-5.6 Sol turn with FIRST_TURN_COMPLETE;
• a content-blind scan finds one read_file marker, no malformed JSON lines, no unpaired surrogates, and no WebSocket/JSON/400/retry failure signatures.

An exact cause-focused metadata search of microsoft/vscode issues was repeated on 2026-08-14. No issue title or exact query matched read_file splitting a surrogate pair, persisting an unpaired surrogate, or causing the resulting JSON failures. The broader results included #253136, a generic Copilot request-failure tracker, and #321578, titled "read_file reads only 2000 characters"; neither metadata result describes this surrogate-boundary corruption.

Related but distinct prior art

github/copilot-sdk#2283 repairs unpaired UTF-16 surrogate escapes in incoming Rust JSON-RPC frames. That is the same failure class but a different boundary. This issue is produced locally by VS Code's read_file line truncation and later fails on outbound CAPI request construction/parsing.

A defense-in-depth outbound/session sanitizer would be useful, but it should accompany rather than replace the producer fix.

vscode-readfile-surrogate-clean-repro-20260814.zip

[Yoyokrazy]

verified with this file.

boundary.txt