Avoid duplicate unused symbols in by-bucket JSON

[Widthdom]

Summary

• Emit each full unused-symbol row once in the canonical symbols array and represent by_bucket membership with lightweight symbol-index references.
• Expose explicit full and compact by_bucket_format contracts in CLI and MCP JSON, while preserving unknown-bucket completeness.
• Add unused --max-json-bytes support with exact UTF-8 budgeting, gap-free byte-truncated pagination, bounded database errors, and readiness metadata derived from emitted rows.
• Reject incompatible verbose/profile output and final-window truncation cases instead of emitting data outside the requested byte cap or stranding omitted results.

Root cause and impact

The full unused --json --by-bucket response serialized complete symbol objects both in the main result set and again in every bucket, so response size grew with duplicated data. The unused command also lacked a byte-budgeted pagination path. Consumers can now reconstruct bucket membership without duplicated full rows and safely page large responses under a requested JSON byte limit.

Validation

• dotnet restore CodeIndex.sln
• dotnet build CodeIndex.sln -c Release -p:UseSharedCompilation=false --no-restore (0 warnings, 0 errors)
• dotnet test CodeIndex.sln -c Release -p:UseSharedCompilation=false --no-build --no-restore --logger "console;verbosity=minimal"
  • net8.0: 11,016 passed, 7 skipped, 0 failed
  • net9.0: 10,542 passed, 420 skipped, 0 failed
• Focused Issue Avoid duplicating unused symbols in by-bucket JSON output #4904 CLI, MCP, byte-budget, and formatter regression tests
• make lint
• Changelog fragment validation (78 passed)
• CLI dogfood checks for bounded output, invalid database errors, verbose incompatibility, and cursor paging
• Codex adversarial review: 2 rounds completed; all 5 reported findings were addressed and revalidated

Documentation and changelog

• Updated the English and Japanese sections of README.md and TESTING_GUIDE.md.
• Added changelog.d/unreleased/4904.fixed.md.

Follow-up candidates

None.

Fixes #4904