Skip to content

refactor(plugin): replace language parsers with source previews - #1391

Merged
mldangelo-oai merged 2 commits into
mainfrom
mdangelo/codex/plain-source-previews
Oct 7, 2026
Merged

mldangelo-oai merged 2 commits into
mainfrom
mdangelo/codex/plain-source-previews

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Helpers that build file inventories include short source previews. Their language-specific parsers can discard useful context: a small Python handler becomes only a function signature, and a configuration setting loses its value. Use bounded source text for every file instead, removing the need to maintain those parsers.

Builds on #1347, which is now merged.

Changes

  • Return complete source text when it fits the preview budget, normalizing line endings and trimming outer blank lines. Larger inputs use bounded line samples. Preserve indentation and literal whitespace in retained lines, and retain source text when a tiny budget cannot accommodate a useful sample.
  • Remove the language parsers, whitespace compactor, and unused filename argument. Retain the shared byte-budget fitter from main, including the source fallback for tiny budgets. Share one Git diff parser between inventory and ranking helpers, preserving their existing symlink behavior.
  • Replace grammar and extension matrices with five representative fixtures used across inventory modes. Keep encoding, binary classification, read limits, selected revisions, deletion, and path coverage.
  • Correct the inventory and workflow documentation, remove preview claims from the path-only inventory instructions, and update the pinned SDK workflow digest.

Testing

On the rebased head:

  • All 212 focused Python inventory, preview, and Git snapshot tests passed. They cover preserved source whitespace, tiny preview budgets, encoding, binary classification, bounded reads, selected revisions, deleted files, and staged/unstaged inventories.
  • All 37 focused SDK tests passed against the rebuilt plugin, covering selected-revision previews, compact diff scans, and custom-validation workflow loading.
  • Ruff checks and formatting, the TypeScript CI build, portable plugin source compatibility checks, and all nine compatibility-check tests passed.
  • git range-diff confirms that both preview commits retain identical patches after rebasing onto main following the merge of fix(plugin): include text files in scans regardless of extension #1347.

Repeat the focused Python checks from plugins/codex-security:

python -m pytest -q --tb=short tests/test_rank_preview.py tests/test_generate_rank_input.py tests/test_generate_in_scope_files.py tests/test_workbench_target.py

With dependencies and the verified native build artifact installed, repeat the SDK checks from sdk/typescript:

pnpm run build:plugin
bun test --timeout 30000 tests-ts/diff-rank-input.test.ts tests-ts/compact-diff-scan.test.ts tests-ts/custom-validation.test.ts

No model-quality benchmark was run on this PR head.

Risk and rollout

Helper commands, JSONL fields, binary handling, read limits, and preview byte budgets remain compatible. Small files retain bodies and values. Large-file samples still omit and truncate lines, so they can expose fewer declarations than structural outlines and do not replace reading the file.

Standard, compact diff, and Deep discovery do not use ranking previews to choose files. Direct helper users see the changed preview content; the legacy diff workflow still copies every selected row into full-file review. Before using previews to select scan coverage, compare discovery recall and false positives on large files, templates, and unfamiliar extensions. No CLI flags or migration are required.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T14:54:25.539231Z e401dda New commits
🔒 Security Review ✅ Completed 2026-10-07T14:55:32.805864Z e401dda New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a02e1c1476

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@zcrab-oai zcrab-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff and relevant surrounding code at 6aa9d8a, with an independent second pass. No obvious correctness or regression issues found. Full local platform suites were not rerun. CI still has failed, canceled, or unfinished checks; those results remain separate from this source-review approval.

@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/text-inventory-coverage branch from 9abc6d7 to 736e787 Compare October 7, 2026 08:01
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/plain-source-previews branch 2 times, most recently from 59a35fc to fbd4d81 Compare October 7, 2026 08:55
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/text-inventory-coverage branch from 736e787 to 3dd9a07 Compare October 7, 2026 08:55
Base automatically changed from mdangelo/codex/text-inventory-coverage to main October 7, 2026 14:42
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: fbd4d810b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/plain-source-previews branch from fbd4d81 to e401dda Compare October 7, 2026 14:51
@mldangelo-oai
mldangelo-oai merged commit eb73cc0 into main Oct 7, 2026
114 of 119 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/plain-source-previews branch October 7, 2026 15:41
@github-actions github-actions Bot mentioned this pull request Oct 7, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants