Skip to content

fix(code): preserve completed Codex review feedback - #204

Open
PNWGMR wants to merge 5 commits into
closedloop-ai:mainfrom
PNWGMR:codex/fix-359-utf8-feedback
Open

PNWGMR wants to merge 5 commits into
closedloop-ai:mainfrom
PNWGMR:codex/fix-359-utf8-feedback

Conversation

@PNWGMR

@PNWGMR PNWGMR commented Sep 26, 2026 •

Copy link
Copy Markdown

Purpose

Fix completed Codex plan reviews whose UTF-8 JSONL contains characters outside the Windows default code page. The existing wrapper can lose the final verdict, truncate feedback, and hide the parser error after the model has already completed.

Tracks pnwgmr-llc/ai-agent-skills#359: https://github.com/pnwgmr-llc/ai-agent-skills/issues/359

Changes

  • Decode the JSONL stream explicitly as UTF-8; require a completed turn and an agent message before publishing feedback.
  • Stage feedback beside its destination and replace it atomically after validation. Preserve the previous feedback on malformed or partial streams.
  • Retain Codex and parser stderr under the log ID, emit a structured failure with log/session identity, and avoid a fresh review when a failed resume emitted nonempty JSONL.
  • Add synthetic shell-level regressions for a non-ASCII completed verdict and malformed/partial streams. No model review was rerun.
  • Bump the code plugin patch version and update the changelog. The documentation workflow also corrected the existing design-inventory command entry in the code plugin README.

Verification

  • Original-script regression: the non-ASCII completed fixture fails under narrow stdout encoding; fixed head passes.
  • pytest plugins/code/skills/codex-review/tests -q — 3 passed (Python 3.13).
  • ruff check on new Python files — passed.
  • pyright --pythonpath <Python 3.13 venv> on new Python files — 0 errors.
  • bash -n plugins/code/skills/codex-review/scripts/run_codex_review.sh — passed.
  • git diff --check — passed.

Risk: Medium. The parser now refuses incomplete JSONL and retains prior feedback; synthetic tests cover that failure path. The upstream Actions run requires maintainer approval for fork code, and maintainer review is pending. A local Windows repository-wide run had 1,798 passed, 380 failed, and 3 skipped; many unrelated shell tests resolved bash to WSL without /bin/bash, and Unix-permission assumptions also failed. The changed codex-review tests all passed. Full upstream Linux CI remains unverified.

- Decode JSONL as UTF-8 and publish validated feedback atomically.
- Retain parser and Codex diagnostics with the log identity.
- Avoid retrying a failed resume that emitted a partial stream.
- Cover completed, malformed, and partial synthetic JSONL.

Testing: Focused pytest, Ruff, Pyright, bash -n, git diff --check.
Risks: Incomplete JSONL now fails closed instead of replacing feedback.
@PNWGMR

PNWGMR commented Sep 26, 2026

Copy link
Copy Markdown
Author

Read-only exact-head review of bbde4b4 against upstream main: no P0-P2 behavioral findings. Two P3 findings were identified: codex-review/SKILL.md still described the prior unconditional resume fallback, and a malformed stream after thread.started could report the previous session ID in CODEX_FAILED. Both are corrected in the next local commit; the synthetic failure fixture now verifies the new session identity while preserving prior feedback. This is an author-posted review receipt, not a maintainer approval. Upstream Actions currently reports action_required for this fork PR, so the maintainer-controlled CI gate is pending.

@PNWGMR

PNWGMR commented Sep 26, 2026

Copy link
Copy Markdown
Author

Exact-head independent read-only review receipt for 6f6d5973f6cfba9e2235f6b6c6406276903ed105 against upstream main at 284077476656dacdfab09408cdaa78e378ad5ea8:

  • No P0-P2 findings in the effective diff.
  • The prior documentation mismatch is resolved in SKILL.md: the resume fallback, UTF-8 validation, diagnostics, and atomic feedback replacement now match the wrapper.
  • The prior partial-stream session loss is resolved: the parser recovers an already emitted thread.started ID for the failure token without accepting incomplete feedback. The synthetic malformed/partial fixture asserts the recovered ID and preserved prior feedback.
  • Focused Python 3.13 test: 3 passed; Ruff and targeted Pyright clean; Bash syntax and diff checks clean. The non-ASCII regression was red against the original parser and green with this change. No paid Codex review was rerun.

This is a contributor-posted independent analysis receipt, not a maintainer approval. Upstream Actions run for this head is action_required pending maintainer approval for fork code, and the PR still requires maintainer review.

@PNWGMR
PNWGMR marked this pull request as ready for review September 26, 2026 19:51
@PNWGMR

PNWGMR commented Sep 26, 2026

Copy link
Copy Markdown
Author

This contribution is ready for maintainer review. GitHub Actions run 36267459568 for the exact head is marked action_required because the workflow requires maintainer approval for a forked PR. Could a maintainer approve that run and review the patch when convenient? The synthetic regression, focused tests, and local verification are recorded above; I have not rerun any paid Codex review.

@mikeangstadt mikeangstadt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right fix for the right bug: gating publication on turn.completed and atomically replacing the file is what this needed. Ran it locally on Linux and your unverified CI is fine (ruff clean, pyright 0 errors, 3 passed), so that risk note can come down. Two things inline about the new strictness, neither blocking.

PNWGMR and others added 2 commits October 7, 2026 01:18
Conflict resolution, plus the review points from closedloop-ai#204:

- run_codex_review.sh: keep upstream's per-run codex_stderr and
  codex_failure_reason, and also append each run's stderr to the retained
  $LOG_ID.stderr. parse_failure_message now reads the JSONL as UTF-8.
- Fresh-session fallback is gated on "no thread.started in the stream"
  (parse_codex_json.py --thread-id) instead of file size, so a dead thread ID
  that prints only an error event no longer wedges every retry.
- parse_codex_json.py skips blank and non-JSON lines (counted on stderr)
  instead of failing the whole stream; turn.completed plus an agent message
  remains the publication gate. An incomplete stream exits 3.
- Failure tokens keep upstream's contract: a failed codex run reports
  "CODEX_FAILED:codex exited with code N: <reason>"; a clean exit with an
  incomplete stream reports CODEX_EMPTY. Prior feedback is never replaced.
- The tools/python fake codex emits turn.completed on success, as
  codex exec --json does.
- README conflicts take upstream's design-inventory text; this PR no longer
  touches README. Version 1.19.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream released code v1.19.4, so this change moves to v1.19.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PNWGMR

PNWGMR commented Oct 7, 2026

Copy link
Copy Markdown
Author

Updated to c7d5862: merged upstream main (through #224, so the PR is mergeable again) and addressed the review points inline.

Notes on the merge, since upstream changed the same wrapper in the meantime:

  • Upstream's per-run codex_stderr and codex_failure_reason are kept. Each run's stderr is also appended to the retained $LOG_ID.stderr. parse_failure_message now opens the JSONL as UTF-8, because the default-encoding open() was the ISS-13320: vibe gives clickable links, never a bare story id #359 bug on Windows.
  • Upstream's token contract is kept. A failed codex run reports CODEX_FAILED:codex exited with code N: <reason>, and a clean exit with an incomplete stream (the parser now exits 3) reports CODEX_EMPTY. Prior feedback is never replaced in either case.
  • The tools/python fake codex now emits turn.completed on success, as codex exec --json does. Without it, the completeness gate correctly refused its "successful" stream.
  • README conflicts take upstream's design-inventory text, so this PR no longer touches README. Version bumped to 1.19.5 in both manifests.

Verification on Linux (Ubuntu via WSL, Python 3.14), from a clean clone of c7d5862's tree:

  • plugins/code/skills/codex-review/tests/test_run_codex_review.py: 8 passed.
  • plugins/code/tools/python/test_run_codex_review.py: all 10 wrapper tests pass. The 2 plan-review hook tests fail identically on pristine main in that environment (no jq), so that's environmental and unrelated to this change.
  • Ruff clean, bash -n clean.

Could a maintainer approve the fork CI run for the new head when convenient?

@shafty023

Copy link
Copy Markdown
Collaborator

Beginning a code-review. Please allow time for me to finish before merging the PR

@@ -0,0 +1,207 @@
"""Synthetic end-to-end checks for the plan review JSONL wrapper."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This filename duplicates plugins/code/tools/python/test_run_codex_review.py. Neither test directory is a package, and CI runs uv run pytest plugins/ without an import-mode override, so pytest imports both as test_run_codex_review and stops collection with an import-file-mismatch. Rename this module or move the cases into the existing one, then check collection with both files present; the isolated directory run does not catch this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants