Repository navigation
Conversation
Describe observation-to-action timing, replay denominators, precision and per-task quality evidence using the existing benchmark owner. Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
Build the native catalog and coverage from source revision d5fc212. Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author)
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision d5fc2121d6d1628111403ba70ec3b70638ab8674 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Selection/task metadata (not a relevance verdict): {"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-boundary-and-quality.md","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/embodied-cpp-source.json"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"d5fc2121d6d1628111403ba70ec3b70638ab8674","status":"ready"}Provider/settings: Uncovered / unsupported paths (87)
|
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Source-only LGTM for the bounded #91 VLA benchmark-guidance amendment at 0f4433b7c9bbc1512a4ef5526f087c2a9730d239, against main 663f3b327998526a13aa31e72ad729025c18983c. No blocking change found in this scope.
Existing review disposition: refreshed the conversation, submitted reviews and inline threads before publication. There are no existing submitted reviews or inline threads to resolve; the automated current-head review is complete with no findings. The SkillEval evidence limitation below remains explicit rather than being counted as a successful evaluation.
Checks performed:
- Inspected all 87 changed paths by their source/generated roles. All 78 unrelated generated records change only source provenance. Source commit
d5fc2121is reachable and its canonical bytes are unchanged at this head; all 12 source-file SHA-256 values in the affected projection match fetched current-head bytes, and projected sidecar fields match their source. - Matched the 17 recorded upstream paths, byte counts and Git blob identities against the complete 213-entry pinned tree. Re-read the relevant timing/pair gate, queue/replay, profiler, aggregation, server response, quality and attribution sources, plus paper v3 §4. Recomputed the public 100+100 latency arrays and memory sample summaries. The guide preserves complete-request versus generated-action timing, per-task loss, precision differences, whole-device memory and missing measured-source/raw-episode evidence; it does not attribute the later SmolVLA experiment to paper v3.
- The existing owner stays
candidate/not_run, with no hardware prerequisite or new SearchSpace mechanism. Public sample arithmetic is not a new device measurement or an independent recount of unavailable episode logs.
Current hosted evidence: all six Ready jobs and the declaration check pass. The inspected Python 3.11 log reports Ruff success, 3,544 passed / 13 skipped / 284 subtests; the isolated wheel check reads 127 source-grounded records. Tested merge 954ff5c6 has zero file differences from this head. I did not rerun repository tests or execute upstream code.
The SkillEval plan selected zero comparisons and was not executed. Its unsupported executable snapshot entry is already present with the identical mode/blob on main; this review neither relaxes that safety boundary nor treats unavailable/unknown budget or evaluation as a pass. No runtime, robot, device, quality-preservation, performance or Agent-efficacy result is established. #339's overlapping pending amendment was not combined or tested here. This COMMENT is not formal APPROVE, merge authorization, or acceptance/closure of the broader #91/#177 goals.
Dong1017
added
the
area:knowledge
ChatGPT Codex Connector
|
Review at head
Not verified by me: the upstream source pins/byte hashes in |
VLA benchmark reports can mistake latency amortized over a generated action chunk for real control cadence, and aggregate success can hide task-specific quality losses. This change adds source-bound observation-to-action, replay-denominator, precision and per-task quality guidance to the existing
inference-benchmark-designowner.Sources are Embodied.cpp at 156bed15, licensed Apache-2.0, and the companion paper v3, CC BY 4.0. The source record identifies 17 inspected files and their exact versions. The later SmolVLA experiment is not presented as a paper-v3 result.
The reference preserves the different measured-script hashes, incomplete dirty patch and absent raw episode records, as well as the reported precision and whole-device memory boundaries. Public sample arithmetic is identified separately from author-reported quality results. The registered owner remains candidate /
not_run, its hardware field remains absent, and its original SearchSpace resource exclusion remains intact.Canonical source M:
d5fc2121d6d1628111403ba70ec3b70638ab8674. Native generated G and current head:0f4433b7c9bbc1512a4ef5526f087c2a9730d239.Native index and coverage write/check modes passed from the actual M. At H, complete checkout and source-package resource reads matched the committed bytes; the generated index and coverage were verified against the actual sources. Before M, four local test modules passed with 77 passed and 5 skipped on the base plus the six source files. An independent Sol/ultra review of the complete H found no blocking source or generated-projection issue. Ready CI passed against a tested merge whose tree equals this head. Python 3.10 and 3.11 each reported 3,544 passed, 13 skipped, 1 warning and 284 subtests; both isolated installed-wheel checks read 127 source-grounded records. The current-head model-policy declaration passed. The nonpaid SkillEval plan selected zero comparisons, with 87 unsupported paths; execution was skipped and no efficacy result follows.
The source-derived candidate does not establish upstream runtime, model, robot, device, quality, performance or agent-efficacy results.
Refs #91. Refs #177.
Knowledge authorship declaration