Skip to content

Qualify MiniMax-H3 Ref2VA five-second Ascend optimization - #294

Merged
Dong1017 merged 42 commits into
mainfrom
amy/h3-ref2va-evidence
Nov 2, 2026
Merged

Dong1017 merged 42 commits into
mainfrom
amy/h3-ref2va-evidence

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #293

Five-second delivery

Deliver the MiniMax-H3 Ref2VA Ascend campaign requested in #293: 407.27 → 108.12 s HTTP E2E, 73.45% lower (3.77× inverse-latency speedup) on eight Ascend 910B1 NPUs. All 26 requests, 14 quality comparisons and 32 actual worker exits pass. Fifteen-second qualification is explicitly deferred.

This adds frozen plans, portable client/coverage/validator scripts, source attribution, raw-value sanitized receipts, an integrity manifest and repeated-trial charts. The final A/B/B/A campaign uses two independent servers per arm, two warmups plus three measured requests per server. Sample SD is across server means, not six independent trials. Mean sampled peak HBM rises 3.10%. The family audio cosine of 0.813411 is close to the 0.80 gate; this is lossy caching, not semantic-quality certification.

Final report and source/benefit table distinguishes the qualified existing Cache-DiT profile from the optional device-KV single-pair pilot (44.86% lower latency, not enabled in final profile). Every adopted optimization has an origin and evidence limit. Rejected audio ownership, failed RPC attempts and the interrupted first final campaign are retained without mixing their samples into the accepted result.

All downstream engine changes are consolidated in vLLM-Omni #9698. Measured runtime: 098b51f6b96dd160a0959e74e8822422527ff06d; baseline: c7f56f0155953c7d4a77276a2f4fcbd844ab0da2. Baseline teardown uses matched diagnostic 60/90-second budgets; candidate uses native production budgets.

Validation: evidence inventory hashes and focused inventory tests; complete target-NPU campaign; all 14 frozen-gate comparisons. Raw media, model weights and host logs stay external, with public hashes and explicit availability limits. No global optimum, p95/p99, throughput, CUDA or 15-second result is claimed.

Agent-assisted-by: Codex

Remaining work: 15-second qualification is deferred. Goal acceptance remains with the requester; this PR does not request issue closure. Knowledge authorship policy reports Not applicable for these campaign evidence paths.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@Dong1017 Dong1017 added the area:workbench CLI workbench: profile, A/B loop, quality gates, evidence reports label Oct 31, 2026 — with ChatGPT Codex Connector

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本轮为 Draft #294 的首次有范围静态评审:对照 #293 的冻结验收契约,检查 10 个新增文件、validator/compare.py / smoke.py / manifest.json,以及 main 的 tools/validate_evidence.py、tools/evidence_inventory.json 与贡献规范。下面 3 项分别涉及现有仓库校验回归、音频时长漏检和 LPIPS 权重身份;建议在把此验证器用于正式 campaign 质量判断前处理。未运行测试、模型或 NPU 实验;PR 中的合成样本结果仅按作者披露记录,不视作本次独立复验。

@@ -0,0 +1,9 @@
{

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1|新 evidence bundle 会使现有库存检查失败(静态可确定)。 此 PR 新增 evidence/h3-ref2va-ascend-293/,但没有同步登记到 tools/evidence_inventory.json。main 的 tools/validate_evidence.py::validate_inventory() 严格比较 evidence/ 顶层目录与 bundles 键,发现新增目录即报 inventory directories differ 并以非零状态退出;贡献约定也将该校验列为 nightly 检查。另需注意:登记该 manifest.json 后,未列入 inventory 会被判为未分类;若直接按 bundle-sha256 校验,清单内两项未随 PR 提交的 .pth 权重会按缺失的本地文件报错。建议登记 kind: campaign,明确 manifest 的本地可验证文件与外部权重来源/不可用边界,并用已有的只读 inventory 检查验证修正(不要求本任务执行)。依据:tools/validate_evidence.py 的 actual != set(entries)、unclassified manifest、verify_file 路径。

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inventory-registration P1 is fixed at 288102b7; this scoped confirmation does not clear the two quality-gate P2s or approve the Draft.

  • The new campaign directory is registered. Exact historical-sha256 mappings cover the client plus five local validator files, and both unvendored weight files are explicitly external. All six local SHA-256 values match their manifests, so those weights are no longer treated as missing local inventory files.
  • The latest increment changes only README/pilot data; the inspected registration and six mapped files are unchanged from 8bb7e797. Both installed-package jobs pass at the new head. Its full-check jobs are still running, and task qualification is skipped for the Draft; the older head's successful Ready run is not a new-head pass.
  • The existing author still owns the audio coverage and loaded LPIPS-weight identity threads. Declaring weights external fixes inventory classification; it does not verify the weights actually used for scoring. No further inventory change is requested by this check.
Evidence and verification limits

Assessed head: 288102b72ef0dc4af8c2687e4f0b54d02c9858f6; API base / observed main: 218565698d850e955f85d0d394a5f0e5e3619812.
Current inventory and validator manifest were checked against the existing validate_manifest local/external contract and raw file bytes. Current Ready run and current-head declaration status were reread before posting.

Static source/byte-hash and retained hosted-status inspection only. I did not run the inventory command, media smoke, LPIPS, models or devices. Campaign measurements remain author-reported with public raw-media/log availability limits; no speedup, lifecycle or whole-campaign acceptance follows. This reply does not mark any reviewer thread resolved.

).stdout
audio = np.frombuffer(pcm, dtype="<f4").reshape(-1, gates["channels"]).T.copy()
audios.append(audio)
if audios[0].shape != audios[1].shape:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2|两臂同样截短音轨时可能错误通过质量门禁。 #293 冻结的是 5s / 15s 视频及音频质量,而 probe() 只核对视频帧数、音频采样率和通道;此处仅要求两臂 decoded PCM 的 shape 相同,没有要求音轨覆盖对应的输出时长或检查 A/V 起止时间戳。因而若两个 15s 视频都只附带相同的短音频(例如 0.5s),其余指标可全部合格并得到 PASS,但完整音视频交付并未验证。建议根据期望时长检查 PCM 样本数(允许明确的编码 padding 容差),同时检查音视频 PTS/覆盖范围,并加一个“两臂都缺尾部音频”的负例。此处是基于代码路径的风险判断,未实际生成媒体复现。

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对本线程 P2 做范围校正(新 5s 证据,不重复要求修复):当前 head 017a5c95 的 results/final-5s/quality-index.json 共 14 条比较均记录每通道 audio_samples=165888;validator/gates.json 冻结采样率为 32 kHz,折算 5.184 s,与 124 帧/24 FPS 的 5.167 s 视频名义时长相符。因而这批 5s 收据没有呈现此前举例的“两臂同样只有 0.5s 音轨”场景;不应把原来的静态反例说成已在本次硬件结果中复现。

剩余边界仍是通用验证器的代码路径:compare.py 仅比较两臂 PCM shape,未核对请求期望时长和 A/V PTS;上述样本数也不能单独证明音轨内容或时间戳完整。对后续 15s/其他输出复用时,仍应按本线程所述建立预期覆盖与负例。这里仅审查提交的源码及 JSON,不代表复验外部 MP4 或 LPIPS 评分器。

import lpips
import torch

loss_fn = lpips.LPIPS(net="alex").eval()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2|冻结的 LPIPS 权重身份尚未在执行入口强制校验。 validator/manifest.json 给出 AlexNet 与 LPIPS 权重的 SHA-256,但这里 lpips.LPIPS(net="alex") 直接依赖环境中的权重获取/缓存路径,compare() 并不核验实际载入的 .pth 内容是否匹配清单。不同缓存/安装状态可能使用未认证的评分器,却仍按同一固定 LPIPS 阈值出结论。建议在评分前验证实际权重字节的固定摘要、将使用的 scorer 身份记录进质量证据,校验失败则不出 PASS;无需为此自动下载或在本任务内运行模型。

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Keep the campaign INCONCLUSIVE pending clean teardown and repeated unprofiled comparisons. Attribute the policy to upstream and link the downstream lifecycle draft.

Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Keep quality gates unchanged and distinguish pilot observations from final AB/BA acceptance.

Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Retain the slower cold request and six terminate signals without claiming a speedup. Quality gates remain unchanged for isolated configuration screening.

Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本轮仅评审 Draft #294 新增的 reference-KV residency pilot 计划(对照 Issue #293 的失败停止与证据要求)。发现一项P2 候选风险:单请求超时与共享设备预约硬上限不一致,可能使失败由外层强制清理而非请求超时判定,并挤占后续比较臂。下附具体行内意见;原有音频完整性与 LPIPS 权重身份线程不重复。本轮仅静态核对文件与已有失败收据,未运行测试或 NPU 实验,也不对任何优化效果作验收。

"profiler": false,
"cache_summary": false,
"block_cache": false,
"wave_timeout_s": 14400,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2|单请求超时长于整轮 NPU 预约,失败归因与多臂覆盖存在风险。 这里 wave_timeout_s、http_timeout_s 都设为 14,400 秒(4 小时),但同一计划第 33 行将全部 kv2 → kvdevice2 → kvdevice8 放进共享 3 小时 hard reservation,且 165 分钟就会 soft TERM。若第一臂的 warmup 像此前 KV2 一样阻塞,外层结束/杀进程将早于内部超时,后续臂可能完全没有样本,且缺少可归因的请求级 timeout 收据;每臂还有 2 次 warmup + 1 次测量。建议冻结**小于剩余预约预算(预留启动/清理空间)**的单请求/单臂截止时间,记录哪一级 timeout/TERM 先触发以及各臂未完成原因;如果有意只用预约守护程序作为故障上限,则在计划中明确它与请求超时的优先级和失败证据格式。该问题是静态计划约束冲突,未复现硬件行为,也不要求本评审任务执行实验。

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充当前 323bf306 的受影响计划:新增的 cache-range-pilot.json 也存在同一 timeout 层级问题,而且四臂共用的预约更短。

  • wave_timeout_s / http_timeout_s 仍为 14,400 秒;共享预约为 2 小时,105 分钟 soft TERM,180 秒后强制终止。每臂还有两次 warmup 和三次测量。
  • 若前臂请求阻塞,预约层会先终止,内部请求 timeout 无法作为该次失败边界,后续臂可能没有样本。请把这份计划一并纳入本线程的处置:按剩余预约时间约束请求/单臂截止时间,或明确预约 watchdog 优先及中断、未执行各臂的证据记录。

这里只补充新计划的静态证据,不重复开问题;没有运行设备,也不判断未公开的执行脚本是否已另行处理。原始失败与未完成记录应保留。

Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new pr-references failure is a PR-body citation issue; this Draft is still awaiting the existing quality-gate review items.

  • P2 — source citation triggers the closing-syntax check. The INT8/RainFusion row at PR-body line 16 places a closing keyword immediately before a bare upstream PR number. The current-head job reports that exact line and exits 1. This is an advisory text check, not a failed media or model test.
  • Minimal fix for the existing author: reword the source reference as “multi-video correction” and link explicitly to vLLM-Omni PR 9241. Preserve the source attribution while removing closing syntax. The next ordinary push will check the current body; no manual workflow dispatch is requested.
  • The audio-coverage and loaded-scorer review items remain separate. Retaining the interrupted A1 attempt and restarting the whole paired campaign does not yet establish final performance or lifecycle acceptance.
Evidence and limits

Reviewed head: 3cdcdb43b9b85e537528e847437bfbd0cf6c3a1e; API base: 218565698d850e955f85d0d394a5f0e5e3619812; observed main: 50c19da43e0b358e6f990c00719891ad308a8ac1.

Current-head failing job logs identify PR-body line 16. The checker and Ready workflow were read; this job is documented as advisory, not a required repository check. Other current-head software jobs succeeded; Draft task qualification was skipped. Static text/source and hosted-log inspection only; no PR code, media scorer, model or device was run by this task. This COMMENTED review does not approve the PR or resolve any reviewer thread.

Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@amy-why-3459 amy-why-3459 changed the title Freeze MiniMax-H3 Ref2VA Ascend quality gates and campaign evidence Qualify MiniMax-H3 Ref2VA five-second Ascend optimization Nov 1, 2026
@amy-why-3459
amy-why-3459 marked this pull request as ready for review November 1, 2026 07:13
Agent-assisted-by: Codex
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR-body reference-check finding is fixed at 017a5c95; this is a scoped repair confirmation, not campaign approval.

  • The rewritten body no longer contains the flagged source wording, and the current-head pr-references job succeeds. No further citation change is requested by this review.
  • Current full-check, installed-package, qualification and declaration/policy checks succeed. These software/admission results do not independently verify the reported media scores, scorer weights or latency measurements.
  • The remaining loaded-weight identity thread and audio-coverage clarification retain their stated scope. The new 5s receipts must not be described as reproducing the earlier short-audio counterexample; whole-campaign acceptance remains with the requester and the existing author retains follow-up ownership.
Evidence and limits

Head: 017a5c95bd77788b92bd3fff1b1648a8e214adb1; API base: 218565698d850e955f85d0d394a5f0e5e3619812; observed main: eabad4ad57c0af9407d33da83bb117201d5e5cce.

Follow-up to the earlier citation review. Current body, exact-head hosted checks/policy, changed paths, submitted final-summary/quality JSON and discussion were inspected. The validator source is unchanged in this increment; full 144-file and downstream/raw-media review remains incomplete. I did not run the validator, LPIPS, media, model, device or benchmark code. External raw media, weights and runtime logs were not independently replayed; the reported 5s result is not a new independent performance or quality acceptance. No reviewer thread is resolved by this COMMENTED review.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for merging at 017a5c95 as five-second campaign evidence. This review is scoped to the PR; acceptance of the #293 campaign stays with its requester.

Checked:

  • All nine checks pass at this head, and the PR merges cleanly. It adds 144 files totaling 0.52 MB, so the evidence-size concern in #273 (R6) does not apply.
  • The headline matches the retained values: in results/final-5s/acceptance.json, the server means of 396.30 s and 418.24 s (baseline) and 106.66 s and 109.58 s (candidate) give 407.27 → 108.12 s, 73.45% lower.
  • All 14 entries in results/final-5s/quality-index.json pass the frozen gates in validator/gates.json. The closest margins are audio spectral cosine 0.813 (gate 0.80, disclosed in the PR body) and SSIM 0.976 (gate 0.97).
  • Every commit carries a DCO sign-off and an Agent-assisted-by trailer.

Open threads; none blocks this merge:

  • LPIPS weight identity (P2): compare.py still loads LPIPS weights from the environment without checking the manifest hash, so the scores are not attested. The largest recorded LPIPS is 0.026 against the 0.20 gate, so a different weight file is unlikely to change these verdicts. Add the hash check before the validator is reused.
  • Audio coverage (P2): the 5 s receipts record 165,888 samples per channel at 32 kHz (5.18 s). The validator still compares only PCM shape, so the 15 s qualification needs the expected-duration check.
  • Pilot timeouts (P2): per that thread, the request timeouts in the two pilot plans exceed their reservations. This affects how pilot failures are attributed, not the final A/B/B/A result.

Not checked: raw media, logs and weights are external, and I did not rerun the validator, models or devices. The measurements remain author-reported.

@Dong1017
Dong1017 merged commit 919ed42 into main Nov 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:workbench CLI workbench: profile, A/B loop, quality gates, evidence reports

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants