Repository navigation

[FEAT] [PERF] Remove per-candidate and metadata host syncs from speculative decoding - #51

Merged

bjf-frz merged 4 commits into

InferMatrixLab:mainfrom

IDEA-V:device-loop/batched-readback

Sep 30, 2026

Merged

[FEAT] [PERF] Remove per-candidate and metadata host syncs from speculative decoding#51
bjf-frz merged 4 commits into
InferMatrixLab:mainfrom
IDEA-V:device-loop/batched-readback

Commits on Sep 29, 2026
- perf: batch greedy draft and verification readbacks
  wyatt-w4ng
  committed
  fc3c7c6
- perf: stage prepared KV metadata in pinned memory
  wyatt-w4ng
  committed
  7f8a2c9
- perf: share depth-independent KV metadata across recurrence depths
  wyatt-w4ng
  committed
  2473666
- perf: keep speculative draft IDs on the device
  wyatt-w4ng
  committed
  e6e9fd0