和AR baseline的推理结果有diff #4

Activity

1. jianc99 commented on Jan 14, 2026

   

   jianc99

   on Jan 14, 2026

   Member

   More actions

   Thanks for reporting this.

   In principle, the accuracy should be very similar. Please also check this PR for more details:
   sgl-project/sglang#16818

   In our own tests, we did observe that on some prompts the generation results can differ slightly, even under greedy decoding. We believe this is caused by numerical randomness introduced by the attention kernel. That said, we do not observe a significant accuracy drop overall—speculative decoding is theoretically lossless, so it should not degrade performance in expectation.

   Could you please share your test environment (hardware, CUDA / PyTorch versions) and the evaluation code or scripts you used? That would help us reproduce the issue and better understand the root cause.
2. dcw02 commented on Jan 14, 2026

   

   dcw02

   on Jan 14, 2026

   More actions

   Yes outputs diverge on some prompts due to kernel differences, this is a known thing for other speculative decoding models as well. This also happens depending on concurrency/batch size for non speculative decoding!
3. mZhenz commented on Jan 15, 2026

   

   mZhenz

   on Jan 15, 2026

   Author

   More actions

   感谢回复，
   diff和transformers版本有关，对齐transformers==4.57.3后指标没有diff了，使用transformers==5.0.0会有比较大的diff。

   👍React with 👍3Reacted by Zhijian Liu, Jian Chen and Junxi Yin
4. mZhenz

   closed this as completedon Jan 15, 2026
5. mvideet

   added a commit that references this issue on May 12, 2026

   HW ideas feasibility loop: 7 studies, 1 shippable win

   ...

   c98245a