Repository navigation
bug: require dflash_config.target_layer_ids instead of guessing it - #157
aqilmarwan wants to merge 1 commit into
Conversation
f35babf to
8f38fe9
Compare
build_target_layer_ids was only ever the default argument of
dflash_config.get("target_layer_ids", ...), and every released
z-lab/*-DFlash checkpoint sets that key, so it never supplied a value.
Had it fired it would have been wrong. It derived the tap count from the
draft's num_hidden_layers, but the number of target taps is an independent
hyperparameter: the two differ in 13 of the 18 released checkpoints, and
the formula only reproduces the shipped ids when given len(target_layer_ids)
rather than the draft depth. Since len(target_layer_ids) sets fc's input
width, a wrong guess surfaces as a checkpoint shape mismatch that points at
the weights instead of the missing config key.
Because dict.get evaluates its default eagerly, the dead call also ran on
every load, making num_target_layers a hard requirement of every draft
config even though nothing reads it. A config that sets target_layer_ids
but omits num_target_layers previously raised AttributeError.
Drop the helper, require the key with an error that names it, and remove
the unused num_target_layers field from the MLX DraftConfig. The MLX
loader already read target_layer_ids with no fallback; the torch path now
matches.
Refs: z-lab#156
8f38fe9 to
fc7f36b
Compare
|
I hit this same helper while porting DFlash to LMDeploy and reviewing the TensorRT-LLM and LMDeploy implementations, so I read this change closely. Verified against the evidence in #156 and my own ports: every published checkpoint sets One data point on ecosystem impact: this helper was copied into downstream engines. SGLang's LGTM. |
|
gentle ping - @jianc99 |
…arget_layer_ids (#4991) * feat: support original z-lab Qwen3 DFlash checkpoints and mixed dtypes Validated dense Qwen3 end-to-end on the DFlash stack (RTX 4070 8 GB, target thewimo/Qwen3-4B-AWQ float16, draft z-lab/Qwen3-4B-DFlash-b16 bfloat16, greedy, 256 new tokens): 33.6-36.8 tok/s vs 25.1 baseline, mean 4.19 of 15 drafts accepted per block, coherent lossless output. Two gaps had to be fixed to get there: - parse_dflash_config required dflash_config['block_size'], but the original z-lab Qwen3 checkpoints (Qwen3-4B/8B-DFlash-b16) store block_size at the top level of config.json. Fall back to the top-level attribute (same resolution order as the reference implementation), for mask_token_id as well, with a regression test. - The draft ran the shared target embedding output and the target aux states in the target dtype through bfloat16 draft weights, which fails with a dtype mismatch for float16 targets (e.g. AWQ). Casting the whole engine to float16 instead is not an option: Qwen3 hidden state outliers overflow float16 during feature fusion and acceptance collapses to zero (measured 0.06 of 15). Cast to the draft dtype at the two ingestion boundaries; logits already cast via get_logits. Signed-off-by: harshal-96 <harshal.dhandrut@gmail.com> * remove build_target_layer_ids fallback; require checkpoint target_layer_ids The evenly spaced fallback cannot be correct: the tap layers are a training-time choice baked into the checkpoint (the fc input width and which target layers the draft was trained against), so any load-time guess reads the wrong features. For 14 of the 20 published DFlash checkpoints the guess differs from the trained layers, and when the counts collide the load succeeds silently with wrong layers (z-lab/dflash#156). Every published DFlash checkpoint sets dflash_config.target_layer_ids, so a missing key now fails loudly. Matches the same removal in the reference repo (z-lab/dflash#157) and sglang (sgl-project/sglang#37476). Signed-off-by: harshal-96 <harshal.dhandrut@gmail.com> * address review: mixed-dtype tests, top-level mask_token_id test, clearer block_size error - Unit-test the two draft-dtype ingestion boundaries (shared target embeddings and projected target hidden states) with a float16 source and a bfloat16 draft; both fail on the uncast code. - Cover the top-level mask_token_id fallback, matching the block_size fallback coverage. - Mention both supported block_size locations in the query-length error. Signed-off-by: harshal-96 <harshal.dhandrut@gmail.com> --------- Signed-off-by: harshal-96 <harshal.dhandrut@gmail.com>
Closes #156.
Makes
dflash_config.target_layer_idsrequired instead of guessing it, and deletesbuild_target_layer_ids. Rebased onto 07ebd93. Full evidence in #156.Why:
fc's input width, so it surfaces as a weight shape mismatch rather than a missing config key.dict.getevaluates its default eagerly, the guess ran on every load, making the unreadnum_target_layersmandatory in every config. That's why the MLXDFlashConfigfield goes too.MLX already required the key with no fallback; torch now matches.
Verified by building the draft model from all 20 reachable configs, including both DFlash 2 ones, and checking
target_layer_ids,fc.in_features,block_sizeandmask_token_idagainst each. All load, same asmain. No released checkpoint changes behaviour.