Skip to content

deepseek4: expert_used_count の override をハッシュルーティング層と両立させる #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ishikawa merged 2 commits into from
Aug 2, 2026
Merged

deepseek4: expert_used_count の override をハッシュルーティング層と両立させる #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9fc4220
deepseek4: expert_used_count の override をハッシュルーティング層と両立させる
ishikawa Aug 2, 2026
87e33dc
レビュー指摘: 集約幅と fail-loud 化を修正する
ishikawa Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions src/llama-graph.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2129,26 +2129,29 @@ ggml_tensor * llm_graph_context::build_moe_ffn(

assert(n_expert_used > 0);

// aggregate over the experts actually computed (n_expert_used may differ from
// hparams.n_expert_used, e.g. for dsv4 hash layers under an expert_used_count override);
// during warmup callers pass n_expert_used == n_expert, so cap to hparams.n_expert_used
// to avoid a large number of add nodes
// ref: https://github.com/ggml-org/llama.cpp/pull/14753
const int64_t n_expert_aggr = cparams.warmup ? std::min(n_expert_used, (int64_t) hparams.n_expert_used) : n_expert_used;

// order the views before the adds
for (uint32_t i = 0; i < hparams.n_expert_used; ++i) {
for (int64_t i = 0; i < n_expert_aggr; ++i) {
cur_experts[i] = ggml_view_2d(ctx0, experts, n_embd, n_tokens, experts->nb[2], i*experts->nb[1]);

ggml_build_forward_expand(gf, cur_experts[i]);
}

// aggregate experts
// note: here we explicitly use hparams.n_expert_used instead of n_expert_used
// to avoid potentially a large number of add nodes during warmup
// ref: https://github.com/ggml-org/llama.cpp/pull/14753
ggml_tensor * moe_out = cur_experts[0];

for (uint32_t i = 1; i < hparams.n_expert_used; ++i) {
for (int64_t i = 1; i < n_expert_aggr; ++i) {
moe_out = ggml_add(ctx0, moe_out, cur_experts[i]);

ggml_build_forward_expand(gf, moe_out);
}

if (hparams.n_expert_used == 1) {
if (n_expert_aggr == 1) {
// avoid returning a non-contiguous tensor
moe_out = ggml_cont(ctx0, moe_out);
}
Expand Down
19 changes: 17 additions & 2 deletions src/models/deepseek4.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,18 @@ void llama_model_deepseek4::load_arch_tensors(llama_model_loader &) {

layer.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0);
if ((uint32_t) i < hparams.dsv4_hash_layer_count) {
layer.ffn_gate_tid2eid = create_tensor(tn(LLM_TENSOR_FFN_GATE_TID2EID, "weight", i), {n_expert_used, n_vocab}, 0);
// the hash-routing table width is baked into the GGUF and must not follow an
// expert_used_count KV override (the routed layers can run with fewer experts,
// the hash layers cannot)
const ggml_tensor * tid2eid_meta = ml->get_tensor_meta(tn(LLM_TENSOR_FFN_GATE_TID2EID, "weight", i).str().c_str());
if (tid2eid_meta == nullptr) {
// metadata-only load paths (llama_model_init_from_user) have no weights map;
// fall back to n_expert_used, which then must not be overridden
LLAMA_LOG_WARN("%s: cannot read the width of %s, assuming expert_used_count (%lld) is not overridden\n",
__func__, tn(LLM_TENSOR_FFN_GATE_TID2EID, "weight", i).str().c_str(), (long long) n_expert_used);
}
const int64_t n_hash_used = tid2eid_meta ? tid2eid_meta->ne[0] : n_expert_used;
layer.ffn_gate_tid2eid = create_tensor(tn(LLM_TENSOR_FFN_GATE_TID2EID, "weight", i), {n_hash_used, n_vocab}, 0);
} else {

@ishikawa ishikawa Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

表の幅は override 後の n_expert_used とは無関係なので、テンソルメタの実寸でロードします。meta が取れない異常系は従来どおり n_expert_used に落として形状検証に任せます

layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, 0);
}
Expand Down Expand Up @@ -1143,9 +1154,13 @@ llama_model_deepseek4::graph::graph(const llama_model & model, const llm_graph_p
const auto & layer = model.layers[il];
ggml_tensor * selected_experts = nullptr;
ggml_tensor * exp_probs_b = layer.ffn_exp_probs_b;
int64_t n_expert_used_layer = hparams.n_expert_used;
if ((uint32_t) il < hparams.dsv4_hash_layer_count) {
selected_experts = ggml_get_rows(ctx0, layer.ffn_gate_tid2eid, res->t_inp_tokens);
exp_probs_b = nullptr;
// hash layers always use the table's expert count, even when
// expert_used_count is overridden for the routed layers
n_expert_used_layer = layer.ffn_gate_tid2eid->ne[0];
}

ggml_tensor * moe_out = build_moe_ffn(cur,
Expand All @@ -1154,7 +1169,7 @@ llama_model_deepseek4::graph::graph(const llama_model & model, const llm_graph_p
layer.ffn_gate_exps,

@ishikawa ishikawa Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_moe_ffn は selected_experts を渡した場合も n_expert_used 引数で weights を reshape するため、ハッシュ層には表の実幅を渡す必要があります

layer.ffn_down_exps,
exp_probs_b,
n_expert, hparams.n_expert_used,
n_expert, n_expert_used_layer,
LLM_FFN_SILU, hparams.expert_weights_norm,
hparams.expert_weights_scale,
(llama_expert_gating_func_type) hparams.expert_gating_func,
Expand Down
Loading