[bong-water-water-bong]

What

An HRX build prints its device and then runs part of the graph on the CPU without saying so. The engine admits a dense MUL_MAT
only when input_size % 256 == 0 (eight gate sites in dispatch_registration/common/dispatch-mul-mat.cpp, stated by
common_is_supported_dense_input_size() in dispatch-mul-mat-common.h), so a model whose hidden size is not a multiple of 256
loses its matmuls silently — and the SSM/scan/norm side stays on HRX, which makes the result look like a normal device run.

Measured with GGML_HRX_STRICT=1 before this change:

model	n_embd	n_embd % 256	nodes off HRX
Zamba2-2.7B	2560	0	0 (strict run clean)
Zamba-7B-v1	3712	128	383 / 5113
BlackMamba-2.8B	1472	192	145 / 1179

Until now the only way to see this was GGML_HRX_STRICT=1, which aborts the run.

How

The placement observer already counted the off nodes for GGML_HRX_REPORT / GGML_HRX_STRICT. It now also counts the off matmuls
whose input_size is not a multiple of 256 and, in the default configuration, prints one line per process on the first graph
that loses a node:

placement_observer: placement: 383 of 5113 nodes on the CPU (MUL_MAT x 281, CONT x 89, UNARY x 13) - 3712 is not a multiple of 256, so 281 matmuls have no HRX dense schedule (input_size % 256 != 0)

Knobs, unchanged in meaning except the new default:

• unset: that line once per process, if a node falls back;
• GGML_HRX_REPORT=0: silent (a model that loses nodes no longer announces itself);
• GGML_HRX_REPORT set to something that is not a number: treated as unset, so a typo cannot hide the line the way atoi()
  would (=x → 0 → quiet); an explicit numeric <= 0 still silences it;
• GGML_HRX_REPORT=1: the pre-existing per-graph report instead of the fallback line (not both);
• GGML_HRX_REPORT>=2: per-graph, every graph;
• GGML_HRX_STRICT=1: unchanged, aborts on the first node off HRX.

Severity is WARN: losing nodes to the CPU is a degraded placement state of the same class as the residency guardrail, so it should
survive a front-end that keeps warnings and drops info.

Cost. The scan is bounded to the first 16 graph evaluations, which is where the prefill graph and the first decode steps are
built. Scanning every graph forever would put a per-node ggml_backend_sched_get_tensor_backend() lookup plus a
std::vector<int64_t>(GGML_OP_COUNT) in the default hot path, and for a fully claimed model (the common case) the scan never
terminates itself. GGML_HRX_STRICT=1 and GGML_HRX_REPORT>=2 remain the exhaustive probes.

No behaviour change: placement, kernels and the split are untouched; only the observer's reporting.

Verification

Build: 211 targets green (build-hrx, HRX enabled, BUILD_EXIT=0). The whole ggml/src/ggml-hrx subtree is byte-identical
between this PR's base d8bd0ff60 and the tree I built and ran (git diff --stat d8bd0ff60 <built> -- ggml/src/ggml-hrx is empty;
ggml-hrx.cpp blob d11ee71e3 on both sides), so the device evidence below is this translation unit.

Device, under gpu-hold, gfx1151, -dev HRX0 -ngl 99:

arm	expected	measured
Zamba2-2.7B (2560, aligned)	nothing	0 lines
Zamba-7B-v1 (3712, unaligned)	one line, with the reason	1 line, exactly the text above
same, GGML_HRX_REPORT=0	silent	0 lines
same, GGML_HRX_REPORT=1	old report form, not both	1 line: 5113 nodes, 383 off HRX (MUL_MAT x 281, CONT x 89, UNARY x 13), 384 graph splits
same, -p 128 -n 32 (many graphs)	still exactly one line	1 line
GGML_HRX_STRICT=1	still aborts, from inside the observer	abort at node_14 MUL_MAT f32 ne=[14848,512,1,1] … 383 of 5113 nodes off HRX … 384 splits

llama-server at default verbosity (the path a user reads), gpu-hold, one greedy /completion per arm so the graph is really built:

arm	expected	measured
Zamba-7B-v1, env unset	one W line	0.06.535.157 W placement_observer: placement: 383 of 5113 nodes on the CPU (…) - 3712 is not a multiple of 256, so 281 matmuls have no HRX dense schedule (input_size % 256 != 0)
Zamba-7B-v1, GGML_HRX_REPORT=x	still prints (a typo must not hide it)	1 line
Zamba-7B-v1, GGML_HRX_REPORT=0	silent	0 lines
Zamba2-2.7B (aligned)	silent	0 lines

Every arm's greedy continuation of The capital of France is is unchanged from the pre-change reference (" Paris.\nThe capital of the United States is Washington, D.C."
for v1, " Paris." for Zamba2), so nothing above was bought with different maths.

The 383/5113 and 281 matmuls reproduce the pre-change GGML_HRX_STRICT=1 measurement exactly.

One thing to know when testing it

llama-bench without -v discards all ggml logging: llama_log_set() forwards to ggml_log_set() (src/llama-impl.cpp:32)
and tools/llama-bench/llama-bench.cpp:2289 installs llama_null_log_callback when !params.verbose. No ggml line of any severity
reaches the terminal there, so llama-bench needs -v. llama-server and llama-cli print it as usual. I lost a verification
cycle to this, and then a second one to a readiness gate that accepted llama-server's 503 "Loading model" as "ready".

Not doing

The real fix — a tail schedule for non-aligned input_size in the dense MUL_MAT path — is the other half of engine ggml-org#991 and
needs its own kernel work; this PR is the observability half only. No tolerance changes, no placement changes.

Refs engine ggml-org#991 (observability half; the tail-schedule half stays open).

[bong-water-water-bong]

#197 ready for review at 37ea88d

Engine ggml-org#991's observability half, on its own branch off 1bit/hrx-vulkan-patched (not stacked on the model PRs). Device evidence in the body, measured on gfx1151 under gpu-hold; the tested file is byte-identical to the base (blob d11ee71e3), so the build proof covers this tree.

[bong-water-water-bong]

VERDICT: READY on the code

37ea88dcf, 1 file (+56/-2, ggml/src/ggml-hrx/ggml-hrx.cpp). A load-time line the placement report was hiding, and
the gating is the part worth checking:

• The old gate silenced it: if (!strict && (level <= 0 || (level < 2 && reported))) returned before any node scan
  whenever nobody asked for GGML_HRX_REPORT, so "this model lost nodes to the CPU" only ever appeared under
  GGML_HRX_STRICT=1. The new gate keeps the cheap exit but only once the line is said, deliberately silenced, or past
  the scan budget — kFallbackScanGraphs = 16, and the counter is only incremented on the level == 0 path, so a
  GGML_HRX_REPORT=1 run does not spend the budget.
• placement_fallback_quiet() fails loud: whole-string strtol, parsed <= 0 only. GGML_HRX_REPORT=0 is quiet,
  GGML_HRX_REPORT=x is not (unlike atoi, which would read 0 and silently hide the line). That is the right
  direction for a knob whose job is to reveal a degraded state.
• The reason is derived, not guessed: src[0]->ne[0] % 256 != 0 on MUL_MAT/MUL_MAT_ID — src[0] is the weight,
  so ne[0] is input_size, which is what the message says ("… is not a multiple of 256, so N matmuls have no HRX
  dense schedule"). With no matching cause the message still prints, just without a reason suffix.
• Once per process via fallback_reported.exchange(true), so a model that loses nodes to the CPU cannot flood a log.
• The GGML_LOG_WARN choice with its own caveat (llama-bench nulls the log callback without -v) is documented at the
  call site, which is where someone will look when the line does not appear.

Not run by me: the HRX device path (this TU only builds under -DONEBIT_HRX=ON and the fork's HRX build is an
ExternalProject). The gate logic I checked by reading the three branches against level/strict; a fork HRX build plus
one model whose hidden size is not a multiple of 256 is what would exercise it end to end.