From 0aaa75465679c7d1d5fe1842f282afa612e64eae Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 00:10:51 +0200 Subject: [PATCH 01/92] Prompt cache for --serve A request continues from the longest prefix of its prompt the engine still holds: the live sequence, checkpoints of the fixed-size sequence state (GDN recurrence and conv history, QSA indexer tail, PLE history) in host memory, or a conversation another request replaced, stashed in RAM with its K/V cells. Checkpoints are taken before a prompt's last <|im_start|>, at the prompt's end and every --cache-every cells of a long prompt. Short prompt remainders go through verify windows instead of the batched prompt path. The decode loop no longer verifies past the output limit or a drafted end-of-turn token, so the live state ends where the emitted text does. The server reports reused tokens (OpenAI prompt_tokens_details.cached_tokens, Anthropic cache_read_input_tokens) and drains the engine to DONE inside the request lock. New engine options: --prompt-cache N, --cache-ram MIB, --cache-every N, --feed-max N. --- CMakeLists.txt | 3 +- README.md | 14 +- include/strata/core/mtp.hpp | 2 + include/strata/core/prompt_cache.hpp | 97 +++++++++ include/strata/core/session.hpp | 16 ++ serve/server.py | 44 +++-- src/core/prompt_cache.cpp | 285 +++++++++++++++++++++++++++ src/core/session.cpp | 63 ++++++ src/program/generate.cpp | 161 ++++++++++++--- 9 files changed, 634 insertions(+), 51 deletions(-) create mode 100644 include/strata/core/prompt_cache.hpp create mode 100644 src/core/prompt_cache.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 0b174608022..1d569254128 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -236,7 +236,8 @@ if(STRATA_ENABLE_CUDA) # library is the honest shape of the dependency graph: core knows about devices and weights, kernels know # about arithmetic, and this one knows the ORDER the arithmetic happens in. add_library(strata_engine STATIC src/core/layer.cpp src/core/session.cpp src/core/expert_source.cpp - src/core/expert_cache.cpp src/core/native_head.cpp src/core/native_dense.cpp src/core/verify.cpp src/core/mtp.cpp) + src/core/expert_cache.cpp src/core/native_head.cpp src/core/native_dense.cpp src/core/verify.cpp src/core/mtp.cpp + src/core/prompt_cache.cpp) target_include_directories(strata_engine PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include) target_link_libraries(strata_engine PUBLIC strata_core strata_kernels strata_kernels_cpu CUDA::cudart) diff --git a/README.md b/README.md index 03db4da8473..381fb306c39 100644 --- a/README.md +++ b/README.md @@ -178,15 +178,23 @@ print(r.choices[0].message.content) think briefly, on hard ones `high` thinks longest and is most accurate. - **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key. - **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut. +- **Prompt cache.** A request continues from the longest start of its prompt the engine still holds, so the next turn + of a chat or an agent processes only what is new (a 16K-token conversation: ~0.3 s instead of ~25 s before the first + token). It keeps checkpoints of the sequence state in RAM (at every turn's start and end, and every 16K tokens of a + long prompt: regenerated and edited answers resume from them) and moves a conversation that another request replaces + (a chat app's title request, a subagent) to RAM with its key/value cache, so switching back continues where it was. + Responses report the reused tokens (`usage.prompt_tokens_details.cached_tokens`; Anthropic: + `cache_read_input_tokens`). Engine options: `--prompt-cache N` checkpoints (default 16, ~110 MB each; 0 = off), + `--cache-ram MIB` for replaced conversations (default 16384), `--cache-every N` (default 16384). A changed system + prompt or reasoning effort changes the prompt from its first tokens, so nothing is reused. - **From other devices / the internet.** The server listens on your PC only (`127.0.0.1`). To reach it from elsewhere, put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/): `cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC: add `"api_key": "some-long-secret"` to `strata-.json` (or set the `STRATA_API_KEY` environment variable); clients then send it as their API key. -**Current limits (v1):** one request at a time; greedy decoding (temperature is ignored); every request processes its -whole prompt again (no conversation cache yet, so long chats have a long time-to-first-token); images only when set up -with them (below); no video. +**Current limits (v1):** one request at a time; greedy decoding (temperature is ignored); requests with images are +not cached; images only when set up with them (below); no video. --- diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index ce2ad113bf3..400d79d3f52 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -46,6 +46,8 @@ class MtpDrafter { /// The prompt's length: prefill() skips the cells the attention window can never reach again. void set_prompt_len(int64_t n) { prompt_len_ = n; } uint64_t vram_bytes() const { return vram_; } + /// The layer's own K/V cells (the prompt cache stashes them with the model's). + QsaState& kv_state() { return st_; } /// The main model's embedding and head, and the verify window's final residuals (T rows, hc*n_embd each). bool bind(const WeightTable& wt, const NativeHead* head, const float* window_R, std::string& err); diff --git a/include/strata/core/prompt_cache.hpp b/include/strata/core/prompt_cache.hpp new file mode 100644 index 00000000000..7d9ed19b455 --- /dev/null +++ b/include/strata/core/prompt_cache.hpp @@ -0,0 +1,97 @@ +// include/strata/core/prompt_cache.hpp - the --serve prompt cache. +// +// A request continues from the longest prefix of its prompt that the engine still holds, instead of processing +// the whole conversation again. Three things hold prefixes: +// +// * the LIVE sequence: the tokens whose cells the session state holds right now; +// * CHECKPOINTS of the live sequence (session_ckpt_*): the fixed-size state at one position. The K/V cells +// are indexed by position, so restoring a checkpoint at `pos` and overwriting the cells from `pos` on +// continues the sequence from there; +// * STASHED sequences: a sequence the live one replaced, kept in host memory with its K/V cells and its +// checkpoints, so a client that alternates between conversations (a chat and its title request, an agent +// and its subagents) does not start over on every switch. +// +// Checkpoint buffers are a shared pool of `max_ckpts`; the least recently used one goes when it is full. The +// stash's K/V copies share a byte budget in the same way. +#pragma once + +#include "strata/core/session.hpp" + +#include +#include +#include +#include + +namespace strata::core { + +class PromptCache { +public: + /// `kv`: every state whose cells a stashed sequence keeps (the QSA layers and the draft layer). + PromptCache(const ModelGeometry& g, SessionState& ss, std::vector kv, void* stream, int max_ckpts, + uint64_t stash_bytes); + ~PromptCache(); + PromptCache(const PromptCache&) = delete; + PromptCache& operator=(const PromptCache&) = delete; + + bool enabled() const { return max_ckpts_ > 0; } + uint64_t ckpt_bytes() const { return ckpt_bytes_; } + + /// Sets the session up for a prompt of `ids`: continues from the longest prefix (at most ids.size() - 1 + /// cells, the last token always goes through a verify window) that the live sequence, a checkpoint or a + /// stashed sequence holds, or zeroes the session. With `cacheable` false (an image request, whose cells the + /// ids do not describe) it zeroes the session and the live sequence is not kept. Returns the cells reused. + int64_t begin(const std::vector& ids, bool cacheable); + /// The cells [pos, pos + n) now hold `tokens`; pos <= live length, and anything after pos is gone. + void set(int64_t pos, const int64_t* tokens, int64_t n); + void set(int64_t pos, const int32_t* tokens, int64_t n); + /// Checkpoints the session, which is at `pos` == the live length. + void checkpoint(int64_t pos); + /// Forgets the live sequence (after an image request). + void forget(); + + /// One line for the log: checkpoints and stashed sequences. + std::string summary() const; + +private: + struct Ckpt { int64_t pos; void* host; uint64_t stamp; }; + struct Seq { + std::vector tokens; + std::vector ckpts; + std::unique_ptr kv; // cells [0, kv_cells) of every K/V state (stashed sequences only) + int64_t kv_cells = 0; + uint64_t kv_bytes = 0; + uint64_t stamp = 0; + }; + + /// One device region of the K/V cells, stored back to back in a stashed sequence's buffer. + struct Part { void* dev; uint64_t bytes; }; + + /// Cells of `seq` a prompt can continue from: the whole sequence when it is a prefix of the prompt (live + /// only: the state is there), else its best checkpoint within the common prefix (its position; -1: none). + int64_t reusable(const Seq& seq, const std::vector& ids, bool live) const; + static Ckpt* find(Seq& seq, int64_t pos); + void* take_buffer(); + void release(Seq& seq, int64_t keep_upto); // frees the checkpoints after `keep_upto` + /// Moves a copy of the live sequence to the stash: its K/V cells, a checkpoint of its end, and its + /// checkpoints after `keep_upto` (the ones before stay with the live sequence, which continues from there). + void stash_live(int64_t keep_upto); + std::vector kv_parts(int64_t cells) const; + bool kv_copy(const std::vector& parts, uint8_t* host, bool to_host); + bool restore(Seq& seq, int64_t pos); // K/V cells (stashed only) and the checkpoint at `pos` + + const ModelGeometry& g_; + SessionState& ss_; + std::vector kv_; + void* stream_; + int max_ckpts_; + uint64_t stash_budget_; + uint64_t ckpt_bytes_; + int allocated_ = 0; + uint64_t clock_ = 0; + Seq live_; + std::vector stash_; + std::vector free_; + std::vector> owned_; // every buffer, and whether it is pinned +}; + +} // namespace strata::core diff --git a/include/strata/core/session.hpp b/include/strata/core/session.hpp index e9c39ced286..3b32942fbd8 100644 --- a/include/strata/core/session.hpp +++ b/include/strata/core/session.hpp @@ -83,6 +83,22 @@ uint64_t session_init(const ModelGeometry& g, int64_t max_cells, int64_t k, void /// from the reference's own `zeros()`. void session_zero(SessionState& s, const ModelGeometry& g, const float* R_init, void* stream); +// ================================ prompt-cache checkpoints ================================ +// +// The K/V caches are indexed by position, so the cells before a position stay valid when a later sequence +// overwrites the ones after it. What a position does NOT recover is the state that folds the whole prefix +// into a fixed size: every GDN layer's recurrence and conv history, each QSA indexer's raw block tail (plus +// its spare key, the pooled row of the open block and the block position), and the PLE's conv history and +// token window. A checkpoint is exactly that state at one position; restoring it and overwriting the cells +// from that position on continues the sequence as if it had been processed up to there. + +/// Bytes of one checkpoint (host memory). +uint64_t session_ckpt_bytes(const ModelGeometry& g); +/// Copies the sequence state at `pos` (positions [0, pos) processed) into `host`; synchronous. +bool session_ckpt_save(const SessionState& s, const ModelGeometry& g, int64_t pos, void* host, void* stream); +/// The inverse: the session continues at `pos`. The K/V cells [0, pos) must still hold the same prefix. +bool session_ckpt_load(SessionState& s, const ModelGeometry& g, int64_t pos, const void* host, void* stream); + /// One token: layers 0..47 in order, each a `block_layer`, and the residual is updated in place. /// /// `parts` is (k, n_embd) DEVICE memory, filled by the caller - by the CPU expert pool and the VRAM-resident diff --git a/serve/server.py b/serve/server.py index 91889ee530e..82d2f8a0090 100644 --- a/serve/server.py +++ b/serve/server.py @@ -85,8 +85,14 @@ def __init__(self, exe: str, args: list[str], cwd: str | None = None, log: str | if self.max_context <= 0: raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "")) + def _done(self, line): + f = line.split() + self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]), + "decode_ms": float(f[4]), "finish": f[5], "cached": int(f[6]) if len(f) > 6 else 0} + def generate(self, ids, max_new, sampling, cancel, embeddings=None): head = f"GENI {int(max_new)} {embeddings}" if embeddings else f"GEN {int(max_new)}" + self.last = {} self.proc.stdin.write(f"{head} {','.join(str(int(t)) for t in ids)}\n") self.proc.stdin.flush() done = False @@ -96,9 +102,7 @@ def generate(self, ids, max_new, sampling, cancel, embeddings=None): if not cancel.is_set(): yield int(line[2:]) elif line.startswith("DONE"): - f = line.split() - self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]), - "decode_ms": float(f[4]), "finish": f[5]} + self._done(line) done = True return elif line.startswith("ERR"): @@ -108,7 +112,10 @@ def generate(self, ids, max_new, sampling, cancel, embeddings=None): finally: if not done: # the consumer stopped early: drain to DONE for line in self.proc.stdout: - if line.startswith("DONE") or line.startswith("ERR"): + if line.startswith("DONE"): + self._done(line) + break + if line.startswith("ERR"): break def close(self): @@ -287,19 +294,23 @@ def run(self, ids, thinking, tools, max_new, sampling, cancel) -> Iterator[tuple with self.fifo: gen = self.engine.generate(ids, max_new, sampling, cancel, embeddings=emb) if emb else \ self.engine.generate(ids, max_new, sampling, cancel) - for t in gen: - n += 1 - if t in self.stop_ids: - finish = "stop" - break - for ev in parser.feed(detok.push(t)): - yield "event", ev + try: + for t in gen: + n += 1 + if t in self.stop_ids: + finish = "stop" + break + for ev in parser.feed(detok.push(t)): + yield "event", ev + finally: + gen.close() # drains the engine to DONE before the next request finally: if emb: Path(emb).unlink(missing_ok=True) for ev in parser.finish(): yield "event", ev - yield "done", {"finish": finish, "completion_tokens": n} + yield "done", {"finish": finish, "completion_tokens": n, + "cached_tokens": getattr(self.engine, "last", {}).get("cached", 0)} # ------------------------------------------------------------------------------------------------ OpenAI @@ -328,7 +339,8 @@ def chunk(delta, finish=None): finish = "tool_calls" if calls and x["finish"] == "stop" else x["finish"] last = chunk({}, finish) last["usage"] = {"prompt_tokens": len(ids), "completion_tokens": x["completion_tokens"], - "total_tokens": len(ids) + x["completion_tokens"]} + "total_tokens": len(ids) + x["completion_tokens"], + "prompt_tokens_details": {"cached_tokens": x["cached_tokens"]}} yield last @@ -393,7 +405,9 @@ def close(): stop = "tool_use" if used_tool and x["finish"] == "stop" else \ {"stop": "end_turn", "length": "max_tokens"}[x["finish"]] yield "message_delta", {"type": "message_delta", "delta": {"stop_reason": stop, "stop_sequence": None}, - "usage": {"output_tokens": x["completion_tokens"]}} + "usage": {"input_tokens": len(ids) - x["cached_tokens"], + "cache_read_input_tokens": x["cached_tokens"], + "output_tokens": x["completion_tokens"]}} yield "message_stop", {"type": "message_stop"} @@ -414,7 +428,7 @@ def anthropic_collect(events) -> dict: b["input"] = json.loads(d["partial_json"]) elif name == "message_delta": msg["stop_reason"] = e["delta"]["stop_reason"] - msg["usage"]["output_tokens"] = e["usage"]["output_tokens"] + msg["usage"].update(e["usage"]) msg["content"] = blocks return msg diff --git a/src/core/prompt_cache.cpp b/src/core/prompt_cache.cpp new file mode 100644 index 00000000000..01af3bda2d3 --- /dev/null +++ b/src/core/prompt_cache.cpp @@ -0,0 +1,285 @@ +// src/core/prompt_cache.cpp - the --serve prompt cache (see include/strata/core/prompt_cache.hpp). +#include "strata/core/prompt_cache.hpp" + +#include "strata/kernels/kv_q8.hpp" +#include "strata/kernels/qsa.hpp" + +#include + +#include +#include +#include +#include + +namespace strata::core { + +namespace { + +/// A replaced sequence is stashed when the switch would drop at least this many of its cells. +constexpr int64_t kStashMin = 1024; + +} // namespace + +PromptCache::PromptCache(const ModelGeometry& g, SessionState& ss, std::vector kv, void* stream, + int max_ckpts, uint64_t stash_bytes) + : g_(g), ss_(ss), kv_(std::move(kv)), stream_(stream), max_ckpts_(max_ckpts), stash_budget_(stash_bytes), + ckpt_bytes_(session_ckpt_bytes(g)) {} + +PromptCache::~PromptCache() { + for (const auto& [p, pinned] : owned_) { + if (pinned) cudaFreeHost(p); + else std::free(p); + } +} + +int64_t PromptCache::reusable(const Seq& seq, const std::vector& ids, bool live) const { + const int64_t m = std::min((int64_t) seq.tokens.size(), (int64_t) ids.size() - 1); + int64_t lcp = 0; + while (lcp < m && seq.tokens[(size_t) lcp] == ids[(size_t) lcp]) ++lcp; + if (live && lcp > 0 && lcp == (int64_t) seq.tokens.size()) return lcp; + int64_t best = -1; + for (const Ckpt& c : seq.ckpts) + if (c.pos <= lcp && c.pos > best) best = c.pos; + return best; +} + +PromptCache::Ckpt* PromptCache::find(Seq& seq, int64_t pos) { + for (Ckpt& c : seq.ckpts) + if (c.pos == pos) return &c; + return nullptr; +} + +void* PromptCache::take_buffer() { + if (!free_.empty()) { + void* h = free_.back(); + free_.pop_back(); + return h; + } + if (allocated_ < max_ckpts_) { + void* h = nullptr; + const bool pinned = cudaMallocHost(&h, (size_t) ckpt_bytes_) == cudaSuccess; + if (!pinned) { + (void) cudaGetLastError(); + h = std::malloc((size_t) ckpt_bytes_); // pageable: slower copies, same result + } + if (h != nullptr) { + owned_.emplace_back(h, pinned); + ++allocated_; + } + return h; + } + // the pool is full: the least recently used checkpoint of any sequence goes + Seq* owner = nullptr; + size_t at = 0; + uint64_t oldest = UINT64_MAX; + auto scan = [&](Seq& s) { + for (size_t i = 0; i < s.ckpts.size(); ++i) + if (s.ckpts[i].stamp < oldest) { + oldest = s.ckpts[i].stamp; + owner = &s; + at = i; + } + }; + scan(live_); + for (Seq& s : stash_) scan(s); + if (owner == nullptr) return nullptr; + void* h = owner->ckpts[at].host; + owner->ckpts.erase(owner->ckpts.begin() + (ptrdiff_t) at); + if (owner != &live_ && owner->ckpts.empty()) // a stashed sequence without a checkpoint cannot be resumed + stash_.erase(stash_.begin() + (owner - stash_.data())); + return h; +} + +void PromptCache::release(Seq& seq, int64_t keep_upto) { + for (size_t i = 0; i < seq.ckpts.size();) { + if (seq.ckpts[i].pos > keep_upto) { + free_.push_back(seq.ckpts[i].host); + seq.ckpts.erase(seq.ckpts.begin() + (ptrdiff_t) i); + } else { + ++i; + } + } +} + +std::vector PromptCache::kv_parts(int64_t cells) const { + // [page][kv_head][page_size][head_dim] with an identity page table: the first cells are one prefix of each + // pool. The indexer's pooled rows are one per idx_block cells. + const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); + const uint64_t hd = (uint64_t) g_.head_dim, row = (uint64_t) g_.idx_key_dim * sizeof(float); + std::vector parts; + for (QsaState* st : kv_) { + const uint64_t pages = (uint64_t) std::min(st->n_pages, (cells + sh.page_size - 1) / sh.page_size); + const uint64_t c = pages * (uint64_t) g_.n_head_kv * (uint64_t) sh.page_size; + if (st->kv_int8) { + const uint64_t scales = c * (hd / strata::kernels::KV_Q8_GROUP) * sizeof(uint16_t); + parts.push_back({st->k_q, c * hd}); + parts.push_back({st->v_q, c * hd}); + parts.push_back({st->k_scale, scales}); + parts.push_back({st->v_scale, scales}); + } else { + parts.push_back({st->k_pool, c * hd * sizeof(uint16_t)}); + parts.push_back({st->v_pool, c * hd * sizeof(uint16_t)}); + } + const int64_t rows = std::min(cells / sh.idx_block + 2, st->max_cells / sh.idx_block + 2); + parts.push_back({st->idx_pooled, (uint64_t) rows * row}); + } + return parts; +} + +bool PromptCache::kv_copy(const std::vector& parts, uint8_t* host, bool to_host) { + cudaStream_t cs = (cudaStream_t) stream_; + for (const Part& p : parts) { + const cudaError_t e = to_host ? cudaMemcpyAsync(host, p.dev, (size_t) p.bytes, cudaMemcpyDeviceToHost, cs) + : cudaMemcpyAsync(p.dev, host, (size_t) p.bytes, cudaMemcpyHostToDevice, cs); + if (e != cudaSuccess) return false; + host += p.bytes; + } + return cudaStreamSynchronize(cs) == cudaSuccess; +} + +void PromptCache::stash_live(int64_t keep_upto) { + const int64_t L = (int64_t) live_.tokens.size(); + if (stash_budget_ == 0 || L < kStashMin) return; + const std::vector parts = kv_parts(L); + uint64_t bytes = 0; + for (const Part& p : parts) bytes += p.bytes; + if (bytes > stash_budget_) return; + uint64_t used = 0; + for (const Seq& s : stash_) used += s.kv_bytes; + while (!stash_.empty() && used + bytes > stash_budget_) { // the least recently used sequences go + auto lru = std::min_element(stash_.begin(), stash_.end(), + [](const Seq& a, const Seq& b) { return a.stamp < b.stamp; }); + used -= lru->kv_bytes; + release(*lru, 0); + stash_.erase(lru); + } + Seq s; + s.kv.reset(new (std::nothrow) uint8_t[bytes]); + if (!s.kv || !kv_copy(parts, s.kv.get(), true)) return; + checkpoint(L); // the end state: the session is still at L + for (size_t i = 0; i < live_.ckpts.size();) { + if (live_.ckpts[i].pos > keep_upto) { + s.ckpts.push_back(live_.ckpts[i]); + live_.ckpts.erase(live_.ckpts.begin() + (ptrdiff_t) i); + } else { + ++i; + } + } + if (s.ckpts.empty()) return; + s.tokens = live_.tokens; + s.kv_cells = L; + s.kv_bytes = bytes; + s.stamp = ++clock_; + stash_.push_back(std::move(s)); +} + +bool PromptCache::restore(Seq& seq, int64_t pos) { + Ckpt* ck = find(seq, pos); + if (ck == nullptr) return false; + if (seq.kv && !kv_copy(kv_parts(seq.kv_cells), seq.kv.get(), false)) return false; + ck->stamp = ++clock_; + return session_ckpt_load(ss_, g_, pos, ck->host, stream_); +} + +int64_t PromptCache::begin(const std::vector& ids, bool cacheable) { + const int64_t L = (int64_t) live_.tokens.size(); + auto zero = [&]() -> int64_t { + release(live_, 0); + live_.tokens.clear(); + session_zero(ss_, g_, nullptr, stream_); + cudaStreamSynchronize((cudaStream_t) stream_); + return 0; + }; + if (!enabled()) return zero(); + if (!cacheable) { + stash_live(0); + return zero(); + } + int64_t best = reusable(live_, ids, true); + int from = -1; // -1: the live sequence, else a stashed one + for (size_t i = 0; i < stash_.size(); ++i) { + const int64_t r = reusable(stash_[i], ids, false); + if (r > best) { + best = r; + from = (int) i; + } + } + if (best <= 0) { + stash_live(0); + return zero(); + } + if (from < 0) { + // The live sequence continues. When the prompt drops most of it (another conversation that shares a + // prefix), the rest goes to the stash; an edit near the end just overwrites it. + const int64_t dropped = L - best; + if (dropped >= kStashMin && dropped > best) { + if (Ckpt* c = find(live_, best)) c->stamp = ++clock_; // keeps it through the stash's evictions + stash_live(best); + } + if (best < L && !restore(live_, best)) return zero(); + release(live_, best); + live_.tokens.resize((size_t) best); + return best; + } + // Another sequence continues: the live one is stashed and the chosen one becomes live. + Seq chosen = std::move(stash_[(size_t) from]); + stash_.erase(stash_.begin() + from); + stash_live(0); + release(live_, 0); + live_.tokens.clear(); + if (!restore(chosen, best)) { + release(chosen, 0); + return zero(); + } + live_.tokens.assign(chosen.tokens.begin(), chosen.tokens.begin() + best); + live_.ckpts = std::move(chosen.ckpts); + release(live_, best); + return best; +} + +void PromptCache::set(int64_t pos, const int64_t* tokens, int64_t n) { + if (!enabled()) return; + release(live_, pos); + live_.tokens.resize((size_t) pos); + live_.tokens.insert(live_.tokens.end(), tokens, tokens + n); +} + +void PromptCache::set(int64_t pos, const int32_t* tokens, int64_t n) { + if (!enabled()) return; + release(live_, pos); + live_.tokens.resize((size_t) pos); + for (int64_t i = 0; i < n; ++i) live_.tokens.push_back(tokens[i]); +} + +void PromptCache::checkpoint(int64_t pos) { + if (!enabled() || pos <= 0 || pos != (int64_t) live_.tokens.size() || find(live_, pos) != nullptr) return; + void* h = take_buffer(); + if (h == nullptr) return; + cudaDeviceSynchronize(); + if (!session_ckpt_save(ss_, g_, pos, h, stream_)) { + free_.push_back(h); + return; + } + live_.ckpts.push_back({pos, h, ++clock_}); +} + +void PromptCache::forget() { + release(live_, 0); + live_.tokens.clear(); +} + +std::string PromptCache::summary() const { + size_t n = live_.ckpts.size(); + uint64_t kv = 0; + for (const Seq& s : stash_) { + n += s.ckpts.size(); + kv += s.kv_bytes; + } + char buf[192]; + std::snprintf(buf, sizeof buf, "live %lld cells; %zu checkpoints (%.0f MiB); %zu stashed sequences (%.0f MiB)", + (long long) live_.tokens.size(), n, (double) (n * ckpt_bytes_) / 1048576.0, stash_.size(), + (double) kv / 1048576.0); + return buf; +} + +} // namespace strata::core diff --git a/src/core/session.cpp b/src/core/session.cpp index cf0bbde2fe6..43fc925770a 100644 --- a/src/core/session.cpp +++ b/src/core/session.cpp @@ -129,6 +129,69 @@ void session_zero(SessionState& s, const ModelGeometry& g, const float* R_init, s.ple_token = -1; } +namespace { + +/// One region of a checkpoint: `bytes` of device memory at `dev`, stored back to back in the host buffer. +struct CkptPart { void* dev; uint64_t bytes; }; + +/// The parts of a checkpoint at `pos`, in storage order (the PLE token window follows them on the host). +std::vector ckpt_parts(const SessionState& s, const ModelGeometry& g, int64_t pos) { + strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); + sh.idx_dim = g.idx_key_dim; + const uint64_t row = (uint64_t) sh.idx_dim * sizeof(float); + std::vector parts; + parts.push_back({s.gdn_state, (uint64_t) g.n_gdn_layers() * gdn_state_floats(g) * sizeof(float)}); + parts.push_back({s.ple_hist, ple_hist_bytes()}); + for (int64_t i = 0; i < g.n_qsa_layers(); ++i) { + const QsaState& st = s.qsa_states[i]; + parts.push_back({st.idx_tail, (uint64_t) (sh.idx_block - 1) * row}); + parts.push_back({st.idx_dead, row}); + parts.push_back({(uint8_t*) st.idx_pooled + (uint64_t) (pos / sh.idx_block) * row, row}); + parts.push_back({st.idx_block_pos, sizeof(int32_t)}); + } + return parts; +} + +constexpr uint64_t kCkptHostInts = 3; // ple_prev[0], ple_prev[1], ple_token + +} // namespace + +uint64_t session_ckpt_bytes(const ModelGeometry& g) { + SessionState probe; // only the sizes are read: null device pointers are never dereferenced here + std::vector qsa((size_t) g.n_qsa_layers()); + probe.qsa_states = qsa.data(); + uint64_t n = kCkptHostInts * sizeof(int32_t); + for (const CkptPart& p : ckpt_parts(probe, g, 0)) n += p.bytes; + return n; +} + +bool session_ckpt_save(const SessionState& s, const ModelGeometry& g, int64_t pos, void* host, void* stream) { + cudaStream_t cs = (cudaStream_t) stream; + uint8_t* h = (uint8_t*) host; + for (const CkptPart& p : ckpt_parts(s, g, pos)) { + if (cudaMemcpyAsync(h, p.dev, (size_t) p.bytes, cudaMemcpyDeviceToHost, cs) != cudaSuccess) return false; + h += p.bytes; + } + const int32_t ints[kCkptHostInts] = {s.ple_prev[0], s.ple_prev[1], s.ple_token}; + std::memcpy(h, ints, sizeof ints); + return cudaStreamSynchronize(cs) == cudaSuccess; +} + +bool session_ckpt_load(SessionState& s, const ModelGeometry& g, int64_t pos, const void* host, void* stream) { + cudaStream_t cs = (cudaStream_t) stream; + const uint8_t* h = (const uint8_t*) host; + for (const CkptPart& p : ckpt_parts(s, g, pos)) { + if (cudaMemcpyAsync(p.dev, h, (size_t) p.bytes, cudaMemcpyHostToDevice, cs) != cudaSuccess) return false; + h += p.bytes; + } + int32_t ints[kCkptHostInts]; + std::memcpy(ints, h, sizeof ints); + s.ple_prev[0] = ints[0]; + s.ple_prev[1] = ints[1]; + s.ple_token = ints[2]; + return cudaStreamSynchronize(cs) == cudaSuccess; +} + /// Sets `s.gdn.state`/`conv_state` for `layer`, which is what makes one layer's GDN state its own. Shared by /// the direct and captured paths so the two cannot disagree about which slice a layer owns. void gdn_point_at(const ModelGeometry& g, int64_t layer, SessionState& s) { diff --git a/src/program/generate.cpp b/src/program/generate.cpp index e116abe52ef..d3b07b581ce 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -38,6 +38,7 @@ #include "strata/core/native_head.hpp" #include "strata/core/verify.hpp" #include "strata/core/mtp.hpp" +#include "strata/core/prompt_cache.hpp" #include "strata/prefill/prefill.hpp" #include "strata/core/native_dense.hpp" #include "strata/program/logits_selection.hpp" @@ -230,6 +231,16 @@ struct Options { bool serve = false; /// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests. bool vision = false; + /// --serve prompt cache: checkpoints of the sequence state kept in host memory (0 = every request starts + /// from an empty sequence). A request continues from the longest prefix the session or a checkpoint holds. + int prompt_cache = 16; + /// Host memory for sequences the live one replaced (their K/V cells), in MiB. + int64_t cache_ram_mib = 16384; + /// Checkpoint interval inside long prompts (positions that are multiples of it). + int64_t cache_every = 16384; + /// Prompt remainders of up to this many tokens go through verify windows instead of the batched prompt path, + /// which lends and refills expert-cache slots on every call. + int64_t feed_max = 256; int adapt_swaps = 96; }; @@ -288,6 +299,14 @@ void usage() { " --shared-late A/B: shared expert after the CPU pool (default: overlapped with it)\n" " --keep-canonical A/B: also load canonical copies of natively served tensors (more VRAM)\n" " --vision --serve takes images too (GENI requests; embeddings from strata-vision)\n" + " --prompt-cache N --serve: keep N checkpoints of the sequence state in host memory and\n" + " continue each request from the longest cached prefix (default 16;\n" + " 0 = every request starts from an empty sequence)\n" + " --cache-every N checkpoint interval inside long prompts (default 16384)\n" + " --cache-ram MIB --serve: host memory for the K/V cells of sequences a request replaced,\n" + " so switching back to them continues where they were (default 16384)\n" + " --feed-max N --serve: prompt remainders up to N tokens run through verify windows\n" + " instead of the batched prompt path (default 256)\n" " --no-token-graph A/B: two graphs per layer (the host launches each) instead of one per token\n" " --no-fused-gr A/B: the six-kernel hyper-connection read and a separate write (native)\n" " --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n" @@ -527,6 +546,10 @@ int main(int argc, char** argv) { else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode"); else if (a == "--serve") o.serve = true; else if (a == "--vision") o.vision = true; + else if (a == "--prompt-cache") o.prompt_cache = std::max(0, std::atoi(next("--prompt-cache"))); + else if (a == "--cache-every") o.cache_every = std::max(1, std::atoll(next("--cache-every"))); + else if (a == "--cache-ram") o.cache_ram_mib = std::max(0, std::atoll(next("--cache-ram"))); + else if (a == "--feed-max") o.feed_max = std::max(0, std::atoll(next("--feed-max"))); else if (a == "--no-spec-split") o.spec_split = false; else if (a == "--eos-ids") { std::string e; @@ -1852,11 +1875,21 @@ int main(int argc, char** argv) { for (float& v : drive.d.usage) v *= 0.7f; return true; }; + // ---- the prompt cache (strata/core/prompt_cache.hpp) + std::vector kv_states; + for (int64_t i = 0; i < g.n_qsa_layers(); ++i) kv_states.push_back(&ss.qsa_states[i]); + kv_states.push_back(&mtp.kv_state()); + strata::core::PromptCache pcache(g, ss, kv_states, main_cs, o.prompt_cache, (uint64_t) o.cache_ram_mib << 20); + if (pcache.enabled()) + std::fprintf(stderr, "strata serve: prompt cache on: up to %d checkpoints of %.1f MiB, %lld MiB for stashed " + "sequences\n", o.prompt_cache, (double) pcache.ckpt_bytes() / 1048576.0, + (long long) o.cache_ram_mib); std::printf("READY %lld\n", (long long) o.max_context); std::fflush(stdout); std::string line; int64_t rounds = 0; const int S = o.spec; + constexpr int64_t kImStart = 248045; // <|im_start|>: a new turn begins here // The vision path (--vision): GENI carries images. The file is one // or more strata-vision records (int32 'SVE1', n, nx, ny, n_embd, then n x n_embd floats) in prompt order; // each image's rows go to its run of <|image_pad|> tokens, whose M-RoPE positions are mtmd's: t = p, @@ -1971,41 +2004,92 @@ int main(int argc, char** argv) { if (bad) { std::printf("ERR a token id is outside the vocabulary\n"); continue; } cur = ids; const Clock::time_point r0 = Clock::now(); - strata::core::session_zero(ss, g, nullptr, main_cs); - cudaStreamSynchronize(main_stream); + auto report = [&](const std::string& e) { + std::fprintf(stderr, "strata serve: %s\n", e.c_str()); + std::printf("ERR %s\n", e.c_str()); + }; + // ---- the prompt cache: continue from the longest prefix it holds (or start from an empty sequence). + // Image requests are not cached: their cells are not identified by the token ids alone. + const bool track = pcache.enabled() && !geni; + const int64_t reuse = pcache.begin(ids, !geni); mtp.set_prompt_len(n); - std::vector> lent_now; - apply_pending(true); - if (lend_first >= 0) { - for (size_t i = 0; i < host_res.size(); ++i) - if (host_res[i] >= lend_first) { - lent_now.emplace_back((int32_t) i, host_res[i]); - host_res[i] = strata::core::kNotResident; + std::vector drafts((size_t) S, 0), window((size_t) S), outv((size_t) S); + std::vector dprob((size_t) S, 0.0f); + // ---- the rest of the prompt, [reuse, n - 1). A turn usually shares everything before its prompt's + // last <|im_start|> with the next one (chat templates re-render the answer, e.g. without its + // reasoning), so that position gets a checkpoint, and so do the prompt's end (a regenerated answer) + // and every `cache_every` cells of a long prompt. A short remainder goes through verify windows. + int64_t q = 0; + if (track) + for (int64_t i = n - 2; i > reuse; --i) + if (ids[(size_t) i] == kImStart) { q = i; break; } + int64_t batched_end = reuse; // [reuse, batched_end) through the batched prompt path + if (n - 1 - reuse > o.feed_max) batched_end = (q > reuse && n - 1 - q <= o.feed_max) ? q : n - 1; + if (batched_end > reuse) { + std::vector> lent_now; + apply_pending(true); + if (lend_first >= 0) { + for (size_t i = 0; i < host_res.size(); ++i) + if (host_res[i] >= lend_first) { + lent_now.emplace_back((int32_t) i, host_res[i]); + host_res[i] = strata::core::kNotResident; + } + cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); + } + for (int64_t a0 = reuse; a0 < batched_end;) { + const int64_t b0 = track ? std::min(batched_end, (a0 / o.cache_every + 1) * o.cache_every) + : batched_end; + if (!sp.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } + if (track) { + pcache.set(a0, ids.data() + a0, b0 - a0); + if (b0 % o.cache_every == 0 || b0 == q || b0 == n - 1) pcache.checkpoint(b0); } - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - } - if (n > 1 && !sp.run(ids.data(), n - 1, 0, err)) { - std::fprintf(stderr, "strata serve: %s\n", err.c_str()); - std::printf("ERR %s\n", err.c_str()); - return 1; - } - for (const auto& [i, slot] : lent_now) { - const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert); - if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err, - (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) { - std::printf("ERR refilling a lent slot failed: %s\n", err.c_str()); - return 1; + a0 = b0; } - host_res[(size_t) i] = slot; + for (const auto& [i, slot] : lent_now) { + const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert); + if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err, + (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) { + std::printf("ERR refilling a lent slot failed: %s\n", err.c_str()); + return 1; + } + host_res[(size_t) i] = slot; + } + if (!lent_now.empty()) + cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); } - if (!lent_now.empty()) - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); + // prompt tokens through verify windows: each window is committed whole, and the draft layer catches up + // on the same cells (its drafts are not used) + auto feed = [&](int64_t from, int64_t to) -> bool { + for (int64_t j = from; j < to;) { + const int T = (int) std::min(S, to - j); + for (int i = 0; i < T; ++i) window[(size_t) i] = (int32_t) ids[(size_t) (j + i)]; + drive.d.layers = 0; + drive.d.experts = 0; + drive.d.failed = false; + apply_pending(false); + if (!ver.run(T, window.data(), j, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) { + if (drive.d.failed && drive.d.fail) err = drive.d.fail; + return false; + } + if (!ver.commit(T, err)) return false; + for (int i = 0; i < T; ++i) outv[(size_t) i] = (int32_t) ids[(size_t) (j + i + 1)]; + if (!mtp.draft(T, outv.data(), j, T - 1, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) + return false; + if (track) pcache.set(j, ids.data() + j, T); + j += T; + } + return true; + }; + const int64_t split = q > batched_end ? q : batched_end; + if (!feed(batched_end, split)) { report(err); return 1; } + if (track && split == q) pcache.checkpoint(q); + if (!feed(split, n - 1)) { report(err); return 1; } + if (track) pcache.checkpoint(n - 1); const double prompt_ms = std::chrono::duration(Clock::now() - r0).count(); // the verify windows: the first holds the last prompt token alone int64_t p = n - 1; int32_t x = (int32_t) ids[(size_t) (n - 1)]; - std::vector drafts((size_t) S, 0), window((size_t) S), outv((size_t) S); - std::vector dprob((size_t) S, 0.0f); bool first_window = true; int64_t produced_n = 0; const char* finish = "length"; @@ -2017,6 +2101,15 @@ int main(int argc, char** argv) { while (T < S && dprob[(size_t) T - 1] >= (float) o.spec_min_p) ++T; } if (first_window) T = 1; + // Never verify past what this request emits: the committed cells then hold exactly the tokens the + // client saw, so its next turn continues from them. So the window stops at the output limit and + // at a drafted end-of-turn token. + T = (int) std::min(T, max_new - produced_n); + for (int i = 1; i < T; ++i) + if (std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) drafts[(size_t) i - 1]) != o.eos_ids.end()) { + T = i + 1; + break; + } if (p + T > o.max_context) break; window[0] = x; for (int i = 1; i < T; ++i) window[(size_t) i] = drafts[(size_t) i - 1]; @@ -2039,6 +2132,7 @@ int main(int argc, char** argv) { std::printf("ERR %s\n", err.c_str()); return 1; } + if (track) pcache.set(p, window.data(), a + 1); // the committed cells p .. p + a first_window = false; bool eos = false; for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) { @@ -2064,12 +2158,15 @@ int main(int argc, char** argv) { p += a + 1; } const double decode_ms = std::chrono::duration(Clock::now() - d0).count(); - std::printf("DONE %lld %lld %.1f %.1f %s\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms, - finish); + std::printf("DONE %lld %lld %.1f %.1f %s %lld\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms, + finish, (long long) reuse); std::fflush(stdout); - std::fprintf(stderr, "strata serve: %lld prompt tokens in %.0f ms (%.1f tok/s), %lld generated in %.0f ms " - "(%.1f tok/s)\n", (long long) n, prompt_ms, prompt_ms > 0 ? 1000.0 * n / prompt_ms : 0.0, - (long long) produced_n, decode_ms, decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0); + const int64_t fresh = n - reuse; + std::fprintf(stderr, "strata serve: %lld prompt tokens (%lld cached) in %.0f ms (%.1f tok/s), %lld generated " + "in %.0f ms (%.1f tok/s)\n", (long long) n, (long long) reuse, prompt_ms, + prompt_ms > 0 ? 1000.0 * fresh / prompt_ms : 0.0, (long long) produced_n, decode_ms, + decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0); + if (pcache.enabled()) std::fprintf(stderr, "strata serve: prompt cache: %s\n", pcache.summary().c_str()); } return 0; } From 8029fa9df743cbcf3f2a309fb36cc07e36e57e4d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 01:54:29 +0200 Subject: [PATCH 02/92] Unsloth Dynamic 3.0 model files The engine now runs unsloth's Qwen3.8-Flash-Next quants (UD-IQ1_S up to UD-Q6_K_XL and Q8_0) as native packs, besides the ISTA-DASLab files. - Split models: --native takes the first shard and finds the others by name (gguf_split_paths, GgufModel). The expert arena is filled from the shards by tensor name and checked against native_experts.txt, whose lines no longer carry shard-1 offsets. The native head and embedding are found in any shard, and --ple-gguf defaults to the shard that holds per_layer_token_embd. - GPU kernels: expert dot products and dequantizers for Q4_K, Q5_K, Q6_K, Q5_1, Q8_0, IQ1_S and IQ4_XS (llama.cpp's), one dispatch list for all formats, and a startup check that every layer's formats have kernels. The Q5_1 min and IQ1_S delta terms use the quantized activation sums, as ggml-cpu does: 1.1% instead of 1.9% relative error per expert. - PLE: the table may be IQ4_NL or Q8_0 and may share its shard with other tensors; the native PLE key may be any native MMVQ type (Q8_0 here). - tools/iq_pack.py: all shards of a split model, and every float tensor in the form the engine takes. The files' F32 router, SSM gate and injection tensors hold BF16 values (exact); their Q8_0 hyper-connection and PLE value weights are re-rounded to BF16. - native_expert_parity: split models, a --synthetic mode for formats with no model file at hand, and a check of the GPU dequantizers. - README: the unsloth files, their RAM needs and how to pack them. UD-Q4_K_XL on an RTX 3090 (160 GB RAM): prompts 529-594 tok/s, generation 49-57 tok/s. Greedy tokens equal llama.cpp's argmax on the same GGUF at 98.2% (code) and 95.6% (thinking) of positions. --- README.md | 22 + include/strata/artifact/gguf_reader.hpp | 30 ++ include/strata/artifact/gguf_split.hpp | 38 ++ include/strata/core/expert_source.hpp | 7 +- include/strata/core/native_head.hpp | 8 +- include/strata/kernels/cpu/expert_layout.hpp | 3 - include/strata/kernels/iq_kernels.hpp | 8 +- include/strata/kernels/ngram.hpp | 13 +- include/strata/ngram/ple_reader.hpp | 17 +- src/core/expert_source.cpp | 67 ++- src/core/native_head.cpp | 25 +- src/kernels/cpu/expert_layout.cpp | 9 +- src/kernels/cuda/iq_kernels.cu | 404 +++++++++++++++++- src/kernels/cuda/ple.cu | 4 +- src/kernels/native_expert_parity.cpp | 409 ++++++++++++------- src/kernels/ngram.cpp | 89 ++-- src/ngram/ple_reader.cpp | 37 +- src/program/generate.cpp | 63 ++- tools/iq_pack.py | 212 +++++++--- 19 files changed, 1122 insertions(+), 343 deletions(-) create mode 100644 include/strata/artifact/gguf_split.hpp diff --git a/README.md b/README.md index 381fb306c39..55de09e3136 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,28 @@ of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next). With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. 32 GB is not enough. +### unsloth's Unsloth Dynamic 3.0 files + +The engine also runs [unsloth's quantizations](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), which +START-HERE does not set up. All experts sit in RAM, pinned when they fit the driver's limit: + +| Model | Download | Experts in RAM | n-gram table | +| --- | ---: | ---: | --- | +| UD-IQ1_S / UD-IQ1_M | 73-75 GB | 40-42 GB | IQ4_NL, 28.8 GB | +| UD-Q2_K_XL / UD-IQ3_XXS | 79-82 GB | 46-49 GB | IQ4_NL | +| UD-Q3_K_XL / UD-IQ4_XS | 90-94 GB | 56-60 GB | IQ4_NL | +| UD-Q4_K_XL | 111 GB | 77 GB | IQ4_NL | +| UD-Q5_K_XL / UD-Q6_K_XL | 158-169 GB | 98-109 GB | Q8_0, 54.4 GB | +| Q8_0 | 188 GB | 128 GB | Q8_0 | + +Pack a model once (seconds: the tokenizer and the float tensors; the experts are read from the GGUF at start): + + python tools/iq_pack.py --gguf /-00001-of-0000N.gguf --out packs/ + +and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, +`--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 529-594 tok/s, generation 49-57 tok/s. + ## Before you start You need **only an NVIDIA driver** (version 580 or newer; update it with the NVIDIA App or from diff --git a/include/strata/artifact/gguf_reader.hpp b/include/strata/artifact/gguf_reader.hpp index eb490b644ed..d32523f952a 100644 --- a/include/strata/artifact/gguf_reader.hpp +++ b/include/strata/artifact/gguf_reader.hpp @@ -23,12 +23,15 @@ #include #include #include +#include #include #include #include #include #include +#include "strata/artifact/gguf_split.hpp" + #ifdef _WIN32 #define WIN32_LEAN_AND_MEAN #include @@ -338,6 +341,7 @@ class GgufFile { uint32_t version() const { return version_; } uint64_t data_start() const { return data_start_; } uint64_t file_size() const { return size_; } + uint64_t alignment() const { return alignment_; } const TensorInfo* find(const std::string& name) const { for (const auto& t : tensors_) @@ -355,6 +359,7 @@ class GgufFile { return it == meta_.end() ? nullptr : &it->second; } const uint8_t* tensor_data(const TensorInfo& t) const { return base_ + data_start_ + t.offset; } + const std::string& path() const { return path_; } private: void open() { @@ -451,6 +456,31 @@ class GgufFile { std::map meta_; }; +// The opened shards of one model, tensors looked up across all of them. Shard 0 holds the metadata. +class GgufModel { +public: + explicit GgufModel(const std::vector& paths) { + if (paths.empty()) throw std::runtime_error("GGUF: a model needs at least one shard"); + for (const auto& p : paths) shards_.push_back(std::make_unique(p)); + } + size_t size() const { return shards_.size(); } + const GgufFile& shard(size_t i) const { return *shards_[i]; } + /// The tensor named `name` (its shard index in `*shard`), or nullptr. Throws when two shards hold it. + const TensorInfo* find(const std::string& name, size_t* shard = nullptr) const { + const TensorInfo* found = nullptr; + for (size_t i = 0; i < shards_.size(); ++i) + if (const TensorInfo* t = shards_[i]->find(name)) { + if (found) throw std::runtime_error("GGUF: tensor " + name + " is in two shards"); + found = t; + if (shard) *shard = i; + } + return found; + } + +private: + std::vector> shards_; +}; + // ---- architecture guard (P1.S2). The engine is specialised to ONE model; anything else must be // refused with a precise error rather than silently mis-run. struct Qwen4ExpGuard { diff --git a/include/strata/artifact/gguf_split.hpp b/include/strata/artifact/gguf_split.hpp new file mode 100644 index 00000000000..5343393db47 --- /dev/null +++ b/include/strata/artifact/gguf_split.hpp @@ -0,0 +1,38 @@ +// include/strata/artifact/gguf_split.hpp - the shards of a split model. Separate from gguf_reader.hpp, which +// includes , so any translation unit can resolve a model's shards. +#pragma once + +#include +#include +#include +#include +#include + +namespace strata { + +// Split models: `-00001-of-0000N.gguf` ... `-0000N-of-0000N.gguf`, as llama.cpp's gguf-split +// names them. The shards in split order, from the first one's path; a file without that suffix is a model of +// one shard. Throws when a shard is missing. +inline std::vector gguf_split_paths(const std::string& first) { + static const char pat[] = "-00001-of-"; + const size_t at = first.rfind(pat); + const size_t digits = at == std::string::npos ? 0 : at + sizeof(pat) - 1; + if (at == std::string::npos || first.size() != digits + 10 || first.compare(digits + 5, 5, ".gguf") != 0) + return {first}; + int n = 0; + for (size_t i = digits; i < digits + 5; ++i) { + if (first[i] < '0' || first[i] > '9') return {first}; + n = n * 10 + (first[i] - '0'); + } + std::vector out; + for (int i = 1; i <= n; ++i) { + char no[8]; + std::snprintf(no, sizeof no, "%05d", i); + std::string p = first.substr(0, at + 1) + no + first.substr(at + 6); + if (!std::ifstream(p, std::ios::binary)) throw std::runtime_error("missing model shard " + p); + out.push_back(std::move(p)); + } + return out; +} + +} // namespace strata diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index c64bf4e7d15..cdc3ad511f6 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -321,8 +321,9 @@ class ArenaExpertSource : public ExpertSource { /// Allocates and loads `/experts.bin`. Prints nothing; the caller reports `note()` and the load /// rate, because those are the two numbers that say whether the arena is the one that was asked for. bool open(const std::string& pack_dir, int64_t n_layers, int64_t n_expert, int threads, std::string& err); - /// Plan v0.3 P6: a native pack without experts.bin takes its experts from the model's shard 1. - void set_gguf(const std::string& shard1) { gguf_ = shard1; } + /// Plan v0.3 P6: a native pack without experts.bin takes its experts from the model's GGUF shards + /// (strata::gguf_split_paths), found by tensor name. + void set_gguf(const std::vector& shards) { gguf_ = shards; } void close(); bool mapped() const { return base_ != nullptr; } @@ -348,7 +349,7 @@ class ArenaExpertSource : public ExpertSource { std::string note_; double gib_per_s_ = 0.0; uint64_t pinned_bytes_ = 0; - std::string gguf_; + std::vector gguf_; }; } // namespace strata::core diff --git a/include/strata/core/native_head.hpp b/include/strata/core/native_head.hpp index 22b4c8db596..cf2bc09988c 100644 --- a/include/strata/core/native_head.hpp +++ b/include/strata/core/native_head.hpp @@ -3,12 +3,14 @@ #include #include #include +#include namespace strata::core { /// Opt-in native output projection. Uploads unchanged GGUF blocks once at startup and owns its /// activation scratch. The normal canonical head remains available for numerical A/B comparisons. -/// Plan v0.3 P6: any type the native MMVQ takes (Q5_K in the Q2_0 / IQ3_XXS files, IQ4_XS in IQ2_XS). +/// Plan v0.3 P6: any type the native MMVQ takes (Q5_K in the Q2_0 / IQ3_XXS files, IQ4_XS in IQ2_XS, Q8_0 in +/// UD-Q4_K_XL). `load` takes the model's shards (strata::gguf_split_paths). class NativeHead { public: NativeHead() = default; @@ -16,7 +18,7 @@ class NativeHead { NativeHead(const NativeHead&) = delete; NativeHead& operator=(const NativeHead&) = delete; - bool load(const std::string& gguf, int64_t n_in, int64_t n_out, std::string& err); + bool load(const std::vector& shards, int64_t n_in, int64_t n_out, std::string& err); bool run(const float* mixed, float* logits, void* stream, std::string& err) const; uint64_t weight_bytes() const { return bytes_; } bool loaded() const { return weights_ != nullptr; } @@ -43,7 +45,7 @@ class NativeEmbed { ~NativeEmbed(); NativeEmbed(const NativeEmbed&) = delete; NativeEmbed& operator=(const NativeEmbed&) = delete; - bool load(const std::string& gguf, int64_t n_embd, int64_t n_vocab, std::string& err); + bool load(const std::vector& shards, int64_t n_embd, int64_t n_vocab, std::string& err); /// Rows for device token ids. void gather_dev(const int32_t* tokens, int64_t n_tok, float* out, void* stream) const; /// One row for a host token id. diff --git a/include/strata/kernels/cpu/expert_layout.hpp b/include/strata/kernels/cpu/expert_layout.hpp index d4b20cb0a49..b96a5c9f1be 100644 --- a/include/strata/kernels/cpu/expert_layout.hpp +++ b/include/strata/kernels/cpu/expert_layout.hpp @@ -20,9 +20,6 @@ struct ExpertLayout { int64_t n_layers = 0, n_expert = NE; std::vector fmt; ///< per layer (native packs) std::vector offset, bytes; ///< per layer: where its 512 blobs start, bytes per blob - /// Plan v0.3 P6: per layer, the absolute offsets of the gate / up / down tensors in the model's shard 1, so - /// the arena can be filled from the GGUF itself when the pack has no experts.bin (3 x n_layers, 0 = unknown). - std::vector gguf_off; uint64_t max_blob = BLOB; uint64_t total = 0; ///< experts.bin size diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index 020d78d522f..8b1baf6955a 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -1,5 +1,6 @@ -// include/strata/kernels/iq_kernels.hpp - the i-quant formats (IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, -// IQ4_NL) and Q2_0 on the GPU for the IQ2_XS / IQ3_XXS model files. +// include/strata/kernels/iq_kernels.hpp - the GGUF expert and embedding formats on the GPU: the i-quants (IQ1_M, +// IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL) and Q2_0 of the IQ2_XS / IQ3_XXS model files, and Q4_K, Q5_K, +// Q5_1 and Q8_0 of the UD-Q4_K_XL file. // // The block layouts, codebook grids and dot products are llama.cpp's (ggml-common.h, ggml-cuda/vecdotq.cuh, // ggml-cuda/dequantize.cuh; MIT, see third_party/ggml/LICENSE and VERSION.txt), so a weight means exactly what it @@ -43,6 +44,9 @@ struct NativeExpertLayout { size_t bytes = 0; // the whole blob }; NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd, int64_t n_ff); +/// Whether the GPU has the dot products (native_expert_grouped) and the dequantizers (the prompt path) for a layer's +/// gate/up and down formats at this geometry. +bool native_expert_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept; /// Bytes of scratch `native_expert_grouped` needs for `cap_entries` entries. size_t native_expert_scratch_bytes(int64_t cap_entries, int64_t n_ff); diff --git a/include/strata/kernels/ngram.hpp b/include/strata/kernels/ngram.hpp index d15491dedc3..b1e90d24a05 100644 --- a/include/strata/kernels/ngram.hpp +++ b/include/strata/kernels/ngram.hpp @@ -5,9 +5,9 @@ // * `ngram_rows` - the HOST hash. Sixteen row indices per token from the last three token ids, computed // with 64-bit multiply/xor and no tensor op at all. `qwen4exp.cpp` L1092-1124, where the source says // outright that it is host-side because "ggml has no int64 and no xor". -// * `PleTable` - the row gather. `per_layer_token_embd.weight` is IQ4_NL and is NOT IN THE PACK: it is -// 51.2e9 elements (28.8 GB) in the ORIGINAL second GGUF shard, and it is the only tensor this engine -// reads from the GGUF rather than from the canonical pack. +// * `PleTable` - the row gather. `per_layer_token_embd.weight` is IQ4_NL (Q8_0 in the unsloth UD-Q5_K_XL and +// larger files) and is NOT IN THE PACK: it is 51.2e9 elements (28.8 GB; 54.4 GB as Q8_0) in the model's +// GGUF, and the engine reads it from there. // // THE HASH IS THE ONE PLACE IN THIS MODEL WHERE A WRONG BIT IS SILENTLY PLAUSIBLE. Every one of its // properties has a rival reading that produces a valid index in range: @@ -24,6 +24,7 @@ #include #include +#include namespace strata::kernels { @@ -45,6 +46,7 @@ inline constexpr float NG_RMS_EPS = 1e-6f; // 5 blocks of 32 at 18 bytes = 90 bytes. The head-slowest flatten then makes 16 rows exactly n_embd = 2560. inline constexpr uint64_t PLE_TABLE_ROWS = 320001536ull; inline constexpr int PLE_ROW_BYTES = (PLE_HEAD_DIM / 32) * 18; // 90 +inline constexpr int PLE_ROW_BYTES_Q8_0 = (PLE_HEAD_DIM / 32) * 34; // 170, the Q8_0 tables /// The artifact's own hash constants, transcribed from `docs/gguf-dump-shard1.txt`: /// @@ -95,6 +97,8 @@ int iq4nl_code(int code); /// so byte j carries elements j and j+16 - NOT 2j and 2j+1, which is the natural reading and which produces /// a perfectly plausible embedding of the wrong 160 values. void iq4nl_dequant_row(const uint8_t* row, float* out160); +/// One row of a table of GGML type `type` (IQ4_NL = 20 or Q8_0 = 8) -> 160 floats. +void ple_dequant_row(int type, const uint8_t* row, float* out160); /// How the table's rows are read (plan v0.3 P2). `Direct` is the default: unbuffered 4 KiB reads from the SSD, /// so the table never occupies RAM or the OS file cache. `Mmap` is the earlier memory-mapped path, kept as the @@ -112,6 +116,9 @@ struct PleIoOptions { bool io_thread = true; ///< reads submitted by a worker thread, not the caller }; +/// The shard among `shards` that holds `per_layer_token_embd.weight`, or "" (headers only). +std::string ple_table_shard(const std::vector& shards); + /// The PLE table. Held by pointer-to-impl so this header does not drag `` into every /// translation unit that wants the hash. class PleTable { diff --git a/include/strata/ngram/ple_reader.hpp b/include/strata/ngram/ple_reader.hpp index 0552ff1e4c5..4a0ee2ddf17 100644 --- a/include/strata/ngram/ple_reader.hpp +++ b/include/strata/ngram/ple_reader.hpp @@ -1,6 +1,7 @@ // include/strata/ngram/ple_reader.hpp - plan v0.3 P2: the n-gram table read straight from the SSD. // -// The table is 320,001,536 rows of 90 bytes (26.8 GiB) and is never held in RAM: every row comes from an +// The table is 320,001,536 rows of 90 bytes (IQ4_NL, 26.8 GiB; 170 bytes as Q8_0) and is never held in RAM: every +// row comes from an // unbuffered 4 KiB read (platform::DirectFile). A token needs 16 rows on 16 different pages, and all 16 depend // on the token itself, so the only time to hide them is the embedding plus layer 0. Hence the split API: // @@ -11,7 +12,7 @@ // `issue` also serves prefill: pass all 16 x N rows of a chunk; pages are deduplicated, sorted by offset and // kept at most `max_inflight` deep, so a chunk's reads can run while the previous chunk computes. // -// THE ROW CACHE IS NOT THE TABLE. It keeps rows this process has already fetched (90 bytes each, bounded, +// THE ROW CACHE IS NOT THE TABLE. It keeps rows this process has already fetched (one row each, bounded, // clock eviction). Measured on the frozen corpus (bench/results/2026-09-23-ngram-io): 1M rows (~95 MB) // would serve up to ~82% of reads; within one long prompt 20-34% of rows recur. Capacity 0 disables it. #pragma once @@ -24,7 +25,7 @@ namespace strata::ngram { -inline constexpr uint32_t ROW_BYTES = 90; +inline constexpr uint32_t ROW_BYTES = 90; ///< an IQ4_NL row, the default inline constexpr uint32_t PAGE = 4096; struct ReaderStats { @@ -52,17 +53,17 @@ class PleReader { PleReader(const PleReader&) = delete; PleReader& operator=(const PleReader&) = delete; - /// `table_offset` is the byte offset of row 0 in the file and `n_rows` the row count; both come from a - /// validated GGUF parse (PleTable::open checks the table exactly fills the file from there). + /// `table_offset` is the byte offset of row 0 in the file, `n_rows` the row count and `row_bytes` a row's + /// size; all come from a validated GGUF parse (PleTable::open checks the table exactly fills its span). /// `io_thread` (default): a worker thread submits and reaps reads, so `issue` costs the caller no ReadFile /// calls. false: the caller's thread does it (A/B arm). bool open(const std::string& path, uint64_t table_offset, uint64_t n_rows, uint32_t max_inflight, - uint64_t cache_rows, std::string& err, bool io_thread = true); + uint64_t cache_rows, std::string& err, bool io_thread = true, uint32_t row_bytes = ROW_BYTES); void close(); bool is_open() const; - /// Start fetching `n` rows; row i's 90 raw bytes land at `out_raw + 90 * i`. `out_raw` must stay valid - /// until `collect` returns. Out-of-range rows produce 90 zero bytes (the mmap path's behaviour). + /// Start fetching `n` rows; row i's raw bytes land at `out_raw + row_bytes * i`. `out_raw` must stay valid + /// until `collect` returns. Out-of-range rows produce zero bytes (the mmap path's behaviour). Ticket issue(const uint32_t* rows, size_t n, uint8_t* out_raw); /// Block until every row of the ticket is in `out_raw`. Returns false on an I/O error (message in `err`). diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index db4465d6fae..b44fc00b8cd 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -1,5 +1,6 @@ // src/core/expert_source.cpp - the adapter. See the header for the three clauses of the contract. #include "strata/core/expert_source.hpp" +#include "strata/artifact/gguf_reader.hpp" #include "strata/kernels/cpu/expert_layout.hpp" #include "strata/core/pinned.hpp" @@ -535,19 +536,57 @@ void expert_hit_run(void* user, void* stream, HitPhase phase, const int32_t* ids // ================================ THE RESIDENT ARENA (R2.1) ================================ -// Plan v0.3 P6: the arena from the model's shard 1. Each layer's gate, up and down tensors hold the 512 experts -// one after another; they are read in chunks and each expert's slice lands at its place in the blob +namespace { +// Plan v0.3 P6: the arena straight from the model's GGUF shards. Each layer's gate, up and down tensors hold the +// 512 experts one after another; they are read in chunks and each expert's slice lands at its place in the blob // [gate rows | up rows | down rows] - the layout tools/iq_pack.py would have written to experts.bin. -LoadStats load_experts_gguf(const std::string& gguf, uint8_t* dst, const strata::kernels::cpu::ExpertLayout& lay, - int threads) { +struct GgufSpan { + size_t shard = 0; + uint64_t offset = 0; ///< in the shard's file +}; + +/// Where each layer's gate, up and down tensors are (3 per layer), found by name in whichever shard holds them +/// and checked against the pack's formats, so a pack and a model that do not belong together are refused. +bool locate_experts_gguf(const std::vector& shards, const strata::kernels::cpu::ExpertLayout& lay, + std::vector& out, std::string& err) { + static const char* roles[3] = {"gate", "up", "down"}; + out.assign((size_t) (3 * lay.n_layers), GgufSpan{}); + try { + const strata::GgufModel model(shards); + for (int64_t l = 0; l < lay.n_layers; ++l) { + const auto& fm = lay.fmt[(size_t) l]; + const uint64_t per[3] = {fm.up_off, fm.up_off, lay.bytes[(size_t) l] - fm.down_off}; + for (int r = 0; r < 3; ++r) { + const std::string name = "blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight"; + size_t s = 0; + const strata::TensorInfo* t = model.find(name, &s); + int be = 0, bb = 0; + if (t == nullptr || (int) t->type != (r < 2 ? fm.gu_type : fm.d_type) || + !strata::block_geometry(t->type, be, bb) || + t->elements() / (uint64_t) be * (uint64_t) bb != per[r] * (uint64_t) lay.n_expert) { + err = "ArenaExpertSource: " + name + " is missing from the model or differs from the pack's " + "native_experts.txt"; + return false; + } + out[(size_t) (3 * l + r)] = GgufSpan{s, model.shard(s).data_start() + t->offset}; + } + } + return true; + } catch (const std::exception& e) { + err = std::string("ArenaExpertSource: ") + e.what(); + return false; + } +} + +LoadStats load_experts_gguf(const std::vector& shards, const std::vector& spans, uint8_t* dst, + const strata::kernels::cpu::ExpertLayout& lay, int threads) { LoadStats st; st.layers = (uint64_t) lay.n_layers; const auto t0 = std::chrono::steady_clock::now(); std::atomic next{0}; std::atomic bad{false}; auto worker = [&]() { - std::ifstream f(gguf, std::ios::binary); - if (!f) { bad = true; return; } + std::vector files(shards.size()); std::vector buf; for (;;) { const int64_t l = next.fetch_add(1); @@ -557,7 +596,11 @@ LoadStats load_experts_gguf(const std::string& gguf, uint8_t* dst, const strata: const uint64_t per[3] = {fm.up_off, fm.up_off, blob - fm.down_off}; const uint64_t at[3] = {0, fm.up_off, fm.down_off}; for (int r = 0; r < 3; ++r) { - const uint64_t src = lay.gguf_off[(size_t) (3 * l + r)]; + const GgufSpan& sp = spans[(size_t) (3 * l + r)]; + std::ifstream& f = files[sp.shard]; + if (!f.is_open()) f.open(shards[sp.shard], std::ios::binary); + if (!f) { bad = true; return; } + const uint64_t src = sp.offset; const uint64_t total = per[r] * (uint64_t) lay.n_expert; const uint64_t chunk = per[r] * 16; // 16 experts per read buf.resize((size_t) chunk); @@ -586,6 +629,7 @@ LoadStats load_experts_gguf(const std::string& gguf, uint8_t* dst, const strata: st.seconds = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); return st; } +} // namespace ArenaExpertSource::~ArenaExpertSource() { close(); } @@ -603,10 +647,13 @@ bool ArenaExpertSource::open(const std::string& pack_dir, int64_t n_layers, int6 const uint64_t want = lay.total; // plan v0.3 P6: no experts.bin in a native pack -> the experts come straight from the GGUF - const bool from_gguf = !std::ifstream(path, std::ios::binary) && lay.native && !lay.gguf_off.empty() && !gguf_.empty(); + const bool from_gguf = !std::ifstream(path, std::ios::binary) && lay.native && !gguf_.empty(); // SIZE CHECK BEFORE THE ALLOCATION, not after. A wrong pack should name the two numbers rather than spend // 34 GB and a minute of loading first. - if (!from_gguf) { + std::vector spans; + if (from_gguf) { + if (!locate_experts_gguf(gguf_, lay, spans, err)) return false; + } else { std::ifstream f(path, std::ios::binary | std::ios::ate); if (!f) { err = "ArenaExpertSource: cannot open " + path; return false; } const uint64_t got = (uint64_t) f.tellg(); @@ -637,7 +684,7 @@ bool ArenaExpertSource::open(const std::string& pack_dir, int64_t n_layers, int6 err = "ArenaExpertSource: the arena could not be reserved (" + std::to_string(want) + " B)"; return false; } - const LoadStats st = from_gguf ? load_experts_gguf(gguf_, a->data(), lay, threads) + const LoadStats st = from_gguf ? load_experts_gguf(gguf_, spans, a->data(), lay, threads) : load_experts_ranges(path, a->data(), loff, lbytes, threads, /*chunk=*/8u << 20); if (st.bytes != want) { delete a; diff --git a/src/core/native_head.cpp b/src/core/native_head.cpp index fc9f694ffcc..e5fb95fa3d7 100644 --- a/src/core/native_head.cpp +++ b/src/core/native_head.cpp @@ -15,22 +15,19 @@ NativeHead::~NativeHead() { if (weights_) cudaFree(weights_); } -bool NativeHead::load(const std::string& path, int64_t n_in, int64_t n_out, std::string& err) { +bool NativeHead::load(const std::vector& shards, int64_t n_in, int64_t n_out, std::string& err) { if (loaded()) { err = "native head is already loaded"; return false; } if (n_in <= 0 || n_out <= 0 || n_in > INT_MAX || n_out > INT_MAX || n_in % 256) { err = "native head requires positive int32 dimensions and whole 256-value rows"; return false; } try { - strata::GgufFile gguf(path); - err = strata::check_architecture(gguf); + const strata::GgufModel model(shards); + err = strata::check_architecture(model.shard(0)); if (!err.empty()) return false; - const strata::TensorInfo* tensor = nullptr; - for (const auto& candidate : gguf.tensors()) { - if (candidate.name != "output.weight") continue; - if (tensor) { err = "native head: duplicate output.weight"; return false; } - tensor = &candidate; - } + size_t at = 0; + const strata::TensorInfo* tensor = model.find("output.weight", &at); + const strata::GgufFile& gguf = model.shard(at); if (!tensor || !strata::kernels::native_mmvq_supported((int) tensor->type) || tensor->shape.size() != 2 || tensor->shape[0] != (uint64_t) n_in || tensor->shape[1] != (uint64_t) n_out) { err = "native head: expected a natively supported output.weight with the canonical head dimensions"; @@ -104,12 +101,12 @@ NativeEmbed::~NativeEmbed() { if (host_) cudaFreeHost(host_); } -bool NativeEmbed::load(const std::string& path, int64_t n_embd, int64_t n_vocab, std::string& err) { +bool NativeEmbed::load(const std::vector& shards, int64_t n_embd, int64_t n_vocab, std::string& err) { try { - strata::GgufFile gguf(path); - const strata::TensorInfo* t = nullptr; - for (const auto& c : gguf.tensors()) - if (c.name == "token_embd.weight") t = &c; + const strata::GgufModel model(shards); + size_t at = 0; + const strata::TensorInfo* t = model.find("token_embd.weight", &at); + const strata::GgufFile& gguf = model.shard(at); if (!t || t->shape.size() != 2 || t->shape[0] != (uint64_t) n_embd || t->shape[1] != (uint64_t) n_vocab || !strata::kernels::iq_supported((int) t->type) || n_embd % 256) { err = "native embedding: token_embd.weight is absent, of another shape, or of a type without a GPU " diff --git a/src/kernels/cpu/expert_layout.cpp b/src/kernels/cpu/expert_layout.cpp index c02cd6577ab..63fe7e47e2b 100644 --- a/src/kernels/cpu/expert_layout.cpp +++ b/src/kernels/cpu/expert_layout.cpp @@ -92,11 +92,12 @@ bool expert_layout_load(const std::string& pack_dir, int64_t n_layers, int64_t n L.bytes.assign((size_t) n_layers, 0); L.max_blob = 0; std::string line; + // `layer gu_type d_type offset blob_bytes`; older packs' extra columns (shard-1 offsets) are not read while (std::getline(in, line)) { if (line.empty() || line[0] == '#') continue; std::istringstream ss(line); long long l = -1, gt = -1, dt = -1; - unsigned long long off = 0, blob = 0, go = 0, uo = 0, dox = 0; + unsigned long long off = 0, blob = 0; if (!(ss >> l >> gt >> dt >> off >> blob) || l < 0 || l >= n_layers) { err = "native_experts.txt: a malformed line: " + line; return false; @@ -108,12 +109,6 @@ bool expert_layout_load(const std::string& pack_dir, int64_t n_layers, int64_t n " B but its formats make " + std::to_string(f.bytes); return false; } - if (ss >> go >> uo >> dox) { // v2 lines: the GGUF offsets - if (L.gguf_off.empty()) L.gguf_off.assign((size_t) (3 * n_layers), 0); - L.gguf_off[(size_t) (3 * l)] = go; - L.gguf_off[(size_t) (3 * l + 1)] = uo; - L.gguf_off[(size_t) (3 * l + 2)] = dox; - } L.fmt[(size_t) l] = f; L.offset[(size_t) l] = off; L.bytes[(size_t) l] = blob; diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index 3525b91ad61..c32ed429246 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -268,6 +268,228 @@ __device__ __forceinline__ float vec_dot_iq4_nl_q8_1(const void* __restrict__ vb return d * sumi; } +// Q4_K / Q5_K gate/up and Q5_1 / Q8_0 down (the unsloth UD-Q4_K_XL file's experts) +__device__ __forceinline__ float vec_dot_q4_K_q8_1_impl_vmmq(const int* __restrict__ v, const int* __restrict__ u, + const uint8_t* __restrict__ sc, const uint8_t* __restrict__ m, + const half2& dm4, const float* __restrict__ d8) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < QR4_K; ++i) { + const int v0i = (v[0] >> (4 * i)) & 0x0F0F0F0F; + const int v1i = (v[1] >> (4 * i)) & 0x0F0F0F0F; + const int dot1 = ggml_cuda_dp4a(v1i, u[2 * i + 1], ggml_cuda_dp4a(v0i, u[2 * i + 0], 0)); + const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 1], ggml_cuda_dp4a(0x01010101, u[2 * i + 0], 0)); + sumf_d += d8[i] * (dot1 * sc[i]); + sumf_m += d8[i] * (dot2 * m[i]); + } + const float2 dm4f = __half22float2(dm4); + return dm4f.x * sumf_d - dm4f.y * sumf_m; +} + +// the 6-bit scale and min of the 32-value group pair `bq8_offset / 2` (vecdotq.cuh, shared by Q4_K and Q5_K) +__device__ __forceinline__ void k_scales(const uint8_t* scales8, int bq8_offset, uint16_t aux[2]) { + const uint16_t* scales = (const uint16_t*) scales8; + const int j = bq8_offset / 2; + const int jm = j & 1; + const uint32_t s0 = scales[jm + 0]; + const uint32_t s2 = scales[jm + 2]; + const uint32_t s4 = scales[jm + 4]; + const uint32_t hi = (uint32_t) -(int32_t) (j >= 2); + aux[0] = (uint16_t) (((s0 & 0x3f3f) & ~hi) | ((((s4 >> 0) & 0x0f0f) | ((s0 & 0xc0c0) >> 2)) & hi)); + aux[1] = (uint16_t) (((s2 & 0x3f3f) & ~hi) | ((((s4 >> 4) & 0x0f0f) | ((s2 & 0xc0c0) >> 2)) & hi)); +} + +__device__ __forceinline__ float vec_dot_q4_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q4_K* bq4_K = (const block_q4_K*) vbq + kbx; + int v[2]; + int u[2 * QR4_K]; + float d8[QR4_K]; + const int bq8_offset = QR4_K * ((iqs / 2) / (QI8_1 / 2)); + const int* q4 = (const int*) (bq4_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); + v[0] = q4[0]; + v[1] = q4[4]; + uint16_t aux[2]; + k_scales(bq4_K->scales, bq8_offset, aux); + const uint8_t* sc = (const uint8_t*) aux; + const uint8_t* m = sc + 2; + for (int i = 0; i < QR4_K; ++i) { + const block_q8_1* bq8i = bq8_1 + bq8_offset + i; + d8[i] = __low2float(bq8i->ds); + const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); + u[2 * i + 0] = q8[0]; + u[2 * i + 1] = q8[4]; + } + return vec_dot_q4_K_q8_1_impl_vmmq(v, u, sc, m, bq4_K->dm, d8); +} + +__device__ __forceinline__ float vec_dot_q5_K_q8_1_impl_vmmq(const int* __restrict__ vl, const int* __restrict__ vh, + const int* __restrict__ u, const uint8_t* __restrict__ sc, + const uint8_t* __restrict__ m, const half2& dm5, + const float* __restrict__ d8) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < QR5_K; ++i) { + const int vl0i = (vl[0] >> (4 * i)) & 0x0F0F0F0F; + const int vl1i = (vl[1] >> (4 * i)) & 0x0F0F0F0F; + const int vh0i = ((vh[0] >> i) << 4) & 0x10101010; + const int vh1i = ((vh[1] >> i) << 4) & 0x10101010; + const int v0i = vl0i | vh0i; + const int v1i = vl1i | vh1i; + const int dot1 = ggml_cuda_dp4a(v0i, u[2 * i + 0], ggml_cuda_dp4a(v1i, u[2 * i + 1], 0)); + const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 0], ggml_cuda_dp4a(0x01010101, u[2 * i + 1], 0)); + sumf_d += d8[i] * (dot1 * sc[i]); + sumf_m += d8[i] * (dot2 * m[i]); + } + const float2 dm5f = __half22float2(dm5); + return dm5f.x * sumf_d - dm5f.y * sumf_m; +} + +__device__ __forceinline__ float vec_dot_q5_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q5_K* bq5_K = (const block_q5_K*) vbq + kbx; + int vl[2]; + int vh[2]; + int u[2 * QR5_K]; + float d8[QR5_K]; + const int bq8_offset = QR5_K * ((iqs / 2) / (QI8_1 / 2)); + const int* ql = (const int*) (bq5_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); + const int* qh = (const int*) (bq5_K->qh + 4 * ((iqs / 2) % 4)); + vl[0] = ql[0]; + vl[1] = ql[4]; + vh[0] = qh[0] >> bq8_offset; + vh[1] = qh[4] >> bq8_offset; + uint16_t aux[2]; + k_scales(bq5_K->scales, bq8_offset, aux); + const uint8_t* sc = (const uint8_t*) aux; + const uint8_t* m = sc + 2; +#pragma unroll + for (int i = 0; i < QR5_K; ++i) { + const block_q8_1* bq8i = bq8_1 + bq8_offset + i; + d8[i] = __low2float(bq8i->ds); + const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); + u[2 * i + 0] = q8[0]; + u[2 * i + 1] = q8[4]; + } + return vec_dot_q5_K_q8_1_impl_vmmq(vl, vh, u, sc, m, bq5_K->dm, d8); +} + +constexpr int VDR_Q5_1 = 2, VDR_Q8_0 = 2; // VDR_Q5_1_Q8_1_MMVQ, VDR_Q8_0_Q8_1_MMVQ + +// The weights are llama.cpp's; the min term is not. llama.cpp multiplies the Q5_1 min by the q8_1 block's sum of +// the ORIGINAL activations while the scaled term uses the quantized ones, which adds min * (sum x - sum x_q) to +// every block: 1.9% instead of 1.2% relative error per expert (native_expert_parity). Like ggml-cpu, and like +// llama.cpp's own Q4_K / Q5_K dots, the min multiplies the quantized sum here. +__device__ __forceinline__ float vec_dot_q5_1_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q5_1* bq5_1 = (const block_q5_1*) vbq + kbx; + int sumi = 0, sumu = 0; +#pragma unroll + for (int i = 0; i < VDR_Q5_1; ++i) { + const int vl = get_int_b4(bq5_1->qs, iqs + i); + const int vh = get_int_b4(bq5_1->qh, 0) >> (4 * (iqs + i)); + const int u0 = get_int_b4(bq8_1->qs, iqs + i), u1 = get_int_b4(bq8_1->qs, iqs + i + QI5_1); + int vi0 = (vl >> 0) & 0x0F0F0F0F; + vi0 |= (vh << 4) & 0x00000010; + vi0 |= (vh << 11) & 0x00001000; + vi0 |= (vh << 18) & 0x00100000; + vi0 |= (vh << 25) & 0x10000000; + sumi = ggml_cuda_dp4a(vi0, u0, sumi); + int vi1 = (vl >> 4) & 0x0F0F0F0F; + vi1 |= (vh >> 12) & 0x00000010; + vi1 |= (vh >> 5) & 0x00001000; + vi1 |= (vh << 2) & 0x00100000; + vi1 |= (vh << 9) & 0x10000000; + sumi = ggml_cuda_dp4a(vi1, u1, sumi); + sumu = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumu)); + } + const float2 dm5 = __half22float2(bq5_1->dm); + const float d8 = __low2float(bq8_1->ds); + return sumi * (dm5.x * d8) + sumu * (dm5.y * d8); +} + +// Q6_K gate/up (UD-Q5_K_XL, UD-Q6_K_XL), VDR_Q6_K_Q8_1_MMVQ = 1 +__device__ __forceinline__ float vec_dot_q6_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q6_K* bq6_K = (const block_q6_K*) vbq + kbx; + const int bq8_offset = 2 * QR6_K * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 4); + const int scale_offset = (QI6_K / 4) * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 8); + const int vh_shift = 2 * ((iqs % (QI6_K / 2)) / (QI6_K / 4)); + const int vl = get_int_b2(bq6_K->ql, iqs); + const int vh = get_int_b2(bq6_K->qh, (QI6_K / 4) * (iqs / (QI6_K / 2)) + iqs % (QI6_K / 4)) >> vh_shift; + const int8_t* scales = bq6_K->scales + scale_offset; + float sumf = 0.0f; +#pragma unroll + for (int i = 0; i < QR6_K; ++i) { + const int u = get_int_b4(bq8_1[bq8_offset + 2 * i].qs, iqs % QI8_1); + const float d8 = __low2float(bq8_1[bq8_offset + 2 * i].ds); + const int sc = scales[4 * i]; + const int vil = (vl >> (4 * i)) & 0x0F0F0F0F; + const int vih = ((vh >> (4 * i)) << 4) & 0x30303030; + const int vi = __vsubss4((vil | vih), 0x20202020); // (vil | vih) - 32 + sumf += d8 * (ggml_cuda_dp4a(vi, u, 0) * sc); + } + return __half2float(bq6_K->d) * sumf; +} + +// IQ1_S gate/up (UD-IQ1_S), VDR 1. As for Q5_1, the delta term takes the quantized activation sum (ggml-cpu's +// bsums) rather than llama.cpp's sum of the original values. +__device__ __forceinline__ float vec_dot_iq1_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq1_s* bq1 = (const block_iq1_s*) vbq + kbx; + const int qs_packed = get_int_b2(bq1->qs, iqs); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq1->qh[iqs]; + int sumi = 0, sumy = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int grid = iq1s_grid_gpu[qs[l0 / 2] | (((qh >> 3 * (l0 / 2)) & 0x07) << 8)]; + const int grid0 = (grid >> 0) & 0x0F0F0F0F; + const int grid1 = (grid >> 4) & 0x0F0F0F0F; + const int u0 = get_int_b4(bq8_1[iqs].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid0, u0, sumi); + sumi = ggml_cuda_dp4a(grid1, u1, sumi); + sumy = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumy)); + } + const float d1q = __half2float(bq1->d) * (((qh >> 11) & 0x0E) + 1); + const float delta = -1.0f + IQ1S_DELTA - (qh & 0x8000) * (2.0f * IQ1S_DELTA / 0x8000); + return d1q * __low2float(bq8_1[iqs].ds) * (sumi + sumy * delta); +} + +// IQ4_XS gate/up (UD-IQ4_XS, UD-Q3_K_XL), VDR 4 +__device__ __forceinline__ float vec_dot_iq4_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq4_xs* bq4 = (const block_iq4_xs*) vbq + kbx; + int sumi = 0; +#pragma unroll + for (int j = 0; j < 4; ++j) { + const int aux_q4 = get_int_b4(bq4->qs, iqs + j); + const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl); + const int u0 = get_int_b4(bq8_1[iqs / 4].qs, j + 0); + const int u1 = get_int_b4(bq8_1[iqs / 4].qs, j + 4); + sumi = ggml_cuda_dp4a(v.x, u0, sumi); + sumi = ggml_cuda_dp4a(v.y, u1, sumi); + } + const int ls = ((bq4->scales_l[iqs / 8] >> (iqs & 0x04)) & 0x0F) | (((bq4->scales_h >> (iqs / 2)) & 0x03) << 4); + sumi *= ls - 32; + const float d = __half2float(bq4->d) * __low2float(bq8_1[iqs / 4].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_q8_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q8_0* bq8_0 = (const block_q8_0*) vbq + kbx; + int sumi = 0; +#pragma unroll + for (int i = 0; i < VDR_Q8_0; ++i) + sumi = ggml_cuda_dp4a(get_int_b2(bq8_0->qs, iqs + i), get_int_b4(bq8_1->qs, iqs + i), sumi); + const float d8_0 = __half2float(bq8_0->d), d8_1 = __low2float(bq8_1->ds); + return d8_0 * d8_1 * ((float) sumi); +} + // ---------------------------------------------------------------- the formats // qk = values per block, ipb = dot calls per block (qi / vdr), step = the iqs stride between calls. template struct Fmt; @@ -287,6 +509,32 @@ template<> struct Fmt<29> { static constexpr int qk = 256, ipb = 8, step = 1; __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_m_q8_1(v, y, kbx, iqs); } }; template<> struct Fmt<42> { static constexpr int qk = 64, ipb = 2, step = 1; __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q2_0_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<12> { static constexpr int qk = 256, ipb = QI4_K / 2, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q4_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<13> { static constexpr int qk = 256, ipb = QI5_K / 2, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<7> { static constexpr int qk = 32, ipb = QI5_1 / VDR_Q5_1, step = VDR_Q5_1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_1_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<8> { static constexpr int qk = 32, ipb = QI8_0 / VDR_Q8_0, step = VDR_Q8_0; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q8_0_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<14> { static constexpr int qk = 256, ipb = QI6_K, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q6_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<19> { static constexpr int qk = 256, ipb = QI1_S, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_s_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<23> { static constexpr int qk = 256, ipb = QI4_XS / 4, step = 4; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_xs_q8_1(v, y, kbx, iqs); } }; + +// Every format above, for the dispatch switches: X(type) once per Fmt. +#define STRATA_FMTS(X) X(7) X(8) X(12) X(13) X(14) X(16) X(17) X(18) X(19) X(20) X(21) X(22) X(23) X(29) X(42) + +int fmt_qk(int t) { + switch (t) { +#define STRATA_QK(T) case T: return Fmt::qk; + STRATA_FMTS(STRATA_QK) +#undef STRATA_QK + default: return 0; + } +} __device__ __forceinline__ float warp_sum(float v) { #pragma unroll @@ -543,6 +791,113 @@ __device__ void dq_q2_0(const void* vx, int64_t ibs, dst_t* yy, int tid) { yy[b * 64 + i] = cvt(d * (float) (code - 1)); } } +__device__ __forceinline__ void scale_min_k4(int j, const uint8_t* q, uint8_t& d, uint8_t& m) { + if (j < 4) { + d = q[j] & 63; m = q[j + 4] & 63; + } else { + d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4); + m = (q[j + 4] >> 4) | ((q[j - 0] >> 6) << 4); + } +} +template +__device__ void dq_q4_k(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_q4_K* x = (const block_q4_K*) vx; + const int64_t il = tid / 8, ir = tid % 8, is = 2 * il; + const int n = 4; + dst_t* y = yy + 64 * il + n * ir; + const float dall = __low2half(x[ibs].dm); + const float dmin = __high2half(x[ibs].dm); + const uint8_t* q = x[ibs].qs + 32 * il + n * ir; + uint8_t sc, m; + scale_min_k4((int) is + 0, x[ibs].scales, sc, m); + const float d1 = dall * sc, m1 = dmin * m; + scale_min_k4((int) is + 1, x[ibs].scales, sc, m); + const float d2 = dall * sc, m2 = dmin * m; + for (int l = 0; l < n; ++l) { + y[l + 0] = cvt(d1 * (q[l] & 0xF) - m1); + y[l + 32] = cvt(d2 * (q[l] >> 4) - m2); + } +} +// llama.cpp's dequantize_q5_K, its 64 threads folded onto 32 +template +__device__ void dq_q5_k(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_q5_K* x = (const block_q5_K*) vx; + for (int tt = tid; tt < 64; tt += 32) { + const int il = tt / 16, ir = tt % 16, is = 2 * il; + dst_t* y = yy + 64 * il + 2 * ir; + const float dall = __low2half(x[ibs].dm); + const float dmin = __high2half(x[ibs].dm); + const uint8_t* ql = x[ibs].qs + 32 * il + 2 * ir; + const uint8_t* qh = x[ibs].qh + 2 * ir; + uint8_t sc, m; + scale_min_k4(is + 0, x[ibs].scales, sc, m); + const float d1 = dall * sc, m1 = dmin * m; + scale_min_k4(is + 1, x[ibs].scales, sc, m); + const float d2 = dall * sc, m2 = dmin * m; + uint8_t hm = (uint8_t) (1 << (2 * il)); + y[0] = cvt(d1 * ((ql[0] & 0xF) + (qh[0] & hm ? 16 : 0)) - m1); + y[1] = cvt(d1 * ((ql[1] & 0xF) + (qh[1] & hm ? 16 : 0)) - m1); + hm <<= 1; + y[32] = cvt(d2 * ((ql[0] >> 4) + (qh[0] & hm ? 16 : 0)) - m2); + y[33] = cvt(d2 * ((ql[1] >> 4) + (qh[1] & hm ? 16 : 0)) - m2); + } +} +// 32-value blocks: a "superblock" is 8 of them; thread tid writes 8 values of block tid % 8 +template +__device__ void dq_q5_1(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_q5_1* x = (const block_q5_1*) vx + ibs * (QK_K / QK5_1); + const int ib = tid % 8, il = tid / 8; + const float2 dm = __half22float2(x[ib].dm); + uint32_t qh; + memcpy(&qh, x[ib].qh, sizeof(qh)); + dst_t* y = yy + 32 * ib; + for (int j = 0; j < 4; ++j) { + const int iqs = 4 * il + j; + const int xh_0 = ((qh >> (iqs + 0)) << 4) & 0x10; + const int xh_1 = ((qh >> (iqs + 12))) & 0x10; + y[iqs] = cvt((float) ((x[ib].qs[iqs] & 0xf) | xh_0) * dm.x + dm.y); + y[iqs + 16] = cvt((float) ((x[ib].qs[iqs] >> 4) | xh_1) * dm.x + dm.y); + } +} +template +__device__ void dq_q8_0(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_q8_0* x = (const block_q8_0*) vx + ibs * (QK_K / QK8_0); + const int ib = tid % 8, il = tid / 8; + const float d = __half2float(x[ib].d); + dst_t* y = yy + 32 * ib + 8 * il; + for (int j = 0; j < 8; ++j) y[j] = cvt((float) x[ib].qs[8 * il + j] * d); +} +// llama.cpp's dequantize_q6_K, its 64 threads folded onto 32 +template +__device__ void dq_q6_k(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_q6_K* x = (const block_q6_K*) vx; + for (int tt = tid; tt < 64; tt += 32) { + const int ip = tt / 32, il = tt - 32 * ip, is = 8 * ip + il / 16; + dst_t* y = yy + 128 * ip + il; + const float d = __half2float(x[ibs].d); + const uint8_t* ql = x[ibs].ql + 64 * ip + il; + const uint8_t qh = x[ibs].qh[32 * ip + il]; + const int8_t* sc = x[ibs].scales + is; + y[0] = cvt(d * sc[0] * ((int8_t) ((ql[0] & 0xF) | (((qh >> 0) & 3) << 4)) - 32)); + y[32] = cvt(d * sc[2] * ((int8_t) ((ql[32] & 0xF) | (((qh >> 2) & 3) << 4)) - 32)); + y[64] = cvt(d * sc[4] * ((int8_t) ((ql[0] >> 4) | (((qh >> 4) & 3) << 4)) - 32)); + y[96] = cvt(d * sc[6] * ((int8_t) ((ql[32] >> 4) | (((qh >> 6) & 3) << 4)) - 32)); + } +} +template +__device__ void dq_iq1_s(const void* vx, int64_t ibs, dst_t* yy, int tid) { + const block_iq1_s* x = (const block_iq1_s*) vx; + const int64_t il = tid / 8, ib = tid % 8; + dst_t* y = yy + 32 * ib + 8 * il; + const float delta = x[ibs].qh[ib] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA; + const float d = (float) x[ibs].d * (2 * ((x[ibs].qh[ib] >> 12) & 7) + 1); + uint32_t grid32[2]; + const int8_t* q = (const int8_t*) grid32; + grid32[0] = iq1s_grid_gpu[x[ibs].qs[4 * ib + il] | (((x[ibs].qh[ib] >> 3 * il) & 7) << 8)]; + grid32[1] = (grid32[0] >> 4) & 0x0f0f0f0f; + grid32[0] &= 0x0f0f0f0f; + for (int j = 0; j < 8; ++j) y[j] = cvt(d * (q[j] + delta)); +} template __device__ __forceinline__ void dq_dispatch(int ty, const void* vx, int64_t ibs, dst_t* y, int tid) { @@ -557,6 +912,12 @@ __device__ __forceinline__ void dq_dispatch(int ty, const void* vx, int64_t ibs, case 23: dq_iq4_xs(vx, ibs, y, tid); break; case 11: dq_q3_k(vx, ibs, y, tid); break; case 42: dq_q2_0(vx, ibs, y, tid); break; + case 12: dq_q4_k(vx, ibs, y, tid); break; + case 13: dq_q5_k(vx, ibs, y, tid); break; + case 7: dq_q5_1(vx, ibs, y, tid); break; + case 8: dq_q8_0(vx, ibs, y, tid); break; + case 14: dq_q6_k(vx, ibs, y, tid); break; + case 19: dq_iq1_s(vx, ibs, y, tid); break; default: break; } } @@ -576,7 +937,11 @@ __global__ void dequant_gu_kernel(int ty, const void* __restrict__ gate, const v dq_dispatch<__half>(ty, parity ? up : gate, i, y + ((2 * r + parity) * per_row + c) * QK_K, threadIdx.x); } -bool is_iq(int t) { return t == 16 || t == 17 || t == 18 || t == 20 || t == 21 || t == 22 || t == 23 || t == 29 || t == 42 || t == 11; } +// the formats dq_dispatch dequantizes +bool is_iq(int t) { + return t == 16 || t == 17 || t == 18 || t == 20 || t == 21 || t == 22 || t == 23 || t == 29 || t == 42 || t == 11 || + t == 12 || t == 13 || t == 14 || t == 19 || t == 7 || t == 8; +} } // namespace @@ -594,6 +959,12 @@ size_t iq_row_bytes(int t, int64_t n) noexcept { case 23: return (size_t) (n / 256) * sizeof(block_iq4_xs); case 11: return (size_t) (n / 256) * sizeof(block_q3_K); case 42: return (size_t) (n / 64) * sizeof(block_q2_0); + case 12: return (size_t) (n / 256) * sizeof(block_q4_K); + case 13: return (size_t) (n / 256) * sizeof(block_q5_K); + case 14: return (size_t) (n / 256) * sizeof(block_q6_K); + case 19: return (size_t) (n / 256) * sizeof(block_iq1_s); + case 7: return (size_t) (n / 32) * sizeof(block_q5_1); + case 8: return (size_t) (n / 32) * sizeof(block_q8_0); default: return 0; } } @@ -612,14 +983,9 @@ void iq_mmvq(int t, const void* w, const void* x_q8_1, float* y, int n_in, int n const auto* W = (const uint8_t*) w; const auto* X = (const block_q8_1*) x_q8_1; switch (t) { - case 16: mmvq_kernel<16><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 17: mmvq_kernel<17><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 18: mmvq_kernel<18><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 20: mmvq_kernel<20><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 21: mmvq_kernel<21><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 22: mmvq_kernel<22><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 29: mmvq_kernel<29><<>>(W, rb, X, y, n_in, n_out, ncols); break; - case 42: mmvq_kernel<42><<>>(W, rb, X, y, n_in, n_out, ncols); break; +#define STRATA_MMVQ(T) case T: mmvq_kernel<<>>(W, rb, X, y, n_in, n_out, ncols); break; + STRATA_FMTS(STRATA_MMVQ) +#undef STRATA_MMVQ default: std::fprintf(stderr, "iq_mmvq: type %d is not supported\n", t); std::exit(1); } check("iq_mmvq"); @@ -663,6 +1029,11 @@ void iq_dequant_gu_f16(int t, const void* gate, const void* up, int64_t n_ff, in check("iq_dequant_gu_f16"); } +bool native_expert_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept { + const int qg = fmt_qk(gu_type), qd = fmt_qk(d_type); + return qg > 0 && qd > 0 && is_iq(gu_type) && is_iq(d_type) && n_embd % qg == 0 && n_ff % qd == 0; +} + NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) { NativeExpertLayout L; L.gu_type = gu_type; @@ -695,13 +1066,9 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long const auto* X = (const block_q8_1*) x_q8_1; const dim3 ggu((unsigned) ((2 * L.n_ff + GU_ROWS - 1) / GU_ROWS), (unsigned) cap_groups); switch (L.gu_type) { - case 16: native_gu_kernel<16><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 17: native_gu_kernel<17><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 18: native_gu_kernel<18><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 21: native_gu_kernel<21><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 22: native_gu_kernel<22><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 29: native_gu_kernel<29><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; - case 42: native_gu_kernel<42><<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; +#define STRATA_GU(T) case T: native_gu_kernel<<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; + STRATA_FMTS(STRATA_GU) +#undef STRATA_GU default: std::fprintf(stderr, "native_expert_grouped: gate/up type %d\n", L.gu_type); std::exit(1); } check("native_expert_grouped/gu"); @@ -710,8 +1077,9 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long quantize_q8_1_kernel<<<(unsigned) ((nh + 255) / 256), 256, 0, s>>>(h, hq, nh); const dim3 gd((unsigned) ((L.n_embd + 7) / 8), (unsigned) cap_groups); switch (L.d_type) { - case 20: native_down_kernel<20><<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; - case 42: native_down_kernel<42><<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; +#define STRATA_DOWN(T) case T: native_down_kernel<<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; + STRATA_FMTS(STRATA_DOWN) +#undef STRATA_DOWN default: std::fprintf(stderr, "native_expert_grouped: down type %d\n", L.d_type); std::exit(1); } check("native_expert_grouped/down"); diff --git a/src/kernels/cuda/ple.cu b/src/kernels/cuda/ple.cu index 5a9174e0246..0e0e914f815 100644 --- a/src/kernels/cuda/ple.cu +++ b/src/kernels/cuda/ple.cu @@ -228,8 +228,8 @@ void ple_block(const float* emb, const float* hidden, const float* hist_rows, co PleOut& out, void* scratch, void* stream) { const bool native_key = w.key_native_data != nullptr && w.key_bf16 == nullptr; if (native_key && (!emb || !hidden || !hist_rows || !out.result || !scratch || !stream || - !w.key_native_q8_1 || w.key_native_type != 42)) - throw std::invalid_argument("ple_block: native key requires Q2_0 weights, input/output, private scratch and explicit stream"); + !w.key_native_q8_1 || !native_mmvq_supported(w.key_native_type))) + throw std::invalid_argument("ple_block: native key requires native MMVQ weights, input/output, private scratch and explicit stream"); if (emb == nullptr || hidden == nullptr || hist_rows == nullptr || out.result == nullptr) return; const int n_embd = NG_N_EMBD, hc = NG_HC, hc_dim = NG_HC_DIM; static_assert(NG_N_EMBD == 2560 && NG_HC_DIM == 10240, "native PLE key geometry changed"); diff --git a/src/kernels/native_expert_parity.cpp b/src/kernels/native_expert_parity.cpp index 9d37dfe3146..ed8dc262cc8 100644 --- a/src/kernels/native_expert_parity.cpp +++ b/src/kernels/native_expert_parity.cpp @@ -1,10 +1,12 @@ -// src/kernels/native_expert_parity.cpp - plan v0.3 P6: one native expert three ways, on real GGUF rows. +// src/kernels/native_expert_parity.cpp - plan v0.3 P6: one native expert three ways. // -// build/native_expert_parity [layer ...] +// build/native_expert_parity [layer ...] real rows (a split model's shards are found by name) +// build/native_expert_parity --synthetic GU/DOWN ... random weights quantized by ggml, e.g. iq1_s/iq4_nl // // (a) float reference: ggml's own dequantizer (`to_float`) and a float SwiGLU expert, (b) the CPU path // (ggml-cpu vec_dot with its quantized activations), (c) the GPU path (`native_expert_grouped`, q8_1 -// activations). (b) and (c) each differ from (a) by their activation rounding only (a few 1e-3 relative). +// activations). (b) and (c) each differ from (a) by their activation rounding only (about 1e-2 relative). +// (d) the GPU dequantizers of the prompt path and the embedding, against `to_float`: exact. #include "strata/artifact/gguf_reader.hpp" #include "strata/kernels/cpu/native_expert.hpp" #include "strata/kernels/cpu/expert.hpp" @@ -16,6 +18,7 @@ #include +#include #include #include #include @@ -26,178 +29,266 @@ namespace cpu = strata::kernels::cpu; -static double rel(const std::vector& a, const std::vector& b) { +namespace { + +constexpr int NT = 3; +constexpr int64_t H = 2560, FF = 640; + +double rel(const std::vector& a, const std::vector& b) { double n = 0, d = 0; for (size_t i = 0; i < a.size(); ++i) { n += std::fabs((double) a[i] - b[i]); d += std::fabs((double) b[i]); } return n / (d + 1e-30); } -int main(int argc, char** argv) { - if (argc < 2) { std::fprintf(stderr, "usage: native_expert_parity [layer ...]\n"); return 2; } - strata::GgufFile gguf(argv[1]); - std::vector layers; - for (int i = 2; i < argc; ++i) layers.push_back(std::atoi(argv[i])); - if (layers.empty()) layers = {0, 1, 2, 3, 20, 47}; - const int NT = 3, E = 7; - const int64_t H = 2560, FF = 640; +double us_since(std::chrono::steady_clock::time_point t0, int it) { + return std::chrono::duration(std::chrono::steady_clock::now() - t0).count() / it; +} + +// The three ways and the dequantizers for one expert blob; returns the number of failed checks. +int check_blob(const cpu::NativeFmt& f, const std::vector& blob, int seed, const std::string& label, + cudaStream_t s) { int failures = 0; - cudaStream_t s; - cudaStreamCreate(&s); - for (int l : layers) { - const strata::TensorInfo* t[3] = {}; - const char* roles[3] = {"gate", "up", "down"}; - for (const auto& ti : gguf.tensors()) - for (int r = 0; r < 3; ++r) - if (ti.name == "blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight") t[r] = &ti; - if (!t[0] || !t[1] || !t[2]) { std::printf("layer %d: no expert tensors\n", l); ++failures; continue; } - cpu::NativeFmt f; - std::string err; - if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, H, FF, f, err)) { - std::printf("layer %d: %s\n", l, err.c_str()); ++failures; continue; - } - std::vector blob(f.bytes); - std::memcpy(blob.data(), gguf.tensor_data(*t[0]) + (size_t) E * f.up_off, f.up_off); - std::memcpy(blob.data() + f.up_off, gguf.tensor_data(*t[1]) + (size_t) E * f.up_off, f.up_off); - std::memcpy(blob.data() + f.down_off, gguf.tensor_data(*t[2]) + (size_t) E * (f.bytes - f.down_off), - f.bytes - f.down_off); - // (a) the float reference - const auto* tg = ggml_get_type_traits((ggml_type) f.gu_type); - const auto* td = ggml_get_type_traits((ggml_type) f.d_type); - std::vector G((size_t) FF * H), U((size_t) FF * H), D((size_t) H * FF); + // (a) the float reference + const auto* tg = ggml_get_type_traits((ggml_type) f.gu_type); + const auto* td = ggml_get_type_traits((ggml_type) f.d_type); + std::vector G((size_t) FF * H), U((size_t) FF * H), D((size_t) H * FF); + for (int64_t r = 0; r < FF; ++r) { + tg->to_float(blob.data() + r * f.gu_row, G.data() + r * H, H); + tg->to_float(blob.data() + f.up_off + r * f.gu_row, U.data() + r * H, H); + } + for (int64_t r = 0; r < H; ++r) td->to_float(blob.data() + f.down_off + r * f.d_row, D.data() + r * FF, FF); + std::mt19937 rng(11 + seed); + std::normal_distribution nd(0.f, 1.f); + std::vector x((size_t) NT * H); + for (auto& v : x) v = nd(rng); + std::vector ref((size_t) NT * H), got_c((size_t) NT * H), got_g((size_t) NT * H); + for (int k = 0; k < NT; ++k) { + std::vector h(FF); for (int64_t r = 0; r < FF; ++r) { - tg->to_float(blob.data() + r * f.gu_row, G.data() + r * H, H); - tg->to_float(blob.data() + f.up_off + r * f.gu_row, U.data() + r * H, H); + double g = 0, u = 0; + for (int64_t i = 0; i < H; ++i) { g += (double) G[r * H + i] * x[k * H + i]; u += (double) U[r * H + i] * x[k * H + i]; } + h[r] = (float) (g / (1.0 + std::exp(-g)) * u); } - for (int64_t r = 0; r < H; ++r) td->to_float(blob.data() + f.down_off + r * f.d_row, D.data() + r * FF, FF); - std::mt19937 rng(11 + l); - std::normal_distribution nd(0.f, 1.f); - std::vector x((size_t) NT * H); - for (auto& v : x) v = nd(rng); - std::vector ref((size_t) NT * H), got_c((size_t) NT * H), got_g((size_t) NT * H); + for (int64_t r = 0; r < H; ++r) { + double o = 0; + for (int64_t i = 0; i < FF; ++i) o += (double) D[r * FF + i] * h[i]; + ref[k * H + r] = (float) o; + } + } + // (b) the CPU + { + std::vector> act(NT, std::vector(cpu::kNativeActBytes)); + std::vector> hq(NT, std::vector(cpu::kNativeHBytes)); + std::vector> ff(NT, std::vector(FF)); + const void* a[NT]; + float* ffp[NT]; + const void* hp[NT]; + float* op[NT]; for (int k = 0; k < NT; ++k) { - std::vector h(FF); - for (int64_t r = 0; r < FF; ++r) { - double g = 0, u = 0; - for (int64_t i = 0; i < H; ++i) { g += (double) G[r * H + i] * x[k * H + i]; u += (double) U[r * H + i] * x[k * H + i]; } - h[r] = (float) (g / (1.0 + std::exp(-g)) * u); - } - for (int64_t r = 0; r < H; ++r) { - double o = 0; - for (int64_t i = 0; i < FF; ++i) o += (double) D[r * FF + i] * h[i]; - ref[k * H + r] = (float) o; - } + cpu::native_quant_act(f, x.data() + k * H, act[k].data()); + a[k] = act[k].data(); + ffp[k] = ff[k].data(); } - // (b) the CPU - { - std::vector> act(NT, std::vector(cpu::kNativeActBytes)); - std::vector> hq(NT, std::vector(cpu::kNativeHBytes)); - std::vector> ff(NT, std::vector(FF)); - const void* a[NT]; - float* ffp[NT]; - const void* hp[NT]; - float* op[NT]; - for (int k = 0; k < NT; ++k) { - cpu::native_quant_act(f, x.data() + k * H, act[k].data()); - a[k] = act[k].data(); - ffp[k] = ff[k].data(); - } - cpu::native_gu_rows(f, blob.data(), a, NT, ffp, 0, (int) FF); - if (cpu::iq512_supported(f.gu_type)) { - // the AVX-512 rows against ggml's own vec_dot, same Q8_K activations: float-order differences only - std::vector g512((size_t) NT * FF), gref((size_t) NT * FF); - float* gp[NT]; - for (int k = 0; k < NT; ++k) gp[k] = g512.data() + k * FF; - cpu::iq512_rows(f.gu_type, blob.data(), f.gu_row, (int) H, a, NT, gp, 0, (int) FF); - const auto* tc = ggml_get_type_traits_cpu((ggml_type) f.gu_type); - for (int k = 0; k < NT; ++k) - for (int64_t r = 0; r < FF; ++r) - tc->vec_dot((int) H, &gref[k * FF + r], 0, blob.data() + r * f.gu_row, 0, a[k], 0, 1); - { // single-thread throughput, the weights cache-resident (compute bound) - float* f1[1] = {ffp[0]}; - const int it = 50; - auto t0 = std::chrono::steady_clock::now(); - for (int i = 0; i < it; ++i) cpu::iq512_gu_rows(f.gu_type, blob.data(), f.gu_row, f.up_off, (int) H, a, 1, f1, 0, (int) FF); - auto t1 = std::chrono::steady_clock::now(); - for (int i = 0; i < it; ++i) - for (int64_t r = 0; r < FF; ++r) { - float gg, uu; - tc->vec_dot((int) H, &gg, 0, blob.data() + r * f.gu_row, 0, a[0], 0, 1); - tc->vec_dot((int) H, &uu, 0, blob.data() + f.up_off + r * f.gu_row, 0, a[0], 0, 1); - } - auto t2 = std::chrono::steady_clock::now(); - const double us512 = std::chrono::duration(t1 - t0).count() / it; - const double usg = std::chrono::duration(t2 - t1).count() / it; - std::printf(" gate+up one token one thread: avx512 %.0f us (%.2f GB/s), ggml %.0f us (%.2f GB/s)\n", - us512, 2.0 * f.up_off / us512 / 1e3, usg, 2.0 * f.up_off / usg / 1e3); - auto t3 = std::chrono::steady_clock::now(); - for (int i = 0; i < it; ++i) cpu::iq512_gu_rows(f.gu_type, blob.data(), f.gu_row, f.up_off, (int) H, a, NT, ffp, 0, (int) FF); - auto t4 = std::chrono::steady_clock::now(); - std::printf(" gate+up %d tokens one thread: avx512 %.0f us vs ggml %d x %.0f us\n", NT, - std::chrono::duration(t4 - t3).count() / it, NT, usg); + cpu::native_gu_rows(f, blob.data(), a, NT, ffp, 0, (int) FF); + const auto* tc = ggml_get_type_traits_cpu((ggml_type) f.gu_type); + const int it = 50; + if (cpu::iq512_supported(f.gu_type)) { + // the AVX-512 rows against ggml's own vec_dot, same Q8_K activations: float-order differences only + std::vector g512((size_t) NT * FF), gref((size_t) NT * FF); + float* gp[NT]; + for (int k = 0; k < NT; ++k) gp[k] = g512.data() + k * FF; + cpu::iq512_rows(f.gu_type, blob.data(), f.gu_row, (int) H, a, NT, gp, 0, (int) FF); + for (int k = 0; k < NT; ++k) + for (int64_t r = 0; r < FF; ++r) + tc->vec_dot((int) H, &gref[k * FF + r], 0, blob.data() + r * f.gu_row, 0, a[k], 0, 1); + std::printf(" %s AVX-512 gate rows vs ggml vec_dot: rel %.2e\n", + ggml_type_name((ggml_type) f.gu_type), rel(g512, gref)); + float* f1[1] = {ffp[0]}; + auto t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < it; ++i) cpu::iq512_gu_rows(f.gu_type, blob.data(), f.gu_row, f.up_off, (int) H, a, 1, f1, 0, (int) FF); + const double us1 = us_since(t0, it); + t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < it; ++i) cpu::iq512_gu_rows(f.gu_type, blob.data(), f.gu_row, f.up_off, (int) H, a, NT, ffp, 0, (int) FF); + std::printf(" gate+up one thread, weights cache-resident: avx512 %.0f us one token (%.2f GB/s), " + "%.0f us %d tokens\n", us1, 2.0 * f.up_off / us1 / 1e3, us_since(t0, it), NT); + } + { // ggml-cpu's own rows, one thread (what the pool runs for formats without an AVX-512 kernel) + auto t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < it; ++i) + for (int64_t r = 0; r < FF; ++r) { + float gg, uu; + tc->vec_dot((int) H, &gg, 0, blob.data() + r * f.gu_row, 0, a[0], 0, 1); + tc->vec_dot((int) H, &uu, 0, blob.data() + f.up_off + r * f.gu_row, 0, a[0], 0, 1); } - std::printf(" %s AVX-512 gate rows vs ggml vec_dot: rel %.2e\n", - ggml_type_name((ggml_type) f.gu_type), rel(g512, gref)); - } + const double usg = us_since(t0, it); + std::printf(" gate+up one thread, weights cache-resident: ggml %.0f us one token (%.2f GB/s)\n", usg, + 2.0 * f.up_off / usg / 1e3); + } + for (int k = 0; k < NT; ++k) { + cpu::native_quant_h(f, ff[k].data(), hq[k].data()); + hp[k] = hq[k].data(); + op[k] = got_c.data() + k * H; + } + cpu::native_down_rows(f, blob.data(), hp, NT, op, 0, (int) H); + if (f.d_type == 42) { + // (b2) the AVX-512 GGUF-layout Q2_0 kernel the pool uses for Q2_0 down projections + std::vector a2(NT); + const cpu::ActQ* ap[NT]; + std::vector alt((size_t) NT * H); + float* altp[NT]; for (int k = 0; k < NT; ++k) { - cpu::native_quant_h(f, ff[k].data(), hq[k].data()); - hp[k] = hq[k].data(); - op[k] = got_c.data() + k * H; - } - cpu::native_down_rows(f, blob.data(), hp, NT, op, 0, (int) H); - if (f.d_type == 42) { - // (b2) the AVX-512 GGUF-layout Q2_0 kernel the pool uses for Q2_0 down projections - std::vector a2(NT); - const cpu::ActQ* ap[NT]; - std::vector alt((size_t) NT * H); - float* altp[NT]; - for (int k = 0; k < NT; ++k) { - cpu::act_quant_q8_1(ff[k].data(), (int) FF, a2[k]); - ap[k] = &a2[k]; - altp[k] = alt.data() + k * H; - } - cpu::q2_0_gguf_rows_multi(blob.data() + f.down_off, f.d_row, (int) (FF / 64), ap, NT, altp, 0, (int) H); - std::printf(" q2_0 AVX-512 down vs ggml down: rel %.2e\n", rel(alt, got_c)); + cpu::act_quant_q8_1(ff[k].data(), (int) FF, a2[k]); + ap[k] = &a2[k]; + altp[k] = alt.data() + k * H; } + cpu::q2_0_gguf_rows_multi(blob.data() + f.down_off, f.d_row, (int) (FF / 64), ap, NT, altp, 0, (int) H); + std::printf(" q2_0 AVX-512 down vs ggml down: rel %.2e\n", rel(alt, got_c)); } - // (c) the GPU: one group holding the NT entries - { - const auto L = strata::kernels::native_expert_layout(f.gu_type, f.d_type, H, FF); - void *dblob, *dx, *dxq, *dscr; - float* dout; - unsigned long long* dptr; - int32_t *dstart, *dn, *ddst, *dtok; - cudaMalloc(&dblob, blob.size()); - cudaMalloc(&dx, x.size() * 4); - cudaMalloc(&dxq, (size_t) NT * H / 32 * 36); - cudaMalloc(&dscr, strata::kernels::native_expert_scratch_bytes(NT, FF)); - cudaMalloc((void**) &dout, (size_t) NT * H * 4); - cudaMalloc((void**) &dptr, 8); - cudaMalloc((void**) &dstart, 8); - cudaMalloc((void**) &dn, 4); - cudaMalloc((void**) &ddst, NT * 4); - cudaMalloc((void**) &dtok, NT * 4); - cudaMemcpy(dblob, blob.data(), blob.size(), cudaMemcpyHostToDevice); - cudaMemcpy(dx, x.data(), x.size() * 4, cudaMemcpyHostToDevice); - const unsigned long long p = (unsigned long long) dblob; - const int32_t st[2] = {0, NT}, one = 1, idx[NT] = {0, 1, 2}; - cudaMemcpy(dptr, &p, 8, cudaMemcpyHostToDevice); - cudaMemcpy(dstart, st, 8, cudaMemcpyHostToDevice); - cudaMemcpy(dn, &one, 4, cudaMemcpyHostToDevice); - cudaMemcpy(ddst, idx, NT * 4, cudaMemcpyHostToDevice); - cudaMemcpy(dtok, idx, NT * 4, cudaMemcpyHostToDevice); - strata::kernels::quantize_q8_1_rows((const float*) dx, NT, H, dxq, s); - strata::kernels::native_expert_grouped(L, dptr, dstart, dn, ddst, dtok, 1, NT, dxq, dscr, dout, s); + } + // (c) the GPU: one group holding the NT entries; (d) its dequantizers + { + const auto L = strata::kernels::native_expert_layout(f.gu_type, f.d_type, H, FF); + void *dblob, *dx, *dxq, *dscr; + float* dout; + unsigned long long* dptr; + int32_t *dstart, *dn, *ddst, *dtok; + cudaMalloc(&dblob, blob.size()); + cudaMalloc(&dx, x.size() * 4); + cudaMalloc(&dxq, (size_t) NT * H / 32 * 36); + cudaMalloc(&dscr, strata::kernels::native_expert_scratch_bytes(NT, FF)); + cudaMalloc((void**) &dout, (size_t) NT * H * 4); + cudaMalloc((void**) &dptr, 8); + cudaMalloc((void**) &dstart, 8); + cudaMalloc((void**) &dn, 4); + cudaMalloc((void**) &ddst, NT * 4); + cudaMalloc((void**) &dtok, NT * 4); + cudaMemcpy(dblob, blob.data(), blob.size(), cudaMemcpyHostToDevice); + cudaMemcpy(dx, x.data(), x.size() * 4, cudaMemcpyHostToDevice); + const unsigned long long p = (unsigned long long) dblob; + const int32_t st[2] = {0, NT}, one = 1, idx[NT] = {0, 1, 2}; + cudaMemcpy(dptr, &p, 8, cudaMemcpyHostToDevice); + cudaMemcpy(dstart, st, 8, cudaMemcpyHostToDevice); + cudaMemcpy(dn, &one, 4, cudaMemcpyHostToDevice); + cudaMemcpy(ddst, idx, NT * 4, cudaMemcpyHostToDevice); + cudaMemcpy(dtok, idx, NT * 4, cudaMemcpyHostToDevice); + strata::kernels::quantize_q8_1_rows((const float*) dx, NT, H, dxq, s); + strata::kernels::native_expert_grouped(L, dptr, dstart, dn, ddst, dtok, 1, NT, dxq, dscr, dout, s); + cudaStreamSynchronize(s); + cudaMemcpy(got_g.data(), dout, got_g.size() * 4, cudaMemcpyDeviceToHost); + float* dq = nullptr; + cudaMalloc((void**) &dq, (size_t) FF * H * 4); + std::vector got_dq((size_t) FF * H); + double dq_err = 0.0; + const struct { const uint8_t* src; int type; const std::vector* want; } mats[3] = { + {(const uint8_t*) dblob, f.gu_type, &G}, {(const uint8_t*) dblob + f.up_off, f.gu_type, &U}, + {(const uint8_t*) dblob + f.down_off, f.d_type, &D}}; + for (const auto& m : mats) { + strata::kernels::iq_dequant_f32(m.type, m.src, FF * H, dq, s); cudaStreamSynchronize(s); - cudaMemcpy(got_g.data(), dout, got_g.size() * 4, cudaMemcpyDeviceToHost); - cudaFree(dblob); cudaFree(dx); cudaFree(dxq); cudaFree(dscr); cudaFree(dout); cudaFree(dptr); - cudaFree(dstart); cudaFree(dn); cudaFree(ddst); cudaFree(dtok); + cudaMemcpy(got_dq.data(), dq, got_dq.size() * 4, cudaMemcpyDeviceToHost); + dq_err = (std::max)(dq_err, rel(got_dq, *m.want)); + } + std::printf(" GPU dequant vs ggml to_float: rel %.2e\n", dq_err); + if (!(dq_err < 1e-6)) ++failures; + cudaFree(dq); + cudaFree(dblob); cudaFree(dx); cudaFree(dxq); cudaFree(dscr); cudaFree(dout); cudaFree(dptr); + cudaFree(dstart); cudaFree(dn); cudaFree(ddst); cudaFree(dtok); + } + const double ec = rel(got_c, ref), eg = rel(got_g, ref), ecg = rel(got_c, got_g); + const bool ok = ec < 3e-2 && eg < 3e-2 && std::isfinite(ec) && std::isfinite(eg); + std::printf("%-9s %-8s/%-7s blob %8zu cpu rel %.2e gpu rel %.2e cpu-gpu %.2e %s\n", label.c_str(), + ggml_type_name((ggml_type) f.gu_type), ggml_type_name((ggml_type) f.d_type), f.bytes, ec, eg, ecg, + ok ? "ok" : "FAIL"); + return failures + (ok ? 0 : 1); +} + +int type_of(std::string name) { + for (auto& c : name) c = (char) std::tolower((unsigned char) c); + for (int t = 0; t < GGML_TYPE_COUNT; ++t) { + const char* n = ggml_type_name((ggml_type) t); + if (n == nullptr) continue; + std::string m(n); + for (auto& c : m) c = (char) std::tolower((unsigned char) c); + if (m == name) return t; + } + return -1; +} + +// Random weights (rows of different magnitude) quantized by ggml into the blob layout. +std::vector synthetic_blob(const cpu::NativeFmt& f, int seed) { + std::vector blob(f.bytes); + std::mt19937 rng(seed); + std::normal_distribution nd(0.f, 1.f); + auto quant = [&](int type, int64_t rows, int64_t cols, uint8_t* dst) { + std::vector w((size_t) (rows * cols)); + for (int64_t r = 0; r < rows; ++r) { + const float scale = 0.02f * (0.5f + (float) (r % 7) / 7.0f); + for (int64_t c = 0; c < cols; ++c) w[(size_t) (r * cols + c)] = scale * nd(rng); + } + const std::vector imatrix((size_t) cols, 1.0f); + ggml_quantize_chunk((ggml_type) type, w.data(), dst, 0, rows, cols, + ggml_quantize_requires_imatrix((ggml_type) type) ? imatrix.data() : nullptr); + }; + quant(f.gu_type, FF, H, blob.data()); + quant(f.gu_type, FF, H, blob.data() + f.up_off); + quant(f.d_type, H, FF, blob.data() + f.down_off); + return blob; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc < 2) { + std::fprintf(stderr, "usage: native_expert_parity [layer ...]\n" + " native_expert_parity --synthetic GU/DOWN ... (ggml type names, e.g. q6_K/q8_0)\n"); + return 2; + } + int failures = 0; + cudaStream_t s; + cudaStreamCreate(&s); + if (std::string(argv[1]) == "--synthetic") { + for (int i = 2; i < argc; ++i) { + const std::string arg = argv[i]; + const size_t slash = arg.find('/'); + const int gu = slash == std::string::npos ? -1 : type_of(arg.substr(0, slash)); + const int dn = slash == std::string::npos ? -1 : type_of(arg.substr(slash + 1)); + cpu::NativeFmt f; + std::string err; + if (gu < 0 || dn < 0 || !cpu::native_fmt(gu, dn, H, FF, f, err) || + !strata::kernels::native_expert_supported(gu, dn, H, FF)) { + std::printf("%s: %s\n", arg.c_str(), err.empty() ? "not a supported pair of GGML types" : err.c_str()); + ++failures; + continue; + } + failures += check_blob(f, synthetic_blob(f, i), i, "synthetic", s); + } + } else { + const strata::GgufModel model(strata::gguf_split_paths(argv[1])); + std::vector layers; + for (int i = 2; i < argc; ++i) layers.push_back(std::atoi(argv[i])); + if (layers.empty()) layers = {0, 1, 2, 3, 20, 47}; + const int E = 7; + for (int l : layers) { + const strata::TensorInfo* t[3] = {}; + const uint8_t* data[3] = {}; + const char* roles[3] = {"gate", "up", "down"}; + for (int r = 0; r < 3; ++r) { + size_t at = 0; + t[r] = model.find("blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight", &at); + if (t[r]) data[r] = model.shard(at).tensor_data(*t[r]); + } + if (!t[0] || !t[1] || !t[2]) { std::printf("layer %d: no expert tensors\n", l); ++failures; continue; } + cpu::NativeFmt f; + std::string err; + if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, H, FF, f, err)) { + std::printf("layer %d: %s\n", l, err.c_str()); ++failures; continue; + } + std::vector blob(f.bytes); + std::memcpy(blob.data(), data[0] + (size_t) E * f.up_off, f.up_off); + std::memcpy(blob.data() + f.up_off, data[1] + (size_t) E * f.up_off, f.up_off); + std::memcpy(blob.data() + f.down_off, data[2] + (size_t) E * (f.bytes - f.down_off), f.bytes - f.down_off); + failures += check_blob(f, blob, l, "layer " + std::to_string(l), s); } - const double ec = rel(got_c, ref), eg = rel(got_g, ref), ecg = rel(got_c, got_g); - const bool ok = ec < 3e-2 && eg < 3e-2 && std::isfinite(ec) && std::isfinite(eg); - std::printf("layer %2d %-8s/%-7s blob %8zu cpu rel %.2e gpu rel %.2e cpu-gpu %.2e %s\n", l, - ggml_type_name((ggml_type) f.gu_type), ggml_type_name((ggml_type) f.d_type), f.bytes, ec, eg, ecg, - ok ? "ok" : "FAIL"); - if (!ok) ++failures; } std::printf("native_expert_parity: %d failures\n", failures); return failures ? 1 : 0; diff --git a/src/kernels/ngram.cpp b/src/kernels/ngram.cpp index 705ced66488..582940ae5bf 100644 --- a/src/kernels/ngram.cpp +++ b/src/kernels/ngram.cpp @@ -1,4 +1,4 @@ -// src/kernels/ngram.cpp - P2.S4: the PLE n-gram hash and the IQ4_NL table read. +// src/kernels/ngram.cpp - P2.S4: the PLE n-gram hash and the table read (IQ4_NL or Q8_0). // // See include/strata/kernels/ngram.hpp for the semantics, the rival readings, and the note on MADV_RANDOM. #include "strata/kernels/ngram.hpp" @@ -107,11 +107,37 @@ void iq4nl_dequant_row(const uint8_t* row, float* out160) { } } +void ple_dequant_row(int type, const uint8_t* row, float* out160) { + if (type != 8) { + iq4nl_dequant_row(row, out160); + return; + } + for (int b = 0; b < PLE_HEAD_DIM / 32; ++b) { // Q8_0: fp16 scale + 32 int8 + const uint8_t* blk = row + (size_t) b * 34; + uint16_t dbits; + std::memcpy(&dbits, blk, 2); + const float d = f32_from_f16(dbits); + for (int j = 0; j < 32; ++j) out160[b * 32 + j] = d * (float) (int8_t) blk[2 + j]; + } +} + +std::string ple_table_shard(const std::vector& shards) { + for (const auto& path : shards) { + try { + if (GgufFile(path).find("per_layer_token_embd.weight") != nullptr) return path; + } catch (const std::exception&) { + } + } + return {}; +} + // --------------------------------------------------------------------------------------------------- struct PleTable::Impl { GgufFile* file = nullptr; const uint8_t* data = nullptr; uint64_t n_rows = 0; + int type = 20; // IQ4_NL or Q8_0 + uint32_t row_bytes = PLE_ROW_BYTES; mutable uint64_t bytes_read = 0; // Direct mode (plan v0.3 P2): the mapping above is released after the header parse and every row comes // from an unbuffered SSD read into `raw`. @@ -120,7 +146,7 @@ struct PleTable::Impl { strata::ngram::PleReader::Ticket ticket; bool pending = false; uint32_t rows[PLE_N_HEADS] = {}; - uint8_t raw[PLE_N_HEADS * PLE_ROW_BYTES] = {}; + uint8_t raw[PLE_N_HEADS * PLE_ROW_BYTES_Q8_0] = {}; }; PleTable::PleTable() : impl_(new Impl) {} @@ -150,11 +176,13 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO close(); return false; } - if (std::strcmp(t->type_name(), "IQ4_NL") != 0) { - err = std::string("per_layer_token_embd.weight is ") + t->type_name() + ", not IQ4_NL"; + if (t->type != 20 && t->type != 8) { + err = std::string("per_layer_token_embd.weight is ") + t->type_name() + ", not IQ4_NL or Q8_0"; close(); return false; } + impl_->type = (int) t->type; + impl_->row_bytes = t->type == 8 ? PLE_ROW_BYTES_Q8_0 : PLE_ROW_BYTES; impl_->n_rows = t->shape[1]; impl_->data = impl_->file->tensor_data(*t); @@ -162,16 +190,21 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO // the offset within the GGUF's DATA SECTION: the file's first 192 bytes are a header, and reading at 0 // would decode the header plus 192 bytes of shifted rows - still plausible IQ4_NL, and wrong for every // row. `GgufFile` parses the header, so `tensor_data` is already correct; this asserts the tensor - // exactly fills the file from there, which is what makes the whole arrangement checkable rather than - // assumed. A wrong data offset would leave a different remainder. - const uint64_t need = impl_->n_rows * (uint64_t) PLE_ROW_BYTES; - const uint64_t have = impl_->file->file_size() - impl_->file->data_start(); - if (need != have) { + // exactly fills its span - up to the next tensor or the end of the file, less the alignment padding - + // which is what makes the whole arrangement checkable rather than assumed. A wrong data offset would + // leave a different remainder. (The table fills the ISTA shard 2 alone; the unsloth shard 2 holds other + // tensors too.) + const uint64_t need = impl_->n_rows * (uint64_t) impl_->row_bytes; + uint64_t end = impl_->file->file_size() - impl_->file->data_start(); + for (const auto& o : impl_->file->tensors()) + if (o.offset > t->offset && o.offset < end) end = o.offset; + const uint64_t have = end - t->offset; + if (need > have || have - need >= impl_->file->alignment()) { char buf[256]; std::snprintf(buf, sizeof buf, - "PLE table size mismatch: %llu rows x %d B = %llu, but the file holds %llu from " - "data_start %llu", - (unsigned long long) impl_->n_rows, PLE_ROW_BYTES, (unsigned long long) need, + "PLE table size mismatch: %llu rows x %d B = %llu, but its span in the file is %llu B " + "(data_start %llu)", + (unsigned long long) impl_->n_rows, (int) impl_->row_bytes, (unsigned long long) need, (unsigned long long) have, (unsigned long long) impl_->file->data_start()); err = buf; close(); @@ -185,7 +218,8 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO delete impl_->file; impl_->file = nullptr; impl_->data = nullptr; - if (!impl_->reader.open(gguf_path, table_offset, n_rows, io.max_inflight, io.cache_rows, err, io.io_thread)) { + if (!impl_->reader.open(gguf_path, table_offset, n_rows, io.max_inflight, io.cache_rows, err, io.io_thread, + impl_->row_bytes)) { close(); return false; } @@ -212,23 +246,23 @@ uint64_t PleTable::bytes_read() const { return impl_->bytes_read; } void PleTable::read_row(uint32_t row, float* out160) const { if (impl_->mode == PleIo::Direct && impl_->reader.is_open()) { - uint8_t raw[PLE_ROW_BYTES]; + uint8_t raw[PLE_ROW_BYTES_Q8_0]; std::string err; const auto t = impl_->reader.issue(&row, 1, raw); if (!impl_->reader.collect(t, err)) { std::memset(out160, 0, (size_t) PLE_HEAD_DIM * sizeof(float)); return; } - iq4nl_dequant_row(raw, out160); - impl_->bytes_read += PLE_ROW_BYTES; + ple_dequant_row(impl_->type, raw, out160); + impl_->bytes_read += impl_->row_bytes; return; } if (impl_->data == nullptr || row >= impl_->n_rows) { std::memset(out160, 0, (size_t) PLE_HEAD_DIM * sizeof(float)); return; } - iq4nl_dequant_row(impl_->data + (size_t) row * PLE_ROW_BYTES, out160); - impl_->bytes_read += PLE_ROW_BYTES; + ple_dequant_row(impl_->type, impl_->data + (size_t) row * impl_->row_bytes, out160); + impl_->bytes_read += impl_->row_bytes; } bool PleTable::issue(const uint32_t* rows16) { @@ -245,8 +279,8 @@ bool PleTable::issue(const uint32_t* rows16) { ULONG_PTR n = 0; for (int h = 0; h < PLE_N_HEADS; ++h) { if (rows16[h] >= impl_->n_rows) continue; - ranges[n].VirtualAddress = (PVOID) (impl_->data + (size_t) rows16[h] * PLE_ROW_BYTES); - ranges[n].NumberOfBytes = PLE_ROW_BYTES; + ranges[n].VirtualAddress = (PVOID) (impl_->data + (size_t) rows16[h] * impl_->row_bytes); + ranges[n].NumberOfBytes = impl_->row_bytes; ++n; } if (n > 0) (void) PrefetchVirtualMemory(GetCurrentProcess(), n, ranges, 0); @@ -262,8 +296,8 @@ bool PleTable::collect(float* out2560, std::string& err) { if (impl_->mode == PleIo::Direct) { if (!impl_->reader.collect(impl_->ticket, err)) return false; for (int h = 0; h < PLE_N_HEADS; ++h) - iq4nl_dequant_row(impl_->raw + (size_t) h * PLE_ROW_BYTES, out2560 + (size_t) h * PLE_HEAD_DIM); - impl_->bytes_read += (uint64_t) PLE_N_HEADS * PLE_ROW_BYTES; + ple_dequant_row(impl_->type, impl_->raw + (size_t) h * impl_->row_bytes, out2560 + (size_t) h * PLE_HEAD_DIM); + impl_->bytes_read += (uint64_t) PLE_N_HEADS * impl_->row_bytes; return true; } for (int h = 0; h < PLE_N_HEADS; ++h) read_row(impl_->rows[h], out2560 + (size_t) h * PLE_HEAD_DIM); @@ -274,11 +308,12 @@ bool PleTable::gather_batch(const uint32_t* rows, size_t n_tokens, float* out, s if (impl_->pending) { err = "PleTable::gather_batch while a token is in flight"; return false; } const size_t n = n_tokens * (size_t) PLE_N_HEADS; if (impl_->mode == PleIo::Direct) { - std::vector raw(n * PLE_ROW_BYTES); + std::vector raw(n * impl_->row_bytes); const auto ticket = impl_->reader.issue(rows, n, raw.data()); if (!impl_->reader.collect(ticket, err)) return false; - for (size_t i = 0; i < n; ++i) iq4nl_dequant_row(raw.data() + i * PLE_ROW_BYTES, out + i * PLE_HEAD_DIM); - impl_->bytes_read += (uint64_t) n * PLE_ROW_BYTES; + for (size_t i = 0; i < n; ++i) + ple_dequant_row(impl_->type, raw.data() + i * impl_->row_bytes, out + i * PLE_HEAD_DIM); + impl_->bytes_read += (uint64_t) n * impl_->row_bytes; return true; } for (size_t i = 0; i < n; ++i) read_row(rows[i], out + i * PLE_HEAD_DIM); @@ -332,8 +367,8 @@ void PleTable::gather(const uint32_t* rows16, float* out2560) const { for (int h = 0; h < PLE_N_HEADS; ++h) { // Out-of-range rows are handled by `read_row` as zeros and have no address to prefetch. if (rows16[h] >= impl_->n_rows) continue; - ranges[n].VirtualAddress = (PVOID) (impl_->data + (size_t) rows16[h] * PLE_ROW_BYTES); - ranges[n].NumberOfBytes = PLE_ROW_BYTES; + ranges[n].VirtualAddress = (PVOID) (impl_->data + (size_t) rows16[h] * impl_->row_bytes); + ranges[n].NumberOfBytes = impl_->row_bytes; ++n; } if (n > 0) (void) PrefetchVirtualMemory(GetCurrentProcess(), n, ranges, 0); diff --git a/src/ngram/ple_reader.cpp b/src/ngram/ple_reader.cpp index 44acd6d4855..3c989ca1a1e 100644 --- a/src/ngram/ple_reader.cpp +++ b/src/ngram/ple_reader.cpp @@ -33,14 +33,16 @@ constexpr uint32_t EMPTY = 0xFFFFFFFFu; struct RowCache { uint64_t sets = 0; std::vector keys; // sets * WAYS - std::vector data; // sets * WAYS * ROW_BYTES + std::vector data; // sets * WAYS * row_bytes std::vector next; // per-set replacement pointer uint64_t used = 0; + uint32_t row_bytes = ROW_BYTES; - void init(uint64_t rows) { + void init(uint64_t rows, uint32_t bytes) { sets = rows / WAYS; + row_bytes = bytes; keys.assign(sets * WAYS, EMPTY); - data.assign(sets * WAYS * ROW_BYTES, 0); + data.assign(sets * WAYS * row_bytes, 0); next.assign(sets, 0); used = 0; } @@ -52,7 +54,7 @@ struct RowCache { if (sets == 0) return nullptr; const uint64_t s = mix(row) % sets; for (uint32_t w = 0; w < WAYS; ++w) - if (keys[s * WAYS + w] == row) return &data[(s * WAYS + w) * ROW_BYTES]; + if (keys[s * WAYS + w] == row) return &data[(s * WAYS + w) * row_bytes]; return nullptr; } void insert(uint32_t row, const uint8_t* bytes) { @@ -62,7 +64,7 @@ struct RowCache { next[s] = (uint8_t) ((w + 1) % WAYS); if (keys[s * WAYS + w] == EMPTY) ++used; keys[s * WAYS + w] = row; - std::memcpy(&data[(s * WAYS + w) * ROW_BYTES], bytes, ROW_BYTES); + std::memcpy(&data[(s * WAYS + w) * row_bytes], bytes, row_bytes); } }; @@ -94,6 +96,7 @@ struct PleReader::Impl { DirectFile file; uint64_t table_offset = 0; uint64_t n_rows = 0; + uint32_t row_bytes = ROW_BYTES; uint32_t max_inflight = 0; uint8_t* slab = nullptr; // max_inflight slots of 2 pages std::vector free_slots; @@ -150,11 +153,11 @@ struct PleReader::Impl { stats.bytes += c.bytes; const uint8_t* buf = slot_buf(s); for (const Use& u : j.uses) { - if (u.in_page + ROW_BYTES > c.bytes) { + if (u.in_page + row_bytes > c.bytes) { error = "PleReader: short read inside the table"; return false; } - std::memcpy(u.dst, buf + u.in_page, ROW_BYTES); + std::memcpy(u.dst, buf + u.in_page, row_bytes); cache.insert(u.row, buf + u.in_page); } auto it = tickets.find(j.ticket); @@ -234,24 +237,26 @@ PleReader::~PleReader() { } bool PleReader::open(const std::string& path, uint64_t table_offset, uint64_t n_rows, uint32_t max_inflight, - uint64_t cache_rows, std::string& err, bool io_thread) { + uint64_t cache_rows, std::string& err, bool io_thread, uint32_t row_bytes) { close(); if (max_inflight == 0 || max_inflight > 1024) { err = "PleReader: max_inflight must be 1..1024"; return false; } + if (row_bytes == 0 || row_bytes > PAGE) { err = "PleReader: a row must fit a page"; return false; } if (!impl_->file.open(path, err)) return false; - if (table_offset + n_rows * (uint64_t) ROW_BYTES > impl_->file.size()) { + if (table_offset + n_rows * (uint64_t) row_bytes > impl_->file.size()) { err = "PleReader: the table extends past the end of " + path; close(); return false; } impl_->table_offset = table_offset; impl_->n_rows = n_rows; + impl_->row_bytes = row_bytes; impl_->max_inflight = max_inflight; impl_->slab = (uint8_t*) DirectFile::alloc_aligned((size_t) max_inflight * 2 * PAGE); if (impl_->slab == nullptr) { err = "PleReader: cannot allocate read buffers"; close(); return false; } impl_->inflight.assign(max_inflight, Job{}); impl_->free_slots.clear(); for (uint32_t s = max_inflight; s-- > 0;) impl_->free_slots.push_back(s); - impl_->cache.init(cache_rows); + impl_->cache.init(cache_rows, row_bytes); impl_->error.clear(); reset_stats(); impl_->stop = false; @@ -291,7 +296,7 @@ void PleReader::close() { m.delayed.clear(); m.inflight.clear(); m.free_slots.clear(); - m.cache.init(0); + m.cache.init(0, ROW_BYTES); m.threaded = false; m.stop = false; } @@ -308,20 +313,20 @@ PleReader::Ticket PleReader::issue(const uint32_t* rows, size_t n, uint8_t* out_ std::unordered_map by_page; // aligned offset -> index in `jobs` std::vector jobs; for (size_t i = 0; i < n; ++i) { - uint8_t* dst = out_raw + i * ROW_BYTES; + uint8_t* dst = out_raw + i * m.row_bytes; ++m.stats.requests; if (rows[i] >= m.n_rows) { - std::memset(dst, 0, ROW_BYTES); + std::memset(dst, 0, m.row_bytes); continue; } if (const uint8_t* hit = m.cache.find(rows[i])) { - std::memcpy(dst, hit, ROW_BYTES); + std::memcpy(dst, hit, m.row_bytes); ++m.stats.cache_hits; continue; } - const uint64_t at = m.table_offset + (uint64_t) rows[i] * ROW_BYTES; + const uint64_t at = m.table_offset + (uint64_t) rows[i] * m.row_bytes; const uint64_t first = at / PAGE * PAGE; - const uint32_t length = (uint32_t) ((at + ROW_BYTES - 1) / PAGE * PAGE - first + PAGE); + const uint32_t length = (uint32_t) ((at + m.row_bytes - 1) / PAGE * PAGE - first + PAGE); auto f = by_page.find(first); if (f != by_page.end()) { Job& j = jobs[f->second]; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index d3b07b581ce..62472b7fdca 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -14,6 +14,7 @@ // AND IT IS PHASE 2, so hit rate is `h = 0` and the number it prints is slow on purpose // (`phase-2-correct-engine.md:5-9`). What it is FOR is the honest tok/s figure and the logit dump. +#include "strata/artifact/gguf_split.hpp" #include "strata/core/expert_cache.hpp" #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" @@ -27,7 +28,9 @@ #include "strata/kernels/s2_expert_grouped.hpp" #include "strata/kernels/sampler.hpp" #include "strata/kernels/shared_expert.hpp" +#include "strata/kernels/iq_kernels.hpp" #include "strata/kernels/native_moe.hpp" +#include "strata/kernels/native_mmvq.hpp" #include "strata/kernels/native_gdn.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/native_qsa.hpp" @@ -251,7 +254,8 @@ void usage() { " --pack DIR the pack directory (default pack/full)\n" " --tokens LIST the prompt as comma-separated token IDS (required)\n" " --tokens-file PATH pretokenized prompt, commas or whitespace (alternative to --tokens)\n" - " --ple-gguf PATH required PLE table (original second GGUF shard)\n" + " --ple-gguf PATH the PLE table: the model shard holding per_layer_token_embd (with --native,\n" + " the --native model's shard that holds it by default)\n" " --no-ple explicit diagnostic ablation of the PLE layer\n" " --ple-io direct|mmap n-gram table reads (plan v0.3 P2). direct (default): unbuffered SSD\n" " reads, the table never enters RAM or the file cache; mmap: A/B arm\n" @@ -277,11 +281,12 @@ void usage() { " --native-ple-postops experimental pinned PLE postprojection arithmetic\n" " --native-router experimental pinned CUDA 512-expert top-10 routing\n" " --cpu-oracle-q8-0 experimental pinned CPU expert quantization and dot reduction\n" - " --native SHARD1 every full-context native path at once (plan v0.3 P1): stream-token,\n" + " --native SHARD1 the model's first GGUF shard (the others of a split model are found by\n" + " name) and every full-context native path at once (plan v0.3 P1): stream-token,\n" " GR MMVF, BF16, head, dense + PLE key, MoE combine, GDN, router, QSA,\n" " indexer, RoPE, PLE postops, and the CPU q8_0 contract unless the\n" " expert cache is on. Individual --native-* flags stay for A/B.\n" - " --native-head-gguf PATH native Q5_K head from model shard 1; requires --stream-token\n" + " --native-head-gguf PATH native head from the model with this first shard; requires --stream-token\n" " --native-dense-gguf PATH native GDN/QSA/shared projections; repeat for each source model shard\n" " --expert-cache-cpu-order experimental GPU expert reduction matching CPU order\n" " --max-new N tokens to generate (default 16)\n" @@ -312,8 +317,6 @@ void usage() { " --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n" " --no-pool skip the CPU expert pool (the GPU-only floor)\n" " --sync-every-layer debug: synchronise after every layer\n" - " --ple-gguf PATH the n-gram/PLE shard. WITHOUT IT LAYER 1's PLE IS SILENTLY SKIPPED,\n" - " which changes every number downstream - pass it for any real run\n" " --dump-mixed PATH write the post-attention residual (n_embd, f32)\n" " --stage-timing per-stage KERNEL-COUNT shares. NOT a time profile: an uncaptured\n" " event interval includes host gaps, so run with --gpu-only-full first\n" @@ -604,9 +607,22 @@ int main(int argc, char** argv) { } strata::core::qsa_set_kv_int8(o.kv == "int8"); strata::core::layer_set_shared_early(!o.shared_late); + std::vector native_shards; // the --native model's GGUF shards if (!o.native_preset.empty()) { - if (o.no_ple || o.ple_gguf.empty()) { - std::fprintf(stderr, "strata generate: --native requires --ple-gguf (the PLE key is native too)\n"); + if (o.no_ple) { + std::fprintf(stderr, "strata generate: --native requires the PLE (the PLE key is native too)\n"); + return 2; + } + try { + native_shards = strata::gguf_split_paths(o.native_preset); + } catch (const std::exception& e) { + std::fprintf(stderr, "strata generate: --native: %s\n", e.what()); + return 2; + } + if (o.ple_gguf.empty()) o.ple_gguf = strata::kernels::ple_table_shard(native_shards); + if (o.ple_gguf.empty()) { + std::fprintf(stderr, "strata generate: no shard of the --native model holds per_layer_token_embd; " + "name the file with --ple-gguf\n"); return 2; } o.stream_token = true; @@ -615,7 +631,11 @@ int main(int argc, char** argv) { o.native_ple_key = o.native_moe_combine = o.native_gdn = o.native_router = true; o.native_qsa = o.native_qsa_indexer = o.native_rope = o.native_ple_postops = true; if (o.native_head_gguf.empty()) o.native_head_gguf = o.native_preset; - if (o.native_dense_gguf.empty()) o.native_dense_gguf = {o.native_preset, o.ple_gguf}; + if (o.native_dense_gguf.empty()) { + o.native_dense_gguf = native_shards; + if (std::find(native_shards.begin(), native_shards.end(), o.ple_gguf) == native_shards.end()) + o.native_dense_gguf.push_back(o.ple_gguf); + } // Plan v0.3 (24 Sep): the CPU experts stay on the VNNI kernel. The llama.cpp-CPU-exact q8_0 contract // cost 27.0 vs 17.2 ms/token of pool time and G-C does not need it; `--cpu-oracle-q8-0` still selects it. } @@ -689,6 +709,15 @@ int main(int argc, char** argv) { } } const bool native_pack = strata::kernels::cpu::expert_layout().native; + if (native_pack) + for (int64_t l = 0; l < strata::kernels::cpu::expert_layout().n_layers; ++l) { + const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) l]; + if (!strata::kernels::native_expert_supported(f.gu_type, f.d_type, f.n_embd, f.n_ff)) { + std::fprintf(stderr, "strata generate: layer %lld's experts (GGML types %d / %d) have no GPU kernels\n", + (long long) l, f.gu_type, f.d_type); + return 1; + } + } // plan v0.3 P6: the PCIe share of the missed experts, measured per kind of pack (the paper, finding on PCIe) if (o.pcie_frac < 0.0) o.pcie_frac = native_pack ? 0.55 : 0.2; // the canonical Q2_0 pack's CPU kernels are AVX-512 only; a native pack runs on AVX2 CPUs as well @@ -704,7 +733,7 @@ int main(int argc, char** argv) { return 2; } const strata::core::ModelGeometry g0; - if (!native_embed.load(o.native_preset, g0.n_embd, 248320, err)) { + if (!native_embed.load(native_shards, g0.n_embd, 248320, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -900,7 +929,8 @@ int main(int argc, char** argv) { ss.ple.w.key_scales = (const float*) ((const uint8_t*) wk->data + wk->codes_bytes); } if (o.native_ple_key && wk->quantized()) { - if (!wk->native_data || wk->native_type != 42 || !wk->native_q8_1) { + // Q2_0 in the Q2_0 file, Q8_0 in UD-Q4_K_XL + if (!wk->native_data || !strata::kernels::native_mmvq_supported(wk->native_type) || !wk->native_q8_1) { std::fprintf(stderr, "strata generate: native PLE key is absent or incompatible\n"); return 1; } @@ -977,7 +1007,7 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: experts via mmap (--mmap-experts; the A/B arm of R2.1)\n"); srcp = &src; } else { - arena_src.set_gguf(o.native_preset); // plan v0.3 P6: a native pack may take its experts from shard 1 + arena_src.set_gguf(native_shards); // plan v0.3 P6: a native pack may take its experts from the GGUF if (!arena_src.open(o.pack, g.n_layers, g.n_expert, /*threads=*/6, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; @@ -1295,11 +1325,18 @@ int main(int argc, char** argv) { const int64_t n_vocab = wo->ne1; strata::core::NativeHead native_head; if (!o.native_head_gguf.empty()) { - if (!native_head.load(o.native_head_gguf, g.n_embd, n_vocab, err)) { + std::vector head_shards; + try { + head_shards = strata::gguf_split_paths(o.native_head_gguf); + } catch (const std::exception& e) { + std::fprintf(stderr, "strata generate: native head: %s\n", e.what()); + return 1; + } + if (!native_head.load(head_shards, g.n_embd, n_vocab, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } - std::fprintf(stderr, "strata generate: experimental native Q5_K head, %llu bytes\n", + std::fprintf(stderr, "strata generate: native head, type %d, %llu bytes\n", native_head.type(), (unsigned long long) native_head.weight_bytes()); } std::vector logits((size_t) n_vocab); diff --git a/tools/iq_pack.py b/tools/iq_pack.py index 2c5d54f113d..6e74216a3f1 100644 --- a/tools/iq_pack.py +++ b/tools/iq_pack.py @@ -1,19 +1,22 @@ -"""tools/iq_pack.py - plan v0.3 P6: a native pack for any of the model files (Q2_0, IQ2_XS, IQ3_XXS). +"""tools/iq_pack.py - plan v0.3 P6: a native pack for a model file whose tensors the engine reads in GGUF form +(Q2_0, IQ2_XS, IQ3_XXS, UD-Q4_K_XL). - python tools/iq_pack.py --gguf -00001-of-00002.gguf --out pack/iq3_xxs (standalone) + python tools/iq_pack.py --gguf -00001-of-0000N.gguf --out pack/iq3_xxs (standalone) python tools/iq_pack.py --gguf -00001-of-00002.gguf --base pack/full --out ... (share dense.bin) -The i-quant experts cannot be re-expressed in the Q2_0 pack form, so this pack keeps every quantized tensor in -its GGUF form: +The other shards of a split model are found by name. The experts cannot be re-expressed in the Q2_0 pack form, +so this pack keeps every quantized tensor in its GGUF form: - experts.bin per layer, 512 blobs of [gate rows | up rows | down rows], the raw GGUF slices. Blob - size is per layer (the files mix IQ1_M ... IQ3_S gate/up and Q2_0 / IQ4_NL down). - native_experts.txt one line per layer: layer gu_type d_type offset blob_bytes + native_experts.txt one line per layer: layer gu_type d_type offset blob_bytes. The engine fills its expert + arena from the GGUF (tensors found by name); `offset` is the layer's place in that arena. + experts.bin optional (--experts-bin): the arena as a file, per layer 512 blobs of [gate rows | up rows | + down rows], the raw GGUF slices. Blob size is per layer (the files mix formats). index.txt the table the engine loads. Quantized dense tensors, token_embd and output are served natively from the GGUF by the engine (--native): their rows carry shape only. - dense.bin standalone: the BF16/F16/F32 tensors exactly as the GGUF stores them (index kinds 4/5/2). + dense.bin standalone: every other tensor in the float form the engine takes (FORM below; index kinds + 4/5/2 = BF16/F16/F32), converted when the file stores it otherwise. With --base: the base (Q2_0) pack's dense.bin, hard-linked - the float tensors are - byte-identical in all three model files (checked) - plus extra.bin for tensors that are + byte-identical in all three ISTA model files (checked) - plus extra.bin for tensors that are float here but quantized in the base pack (blk.1.ple_key). tokenizer/ exported from the GGUF (tools/strata_tokenizer.py). """ @@ -23,6 +26,7 @@ import json import os import pathlib +import re import shutil import subprocess import sys @@ -37,6 +41,97 @@ ROLES = ("gate", "up", "down") N_EXPERT = 512 ALIGN = 64 +KIND = {"BF16": "4", "F16": "5", "F32": "2"} + +# The float form the engine takes for each tensor it does not read from the GGUF (name without `blk.N.`). A file +# that stores one otherwise is converted: UD-Q4_K_XL keeps the router, SSM gates and injections in F32 (their +# values are BF16's, so the conversion is exact), the hyper-connection projections and the PLE value in Q8_0 +# (re-rounded to BF16) and the PLE conv in F32. +FORM = { + "ffn_gate_inp.weight": "BF16", "ffn_gate_inp_shexp.weight": "BF16", + "hc_attn_down.weight": "BF16", "hc_attn_up.weight": "BF16", "hc_attn_inject.weight": "BF16", + "hc_ffn_down.weight": "BF16", "hc_ffn_up.weight": "BF16", "hc_ffn_inject.weight": "BF16", + "output_hc_down.weight": "BF16", "output_hc_up.weight": "BF16", + "indexer.k_proj.weight": "BF16", "indexer.q_proj.weight": "BF16", + "ssm_alpha.weight": "BF16", "ssm_beta.weight": "BF16", + "ple_key.weight": "BF16", "ple_value.weight": "BF16", "ple_conv1d.weight": "F16", + "attn_q_norm.weight": "F32", "attn_k_norm.weight": "F32", "hc_attn_norm.weight": "F32", + "hc_ffn_norm.weight": "F32", "output_hc_norm.weight": "F32", "indexer.q_norm.weight": "F32", + "indexer.k_norm.weight": "F32", "ple_norm_conv.weight": "F32", "ple_norm_key.weight": "F32", + "ple_norm_query.weight": "F32", "ssm_a": "F32", "ssm_conv1d.weight": "F32", "ssm_dt.bias": "F32", + "ssm_norm.weight": "F32", +} +# Served from the GGUF when quantized in a type the engine has kernels for: the dense projections and the PLE key +# (NativeDense and NativeHead: native_mmvq_supported), the token embedding (NativeEmbed: iq_supported). +MMVQ_TYPES = {"Q4_0", "Q5_0", "Q8_0", "Q3_K", "Q4_K", "Q5_K", "Q6_K", "IQ4_NL", "IQ4_XS", "Q2_0", "IQ2_XXS", + "IQ2_XS", "IQ3_XXS", "IQ3_S", "IQ2_S", "IQ1_M"} +EMBED_TYPES = {"IQ2_XXS", "IQ2_XS", "IQ3_XXS", "IQ4_NL", "IQ3_S", "IQ2_S", "IQ4_XS", "IQ1_S", "IQ1_M", "Q2_0", "Q3_K", + "Q4_K", "Q5_K", "Q6_K", "Q5_1", "Q8_0"} +NATIVE = {"attn_qkv.weight", "attn_gate.weight", "ssm_out.weight", "attn_q.weight", "attn_k.weight", + "attn_v.weight", "attn_output.weight", "ffn_gate_shexp.weight", "ffn_up_shexp.weight", + "ffn_down_shexp.weight", "ple_key.weight", "output.weight"} +PLE_TABLE = "per_layer_token_embd.weight" # read from the GGUF by the engine (--ple-gguf) + + +class Model: + """A model's shards (a split model's are found by name from the first), tensors looked up across them.""" + + def __init__(self, first: pathlib.Path): + m = re.fullmatch(r"(.*-)00001(-of-(\d{5})\.gguf)", first.name) + names = [f"{m.group(1)}{i:05d}{m.group(2)}" for i in range(1, int(m.group(3)) + 1)] if m else [first.name] + self.shards = [] + self.where = {} + for name in names: + path = first.parent / name + if not path.exists(): + raise FileNotFoundError(f"missing model shard {path}") + g = G.GGUFFile(path) + mm = np.memmap(path, dtype=np.uint8, mode="r") + for t in g.tensors: + if t.name in self.where: + raise ValueError(f"tensor {t.name} is in two shards") + self.where[t.name] = (g, mm, t) + self.shards.append(g) + + def tensors(self): + return [t for g in self.shards for t in g.tensors] + + def get(self, name): + return self.where[name][2] if name in self.where else None + + def bytes(self, t) -> np.ndarray: + g, mm, _ = self.where[t.name] + return mm[g.data_start + t.offset: g.data_start + t.offset + t.expected_bytes()] + + +def suffix(name: str) -> str: + return re.sub(r"^blk\.\d+\.", "", name) + + +def to_f32(raw: np.ndarray, type_name: str) -> np.ndarray: + if type_name == "F32": + return raw.view(" bytes: + if form == "F32": + return x.astype("> 16) & 1)) >> 16).astype(" bool: + return name.startswith("blk.") and name.endswith(("_exps.weight",)) def read_index(path: pathlib.Path): @@ -50,40 +145,52 @@ def read_index(path: pathlib.Path): return header, rows -def tensor_bytes(mm, g, t) -> np.ndarray: - n = t.expected_bytes() - return mm[g.data_start + t.offset: g.data_start + t.offset + n] - - -def is_expert(name: str) -> bool: - return name.startswith("blk.") and name.endswith(("_exps.weight",)) +def native_row(t) -> list[str]: + ne0 = int(t.shape[0]) + ne1 = int(t.shape[1]) if len(t.shape) > 1 else 0 + return [t.name, "0", "0", "0", "0", "0", "0", str(ne0), str(ne1), "8", "0", "32"] + ["0"] * 7 -def index_standalone(src, out, g, T, mm) -> int: - """Every non-expert tensor of shard 1: floats into dense.bin as stored, quantized ones native-only.""" - rows, at = [], 0 - served = 0 +def index_standalone(src, out, model: Model) -> int: + """Every non-expert tensor: quantized ones the engine serves natively as shape-only rows, the others into + dense.bin in their engine form.""" + rows, at, served, converted = [], 0, 0, 0 with open(out / "dense.bin", "wb") as fo: - for t in g.tensors: - if is_expert(t.name): + for t in model.tensors(): + if is_expert(t.name) or t.name == PLE_TABLE: continue if len(t.shape) > 2: print("tensor %s has %d dimensions; the index holds two" % (t.name, len(t.shape))) return 1 + sfx = suffix(t.name) + if t.name == "token_embd.weight" or (sfx in NATIVE and t.type_name not in FLOAT): + ok = EMBED_TYPES if t.name == "token_embd.weight" else MMVQ_TYPES + if t.type_name not in ok: + print("%s is %s, a type the engine has no native kernels for" % (t.name, t.type_name)) + return 1 + served += 1 + rows.append(native_row(t)) + continue + form = FORM.get(sfx) + if form is None: + print("tensor %s: no engine form known for it" % t.name) + return 1 + raw = model.bytes(t) + if t.type_name == form: + data = raw.tobytes() + else: + data = encode(to_f32(raw, t.type_name), form) + converted += 1 ne0 = int(t.shape[0]) ne1 = int(t.shape[1]) if len(t.shape) > 1 else 0 - if t.type_name in FLOAT: - raw = tensor_bytes(mm, g, t).tobytes() - kind = {"BF16": "4", "F16": "5", "F32": "2"}[t.type_name] - rows.append([t.name, "0", kind, str(at), str(len(raw)), "0", str(len(raw)), str(ne0), str(ne1), - "0", "0", "1"] + ["0"] * 7) - fo.write(raw) - pad = (-len(raw)) % ALIGN - fo.write(b"\0" * pad) - at += len(raw) + pad - else: - served += 1 - rows.append([t.name, "0", "0", "0", "0", "0", "0", str(ne0), str(ne1), "8", "0", "32"] + ["0"] * 7) + rows.append([t.name, "0", KIND[form], str(at), str(len(data)), "0", str(len(data)), str(ne0), str(ne1), + "0", "0", "1"] + ["0"] * 7) + fo.write(data) + pad = (-len(data)) % ALIGN + fo.write(b"\0" * pad) + at += len(data) + pad + if converted: + print("dense.bin: %d tensors converted to their engine form" % converted) write_index(out, rows, src, served, 0) return 0 @@ -102,25 +209,23 @@ def write_index(out, rows, src, served, n_extra): % (len(rows), served, n_extra, at / 2**30)) -def index_from_base(a, src, base, out, g, T, mm) -> int: +def index_from_base(src, base, out, model: Model) -> int: base_src = pathlib.Path(json.loads((base / "manifest.json").read_text(encoding="utf-8"))["source"]["shard1"]) if not base_src.exists(): print("cannot find the base pack's shard 1 from its manifest.json") return 1 - bg = G.GGUFFile(base_src) - BT = {t.name: t for t in bg.tensors} - bmm = np.memmap(base_src, dtype=np.uint8, mode="r") + bm = Model(base_src) header, rows = read_index(base / "index.txt") new_rows, extra = [], [] served = 0 for name, f in rows.items(): - t, bt = T.get(name), BT.get(name) + t, bt = model.get(name), bm.get(name) if t is None or bt is None: print("tensor %s missing from one of the models" % name) return 1 if t.type_name in FLOAT and bt.type_name in FLOAT: if t.type_name != bt.type_name or t.shape != bt.shape or \ - not np.array_equal(tensor_bytes(mm, g, t), tensor_bytes(bmm, bg, bt)): + not np.array_equal(model.bytes(t), bm.bytes(bt)): print("float tensor %s differs from the base model; this pack cannot reuse its dense.bin" % name) return 1 new_rows.append(list(f)) @@ -130,7 +235,7 @@ def index_from_base(a, src, base, out, g, T, mm) -> int: return 1 nbytes = t.expected_bytes() off = sum(len(b) + (-len(b)) % ALIGN for b in extra) - extra.append(tensor_bytes(mm, g, t).tobytes()) + extra.append(model.bytes(t).tobytes()) # file 3 = extra.bin, raw BF16 (index kind 4) new_rows.append([name, "3", "4", str(off), str(nbytes), "0", str(nbytes), f[7], f[8]] + ["0"] * 10) else: @@ -154,7 +259,7 @@ def index_from_base(a, src, base, out, g, T, mm) -> int: def main() -> int: ap = argparse.ArgumentParser() - ap.add_argument("--gguf", required=True, help="the model's shard 1") + ap.add_argument("--gguf", required=True, help="the model's first shard (the others are found by name)") ap.add_argument("--base", help="optional: a Q2_0 canonical pack whose dense.bin holds the shared float tensors") ap.add_argument("--out", required=True) ap.add_argument("--skip-experts", action="store_true", help="rewrite the index only") @@ -166,13 +271,8 @@ def main() -> int: out = pathlib.Path(a.out) out.mkdir(parents=True, exist_ok=True) - g = G.GGUFFile(src) - T = {t.name: t for t in g.tensors} - mm = np.memmap(src, dtype=np.uint8, mode="r") - if a.base: - rc = index_from_base(a, src, base, out, g, T, mm) - else: - rc = index_standalone(src, out, g, T, mm) + model = Model(src) + rc = index_from_base(src, base, out, model) if a.base else index_standalone(src, out, model) if rc: return rc if not (out / "tokenizer" / "vocab.json").exists(): @@ -180,10 +280,11 @@ def main() -> int: check=True) # writes /tokenizer/ # ---- the experts - n_layers = 1 + max(int(n.split(".")[1]) for n in T if n.startswith("blk.") and n.endswith("_exps.weight")) + names = [t.name for t in model.tensors()] + n_layers = 1 + max(int(n.split(".")[1]) for n in names if n.startswith("blk.") and n.endswith("_exps.weight")) layout, offset = [], 0 for l in range(n_layers): - ts = [T["blk.%d.ffn_%s_exps.weight" % (l, r)] for r in ROLES] + ts = [model.get("blk.%d.ffn_%s_exps.weight" % (l, r)) for r in ROLES] per = [t.expected_bytes() // N_EXPERT for t in ts] if per[0] != per[1] or ts[0].type_name != ts[1].type_name: print("layer %d: gate and up differ in type" % l) @@ -192,10 +293,11 @@ def main() -> int: layout.append((l, ts[0].type_id, ts[2].type_id, offset, blob, ts)) offset += blob * N_EXPERT with open(out / "native_experts.txt", "w", encoding="utf-8", newline="\n") as fo: - fo.write("# strata native experts v2: layer gu_type d_type offset blob_bytes gate_off up_off down_off " - "(n_expert %d, total %d; the last three are absolute offsets in %s)\n" % (N_EXPERT, offset, src.name)) + fo.write("# strata native experts v3: layer gu_type d_type offset blob_bytes (n_expert %d, total %d; the " + "engine takes the experts from %s by tensor name, or from experts.bin)\n" + % (N_EXPERT, offset, src.name)) for l, gt, dt, off, blob, ts in layout: - fo.write("%d %d %d %d %d %d %d %d\n" % (l, gt, dt, off, blob, *[g.data_start + t.offset for t in ts])) + fo.write("%d %d %d %d %d\n" % (l, gt, dt, off, blob)) if a.skip_experts or not a.experts_bin: if (out / "experts.bin").exists() and not a.experts_bin: print("note: %s/experts.bin exists; the engine reads it instead of the GGUF" % out) @@ -206,7 +308,7 @@ def main() -> int: return 0 with open(path, "wb") as fo: for l, gt, dt, off, blob, ts in layout: - parts = [tensor_bytes(mm, g, t).reshape(N_EXPERT, -1) for t in ts] + parts = [model.bytes(t).reshape(N_EXPERT, -1) for t in ts] chunk = np.concatenate(parts, axis=1) # (512, blob): gate | up | down per expert assert chunk.shape == (N_EXPERT, blob) fo.write(chunk.tobytes()) From 240c8be7e6927caf56d46857f47972151941706e Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 02:58:31 +0200 Subject: [PATCH 03/92] Short prompt parts through verify windows A batched prompt call costs a fixed 4.7-6.6 s on an RTX 3090 behind a PCIe 4.0 x4 link: it lends and refills expert-cache slots and streams every expert its tokens use, however few tokens it holds. Verify windows cost ~15 ms (IQ3_XXS) and ~21 ms (UD-Q4_K_XL) per token, so the two break even at ~480 and ~530 tokens. - --feed-max defaults to 512 (was 256). - A last prompt chunk of up to --feed-max tokens goes through verify windows as well. A prompt just past a multiple of --prefill used to run its last tokens as a chunk of their own (3.6 s on UD-Q4_K_XL). - Chunks and checkpoint segments count from the request's first new token, so a follow-up that starts inside a segment no longer gets a short first chunk. --cache-every is rounded down to whole --prefill chunks. - Image requests always take the batched path, the only one that reads image rows. A request of up to --feed-max tokens with an image used to lose the image. Prompt speed with the prompt cache off, 16K-131K tokens: IQ3_XXS 703-766 tok/s (was 674-747), UD-Q4_K_XL 570-603 tok/s (was 529-594). --- README.md | 2 +- src/program/generate.cpp | 34 ++++++++++++++++++++++------------ 2 files changed, 23 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 55de09e3136..af573b3fdd0 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 529-594 tok/s, generation 49-57 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 570-603 tok/s, generation 49-59 tok/s. ## Before you start diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 62472b7fdca..ce9be41317a 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -239,11 +239,13 @@ struct Options { int prompt_cache = 16; /// Host memory for sequences the live one replaced (their K/V cells), in MiB. int64_t cache_ram_mib = 16384; - /// Checkpoint interval inside long prompts (positions that are multiples of it). + /// Checkpoint interval inside long prompts, in whole prompt chunks (counted from the request's first new cell). int64_t cache_every = 16384; - /// Prompt remainders of up to this many tokens go through verify windows instead of the batched prompt path, - /// which lends and refills expert-cache slots on every call. - int64_t feed_max = 256; + /// Prompt parts of up to this many tokens go through verify windows instead of the batched prompt path: a new + /// part that short, and a last prompt chunk that short. A batched call costs a fixed ~5-7 s on the 3090 (it + /// lends and refills expert-cache slots and streams every expert its tokens use); verify windows cost 15-22 ms + /// per token. They break even at ~480 (IQ3_XXS) and ~530 tokens (UD-Q4_K_XL; bench/feed_test.py). + int64_t feed_max = 512; int adapt_swaps = 96; }; @@ -307,11 +309,12 @@ void usage() { " --prompt-cache N --serve: keep N checkpoints of the sequence state in host memory and\n" " continue each request from the longest cached prefix (default 16;\n" " 0 = every request starts from an empty sequence)\n" - " --cache-every N checkpoint interval inside long prompts (default 16384)\n" + " --cache-every N checkpoint interval inside long prompts, rounded down to whole --prefill\n" + " chunks (default 16384)\n" " --cache-ram MIB --serve: host memory for the K/V cells of sequences a request replaced,\n" " so switching back to them continues where they were (default 16384)\n" - " --feed-max N --serve: prompt remainders up to N tokens run through verify windows\n" - " instead of the batched prompt path (default 256)\n" + " --feed-max N --serve: new prompt parts and last prompt chunks of up to N tokens run\n" + " through verify windows instead of the batched prompt path (default 512)\n" " --no-token-graph A/B: two graphs per layer (the host launches each) instead of one per token\n" " --no-fused-gr A/B: the six-kernel hyper-connection read and a separate write (native)\n" " --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n" @@ -2055,13 +2058,21 @@ int main(int argc, char** argv) { // ---- the rest of the prompt, [reuse, n - 1). A turn usually shares everything before its prompt's // last <|im_start|> with the next one (chat templates re-render the answer, e.g. without its // reasoning), so that position gets a checkpoint, and so do the prompt's end (a regenerated answer) - // and every `cache_every` cells of a long prompt. A short remainder goes through verify windows. + // and the end of every `step` cells of a long prompt. Short parts go through verify windows. int64_t q = 0; if (track) for (int64_t i = n - 2; i > reuse; --i) if (ids[(size_t) i] == kImStart) { q = i; break; } int64_t batched_end = reuse; // [reuse, batched_end) through the batched prompt path - if (n - 1 - reuse > o.feed_max) batched_end = (q > reuse && n - 1 - q <= o.feed_max) ? q : n - 1; + if (geni) batched_end = n - 1; // image rows reach the batched path only + else if (n - 1 - reuse > o.feed_max) batched_end = (q > reuse && n - 1 - q <= o.feed_max) ? q : n - 1; + // Every batched chunk streams each expert its tokens use, however few tokens it holds, so a short last + // chunk goes through verify windows too. Chunks and checkpoint segments start at `reuse`. + const int64_t step = std::max(1, o.cache_every / o.prefill_chunk) * o.prefill_chunk; + if (!geni && batched_end > reuse) { + const int64_t last = reuse + (batched_end - 1 - reuse) / o.prefill_chunk * o.prefill_chunk; + if (last > reuse && batched_end - last <= o.feed_max) batched_end = last; + } if (batched_end > reuse) { std::vector> lent_now; apply_pending(true); @@ -2074,12 +2085,11 @@ int main(int argc, char** argv) { cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); } for (int64_t a0 = reuse; a0 < batched_end;) { - const int64_t b0 = track ? std::min(batched_end, (a0 / o.cache_every + 1) * o.cache_every) - : batched_end; + const int64_t b0 = track ? std::min(batched_end, a0 + step) : batched_end; if (!sp.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } if (track) { pcache.set(a0, ids.data() + a0, b0 - a0); - if (b0 % o.cache_every == 0 || b0 == q || b0 == n - 1) pcache.checkpoint(b0); + pcache.checkpoint(b0); } a0 = b0; } From 83d83e77ce00da360baa11ba73a45a35e63e6f38 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 03:45:04 +0200 Subject: [PATCH 04/92] Build without compiler warnings - MSVC: _CRT_SECURE_NO_WARNINGS (fopen, sscanf and getenv are standard C), /wd4211 for the host stubs nvcc generates for kernels with internal linkage, and C4324 off around ExpertPool, whose alignas padding is the point. - Unused locals, device functions, a kernel and a constant removed; shadowed locals renamed; the router clamp bound is an if constexpr. - qsa_layer's prologue, a single 917-character line, split into lines. --- CMakeLists.txt | 6 ++++- include/strata/kernels/cpu/pool.hpp | 7 ++++++ src/artifact/dequant.cpp | 1 - src/core/layer.cpp | 15 ++++++++++-- src/kernels/cuda/gdn.cu | 2 -- src/kernels/cuda/gr.cu | 34 +++------------------------- src/kernels/cuda/s2_gemv_quads.cu | 1 - src/kernels/cuda/shared_expert.cu | 5 ---- src/kernels/qsa_parity.cpp | 10 ++++---- src/kernels/quantize_act_parity.cpp | 16 ++++++------- src/kernels/router_top10_parity.cpp | 4 ++-- src/kernels/shared_expert_parity.cpp | 14 ++++++------ src/program/generate.cpp | 10 ++++---- 13 files changed, 55 insertions(+), 70 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1d569254128..cdde42086fb 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -71,7 +71,11 @@ target_compile_options(strata_warnings INTERFACE $<$,$>:/utf-8> $<$,$>>:-Wall> $<$,$>>:-Wextra> - $<$,$>:-Xcompiler=/W4>) + $<$,$>:-Xcompiler=/W4> + # C4211 comes from the host stubs nvcc generates for kernels with internal linkage (anonymous namespaces). + $<$,$>:-Xcompiler=/wd4211>) +# fopen, sscanf and getenv are standard C; MSVC deprecates them in favour of its own _s variants. +target_compile_definitions(strata_warnings INTERFACE $<$:_CRT_SECURE_NO_WARNINGS>) if(STRATA_WERROR) target_compile_options(strata_warnings INTERFACE $<$,$>:/WX> diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp index 7c1a9d2cd5f..c138b2b231c 100644 --- a/include/strata/kernels/cpu/pool.hpp +++ b/include/strata/kernels/cpu/pool.hpp @@ -72,6 +72,10 @@ std::vector physical_cores(bool skip_first); long long pin_current_thread(int core); void restore_thread_affinity(long long previous); +#ifdef _MSC_VER +#pragma warning(push) +#pragma warning(disable : 4324) // the alignas(64) members pad the class on purpose (one cache line each) +#endif class ExpertPool { public: /// `n_workers <= 0` means "every physical core except the first". Workers are pinned to physical cores @@ -191,5 +195,8 @@ class ExpertPool { const NativeFmt* nfmt_ = nullptr; std::vector split_multi_; }; +#ifdef _MSC_VER +#pragma warning(pop) +#endif } // namespace strata::kernels::cpu diff --git a/src/artifact/dequant.cpp b/src/artifact/dequant.cpp index ff058c0ac71..8419ad6a54e 100644 --- a/src/artifact/dequant.cpp +++ b/src/artifact/dequant.cpp @@ -38,7 +38,6 @@ int main(int argc, char** argv) { // implementation on identical bytes. A single implementation being self-consistent // proves little - two disagreeing is what finds bugs (round 67). double sum = 0, sumabs = 0; - const uint8_t* base = g.tensor_data(t); for (uint64_t b = 0; b < nblocks && b < 64; ++b) { float out[64]; strata::dequantize_q2_0(base + b * 18, out); diff --git a/src/core/layer.cpp b/src/core/layer.cpp index c662b179f02..9d8d82ce6d7 100644 --- a/src/core/layer.cpp +++ b/src/core/layer.cpp @@ -623,7 +623,19 @@ void stage_timing_report(int64_t n_layers) { // needs nothing from the layer it is called for beyond its index. static void dump_slot(float* dump, const ModelGeometry& g, int64_t layer, const float* src, uint64_t off, uint64_t n, void* stream); -bool qsa_layer(const WeightTable& tables, const ModelGeometry& g, int64_t layer, int64_t pos, int32_t pos_base, const QsaState& st, const QsaBuffers& b, const float* x, float* out, void* stream, std::string& err, float* dump) { using namespace strata::kernels; const QsaShapes s = qsa_shapes(g); const LayerView v(tables, layer); const int64_t n_kv = pos + 1; /* P7 audit: RoPE reads cos/sin row pos_base + pos, and the table holds max_cells rows. */ if ((int64_t) pos_base + pos >= st.max_cells || pos_base < 0) { err = "qsa_layer: position " + std::to_string((long long) pos_base + pos) + " is outside the RoPE table (" + std::to_string((long long) st.max_cells) + " rows)"; return false; } const int64_t n_bid = n_kv / s.idx_block; const int64_t width = qsa_selection_width(n_kv, s); const int64_t cap = qsa_selection_width(kTopkMaxCells, s); +bool qsa_layer(const WeightTable& tables, const ModelGeometry& g, int64_t layer, int64_t pos, int32_t pos_base, + const QsaState& st, const QsaBuffers& b, const float* x, float* out, void* stream, + std::string& err, float* dump) { + using namespace strata::kernels; + const QsaShapes s = qsa_shapes(g); + const LayerView v(tables, layer); + /* P7 audit: RoPE reads cos/sin row pos_base + pos, and the table holds max_cells rows. */ + if ((int64_t) pos_base + pos >= st.max_cells || pos_base < 0) { + err = "qsa_layer: position " + std::to_string((long long) pos_base + pos) + " is outside the RoPE table (" + + std::to_string((long long) st.max_cells) + " rows)"; + return false; + } + const int64_t cap = qsa_selection_width(kTopkMaxCells, s); const auto normalize_rotate = [&](float* data, const WeightRef* norm, int rows, int cols) { try { if (native_qsa_enabled()) native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, rows, RMS_EPS, stream); @@ -922,7 +934,6 @@ bool block_layer_pre(const WeightTable& tables, const ModelGeometry& g, int64_t gr_write(bb.R, bb.block_out, bb.inject2, gs0, bb.R, stream); pending_ffn = false; } - const int64_t hcd = strata::kernels::NG_HC_DIM; strata::kernels::PleOut po; // Exports must not alias the block's internal workspace. The previous diagnostic views used a // different layout inside that workspace: exporting gated values overwrote normalized values diff --git a/src/kernels/cuda/gdn.cu b/src/kernels/cuda/gdn.cu index aad247a797e..5111afda106 100644 --- a/src/kernels/cuda/gdn.cu +++ b/src/kernels/cuda/gdn.cu @@ -31,8 +31,6 @@ constexpr int JTHREADS = 32; ///< threads along j, the state's fast axis /// The arithmetic does not change at all: the same columns are computed by the same code, in different blocks. constexpr int MAX_H = 1; ///< heads staged in shared memory per block -/// ggml_compute_softplus_f32: log1p(exp(x)), with the large-x branch that avoids overflow. -__device__ __forceinline__ float softplus_f(float x) { return x > 20.0f ? x : log1pf(expf(x)); } __device__ __forceinline__ float sigmoid_f(float x) { return 1.0f / (1.0f + expf(-x)); } /// One step of the recurrence. One thread per (h, j) column of the state. diff --git a/src/kernels/cuda/gr.cu b/src/kernels/cuda/gr.cu index 7eea7f49f43..24d615b2fb2 100644 --- a/src/kernels/cuda/gr.cu +++ b/src/kernels/cuda/gr.cu @@ -66,11 +66,6 @@ __device__ __forceinline__ void store_activation(uint16_t* dst, int i, float x) __device__ __forceinline__ float silu_f(float x) { return x / (1.0f + expf(-x)); } __device__ __forceinline__ float sigmoid_f(float x) { return 1.0f / (1.0f + expf(-x)); } -__device__ __forceinline__ double warp_sum(double v) { - for (int off = 16; off > 0; off >>= 1) v += __shfl_down_sync(0xFFFFFFFFu, v, off); - return __shfl_sync(0xFFFFFFFFu, v, 0); -} - __device__ __forceinline__ float warp_sumf(float v) { for (int off = 16; off > 0; off >>= 1) v += __shfl_down_sync(0xFFFFFFFFu, v, off); return __shfl_sync(0xFFFFFFFFu, v, 0); @@ -84,9 +79,9 @@ __device__ __forceinline__ float warp_sumf(float v) { /// the cast cannot overrun, and it keeps the change to one function instead of one per caller. __device__ float block_sumf(float v, double* scratch_raw) { float* scratch = (float*) scratch_raw; - // The leading barrier is not decoration - see the note on `block_sum` above: the result is read straight out - // of `scratch[0]` by every thread and a later call reuses the array, so without it a fast thread can - // overwrite `scratch[0]` before a slow one has read the previous result. + // The leading barrier is not decoration: the result is read straight out of `scratch[0]` by every thread and + // a later call reuses the array, so without it a fast thread can overwrite `scratch[0]` before a slow one has + // read the previous result - a race that is invisible in most runs. __syncthreads(); const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; v = warp_sumf(v); @@ -100,29 +95,6 @@ __device__ float block_sumf(float v, double* scratch_raw) { return scratch[0]; } -/// Block-wide sum in DOUBLE, broadcast to every thread. -/// -/// Double because `ref/gr.py` reduces in float64 for the norm and the bottleneck, and because a block-wide -/// f32 sum of 2560 squares is a different number from the reference's; the shuffle tree here also reorders -/// the additions, so the only defence is enough precision that the order stops mattering. -/// -/// The leading `__syncthreads()` is not decoration: the result is read straight out of `scratch[0]` by every -/// thread, and a later call reuses the same array. Without a barrier on entry a fast thread can overwrite -/// `scratch[0]` before a slow one has read the previous result - a race that is invisible in most runs. -__device__ double block_sum(double v, double* scratch) { - __syncthreads(); - const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; - v = warp_sum(v); - if (lane == 0) scratch[warp] = v; - __syncthreads(); - const int nw = ((int) blockDim.x + 31) >> 5; - v = (threadIdx.x < nw) ? scratch[threadIdx.x] : 0.0; - if (warp == 0) v = warp_sum(v); - if (threadIdx.x == 0) scratch[0] = v; - __syncthreads(); - return scratch[0]; -} - /// `hc` blocks, one per residual stream: per-stream RMSNorm, then the activated stream in both forms. /// /// ggml_rms_norm reduces over ne[0], and ne[0] is n_embd, so the reduction is PER STREAM. The `[n_embd, hc]` diff --git a/src/kernels/cuda/s2_gemv_quads.cu b/src/kernels/cuda/s2_gemv_quads.cu index 7cb7cb4422c..90973ab70a8 100644 --- a/src/kernels/cuda/s2_gemv_quads.cu +++ b/src/kernels/cuda/s2_gemv_quads.cu @@ -28,7 +28,6 @@ namespace { // QK = 64 elements per group; a quad of 4 elements is 1/16 of a group, so // group = (quad * 4) >> 6 = quad >> 4 constexpr int QK_S2 = 64; -constexpr int CODES_PER_BYTE_S2 = 4; __global__ void s2_gemv_quads_kernel(const uint16_t* __restrict__ x, const uint8_t* __restrict__ codes, const float* __restrict__ scales, float* __restrict__ y, long long n_in, diff --git a/src/kernels/cuda/shared_expert.cu b/src/kernels/cuda/shared_expert.cu index 81402a9fd1c..aba29bbb4c3 100644 --- a/src/kernels/cuda/shared_expert.cu +++ b/src/kernels/cuda/shared_expert.cu @@ -68,11 +68,6 @@ __global__ void native_swiglu_kernel(const float* gate, const float* up, float* out[i] = __fdividef(gate[i], 1.0f + __expf(-gate[i])) * up[i]; } -__global__ void to_f16_kernel(const float* __restrict__ in, uint16_t* __restrict__ out, int n) { - const int i = blockIdx.x * blockDim.x + threadIdx.x; - if (i < n) out[i] = f16_from_f32(in[i]); -} - // The per-token scalar gate: sigmoid(dot(x, w)) with w = `ffn_gate_inp_shexp` (n_embd,). // // **ONE THREAD WAS THE LARGEST SINGLE KERNEL IN THE MODEL, AND THE COMMENT DEFENDING IT SAID WHY IT SHOULD diff --git a/src/kernels/qsa_parity.cpp b/src/kernels/qsa_parity.cpp index 5a8186ddae1..517db948aa9 100644 --- a/src/kernels/qsa_parity.cpp +++ b/src/kernels/qsa_parity.cpp @@ -763,7 +763,7 @@ int main(int argc, char** argv) { const int64_t max_blocks = max_n / R + 1; const int64_t cap = strata::kernels::qsa_selection_width(max_n, S); Dev pooled, query, scores((size_t) max_n); - Dev ids((size_t) cap), step(strata::kernels::kStepCount); + Dev sel((size_t) cap), step(strata::kernels::kStepCount); pooled.put(std::vector((size_t) max_blocks * IDXD, 0.0f)); query.put(std::vector((size_t) IDXN * IDXD, 0.0f)); @@ -773,7 +773,7 @@ int main(int argc, char** argv) { check(cudaStreamCreate(&stream), "tail stream"); check(cudaStreamBeginCapture(stream, cudaStreamCaptureModeThreadLocal), "tail capture begin"); strata::kernels::qsa_index_step(pooled.p, query.p, nullptr, S, step.p, max_blocks, scores.p, stream); - strata::kernels::topk_512_step(scores.p, S, cap, step.p, ids.p, stream); + strata::kernels::topk_512_step(scores.p, S, cap, step.p, sel.p, stream); check(cudaStreamEndCapture(stream, &graph), "tail capture end"); check(cudaGraphInstantiate(&exec, graph, 0), "tail instantiate"); @@ -789,15 +789,15 @@ int main(int argc, char** argv) { for (int64_t j = n - tail; j < n; ++j) want.push_back((int32_t) j); strata::kernels::qsa_index(pooled.p, n / R, query.p, nullptr, S, n, scores.p, nullptr); - strata::kernels::topk_512(scores.p, n, S, cap, ids.p, nullptr); - require("tail static n=" + std::to_string(n), ids.get((size_t) width) == want); + strata::kernels::topk_512(scores.p, n, S, cap, sel.p, nullptr); + require("tail static n=" + std::to_string(n), sel.get((size_t) width) == want); std::vector values(strata::kernels::kStepCount); strata::kernels::qsa_step_fill(values.data(), n - 1, S); step.put(values); check(cudaGraphLaunch(exec, stream), "tail replay"); check(cudaStreamSynchronize(stream), "tail replay sync"); - require("tail captured n=" + std::to_string(n), ids.get((size_t) width) == want); + require("tail captured n=" + std::to_string(n), sel.get((size_t) width) == want); const auto actual_scores = scores.get((size_t) n); bool exact_scores = true; for (int64_t j = 0; j < n; ++j) { diff --git a/src/kernels/quantize_act_parity.cpp b/src/kernels/quantize_act_parity.cpp index b2cd7ee4149..e9c794e22c4 100644 --- a/src/kernels/quantize_act_parity.cpp +++ b/src/kernels/quantize_act_parity.cpp @@ -318,10 +318,10 @@ int main(int argc, char** argv) { {0.0f, 0x0000u, "0.0"}, }; int fp16_bad = 0; - for (const F16Case& c : cases) { - const uint16_t got = strata::kernels::f16_from_f32(c.in); - if (got != c.want) { - std::printf(" *** %s: want 0x%04X got 0x%04X\n", c.what, c.want, got); + for (const F16Case& fc : cases) { + const uint16_t got = strata::kernels::f16_from_f32(fc.in); + if (got != fc.want) { + std::printf(" *** %s: want 0x%04X got 0x%04X\n", fc.what, fc.want, got); ++fp16_bad; } } @@ -388,16 +388,16 @@ int main(int argc, char** argv) { std::vector b127, b128; reference_q8_K(ka.data(), NK, r127, b127, -127.0f); reference_q8_K(ka.data(), NK, r128, b128, -128.0f); - double d = 0, m = 0; + double diff = 0, m = 0; long long qdiffer = 0; - for (long long i = 0; i < NK; ++i) { d += std::fabs(b127[(size_t) i] - b128[(size_t) i]); m += std::fabs(b127[(size_t) i]); } + for (long long i = 0; i < NK; ++i) { diff += std::fabs(b127[(size_t) i] - b128[(size_t) i]); m += std::fabs(b127[(size_t) i]); } for (size_t i = 4; i < r127.size(); ++i) if (r127[i] != r128[i]) ++qdiffer; // The expected size of the effect is exactly (1 - 127/128) = 0.78% of the magnitude, so the floor is // 0.5% - below what the rival constant must produce and far above any noise. The byte count is the // blunter evidence and is reported alongside it: 172,017 bytes differ, i.e. a third of the quants. std::printf(" %-26s %-4s (%.3f%% of magnitude, %lld bytes differ)\n", "-127 vs -128 scale observable", - d / m > 0.005 ? "yes" : "*** NO ***", 100.0 * d / m, qdiffer); - if (!(d / m > 0.005)) ++bad_k; + diff / m > 0.005 ? "yes" : "*** NO ***", 100.0 * diff / m, qdiffer); + if (!(diff / m > 0.005)) ++bad_k; } std::printf("\nquantize_act: %d failures over 5 Q8_0 + 4 Q8_K distributions\n", bad + bad_k); diff --git a/src/kernels/router_top10_parity.cpp b/src/kernels/router_top10_parity.cpp index ac7e652eb53..bcb34f2236b 100644 --- a/src/kernels/router_top10_parity.cpp +++ b/src/kernels/router_top10_parity.cpp @@ -150,11 +150,11 @@ int main(int argc, char** argv) { // distribution). Here k/n = 10/512 = 0.01953125, which is 320x the 2**-14 = 6.1035e-05 clamp, so no input // whatsoever can make this model's renormalisation clamp - not an extreme one, not an adversarial one. // Asserting the bound is worth more than a test that cannot reach it. - const double min_possible = (double) K / (double) NE; + constexpr double min_possible = (double) K / (double) NE; std::printf(" %-26s top-%d sum >= k/n = %.5f, clamp is %.3e -> margin %.0fx %s\n", "clamp reachability", K, min_possible, RENORM_CLAMP, min_possible / RENORM_CLAMP, min_possible > RENORM_CLAMP ? "(CLAMP IS UNREACHABLE, asserted)" : "*** REACHABLE ***"); - if (min_possible <= RENORM_CLAMP) { + if constexpr (min_possible <= RENORM_CLAMP) { std::fprintf(stderr, "the clamp CAN trigger, so it needs a test\n"); ++bad; } diff --git a/src/kernels/shared_expert_parity.cpp b/src/kernels/shared_expert_parity.cpp index 92de10ad00b..e469593a0aa 100644 --- a/src/kernels/shared_expert_parity.cpp +++ b/src/kernels/shared_expert_parity.cpp @@ -436,7 +436,7 @@ int main(int argc, char** argv) { } return y; }; - const std::vector want = ref(true, false); + const std::vector want_mc = ref(true, false); float *d_p = nullptr, *d_w = nullptr, *d_s = nullptr, *d_y = nullptr; check(cudaMalloc(&d_p, parts.size() * 4), "mc p"); @@ -447,12 +447,12 @@ int main(int argc, char** argv) { check(cudaMemcpy(d_w, w.data(), w.size() * 4, cudaMemcpyHostToDevice), "mc cw"); check(cudaMemcpy(d_s, shared.data(), shared.size() * 4, cudaMemcpyHostToDevice), "mc cs"); strata::kernels::moe_combine(d_p, d_w, d_s, d_y, n_embd2, k2, nullptr); - std::vector got((size_t) n_embd2); - check(cudaMemcpy(got.data(), d_y, got.size() * 4, cudaMemcpyDeviceToHost), "mc cy"); + std::vector got_mc((size_t) n_embd2); + check(cudaMemcpy(got_mc.data(), d_y, got_mc.size() * 4, cudaMemcpyDeviceToHost), "mc cy"); for (const auto& trap : {std::make_pair(false, false), std::make_pair(true, true)}) { const std::vector wrong = ref(trap.first, trap.second); - const double rel = rel_l1(want, wrong); + const double rel = rel_l1(want_mc, wrong); const char* what = (!trap.first && !trap.second) ? "routed sum left UNWEIGHTED is observable" : "shared output router-WEIGHTED is observable"; const bool visible = rel > 0.05; @@ -460,15 +460,15 @@ int main(int argc, char** argv) { if (!visible) ++bad; } - const double rel_k = rel_l1(want, got); + const double rel_k = rel_l1(want_mc, got_mc); std::printf(" %-42s rel %.3e\n", "moe_combine vs reference", rel_k); // Both sides accumulate in double over the same k terms, so this is order-only. if (!(rel_k <= 1e-6)) { std::printf(" *** over 1e-6 ***\n"); ++bad; } // a null `shared` must mean "no shared expert", not "add nothing but leave it undefined" strata::kernels::moe_combine(d_p, d_w, nullptr, d_y, n_embd2, k2, nullptr); - check(cudaMemcpy(got.data(), d_y, got.size() * 4, cudaMemcpyDeviceToHost), "mc cy2"); - const double rel_no_shared = rel_l1(want, got); + check(cudaMemcpy(got_mc.data(), d_y, got_mc.size() * 4, cudaMemcpyDeviceToHost), "mc cy2"); + const double rel_no_shared = rel_l1(want_mc, got_mc); const bool differs = rel_no_shared > 0.05; std::printf(" %-42s %s (%.2f%% apart - null shared really omits it)\n", "a null `shared` is observable", differs ? "yes" : "*** NO ***", rel_no_shared * 100); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index ce9be41317a..14d9e7186b4 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -1809,7 +1809,7 @@ int main(int argc, char** argv) { "--expert-cache\n"); return 2; } - strata::prefill::Prefill sp; + strata::prefill::Prefill prefill; void* borrow = nullptr; uint64_t borrow_bytes = 0; int32_t lend_first = -1; @@ -1829,7 +1829,7 @@ int main(int argc, char** argv) { : (uint64_t) k * (uint64_t) blob; } } - if (!sp.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, borrow, borrow_bytes)) { + if (!prefill.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, borrow, borrow_bytes)) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; } @@ -1847,7 +1847,7 @@ int main(int argc, char** argv) { ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 std::vector cur; - sp.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { + prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { std::vector nxt((size_t) T); for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)]; return mtp.prefill(R_rows, nxt.data(), T, p0, e); @@ -2033,7 +2033,7 @@ int main(int argc, char** argv) { } mrope_identity = !geni; } - sp.embd_rows = geni ? row_ptr.data() : nullptr; + prefill.embd_rows = geni ? row_ptr.data() : nullptr; if (n + max_new + 8 > o.max_context) { std::printf("ERR prompt (%lld tokens) + max_new (%lld) exceeds the context (%lld)\n", (long long) n, (long long) max_new, (long long) o.max_context); @@ -2086,7 +2086,7 @@ int main(int argc, char** argv) { } for (int64_t a0 = reuse; a0 < batched_end;) { const int64_t b0 = track ? std::min(batched_end, a0 + step) : batched_end; - if (!sp.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } + if (!prefill.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } if (track) { pcache.set(a0, ids.data() + a0, b0 - a0); pcache.checkpoint(b0); From 8d4210ff1abc345211a1695de9c79dbf5358150f Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 04:24:19 +0200 Subject: [PATCH 05/92] MTP draft head from Q6_K output heads The draft head copies rows of the model's output head, and the copy took rows in 16-byte units only. Q6_K rows of 2,560 values are 2,100 bytes, so MTP exited at startup with the Q6_K heads of unsloth's UD-IQ3_XXS, UD-IQ4_XS and UD-Q3_K_XL. Rows that are a multiple of 4 bytes are now copied in 4-byte units. --- include/strata/kernels/verify_kernels.hpp | 2 +- src/kernels/cuda/verify_kernels.cu | 20 +++++++++++++------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index d66093221d2..5def3a7c17f 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -65,7 +65,7 @@ void ident_hits(const int32_t* ids, int n, int32_t* slot, int32_t* dst, int32_t* void mtp_select(const float* R_src, int64_t R_stride, const int32_t* ids, const int32_t* row_dev, float* R_dst, int32_t* tok_dst, int32_t* out, int j, void* stream, const float* probs = nullptr, float* out_p = nullptr); -/// dst row i = src row ids[i] (row_bytes each, multiple of 16), for n rows. +/// dst row i = src row ids[i] (row_bytes each, a multiple of 4), for n rows. void gather_rows(const uint8_t* src, int64_t row_bytes, const int32_t* ids, int64_t n, uint8_t* dst, void* stream); /// ids[t] = table[ids[t]] for n entries (a subset index back to a token id). void map_ids(int32_t* ids, const int32_t* table, int n, void* stream); diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index f9abe84dcc3..bca54aee6f9 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -244,12 +244,13 @@ __global__ void ident_hits_kernel(const int32_t* __restrict__ ids, int n, int32_ if (i == 0) *count = n; } -__global__ void gather_rows_kernel(const uint4* __restrict__ src, long long row16, const int32_t* __restrict__ ids, - long long n, uint4* __restrict__ dst) { - const long long total = n * row16; +template +__global__ void gather_rows_kernel(const U* __restrict__ src, long long row_units, const int32_t* __restrict__ ids, + long long n, U* __restrict__ dst) { + const long long total = n * row_units; for (long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; i < total; i += (long long) gridDim.x * blockDim.x) { - const long long r = i / row16, o = i - r * row16; - dst[i] = src[(long long) ids[r] * row16 + o]; + const long long r = i / row_units, o = i - r * row_units; + dst[i] = src[(long long) ids[r] * row_units + o]; } } @@ -334,8 +335,13 @@ void mtp_select(const float* R_src, int64_t R_stride, const int32_t* ids, const } void gather_rows(const uint8_t* src, int64_t row_bytes, const int32_t* ids, int64_t n, uint8_t* dst, void* stream) { - if (row_bytes % 16 != 0) { std::fprintf(stderr, "gather_rows: row size must be a multiple of 16\n"); std::exit(1); } - gather_rows_kernel<<<48 * 8, 256, 0, (cudaStream_t) stream>>>((const uint4*) src, row_bytes / 16, ids, n, (uint4*) dst); + if (n <= 0) return; + if (row_bytes % 16 == 0) + gather_rows_kernel<<<48 * 8, 256, 0, (cudaStream_t) stream>>>((const uint4*) src, row_bytes / 16, ids, n, (uint4*) dst); + else if (row_bytes % 4 == 0) // e.g. Q6_K rows of 2,560 values: 2,100 bytes + gather_rows_kernel<<<48 * 8, 256, 0, (cudaStream_t) stream>>>((const uint32_t*) src, row_bytes / 4, ids, n, + (uint32_t*) dst); + else { std::fprintf(stderr, "gather_rows: row size must be a multiple of 4\n"); std::exit(1); } check("gather_rows"); } From 7317dadbd888d8190a5e0fe8fefa554d45b7af23 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 04:24:19 +0200 Subject: [PATCH 06/92] Speculation benchmarks on fixed text (--spec-follow) With --spec-follow, a generate run emits a given continuation instead of the window's argmax and accepts the drafts that match it. Runs with different --spec, --spec-min-p or draft-layer settings then process the same tokens and route the same experts, so their speeds compare within ~1% instead of the ~4% two identical free runs differ by once their text diverges at a near tie. The usage text also lists the MTP flags. --- src/program/generate.cpp | 52 ++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 14d9e7186b4..eaed5acd7d1 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -213,6 +213,9 @@ struct Options { int spec = 0; std::string spec_oracle; int spec_corrupt = 0; + /// Benchmarks: the run emits this continuation (token ids) instead of the window's argmax, and a draft is + /// accepted when it matches it, so runs with different speculation settings process the same text. + std::string spec_follow; /// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it. std::string mtp; int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell) @@ -315,6 +318,14 @@ void usage() { " so switching back to them continues where they were (default 16384)\n" " --feed-max N --serve: new prompt parts and last prompt chunks of up to N tokens run\n" " through verify windows instead of the batched prompt path (default 512)\n" + " --spec T verify windows of up to T tokens: the last one and T-1 MTP drafts (2..8)\n" + " --spec-min-p P a draft enters the window only while it and the drafts before it have at\n" + " least probability P under the draft layer (default 0: always T-1 drafts)\n" + " --mtp DIR the MTP draft layer's runtime files (tools/mtp_rt.py)\n" + " --mtp-window N the draft layer attends to the last N cells (default 32768; 0 = every cell)\n" + " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" + " accept the drafts that match it, so speculation settings compare on the\n" + " same text (not --serve)\n" " --no-token-graph A/B: two graphs per layer (the host launches each) instead of one per token\n" " --no-fused-gr A/B: the six-kernel hyper-connection read and a separate write (native)\n" " --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n" @@ -541,6 +552,7 @@ int main(int argc, char** argv) { else if (a == "--dump-final-r") o.dump_final_r = next("--dump-final-r"); else if (a == "--spec") o.spec = std::atoi(next("--spec")); else if (a == "--spec-oracle") o.spec_oracle = next("--spec-oracle"); + else if (a == "--spec-follow") o.spec_follow = next("--spec-follow"); else if (a == "--spec-corrupt") o.spec_corrupt = std::atoi(next("--spec-corrupt")); else if (a == "--mtp") o.mtp = next("--mtp"); else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window")); @@ -2532,16 +2544,24 @@ int main(int argc, char** argv) { const bool ended = o.stop_eos && !produced.empty() && std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) produced.back()) != o.eos_ids.end(); if (spec_pos > 0 && (int64_t) produced.size() < o.max_new && !ended) { - std::vector oracle; - if (!o.spec_oracle.empty()) { - std::ifstream in(o.spec_oracle); + auto read_ids = [](const std::string& path, std::vector& ids) { + std::ifstream in(path); std::string text((std::istreambuf_iterator(in)), std::istreambuf_iterator()); std::string e; - if (!in || !parse_i64_list(text.c_str(), oracle, e)) { - std::fprintf(stderr, "strata generate: cannot read --spec-oracle %s\n", o.spec_oracle.c_str()); - return 2; - } + return (bool) in && parse_i64_list(text.c_str(), ids, e); + }; + std::vector oracle, follow; + if (!o.spec_oracle.empty() && !read_ids(o.spec_oracle, oracle)) { + std::fprintf(stderr, "strata generate: cannot read --spec-oracle %s\n", o.spec_oracle.c_str()); + return 2; } + if (!o.spec_follow.empty() && !read_ids(o.spec_follow, follow)) { + std::fprintf(stderr, "strata generate: cannot read --spec-follow %s\n", o.spec_follow.c_str()); + return 2; + } + // --spec-follow: the run ends with the continuation + const int64_t max_new = follow.empty() ? o.max_new : std::min(o.max_new, (int64_t) follow.size()); + int64_t follow_differ = 0, follow_emitted = 0; if (thits.d_res == nullptr) { std::fprintf(stderr, "strata generate: --spec needs the device residency table (--expert-profile, " "--expert-cache and the token graph)\n"); @@ -2654,11 +2674,12 @@ int main(int argc, char** argv) { return 1; } std::vector window((size_t) o.spec), outv((size_t) o.spec); + std::vector fdiff((size_t) o.spec, 0); std::vector accepted_hist((size_t) o.spec, 0); int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; const int64_t misses0 = drive.d.multi_misses, entries0 = drive.d.multi_entries; - while ((int64_t) produced.size() < o.max_new) { + while ((int64_t) produced.size() < max_new) { const Clock::time_point t0 = Clock::now(); int T = o.spec; if (use_mtp && o.spec_min_p > 0.0) { @@ -2692,6 +2713,13 @@ int main(int argc, char** argv) { drive.d.fail ? drive.d.fail : "(no message)"); return 1; } + if (!follow.empty()) { // the continuation stands in for the argmax + const size_t k0 = produced.size(); + for (int i = 0; i < T && k0 + (size_t) i < follow.size(); ++i) { + fdiff[(size_t) i] = outv[(size_t) i] != (int32_t) follow[k0 + (size_t) i]; + outv[(size_t) i] = (int32_t) follow[k0 + (size_t) i]; + } + } int a = 0; while (a < T - 1 && window[(size_t) a + 1] == outv[(size_t) a]) ++a; if (first_window) { @@ -2714,16 +2742,17 @@ int main(int argc, char** argv) { drafts_ok += a; ++accepted_hist[(size_t) a]; bool eos = false; - for (int i = 0; i <= a && (int64_t) produced.size() < o.max_new && !eos; ++i) { + for (int i = 0; i <= a && (int64_t) produced.size() < max_new && !eos; ++i) { produced.push_back(outv[(size_t) i]); eos = o.stop_eos && std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) outv[(size_t) i]) != o.eos_ids.end(); + if (!follow.empty()) { follow_differ += fdiff[(size_t) i]; ++follow_emitted; } } if (eos) { if (adapt_thr.joinable()) adapt_thr.join(); total_ms += std::chrono::duration(Clock::now() - t0).count(); break; } - const bool drafted = !use_mtp || (int64_t) produced.size() >= o.max_new || + const bool drafted = !use_mtp || (int64_t) produced.size() >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p); if (adapt_thr.joinable()) adapt_thr.join(); if (!adapt_ok) return 1; @@ -2742,6 +2771,9 @@ int main(int argc, char** argv) { "speculation", (long long) rounds, o.spec, (long long) drafts_ok, (long long) drafts_total, drafts_total > 0 ? (double) drafts_ok / (double) drafts_total : 0.0, rounds > 0 ? (double) (drafts_ok + rounds) / (double) rounds : 0.0); + if (!follow.empty()) + std::printf("%-24s %lld of %lld emitted tokens differ from the argmax\n", "follow", + (long long) follow_differ, (long long) follow_emitted); if (o.spec_min_p > 0.0) { std::printf("%-24s", "window sizes"); for (size_t i = 1; i < window_hist.size(); ++i) std::printf(" T%zu:%lld", i, (long long) window_hist[i]); From 7efc36cf3bc436a54093f05bf2fd09ef5de3b00b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 05:55:15 +0200 Subject: [PATCH 07/92] One adaptive VRAM tier for both decode loops; routing concentration in --stats - The serve loop and the speculative loop each carried the same adaptive expert tier (ranking, refill copies, admission once landed); it is now strata::core::AdaptiveTier. - --stats prints the share of the routed entries the N most-routed experts of the run take, which sizes what a larger expert cache could serve. --- CMakeLists.txt | 2 +- include/strata/core/adaptive_tier.hpp | 54 +++++++++ include/strata/core/expert_source.hpp | 3 + src/core/adaptive_tier.cpp | 93 +++++++++++++++ src/core/expert_source.cpp | 3 + src/program/generate.cpp | 166 ++++++-------------------- 6 files changed, 188 insertions(+), 133 deletions(-) create mode 100644 include/strata/core/adaptive_tier.hpp create mode 100644 src/core/adaptive_tier.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index cdde42086fb..3e3c5b2a389 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -241,7 +241,7 @@ if(STRATA_ENABLE_CUDA) # about arithmetic, and this one knows the ORDER the arithmetic happens in. add_library(strata_engine STATIC src/core/layer.cpp src/core/session.cpp src/core/expert_source.cpp src/core/expert_cache.cpp src/core/native_head.cpp src/core/native_dense.cpp src/core/verify.cpp src/core/mtp.cpp - src/core/prompt_cache.cpp) + src/core/prompt_cache.cpp src/core/adaptive_tier.cpp) target_include_directories(strata_engine PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include) target_link_libraries(strata_engine PUBLIC strata_core strata_kernels strata_kernels_cpu CUDA::cudart) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp new file mode 100644 index 00000000000..d3f52bf7f1a --- /dev/null +++ b/include/strata/core/adaptive_tier.hpp @@ -0,0 +1,54 @@ +// include/strata/core/adaptive_tier.hpp - plan v0.3 P6: the VRAM expert tier follows the conversation. +// +// Every few rounds the missing experts routed most often (decayed counts, at least twice) take the slots of their +// layer's least-routed residents when they were routed clearly more often. The copies run on their own stream +// between rounds; an evicted expert is a CPU miss at once, a new one resident once its copy has landed +// (`apply_pending`). Both decode loops (--serve and the speculative loop) use it. +#pragma once + +#include "strata/core/expert_cache.hpp" +#include "strata/core/expert_source.hpp" + +#include + +#include +#include +#include +#include + +namespace strata::core { + +class AdaptiveTier { +public: + AdaptiveTier() = default; + ~AdaptiveTier(); + AdaptiveTier(const AdaptiveTier&) = delete; + AdaptiveTier& operator=(const AdaptiveTier&) = delete; + + /// `host_res` is the residency table (n_layers x n_expert, slot or kNotResident) the dispatch reads, `d_res` + /// its device copy; up to `max_swaps` experts move per call. + bool init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, int64_t n_layers, + int64_t n_expert, int max_swaps, std::string& err); + + /// Ranks and submits this call's swaps, then decays `usage` (n_layers x n_expert). Nothing to do while the + /// previous swaps are in flight. False when a copy could not be submitted. + bool adapt(std::vector& usage, std::string& err); + /// Admits the landed swaps into `host_res` and `d_res`; `wait` blocks until they have landed. + void apply_pending(bool wait); + + int64_t swaps = 0; ///< experts swapped in + double ms = 0; ///< host time in `adapt` + +private: + ExpertCache* cache_ = nullptr; + ExpertSource* src_ = nullptr; + std::vector* res_ = nullptr; + int32_t* d_res_ = nullptr; + int64_t n_layers_ = 0, n_expert_ = 0; + int max_swaps_ = 0; + std::vector> pending_; // (residency index, slot) once the copies have landed + cudaStream_t stream_ = nullptr; + cudaEvent_t ev_ = nullptr; +}; + +} // namespace strata::core diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index cdc3ad511f6..60c994a94b1 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -212,6 +212,9 @@ struct ExpertDispatch { std::vector usage; int64_t multi_misses = 0; ///< distinct (layer, expert) pairs the CPU computed in verify windows int64_t multi_entries = 0; ///< routed (token, expert) entries the CPU served in verify windows + /// Routed (token, expert) entries per (layer, expert) in verify windows, when the caller sizes it: how many of + /// the entries a cache of the N most-routed experts would serve. + std::vector routed; /// Set when `dispatch` could not produce an answer. The loop itself has no error channel, so this is /// where a source failure surfaces: the driver checks it after `session_loop` returns rather than the /// engine computing from a half-filled `parts`. diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp new file mode 100644 index 00000000000..41c1c282ce6 --- /dev/null +++ b/src/core/adaptive_tier.cpp @@ -0,0 +1,93 @@ +// src/core/adaptive_tier.cpp - see include/strata/core/adaptive_tier.hpp. +#include "strata/core/adaptive_tier.hpp" + +#include "strata/kernels/cpu/expert_layout.hpp" + +#include +#include + +namespace strata::core { + +AdaptiveTier::~AdaptiveTier() { + if (stream_) cudaStreamSynchronize(stream_); + if (ev_) cudaEventDestroy(ev_); + if (stream_) cudaStreamDestroy(stream_); +} + +bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, + int64_t n_layers, int64_t n_expert, int max_swaps, std::string& err) { + cache_ = &cache; + src_ = &src; + res_ = &host_res; + d_res_ = d_res; + n_layers_ = n_layers; + n_expert_ = n_expert; + max_swaps_ = max_swaps; + if (cudaStreamCreateWithFlags(&stream_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&ev_, cudaEventDisableTiming) != cudaSuccess) { + err = "adaptive tier: cannot create the refill stream"; + return false; + } + return true; +} + +bool AdaptiveTier::adapt(std::vector& usage, std::string& err) { + const auto t0 = std::chrono::steady_clock::now(); + if (!pending_.empty()) return true; // the previous swaps are still in flight + struct Swap { float gain; int32_t layer, in, out; }; + std::vector todo; + std::vector> cand, vict; + std::vector& res = *res_; + for (int64_t l = 0; l < n_layers_; ++l) { + cand.clear(); + vict.clear(); + const float* u = usage.data() + l * n_expert_; + const int32_t* r = res.data() + l * n_expert_; + for (int32_t e = 0; e < (int32_t) n_expert_; ++e) { + if (r[e] < 0) { if (u[e] >= 2.0f) cand.emplace_back(u[e], e); } + else vict.emplace_back(u[e], e); + } + if (cand.empty() || vict.empty()) continue; + std::sort(cand.begin(), cand.end(), [](auto& a, auto& b) { return a.first > b.first; }); + const size_t nc = std::min(cand.size(), vict.size()); + std::partial_sort(vict.begin(), vict.begin() + (ptrdiff_t) nc, vict.end(), + [](auto& a, auto& b) { return a.first < b.first; }); + for (size_t i = 0; i < nc; ++i) { + if (cand[i].first < vict[i].first + 1.5f) break; + todo.push_back({cand[i].first - vict[i].first, (int32_t) l, cand[i].second, vict[i].second}); + } + } + std::sort(todo.begin(), todo.end(), [](const Swap& a, const Swap& b) { return a.gain > b.gain; }); + if ((int) todo.size() > max_swaps_) todo.resize((size_t) max_swaps_); + const auto& lay = strata::kernels::cpu::expert_layout(); + for (const Swap& s : todo) { + const size_t in = (size_t) (s.layer * n_expert_ + s.in), out = (size_t) (s.layer * n_expert_ + s.out); + const int32_t slot = res[out]; + const uint8_t* b = src_->blob(s.layer, s.in); + // asynchronous: the copies run while the MTP drafts; the next window waits for them + if (slot < 0 || b == nullptr || cudaMemcpyAsync(cache_->device_slot(slot), b, (size_t) lay.blob_bytes(s.layer), + cudaMemcpyHostToDevice, stream_) != cudaSuccess) { + err = "adaptive tier: a refill copy failed"; + return false; + } + res[out] = kNotResident; // evicted now: the CPU computes it meanwhile + pending_.emplace_back((int32_t) in, slot); // resident once the copy has landed + } + if (!todo.empty()) cudaEventRecord(ev_, stream_); + for (float& v : usage) v *= 0.7f; + swaps += (int64_t) todo.size(); + ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + return true; +} + +void AdaptiveTier::apply_pending(bool wait) { + if (pending_.empty()) return; + if (wait) cudaEventSynchronize(ev_); + else if (cudaEventQuery(ev_) != cudaSuccess) return; + std::vector& res = *res_; + for (const auto& [i, slot] : pending_) res[(size_t) i] = slot; + pending_.clear(); + if (d_res_ != nullptr) cudaMemcpy(d_res_, res.data(), res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); +} + +} // namespace strata::core diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index b44fc00b8cd..bcd18c683a4 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -269,6 +269,9 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (!d.usage.empty()) for (int64_t i = 0; i < n_tok * k; ++i) if (ids[i] >= 0 && ids[i] < d.n_expert) d.usage[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]] += 1.0f; + if (!d.routed.empty()) + for (int64_t i = 0; i < n_tok * k; ++i) + if (ids[i] >= 0 && ids[i] < d.n_expert) ++d.routed[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]]; // ---- plan v0.3 P6: the GPU's share, decided and published FIRST so the GPU starts while the CPU works. // Distinct experts in routing order; resident ones and the last pcie_num/256 of the missed ones go to the GPU. const int64_t n = n_tok * k; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index eaed5acd7d1..64ab25b9891 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -15,6 +15,7 @@ // (`phase-2-correct-engine.md:5-9`). What it is FOR is the honest tok/s figure and the logit dump. #include "strata/artifact/gguf_split.hpp" +#include "strata/core/adaptive_tier.hpp" #include "strata/core/expert_cache.hpp" #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" @@ -58,6 +59,7 @@ #include #include #include +#include #include #include #include @@ -1776,6 +1778,17 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: token graph hit path: %lld resident experts, decided on the device\n", (long long) resident); } + // plan v0.3 P6: the VRAM tier follows the conversation in both decode loops (--serve and the speculative loop): + // swaps every --adapt-every rounds on the adapt thread, admitted once their copies have landed + strata::core::AdaptiveTier tier; + if (!host_res.empty() && o.adapt_every > 0 && o.adapt_swaps > 0 && + !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } + std::string adapt_err; + auto adapt = [&]() -> bool { return tier.adapt(drive.d.usage, adapt_err); }; + auto apply_pending = [&](bool wait) { tier.apply_pending(wait); }; if (!o.no_capture && !o.no_token_graph && layer_dump == nullptr && half_dump == nullptr && (hit_fn == nullptr || thits.on()) && !native_pack) { if (!strata::core::session_capture_token(wt, g, ss, d_parts, loop_scratch.y_miss, loop_scratch.parts_bytes, @@ -1867,66 +1880,6 @@ int main(int argc, char** argv) { drive.d.plan = ver.plan_sink(); drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5))); if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); - cudaStream_t adapt_stream = nullptr; - if (cudaStreamCreateWithFlags(&adapt_stream, cudaStreamNonBlocking) != cudaSuccess) { - std::fprintf(stderr, "strata serve: cannot create the refill stream\n"); - return 1; - } - // plan v0.3 P6: swaps in flight - (residency index, slot) admitted when adapt_ev has completed - std::vector> pending; - cudaEvent_t adapt_ev = nullptr; - cudaEventCreateWithFlags(&adapt_ev, cudaEventDisableTiming); - auto apply_pending = [&](bool wait) { - if (pending.empty()) return; - if (wait) cudaEventSynchronize(adapt_ev); - else if (cudaEventQuery(adapt_ev) != cudaSuccess) return; - for (const auto& [i, slot] : pending) host_res[(size_t) i] = slot; - pending.clear(); - if (d_res != nullptr) - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - }; - // the VRAM tier follows the conversation (the same rule as the speculative loop below) - auto adapt = [&]() -> bool { - if (!pending.empty()) return true; // the previous swaps are still in flight - struct Swap { float gain; int32_t layer, in, out; }; - std::vector swaps; - std::vector> cand, vict; - for (int64_t l = 0; l < g.n_layers; ++l) { - cand.clear(); - vict.clear(); - const float* u = drive.d.usage.data() + l * g.n_expert; - const int32_t* r = host_res.data() + l * g.n_expert; - for (int32_t e = 0; e < (int32_t) g.n_expert; ++e) { - if (r[e] < 0) { if (u[e] >= 2.0f) cand.emplace_back(u[e], e); } - else vict.emplace_back(u[e], e); - } - if (cand.empty() || vict.empty()) continue; - std::sort(cand.begin(), cand.end(), [](auto& a, auto& b) { return a.first > b.first; }); - const size_t nc = std::min(cand.size(), vict.size()); - std::partial_sort(vict.begin(), vict.begin() + (ptrdiff_t) nc, vict.end(), - [](auto& a, auto& b) { return a.first < b.first; }); - for (size_t i = 0; i < nc; ++i) { - if (cand[i].first < vict[i].first + 1.5f) break; - swaps.push_back({cand[i].first - vict[i].first, (int32_t) l, cand[i].second, vict[i].second}); - } - } - std::sort(swaps.begin(), swaps.end(), [](const Swap& a, const Swap& b) { return a.gain > b.gain; }); - if ((int) swaps.size() > o.adapt_swaps) swaps.resize((size_t) o.adapt_swaps); - for (const Swap& s : swaps) { - const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out; - const int32_t slot = host_res[out]; - const uint8_t* b = srcp->blob(s.layer, s.in); - if (slot < 0 || b == nullptr || - cudaMemcpyAsync(xcache.device_slot(slot), b, (size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer), - cudaMemcpyHostToDevice, adapt_stream) != cudaSuccess) - return false; - host_res[out] = strata::core::kNotResident; // evicted now: the CPU computes it meanwhile - pending.emplace_back((int32_t) in, slot); // resident once the copy has landed - } - if (!swaps.empty()) cudaEventRecord(adapt_ev, adapt_stream); - for (float& v : drive.d.usage) v *= 0.7f; - return true; - }; // ---- the prompt cache (strata/core/prompt_cache.hpp) std::vector kv_states; for (int64_t i = 0; i < g.n_qsa_layers(); ++i) kv_states.push_back(&ss.qsa_states[i]); @@ -2590,76 +2543,6 @@ int main(int argc, char** argv) { if (drive.d.pcie_num > 256) drive.d.pcie_num = 256; const int64_t pcie0 = drive.d.pcie_experts; if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); - int64_t swaps_total = 0; - double ms_adapt = 0; - cudaStream_t adapt_stream = nullptr; - if (!drive.d.usage.empty() && cudaStreamCreateWithFlags(&adapt_stream, cudaStreamNonBlocking) != cudaSuccess) { - std::fprintf(stderr, "strata generate: cannot create the refill stream\n"); - return 1; - } - // plan v0.3 P6: swaps in flight - (residency index, slot) admitted when adapt_ev has completed - std::vector> pending; - cudaEvent_t adapt_ev = nullptr; - cudaEventCreateWithFlags(&adapt_ev, cudaEventDisableTiming); - auto apply_pending = [&](bool wait) { - if (pending.empty()) return; - if (wait) cudaEventSynchronize(adapt_ev); - else if (cudaEventQuery(adapt_ev) != cudaSuccess) return; - for (const auto& [i, slot] : pending) host_res[(size_t) i] = slot; - pending.clear(); - if (d_res != nullptr) - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - }; - // Plan v0.3 P6: the VRAM tier follows the conversation. Candidates are missing experts routed at least - // twice (decayed); each is paired with its layer's least-routed resident expert and swapped when it was - // routed clearly more often. Copies run between rounds, when the GPU is idle. - auto adapt = [&]() -> bool { - const Clock::time_point ta = Clock::now(); - if (!pending.empty()) return true; // the previous swaps are still in flight - struct Swap { float gain; int32_t layer, in, out; }; - std::vector swaps; - std::vector> cand, vict; - for (int64_t l = 0; l < g.n_layers; ++l) { - cand.clear(); - vict.clear(); - const float* u = drive.d.usage.data() + l * g.n_expert; - const int32_t* r = host_res.data() + l * g.n_expert; - for (int32_t e = 0; e < (int32_t) g.n_expert; ++e) { - if (r[e] < 0) { if (u[e] >= 2.0f) cand.emplace_back(u[e], e); } - else vict.emplace_back(u[e], e); - } - if (cand.empty() || vict.empty()) continue; - std::sort(cand.begin(), cand.end(), [](auto& a, auto& b) { return a.first > b.first; }); - const size_t nc = std::min(cand.size(), vict.size()); - std::partial_sort(vict.begin(), vict.begin() + (ptrdiff_t) nc, vict.end(), - [](auto& a, auto& b) { return a.first < b.first; }); - for (size_t i = 0; i < nc; ++i) { - if (cand[i].first < vict[i].first + 1.5f) break; - swaps.push_back({cand[i].first - vict[i].first, (int32_t) l, cand[i].second, vict[i].second}); - } - } - std::sort(swaps.begin(), swaps.end(), [](const Swap& a, const Swap& b) { return a.gain > b.gain; }); - if ((int) swaps.size() > o.adapt_swaps) swaps.resize((size_t) o.adapt_swaps); - for (const Swap& s : swaps) { - const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out; - const int32_t slot = host_res[out]; - const uint8_t* b = srcp->blob(s.layer, s.in); - // asynchronous: the copies run while the MTP drafts; the next window waits for them - if (slot < 0 || b == nullptr || - cudaMemcpyAsync(xcache.device_slot(slot), b, (size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer), - cudaMemcpyHostToDevice, adapt_stream) != cudaSuccess) { - std::fprintf(stderr, "strata generate: an adaptive refill failed\n"); - return false; - } - host_res[out] = strata::core::kNotResident; // evicted now: the CPU computes it meanwhile - pending.emplace_back((int32_t) in, slot); // resident once the copy has landed - } - if (!swaps.empty()) cudaEventRecord(adapt_ev, adapt_stream); - for (float& v : drive.d.usage) v *= 0.7f; - swaps_total += (int64_t) swaps.size(); - ms_adapt += std::chrono::duration(Clock::now() - ta).count(); - return true; - }; int64_t p = spec_pos; int32_t x = (int32_t) tok; std::vector drafts((size_t) o.spec, 0); @@ -2679,6 +2562,7 @@ int main(int argc, char** argv) { int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; const int64_t misses0 = drive.d.multi_misses, entries0 = drive.d.multi_entries; + if (o.stats) drive.d.routed.assign((size_t) (g.n_layers * g.n_expert), 0); while ((int64_t) produced.size() < max_new) { const Clock::time_point t0 = Clock::now(); int T = o.spec; @@ -2755,7 +2639,10 @@ int main(int argc, char** argv) { const bool drafted = !use_mtp || (int64_t) produced.size() >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p); if (adapt_thr.joinable()) adapt_thr.join(); - if (!adapt_ok) return 1; + if (!adapt_ok) { + std::fprintf(stderr, "strata generate: %s\n", adapt_err.c_str()); + return 1; + } if (!drafted) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; @@ -2801,7 +2688,22 @@ int main(int argc, char** argv) { drive.d.ms_run / rounds); if (rounds > 0 && !drive.d.usage.empty()) std::printf("%-24s %lld experts swapped into the VRAM tier (every %d rounds, %.3f ms/round)\n", "adaptive tier", - (long long) swaps_total, o.adapt_every, ms_adapt / rounds); + (long long) tier.swaps, o.adapt_every, tier.ms / rounds); + if (rounds > 0 && !drive.d.routed.empty()) { + // the share of the routed entries the N most-routed experts of this run take + std::vector c = drive.d.routed; + std::sort(c.begin(), c.end(), std::greater()); + double total = 0; + for (uint32_t v : c) total += v; + std::printf("%-24s", "routing concentration"); + double acc = 0; + size_t at = 0; + for (size_t n : {1000, 2000, 4000, 6000, 8000, 10000, 12000, 16000, 20000}) { + for (; at < n && at < c.size(); ++at) acc += c[at]; + std::printf(" %zu:%.3f", n, total > 0 ? acc / total : 0.0); + } + std::printf(" (top-N experts' share of %.0f routed entries)\n", total); + } if (rounds > 0 && drive.d.pcie_num > 0) std::printf("%-24s %.2f distinct experts per layer read over PCIe (share %d/256 of the misses)\n", "pcie experts", (double) (drive.d.pcie_experts - pcie0) / (double) (rounds * g.n_layers), From cce62942c1ea7cda461e9a5bbd31d51e5c8410f2 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 07:13:05 +0200 Subject: [PATCH 08/92] A second GPU as another expert tier (--second-gpu) The GPU that holds the expert cache is one per-layer bottleneck and the CPU pool, which computes the missed experts, the other. With --second-gpu N, device N holds more experts (the profile's pairs the first cache lacks, then empty slots its own adaptive tier fills with the conversation's) and computes its share of each verify-window layer while the CPU pool computes the rest. It never spins on host memory: per layer the host launches one captured graph (the plan copied in by a kernel, the activations read from mapped memory, the grouped expert kernels of the first GPU's hit path, the rows written to mapped memory) before the pool and collects the rows after it, so a GPU that also drives the display keeps its latency. A layer whose misses are below --second-gpu-min-mb (4) stays on the CPU, which is then quicker than the ~100 us round trip. UD-Q4_K_XL, RTX 3090 + RTX 5070 Ti (12.6 GiB of experts): generation +9% on a 3000-token thinking answer, +14-19% on code; IQ3_XXS, whose rounds wait on the first GPU, gains nothing. The expert arena is now one cudaHostAlloc block per layer instead of a registered reservation. With a second CUDA context in the process, Windows pins at most ~42 GiB through cudaHostRegister, and past that every CUDA allocation fails; allocated blocks reach ~75 GiB with one GPU or two. The layers past the budget share one locked, pageable range as before. --- CMakeLists.txt | 2 +- include/strata/core/adaptive_tier.hpp | 42 ++++-- include/strata/core/expert_source.hpp | 32 +++-- include/strata/core/pinned.hpp | 4 + include/strata/core/second_gpu.hpp | 62 +++++++++ src/core/adaptive_tier.cpp | 90 +++++++++---- src/core/expert_source.cpp | 182 ++++++++++++++++++-------- src/core/pinned.cu | 10 +- src/core/second_gpu.cpp | 170 ++++++++++++++++++++++++ src/program/generate.cpp | 121 ++++++++++++++++- 10 files changed, 602 insertions(+), 113 deletions(-) create mode 100644 include/strata/core/second_gpu.hpp create mode 100644 src/core/second_gpu.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 3e3c5b2a389..1ce11f23e92 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -241,7 +241,7 @@ if(STRATA_ENABLE_CUDA) # about arithmetic, and this one knows the ORDER the arithmetic happens in. add_library(strata_engine STATIC src/core/layer.cpp src/core/session.cpp src/core/expert_source.cpp src/core/expert_cache.cpp src/core/native_head.cpp src/core/native_dense.cpp src/core/verify.cpp src/core/mtp.cpp - src/core/prompt_cache.cpp src/core/adaptive_tier.cpp) + src/core/prompt_cache.cpp src/core/adaptive_tier.cpp src/core/second_gpu.cpp) target_include_directories(strata_engine PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include) target_link_libraries(strata_engine PUBLIC strata_core strata_kernels strata_kernels_cpu CUDA::cudart) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp index d3f52bf7f1a..a91c7cf46bd 100644 --- a/include/strata/core/adaptive_tier.hpp +++ b/include/strata/core/adaptive_tier.hpp @@ -1,9 +1,10 @@ -// include/strata/core/adaptive_tier.hpp - plan v0.3 P6: the VRAM expert tier follows the conversation. +// include/strata/core/adaptive_tier.hpp - plan v0.3 P6: a VRAM expert tier follows the conversation. // -// Every few rounds the missing experts routed most often (decayed counts, at least twice) take the slots of their -// layer's least-routed residents when they were routed clearly more often. The copies run on their own stream -// between rounds; an evicted expert is a CPU miss at once, a new one resident once its copy has landed -// (`apply_pending`). Both decode loops (--serve and the speculative loop) use it. +// Every few rounds the missing experts routed most often (decayed counts, at least twice) move into the tier: into +// their layer's empty slots first, then into the slots of the layer's least-routed residents when they were routed +// clearly more often. The copies run on their own stream between rounds; an evicted expert is a miss at once, a new +// one resident once its copy has landed (`apply_pending`). Both decode loops (--serve and the speculative loop) use +// it, for the first GPU's cache and for a second GPU's. #pragma once #include "strata/core/expert_cache.hpp" @@ -26,18 +27,27 @@ class AdaptiveTier { AdaptiveTier& operator=(const AdaptiveTier&) = delete; /// `host_res` is the residency table (n_layers x n_expert, slot or kNotResident) the dispatch reads, `d_res` - /// its device copy; up to `max_swaps` experts move per call. + /// its device copy (may be null); up to `max_moves` experts move per call. `device`: the cache's GPU (its + /// stream and copies), `main_device` current again afterwards. bool init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, int64_t n_layers, - int64_t n_expert, int max_swaps, std::string& err); + int64_t n_expert, int max_moves, std::string& err, int device = -1, int main_device = 0); + bool on() const { return res_ != nullptr; } + /// An empty slot sized for `layer`'s experts. + void add_free(int64_t layer, int32_t slot) { free_[(size_t) layer].push_back(slot); } + int64_t free_slots() const; + /// A tier in front of this one: the experts it holds or is loading are not candidates here, and this tier's + /// copies of them are evicted first. + void set_upper(const AdaptiveTier* upper) { upper_ = upper; } - /// Ranks and submits this call's swaps, then decays `usage` (n_layers x n_expert). Nothing to do while the - /// previous swaps are in flight. False when a copy could not be submitted. - bool adapt(std::vector& usage, std::string& err); - /// Admits the landed swaps into `host_res` and `d_res`; `wait` blocks until they have landed. + /// Ranks and submits this call's moves, then decays `usage` (n_layers x n_expert) unless `decay` is false (a + /// tier ranked before another on the same counts). Nothing to do while the previous moves are in flight. + /// False when a copy could not be submitted. + bool adapt(std::vector& usage, std::string& err, bool decay = true); + /// Admits the landed moves into `host_res` (and `d_res`); `wait` blocks until they have landed. void apply_pending(bool wait); - int64_t swaps = 0; ///< experts swapped in - double ms = 0; ///< host time in `adapt` + int64_t swaps = 0, fills = 0; ///< experts moved into an occupied / an empty slot + double ms = 0; ///< host time in `adapt` private: ExpertCache* cache_ = nullptr; @@ -45,7 +55,11 @@ class AdaptiveTier { std::vector* res_ = nullptr; int32_t* d_res_ = nullptr; int64_t n_layers_ = 0, n_expert_ = 0; - int max_swaps_ = 0; + int max_moves_ = 0; + int dev_ = -1, main_ = 0; + const AdaptiveTier* upper_ = nullptr; + std::vector upper_has_; // per (layer, expert), rebuilt each call + std::vector> free_; // per layer std::vector> pending_; // (residency index, slot) once the copies have landed cudaStream_t stream_ = nullptr; cudaEvent_t ev_ = nullptr; diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index 60c994a94b1..283f6cf4308 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -31,6 +31,8 @@ namespace strata::core { +class SecondGpu; + /// Where one routed expert's bytes come from. /// /// Phase 2 has NO cache (`phase-2-correct-engine.md`: hit rate `h = 0`), so the only implementation is a @@ -215,6 +217,15 @@ struct ExpertDispatch { /// Routed (token, expert) entries per (layer, expert) in verify windows, when the caller sizes it: how many of /// the entries a cache of the N most-routed experts would serve. std::vector routed; + /// A second GPU's expert tier (verify windows) and its residency table (n_layers x n_expert, slot or -1): the + /// experts it holds and the first GPU's cache does not are computed there, beside the CPU pool. + SecondGpu* gpu2 = nullptr; + const int32_t* host_res2 = nullptr; + int64_t gpu2_entries = 0; ///< routed entries the second GPU served + int64_t gpu2_skipped = 0; ///< layers the CPU took the second GPU's share of (it was quicker) + bool gpu2_used = false; ///< this layer's decision + uint64_t gpu2_min_bytes = 0; ///< a layer's miss bytes from which the second GPU takes its share + std::string gpu2_err; ///< `fail` points here when the second GPU failed /// Set when `dispatch` could not produce an answer. The loop itself has no error channel, so this is /// where a source failure surfaces: the driver checks it after `session_loop` returns rather than the /// engine computing from a half-filled `parts`. @@ -311,9 +322,12 @@ class FileExpertSource : public ExpertSource { // ordinary (anonymous, resident) memory; the engine reads `MapViewOfFile`. That is the whole difference, and it // is why this class exists. // -// It reads `experts.bin` into a `PinnedArena` once at startup, so the expert stream comes from anonymous memory -// the OS has no cheaper reason to evict. `PinnedArena` also tries `cudaHostRegister`, which the GPU needs for -// Phase 3's cache fills and the CPU/PCIe miss split - but registration is best-effort and reported, not assumed. +// It reads the experts once at startup into anonymous memory the OS has no cheaper reason to evict: one pinned +// block per layer (cudaHostAlloc, portable and mapped), which the GPUs need for cache fills, the prompt path and the +// PCIe share of the misses. Registering one reserved range instead (cudaHostRegister) is limited to ~42 GiB on +// Windows once a second GPU's context exists, and past that limit every CUDA allocation fails; allocated blocks reach +// ~75 GiB with one or two GPUs. The layers past the pinning budget share one locked, pageable range; the note says +// how many layers were pinned. class ArenaExpertSource : public ExpertSource { public: ArenaExpertSource() = default; @@ -329,7 +343,7 @@ class ArenaExpertSource : public ExpertSource { void set_gguf(const std::vector& shards) { gguf_ = shards; } void close(); - bool mapped() const { return base_ != nullptr; } + bool mapped() const { return !layer_base_.empty(); } int64_t blobs() const { return blobs_; } const uint8_t* blob(int64_t layer, int64_t expert) override; int64_t reads() const { return reads_; } @@ -342,16 +356,16 @@ class ArenaExpertSource : public ExpertSource { double load_gib_per_second() const { return gib_per_s_; } private: - void* arena_ = nullptr; ///< the PinnedArena, owned - std::vector dev_slice_; ///< device alias of each registered slice (or of the whole range) - uint64_t slice_bytes_ = 0; - const uint8_t* base_ = nullptr; + std::vector layer_base_; ///< each layer's experts: a pinned block, or a part of `tail_` + std::vector layer_dev_; ///< the device alias of each pinned layer (null for the others) + std::vector blocks_; ///< the pinned blocks, owned (layers [0, blocks_.size())) + void* tail_ = nullptr; ///< the layers past the pinning budget, locked resident, owned + uint64_t tail_bytes_ = 0; int64_t blobs_ = 0; int64_t n_expert_ = 0; int64_t reads_ = 0; std::string note_; double gib_per_s_ = 0.0; - uint64_t pinned_bytes_ = 0; std::vector gguf_; }; diff --git a/include/strata/core/pinned.hpp b/include/strata/core/pinned.hpp index af704691e93..b4c094d76c6 100644 --- a/include/strata/core/pinned.hpp +++ b/include/strata/core/pinned.hpp @@ -63,6 +63,10 @@ struct LoadStats { LoadStats load_experts(const std::string& path, uint8_t* dst, uint64_t blob_bytes, uint64_t blobs_per_layer, uint64_t layers, int threads, uint64_t chunk); /// Plan v0.3 P6: the same with one byte range per layer (`layer_off[L]`, `layer_bytes[L]`). +/// Layer L's `layer_bytes[L]` bytes at file offset `layer_off[L]` go to `layer_dst[L]`. +LoadStats load_experts_ranges(const std::string& path, const std::vector& layer_dst, + const std::vector& layer_off, const std::vector& layer_bytes, int threads, + uint64_t chunk); LoadStats load_experts_ranges(const std::string& path, uint8_t* dst, const std::vector& layer_off, const std::vector& layer_bytes, int threads, uint64_t chunk); diff --git a/include/strata/core/second_gpu.hpp b/include/strata/core/second_gpu.hpp new file mode 100644 index 00000000000..cb500429d12 --- /dev/null +++ b/include/strata/core/second_gpu.hpp @@ -0,0 +1,62 @@ +// include/strata/core/second_gpu.hpp - a second GPU as another expert tier. +// +// It holds experts the first GPU's cache does not (the profile's next ranks, then the conversation's, through its +// own AdaptiveTier) and computes them for each layer of a verify window while the CPU pool computes the misses. +// It never spins on host memory: per layer the host launches one captured graph (copy the plan in, quantize the +// activations, the grouped experts; activations, plan and rows in mapped host memory) before the CPU pool and +// collects the rows after it, so it keeps its latency on a GPU that also drives a display (a graph spinning there +// stalls whenever the desktop draws). One graph per format pair and group-count bucket: the grouped kernels' +// grids are sized for the bucket. Native packs only: the kernels are the first GPU's hit path. +#pragma once + +#include "strata/core/expert_cache.hpp" + +#include + +#include +#include +#include + +namespace strata::core { + +class SecondGpu { +public: + SecondGpu() = default; + ~SecondGpu(); + SecondGpu(const SecondGpu&) = delete; + SecondGpu& operator=(const SecondGpu&) = delete; + + /// Buffers for windows of up to `max_t` tokens with `k` routed experts each, on `device`; the caller then opens + /// and fills `cache()` with that device current. `main_device` is made current again after every call. + bool init(int device, int main_device, int64_t n_embd, int64_t n_ff, int max_t, int64_t k, std::string& err); + ExpertCache& cache() { return cache_; } + int device() const { return dev_; } + + /// One layer's share: group g is the expert in slot `slots[g]`, serving the routed entries + /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations. + bool submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, + const int32_t* entries, int n_groups, std::string& err); + /// Waits for the submitted layer and writes its rows into `out` (row i = entry i, n_embd floats). + bool finish(float* out, std::string& err); + + int64_t layers = 0, experts = 0, entries_done = 0; + double ms_wait = 0; ///< host time waiting for it after the CPU pool + +private: + bool graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& exec, std::string& err); + + int dev_ = -1, main_ = 0, max_t_ = 0; + int64_t n_embd_ = 0, n_ff_ = 0, cap_ = 0; + ExpertCache cache_; + cudaStream_t s_ = nullptr; + cudaEvent_t ev_ = nullptr; + // pinned staging (portable, mapped): the activations, the plan, and the rows the down kernel writes + float *h_x_ = nullptr, *m_x_ = nullptr, *h_out_ = nullptr, *m_out_ = nullptr; + uint8_t *h_plan_ = nullptr, *m_plan_ = nullptr; + uint8_t *d_xq_ = nullptr, *d_plan_ = nullptr, *d_scratch_ = nullptr; + struct Graph { int gu, d, groups; cudaGraphExec_t exec; }; + std::vector graphs_; // per (gate/up, down) format pair and group-count bucket + std::vector pending_; // the submitted layer's entries, in row order +}; + +} // namespace strata::core diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 41c1c282ce6..81a8e29ed08 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -7,22 +7,35 @@ #include namespace strata::core { +namespace { +struct OnDevice { // the tier's GPU current for one call (when it has one), the main one again afterwards + int main; + bool set; + OnDevice(int dev, int main_dev) : main(main_dev), set(dev >= 0) { if (set) cudaSetDevice(dev); } + ~OnDevice() { if (set) cudaSetDevice(main); } +}; +} // namespace AdaptiveTier::~AdaptiveTier() { + OnDevice on(dev_, main_); if (stream_) cudaStreamSynchronize(stream_); if (ev_) cudaEventDestroy(ev_); if (stream_) cudaStreamDestroy(stream_); } bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, - int64_t n_layers, int64_t n_expert, int max_swaps, std::string& err) { + int64_t n_layers, int64_t n_expert, int max_moves, std::string& err, int device, int main_device) { cache_ = &cache; src_ = &src; res_ = &host_res; d_res_ = d_res; n_layers_ = n_layers; n_expert_ = n_expert; - max_swaps_ = max_swaps; + max_moves_ = max_moves; + dev_ = device; + main_ = main_device; + free_.assign((size_t) n_layers, {}); + OnDevice on(dev_, main_); if (cudaStreamCreateWithFlags(&stream_, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&ev_, cudaEventDisableTiming) != cudaSuccess) { err = "adaptive tier: cannot create the refill stream"; @@ -31,11 +44,22 @@ bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& usage, std::string& err) { +int64_t AdaptiveTier::free_slots() const { + int64_t n = 0; + for (const auto& f : free_) n += (int64_t) f.size(); + return n; +} + +bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay) { const auto t0 = std::chrono::steady_clock::now(); - if (!pending_.empty()) return true; // the previous swaps are still in flight - struct Swap { float gain; int32_t layer, in, out; }; - std::vector todo; + if (!pending_.empty()) return true; // the previous moves are still in flight + if (upper_ != nullptr) { + upper_has_.assign((size_t) (n_layers_ * n_expert_), 0); + for (size_t i = 0; i < upper_has_.size(); ++i) upper_has_[i] = (*upper_->res_)[i] >= 0; + for (const auto& pr : upper_->pending_) upper_has_[(size_t) pr.first] = 1; + } + struct Move { float gain; int32_t layer, in, out, slot; }; // out < 0: an empty slot + std::vector moves; std::vector> cand, vict; std::vector& res = *res_; for (int64_t l = 0; l < n_layers_; ++l) { @@ -43,39 +67,51 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err) { vict.clear(); const float* u = usage.data() + l * n_expert_; const int32_t* r = res.data() + l * n_expert_; + const uint8_t* up = upper_ != nullptr ? upper_has_.data() + l * n_expert_ : nullptr; for (int32_t e = 0; e < (int32_t) n_expert_; ++e) { - if (r[e] < 0) { if (u[e] >= 2.0f) cand.emplace_back(u[e], e); } - else vict.emplace_back(u[e], e); + if (r[e] < 0) { if (u[e] >= 2.0f && (up == nullptr || !up[e])) cand.emplace_back(u[e], e); } + else vict.emplace_back(up != nullptr && up[e] ? -1.0f : u[e], e); // the upper tier's copy goes first } - if (cand.empty() || vict.empty()) continue; + if (cand.empty()) continue; std::sort(cand.begin(), cand.end(), [](auto& a, auto& b) { return a.first > b.first; }); - const size_t nc = std::min(cand.size(), vict.size()); - std::partial_sort(vict.begin(), vict.begin() + (ptrdiff_t) nc, vict.end(), + const std::vector& fr = free_[(size_t) l]; + size_t c = 0; + for (; c < cand.size() && c < fr.size(); ++c) + moves.push_back({cand[c].first, (int32_t) l, cand[c].second, -1, fr[fr.size() - 1 - c]}); + const size_t nv = std::min(cand.size() - c, vict.size()); + std::partial_sort(vict.begin(), vict.begin() + (ptrdiff_t) nv, vict.end(), [](auto& a, auto& b) { return a.first < b.first; }); - for (size_t i = 0; i < nc; ++i) { - if (cand[i].first < vict[i].first + 1.5f) break; - todo.push_back({cand[i].first - vict[i].first, (int32_t) l, cand[i].second, vict[i].second}); + for (size_t i = 0; i < nv; ++i, ++c) { + if (cand[c].first < vict[i].first + 1.5f) break; + moves.push_back({cand[c].first - vict[i].first, (int32_t) l, cand[c].second, vict[i].second, + r[vict[i].second]}); } } - std::sort(todo.begin(), todo.end(), [](const Swap& a, const Swap& b) { return a.gain > b.gain; }); - if ((int) todo.size() > max_swaps_) todo.resize((size_t) max_swaps_); + std::sort(moves.begin(), moves.end(), [](const Move& a, const Move& b) { return a.gain > b.gain; }); + if ((int) moves.size() > max_moves_) moves.resize((size_t) max_moves_); const auto& lay = strata::kernels::cpu::expert_layout(); - for (const Swap& s : todo) { - const size_t in = (size_t) (s.layer * n_expert_ + s.in), out = (size_t) (s.layer * n_expert_ + s.out); - const int32_t slot = res[out]; - const uint8_t* b = src_->blob(s.layer, s.in); + OnDevice on(dev_, main_); + for (const Move& m : moves) { + const uint8_t* b = src_->blob(m.layer, m.in); // asynchronous: the copies run while the MTP drafts; the next window waits for them - if (slot < 0 || b == nullptr || cudaMemcpyAsync(cache_->device_slot(slot), b, (size_t) lay.blob_bytes(s.layer), - cudaMemcpyHostToDevice, stream_) != cudaSuccess) { + if (b == nullptr || cudaMemcpyAsync(cache_->device_slot(m.slot), b, (size_t) lay.blob_bytes(m.layer), + cudaMemcpyHostToDevice, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } - res[out] = kNotResident; // evicted now: the CPU computes it meanwhile - pending_.emplace_back((int32_t) in, slot); // resident once the copy has landed + if (m.out >= 0) { + res[(size_t) (m.layer * n_expert_ + m.out)] = kNotResident; // evicted now: a miss meanwhile + ++swaps; + } else { + std::vector& fr = free_[(size_t) m.layer]; + fr.erase(std::find(fr.begin(), fr.end(), m.slot)); + ++fills; + } + pending_.emplace_back((int32_t) (m.layer * n_expert_ + m.in), m.slot); // resident once the copy has landed } - if (!todo.empty()) cudaEventRecord(ev_, stream_); - for (float& v : usage) v *= 0.7f; - swaps += (int64_t) todo.size(); + if (!moves.empty()) cudaEventRecord(ev_, stream_); + if (decay) + for (float& v : usage) v *= 0.7f; ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); return true; } diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index bcd18c683a4..aeef419ed69 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -1,9 +1,11 @@ // src/core/expert_source.cpp - the adapter. See the header for the three clauses of the contract. #include "strata/core/expert_source.hpp" +#include "strata/core/second_gpu.hpp" #include "strata/artifact/gguf_reader.hpp" #include "strata/kernels/cpu/expert_layout.hpp" #include "strata/core/pinned.hpp" +#include "strata/platform/memory.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/quantize_act.hpp" #include "strata/kernels/s2_expert_grouped.hpp" @@ -275,7 +277,31 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 // ---- plan v0.3 P6: the GPU's share, decided and published FIRST so the GPU starts while the CPU works. // Distinct experts in routing order; resident ones and the last pcie_num/256 of the missed ones go to the GPU. const int64_t n = n_tok * k; - int32_t kind[128]; // per entry: -1 CPU, 0 VRAM, 1 PCIe + int32_t kind[128]; // per entry: -1 CPU, 0 VRAM, 1 PCIe, 2 the second GPU + // the second GPU's share: group g = the expert in its slot g2_slot[g], entries g2_ent[g2_start[g] ..) + int n2g = 0, n2e = 0; + int32_t g2_slot[128], g2_start[129], g2_ent[128]; + auto on_gpu2 = [&](int32_t e) { + return d.gpu2 != nullptr && d.host_res2[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0; + }; + // The second GPU takes this layer's share only when the layer's misses are big enough that the CPU would need + // longer for all of them than its round trip (~100 us: ~4 MB of experts at the pool's ~45 GB/s); otherwise it + // would only add its latency. + if (d.gpu2 != nullptr) { + uint64_t miss_bytes = 0, gpu2_bytes = 0; + for (int64_t i = 0; i < n_tok * k; ++i) { + const int32_t e = ids[i]; + if (e < 0 || e >= d.n_expert || d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) continue; + bool first = true; + for (int64_t j = 0; j < i && first; ++j) first = ids[j] != e; + if (!first) continue; + miss_bytes += lay.blob_bytes(d.layers); + if (on_gpu2(e)) gpu2_bytes += lay.blob_bytes(d.layers); + } + d.gpu2_used = gpu2_bytes > 0 && miss_bytes >= d.gpu2_min_bytes; + if (gpu2_bytes > 0 && !d.gpu2_used) ++d.gpu2_skipped; + } + auto on_gpu2_now = [&](int32_t e) { return d.gpu2_used && on_gpu2(e); }; if (d.plan != nullptr && n <= 128 && n <= d.plan->cap) { int64_t distinct[128], first_of[128]; int nd = 0, nmiss = 0; @@ -286,7 +312,9 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (first_of[i] == i) { distinct[nd++] = i; const int32_t e = ids[i]; - if (e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] < 0) ++nmiss; + if (e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] < 0 && + !on_gpu2_now(e)) + ++nmiss; } } const bool pcie_ok = d.pcie_num > 0 && d.src->device_alias(d.layers, 0) != nullptr; @@ -306,6 +334,12 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 kd = 0; ptr = (unsigned long long) (d.cache_base + (d.cache_slot_off ? (size_t) d.cache_slot_off[slot] : (size_t) slot * (size_t) d.cache_blob)); + } else if (on_gpu2_now(e)) { + kd = 2; + g2_slot[n2g] = d.host_res2[(size_t) d.layers * (size_t) d.n_expert + (size_t) e]; + g2_start[n2g++] = n2e; + for (int64_t i = i0; i < n; ++i) + if (first_of[i] == i0) g2_ent[n2e++] = (int32_t) i; } else { if (miss_rank >= nmiss - m && fetches < P.staging_cap && fetches < 64) { const uint8_t* src = d.src->blob(d.layers, e); @@ -354,6 +388,13 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 std::atomic_thread_fence(std::memory_order_seq_cst); if (P.publish) P.publish(P.ctx); if (P.fetch) P.fetch(P.ctx, dma_src, P.pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work + g2_start[n2g] = n2e; + if (n2g > 0 && !d.gpu2->submit(d.layers, x_f, (int) n_tok, k, g2_slot, g2_start, g2_ent, n2g, d.gpu2_err)) { + d.failed = true; + d.fail = d.gpu2_err.c_str(); + d.fail_layer = d.layers; + return; + } } else { for (int64_t i = 0; i < n; ++i) { const int32_t e = ids[i]; @@ -383,8 +424,9 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 d.fail_expert = e; return; } - if (kind[i] >= 0) { // the GPU computes this entry (a VRAM hit or a PCIe read) + if (kind[i] >= 0) { // a GPU computes this entry (a VRAM hit, a PCIe read, the second GPU) if (kind[i] == 0) ++d.cache_hits; + if (kind[i] == 2) ++d.gpu2_entries; std::memset(row, 0, (size_t) H * sizeof(float)); continue; } @@ -415,6 +457,12 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 const auto c3 = std::chrono::steady_clock::now(); if (native) d.pool->run_split_multi_native(lay.fmt[(size_t) d.layers], d.jobs_multi.data(), njobs); else d.pool->run_split_multi(d.jobs_multi.data(), njobs); + if (n2g > 0 && !d.gpu2->finish(out, d.gpu2_err)) { // the second GPU's rows, over the zeroed ones + d.failed = true; + d.fail = d.gpu2_err.c_str(); + d.fail_layer = d.layers; + return; + } const auto c4 = std::chrono::steady_clock::now(); auto ms = [](auto a, auto b) { return std::chrono::duration(b - a).count(); }; d.ms_plan += ms(c0, c1); @@ -581,8 +629,9 @@ bool locate_experts_gguf(const std::vector& shards, const strata::k } } -LoadStats load_experts_gguf(const std::vector& shards, const std::vector& spans, uint8_t* dst, - const strata::kernels::cpu::ExpertLayout& lay, int threads) { +LoadStats load_experts_gguf(const std::vector& shards, const std::vector& spans, + const std::vector& layer_base, const strata::kernels::cpu::ExpertLayout& lay, + int threads) { LoadStats st; st.layers = (uint64_t) lay.n_layers; const auto t0 = std::chrono::steady_clock::now(); @@ -614,7 +663,7 @@ LoadStats load_experts_gguf(const std::vector& shards, const std::v if ((uint64_t) f.gcount() != n) { bad = true; return; } for (uint64_t k = 0; k < n / per[r]; ++k) { const uint64_t e = done / per[r] + k; - std::memcpy(dst + lay.blob_offset(l, (int64_t) e) + at[r], buf.data() + k * per[r], (size_t) per[r]); + std::memcpy(layer_base[(size_t) l] + e * blob + at[r], buf.data() + k * per[r], (size_t) per[r]); } } } @@ -632,6 +681,24 @@ LoadStats load_experts_gguf(const std::vector& shards, const std::v st.seconds = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); return st; } +// Pageable memory for the arena's layers past the pinning budget. +void* reserve_pageable(uint64_t bytes) { +#if defined(_WIN32) + return VirtualAlloc(nullptr, (SIZE_T) bytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE); +#else + void* p = mmap(nullptr, (size_t) bytes, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + return p == MAP_FAILED ? nullptr : p; +#endif +} + +void release_pageable(void* p, uint64_t bytes) { +#if defined(_WIN32) + (void) bytes; + VirtualFree(p, 0, MEM_RELEASE); +#else + munmap(p, (size_t) bytes); +#endif +} } // namespace ArenaExpertSource::~ArenaExpertSource() { close(); } @@ -672,88 +739,91 @@ bool ArenaExpertSource::open(const std::string& pack_dir, int64_t n_layers, int6 } } - // one layer per registration slice, so no expert straddles two registrations. The arena is one blob - // longer than the file: a copy of a whole VRAM slot (the largest blob) may then start at any expert. - std::vector bounds, loff, lbytes; + // one pinned block per layer, each followed by room for a whole slot-sized copy (the largest blob) from its last + // expert; the layers past the pinning budget in one pageable range, locked resident + const uint64_t maxb = lay.max_blob; + std::vector loff, lbytes, room; for (int64_t l = 0; l < n_layers; ++l) { - bounds.push_back(lay.layer_offset(l)); loff.push_back(lay.layer_offset(l)); lbytes.push_back(lay.blob_bytes(l) * (uint64_t) n_expert); + room.push_back(lbytes.back() + (maxb - lay.blob_bytes(l))); } - bounds.push_back(want); - PinnedArena* a = new PinnedArena(want + (uint64_t) blob, bounds); - if (!a->valid()) { - delete a; - err = "ArenaExpertSource: the arena could not be reserved (" + std::to_string(want) + " B)"; - return false; + uint64_t pinned_total = 0; + std::string tail_note; + for (int64_t l = 0; l < n_layers; ++l) { + void* p = nullptr; + if (cudaHostAlloc(&p, (size_t) room[(size_t) l], cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess) { + (void) cudaGetLastError(); + for (int64_t m = l; m < n_layers; ++m) tail_bytes_ += room[(size_t) m]; + tail_ = reserve_pageable(tail_bytes_); + if (tail_ == nullptr) { + close(); + err = "ArenaExpertSource: the arena could not be reserved (" + std::to_string(tail_bytes_) + " B)"; + return false; + } + tail_note = strata::platform::lock_resident(tail_, tail_bytes_).note; + uint8_t* at = (uint8_t*) tail_; + for (int64_t m = l; m < n_layers; ++m) { + layer_base_.push_back(at); + layer_dev_.push_back(nullptr); + at += room[(size_t) m]; + } + break; + } + blocks_.push_back((uint8_t*) p); + layer_base_.push_back((uint8_t*) p); + void* d = nullptr; + layer_dev_.push_back(cudaHostGetDevicePointer(&d, p, 0) == cudaSuccess ? (const uint8_t*) d : nullptr); + (void) cudaGetLastError(); + pinned_total += room[(size_t) l]; } - const LoadStats st = from_gguf ? load_experts_gguf(gguf_, spans, a->data(), lay, threads) - : load_experts_ranges(path, a->data(), loff, lbytes, threads, /*chunk=*/8u << 20); + const LoadStats st = from_gguf ? load_experts_gguf(gguf_, spans, layer_base_, lay, threads) + : load_experts_ranges(path, layer_base_, loff, lbytes, threads, /*chunk=*/8u << 20); if (st.bytes != want) { - delete a; + close(); err = "ArenaExpertSource: the load read " + std::to_string(st.bytes) + " B of " + std::to_string(want); return false; } - arena_ = a; - base_ = a->data(); - pinned_bytes_ = a->registered_bytes; - // plan v0.3 P6: device aliases of the mapped registration, for the PCIe share of the misses - dev_slice_.clear(); - slice_bytes_ = a->slice_bytes; - if (a->registered_bytes > 0) { - std::vector starts = a->slice_bytes > 0 ? a->slice_starts : std::vector{0}; - for (uint64_t off : starts) { - void* d = nullptr; - if (cudaHostGetDevicePointer(&d, (void*) (base_ + off), 0) != cudaSuccess) { - (void) cudaGetLastError(); - dev_slice_.clear(); - break; - } - dev_slice_.push_back((const uint8_t*) d); - } - } + note_ = std::to_string(blocks_.size()) + " of " + std::to_string(n_layers) + " layers pinned (" + + std::to_string(pinned_total >> 30) + " GiB, a cudaHostAlloc block each)"; + if (tail_ != nullptr) note_ += "; the others pageable: " + tail_note; blobs_ = n_layers * n_expert; n_expert_ = n_expert; reads_ = 0; - note_ = a->note; gib_per_s_ = st.gib_per_second(); return true; } void ArenaExpertSource::close() { - if (arena_ != nullptr) { - delete (PinnedArena*) arena_; - arena_ = nullptr; + for (uint8_t* p : blocks_) cudaFreeHost(p); + blocks_.clear(); + if (tail_ != nullptr) { + strata::platform::unlock_resident(tail_, tail_bytes_); + release_pageable(tail_, tail_bytes_); + tail_ = nullptr; + tail_bytes_ = 0; } - base_ = nullptr; + layer_base_.clear(); + layer_dev_.clear(); blobs_ = 0; n_expert_ = 0; } bool ArenaExpertSource::pinned(int64_t layer, int64_t expert) const { - if (base_ == nullptr || layer < 0 || expert < 0 || expert >= n_expert_) return false; - const auto& lay = strata::kernels::cpu::expert_layout(); - return lay.blob_offset(layer, expert) + lay.blob_bytes(layer) <= pinned_bytes_; + return layer >= 0 && layer < (int64_t) blocks_.size() && expert >= 0 && expert < n_expert_; } const uint8_t* ArenaExpertSource::device_alias(int64_t layer, int64_t expert) const { - if (dev_slice_.empty() || !pinned(layer, expert)) return nullptr; - const auto& lay = strata::kernels::cpu::expert_layout(); - if (slice_bytes_ == 0) return dev_slice_[0] + lay.blob_offset(layer, expert); - // one registration slice per layer - if ((size_t) layer >= dev_slice_.size()) return nullptr; - return dev_slice_[(size_t) layer] + (uint64_t) expert * lay.blob_bytes(layer); + if (!pinned(layer, expert) || layer_dev_[(size_t) layer] == nullptr) return nullptr; + return layer_dev_[(size_t) layer] + (uint64_t) expert * strata::kernels::cpu::expert_layout().blob_bytes(layer); } const uint8_t* ArenaExpertSource::blob(int64_t layer, int64_t expert) { - if (base_ == nullptr) return nullptr; - if (layer < 0 || expert < 0 || expert >= n_expert_) return nullptr; - const int64_t idx = layer * n_expert_ + expert; - if (idx < 0 || idx >= blobs_) return nullptr; + if (layer < 0 || layer >= (int64_t) layer_base_.size() || expert < 0 || expert >= n_expert_) return nullptr; ++reads_; // Pointer arithmetic into resident memory. No fault, no copy, no mapping - which is the entire point of // this class over `FileExpertSource`. - return base_ + strata::kernels::cpu::expert_layout().blob_offset(layer, expert); + return layer_base_[(size_t) layer] + (uint64_t) expert * strata::kernels::cpu::expert_layout().blob_bytes(layer); } } // namespace strata::core diff --git a/src/core/pinned.cu b/src/core/pinned.cu index 56573ce3257..3d012df2d53 100644 --- a/src/core/pinned.cu +++ b/src/core/pinned.cu @@ -192,6 +192,14 @@ LoadStats load_experts(const std::string& path, uint8_t* dst, uint64_t blob_byte LoadStats load_experts_ranges(const std::string& path, uint8_t* dst, const std::vector& layer_off, const std::vector& layer_bytes, int threads, uint64_t chunk) { + std::vector layer_dst; + for (uint64_t off : layer_off) layer_dst.push_back(dst + off); + return load_experts_ranges(path, layer_dst, layer_off, layer_bytes, threads, chunk); +} + +LoadStats load_experts_ranges(const std::string& path, const std::vector& layer_dst, + const std::vector& layer_off, const std::vector& layer_bytes, int threads, + uint64_t chunk) { LoadStats st; const uint64_t layers = (uint64_t) layer_off.size(); st.layers = layers; @@ -231,7 +239,7 @@ LoadStats load_experts_ranges(const std::string& path, uint8_t* dst, const std:: err = "short read in layer " + std::to_string(L); return; } - std::memcpy(dst + off + pos, buf.data(), (size_t) n); + std::memcpy(layer_dst[(size_t) L] + pos, buf.data(), (size_t) n); h = fnv1a64(buf.data(), n, h); pos += n; remaining -= n; diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp new file mode 100644 index 00000000000..f38baf2b225 --- /dev/null +++ b/src/core/second_gpu.cpp @@ -0,0 +1,170 @@ +// src/core/second_gpu.cpp - see include/strata/core/second_gpu.hpp. +#include "strata/core/second_gpu.hpp" + +#include "strata/kernels/cpu/expert_layout.hpp" +#include "strata/kernels/elementwise.hpp" +#include "strata/kernels/iq_kernels.hpp" + +#include + +#include +#include +#include + +namespace strata::core { +namespace { + +// the plan block: [n_groups, pad x3] [starts: cap+1] [dst: cap] [tok: cap] (8-byte aligned) [slot addresses: cap] +int64_t ptr_off(int64_t cap) { return (4 + (cap + 1) + 2 * cap + 1) / 2 * 2; } // in int32 units +size_t plan_bytes(int64_t cap) { return (size_t) ptr_off(cap) * 4 + (size_t) cap * 8; } + +// the grouped kernels' grids are sized for the bucket a layer's group count falls in +int bucket(int groups) { + for (int b : {1, 2, 3, 4, 6, 8, 12, 16, 24, 32}) if (groups <= b) return b; + return groups <= 48 ? 48 : groups; +} + +struct DeviceScope { // the second GPU current for one call, the main one again afterwards + int main; + DeviceScope(int dev, int main_dev) : main(main_dev) { cudaSetDevice(dev); } + ~DeviceScope() { cudaSetDevice(main); } +}; + +} // namespace + +SecondGpu::~SecondGpu() { + if (dev_ < 0) return; + DeviceScope scope(dev_, main_); + if (s_) cudaStreamSynchronize(s_); + for (auto& gr : graphs_) cudaGraphExecDestroy(gr.exec); + cache_.close(); + void* dev[] = {d_xq_, d_plan_, d_scratch_}; + for (void* p : dev) if (p) cudaFree(p); + void* host[] = {h_x_, h_out_, h_plan_}; + for (void* p : host) if (p) cudaFreeHost(p); + if (ev_) cudaEventDestroy(ev_); + if (s_) cudaStreamDestroy(s_); +} + +bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, int max_t, int64_t k, std::string& err) { + dev_ = device; + main_ = main_device; + max_t_ = max_t; + n_embd_ = n_embd; + n_ff_ = n_ff; + cap_ = (int64_t) max_t * k; + DeviceScope scope(dev_, main_); + const size_t xb = (size_t) max_t * (size_t) n_embd * sizeof(float), ob = (size_t) cap_ * (size_t) n_embd * sizeof(float); + const unsigned pm = cudaHostAllocPortable | cudaHostAllocMapped; + const char* step = nullptr; + cudaError_t e = cudaSuccess; + auto run = [&](const char* what, cudaError_t r) { if (e == cudaSuccess && r != cudaSuccess) { e = r; step = what; } }; + run("context", cudaFree(nullptr)); + run("stream", cudaStreamCreateWithFlags(&s_, cudaStreamNonBlocking)); + run("event", cudaEventCreateWithFlags(&ev_, cudaEventDisableTiming)); + run("pinned staging", cudaHostAlloc((void**) &h_x_, xb, pm)); + run("pinned staging", cudaHostAlloc((void**) &h_out_, ob, pm)); + run("pinned staging", cudaHostAlloc((void**) &h_plan_, plan_bytes(cap_), pm)); + run("mapped staging", cudaHostGetDevicePointer((void**) &m_x_, h_x_, 0)); + run("mapped staging", cudaHostGetDevicePointer((void**) &m_out_, h_out_, 0)); + run("mapped staging", cudaHostGetDevicePointer((void**) &m_plan_, h_plan_, 0)); + run("device buffers", cudaMalloc((void**) &d_xq_, (size_t) max_t * (size_t) (n_embd / 32) * 36)); + run("device buffers", cudaMalloc((void**) &d_plan_, plan_bytes(cap_))); + run("device buffers", cudaMalloc((void**) &d_scratch_, strata::kernels::native_expert_scratch_bytes(cap_, n_ff))); + if (e != cudaSuccess) { + err = std::string("second GPU: ") + step + ": " + cudaGetErrorString(e); + return false; + } + std::memset(h_x_, 0, xb); + std::memset(h_plan_, 0, plan_bytes(cap_)); + pending_.reserve((size_t) cap_); + return true; +} + +bool SecondGpu::graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& exec, std::string& err) { + for (const auto& gr : graphs_) + if (gr.gu == gu_type && gr.d == d_type && gr.groups == groups) { exec = gr.exec; return true; } + const strata::kernels::NativeExpertLayout L = strata::kernels::native_expert_layout(gu_type, d_type, n_embd_, n_ff_); + const auto* pi = (const int32_t*) d_plan_; + cudaGraph_t graph = nullptr; + if (cudaStreamBeginCapture(s_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { + err = "second GPU: cannot capture"; + return false; + } + try { + // a kernel copies the plan (a copy node costs ~15 us on this link); the grouped kernels read the group + // count from it and skip the groups past it + strata::kernels::copy_i32_from_mapped((int32_t*) d_plan_, (const int32_t*) m_plan_, (int64_t) plan_bytes(cap_) / 4, s_); + strata::kernels::quantize_q8_1_rows(m_x_, max_t_, n_embd_, d_xq_, s_); + strata::kernels::native_expert_grouped(L, (const unsigned long long*) (pi + ptr_off(cap_)), pi + 4, pi, + pi + 4 + cap_ + 1, pi + 4 + 2 * cap_ + 1, groups, cap_, d_xq_, d_scratch_, + m_out_, s_); + } catch (const std::exception& ex) { + cudaStreamEndCapture(s_, &graph); + if (graph) cudaGraphDestroy(graph); + err = std::string("second GPU: ") + ex.what(); + return false; + } + const cudaError_t ce = cudaStreamEndCapture(s_, &graph); + if (ce != cudaSuccess || cudaGraphInstantiate(&exec, graph, 0) != cudaSuccess) { + if (graph) cudaGraphDestroy(graph); + err = std::string("second GPU: graph capture: ") + cudaGetErrorString(ce); + return false; + } + cudaGraphDestroy(graph); + cudaGraphUpload(exec, s_); + cudaStreamSynchronize(s_); + graphs_.push_back({gu_type, d_type, groups, exec}); + return true; +} + +bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, + const int32_t* entries, int n_groups, std::string& err) { + const int n = starts[n_groups]; + if (n_groups <= 0 || n > cap_ || n_tok > max_t_) { err = "second GPU: a layer's share is out of range"; return false; } + // the plan: rows 0..n-1 in entry order + auto* pi = (int32_t*) h_plan_; + int32_t* start = pi + 4; + int32_t* dst = start + cap_ + 1; + int32_t* tok = dst + cap_; + auto* ptr = (unsigned long long*) (pi + ptr_off(cap_)); + pi[0] = n_groups; + pending_.assign(entries, entries + n); + for (int g = 0; g <= n_groups; ++g) start[g] = starts[g]; + for (int j = 0; j < n; ++j) { + dst[j] = j; + tok[j] = (int32_t) (entries[j] / k); + } + for (int g = 0; g < n_groups; ++g) ptr[g] = (unsigned long long) cache_.device_slot(slots[g]); + std::memcpy(h_x_, x, (size_t) n_tok * (size_t) n_embd_ * sizeof(float)); + const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; + DeviceScope scope(dev_, main_); + cudaGraphExec_t exec = nullptr; + if (!graph_for(f.gu_type, f.d_type, bucket(n_groups), exec, err)) return false; + if (cudaGraphLaunch(exec, s_) != cudaSuccess || cudaEventRecord(ev_, s_) != cudaSuccess) { + err = std::string("second GPU: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + ++layers; + experts += n_groups; + entries_done += n; + return true; +} + +bool SecondGpu::finish(float* out, std::string& err) { + const auto t0 = std::chrono::steady_clock::now(); + cudaError_t q; + while ((q = cudaEventQuery(ev_)) == cudaErrorNotReady) _mm_pause(); + ms_wait += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + if (q != cudaSuccess) { + err = std::string("second GPU: ") + cudaGetErrorString(q); + return false; + } + for (size_t j = 0; j < pending_.size(); ++j) + std::memcpy(out + (size_t) pending_[j] * (size_t) n_embd_, h_out_ + j * (size_t) n_embd_, + (size_t) n_embd_ * sizeof(float)); + pending_.clear(); + return true; +} + +} // namespace strata::core diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 64ab25b9891..af08000834a 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -17,6 +17,7 @@ #include "strata/artifact/gguf_split.hpp" #include "strata/core/adaptive_tier.hpp" #include "strata/core/expert_cache.hpp" +#include "strata/core/second_gpu.hpp" #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" #include "strata/core/layout.hpp" @@ -221,6 +222,12 @@ struct Options { /// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it. std::string mtp; int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell) + /// A second GPU (device ordinal) as another expert tier, with this much VRAM for experts (0 = all but + /// `second_gpu_reserve_mib`). + int second_gpu = -1; + double second_gpu_gib = 0.0; + int second_gpu_reserve_mib = 2048; + double second_gpu_min_mb = 4.0; ///< a layer's misses from which it takes its share (smaller: the CPU is quicker) /// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the /// pinned arena while the CPU computes the rest (verify windows). double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs) @@ -325,6 +332,12 @@ void usage() { " least probability P under the draft layer (default 0: always T-1 drafts)\n" " --mtp DIR the MTP draft layer's runtime files (tools/mtp_rt.py)\n" " --mtp-window N the draft layer attends to the last N cells (default 32768; 0 = every cell)\n" + " --second-gpu N CUDA device N (of the visible ones; the engine runs on device 0) as another\n" + " expert tier: it holds the profile's next experts, then the conversation's,\n" + " and computes them beside the CPU pool (native packs, verify windows)\n" + " --second-gpu-gib G its VRAM for experts (default: all but --second-gpu-reserve-mib, 2048)\n" + " --second-gpu-min-mb M a layer's missed experts from which it takes its share (default 4; below\n" + " it the CPU is quicker than its ~100 us round trip)\n" " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" " accept the drafts that match it, so speculation settings compare on the\n" " same text (not --serve)\n" @@ -558,6 +571,10 @@ int main(int argc, char** argv) { else if (a == "--spec-corrupt") o.spec_corrupt = std::atoi(next("--spec-corrupt")); else if (a == "--mtp") o.mtp = next("--mtp"); else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window")); + else if (a == "--second-gpu") o.second_gpu = std::atoi(next("--second-gpu")); + else if (a == "--second-gpu-gib") o.second_gpu_gib = std::atof(next("--second-gpu-gib")); + else if (a == "--second-gpu-reserve-mib") o.second_gpu_reserve_mib = std::atoi(next("--second-gpu-reserve-mib")); + else if (a == "--second-gpu-min-mb") o.second_gpu_min_mb = std::atof(next("--second-gpu-min-mb")); else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac")); else if (a == "--adapt-every") o.adapt_every = std::atoi(next("--adapt-every")); else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); @@ -1779,16 +1796,102 @@ int main(int argc, char** argv) { (long long) resident); } // plan v0.3 P6: the VRAM tier follows the conversation in both decode loops (--serve and the speculative loop): - // swaps every --adapt-every rounds on the adapt thread, admitted once their copies have landed + // moves every --adapt-every rounds on the adapt thread, admitted once their copies have landed strata::core::AdaptiveTier tier; - if (!host_res.empty() && o.adapt_every > 0 && o.adapt_swaps > 0 && - !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err)) { + const bool adaptive = !host_res.empty() && o.adapt_every > 0 && o.adapt_swaps > 0; + if (adaptive && !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } + // ---- a second GPU as another expert tier: the profile's pairs the first GPU's cache does not hold, then empty + // slots for every layer, which its own adaptive tier fills with the conversation's experts the first tier has not + strata::core::SecondGpu gpu2; + std::vector host_res2; + strata::core::AdaptiveTier tier2; + if (o.second_gpu >= 0) { + if (!native_pack || host_res.empty() || o.spec < 2) { + std::fprintf(stderr, "strata generate: --second-gpu needs a native pack, --expert-profile and --spec\n"); + return 2; + } + int main_dev = 0; + cudaGetDevice(&main_dev); + cudaDeviceProp prop{}; + if (o.second_gpu == main_dev || cudaGetDeviceProperties(&prop, o.second_gpu) != cudaSuccess) { + std::fprintf(stderr, "strata generate: --second-gpu %d is not another visible CUDA device\n", o.second_gpu); + return 2; + } + if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } + cudaSetDevice(o.second_gpu); + size_t free_b = 0, total_b = 0; + cudaMemGetInfo(&free_b, &total_b); + const uint64_t reserve = (uint64_t) o.second_gpu_reserve_mib << 20; + const uint64_t budget = o.second_gpu_gib > 0 ? (uint64_t) (o.second_gpu_gib * 1073741824.0) + : (free_b > reserve ? (uint64_t) free_b - reserve : 0); + const auto& lay = strata::kernels::cpu::expert_layout(); + std::vector sizes; + std::vector> pre; + std::vector empty2; + uint64_t used = 0; + for (size_t i = (size_t) prefilled; i < profile.size(); ++i) { + const auto& pr = profile[i]; + if (host_res[(size_t) (pr.first * g.n_expert + pr.second)] >= 0) continue; // the first GPU holds it + const uint64_t b = (lay.blob_bytes(pr.first) + 255) / 256 * 256; + if (used + b > budget) break; + used += b; + sizes.push_back((int64_t) lay.blob_bytes(pr.first)); + pre.push_back(pr); + } + if (adaptive) { + uint64_t round = 0; + for (int64_t l = 0; l < g.n_layers; ++l) round += (lay.blob_bytes(l) + 255) / 256 * 256; + for (uint64_t q = (budget - used) / round; q > 0; --q) + for (int64_t l = 0; l < g.n_layers; ++l) { + sizes.push_back((int64_t) lay.blob_bytes(l)); + empty2.push_back((int32_t) l); + } + } + bool ok = !sizes.empty() && gpu2.cache().open_sized(sizes, g.n_layers, g.n_expert, err); + for (size_t i = 0; ok && i < pre.size(); ++i) { + const int32_t slot = gpu2.cache().admit(pre[i].first, pre[i].second); + const uint8_t* b = srcp->blob(pre[i].first, pre[i].second); + ok = slot != strata::core::kNotResident && b != nullptr && + gpu2.cache().fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pre[i].first)); + } + cudaSetDevice(main_dev); + if (ok) { + host_res2.assign((size_t) (g.n_layers * g.n_expert), strata::core::kNotResident); + for (int64_t l = 0; l < g.n_layers; ++l) + for (int64_t e = 0; e < g.n_expert; ++e) host_res2[(size_t) (l * g.n_expert + e)] = gpu2.cache().slot_of(l, e); + } + if (ok && adaptive) { + ok = tier2.init(gpu2.cache(), *srcp, host_res2, nullptr, g.n_layers, g.n_expert, o.adapt_swaps, err, + o.second_gpu, main_dev); + for (size_t i = 0; ok && i < empty2.size(); ++i) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); + tier2.set_upper(&tier); + } + if (!ok) { + std::fprintf(stderr, "strata generate: second GPU: %s\n", sizes.empty() ? "no VRAM for experts" : err.c_str()); + return 1; + } + drive.d.gpu2 = &gpu2; + drive.d.host_res2 = host_res2.data(); + drive.d.gpu2_min_bytes = (uint64_t) (o.second_gpu_min_mb * 1048576.0); + std::fprintf(stderr, "strata generate: second GPU %d (%s): %zu experts from the profile and %zu empty slots, " + "%.2f GiB\n", o.second_gpu, prop.name, pre.size(), empty2.size(), gpu2.cache().gib()); + } + // the second tier ranks after the first on the same counts, and decays them std::string adapt_err; - auto adapt = [&]() -> bool { return tier.adapt(drive.d.usage, adapt_err); }; - auto apply_pending = [&](bool wait) { tier.apply_pending(wait); }; + auto adapt = [&]() -> bool { + const bool two = tier2.on(); + return tier.adapt(drive.d.usage, adapt_err, !two) && (!two || tier2.adapt(drive.d.usage, adapt_err)); + }; + auto apply_pending = [&](bool wait) { + tier.apply_pending(wait); + tier2.apply_pending(wait); + }; if (!o.no_capture && !o.no_token_graph && layer_dump == nullptr && half_dump == nullptr && (hit_fn == nullptr || thits.on()) && !native_pack) { if (!strata::core::session_capture_token(wt, g, ss, d_parts, loop_scratch.y_miss, loop_scratch.parts_bytes, @@ -2689,6 +2792,14 @@ int main(int argc, char** argv) { if (rounds > 0 && !drive.d.usage.empty()) std::printf("%-24s %lld experts swapped into the VRAM tier (every %d rounds, %.3f ms/round)\n", "adaptive tier", (long long) tier.swaps, o.adapt_every, tier.ms / rounds); + if (rounds > 0 && drive.d.gpu2 != nullptr) + std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " + "%.3f ms/round waiting for it after the CPU pool; %lld experts swapped in and %lld into empty " + "slots (%lld still empty)\n", "second GPU", + (double) gpu2.entries_done / (double) rounds, (double) gpu2.experts / (double) rounds, + (double) gpu2.layers / (double) rounds, (double) drive.d.gpu2_skipped / (double) rounds, + gpu2.ms_wait / (double) rounds, (long long) tier2.swaps, (long long) tier2.fills, + (long long) tier2.free_slots()); if (rounds > 0 && !drive.d.routed.empty()) { // the share of the routed entries the N most-routed experts of this run take std::vector c = drive.d.routed; From 35036d9935fe3a6a1c4cbb2d87ff7ef7775498e6 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 07:20:17 +0200 Subject: [PATCH 09/92] README: the second GPU; UD-Q4_K_XL speeds with --adapt-every 2 --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index af573b3fdd0..220358db88c 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,15 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 570-603 tok/s, generation 49-59 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 572-608 tok/s, generation 52-60 tok/s. + +### A second GPU + +With `--second-gpu N`, CUDA device N (the engine runs on device 0) holds more experts and computes its share of +each layer while the CPU computes the rest; it may be the card that drives the display. Build for both GPUs' +architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). UD-Q4_K_XL on an RTX 3090 plus an RTX 5070 Ti (12.6 GB of +its VRAM): generation 59-69 tok/s, +9-19%. It pays when the CPU's share of a token is the larger one: with +IQ3_XXS, whose experts mostly fit the first GPU, it gains nothing. ## Before you start From 5a84ece95546700f805027eb03d520e8aeabfb93 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 12:29:50 +0200 Subject: [PATCH 10/92] --main-gpu: the CUDA device the engine runs on --- README.md | 2 +- src/program/generate.cpp | 28 ++++++++++++++++++++-------- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 220358db88c..68dca8a0956 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 572-608 tok/s, generation ### A second GPU -With `--second-gpu N`, CUDA device N (the engine runs on device 0) holds more experts and computes its share of +With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0) holds more experts and computes its share of each layer while the CPU computes the rest; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). UD-Q4_K_XL on an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its VRAM): generation 59-69 tok/s, +9-19%. It pays when the CPU's share of a token is the larger one: with diff --git a/src/program/generate.cpp b/src/program/generate.cpp index af08000834a..261b68e3563 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -222,6 +222,7 @@ struct Options { /// Plan v0.3 P6: the MTP draft layer's runtime directory (tools/mtp_rt.py); drafts come from it. std::string mtp; int64_t mtp_window = 32768; ///< the draft layer attends to the last N cells (0 = every cell) + int main_gpu = 0; ///< the CUDA device (of the visible ones) the engine runs on /// A second GPU (device ordinal) as another expert tier, with this much VRAM for experts (0 = all but /// `second_gpu_reserve_mib`). int second_gpu = -1; @@ -332,7 +333,8 @@ void usage() { " least probability P under the draft layer (default 0: always T-1 drafts)\n" " --mtp DIR the MTP draft layer's runtime files (tools/mtp_rt.py)\n" " --mtp-window N the draft layer attends to the last N cells (default 32768; 0 = every cell)\n" - " --second-gpu N CUDA device N (of the visible ones; the engine runs on device 0) as another\n" + " --main-gpu N run on CUDA device N of the visible ones (default 0)\n" + " --second-gpu N CUDA device N (of the visible ones, not --main-gpu) as another\n" " expert tier: it holds the profile's next experts, then the conversation's,\n" " and computes them beside the CPU pool (native packs, verify windows)\n" " --second-gpu-gib G its VRAM for experts (default: all but --second-gpu-reserve-mib, 2048)\n" @@ -571,6 +573,7 @@ int main(int argc, char** argv) { else if (a == "--spec-corrupt") o.spec_corrupt = std::atoi(next("--spec-corrupt")); else if (a == "--mtp") o.mtp = next("--mtp"); else if (a == "--mtp-window") o.mtp_window = std::atoll(next("--mtp-window")); + else if (a == "--main-gpu") o.main_gpu = std::atoi(next("--main-gpu")); else if (a == "--second-gpu") o.second_gpu = std::atoi(next("--second-gpu")); else if (a == "--second-gpu-gib") o.second_gpu_gib = std::atof(next("--second-gpu-gib")); else if (a == "--second-gpu-reserve-mib") o.second_gpu_reserve_mib = std::atoi(next("--second-gpu-reserve-mib")); @@ -639,6 +642,18 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: --kv must be fp16 or int8\n"); return 2; } + // every CUDA call from here on uses this device; the adaptive tier sets it on its own thread too + int n_dev = 0; + if (cudaGetDeviceCount(&n_dev) != cudaSuccess || o.main_gpu < 0 || o.main_gpu >= n_dev || + cudaSetDevice(o.main_gpu) != cudaSuccess) { + std::fprintf(stderr, "strata generate: --main-gpu %d is not a visible CUDA device (%d visible)\n", o.main_gpu, + n_dev); + return 2; + } + if (o.second_gpu >= 0 && (o.second_gpu == o.main_gpu || o.second_gpu >= n_dev)) { + std::fprintf(stderr, "strata generate: --second-gpu %d is not another visible CUDA device\n", o.second_gpu); + return 2; + } strata::core::qsa_set_kv_int8(o.kv == "int8"); strata::core::layer_set_shared_early(!o.shared_late); std::vector native_shards; // the --native model's GGUF shards @@ -1799,7 +1814,8 @@ int main(int argc, char** argv) { // moves every --adapt-every rounds on the adapt thread, admitted once their copies have landed strata::core::AdaptiveTier tier; const bool adaptive = !host_res.empty() && o.adapt_every > 0 && o.adapt_swaps > 0; - if (adaptive && !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err)) { + if (adaptive && !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err, + o.main_gpu, o.main_gpu)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -1813,13 +1829,9 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: --second-gpu needs a native pack, --expert-profile and --spec\n"); return 2; } - int main_dev = 0; - cudaGetDevice(&main_dev); + const int main_dev = o.main_gpu; cudaDeviceProp prop{}; - if (o.second_gpu == main_dev || cudaGetDeviceProperties(&prop, o.second_gpu) != cudaSuccess) { - std::fprintf(stderr, "strata generate: --second-gpu %d is not another visible CUDA device\n", o.second_gpu); - return 2; - } + cudaGetDeviceProperties(&prop, o.second_gpu); if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; From 48ba798b187ce6479e5e05dd78aab9d1685c8abe Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 12:36:13 +0200 Subject: [PATCH 11/92] Device ordinals in nvidia-smi's order (CUDA_DEVICE_ORDER=PCI_BUS_ID unless set) --- src/program/generate.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 261b68e3563..01d37c19504 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -333,7 +333,8 @@ void usage() { " least probability P under the draft layer (default 0: always T-1 drafts)\n" " --mtp DIR the MTP draft layer's runtime files (tools/mtp_rt.py)\n" " --mtp-window N the draft layer attends to the last N cells (default 32768; 0 = every cell)\n" - " --main-gpu N run on CUDA device N of the visible ones (default 0)\n" + " --main-gpu N run on CUDA device N of the visible ones, in nvidia-smi's order unless\n" + " CUDA_DEVICE_ORDER says otherwise (default 0)\n" " --second-gpu N CUDA device N (of the visible ones, not --main-gpu) as another\n" " expert tier: it holds the profile's next experts, then the conversation's,\n" " and computes them beside the CPU pool (native packs, verify windows)\n" @@ -472,6 +473,14 @@ int main(int argc, char** argv) { // pipe or a file is block-buffered, so a program that dies loses every line it had already printed - which // turns "it crashed at step 7" into "it crashed somewhere", and the difference is a debugging session. std::setvbuf(stdout, nullptr, _IONBF, 0); + // device ordinals in nvidia-smi's order, not CUDA's fastest-first guess (before the first CUDA call) + if (std::getenv("CUDA_DEVICE_ORDER") == nullptr) { +#if defined(_WIN32) + _putenv_s("CUDA_DEVICE_ORDER", "PCI_BUS_ID"); +#else + setenv("CUDA_DEVICE_ORDER", "PCI_BUS_ID", 0); +#endif + } Options o; bool have_tokens = false; bool have_logits_stride = false; From dd97f87084af186e786e64706b09c709acc3cffe Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 22:03:24 +0200 Subject: [PATCH 12/92] --window-profile: GPU time of each stage of the verify window --- include/strata/core/verify.hpp | 14 +++++ include/strata/kernels/verify_kernels.hpp | 2 + src/core/verify.cpp | 71 +++++++++++++++++++++++ src/kernels/cuda/verify_kernels.cu | 13 +++++ src/program/generate.cpp | 7 +++ 5 files changed, 107 insertions(+) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 384ccc089ff..d81c9b63eee 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -29,6 +29,7 @@ #include #include +#include namespace strata::core { @@ -77,6 +78,10 @@ class Verifier { /// arena directly, 2 = a copy kernel stages it inside the graph (no API calls on the pool's thread; best when /// the CPU is RAM-bound, Q2_0). Set before the first `run`. void set_pcie_mode(int mode) { sink_.pcie_mode = mode; } + /// Diagnostics: GPU timestamps between the stages of every layer (`--window-profile`; each costs a kernel + /// launch, ~2 us). Set before the first `run`; `print_profile` writes the per-stage table to stdout. + void set_profile(bool on) { profile_ = on; } + void print_profile() const; double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0; int64_t windows = 0; @@ -155,6 +160,15 @@ class Verifier { float *sh_gate_ = nullptr, *sh_up_ = nullptr, *sh_g_ = nullptr; float* hist_snap_ = nullptr; // T * NG_HIST * NG_HC_DIM int64_t cap_ = 0, max_blocks_ = 0, attn_scratch_floats_ = 0; + + // --window-profile: stamps [window size][layer, then one row for the window][stage] + static constexpr int kStamps = 13; + bool profile_ = false; + unsigned long long* stamps_ = nullptr; + std::vector h_stamps_; + std::vector prof_ns_; // [mixer kind (GDN, QSA)][stage], summed + double prof_window_ns_[4] = {}; // inputs + embeddings, head, whole window, stamp gap + int64_t prof_windows_ = 0, prof_layers_[2] = {}; }; } // namespace strata::core diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index 5def3a7c17f..31e3e813399 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -38,6 +38,8 @@ void gdn_step_norm_multi(float* state, const float* h, int conv_channels, const /// Spin until *flag >= value (a mapped host flag). The value is fixed at capture, so several rings can be /// outstanding at once (the split verify window keeps two). void wait_flag_ge(const uint32_t* flag, uint32_t value, void* stream); +/// *dst = the GPU's global nanosecond timer when the stream reaches this point (`--window-profile`). +void gpu_stamp(unsigned long long* dst, void* stream); /// Rows of the S2/S4/S8 embedding for T token ids read from DEVICE memory; out (T, n). Bitwise `embedding_gather`. void embedding_gather_dev(const uint8_t* codes, const float* scales, const float* offsets, const int32_t* tokens, diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 35a73a213e6..600296c711f 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -97,6 +97,7 @@ Verifier::~Verifier() { if (cs_) cudaStreamDestroy(cs_); if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (arena_) cudaFree(arena_); + if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, h_flagA_, h_plan_, h_flagB_}; for (void* h : hosts) @@ -278,6 +279,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { const int G = (split_ && T >= 2) ? 2 : 1; const int tb_[2] = {0, (T + 1) / 2}, te_[2] = {G == 2 ? (T + 1) / 2 : T, T}; groups_[T] = G; + // --window-profile: row l < n_layers holds layer l's stages, row n_layers the window's own stamps + auto stamp = [&](int64_t row, int i) { + if (stamps_ != nullptr && G == 1) + gpu_stamp(stamps_ + ((size_t) T * (size_t) (g.n_layers + 1) + (size_t) row) * kStamps + i, cs); + }; + stamp(g.n_layers, 0); // ---- the window's inputs, from mapped staging copy_i32_from_mapped(tok_, m_tok_, T, cs); @@ -318,6 +325,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { // ---------------------------------------------------------------- pre(l, group): up to the ring auto pre = [&](int64_t l, int grp) -> bool { const int tb = tb_[grp], te = te_[grp], n = te - tb; + stamp(l, 0); const LayerView v(wt, l); const char* pfx[2] = {"hc_attn_", "hc_ffn_"}; const WeightRef *wn[2], *wd[2], *wu[2], *wi[2]; @@ -361,6 +369,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { fused_gr_read_multi(fa, n, xn_ + (size_t) tb * HC * N, cs); }; gr_read_group(0, pending, inj2_, inj_); + stamp(l, 1); float* xm = mixed_ + tb * N; try { if (!is_qsa_layer(g, l)) { @@ -474,14 +483,18 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { err = "verify layer " + std::to_string(l) + ": " + e.what(); return false; } + stamp(l, 2); gr_read_group(1, true, inj_, inj2_); + stamp(l, 3); for (int t = tb; t < te; ++t) { MoEBuffers mb = ss.moe; mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K; if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false; } + stamp(l, 4); doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, m_ids_ + tb * K, m_w_ + tb * K, m_seq_, cs); + stamp(l, 5); { const WeightRef *wgi = need(v, "ffn_gate_inp_shexp.weight", err), *wsg = need(v, "ffn_gate_shexp.weight", err), *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); @@ -507,6 +520,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, cs); else quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, cs); + stamp(l, 6); return true; }; @@ -515,6 +529,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { const int tb = tb_[grp], te = te_[grp], n = te - tb; const uint32_t ring = (uint32_t) (l * G + grp + 1); wait_flag_ge(m_flagA_, ring, cs); // the pool published this group's GPU plan + stamp(l, 7); const int64_t cap = (int64_t) n * K, capx = (int64_t) max_t_ * K; int32_t* pl = plan_ + (size_t) grp * (size_t) (plan_i32_ + 16); copy_i32_from_mapped(pl, m_plan_ + (size_t) grp * (size_t) plan_i32_, plan_i32_, cs); @@ -542,6 +557,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } }; grouped(p_ptr, p_start, p_counts); + stamp(l, 8); wait_flag_ge(m_flagB_, ring, cs); // the PCIe share is in staging (DMA) or mapped if (sink_.pcie_mode == 2) { // stage it with a copy kernel, then point at staging const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; @@ -550,8 +566,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs); } grouped(p_ptr2, p_start2, p_counts + 2); + stamp(l, 9); wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows + stamp(l, 10); copy_from_mapped(parts_ + (size_t) tb * K * N, m_ymiss_ + (size_t) tb * K * N, (int64_t) n * K * N, cs); + stamp(l, 11); moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, p_dst, p_counts + 1, cap, N, cs); for (int t = tb; t < te; ++t) { MoEBuffers mb = ss.moe; @@ -560,9 +579,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); + stamp(l, 12); return true; }; + stamp(g.n_layers, 1); for (int grp = 0; grp < G; ++grp) if (!pre(0, grp)) return false; for (int64_t l = 0; l < g.n_layers; ++l) @@ -570,6 +591,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!post(l, grp)) return false; if (l + 1 < g.n_layers && !pre(l + 1, grp)) return false; } + stamp(g.n_layers, 2); // ---- the head, T columns, and the argmax of each { @@ -600,11 +622,19 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { sp.temperature = 0.0f; sample_tokens(head_logits_, T, (int) n_vocab_, nullptr, 0, sp, m_out_, cs); } + stamp(g.n_layers, 3); + stamp(g.n_layers, 4); // right after the last: one stamp's own cost return true; } bool Verifier::capture(int T, std::string& err) { if (exec_[T] != nullptr) return true; + if (profile_ && stamps_ == nullptr && + cudaMalloc(&stamps_, (size_t) (strata::kernels::kVerifyMaxT + 1) * (size_t) (g_->n_layers + 1) * kStamps * + sizeof(unsigned long long)) != cudaSuccess) { + err = "verify: the profile's stamp buffer does not fit"; + return false; + } if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { err = "verify: begin capture failed"; return false; @@ -795,11 +825,52 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, if (se != cudaSuccess) { err = std::string("verify: ") + cudaGetErrorString(se); return false; } cudaStreamSynchronize(copy_); // no host function of this window may raise flag B in the next one for (int t = 0; t < T; ++t) out[t] = ((volatile int32_t*) h_out_)[t]; + if (stamps_ != nullptr && G == 1) { + const int64_t NL = g.n_layers; + h_stamps_.resize((size_t) (NL + 1) * kStamps); + cudaMemcpy(h_stamps_.data(), stamps_ + (size_t) T * (size_t) (NL + 1) * kStamps, + h_stamps_.size() * sizeof(unsigned long long), cudaMemcpyDeviceToHost); + if (prof_ns_.empty()) prof_ns_.assign(2 * kStamps, 0.0); + const unsigned long long* w = h_stamps_.data() + (size_t) NL * kStamps; + for (int64_t l = 0; l < NL; ++l) { + const unsigned long long* st = h_stamps_.data() + (size_t) l * kStamps; + const int kind = is_qsa_layer(g, l) ? 1 : 0; + const unsigned long long before = l == 0 ? w[1] : (st - kStamps)[kStamps - 1]; + prof_ns_[(size_t) kind * kStamps] += (double) (st[0] - before); + for (int i = 1; i < kStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); + ++prof_layers_[kind]; + } + prof_window_ns_[0] += (double) (w[1] - w[0]); + prof_window_ns_[1] += (double) (w[3] - w[2]); + prof_window_ns_[2] += (double) (w[3] - w[0]); + prof_window_ns_[3] += (double) (w[4] - w[3]); + ++prof_windows_; + } VDBG("window done\n"); ++windows; return true; } +void Verifier::print_profile() const { + if (prof_windows_ == 0) return; + static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", + "router", "doorbell", "shared expert", "wait for plan", + "VRAM experts", "PCIe experts", "wait for CPU", "CPU rows copy", + "combine"}; + const double nw = (double) prof_windows_; + std::printf("%-24s GPU time over %lld windows, ms per window: inputs %.3f, head %.3f, whole window %.3f; " + "one stamp %.2f us (each stage below includes one)\n", "window profile", (long long) prof_windows_, + prof_window_ns_[0] / nw / 1e6, prof_window_ns_[1] / nw / 1e6, prof_window_ns_[2] / nw / 1e6, + prof_window_ns_[3] / nw / 1e3); + std::printf(" %-22s %12s %12s %14s\n", "stage", "GDN us/layer", "QSA us/layer", "ms per window"); + for (int i = 0; i < kStamps; ++i) { + const double a = prof_ns_[(size_t) i], b = prof_ns_[(size_t) kStamps + i]; + std::printf(" %-22s %12.1f %12.1f %14.3f\n", names[i], + prof_layers_[0] ? a / (double) prof_layers_[0] / 1e3 : 0.0, + prof_layers_[1] ? b / (double) prof_layers_[1] / 1e3 : 0.0, (a + b) / nw / 1e6); + } +} + void Verifier::set_plan_slot(int grp) { const int64_t cap = sink_.cap; int32_t* base = h_plan_ + (size_t) grp * (size_t) plan_i32_; diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index bca54aee6f9..a5216da28c4 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -432,6 +432,19 @@ void wait_flag_ge(const uint32_t* flag, uint32_t value, void* stream) { check("wait_flag_ge"); } +namespace { +__global__ void gpu_stamp_kernel(unsigned long long* dst) { + unsigned long long t; + asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(t)); + *dst = t; +} +} // namespace + +void gpu_stamp(unsigned long long* dst, void* stream) { + gpu_stamp_kernel<<<1, 1, 0, (cudaStream_t) stream>>>(dst); + check("gpu_stamp"); +} + void embedding_gather_dev(const uint8_t* codes, const float* scales, const float* offsets, const int32_t* tokens, int n_tok, int64_t n, int code_bits, int code_bias, int group_elems, uint64_t row_codes, uint64_t row_groups, float* out, void* stream) { diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 01d37c19504..7a53e96da7b 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -243,6 +243,7 @@ struct Options { bool stop_eos = false; std::vector eos_ids = {248044, 248046}; bool spec_split = false; ///< opt-in split verify window (the overlap study: exact, ~7% slower) + bool window_profile = false; ///< GPU timestamps between the verify window's stages, printed by --stats /// Plan v0.3 P8: stay resident and take requests on stdin (see the --serve block in main). bool serve = false; /// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests. @@ -357,6 +358,8 @@ void usage() { " --gpu-only-full MEASURE: replay pre+post for all 48 layers plus the LM head, no pool.\n" " THE TRUE PER-TOKEN GPU FLOOR. Quote this one, not --graph-only.\n" " --stats print the per-stage breakdown\n" + " --window-profile with --stats: the GPU time of each stage of the verify window (timestamps\n" + " inside the graph; each costs ~2 us)\n" " --gpu-stages R0.9: capture the layer as three graphs (mixer / ffn+router / post)\n" " and time them from OUTSIDE the capture. The per-stage table on the\n" " real graph that --stage-timing cannot give. Prints and exits.\n" @@ -592,6 +595,7 @@ int main(int argc, char** argv) { else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); else if (a == "--stop-eos") o.stop_eos = true; else if (a == "--spec-split") o.spec_split = true; + else if (a == "--window-profile") o.window_profile = true; else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode"); else if (a == "--serve") o.serve = true; else if (a == "--vision") o.vision = true; @@ -1993,6 +1997,7 @@ int main(int argc, char** argv) { return 1; } ver.set_split(o.spec_split); + ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 std::vector cur; @@ -2659,6 +2664,7 @@ int main(int argc, char** argv) { return 1; } ver.set_split(o.spec_split); + ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 drive.d.plan = ver.plan_sink(); @@ -2810,6 +2816,7 @@ int main(int argc, char** argv) { std::printf("%-24s plan %.3f activation quantize %.3f jobs %.3f run %.3f ms/round\n", "dispatch", drive.d.ms_plan / rounds, drive.d.ms_actq / rounds, drive.d.ms_jobs / rounds, drive.d.ms_run / rounds); + ver.print_profile(); if (rounds > 0 && !drive.d.usage.empty()) std::printf("%-24s %lld experts swapped into the VRAM tier (every %d rounds, %.3f ms/round)\n", "adaptive tier", (long long) tier.swaps, o.adapt_every, tier.ms / rounds); From 6bc67debdcc7c952b568a25bce02083179c281a8 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 22:22:07 +0200 Subject: [PATCH 13/92] Verify window: one gather-combine reads only the host's rows over PCIe; no PCIe stage without a PCIe share --- CMakeLists.txt | 6 + include/strata/core/expert_source.hpp | 2 + include/strata/core/verify.hpp | 7 +- include/strata/kernels/native_moe.hpp | 8 ++ src/core/expert_source.cpp | 3 +- src/core/verify.cpp | 47 +++++--- src/kernels/cuda/native_moe.cu | 55 ++++++++++ src/kernels/verify_parity.cpp | 151 ++++++++++++++++++++++++++ src/program/generate.cpp | 2 + 9 files changed, 263 insertions(+), 18 deletions(-) create mode 100644 src/kernels/verify_parity.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 1ce11f23e92..2f838b33507 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -361,6 +361,12 @@ if(STRATA_ENABLE_CUDA) target_link_libraries(kv_q8_parity PRIVATE strata_kernels CUDA::cudart) add_test(NAME kv_q8_parity COMMAND kv_q8_parity) + # The verify window's batched kernels vs the per-token kernel sequences they replace, bitwise (synthetic data). + add_executable(verify_parity src/kernels/verify_parity.cpp) + target_include_directories(verify_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) + target_link_libraries(verify_parity PRIVATE strata_kernels CUDA::cudart) + add_test(NAME verify_parity COMMAND verify_parity --selftest) + # Plan v0.3 P3: native MMVQ for 1..8 tokens (bitwise selftest + --bench timing); synthetic data, no model. if(STRATA_BUILD_TESTS) add_executable(native_mmvq_multi bench/micro/native_mmvq_multi.cpp) diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index 283f6cf4308..e2f97d7f23f 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -88,6 +88,8 @@ struct GpuPlanSink { /// kernel reads the mapped arena directly; 2 = a copy kernel stages it inside the graph. For 1 and 2 `ptr2` /// holds the arena's device alias. int pcie_mode = 0; + /// The GPU reads only the rows it did not compute, so the pool leaves the GPU's rows of `out` unwritten. + bool host_rows_only = false; }; /// The adapter's own state. One per session, reused every layer so the token path allocates nothing (P2.T10). diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index d81c9b63eee..a6a6c26215c 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -35,7 +35,8 @@ namespace strata::core { class NativeHead; -/// The CPU pool for a window: x_f (n_tok, n_embd), ids (n_tok, k) -> out (n_tok * k, n_embd), hit rows zeroed. +/// The CPU pool for a window: x_f (n_tok, n_embd), ids (n_tok, k) -> out (n_tok * k, n_embd): the CPU's and the +/// second GPU's rows, the main GPU's zeroed unless `GpuPlanSink::host_rows_only`. using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out, int64_t layer); @@ -78,6 +79,9 @@ class Verifier { /// arena directly, 2 = a copy kernel stages it inside the graph (no API calls on the pool's thread; best when /// the CPU is RAM-bound, Q2_0). Set before the first `run`. void set_pcie_mode(int mode) { sink_.pcie_mode = mode; } + /// Whether the pool may give the GPU a PCIe share of the misses (`--pcie-frac` > 0); without one the window + /// has no PCIe stage. Set before the first `run`. + void set_pcie_share(bool on) { pcie_share_ = on; } /// Diagnostics: GPU timestamps between the stages of every layer (`--window-profile`; each costs a kernel /// launch, ~2 us). Set before the first `run`; `print_profile` writes the per-stage table to stdout. void set_profile(bool on) { profile_ = on; } @@ -131,6 +135,7 @@ class Verifier { static void publish_plan(void* ctx); void set_plan_slot(int grp); bool split_ = false; // opt-in (--spec-split): exact but slower, see the overlap study + bool pcie_share_ = true; int groups_[9] = {}; float* h_ymiss_ = nullptr; float* m_ymiss_ = nullptr; // T * k * n_embd diff --git a/include/strata/kernels/native_moe.hpp b/include/strata/kernels/native_moe.hpp index 0c48d6ae8b0..793231770dd 100644 --- a/include/strata/kernels/native_moe.hpp +++ b/include/strata/kernels/native_moe.hpp @@ -15,4 +15,12 @@ bool native_moe_combine_enabled(); // Requires a nonnull ordered stream and disjoint output. No allocation or sync. void native_moe_combine(const float* parts, const float* weights, const float* shared, float* output, int64_t n_embd, int64_t k, void* stream); + +// The verify window's combine for n_tok tokens, bitwise `native_moe_combine` per token over the rows the window +// used to assemble: row t*k + j is the GPU's (`gpu_rows`, row-indexed, as the host's zeroed row plus the hit) when +// an entry of dst[0..*count) names it, and otherwise the host's (`host_rows`, mapped memory: only these rows cross +// PCIe). weights (n_tok, k), shared and output (n_tok, n_embd). Graph-capturable. +void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, + const int32_t* count, const float* weights, const float* shared, float* output, + int64_t n_embd, int64_t k, int n_tok, void* stream); } diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index aeef419ed69..b891dded379 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -402,6 +402,7 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) ? 0 : -1; } } + const bool skip_gpu_rows = d.plan != nullptr && d.plan->host_rows_only && n <= 128 && n <= d.plan->cap; const auto c1 = std::chrono::steady_clock::now(); if (native && lay.fmt[(size_t) d.layers].gu_type == 42) // a native Q2_0 pack: the Q2_0 kernels' activations for (int64_t t = 0; t < n_tok; ++t) act_quant_any(x_f + (size_t) t * H, H, d.act_multi[(size_t) t]); @@ -427,7 +428,7 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (kind[i] >= 0) { // a GPU computes this entry (a VRAM hit, a PCIe read, the second GPU) if (kind[i] == 0) ++d.cache_hits; if (kind[i] == 2) ++d.gpu2_entries; - std::memset(row, 0, (size_t) H * sizeof(float)); + if (!skip_gpu_rows) std::memset(row, 0, (size_t) H * sizeof(float)); continue; } ++d.cache_refused; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 600296c711f..893672c925c 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -14,6 +14,7 @@ #include "strata/kernels/gr.hpp" #include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_mmvq.hpp" +#include "strata/kernels/native_moe.hpp" #include "strata/kernels/native_qsa.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_rope.hpp" @@ -183,6 +184,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sink_.start2 = h_plan_ + ptr_off + 4 * cap; sink_.cap = cap; sink_.publish = &Verifier::publish_plan; + sink_.host_rows_only = strata::kernels::native_moe_combine_enabled(); // the gather-combine (post) sink_.fetch = &Verifier::fetch_dma; sink_.ctx = this; } @@ -558,25 +560,38 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { }; grouped(p_ptr, p_start, p_counts); stamp(l, 8); - wait_flag_ge(m_flagB_, ring, cs); // the PCIe share is in staging (DMA) or mapped - if (sink_.pcie_mode == 2) { // stage it with a copy kernel, then point at staging - const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; - uint8_t* stage = staging_ + (size_t) (grp * per) * lay.max_blob; - fetch_blobs(p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs); - rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs); + if (pcie_share_) { + wait_flag_ge(m_flagB_, ring, cs); // the PCIe share is in staging (DMA) or mapped + if (sink_.pcie_mode == 2) { // stage it with a copy kernel, then point at staging + const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; + uint8_t* stage = staging_ + (size_t) (grp * per) * lay.max_blob; + fetch_blobs(p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs); + rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs); + } + grouped(p_ptr2, p_start2, p_counts + 2); } - grouped(p_ptr2, p_start2, p_counts + 2); stamp(l, 9); wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows stamp(l, 10); - copy_from_mapped(parts_ + (size_t) tb * K * N, m_ymiss_ + (size_t) tb * K * N, (int64_t) n * K * N, cs); - stamp(l, 11); - moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, p_dst, p_counts + 1, cap, N, cs); - for (int t = tb; t < te; ++t) { - MoEBuffers mb = ss.moe; - mb.weights = w_ + t * K; mb.shared = shared_ + t * N; - if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; + if (native_moe_combine_enabled()) { + // only the host's rows cross PCIe; the GPU's come from `hit_out` (the pool leaves their rows unwritten) + try { + native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, p_dst, p_counts + 1, w_ + tb * K, + shared_ + tb * N, bo_ + tb * N, N, K, n, cs); + } catch (const std::exception& e) { + err = "verify layer " + std::to_string(l) + " combine: " + e.what(); + return false; + } + } else { + copy_from_mapped(parts_ + (size_t) tb * K * N, m_ymiss_ + (size_t) tb * K * N, (int64_t) n * K * N, cs); + moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, p_dst, p_counts + 1, cap, N, cs); + for (int t = tb; t < te; ++t) { + MoEBuffers mb = ss.moe; + mb.weights = w_ + t * K; mb.shared = shared_ + t * N; + if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; + } } + stamp(l, 11); if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); stamp(l, 12); @@ -855,8 +870,8 @@ void Verifier::print_profile() const { if (prof_windows_ == 0) return; static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", "router", "doorbell", "shared expert", "wait for plan", - "VRAM experts", "PCIe experts", "wait for CPU", "CPU rows copy", - "combine"}; + "VRAM experts", "PCIe experts", "wait for CPU", "rows + combine", + "last HC write"}; const double nw = (double) prof_windows_; std::printf("%-24s GPU time over %lld windows, ms per window: inputs %.3f, head %.3f, whole window %.3f; " "one stamp %.2f us (each stage below includes one)\n", "window profile", (long long) prof_windows_, diff --git a/src/kernels/cuda/native_moe.cu b/src/kernels/cuda/native_moe.cu index 42b57ca0f84..ebc912d97c3 100644 --- a/src/kernels/cuda/native_moe.cu +++ b/src/kernels/cuda/native_moe.cu @@ -43,6 +43,48 @@ __global__ void combine(const float* __restrict__ parts, const float* __restrict if (shared) sum += shared[col]; output[col] = sum; } +constexpr int kMaxK = 15; +// One block row per token. The rows a GPU computed are marked from the plan's entries first; each column then +// loads its k values (the host's over PCIe, all in flight at once) and sums them as `combine` does. +__global__ void gather_combine(const float* __restrict__ gpu_rows, const float* host_rows, + const int32_t* __restrict__ dst, const int32_t* __restrict__ count, + const float* __restrict__ weights, const float* __restrict__ shared, + float* __restrict__ output, int64_t n_embd, int k) { + __shared__ unsigned on_gpu; // bit j: row t*k + j is the GPU's + const int t = blockIdx.y; + if (threadIdx.x == 0) on_gpu = 0u; + __syncthreads(); + const int c = *count; + for (int i = threadIdx.x; i < c; i += blockDim.x) { + const int r = dst[i] - t * k; + if (r >= 0 && r < k) atomicOr(&on_gpu, 1u << r); + } + __syncthreads(); + const int64_t col = int64_t(blockIdx.x) * blockDim.x + threadIdx.x; + if (col >= n_embd) return; + const unsigned mask = on_gpu; + const float* w = weights + int64_t(t) * k; + float v[kMaxK]; +#pragma unroll + for (int j = 0; j < kMaxK; ++j) { + if (j >= k) break; + const int64_t at = (int64_t(t) * k + j) * n_embd + col; + if ((mask >> j) & 1u) { + const float h = gpu_rows[at]; + v[j] = __float_as_uint(h) == 0x80000000u ? 0.0f : h; // 0 + h: the host's zeroed row plus the hit + } else { + v[j] = host_rows[at]; + } + } + float sum = v[0] * w[0]; +#pragma unroll + for (int j = 1; j < kMaxK; ++j) { + if (j >= k) break; + sum += v[j] * w[j]; + } + if (shared) sum += shared[int64_t(t) * n_embd + col]; + output[int64_t(t) * n_embd + col] = sum; +} bool valid_span(const void* p, size_t bytes) { const auto address = reinterpret_cast(p); return p && address % alignof(float) == 0 && bytes <= UINTPTR_MAX - address; @@ -71,4 +113,17 @@ void native_moe_combine(const float* parts, const float* weights, const float* s const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } +void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, + const int32_t* count, const float* weights, const float* shared, float* output, + int64_t n_embd, int64_t k, int n_tok, void* stream) { + if (!stream || n_embd <= 0 || n_embd > std::numeric_limits::max() || k < 1 || k > kMaxK || n_tok < 1 || + !gpu_rows || !host_rows || !dst || !count || !weights || !output) + throw std::invalid_argument("native MoE gather-combine requires a stream, positive width, 1..15 experts " + "and its buffers"); + const dim3 grid(unsigned((n_embd + 255) / 256), unsigned(n_tok)); + gather_combine<<(stream)>>>(gpu_rows, host_rows, dst, count, weights, + shared, output, n_embd, int(k)); + const auto error = cudaGetLastError(); + if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); +} } diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp new file mode 100644 index 00000000000..0a4f21a244e --- /dev/null +++ b/src/kernels/verify_parity.cpp @@ -0,0 +1,151 @@ +// src/kernels/verify_parity.cpp - the verify window's batched kernels against the kernel sequences they replace. +// +// A verify window's token t must come out bit for bit as it would in a window of any other size (the drafts are +// accepted exactly when greedy decode would have produced them), so every batched kernel is checked BITWISE against +// the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. +#include "strata/kernels/elementwise.hpp" +#include "strata/kernels/native_moe.hpp" +#include "strata/kernels/s2_expert_grouped.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +void check(cudaError_t e, const char* what) { + if (e != cudaSuccess) { + std::fprintf(stderr, "%s: %s\n", what, cudaGetErrorString(e)); + std::exit(1); + } +} + +template T* dev(size_t n) { + T* p = nullptr; + check(cudaMalloc(&p, n * sizeof(T)), "cudaMalloc"); + return p; +} +template void up(T* d, const std::vector& h) { + check(cudaMemcpy(d, h.data(), h.size() * sizeof(T), cudaMemcpyHostToDevice), "upload"); +} +template std::vector down(const T* d, size_t n) { + std::vector h(n); + check(cudaMemcpy(h.data(), d, n * sizeof(T), cudaMemcpyDeviceToHost), "download"); + return h; +} + +// values that exercise the rounding edges: ordinary, large, tiny, denormal, -0.0 +float edgy(std::mt19937& rng) { + std::uniform_real_distribution u(-1.0f, 1.0f); + const int kind = (int) (rng() % 16); + if (kind == 0) return -0.0f; + if (kind == 1) return 0.0f; + if (kind == 2) return u(rng) * 1e-39f; // denormal + if (kind == 3) return u(rng) * 1e4f; + return u(rng); +} + +int bitwise_diff(const std::vector& a, const std::vector& b, const char* what) { + int bad = 0; + for (size_t i = 0; i < a.size(); ++i) { + uint32_t x, y; + std::memcpy(&x, &a[i], 4); + std::memcpy(&y, &b[i], 4); + if (x != y && !(std::isnan(a[i]) && std::isnan(b[i]))) { + if (bad < 5) std::fprintf(stderr, " %s: element %zu: %.9g (0x%08x) vs %.9g (0x%08x)\n", what, i, a[i], x, b[i], y); + ++bad; + } + } + return bad; +} + +// ---- the MoE combine: copy all rows + add the hits + one combine per token vs native_moe_gather_combine +int test_gather_combine(std::mt19937& rng, cudaStream_t s) { + const int64_t N = 2560, K = 10; + int bad = 0; + for (int n_tok : {1, 2, 3, 4, 8}) { + for (int rep = 0; rep < 8; ++rep) { + const int64_t rows = n_tok * K; + std::vector hit((size_t) (rows * N)), host((size_t) (rows * N)), w((size_t) rows), + shared((size_t) (n_tok * N)); + for (auto& v : hit) v = edgy(rng); + for (auto& v : host) v = edgy(rng); + for (auto& v : w) v = std::uniform_real_distribution(0.0f, 1.0f)(rng); + for (auto& v : shared) v = edgy(rng); + // the GPU's entries, in a random order and a random share (none, some, all) + std::vector dst; + for (int32_t r = 0; r < rows; ++r) + if (rep == 1 || (rep != 0 && rng() % 3 != 0)) dst.push_back(r); + std::shuffle(dst.begin(), dst.end(), rng); + const int32_t count = (int32_t) dst.size(); + // the old path's host rows: the pool zeroed the GPU's + std::vector host_zeroed = host; + for (int32_t r : dst) std::fill(host_zeroed.begin() + r * N, host_zeroed.begin() + (r + 1) * N, 0.0f); + // the new path's mapped rows: the GPU's rows hold garbage it must not read + std::vector host_poison = host; + for (int32_t r : dst) std::fill(host_poison.begin() + r * N, host_poison.begin() + (r + 1) * N, NAN); + + float *d_hit = dev(hit.size()), *d_parts = dev(hit.size()), *d_w = dev(w.size()), + *d_sh = dev(shared.size()), *d_out_old = dev(shared.size()), + *d_out_new = dev(shared.size()); + int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(1); + up(d_hit, hit); + up(d_w, w); + up(d_sh, shared); + if (!dst.empty()) up(d_dst, dst); + up(d_count, std::vector{count}); + float* h_map = nullptr; + float* m_map = nullptr; + check(cudaHostAlloc((void**) &h_map, host.size() * sizeof(float), cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostGetDevicePointer((void**) &m_map, h_map, 0), "cudaHostGetDevicePointer"); + + // old: copy all rows, add the hits, combine per token + std::memcpy(h_map, host_zeroed.data(), host.size() * sizeof(float)); + strata::kernels::copy_from_mapped(d_parts, m_map, rows * N, s); + strata::kernels::moe_hit_add(d_parts, d_hit, d_dst, d_count, rows, N, s); + for (int t = 0; t < n_tok; ++t) + strata::kernels::native_moe_combine(d_parts + t * K * N, d_w + t * K, d_sh + t * N, d_out_old + t * N, + N, K, s); + check(cudaStreamSynchronize(s), "old path"); + // new + std::memcpy(h_map, host_poison.data(), host.size() * sizeof(float)); + strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count, d_w, d_sh, d_out_new, N, K, n_tok, s); + check(cudaStreamSynchronize(s), "new path"); + const int b = bitwise_diff(down(d_out_old, shared.size()), down(d_out_new, shared.size()), "gather_combine"); + if (b) std::fprintf(stderr, "gather_combine: n_tok %d rep %d: %d of %zu differ\n", n_tok, rep, b, shared.size()); + bad += b; + cudaFreeHost(h_map); + for (void* p : {(void*) d_hit, (void*) d_parts, (void*) d_w, (void*) d_sh, (void*) d_out_old, + (void*) d_out_new, (void*) d_dst, (void*) d_count}) + cudaFree(p); + } + } + std::printf("gather_combine: %s\n", bad ? "MISMATCH" : "bitwise equal (n_tok 1-4, 8; 8 plans each)"); + return bad; +} + +} // namespace + +int main(int argc, char** argv) { + for (int i = 1; i < argc; ++i) { + if (std::string(argv[i]) != "--selftest") { + std::fprintf(stderr, "usage: verify_parity [--selftest]\n"); + return 2; + } + } + std::mt19937 rng(20260925); + cudaStream_t s = nullptr; + check(cudaStreamCreate(&s), "stream"); + int bad = 0; + bad += test_gather_combine(rng, s); + cudaStreamDestroy(s); + std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); + return bad ? 1 : 0; +} diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 7a53e96da7b..de3e5da4d5c 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2008,6 +2008,7 @@ int main(int argc, char** argv) { }; drive.d.plan = ver.plan_sink(); drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5))); + ver.set_pcie_share(drive.d.pcie_num > 0); if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); // ---- the prompt cache (strata/core/prompt_cache.hpp) std::vector kv_states; @@ -2671,6 +2672,7 @@ int main(int argc, char** argv) { drive.d.pcie_num = (int) (o.pcie_frac * 256.0 + 0.5); if (drive.d.pcie_num < 0) drive.d.pcie_num = 0; if (drive.d.pcie_num > 256) drive.d.pcie_num = 256; + ver.set_pcie_share(drive.d.pcie_num > 0); const int64_t pcie0 = drive.d.pcie_experts; if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); int64_t p = spec_pos; From 22f4b8c7051e8f3a4b2368d8ab577bacf5359f61 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 22:29:52 +0200 Subject: [PATCH 14/92] Verify window: multi-token router, indexer projections and shared-expert gate (one weight read, one launch) --- include/strata/kernels/bf16_gemv.hpp | 4 ++ include/strata/kernels/native_router.hpp | 2 + src/core/verify.cpp | 31 +++++---- src/kernels/cuda/native_bf16.cu | 81 ++++++++++++++++++++++++ src/kernels/cuda/native_router.cu | 14 +++- src/kernels/cuda/shared_expert.cu | 14 ++-- src/kernels/verify_parity.cpp | 70 ++++++++++++++++++++ 7 files changed, 197 insertions(+), 19 deletions(-) diff --git a/include/strata/kernels/bf16_gemv.hpp b/include/strata/kernels/bf16_gemv.hpp index 6f3619d98c1..020bf30b7f0 100644 --- a/include/strata/kernels/bf16_gemv.hpp +++ b/include/strata/kernels/bf16_gemv.hpp @@ -44,5 +44,9 @@ void bf16_gemv_split(const uint16_t* x, const uint16_t* w, float* y, int64_t n_i /// No allocations or synchronization, including when stream is null (the CUDA default stream). void bf16_gemv_fp32_mmvf(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, void* stream); +/// `bf16_gemv_fp32_mmvf` for n_tok (1..8) activation rows x (n_tok, n_in) -> y (n_tok, n_out), each weight read once; +/// every row bitwise what the single-row call gives. +void bf16_gemv_fp32_mmvf_multi(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, int n_tok, + void* stream); } // namespace strata::kernels diff --git a/include/strata/kernels/native_router.hpp b/include/strata/kernels/native_router.hpp index 4b3dccdabdc..0ed95bac79c 100644 --- a/include/strata/kernels/native_router.hpp +++ b/include/strata/kernels/native_router.hpp @@ -14,4 +14,6 @@ bool native_router_enabled(); // four-byte aligned and outputs disjoint from each other and the input. // Requires a nonnull ordered CUDA stream. No allocation or synchronization. void native_router_top10(const float* logits, int32_t* ids, float* weights, void* stream); +// `native_router_top10` for n_tok tokens in one launch: logits (n_tok, 512) -> ids, weights (n_tok, 10). +void native_router_top10_multi(const float* logits, int32_t* ids, float* weights, int n_tok, void* stream); } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 893672c925c..c86db750535 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -17,6 +17,7 @@ #include "strata/kernels/native_moe.hpp" #include "strata/kernels/native_qsa.hpp" #include "strata/kernels/native_qsa_indexer.hpp" +#include "strata/kernels/native_router.hpp" #include "strata/kernels/native_rope.hpp" #include "strata/kernels/ngram.hpp" #include "strata/kernels/ple.hpp" @@ -425,8 +426,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { }; float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; native_quantize_q8_1(xm, xq_, (int) N, n, cs); - for (int t = tb; t < te; ++t) - bf16_gemv_fp32_mmvf(mixed_ + t * N, (const uint16_t*) wik->data, idx_raw + t * ID, (int) N, (int) ID, cs); + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, cs); native_mmvq(wk->native_type, wk->native_data, xq_, kcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); native_mmvq(wv->native_type, wv->native_data, xq_, vcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH); @@ -456,11 +456,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } norm_rope(qc, wqn, (int) NH, (int) HD, pos_ + t * NH); } - for (int t = tb; t < te; ++t) { - float* qx = qidx_ + t * IQ * ID; - bf16_gemv_fp32_mmvf(mixed_ + t * N, (const uint16_t*) wiq->data, qx, (int) N, (int) (IQ * ID), cs); - norm_rope(qx, wiqn, (int) IQ, (int) ID, pos_ + t * NH); - } + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, N, IQ * ID, n, cs); + for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH); qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, s, scores_ + (size_t) tb * max_blocks_, cs); qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, @@ -488,10 +485,22 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 2); gr_read_group(1, true, inj_, inj2_); stamp(l, 3); - for (int t = tb; t < te; ++t) { - MoEBuffers mb = ss.moe; - mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K; - if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false; + if (native_router_enabled()) { // one weight read and one launch each for the window's tokens + const WeightRef* wr = need(v, "ffn_gate_inp.weight", err); + if (!wr) return false; + try { + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wr->data, logits_ + tb * NE, N, NE, n, cs); + native_router_top10_multi(logits_ + tb * NE, ids_ + tb * K, w_ + tb * K, n, cs); + } catch (const std::exception& e) { + err = "verify layer " + std::to_string(l) + " router: " + e.what(); + return false; + } + } else { + for (int t = tb; t < te; ++t) { + MoEBuffers mb = ss.moe; + mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K; + if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false; + } } stamp(l, 4); doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, diff --git a/src/kernels/cuda/native_bf16.cu b/src/kernels/cuda/native_bf16.cu index 5131b20b9b1..7471fedb83d 100644 --- a/src/kernels/cuda/native_bf16.cu +++ b/src/kernels/cuda/native_bf16.cu @@ -70,6 +70,56 @@ __global__ void bf16_f32_mmvf_kernel(const float* __restrict__ x, const uint16_t if (t == 0) y[blockIdx.x] = acc; } +// `bf16_f32_mmvf_kernel` for up to kMultiT activation rows: each thread walks the same pairs in the same order for +// every row, and each row is reduced exactly as the single-row kernel reduces it. +constexpr int kMultiT = 8; +template +__global__ void bf16_f32_mmvf_multi_kernel(const float* __restrict__ x, const uint16_t* __restrict__ w, + float* __restrict__ y, int n_in, int n_out, int n_tok) { + const int t = threadIdx.x; + const uint16_t* row = w + (size_t) blockIdx.x * n_in; + const uint32_t* weights2 = reinterpret_cast(row); + __shared__ float partials[kMultiT][32]; + if constexpr (BLOCK_SIZE > 32) { + if (t < 32) + for (int k = 0; k < kMultiT; ++k) partials[k][t] = 0.0f; + __syncthreads(); + } + float acc[kMultiT]; +#pragma unroll + for (int k = 0; k < kMultiT; ++k) acc[k] = 0.0f; + for (int pair = t; pair < n_in / 2; pair += BLOCK_SIZE) { + const uint32_t weight = weights2[pair]; + const float w0 = f32_from_bf16((uint16_t) weight), w1 = f32_from_bf16((uint16_t) (weight >> 16)); +#pragma unroll + for (int k = 0; k < kMultiT; ++k) { + if (k >= n_tok) break; + const float2 input = reinterpret_cast(x + (size_t) k * n_in)[pair]; + acc[k] = __fmaf_rn(w0, input.x, acc[k]); + acc[k] = __fmaf_rn(w1, input.y, acc[k]); + } + } +#pragma unroll + for (int k = 0; k < kMultiT; ++k) { + if (k >= n_tok) break; + acc[k] = mmvf_warp_sum(acc[k]); + } + if constexpr (BLOCK_SIZE > 32) { + if ((t & 31) == 0) + for (int k = 0; k < n_tok; ++k) partials[k][t / 32] = acc[k]; + __syncthreads(); + if (t < 32) { +#pragma unroll + for (int k = 0; k < kMultiT; ++k) { + if (k >= n_tok) break; + acc[k] = mmvf_warp_sum(partials[k][t]); + } + } + } + if (t == 0) + for (int k = 0; k < n_tok; ++k) y[(size_t) k * n_out + blockIdx.x] = acc[k]; +} + int mmvf_block_size(int64_t n_in) { int best = 32; int64_t best_iterations = (n_in + 63) / 64; @@ -115,4 +165,35 @@ void bf16_gemv_fp32_mmvf(const float* x, const uint16_t* w, float* y, } +void bf16_gemv_fp32_mmvf_multi(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, int n_tok, + void* stream) { + if (n_tok < 1 || n_tok > kMultiT) + throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: 1..8 activation rows"); + if (n_in <= 0 || (n_in & 1) != 0 || n_in > std::numeric_limits::max() || + n_out <= 0 || n_out > std::numeric_limits::max()) + throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: require positive even n_in and positive n_out <= INT_MAX"); + if (x == nullptr || w == nullptr || y == nullptr || + (reinterpret_cast(x) & 7u) != 0 || + (reinterpret_cast(w) & 3u) != 0 || + (reinterpret_cast(y) & 3u) != 0) + throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: null or misaligned pointer"); + const cudaStream_t st = (cudaStream_t) stream; +#define STRATA_MMVF_CASE(N) case N: \ + bf16_f32_mmvf_multi_kernel<<<(unsigned) n_out, N, 0, st>>>(x, w, y, (int) n_in, (int) n_out, n_tok); break + switch (mmvf_block_size(n_in)) { + STRATA_MMVF_CASE(32); + STRATA_MMVF_CASE(64); + STRATA_MMVF_CASE(96); + STRATA_MMVF_CASE(128); + STRATA_MMVF_CASE(160); + STRATA_MMVF_CASE(192); + STRATA_MMVF_CASE(224); + STRATA_MMVF_CASE(256); + } +#undef STRATA_MMVF_CASE + const cudaError_t result = cudaGetLastError(); + if (result != cudaSuccess) + throw std::runtime_error(std::string("bf16_gemv_fp32_mmvf_multi launch: ") + cudaGetErrorString(result)); +} + } // namespace strata::kernels diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index 12456371266..00e062996fd 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -44,8 +44,11 @@ __device__ __forceinline__ float warp_max(float value) { __launch_bounds__(256, 1) __global__ void route(const float* __restrict__ logits, int32_t* __restrict__ ids, float* __restrict__ weights) { - // Preserve the pinned 32x8 block geometry; only row zero is active here. + // Preserve the pinned 32x8 block geometry; only row zero is active here. Block b routes token b. if (threadIdx.y != 0) return; + logits += blockIdx.x * 512; + ids += blockIdx.x * 10; + weights += blockIdx.x * 10; const int lane = threadIdx.x; float values[16]; #pragma unroll @@ -114,4 +117,13 @@ void native_router_top10(const float* logits, int32_t* ids, float* weights, void const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } +void native_router_top10_multi(const float* logits, int32_t* ids, float* weights, int n_tok, void* stream) { + const size_t nl = size_t(n_tok) * 512 * 4, no = size_t(n_tok) * 10 * 4; + if (!stream || n_tok < 1 || !valid(logits, nl) || !valid(ids, no) || !valid(weights, no) + || overlap(logits, nl, ids, no) || overlap(logits, nl, weights, no) || overlap(ids, no, weights, no)) + throw std::invalid_argument("native router requires a stream, aligned spans, and disjoint outputs"); + route<<(stream)>>>(logits, ids, weights); + const auto error = cudaGetLastError(); + if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); +} } diff --git a/src/kernels/cuda/shared_expert.cu b/src/kernels/cuda/shared_expert.cu index aba29bbb4c3..52ce6c044e8 100644 --- a/src/kernels/cuda/shared_expert.cu +++ b/src/kernels/cuda/shared_expert.cu @@ -128,7 +128,8 @@ __global__ void scale_kernel(float* __restrict__ out, const float* __restrict__ __global__ void native_scalar_sigmoid_kernel(float* gate) { // Match the single-token CUDA sigmoid's FP32 fast-math operations without changing legacy kernels' // compilation flags. The dot product was already reduced by the pinned native MMVF implementation. - gate[0] = __fdividef(1.0f, 1.0f + __expf(-gate[0])); + // One thread per token. + gate[threadIdx.x] = __fdividef(1.0f, 1.0f + __expf(-gate[threadIdx.x])); } /// The MoE block's final combination. See the header for the two readings it exists to pin. @@ -171,13 +172,12 @@ void shared_expert_multi(int n_tok, const float* x, const uint16_t* x_bf16, cons native_swiglu_kernel<<<(unsigned) ((n + THREADS - 1) / THREADS), THREADS, 0, cs>>>(gate, up, gate, n); native_quantize_q8_1(gate, nw.q8_1, (int) n_ff, n_tok, stream); native_mmvq(nw.down_type, nw.down_data, nw.q8_1, out, (int) n_ff, (int) n_embd, n_tok, stream); - for (int t = 0; t < n_tok; ++t) { - if (native_bf16) { - bf16_gemv_fp32_mmvf(x + (size_t) t * n_embd, gate_inp_bf16, g + t, n_embd, 1, stream); - native_scalar_sigmoid_kernel<<<1, 1, 0, cs>>>(g + t); - } else { + if (native_bf16) { + bf16_gemv_fp32_mmvf_multi(x, gate_inp_bf16, g, n_embd, 1, n_tok, stream); + native_scalar_sigmoid_kernel<<<1, (unsigned) n_tok, 0, cs>>>(g); + } else { + for (int t = 0; t < n_tok; ++t) scalar_gate_kernel<<<1, 256, 0, cs>>>(x_bf16 + (size_t) t * n_embd, gate_inp_bf16, g + t, (int) n_embd); - } } scale_rows_kernel<<>>( out, g, (int) n_embd); diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 0a4f21a244e..1c6b5b838db 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -3,8 +3,10 @@ // A verify window's token t must come out bit for bit as it would in a window of any other size (the drafts are // accepted exactly when greedy decode would have produced them), so every batched kernel is checked BITWISE against // the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. +#include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/native_moe.hpp" +#include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" #include @@ -131,6 +133,72 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- BF16 x FP32 MMVF: one call per row vs bf16_gemv_fp32_mmvf_multi (the router, indexer and gate shapes) +int test_mmvf_multi(std::mt19937& rng, cudaStream_t s) { + int bad = 0; + for (int64_t n_out : {512, 128, 1}) { + const int64_t n_in = 2560; + std::vector w((size_t) (n_in * n_out)); + for (auto& v : w) { + const float f = edgy(rng); + uint32_t b; + std::memcpy(&b, &f, 4); + v = (uint16_t) (b >> 16); + } + uint16_t* d_w = dev(w.size()); + up(d_w, w); + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + std::vector x((size_t) (n_tok * n_in)); + for (auto& v : x) v = edgy(rng); + float *d_x = dev(x.size()), *d_a = dev((size_t) (n_tok * n_out)), + *d_b = dev((size_t) (n_tok * n_out)); + up(d_x, x); + for (int t = 0; t < n_tok; ++t) + strata::kernels::bf16_gemv_fp32_mmvf(d_x + t * n_in, d_w, d_a + t * n_out, n_in, n_out, s); + strata::kernels::bf16_gemv_fp32_mmvf_multi(d_x, d_w, d_b, n_in, n_out, n_tok, s); + check(cudaStreamSynchronize(s), "mmvf"); + const int b = bitwise_diff(down(d_a, (size_t) (n_tok * n_out)), down(d_b, (size_t) (n_tok * n_out)), "mmvf_multi"); + if (b) std::fprintf(stderr, "mmvf_multi: n_out %lld n_tok %d: %d differ\n", (long long) n_out, n_tok, b); + bad += b; + cudaFree(d_x); + cudaFree(d_a); + cudaFree(d_b); + } + cudaFree(d_w); + } + std::printf("mmvf_multi: %s\n", bad ? "MISMATCH" : "bitwise equal (n_out 512, 128, 1; 1-8 rows)"); + return bad; +} + +// ---- the top-10 router: one launch per token vs native_router_top10_multi +int test_router_multi(std::mt19937& rng, cudaStream_t s) { + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + std::vector logits((size_t) n_tok * 512); + std::normal_distribution nd(0.0f, 2.0f); + for (auto& v : logits) v = nd(rng); + for (int t = 0; t < n_tok; ++t) // exact ties, which the lower expert index must win + for (int i = 0; i < 6; ++i) logits[(size_t) t * 512 + rng() % 512] = logits[(size_t) t * 512 + 7]; + float *d_l = dev(logits.size()), *d_wa = dev((size_t) n_tok * 10), + *d_wb = dev((size_t) n_tok * 10); + int32_t *d_ia = dev((size_t) n_tok * 10), *d_ib = dev((size_t) n_tok * 10); + up(d_l, logits); + for (int t = 0; t < n_tok; ++t) + strata::kernels::native_router_top10(d_l + t * 512, d_ia + t * 10, d_wa + t * 10, s); + strata::kernels::native_router_top10_multi(d_l, d_ib, d_wb, n_tok, s); + check(cudaStreamSynchronize(s), "router"); + int b = bitwise_diff(down(d_wa, (size_t) n_tok * 10), down(d_wb, (size_t) n_tok * 10), "router weights"); + if (down(d_ia, (size_t) n_tok * 10) != down(d_ib, (size_t) n_tok * 10)) { + std::fprintf(stderr, "router_multi: n_tok %d: the ids differ\n", n_tok); + ++b; + } + bad += b; + for (void* p : {(void*) d_l, (void*) d_wa, (void*) d_wb, (void*) d_ia, (void*) d_ib}) cudaFree(p); + } + std::printf("router_multi: %s\n", bad ? "MISMATCH" : "bitwise equal (1-8 tokens, with ties)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -145,6 +213,8 @@ int main(int argc, char** argv) { check(cudaStreamCreate(&s), "stream"); int bad = 0; bad += test_gather_combine(rng, s); + bad += test_mmvf_multi(rng, s); + bad += test_router_multi(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; From 9ef275e6bf83b2afb41bd2111bf8c2732df6741e Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 22:50:46 +0200 Subject: [PATCH 15/92] Faster multi-token hyper-connection read: norm per (token, stream), prefetched and double-buffered down tiles, two up rows per warp (bitwise the single-token read) --- src/kernels/cuda/fused_gr.cu | 235 +++++++++++++++++++++------------- src/kernels/verify_parity.cpp | 81 ++++++++++++ 2 files changed, 226 insertions(+), 90 deletions(-) diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu index c439ee48ca3..62cb50d370b 100644 --- a/src/kernels/cuda/fused_gr.cu +++ b/src/kernels/cuda/fused_gr.cu @@ -137,87 +137,123 @@ struct GrMulti { int T; }; -// Step 1 of `gr_down_kernel`, one block per token, same threads and reduction order: rs[t] and xn[t] to global. +// Step 1 of `gr_down_kernel`, one block per (token, stream): each thread visits its elements of that stream in the +// order `gr_down_kernel`'s thread does, and the stream's warp partials are summed in the same order. rs and xn to +// global. __global__ void __launch_bounds__(THREADS) gr_norm_multi_kernel(GrMulti m) { - __shared__ float part[WARPS][HC]; - __shared__ float s_rs[HC]; - const FusedGrArgs& a = m.a[blockIdx.x]; - float* xn = m.xn + (size_t) blockIdx.x * D; + __shared__ float part[WARPS]; + const int tok = blockIdx.x, c = blockIdx.y; + const FusedGrArgs& a = m.a[tok]; + float* xn = m.xn + (size_t) tok * D; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - float gw[HC]; + const float gw = a.apply ? 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC) : 0.0f; + constexpr int STEPS = D / (THREADS * 4); // 10 + float4 p[STEPS]; + float ss = 0.0f; #pragma unroll - for (int c = 0; c < HC; ++c) gw[c] = a.apply ? 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC) : 0.0f; - float ss[HC] = {0.0f, 0.0f, 0.0f, 0.0f}; - for (int i = t * 4; i < D; i += THREADS * 4) { - const int c = i / N, d = i - c * N; + for (int j = 0; j < STEPS; ++j) { + const int i = t * 4 + j * THREADS * 4; + if (i / N != c) continue; float4 r = *reinterpret_cast(a.R + i); if (a.apply) { - const float4 b = *reinterpret_cast(a.bo_prev + d); - r.x = fmaf(b.x, gw[c], r.x); r.y = fmaf(b.y, gw[c], r.y); - r.z = fmaf(b.z, gw[c], r.z); r.w = fmaf(b.w, gw[c], r.w); + const float4 bo = *reinterpret_cast(a.bo_prev + (i - c * N)); + r.x = fmaf(bo.x, gw, r.x); r.y = fmaf(bo.y, gw, r.y); + r.z = fmaf(bo.z, gw, r.z); r.w = fmaf(bo.w, gw, r.w); } const float4 g = *reinterpret_cast(a.w_norm + i); - float sq = r.x * r.x + r.y * r.y + r.z * r.z + r.w * r.w; -#pragma unroll - for (int cc = 0; cc < HC; ++cc) if (cc == c) ss[cc] += sq; - *reinterpret_cast(xn + i) = make_float4(r.x * g.x, r.y * g.y, r.z * g.z, r.w * g.w); - } -#pragma unroll - for (int c = 0; c < HC; ++c) { - const float v = warp_sum(ss[c]); - if (lane == 0) part[warp][c] = v; + const float sq = r.x * r.x + r.y * r.y + r.z * r.z + r.w * r.w; + ss += sq; + p[j] = make_float4(r.x * g.x, r.y * g.y, r.z * g.z, r.w * g.w); } + ss = warp_sum(ss); + if (lane == 0) part[warp] = ss; __syncthreads(); - if (t < HC) { - float s = 0.0f; - for (int w = 0; w < WARPS; ++w) s += part[w][t]; - s_rs[t] = rsqrtf(s / (float) N + a.eps); - a.rs[t] = s_rs[t]; + float sum = 0.0f; + for (int w = 0; w < WARPS; ++w) sum += part[w]; + const float rs = rsqrtf(sum / (float) N + a.eps); + if (t == 0) a.rs[c] = rs; +#pragma unroll + for (int j = 0; j < STEPS; ++j) { + const int i = t * 4 + j * THREADS * 4; + if (i / N != c) continue; + *reinterpret_cast(xn + i) = make_float4(p[j].x * rs, p[j].y * rs, p[j].z * rs, p[j].w * rs); } - __syncthreads(); - for (int i = t; i < D; i += THREADS) xn[i] *= s_rs[i / N]; } -constexpr int TILE = 2560; // xn floats per token staged at a time: 320 chunks of 8, 10 per lane -constexpr int TQ = TILE / 8 / 32; // uint4 weight chunks per lane per tile +// 8 bf16 packed in a uint4 against 8 floats held as two float4 (the halves of a staged chunk): `dot8`'s order. +__device__ __forceinline__ float dot8h(const uint4 w, const float4 x0, const float4 x1) { + float acc = 0.0f; + acc = fmaf(__uint_as_float(w.x << 16), x0.x, acc); + acc = fmaf(__uint_as_float(w.x & 0xffff0000u), x0.y, acc); + acc = fmaf(__uint_as_float(w.y << 16), x0.z, acc); + acc = fmaf(__uint_as_float(w.y & 0xffff0000u), x0.w, acc); + acc = fmaf(__uint_as_float(w.z << 16), x1.x, acc); + acc = fmaf(__uint_as_float(w.z & 0xffff0000u), x1.y, acc); + acc = fmaf(__uint_as_float(w.w << 16), x1.z, acc); + acc = fmaf(__uint_as_float(w.w & 0xffff0000u), x1.w, acc); + return acc; +} -// Step 2 of `gr_down_kernel` for T tokens. One warp per row (so each lane accumulates the same chunks in the -// same order as the single-token kernel); per tile the lane's 10 weight chunks are loaded BEFORE the activation -// tile is staged, so the DRAM and L2 traffic are in flight together. +// Step 2 of `gr_down_kernel` for T tokens. One warp per row, so each lane accumulates the same chunks in the same +// order as the single-token kernel. The xn tiles of every token are staged split into the first and second halves +// of the 8-float chunks (conflict-free 16-byte reads) and double-buffered; each tile's weights are loaded one tile +// ahead; blocks start their staging at different offsets, which spreads the reads of the same lines over time. +// TC chunks per tile: 320 for up to 4 tokens, 160 above (the two buffers must fit shared memory). +template __global__ void __launch_bounds__(THREADS) gr_down_multi_kernel(GrMulti m) { - extern __shared__ __align__(16) float tile[]; // [T][TILE] + extern __shared__ __align__(16) float4 buf[]; // [2][T][2][TC] + constexpr int TQ = TC / 32, NT = D / 8 / TC; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; const int T = m.T; const bool inject_block = blockIdx.x == DOWN_BLOCKS; const int row = inject_block ? warp : blockIdx.x * WARPS + warp; const bool active = !(inject_block && (m.a[0].w_inject == nullptr || warp >= HC)); - const uint16_t* wrow = (inject_block ? m.a[0].w_inject : m.a[0].w_down) + (size_t) (active ? row : 0) * D; - const uint4* w4 = reinterpret_cast(wrow); + const uint4* w4 = reinterpret_cast((inject_block ? m.a[0].w_inject : m.a[0].w_down) + + (size_t) (active ? row : 0) * D); float acc[kFusedGrMaxT]; #pragma unroll for (int k = 0; k < kFusedGrMaxT; ++k) acc[k] = 0.0f; - for (int base = 0; base < D; base += TILE) { - uint4 wv[TQ]; - if (active) { -#pragma unroll - for (int q = 0; q < TQ; ++q) wv[q] = __ldg(w4 + base / 8 + lane + 32 * q); + const int per_buf = T * 2 * TC, cnt = T * 2 * TC; + const int rot = (int) ((blockIdx.x * 331u) % (unsigned) cnt); + const float4* src4 = reinterpret_cast(m.xn); + auto stage = [&](int tile, float4* dst) { + for (int ii = t; ii < cnt; ii += THREADS) { + int i = ii + rot; + if (i >= cnt) i -= cnt; + const int k = i / (2 * TC), rem = i - k * 2 * TC, j = rem >> 1, h = rem & 1; + dst[(k * 2 + h) * TC + j] = src4[((size_t) k * D + (size_t) tile * TC * 8) / 4 + j * 2 + h]; } - __syncthreads(); // the previous tile is consumed - const float4* src4 = reinterpret_cast(m.xn); - float4* tile4 = reinterpret_cast(tile); - for (int i = t; i < T * (TILE / 4); i += THREADS) { - const int k = i / (TILE / 4), off = i - k * (TILE / 4); - tile4[i] = src4[((size_t) k * D + base) / 4 + off]; + }; + uint4 wv[TQ], wn[TQ]; + if (active) { +#pragma unroll + for (int q = 0; q < TQ; ++q) wn[q] = __ldg(w4 + lane + 32 * q); + } + stage(0, buf); + __syncthreads(); + for (int tile = 0; tile < NT; ++tile) { + const float4* xb = buf + (size_t) (tile & 1) * per_buf; +#pragma unroll + for (int q = 0; q < TQ; ++q) wv[q] = wn[q]; + if (tile + 1 < NT) { + if (active) { +#pragma unroll + for (int q = 0; q < TQ; ++q) wn[q] = __ldg(w4 + (tile + 1) * TC + lane + 32 * q); + } + stage(tile + 1, buf + (size_t) ((tile + 1) & 1) * per_buf); } - __syncthreads(); - if (!active) continue; + if (active) { #pragma unroll - for (int q = 0; q < TQ; ++q) { - const int j = lane + 32 * q; + for (int q = 0; q < TQ; ++q) { + const int j = lane + 32 * q; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) - if (k < T) acc[k] += dot8(wv[q], tile + k * TILE + j * 8); + for (int k = 0; k < kFusedGrMaxT; ++k) { + if (k >= T) break; + acc[k] += dot8h(wv[q], xb[(k * 2) * TC + j], xb[(k * 2 + 1) * TC + j]); + } + } } + __syncthreads(); } if (!active) return; float s[kFusedGrMaxT]; @@ -238,49 +274,64 @@ __global__ void __launch_bounds__(THREADS) gr_down_multi_kernel(GrMulti m) { constexpr int UPM_COLS = 16; // columns per block (x 4 streams = 64 rows, 8 per warp) constexpr int UPM_BLOCKS = N / UPM_COLS; // 160 +constexpr int UPM_RPW = 2; // rows a warp has in flight at once -// `gr_up_kernel` for T tokens: each row of w_up read once; the T dots reduced by xor so every lane holds every -// sum, and lane k runs token k's epilogue - the T epilogues in parallel instead of one after another. +// `gr_up_kernel` for T tokens: each row of w_up read once, two rows per warp in flight; the lo vectors staged split +// into chunk halves; the T dots of a row reduced by xor so every lane holds every sum, and lane k runs token k's +// epilogue - the T epilogues in parallel instead of one after another. __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { - __shared__ __align__(16) float lo[kFusedGrMaxT][LR]; + __shared__ __align__(16) float4 lo4[kFusedGrMaxT][2][LR / 8]; __shared__ float g[kFusedGrMaxT][HC][UPM_COLS]; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; const int T = m.T; const int d0 = blockIdx.x * UPM_COLS; - for (int i = t; i < T * LR; i += THREADS) lo[i / LR][i % LR] = m.a[i / LR].lo[i % LR]; + for (int i = t; i < T * LR / 4; i += THREADS) { + const int k = i / (LR / 4), f = i - k * (LR / 4); + lo4[k][f & 1][f >> 1] = reinterpret_cast(m.a[k].lo)[f]; + } __syncthreads(); - for (int r = warp; r < HC * UPM_COLS; r += WARPS) { - const int c = r / UPM_COLS, dd = r - c * UPM_COLS, i = c * N + d0 + dd; - const uint4* w4 = reinterpret_cast(m.a[0].w_up + (size_t) i * LR); - const uint4 wa = __ldg(w4 + lane); - const uint4 wb = lane < LR / 8 - 32 ? __ldg(w4 + 32 + lane) : make_uint4(0, 0, 0, 0); + for (int r0 = warp * UPM_RPW; r0 < HC * UPM_COLS; r0 += WARPS * UPM_RPW) { + uint4 wa[UPM_RPW], wb[UPM_RPW]; // the epilogue inputs of this lane's token, fetched while the dots run - float rv = 0.0f, wn = 0.0f, rsc = 0.0f, bo = 0.0f, ip = 0.0f; + float rv[UPM_RPW], wn[UPM_RPW], rsc[UPM_RPW], bo[UPM_RPW], ip[UPM_RPW]; bool apply = false; - if (lane < T) { - const FusedGrArgs& a = m.a[lane]; - rv = a.R[i]; - wn = a.w_norm[i]; - rsc = a.rs[c]; - apply = a.apply; - if (apply) { bo = a.bo_prev[d0 + dd]; ip = a.inj_prev[c]; } - } - float mine = 0.0f; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) { - if (k >= T) break; - float acc = dot8(wa, lo[k] + lane * 8); - if (lane < LR / 8 - 32) acc += dot8(wb, lo[k] + (32 + lane) * 8); - acc = warp_sum(acc); - if (lane == k) mine = acc; + for (int q = 0; q < UPM_RPW; ++q) { + const int r = r0 + q, c = r / UPM_COLS, dd = r - c * UPM_COLS, i = c * N + d0 + dd; + const uint4* w4 = reinterpret_cast(m.a[0].w_up + (size_t) i * LR); + wa[q] = __ldg(w4 + lane); + wb[q] = lane < LR / 8 - 32 ? __ldg(w4 + 32 + lane) : make_uint4(0, 0, 0, 0); + rv[q] = wn[q] = rsc[q] = bo[q] = ip[q] = 0.0f; + if (lane < T) { + const FusedGrArgs& a = m.a[lane]; + rv[q] = a.R[i]; + wn[q] = a.w_norm[i]; + rsc[q] = a.rs[c]; + apply = a.apply; + if (apply) { bo[q] = a.bo_prev[d0 + dd]; ip[q] = a.inj_prev[c]; } + } } - if (lane < T) { - if (apply) { - rv = fmaf(bo, 2.0f * sigmoidf_(ip / (float) HC), rv); - m.a[lane].R_out[i] = rv; +#pragma unroll + for (int q = 0; q < UPM_RPW; ++q) { + const int r = r0 + q, c = r / UPM_COLS, dd = r - c * UPM_COLS, i = c * N + d0 + dd; + float mine = 0.0f; +#pragma unroll + for (int k = 0; k < kFusedGrMaxT; ++k) { + if (k >= T) break; + float acc = dot8h(wa[q], lo4[k][0][lane], lo4[k][1][lane]); + if (lane < LR / 8 - 32) acc += dot8h(wb[q], lo4[k][0][32 + lane], lo4[k][1][32 + lane]); + acc = warp_sum(acc); + if (lane == k) mine = acc; + } + if (lane < T) { + float x = rv[q]; + if (apply) { + x = fmaf(bo[q], 2.0f * sigmoidf_(ip[q] / (float) HC), x); + m.a[lane].R_out[i] = x; + } + const float y = x * wn[q] * rsc[q]; + g[lane][c][dd] = y * sigmoidf_(mine); } - const float x = rv * wn * rsc; - g[lane][c][dd] = x * sigmoidf_(mine); } } __syncthreads(); @@ -314,14 +365,18 @@ void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* xn_scratch, voi m.xn = xn_scratch; m.T = n_tok; cudaStream_t st = (cudaStream_t) stream; - gr_norm_multi_kernel<<>>(m); + gr_norm_multi_kernel<<>>(m); static bool attr = false; - if (!attr) { - cudaFuncSetAttribute(gr_down_multi_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, - (int) (kFusedGrMaxT * TILE * sizeof(float))); + if (!attr) { // two buffers of the largest tile: 4 tokens x 320 chunks, or 8 x 160 + const int bytes = 2 * 4 * 2 * 320 * (int) sizeof(float4); + cudaFuncSetAttribute(gr_down_multi_kernel<320>, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes); + cudaFuncSetAttribute(gr_down_multi_kernel<160>, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes); attr = true; } - gr_down_multi_kernel<<>>(m); + if (n_tok <= 4) + gr_down_multi_kernel<320><<>>(m); + else + gr_down_multi_kernel<160><<>>(m); gr_up_multi_kernel<<>>(m); const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 1c6b5b838db..d45bf55b16e 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -5,6 +5,7 @@ // the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" +#include "strata/kernels/fused_gr.hpp" #include "strata/kernels/native_moe.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" @@ -199,6 +200,85 @@ int test_router_multi(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the hyper-connection read: fused_gr_read per token vs fused_gr_read_multi +int test_gr_read(std::mt19937& rng, cudaStream_t s) { + const int64_t N = 2560, HC = 4, D = N * HC, LR = 320; + auto bf16 = [&](size_t n) { + std::vector v(n); + for (auto& x : v) { + const float f = edgy(rng) * 0.05f; + uint32_t b; + std::memcpy(&b, &f, 4); + x = (uint16_t) (b >> 16); + } + return v; + }; + std::vector wn((size_t) D); + for (auto& v : wn) v = edgy(rng); + uint16_t *d_wd = dev((size_t) (LR * D)), *d_wu = dev((size_t) (D * LR)), + *d_wi = dev((size_t) (HC * D)); + float* d_wn = dev(wn.size()); + up(d_wd, bf16((size_t) (LR * D))); + up(d_wu, bf16((size_t) (D * LR))); + up(d_wi, bf16((size_t) (HC * D))); + up(d_wn, wn); + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + for (int variant = 0; variant < 3; ++variant) { // apply + inject, no apply, no inject + const bool apply = variant != 1, inject = variant != 2; + std::vector R((size_t) (n_tok * D)), bo((size_t) (n_tok * N)), inj((size_t) (n_tok * HC)); + for (auto& v : R) v = edgy(rng); + for (auto& v : bo) v = edgy(rng); + for (auto& v : inj) v = edgy(rng); + // one set of outputs per path; R is updated in place when apply, so each path has its own copy + float* d[2][6]; // R, lo, rs, inject, mixed, xn + for (int p = 0; p < 2; ++p) { + d[p][0] = dev(R.size()); + up(d[p][0], R); + d[p][1] = dev((size_t) (n_tok * LR)); + d[p][2] = dev((size_t) (n_tok * HC)); + d[p][3] = dev((size_t) (n_tok * HC)); + d[p][4] = dev((size_t) (n_tok * N)); + d[p][5] = dev((size_t) (n_tok * D)); + check(cudaMemset(d[p][3], 0, (size_t) (n_tok * HC) * 4), "memset"); + } + float *d_bo = dev(bo.size()), *d_inj = dev(inj.size()); + up(d_bo, bo); + up(d_inj, inj); + std::vector args[2]; + for (int p = 0; p < 2; ++p) + for (int t = 0; t < n_tok; ++t) { + strata::kernels::FusedGrArgs a; + a.R = d[p][0] + t * D; a.R_out = d[p][0] + t * D; a.apply = apply; + a.bo_prev = d_bo + t * N; a.inj_prev = d_inj + t * HC; + a.w_norm = d_wn; a.w_down = d_wd; a.w_up = d_wu; a.w_inject = inject ? d_wi : nullptr; + a.eps = 1e-6f; + a.lo = d[p][1] + t * LR; a.rs = d[p][2] + t * HC; a.inject_out = d[p][3] + t * HC; + a.mixed = d[p][4] + t * N; + args[p].push_back(a); + } + for (int t = 0; t < n_tok; ++t) strata::kernels::fused_gr_read(args[0][(size_t) t], s); + strata::kernels::fused_gr_read_multi(args[1].data(), n_tok, d[1][5], s); + check(cudaStreamSynchronize(s), "gr read"); + const char* names[5] = {"R", "lo", "rs", "inject", "mixed"}; + const size_t sizes[5] = {R.size(), (size_t) (n_tok * LR), (size_t) (n_tok * HC), (size_t) (n_tok * HC), + (size_t) (n_tok * N)}; + for (int o = 0; o < 5; ++o) { + const int b = bitwise_diff(down(d[0][o], sizes[o]), down(d[1][o], sizes[o]), names[o]); + if (b) std::fprintf(stderr, "gr_read: n_tok %d variant %d: %s: %d differ\n", n_tok, variant, names[o], b); + bad += b; + } + for (int p = 0; p < 2; ++p) + for (float* q : d[p]) cudaFree(q); + cudaFree(d_bo); + cudaFree(d_inj); + } + } + for (void* p : {(void*) d_wd, (void*) d_wu, (void*) d_wi, (void*) d_wn}) cudaFree(p); + std::printf("gr_read: %s\n", bad ? "MISMATCH" : "bitwise equal to the single-token read (1-8 tokens, 3 variants)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -215,6 +295,7 @@ int main(int argc, char** argv) { bad += test_gather_combine(rng, s); bad += test_mmvf_multi(rng, s); bad += test_router_multi(rng, s); + bad += test_gr_read(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; From 939add21bb033953052cfaeeef3806aac9ffa6ae Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 23:32:43 +0200 Subject: [PATCH 16/92] Verify window: the QSA layer's per-token kernels batched; --window-hashes RoPE (native_rope_apply_tokens), the int8 K/V append (kv_append_q8_steps), the indexer append (native_qsa_indexer_append_multi), the q/gate split, the q/k norms and the gate now run once per window instead of once per token. Each token's arithmetic is unchanged: verify_parity checks the new kernels bitwise against the per-token ones, and --window-hashes writes a hash of each window's final residual rows so two builds can be compared end to end. --- include/strata/kernels/kv_q8.hpp | 5 + include/strata/kernels/native_qsa_indexer.hpp | 6 + include/strata/kernels/native_rope.hpp | 5 + src/core/verify.cpp | 67 +++---- src/kernels/cuda/kv_q8.cu | 21 ++- src/kernels/cuda/native_qsa_indexer.cu | 71 +++++--- src/kernels/cuda/native_rope.cu | 20 +- src/kernels/verify_parity.cpp | 171 ++++++++++++++++++ src/program/generate.cpp | 23 +++ 9 files changed, 324 insertions(+), 65 deletions(-) diff --git a/include/strata/kernels/kv_q8.hpp b/include/strata/kernels/kv_q8.hpp index f6d5045e634..e94d23abe9f 100644 --- a/include/strata/kernels/kv_q8.hpp +++ b/include/strata/kernels/kv_q8.hpp @@ -28,6 +28,11 @@ inline uint64_t kv_q8_bytes_per_cell(const QsaShapes& s) { /// Append the cell at step[kStepPos] (graph-capturable: position and page come from device memory). void kv_append_q8_step(int8_t* k_q, int8_t* v_q, uint16_t* k_scale, uint16_t* v_scale, const int32_t* page_table, const int32_t* step, const float* kcur, const float* vcur, const QsaShapes& s, void* stream); +/// `kv_append_q8_step` for n_tok cells in one launch: token t's step record at step + t * step_stride, its K and V at +/// kcur/vcur + t * n_head_kv * head_dim. Bitwise the per-token appends. +void kv_append_q8_steps(int8_t* k_q, int8_t* v_q, uint16_t* k_scale, uint16_t* v_scale, const int32_t* page_table, + const int32_t* step, int step_stride, const float* kcur, const float* vcur, int n_tok, + const QsaShapes& s, void* stream); /// Gather step[kStepWidth] cells named by `ids` into FP16 scratch `[id][kv_head][head_dim]`; the grid is sized by /// `max_ids` (capacity), the kernel reads the real count from `step`. diff --git a/include/strata/kernels/native_qsa_indexer.hpp b/include/strata/kernels/native_qsa_indexer.hpp index a79fd47cef7..d51f57a560d 100644 --- a/include/strata/kernels/native_qsa_indexer.hpp +++ b/include/strata/kernels/native_qsa_indexer.hpp @@ -31,5 +31,11 @@ void native_qsa_indexer_append(const float* raw, const int32_t* relative_pos_dev int32_t pos_base, const float* gamma, float epsilon, const QsaIndexerBuffers& buffers, const QsaShapes& shapes, int64_t max_cells, float freq_base, void* stream); +// n_tok appends in one launch, in order: token t's raw key at raw + t * 128, its position at +// relative_pos_device[t * pos_stride]. Bitwise n_tok calls of native_qsa_indexer_append. +void native_qsa_indexer_append_multi(const float* raw, const int32_t* relative_pos_device, int pos_stride, + int n_tok, int32_t pos_base, const float* gamma, float epsilon, + const QsaIndexerBuffers& buffers, const QsaShapes& shapes, + int64_t max_cells, float freq_base, void* stream); } // namespace strata::kernels diff --git a/include/strata/kernels/native_rope.hpp b/include/strata/kernels/native_rope.hpp index 24bffe61e67..c4ab78c6605 100644 --- a/include/strata/kernels/native_rope.hpp +++ b/include/strata/kernels/native_rope.hpp @@ -11,4 +11,9 @@ bool native_rope_enabled(); // Explicit stream required. No allocation or synchronization. void native_rope_apply(const float* x, float* out, int rows, int head_dim, int n_rot, float freq_base, const int* positions, void* stream); +// The same for several tokens' heads in one launch: row r is head r % heads of token r / heads, and takes +// positions[(r / heads) * pos_stride + r % heads] (a verify window's per-token position vectors). Each row is +// bitwise what native_rope_apply gives it. +void native_rope_apply_tokens(const float* x, float* out, int rows, int head_dim, int n_rot, float freq_base, + const int* positions, int heads, int pos_stride, void* stream); } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index c86db750535..bf312d8a63e 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -424,40 +424,45 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, cs); else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st.cos_tab, st.sin_tab, pos, cs); }; + // the same for the n tokens' heads in one launch each: token t's positions at pos + t * NH + const bool batched = native_qsa_enabled() && native_rope_enabled(); + auto norm_rope_tokens = [&](float* data, const WeightRef* norm, int heads, int cols, const int32_t* pos) { + native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, n * heads, EPS, cs); + native_rope_apply_tokens(data, data, n * heads, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, heads, + (int) NH, cs); + }; float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; native_quantize_q8_1(xm, xq_, (int) N, n, cs); bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, cs); native_mmvq(wk->native_type, wk->native_data, xq_, kcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); native_mmvq(wv->native_type, wv->native_data, xq_, vcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); - for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH); + if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH); + else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH); if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, cs); - for (int t = tb; t < te; ++t) { - const int32_t* step_t = step_ + t * kStepCount; - if (st.kv_int8) - kv_append_q8_step(st.k_q, st.v_q, st.k_scale, st.v_scale, st.page_table, step_t, - kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); - else - kv_append_step(st.k_pool, st.v_pool, st.page_table, step_t, kcur_ + t * NKV * HD, + if (st.kv_int8) + kv_append_q8_steps(st.k_q, st.v_q, st.k_scale, st.v_scale, st.page_table, step_ + tb * kStepCount, + (int) kStepCount, kcur_ + tb * NKV * HD, vcur_ + tb * NKV * HD, n, s, cs); + else + for (int t = tb; t < te; ++t) + kv_append_step(st.k_pool, st.v_pool, st.page_table, step_ + t * kStepCount, kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); - } const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; - for (int t = tb; t < te; ++t) - native_qsa_indexer_append(idx_raw + t * ID, step_ + t * kStepCount + kStepPos, 0, - (const float*) wikn->data, EPS, ib, s, st.max_cells, - (float) qsa_freq_base(), cs); + native_qsa_indexer_append_multi(idx_raw + tb * ID, step_ + tb * kStepCount + kStepPos, (int) kStepCount, n, + 0, (const float*) wikn->data, EPS, ib, s, st.max_cells, + (float) qsa_freq_base(), cs); native_mmvq(wq->native_type, wq->native_data, xq_, qfull_ + tb * NH * 2 * HD, (int) N, (int) (NH * 2 * HD), n, cs); - for (int t = tb; t < te; ++t) { - float* qc = qcur_ + t * NH * HD; - if (cudaMemcpy2DAsync(qc, (size_t) HD * 4, qfull_ + t * NH * 2 * HD, (size_t) HD * 2 * 4, - (size_t) HD * 4, (size_t) NH, cudaMemcpyDeviceToDevice, cs) != cudaSuccess) { - err = "verify: the q/gate split failed"; - return false; - } - norm_rope(qc, wqn, (int) NH, (int) HD, pos_ + t * NH); + // the query halves of every token's heads (rows of 2 * HD: query, then gate) + if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, + (size_t) HD * 4, (size_t) (n * NH), cudaMemcpyDeviceToDevice, cs) != cudaSuccess) { + err = "verify: the q/gate split failed"; + return false; } + if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH); + else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH); bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, N, IQ * ID, n, cs); - for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH); + if (batched) norm_rope_tokens(qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + tb * NH); + else for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH); qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, s, scores_ + (size_t) tb * max_blocks_, cs); qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, @@ -468,13 +473,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } qsa_decode_attn_batch(qcur_ + tb * NH * HD, pools, sel_ + (size_t) tb * cap_, step_ + tb * kStepCount, cap_, s, attn_scratch_ + (size_t) tb * attn_scratch_floats_, attn_ + tb * NH * HD, n, cs); - for (int t = tb; t < te; ++t) { - if (native_qsa_enabled()) - native_qsa_gate_apply(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, attn32_ + t * NH * HD, - (int) NH, (int) HD, cs); - else + if (native_qsa_enabled()) + native_qsa_gate_apply(attn_ + tb * NH * HD, qfull_ + tb * NH * 2 * HD, attn32_ + tb * NH * HD, + (int) (n * NH), (int) HD, cs); + else + for (int t = tb; t < te; ++t) qsa_gate_apply_f32(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, s, attn32_ + t * NH * HD, cs); - } native_quantize_q8_1(attn32_ + tb * NH * HD, xq_, (int) (NH * HD), n, cs); native_mmvq(wo->native_type, wo->native_data, xq_, bo_ + tb * N, (int) (NH * HD), (int) N, n, cs); } @@ -727,10 +731,9 @@ bool Verifier::capture_commit(std::string& err) { if (!wikn) { ok = false; break; } copy_from_mapped(st.idx_tail, tail_snap_ + (size_t) qsa_index * TS, TS, cs_); const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; - for (int64_t t = 0; t < MT; ++t) - native_qsa_indexer_append(idx_raw_L_ + (size_t) (qsa_index * MT + t) * ID, commit_ + 2 + t, 0, - (const float*) wikn->data, EPS, ib, s, st.max_cells, - (float) qsa_freq_base(), cs_); + native_qsa_indexer_append_multi(idx_raw_L_ + (size_t) (qsa_index * MT) * ID, commit_ + 2, 1, (int) MT, 0, + (const float*) wikn->data, EPS, ib, s, st.max_cells, + (float) qsa_freq_base(), cs_); ++qsa_index; } } diff --git a/src/kernels/cuda/kv_q8.cu b/src/kernels/cuda/kv_q8.cu index 09f65ecf34a..78e9eed86b7 100644 --- a/src/kernels/cuda/kv_q8.cu +++ b/src/kernels/cuda/kv_q8.cu @@ -31,10 +31,15 @@ __global__ void kv_append_q8_kernel(int8_t* __restrict__ k_q, int8_t* __restrict uint16_t* __restrict__ k_scale, uint16_t* __restrict__ v_scale, const int32_t* __restrict__ table, const int32_t* __restrict__ step, const float* __restrict__ kcur, const float* __restrict__ vcur, int kv_heads, - int head_dim, int page_size) { + int head_dim, int page_size, int step_stride) { + // blockIdx.z = 2 * token + (0 K, 1 V) + const int tok = blockIdx.z >> 1; + step += (size_t) tok * step_stride; + kcur += (size_t) tok * kv_heads * head_dim; + vcur += (size_t) tok * kv_heads * head_dim; const long long pos = (long long) __ldg(step + kStepPos); const int h = blockIdx.x, g = blockIdx.y, t = threadIdx.x; - const bool is_v = blockIdx.z == 1; + const bool is_v = (blockIdx.z & 1) == 1; const int groups = head_dim / KV_Q8_GROUP; const float x = (is_v ? vcur : kcur)[h * head_dim + g * KV_Q8_GROUP + t]; // max |x| over the 64 values: two warps, then combine through shared memory in a fixed order @@ -95,10 +100,18 @@ __global__ void kv_gather_q8_kernel(const int8_t* __restrict__ k_q, const int8_t void kv_append_q8_step(int8_t* k_q, int8_t* v_q, uint16_t* k_scale, uint16_t* v_scale, const int32_t* page_table, const int32_t* step, const float* kcur, const float* vcur, const QsaShapes& s, void* stream) { validate(s, "kv_append_q8"); - const dim3 grid((unsigned) s.n_head_kv, (unsigned) (s.head_dim / KV_Q8_GROUP), 2); + kv_append_q8_steps(k_q, v_q, k_scale, v_scale, page_table, step, 0, kcur, vcur, 1, s, stream); +} + +void kv_append_q8_steps(int8_t* k_q, int8_t* v_q, uint16_t* k_scale, uint16_t* v_scale, const int32_t* page_table, + const int32_t* step, int step_stride, const float* kcur, const float* vcur, int n_tok, + const QsaShapes& s, void* stream) { + validate(s, "kv_append_q8"); + if (n_tok < 1) return; + const dim3 grid((unsigned) s.n_head_kv, (unsigned) (s.head_dim / KV_Q8_GROUP), (unsigned) (2 * n_tok)); kv_append_q8_kernel<<>>( k_q, v_q, k_scale, v_scale, page_table, step, kcur, vcur, (int) s.n_head_kv, (int) s.head_dim, - (int) s.page_size); + (int) s.page_size, step_stride); check("kv_append_q8 launch"); } diff --git a/src/kernels/cuda/native_qsa_indexer.cu b/src/kernels/cuda/native_qsa_indexer.cu index b80009b52b6..3a2d51cd0c7 100644 --- a/src/kernels/cuda/native_qsa_indexer.cu +++ b/src/kernels/cuda/native_qsa_indexer.cu @@ -39,12 +39,14 @@ __device__ float warp_sum(float x) { x += __shfl_xor_sync(0xffffffffu, x, offset); return x; } -__global__ void append(const float* __restrict__ raw, const int32_t* __restrict__ pos_dev, - int pos_base, const float* __restrict__ gamma, float epsilon, - float* __restrict__ tail, float* __restrict__ dead, - float* __restrict__ pooled, int32_t* __restrict__ block_pos, - int max_cells, float theta_scale, const int32_t* __restrict__ mtab) { - const int pos = *pos_dev, d = threadIdx.x; +// One token's append; every thread of the block calls it (it synchronizes the block), a later call sees this one's +// tail writes. +__device__ void append_one(const float* __restrict__ raw, const int pos, + int pos_base, const float* __restrict__ gamma, float epsilon, + float* __restrict__ tail, float* __restrict__ dead, + float* __restrict__ pooled, int32_t* __restrict__ block_pos, + int max_cells, float theta_scale, const int32_t* __restrict__ mtab) { + const int d = threadIdx.x; if (pos < 0 || pos >= max_cells) return; const int slot = pos % R; float incoming = 0.0f; @@ -77,21 +79,34 @@ __global__ void append(const float* __restrict__ raw, const int32_t* __restrict_ const float scale = rsqrtf(square_sum / D + epsilon); if (d < D) values[d] = scale * mean * gamma[d]; __syncthreads(); - if (d >= D) return; - const int b = pos / R; - const int rope_pos = pos == 0 ? 0 : pos_base + R * b; - float y = values[d]; - if (d < ROT) { - const int pair = d % (ROT / 2); - const float theta = (pos == 0 ? 0 : mrope_pos(mtab, rope_pos, pair)) * powf(theta_scale, float(pair)); - const float c = cosf(theta), s = sinf(theta); - const float a = values[pair], z = values[pair + ROT / 2]; - y = d < ROT / 2 ? a * c - z * s : a * s + z * c; + if (d < D) { + const int b = pos / R; + const int rope_pos = pos == 0 ? 0 : pos_base + R * b; + float y = values[d]; + if (d < ROT) { + const int pair = d % (ROT / 2); + const float theta = (pos == 0 ? 0 : mrope_pos(mtab, rope_pos, pair)) * powf(theta_scale, float(pair)); + const float c = cosf(theta), s = sinf(theta); + const float a = values[pair], z = values[pair + ROT / 2]; + y = d < ROT / 2 ? a * c - z * s : a * s + z * c; + } + pooled[std::size_t(b) * D + d] = y; + if (pos == 0) dead[d] = y; + else pooled[std::size_t(b + 1) * D + d] = dead[d]; + if (d == 0 && pos != 0) *block_pos = rope_pos; + } + __syncthreads(); // values and partials are reused by the next call +} +__global__ void append(const float* __restrict__ raw, const int32_t* __restrict__ pos_dev, int pos_stride, + int n_tok, int pos_base, const float* __restrict__ gamma, float epsilon, + float* __restrict__ tail, float* __restrict__ dead, + float* __restrict__ pooled, int32_t* __restrict__ block_pos, + int max_cells, float theta_scale, const int32_t* __restrict__ mtab) { + for (int t = 0; t < n_tok; ++t) { + append_one(raw + std::size_t(t) * D, pos_dev[std::size_t(t) * pos_stride], pos_base, gamma, epsilon, tail, + dead, pooled, block_pos, max_cells, theta_scale, mtab); + __syncthreads(); // this token's tail and pooled writes before the next token reads them } - pooled[std::size_t(b) * D + d] = y; - if (pos == 0) dead[d] = y; - else pooled[std::size_t(b + 1) * D + d] = dead[d]; - if (d == 0 && pos != 0) *block_pos = rope_pos; } struct Span { const void* p; std::size_t n; }; void validate(Span s) { @@ -110,19 +125,29 @@ bool native_qsa_indexer_enabled() { return enabled.load(std::memory_order_relaxe void native_qsa_indexer_append(const float* raw, const int32_t* relative_pos_device, int32_t pos_base, const float* gamma, float epsilon, const QsaIndexerBuffers& b, const QsaShapes& s, int64_t max_cells, float freq_base, void* stream) { + native_qsa_indexer_append_multi(raw, relative_pos_device, 0, 1, pos_base, gamma, epsilon, b, s, max_cells, + freq_base, stream); +} +void native_qsa_indexer_append_multi(const float* raw, const int32_t* relative_pos_device, int pos_stride, + int n_tok, int32_t pos_base, const float* gamma, float epsilon, + const QsaIndexerBuffers& b, const QsaShapes& s, int64_t max_cells, + float freq_base, void* stream) { + if (n_tok < 1 || pos_stride < 0 || (n_tok > 1 && pos_stride < 1)) + throw std::invalid_argument("native QSA indexer: n_tok >= 1 appends with a positive position stride"); if (!stream || s.idx_dim != D || s.idx_block != R || s.n_rot != ROT || max_cells < 1 || max_cells > INT32_MAX || pos_base < 0 || pos_base % R || int64_t(pos_base) + max_cells > INT32_MAX || !std::isfinite(epsilon) || epsilon <= 0.0f || !std::isfinite(freq_base) || freq_base <= 1.0f) throw std::invalid_argument("native QSA indexer requires fixed geometry, aligned position base, positive capacity/epsilon, valid frequency and explicit stream"); - const Span spans[] = {{raw,D*4},{relative_pos_device,4},{gamma,D*4},{b.tail,(R-1)*D*4}, + const Span spans[] = {{raw,std::size_t(n_tok)*D*4},{relative_pos_device,(std::size_t(n_tok-1)*pos_stride+1)*4}, + {gamma,D*4},{b.tail,(R-1)*D*4}, {b.dead,D*4},{b.pooled,std::size_t(max_cells/R+1)*D*4},{b.block_pos,4}}; for (const auto& span : spans) validate(span); for (int i = 0; i < 7; ++i) for (int j = i + 1; j < 7; ++j) if (overlaps(spans[i], spans[j])) throw std::invalid_argument("native QSA indexer buffers overlap"); const float theta_scale = powf(freq_base, -2.0f / ROT); - append<<<1,THREADS,0,static_cast(stream)>>>(raw,relative_pos_device,pos_base,gamma,epsilon, - b.tail,b.dead,b.pooled,b.block_pos,int(max_cells),theta_scale,mrope_table()); + append<<<1,THREADS,0,static_cast(stream)>>>(raw,relative_pos_device,pos_stride,n_tok,pos_base,gamma, + epsilon,b.tail,b.dead,b.pooled,b.block_pos,int(max_cells),theta_scale,mrope_table()); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/cuda/native_rope.cu b/src/kernels/cuda/native_rope.cu index d3e9225c208..51ac0a94f49 100644 --- a/src/kernels/cuda/native_rope.cu +++ b/src/kernels/cuda/native_rope.cu @@ -39,10 +39,12 @@ bool overlaps(const void* a, size_t an, const void* b, size_t bn) { return x <= y ? y - x < an : x - y < bn; } __global__ void apply(const float* x, float* out, int rows, int width, - int n_rot, float theta_scale, const int* positions, const int32_t* mtab) { + int n_rot, float theta_scale, const int* positions, const int32_t* mtab, int heads, + int pos_stride) { const int row = blockIdx.y; const int pair = blockIdx.x * blockDim.x + threadIdx.x; if (row >= rows || pair >= width / 2) return; + const int position = positions[(row / heads) * pos_stride + row % heads]; const size_t start = size_t(row) * width; if (pair >= n_rot / 2) { if (x != out) { @@ -51,7 +53,7 @@ __global__ void apply(const float* x, float* out, int rows, int width, } return; } - const float theta = mrope_pos(mtab, positions[row], pair) * powf(theta_scale, float(pair)); + const float theta = mrope_pos(mtab, position, pair) * powf(theta_scale, float(pair)); const float c = cosf(theta), s = sinf(theta); const float a = x[start + pair], b = x[start + pair + n_rot / 2]; out[start + pair] = a * c - b * s; @@ -65,7 +67,11 @@ void native_rope_set_enabled(bool value) { enabled.store(value, std::memory_orde bool native_rope_enabled() { return enabled.load(std::memory_order_relaxed); } void native_rope_apply(const float* x, float* out, int rows, int head_dim, int n_rot, float freq_base, const int* positions, void* stream) { - if (!x || !out || !positions || !stream || rows < 1 || rows > 65535 || + native_rope_apply_tokens(x, out, rows, head_dim, n_rot, freq_base, positions, rows, rows, stream); +} +void native_rope_apply_tokens(const float* x, float* out, int rows, int head_dim, int n_rot, float freq_base, + const int* positions, int heads, int pos_stride, void* stream) { + if (!x || !out || !positions || !stream || rows < 1 || rows > 65535 || heads < 1 || pos_stride < heads || (head_dim != 128 && head_dim != 256) || n_rot != 64 || !std::isfinite(freq_base) || freq_base <= 1.0f || reinterpret_cast(x) % 4 || reinterpret_cast(out) % 4 || @@ -73,15 +79,17 @@ void native_rope_apply(const float* x, float* out, int rows, int head_dim, throw std::invalid_argument("native RoPE requires aligned F32 rows, width 128/256, rotation 64, valid base and explicit stream"); } const size_t bytes = size_t(rows) * head_dim * sizeof(float); + const size_t pos_bytes = (size_t((rows - 1) / heads) * pos_stride + heads) * sizeof(int); if ((x != out && overlaps(x, bytes, out, bytes)) || - overlaps(positions, size_t(rows) * sizeof(int), out, bytes) || - overlaps(positions, size_t(rows) * sizeof(int), x, bytes)) { + overlaps(positions, pos_bytes, out, bytes) || + overlaps(positions, pos_bytes, x, bytes)) { throw std::invalid_argument("native RoPE buffers partially overlap"); } // Match pinned host-side float powf before device fast powf/trigonometry. const float theta_scale = powf(freq_base, -2.0f / n_rot); apply<<(stream)>>>(x, out, rows, head_dim, n_rot, theta_scale, positions, mrope_table()); + static_cast(stream)>>>(x, out, rows, head_dim, n_rot, theta_scale, positions, mrope_table(), + heads, pos_stride); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index d45bf55b16e..3d75422d1e8 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -6,7 +6,10 @@ #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/fused_gr.hpp" +#include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_moe.hpp" +#include "strata/kernels/native_qsa_indexer.hpp" +#include "strata/kernels/native_rope.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" @@ -279,6 +282,171 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- RoPE over several tokens' heads: native_rope_apply per token vs native_rope_apply_tokens +int test_rope_tokens(std::mt19937& rng, cudaStream_t s) { + const int NH = 24; + int bad = 0; + for (int heads : {2, 4, 24}) { + for (int head_dim : {128, 256}) { + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + std::vector x((size_t) n_tok * heads * head_dim); + for (auto& v : x) v = edgy(rng); + std::vector pos((size_t) n_tok * NH); // a window's per-token position vectors + const int p0 = (int) (rng() % 200000); + for (int t = 0; t < n_tok; ++t) + for (int h = 0; h < NH; ++h) pos[(size_t) t * NH + h] = p0 + t; + float *d_a = dev(x.size()), *d_b = dev(x.size()); + int32_t* d_pos = dev(pos.size()); + up(d_a, x); + up(d_b, x); + up(d_pos, pos); + for (int t = 0; t < n_tok; ++t) + strata::kernels::native_rope_apply(d_a + (size_t) t * heads * head_dim, d_a + (size_t) t * heads * head_dim, + heads, head_dim, 64, 5000000.0f, d_pos + t * NH, s); + strata::kernels::native_rope_apply_tokens(d_b, d_b, n_tok * heads, head_dim, 64, 5000000.0f, d_pos, heads, + NH, s); + check(cudaStreamSynchronize(s), "rope"); + bad += bitwise_diff(down(d_a, x.size()), down(d_b, x.size()), "rope_tokens"); + cudaFree(d_a); + cudaFree(d_b); + cudaFree(d_pos); + } + } + } + std::printf("rope_tokens: %s\n", bad ? "MISMATCH" : "bitwise equal (2, 4, 24 heads; 1-8 tokens)"); + return bad; +} + +// ---- the int8 KV append: kv_append_q8_step per token vs kv_append_q8_steps +int test_kv_append(std::mt19937& rng, cudaStream_t s) { + const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); + const int cells = 4096, pages = cells / (int) sh.page_size, per_tok = (int) (sh.n_head_kv * sh.head_dim); + const size_t codes = (size_t) cells * per_tok, scales = codes / strata::kernels::KV_Q8_GROUP; + std::vector table((size_t) pages); + for (int i = 0; i < pages; ++i) table[(size_t) i] = pages - 1 - i; // a permuted page table + int32_t* d_table = dev(table.size()); + up(d_table, table); + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + const int p0 = 500 + (int) (rng() % 2000); + std::vector k((size_t) n_tok * per_tok), v((size_t) n_tok * per_tok); + for (auto& x : k) x = edgy(rng); + for (auto& x : v) x = edgy(rng); + std::vector steps((size_t) n_tok * strata::kernels::kStepCount, 0); + for (int t = 0; t < n_tok; ++t) steps[(size_t) t * strata::kernels::kStepCount + strata::kernels::kStepPos] = p0 + t; + float *d_k = dev(k.size()), *d_v = dev(v.size()); + int32_t* d_steps = dev(steps.size()); + up(d_k, k); + up(d_v, v); + up(d_steps, steps); + int8_t* q[2][2]; + uint16_t* sc[2][2]; + for (int p = 0; p < 2; ++p) + for (int kv = 0; kv < 2; ++kv) { + q[p][kv] = dev(codes); + sc[p][kv] = dev(scales); + check(cudaMemset(q[p][kv], 0, codes), "memset"); + check(cudaMemset(sc[p][kv], 0, scales * 2), "memset"); + } + for (int t = 0; t < n_tok; ++t) + strata::kernels::kv_append_q8_step(q[0][0], q[0][1], sc[0][0], sc[0][1], d_table, + d_steps + t * strata::kernels::kStepCount, d_k + (size_t) t * per_tok, + d_v + (size_t) t * per_tok, sh, s); + strata::kernels::kv_append_q8_steps(q[1][0], q[1][1], sc[1][0], sc[1][1], d_table, d_steps, + strata::kernels::kStepCount, d_k, d_v, n_tok, sh, s); + check(cudaStreamSynchronize(s), "kv append"); + for (int kv = 0; kv < 2; ++kv) { + if (down(q[0][kv], codes) != down(q[1][kv], codes) || down(sc[0][kv], scales) != down(sc[1][kv], scales)) { + std::fprintf(stderr, "kv_append: n_tok %d: the %s cells differ\n", n_tok, kv ? "V" : "K"); + ++bad; + } + } + for (int p = 0; p < 2; ++p) + for (int kv = 0; kv < 2; ++kv) { + cudaFree(q[p][kv]); + cudaFree(sc[p][kv]); + } + cudaFree(d_k); + cudaFree(d_v); + cudaFree(d_steps); + } + cudaFree(d_table); + std::printf("kv_append_steps: %s\n", bad ? "MISMATCH" : "bitwise equal (1-8 tokens)"); + return bad; +} + +// ---- the indexer append: one call per token vs native_qsa_indexer_append_multi (blocks completed inside the +// window, the first cell, and rejected tokens' -1 positions) +int test_indexer_append(std::mt19937& rng, cudaStream_t s) { + const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); + const int D = 128, max_cells = 4096; + const size_t pooled_n = (size_t) (max_cells / 4 + 1) * D; + std::vector gamma((size_t) D); + for (auto& g : gamma) g = 0.5f + std::uniform_real_distribution(0.0f, 1.0f)(rng); + float* d_gamma = dev(gamma.size()); + up(d_gamma, gamma); + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + for (int start : {0, 1, 2, 3, 5, 6, 7, 1022}) { + for (int rejected = 0; rejected < 2; ++rejected) { + std::vector raw((size_t) n_tok * D); + for (auto& x : raw) x = std::normal_distribution(0.0f, 1.0f)(rng); + std::vector pos((size_t) n_tok); + for (int t = 0; t < n_tok; ++t) pos[(size_t) t] = rejected && t >= (n_tok + 1) / 2 ? -1 : start + t; + // the state both paths start from: a random earlier tail, spare and pooled blocks + std::vector tail((size_t) 3 * D), dead((size_t) D), pooled(pooled_n); + for (auto& x : tail) x = edgy(rng); + for (auto& x : dead) x = edgy(rng); + for (auto& x : pooled) x = edgy(rng); + float *d_raw = dev(raw.size()); + int32_t* d_pos = dev(pos.size()); + up(d_raw, raw); + up(d_pos, pos); + float* bufs[2][3]; + int32_t* bp[2]; + for (int p = 0; p < 2; ++p) { + bufs[p][0] = dev(tail.size()); + bufs[p][1] = dev(dead.size()); + bufs[p][2] = dev(pooled.size()); + bp[p] = dev(1); + up(bufs[p][0], tail); + up(bufs[p][1], dead); + up(bufs[p][2], pooled); + up(bp[p], std::vector{-7}); + } + const strata::kernels::QsaIndexerBuffers ib0{bufs[0][0], bufs[0][1], bufs[0][2], bp[0]}; + const strata::kernels::QsaIndexerBuffers ib1{bufs[1][0], bufs[1][1], bufs[1][2], bp[1]}; + for (int t = 0; t < n_tok; ++t) + strata::kernels::native_qsa_indexer_append(d_raw + (size_t) t * D, d_pos + t, 0, d_gamma, 1e-6f, ib0, sh, + max_cells, 5000000.0f, s); + strata::kernels::native_qsa_indexer_append_multi(d_raw, d_pos, 1, n_tok, 0, d_gamma, 1e-6f, ib1, sh, + max_cells, 5000000.0f, s); + check(cudaStreamSynchronize(s), "indexer"); + const size_t sizes[3] = {tail.size(), dead.size(), pooled.size()}; + const char* names[3] = {"tail", "dead", "pooled"}; + for (int o = 0; o < 3; ++o) { + const int b = bitwise_diff(down(bufs[0][o], sizes[o]), down(bufs[1][o], sizes[o]), names[o]); + if (b) std::fprintf(stderr, "indexer: n_tok %d start %d rejected %d: %s differ\n", n_tok, start, rejected, names[o]); + bad += b; + } + if (down(bp[0], 1) != down(bp[1], 1)) { + std::fprintf(stderr, "indexer: n_tok %d start %d: block_pos differs\n", n_tok, start); + ++bad; + } + for (int p = 0; p < 2; ++p) { + for (float* q : bufs[p]) cudaFree(q); + cudaFree(bp[p]); + } + cudaFree(d_raw); + cudaFree(d_pos); + } + } + } + cudaFree(d_gamma); + std::printf("indexer_append_multi: %s\n", bad ? "MISMATCH" : "bitwise equal (1-8 tokens, 8 starts, rejected cells)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -296,6 +464,9 @@ int main(int argc, char** argv) { bad += test_mmvf_multi(rng, s); bad += test_router_multi(rng, s); bad += test_gr_read(rng, s); + bad += test_rope_tokens(rng, s); + bad += test_kv_append(rng, s); + bad += test_indexer_append(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index de3e5da4d5c..898d73fd159 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -244,6 +244,7 @@ struct Options { std::vector eos_ids = {248044, 248046}; bool spec_split = false; ///< opt-in split verify window (the overlap study: exact, ~7% slower) bool window_profile = false; ///< GPU timestamps between the verify window's stages, printed by --stats + std::string window_hashes; ///< per verify window: a hash of its final residual rows and its argmaxes /// Plan v0.3 P8: stay resident and take requests on stdin (see the --serve block in main). bool serve = false; /// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests. @@ -358,6 +359,9 @@ void usage() { " --gpu-only-full MEASURE: replay pre+post for all 48 layers plus the LM head, no pool.\n" " THE TRUE PER-TOKEN GPU FLOOR. Quote this one, not --graph-only.\n" " --stats print the per-stage breakdown\n" + " --window-hashes PATH per verify window, a line with its size, a 64-bit hash of its final residual\n" + " rows and its argmaxes: two builds that do the same arithmetic write the same\n" + " file (fixed text through --spec-follow, --adapt-every 0; not --serve)\n" " --window-profile with --stats: the GPU time of each stage of the verify window (timestamps\n" " inside the graph; each costs ~2 us)\n" " --gpu-stages R0.9: capture the layer as three graphs (mixer / ffn+router / post)\n" @@ -596,6 +600,7 @@ int main(int argc, char** argv) { else if (a == "--stop-eos") o.stop_eos = true; else if (a == "--spec-split") o.spec_split = true; else if (a == "--window-profile") o.window_profile = true; + else if (a == "--window-hashes") o.window_hashes = next("--window-hashes"); else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode"); else if (a == "--serve") o.serve = true; else if (a == "--vision") o.vision = true; @@ -2690,6 +2695,12 @@ int main(int argc, char** argv) { } std::vector window((size_t) o.spec), outv((size_t) o.spec); std::vector fdiff((size_t) o.spec, 0); + std::FILE* hashes = o.window_hashes.empty() ? nullptr : std::fopen(o.window_hashes.c_str(), "w"); + if (!o.window_hashes.empty() && hashes == nullptr) { + std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_hashes.c_str()); + return 1; + } + std::vector hash_rows; std::vector accepted_hist((size_t) o.spec, 0); int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; @@ -2729,6 +2740,17 @@ int main(int argc, char** argv) { drive.d.fail ? drive.d.fail : "(no message)"); return 1; } + if (hashes != nullptr) { // FNV-1a over the window's final residual rows + const size_t n_floats = (size_t) T * (size_t) (g.hc * g.n_embd); + hash_rows.resize(n_floats); + cudaMemcpy(hash_rows.data(), ver.final_R_all(), n_floats * sizeof(float), cudaMemcpyDeviceToHost); + uint64_t h = 1469598103934665603ull; + const auto* bytes = (const uint8_t*) hash_rows.data(); + for (size_t i = 0; i < n_floats * sizeof(float); ++i) h = (h ^ bytes[i]) * 1099511628211ull; + std::fprintf(hashes, "%lld %d %016llx", (long long) rounds, T, (unsigned long long) h); + for (int i = 0; i < T; ++i) std::fprintf(hashes, " %d", (int) outv[(size_t) i]); + std::fprintf(hashes, "\n"); + } if (!follow.empty()) { // the continuation stands in for the argmax const size_t k0 = produced.size(); for (int i = 0; i < T && k0 + (size_t) i < follow.size(); ++i) { @@ -2786,6 +2808,7 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: position %lld, %lld tokens, %lld rounds\n", (long long) p, (long long) produced.size(), (long long) rounds); } + if (hashes != nullptr) std::fclose(hashes); std::printf("%-24s %lld rounds of %d, drafts accepted %lld of %lld (%.3f), %.2f tokens per round\n", "speculation", (long long) rounds, o.spec, (long long) drafts_ok, (long long) drafts_total, drafts_total > 0 ? (double) drafts_ok / (double) drafts_total : 0.0, From 7671fd2b84e17ca2341918bcb0c4c089263e6ce9 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Fri, 25 Sep 2026 23:55:34 +0200 Subject: [PATCH 17/92] Prompt path: the PLE block and the indexer appends over the whole chunk; --prefill-profile The PLE block ran once per prompt token (a 52 MB key GEMV each); its key and value projections are now GEMMs over the chunk, followed by the decode path's per-token arithmetic in one pass (native_ple_postops_tokens, bitwise n_tok native_ple_postops + ple_history_advance calls; verify_parity checks it). The indexer appends of a chunk are one native_qsa_indexer_append_multi launch. 16K-token prompt, IQ3_XXS: 22.8 -> 18.3 s. --prefill-profile prints the GPU time of each section of the prompt path; --window-hashes lines carry the window's first position. --- include/strata/kernels/native_gr_norm.hpp | 4 + include/strata/kernels/native_ple_postops.hpp | 16 ++++ include/strata/prefill/prefill.hpp | 12 +++ src/kernels/cuda/native_gr_norm.cu | 17 ++-- src/kernels/cuda/native_ple_postops.cu | 70 +++++++++++++++ src/kernels/verify_parity.cpp | 63 ++++++++++++- src/prefill/prefill.cpp | 89 +++++++++++++------ src/program/generate.cpp | 20 ++++- 8 files changed, 255 insertions(+), 36 deletions(-) diff --git a/include/strata/kernels/native_gr_norm.hpp b/include/strata/kernels/native_gr_norm.hpp index b8bb4ed1b6c..dc5b3ab9cc3 100644 --- a/include/strata/kernels/native_gr_norm.hpp +++ b/include/strata/kernels/native_gr_norm.hpp @@ -14,5 +14,9 @@ namespace strata::kernels { // default stream, preserving gr_read's existing API; this function never syncs. void native_gr_rms_norm_weighted(const float* input, const float* gamma, float* output, int n_cols, int n_rows, float epsilon, void* stream); +// The same with gamma holding gamma_rows rows: row r is scaled by gamma row r % gamma_rows (several tokens' +// rows against one token's gamma). +void native_gr_rms_norm_weighted_repeat(const float* input, const float* gamma, float* output, + int n_cols, int n_rows, int gamma_rows, float epsilon, void* stream); } // namespace strata::kernels diff --git a/include/strata/kernels/native_ple_postops.hpp b/include/strata/kernels/native_ple_postops.hpp index 9e947878084..ad6d0c4f881 100644 --- a/include/strata/kernels/native_ple_postops.hpp +++ b/include/strata/kernels/native_ple_postops.hpp @@ -27,4 +27,20 @@ void native_ple_postops(const float* projected_key, const float* hidden, const PleWeights& weights, const NativePlePostopsBuffers& buffers, void* stream); +// n_tok consecutive tokens (the prompt path), row-major with the token slow: projected_key, hidden, key, query, +// gated, normalized and result [n_tok, 10240], value [n_tok, 2560], gate [n_tok, 4]. Bitwise n_tok calls of +// native_ple_postops, each followed by ple_history_advance: history is read, then left holding the last nine +// normalized rows. result may equal hidden and query may equal normalized; nothing else may overlap. +struct NativePleTokensBuffers { + float* key; + float* query; + float* gate; + float* gated; + float* normalized; + float* result; +}; +void native_ple_postops_tokens(const float* projected_key, const float* hidden, const float* value, + float* history, const PleWeights& weights, const NativePleTokensBuffers& buffers, + int n_tok, void* stream); + } // namespace strata::kernels diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index 11c0baef5ce..0b3bf8e6e47 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -23,6 +23,13 @@ namespace strata::prefill { +/// The sections `Prefill::profile` times, in the order a layer runs them. +enum PrefillSection { + kPsPle, kPsHcRead, kPsGdn, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, + kPsExperts, kPsCombine, kPsCount +}; +const char* prefill_section_name(int section); + struct PrefillStats { int64_t tokens = 0; int64_t chunks = 0; @@ -32,6 +39,7 @@ struct PrefillStats { int64_t experts_dma = 0; ///< ...of which straight from the pinned arena (no CPU copy) int64_t experts_resident = 0; ///< expert-layer groups served from the VRAM tier double ms_ple = 0; + double ms_section[kPsCount] = {}; ///< GPU ms per section, with `Prefill::profile` }; class Prefill { @@ -66,6 +74,10 @@ class Prefill { /// embedding where non-null (an image's <|image_pad|> cells). Null (default): every position embeds its token. const float* const* embd_rows = nullptr; + /// GPU time per section into `stats().ms_section` (events between the sections; the host waits for the last + /// one after every chunk). + bool profile = false; + private: struct Impl; std::unique_ptr impl_; diff --git a/src/kernels/cuda/native_gr_norm.cu b/src/kernels/cuda/native_gr_norm.cu index 5abe013ecf3..9a1d1593dc1 100644 --- a/src/kernels/cuda/native_gr_norm.cu +++ b/src/kernels/cuda/native_gr_norm.cu @@ -45,11 +45,11 @@ __device__ __forceinline__ float norm_warp_sum(float value) { template __global__ void weighted_rms_norm(const float* __restrict__ input, const float* __restrict__ gamma, - float* __restrict__ output, int n_cols, float epsilon) { + float* __restrict__ output, int n_cols, int gamma_rows, float epsilon) { const int tid = threadIdx.x; const std::size_t row_offset = std::size_t(blockIdx.x) * n_cols; input += row_offset; - gamma += row_offset; + gamma += std::size_t(blockIdx.x % gamma_rows) * n_cols; output += row_offset; float partial = 0.0f; for (int col = tid; col < n_cols; col += BlockSize) { @@ -84,16 +84,23 @@ void check_pointer(const void* p) { void native_gr_rms_norm_weighted(const float* input, const float* gamma, float* output, int n_cols, int n_rows, float epsilon, void* stream) { - if (n_cols <= 0 || n_rows <= 0 || !std::isfinite(epsilon) || epsilon < 0.0f) + native_gr_rms_norm_weighted_repeat(input, gamma, output, n_cols, n_rows, n_rows, epsilon, stream); +} + +void native_gr_rms_norm_weighted_repeat(const float* input, const float* gamma, float* output, + int n_cols, int n_rows, int gamma_rows, float epsilon, void* stream) { + if (n_cols <= 0 || n_rows <= 0 || gamma_rows <= 0 || !std::isfinite(epsilon) || epsilon < 0.0f) throw std::invalid_argument("native GR RMSNorm requires positive dimensions and finite nonnegative epsilon"); check_pointer(input); check_pointer(gamma); check_pointer(output); const auto cuda_stream = static_cast(stream); if (n_cols < 1024) - weighted_rms_norm<256><<>>(input, gamma, output, n_cols, epsilon); + weighted_rms_norm<256><<>>(input, gamma, output, n_cols, gamma_rows, + epsilon); else - weighted_rms_norm<1024><<>>(input, gamma, output, n_cols, epsilon); + weighted_rms_norm<1024><<>>(input, gamma, output, n_cols, gamma_rows, + epsilon); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(std::string("native GR RMSNorm launch: ") + cudaGetErrorString(error)); diff --git a/src/kernels/cuda/native_ple_postops.cu b/src/kernels/cuda/native_ple_postops.cu index f8d2df32e7c..7a2cf51d7ee 100644 --- a/src/kernels/cuda/native_ple_postops.cu +++ b/src/kernels/cuda/native_ple_postops.cu @@ -86,6 +86,43 @@ __global__ void conv_residual_kernel(const float* history, const float* normaliz // Exact hidden/result alias is safe: each thread owns one element. result[c] = __fadd_rn(hidden[c], __fadd_rn(gated[c], activation)); } +// n_tok tokens: token t's taps read the normalized rows of tokens t-9, t-6 and t-3 (history rows t, t+3 and t+6 +// before the chunk while those tokens precede it) and its own. +__global__ void broadcast_tokens_kernel(const float* value, const float* gate, float* gated, int n_tok) { + const size_t i = size_t(blockIdx.x) * blockDim.x + threadIdx.x; + if (i < size_t(n_tok) * D) gated[i] = __fmul_rn(value[i / D * N + i % N], gate[i / N]); +} +__global__ void conv_residual_tokens_kernel(const float* history, const float* normalized, + const uint16_t* weights, const float* hidden, + const float* gated, float* result, int n_tok) { + const size_t i = size_t(blockIdx.x) * blockDim.x + threadIdx.x; + if (i >= size_t(n_tok) * D) return; + const int t = int(i / D), c = int(i % D); + float sum = 0; +#pragma unroll + for (int k = 0; k < 4; ++k) { + const int j = t - HISTORY + 3 * k; + const float x = j >= 0 ? normalized[size_t(j) * D + c] : history[c * HISTORY + j + HISTORY]; + const float w = __half2float(__ushort_as_half(weights[c * 4 + k])); + const float term = __fmul_rn(x, w); + sum = k == 0 ? term : __fadd_rn(sum, term); + } + const float activation = sum / (1.0f + expf(-sum)); + result[i] = __fadd_rn(hidden[i], __fadd_rn(gated[i], activation)); +} +// The history after the chunk: its last nine normalized rows, older rows moved up when the chunk is shorter. +__global__ void history_tokens_kernel(float* history, const float* normalized, int n_tok) { + const int c = blockIdx.x * blockDim.x + threadIdx.x; + if (c >= D) return; + float old[HISTORY]; +#pragma unroll + for (int r = 0; r < HISTORY; ++r) old[r] = history[c * HISTORY + r]; +#pragma unroll + for (int r = 0; r < HISTORY; ++r) { + const int j = n_tok - HISTORY + r; + history[c * HISTORY + r] = j >= 0 ? normalized[size_t(j) * D + c] : old[r + n_tok]; + } +} struct Span { const void* p; size_t bytes; size_t alignment; }; bool overlaps(Span a, Span b) { const auto x = reinterpret_cast(a.p), y = reinterpret_cast(b.p); @@ -131,4 +168,37 @@ void native_ple_postops(const float* projected_key, const float* hidden, conv_residual_kernel<<>>(history,b.normalized,w.conv1d_f16,hidden,b.gated,b.conv,b.result); launch_check(); } + +void native_ple_postops_tokens(const float* projected_key, const float* hidden, const float* value, float* history, + const PleWeights& w, const NativePleTokensBuffers& b, int n_tok, void* stream) { + if (!stream || n_tok < 1) throw std::invalid_argument("native PLE postops: tokens need a stream and n_tok >= 1"); + const size_t rows = size_t(n_tok) * D * 4; + const Span inputs[] = {{projected_key,rows,4}, {hidden,rows,4}, {value,size_t(n_tok)*N*4,4}, + {history,HISTORY*D*4,4}, {w.norm_key,D*4,4}, {w.norm_query,D*4,4}, + {w.norm_conv,D*4,4}, {w.conv1d_f16,4*D*2,2}}; + const Span outputs[] = {{b.key,rows,4}, {b.query,rows,4}, {b.gate,size_t(n_tok)*H*4,4}, + {b.gated,rows,4}, {b.normalized,rows,4}, {b.result,rows,4}}; + for (const auto& span : inputs) validate(span); + for (const auto& span : outputs) validate(span); + for (size_t i = 0; i < 6; ++i) { + for (size_t j = 0; j < 8; ++j) + if (!(i == 5 && j == 1 && b.result == hidden) && overlaps(outputs[i], inputs[j])) + throw std::invalid_argument("native PLE postops output overlaps an input, the history or a weight"); + for (size_t j = i + 1; j < 6; ++j) + if (!(i == 1 && j == 4 && b.query == b.normalized) && overlaps(outputs[i], outputs[j])) + throw std::invalid_argument("native PLE postops writable spans overlap"); + } + const int R = H * n_tok; + const unsigned blocks = unsigned((size_t(n_tok) * D + 255) / 256); + auto st = static_cast(stream); + native_gr_rms_norm_weighted_repeat(projected_key,w.norm_key,b.key,N,R,H,NG_RMS_EPS,stream); + native_gr_rms_norm_weighted_repeat(hidden,w.norm_query,b.query,N,R,H,NG_RMS_EPS,stream); + gate_kernel<<>>(b.key,b.query,b.gate,1.0f / std::sqrt(float(N))); + broadcast_tokens_kernel<<>>(value,b.gate,b.gated,n_tok); + launch_check(); + native_gr_rms_norm_weighted_repeat(b.gated,w.norm_conv,b.normalized,N,R,H,NG_RMS_EPS,stream); + conv_residual_tokens_kernel<<>>(history,b.normalized,w.conv1d_f16,hidden,b.gated,b.result,n_tok); + history_tokens_kernel<<>>(history,b.normalized,n_tok); + launch_check(); +} } // namespace strata::kernels diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 3d75422d1e8..ae1aece1116 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -1,13 +1,15 @@ -// src/kernels/verify_parity.cpp - the verify window's batched kernels against the kernel sequences they replace. +// src/kernels/verify_parity.cpp - batched kernels against the per-token kernel sequences they replace. // // A verify window's token t must come out bit for bit as it would in a window of any other size (the drafts are // accepted exactly when greedy decode would have produced them), so every batched kernel is checked BITWISE against -// the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. +// the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. The +// prompt path's batched PLE arithmetic is held to the same standard. #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/fused_gr.hpp" #include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_moe.hpp" +#include "strata/kernels/native_ple_postops.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_rope.hpp" #include "strata/kernels/native_router.hpp" @@ -447,6 +449,62 @@ int test_indexer_append(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the PLE block after its projections: native_ple_postops + ple_history_advance per token vs +// native_ple_postops_tokens (chunks shorter and longer than the nine-row history) +int test_ple_tokens(std::mt19937& rng, cudaStream_t s) { + const int N = 2560, D = 10240, H = 4, HIST = 9; + std::normal_distribution nd(0.0f, 1.0f); + std::vector nk((size_t) D), nq((size_t) D), nc((size_t) D); + for (auto* v : {&nk, &nq, &nc}) + for (auto& x : *v) x = 0.5f + std::uniform_real_distribution(0.0f, 1.0f)(rng); + std::vector taps((size_t) 4 * D); + for (auto& t : taps) { // F16 in [-1, 1): sign, exponent 0..14, random mantissa + const uint16_t sign = (uint16_t) ((rng() & 1u) << 15), e = (uint16_t) (rng() % 15), m = (uint16_t) (rng() & 0x3FFu); + t = (uint16_t) (sign | (e << 10) | m); + } + float *d_nk = dev(D), *d_nq = dev(D), *d_nc = dev(D); + uint16_t* d_taps = dev(taps.size()); + up(d_nk, nk); up(d_nq, nq); up(d_nc, nc); up(d_taps, taps); + strata::kernels::PleWeights w; + w.norm_key = d_nk; w.norm_query = d_nq; w.norm_conv = d_nc; w.conv1d_f16 = d_taps; + int bad = 0; + for (int n_tok : {1, 2, 5, 9, 10, 23}) { + std::vector key((size_t) n_tok * D), hidden((size_t) n_tok * D), value((size_t) n_tok * N), + hist((size_t) HIST * D); + for (auto& x : key) x = edgy(rng); + for (auto& x : hidden) x = edgy(rng); + for (auto& x : value) x = nd(rng); + for (auto& x : hist) x = nd(rng); + float *d_key = dev(key.size()), *d_hidden = dev(hidden.size()), *d_value = dev(value.size()); + up(d_key, key); up(d_hidden, hidden); up(d_value, value); + // per token: the decode path's calls + float *h0 = dev(hist.size()), *r0 = dev(hidden.size()); + float *k1 = dev(D), *q1 = dev(D), *g1 = dev(H), *gd1 = dev(D), *c1 = dev(D); + up(h0, hist); + for (int t = 0; t < n_tok; ++t) { + const strata::kernels::NativePlePostopsBuffers b{k1, q1, g1, gd1, q1, c1, r0 + (size_t) t * D}; + strata::kernels::native_ple_postops(d_key + (size_t) t * D, d_hidden + (size_t) t * D, + d_value + (size_t) t * N, h0, w, b, s); + strata::kernels::ple_history_advance(h0, q1, s); + } + // the chunk at once + float *h2 = dev(hist.size()), *r2 = dev(hidden.size()), *k2 = dev(key.size()), + *q2 = dev(key.size()), *g2 = dev((size_t) n_tok * H), *gd2 = dev(key.size()); + up(h2, hist); + strata::kernels::native_ple_postops_tokens(d_key, d_hidden, d_value, h2, w, {k2, q2, g2, gd2, q2, r2}, n_tok, s); + check(cudaStreamSynchronize(s), "ple tokens"); + const int br = bitwise_diff(down(r0, hidden.size()), down(r2, hidden.size()), "result"); + const int bh = bitwise_diff(down(h0, hist.size()), down(h2, hist.size()), "history"); + if (br || bh) std::fprintf(stderr, "ple tokens: n_tok %d: result or history differ\n", n_tok); + bad += br + bh; + for (float* p : {d_key, d_hidden, d_value, h0, r0, k1, q1, g1, gd1, c1, h2, r2, k2, q2, g2, gd2}) cudaFree(p); + } + for (float* p : {d_nk, d_nq, d_nc}) cudaFree(p); + cudaFree(d_taps); + std::printf("ple_postops_tokens: %s\n", bad ? "MISMATCH" : "bitwise equal (1-23 tokens, results and history)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -467,6 +525,7 @@ int main(int argc, char** argv) { bad += test_rope_tokens(rng, s); bad += test_kv_append(rng, s); bad += test_indexer_append(rng, s); + bad += test_ple_tokens(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 6c9aa5e7cfb..e83dfc21756 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -4,8 +4,8 @@ #include "strata/core/layout.hpp" #include "strata/kernels/cpu/expert.hpp" #include "strata/kernels/native_qsa_indexer.hpp" +#include "strata/kernels/native_ple_postops.hpp" #include "strata/kernels/ngram.hpp" -#include "strata/kernels/ple.hpp" #include "strata/kernels/iq_kernels.hpp" #include "strata/kernels/cpu/expert_layout.hpp" #include "strata/kernels/qsa.hpp" @@ -109,7 +109,6 @@ struct Prefill::Impl { float* ple_emb = nullptr; std::vector ple_emb_host; std::vector ple_rows; - float* ple_norm = nullptr; PrefillStats* stats = nullptr; }; @@ -188,7 +187,6 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co if (cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) != cudaSuccess) ok = false; } m.ple_emb = o.take(T * N, ok); - m.ple_norm = o.take((size_t) strata::kernels::NG_HC_DIM, ok); m.ids_host.resize(T * K); m.slot_host.resize(T * K); m.src_host.resize(T * K); m.cnt.resize(NE); m.off.resize(NE + 1); m.ple_emb_host.resize(T * N); m.ple_rows.resize(T * strata::kernels::PLE_N_HEADS); if (!ok) { err = "prefill: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; return false; } @@ -222,7 +220,6 @@ uint64_t Prefill::bytes_needed(const core::ModelGeometry& g, const core::Session for (int i = 0; i < DQ; ++i) { o.take(1280 * 2560, ok); o.take(2560 * 640, ok); } for (int i = 0; i < STAGE; ++i) o.take((size_t) MAXBLOB(), ok); f(T * N); - f((size_t) strata::kernels::NG_HC_DIM); return o.used + (8u << 20); // alignment slack } @@ -259,6 +256,15 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const uint64_t gdn_floats = (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size + (uint64_t) g.ssm_conv_channels * (g.ssm_d_conv - 1); int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]}; + // --prefill-profile: an event at the start of each section; the time to the next event is charged to it + std::vector> ev; + auto mark = [&](int section) { + if (!profile) return; + cudaEvent_t e; + cudaEventCreate(&e); + cudaEventRecord(e, m.cs); + ev.push_back({section, e}); + }; for (int64_t c0 = 0; c0 < n; c0 += m.T) { const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0; @@ -300,22 +306,29 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e int64_t qsa_index = 0, gdn_index = 0; for (int64_t l = 0; l < g.n_layers; ++l) { const core::LayerView v(*m.wt, l); - // ---- the PLE block at layer 1, token by token (its conv reads the previous tokens' rows) + mark(kPsPle); + // ---- the PLE block at layer 1: the key and value projections of the chunk, then the decode path's + // per-token arithmetic (the conv reads the previous tokens' normalized rows) if (l == 1 && ple_on) { - const auto tp = Clock::now(); - for (int64_t t = 0; t < T; ++t) { - strata::kernels::PleOut po; - po.normalized = m.ple_norm; - po.result = m.R + t * D; - try { - strata::kernels::ple_block(m.ple_emb + t * N, m.R + t * D, ss.ple.hist, ss.ple.w, po, - ss.ple.scratch, m.cs); - } catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; } - strata::kernels::ple_history_advance(ss.ple.hist, m.ple_norm, m.cs); + const strata::kernels::PleWeights& pw = ss.ple.w; + to_bf16(m.ple_emb, m.mixed_bf, T * N, m.cs); + if (pw.key_bf16 != nullptr) { + m.gemm.bf16(m.mixed_bf, pw.key_bf16, m.xn, T, D, N); + } else if (pw.key_native_data != nullptr) { + to_f16(m.ple_emb, m.mixed_h, T * N, m.cs); + m.gemm.native(m.mixed_h, pw.key_native_type, pw.key_native_data, m.xn, T, D, N); + } else { + err = "prefill: the PLE key has neither a BF16 nor a native GGUF form (run with --native)"; + return false; } - stats_.ms_ple += ms_since(tp); + m.gemm.bf16(m.mixed_bf, pw.value_bf16, m.bo, T, N, N); + try { + strata::kernels::native_ple_postops_tokens(m.xn, m.R, m.bo, ss.ple.hist, pw, + {m.qkv, m.hbuf, m.inj, m.gated, m.hbuf, m.R}, (int) T, m.cs); + } catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; } } for (int half = 0; half < 2; ++half) { + if (half == 0) mark(kPsHcRead); // ---- the hyper-connection read of this half const char* pre = half == 0 ? "hc_attn_" : "hc_ffn_"; const std::string sn = std::string(pre) + "norm.weight", sd = std::string(pre) + "down.weight", @@ -338,6 +351,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wnm = need(v, "ssm_norm.weight", err), *wdt = need(v, "ssm_dt.bias", err), *wsa = need(v, "ssm_a", err); if (!wqkv || !wg || !wo || !wa || !wb || !wc || !wnm || !wdt || !wsa) return false; + mark(kPsGdn); float* state = ss.gdn_state + (size_t) gdn_index * gdn_floats; float* conv = state + (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size; if (!native_proj(m.gemm, wqkv, m.mixed_h, m.qkv, T, v.name("attn_qkv.weight"), err)) return false; @@ -360,6 +374,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wiqn = need(v, "indexer.q_norm.weight", err), *wikn = need(v, "indexer.k_norm.weight", err); if (!wq || !wk || !wv || !wo || !wik || !wiq || !wqn || !wkn || !wiqn || !wikn) return false; + mark(kPsQsaProj); if (!native_proj(m.gemm, wk, m.mixed_h, m.Kc, T, v.name("attn_k.weight"), err)) return false; if (!native_proj(m.gemm, wv, m.mixed_h, m.Vc, T, v.name("attn_v.weight"), err)) return false; if (!native_proj(m.gemm, wq, m.mixed_h, m.Qf, T, v.name("attn_q.weight"), err)) return false; @@ -374,17 +389,16 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e rope(m.q, T, 24, 256, 6144, p0, (float) strata::kernels::qsa_freq_base(), m.cs); rms_rows(m.q_idx, (const float*) wiqn->data, T * 4, 128, 128, EPS, m.cs); rope(m.q_idx, T, 4, 128, 512, p0, (float) strata::kernels::qsa_freq_base(), m.cs); - // the indexer appends, token by token; then scores + selection for many queries at once: + // the indexer appends of the chunk; then scores + selection for many queries at once: // a query reads completed blocks (final once completed) and `dead` for its own tail block const strata::kernels::QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; - for (int64_t t = 0; t < T; ++t) { - const int32_t* step_t = m.steps_dev + t * strata::kernels::kStepCount; - try { - strata::kernels::native_qsa_indexer_append(m.idx_raw + t * 128, step_t + strata::kernels::kStepPos, 0, - (const float*) wikn->data, EPS, ib, s, st.max_cells, - (float) strata::kernels::qsa_freq_base(), m.cs); - } catch (const std::exception& e) { err = std::string("prefill indexer: ") + e.what(); return false; } - } + mark(kPsQsaIndexer); + try { + strata::kernels::native_qsa_indexer_append_multi( + m.idx_raw, m.steps_dev + strata::kernels::kStepPos, (int) strata::kernels::kStepCount, (int) T, 0, + (const float*) wikn->data, EPS, ib, s, st.max_cells, (float) strata::kernels::qsa_freq_base(), m.cs); + } catch (const std::exception& e) { err = std::string("prefill indexer: ") + e.what(); return false; } + mark(kPsQsaScores); for (int64_t t0 = 0; t0 < T; t0 += m.sel_batch) { const int64_t nb = std::min(m.sel_batch, T - t0); const int32_t* steps0 = m.steps_dev + t0 * strata::kernels::kStepCount; @@ -397,12 +411,14 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e pools.page_table = st.page_table; if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } + mark(kPsQsaAttn); for (int64_t t0 = 0; t0 < T; t0 += m.attn_batch) { const int64_t nb = std::min(m.attn_batch, T - t0); strata::kernels::qsa_decode_attn_batch(m.q + t0 * ZV, pools, m.sel_ids + t0 * m.cap, m.steps_dev + t0 * strata::kernels::kStepCount, m.cap, s, m.attn_scratch, m.attn + t0 * ZV, nb, m.cs); } + mark(kPsQsaOut); gate_attn(m.attn, m.Qf, m.attn_h, T, m.cs); if (!native_proj(m.gemm, wo, m.attn_h, m.bo, T, v.name("attn_output.weight"), err)) return false; ++qsa_index; @@ -414,6 +430,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); if (!wr || !wgi || !wsg || !wsu || !wsd) return false; + mark(kPsRouter); if (!bf16_proj(m.gemm, wr, m.mixed_bf, m.logits, T, v.name("ffn_gate_inp.weight"), err)) return false; route(m.logits, m.ids, m.w, T, m.cs); // the shared expert and its scalar gate @@ -444,6 +461,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e cudaMemcpyAsync(m.slot_dev, m.slot_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); cudaMemcpyAsync(m.src_dev, m.src_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); gather_rows16(m.mixed_h, m.src_dev, m.Xs, T * K, N, m.cs); + mark(kPsExperts); // the experts, in id order: resident ones from VRAM, the others through the staging ring std::vector order; for (int32_t e = 0; e < NE; ++e) if (m.cnt[(size_t) e] > 0) order.push_back(e); @@ -509,12 +527,25 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e swiglu_interleaved(m.GU + o0 * 1280, m.Hh + o0 * 640, ne, m.cs); m.gemm.f16(m.Hh + o0 * 640, m.dq_d[q], m.Dm + o0 * N, ne, N, 640); } + mark(kPsCombine); moe_combine(m.Dm, m.slot_dev, m.w, m.shared, m.sg, m.bo, T, m.cs); } // ---- the hyper-connection write of this half + if (half == 0) mark(kPsHcFfn); gr_write(m.R, m.bo, m.inj, HC, T, m.cs); } } + if (profile) { + mark(kPsPle); + cudaEventSynchronize(ev.back().second); + for (size_t i = 0; i + 1 < ev.size(); ++i) { + float ms = 0; + cudaEventElapsedTime(&ms, ev[i].second, ev[i + 1].second); + stats_.ms_section[ev[i].first] += ms; + } + for (auto& e : ev) cudaEventDestroy(e.second); + ev.clear(); + } stats_.tokens += T; if (on_chunk) { if (cudaStreamSynchronize(m.cs) != cudaSuccess) { @@ -534,4 +565,12 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e return true; } +const char* prefill_section_name(int section) { + static const char* const names[kPsCount] = { + "PLE block", "HC read (mixer)", "GDN mixer", "QSA projections, norms, KV", "QSA indexer appends", + "QSA block scores + top-k", "QSA attention", "QSA gate + output", "HC write + HC read (FFN)", + "router, shared, grouping", "experts", "combine + HC write"}; + return section >= 0 && section < kPsCount ? names[section] : "?"; +} + } // namespace strata::prefill diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 898d73fd159..41deb1f8695 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -245,6 +245,7 @@ struct Options { bool spec_split = false; ///< opt-in split verify window (the overlap study: exact, ~7% slower) bool window_profile = false; ///< GPU timestamps between the verify window's stages, printed by --stats std::string window_hashes; ///< per verify window: a hash of its final residual rows and its argmaxes + bool prefill_profile = false; ///< GPU time per section of the batched prompt path /// Plan v0.3 P8: stay resident and take requests on stdin (see the --serve block in main). bool serve = false; /// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests. @@ -359,11 +360,13 @@ void usage() { " --gpu-only-full MEASURE: replay pre+post for all 48 layers plus the LM head, no pool.\n" " THE TRUE PER-TOKEN GPU FLOOR. Quote this one, not --graph-only.\n" " --stats print the per-stage breakdown\n" - " --window-hashes PATH per verify window, a line with its size, a 64-bit hash of its final residual\n" - " rows and its argmaxes: two builds that do the same arithmetic write the same\n" - " file (fixed text through --spec-follow, --adapt-every 0; not --serve)\n" + " --window-hashes PATH per verify window, a line with its round, first position and size, a 64-bit hash\n" + " of its final residual rows and its argmaxes: two builds that do the same\n" + " arithmetic write the same file (fixed text through --spec-follow,\n" + " --adapt-every 0; not --serve)\n" " --window-profile with --stats: the GPU time of each stage of the verify window (timestamps\n" " inside the graph; each costs ~2 us)\n" + " --prefill-profile the GPU time of each section of the batched prompt path (not --serve)\n" " --gpu-stages R0.9: capture the layer as three graphs (mixer / ffn+router / post)\n" " and time them from OUTSIDE the capture. The per-stage table on the\n" " real graph that --stage-timing cannot give. Prints and exits.\n" @@ -600,6 +603,7 @@ int main(int argc, char** argv) { else if (a == "--stop-eos") o.stop_eos = true; else if (a == "--spec-split") o.spec_split = true; else if (a == "--window-profile") o.window_profile = true; + else if (a == "--prefill-profile") o.prefill_profile = true; else if (a == "--window-hashes") o.window_hashes = next("--window-hashes"); else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode"); else if (a == "--serve") o.serve = true; @@ -2374,6 +2378,7 @@ int main(int argc, char** argv) { return mtp.prefill(R_rows, nxt.data(), T, p0, e); }; } + prefill.profile = o.prefill_profile; const Clock::time_point tp0 = Clock::now(); const int64_t n_batched = (o.prefill_until > 0 && o.prefill_until < n_prompt - 1) ? o.prefill_until : n_prompt - 1; if (!prefill.run(o.tokens.data(), n_batched, 0, err)) { @@ -2409,6 +2414,13 @@ int main(int argc, char** argv) { (long long) ps.tokens, (long long) ps.chunks, ps.ms_total, ps.ms_total > 0 ? 1000.0 * (double) ps.tokens / ps.ms_total : 0.0, (long long) ps.experts_streamed, (long long) ps.experts_dma, ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); + if (o.prefill_profile) { + std::fprintf(stderr, "strata generate: prefill GPU ms by section:"); + for (int i = 0; i < strata::prefill::kPsCount; ++i) + std::fprintf(stderr, "%s %s %.0f", i ? ";" : "", strata::prefill::prefill_section_name(i), + ps.ms_section[i]); + std::fprintf(stderr, "\n"); + } } for (int64_t pos = pos_start;; ++pos) { @@ -2747,7 +2759,7 @@ int main(int argc, char** argv) { uint64_t h = 1469598103934665603ull; const auto* bytes = (const uint8_t*) hash_rows.data(); for (size_t i = 0; i < n_floats * sizeof(float); ++i) h = (h ^ bytes[i]) * 1099511628211ull; - std::fprintf(hashes, "%lld %d %016llx", (long long) rounds, T, (unsigned long long) h); + std::fprintf(hashes, "%lld %lld %d %016llx", (long long) rounds, (long long) p, T, (unsigned long long) h); for (int i = 0; i < T; ++i) std::fprintf(hashes, " %d", (int) outv[(size_t) i]); std::fprintf(hashes, "\n"); } From af76f64ebdfe7ce01fd61c1e76aeff49b43c3dbe Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 00:56:06 +0200 Subject: [PATCH 18/92] QSA: block scores sized to the blocks in use; the prompt path's attention in one pass per query qsa_block_scores launched a warp for every block of the full context (max_cells / 4, 50K blocks at 200K) for every query, nearly all of them idle. Its warps now go round over the blocks: the verify window and the token path use a fixed grid of 1024 per query, the prompt path the chunk's largest n_bid + 1. Each score is computed as before (bitwise). qsa_prefill_attn: the prompt path's attention, a block per (KV head, query) walking the query's selected cells in tiles of 64 with an online softmax, instead of the decode kernel's split-K partial sums through memory. FP32; int8 codes are converted without I2F. verify_parity checks it against the decode kernel (relative difference 3.9e-6). 16K-token prompt: IQ3_XXS 18.3 -> 15.9 s, UD-Q4_K_XL 24.8 -> 22.4 s. --- include/strata/kernels/qsa_decode_attn.hpp | 6 + include/strata/kernels/qsa_select.hpp | 7 +- src/core/layer.cpp | 3 +- src/core/verify.cpp | 2 +- src/kernels/cuda/qsa_decode_attn.cu | 203 +++++++++++++++++++++ src/kernels/cuda/qsa_select.cu | 40 ++-- src/kernels/verify_parity.cpp | 86 ++++++++- src/prefill/prefill.cpp | 17 +- 8 files changed, 332 insertions(+), 32 deletions(-) diff --git a/include/strata/kernels/qsa_decode_attn.hpp b/include/strata/kernels/qsa_decode_attn.hpp index 2044ac5c953..268f7343f20 100644 --- a/include/strata/kernels/qsa_decode_attn.hpp +++ b/include/strata/kernels/qsa_decode_attn.hpp @@ -41,4 +41,10 @@ void qsa_decode_attn_step(const float* q, const QsaAttnPools& pools, const int32 void qsa_decode_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream); +/// The prompt path's form of qsa_decode_attn_batch (no scratch): a block per (KV head, query) walks the query's +/// cells in tiles of 64 with an online softmax, so no partial sums go through memory. FP32 like the decode +/// kernel; the sums are ordered differently. +void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, + const QsaShapes& s, float* attn, int64_t n_q, void* stream); + } // namespace strata::kernels diff --git a/include/strata/kernels/qsa_select.hpp b/include/strata/kernels/qsa_select.hpp index d04009d5814..51708359779 100644 --- a/include/strata/kernels/qsa_select.hpp +++ b/include/strata/kernels/qsa_select.hpp @@ -23,8 +23,13 @@ namespace strata::kernels { /// scores [nq, max_blocks]; q_idx [nq, idx_n_head, idx_dim] (normed and rotated); steps [nq, kStepCount]. +/// The grid holds a warp for each of the first `grid_blocks` blocks of every query and its warps go round for the +/// rest: a captured graph passes a fixed size (qsa_score_grid_blocks), the prompt path the chunk's largest n_bid + 1. void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx, const int32_t* steps, int64_t nq, - int64_t max_blocks, const QsaShapes& s, float* scores, void* stream); + int64_t max_blocks, const QsaShapes& s, float* scores, void* stream, int64_t grid_blocks); + +/// The block-score grid of the decode paths, whatever the context: 1024 warps per query. +constexpr int64_t qsa_score_grid_blocks = 1024; /// ids [nq, cap] (cells, ascending); `cap` >= the largest selection width. void qsa_block_topk(const float* scores, const int32_t* steps, int64_t nq, int64_t max_blocks, int64_t cap, diff --git a/src/core/layer.cpp b/src/core/layer.cpp index 9d8d82ce6d7..da241879c38 100644 --- a/src/core/layer.cpp +++ b/src/core/layer.cpp @@ -715,7 +715,8 @@ if (!normalize_rotate(b.q_idx, w_iqn, (int) g.idx_q_heads, (int) g.idx_key_dim)) int64_t max_blocks = (st.max_cells / s.idx_block) + 2; if (g_fast_select) { // Plan v0.3 P7: block-level FP32 scores and a radix selection over blocks (qsa_select.hpp). - qsa_block_scores(st.idx_pooled, st.idx_dead, b.q_idx, st.step, 1, max_blocks, s, b.cell_scores, stream); + qsa_block_scores(st.idx_pooled, st.idx_dead, b.q_idx, st.step, 1, max_blocks, s, b.cell_scores, stream, + qsa_score_grid_blocks); qsa_block_topk(b.cell_scores, st.step, 1, max_blocks, cap, s, b.ids, stream); } else { qsa_index_step(st.idx_pooled, b.q_idx, nullptr, s, st.step, max_blocks, b.cell_scores, stream); topk_512_step(b.cell_scores, s, cap, st.step, b.ids, stream); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index bf312d8a63e..29e65aef86c 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -464,7 +464,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (batched) norm_rope_tokens(qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + tb * NH); else for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH); qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, - s, scores_ + (size_t) tb * max_blocks_, cs); + s, scores_ + (size_t) tb * max_blocks_, cs, qsa_score_grid_blocks); qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, sel_ + (size_t) tb * cap_, cs); QsaAttnPools pools; diff --git a/src/kernels/cuda/qsa_decode_attn.cu b/src/kernels/cuda/qsa_decode_attn.cu index 9fe92b6ab13..577b78018f1 100644 --- a/src/kernels/cuda/qsa_decode_attn.cu +++ b/src/kernels/cuda/qsa_decode_attn.cu @@ -166,8 +166,211 @@ __global__ void __launch_bounds__(HD) attn_merge_kernel(const float* __restrict_ attn[(size_t) h * HD + d] = L > 0.0f ? acc / L : 0.0f; } +// ---- the prompt path: a block per (KV head, query) walks all of the query's cells in tiles, online softmax +constexpr int PT = 64; // cells per tile +static_assert(KV_Q8_GROUP == 64, "a score thread's 64 dimensions are one int8 scale group"); + +// Four int8 codes as floats without I2F (a quarter-rate instruction on sm_86): flip the sign bits, put each byte +// in the mantissa of 2^23 and subtract 2^23 + 128. Exact. +__device__ __forceinline__ void i8x4(uint32_t w, float* f) { + const uint32_t u = w ^ 0x80808080u; +#pragma unroll + for (int j = 0; j < 4; ++j) f[j] = __int_as_float((int) __byte_perm(u, 0x4B000000u, 0x7650u | j)) - 8388736.0f; +} + +// Scores: thread (cell, quarter) dots its cell's 64 dimensions of that quarter with the 12 query heads, the quarters +// summed through shared memory. Values: thread (dimension pair, cell parity) accumulates its two dimensions of the +// 12 heads over every other cell; the parities are added at the end. +template +__global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __restrict__ q, QsaAttnPools p, + const int32_t* __restrict__ ids, + const int32_t* __restrict__ steps, int n_kv_heads, + int page_size, float scale, float* __restrict__ attn, + int cap) { + const int kvh = blockIdx.x; + const size_t qi = blockIdx.y; + const size_t head0 = qi * (size_t) (n_kv_heads * G) + (size_t) kvh * G; + q += head0 * HD; + attn += head0 * HD; + ids += qi * (size_t) cap; + const int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + __shared__ __align__(16) float sq[G][HD]; // 12 KB: this KV head's query heads + __shared__ __align__(16) float spart[4][G][PT]; // 12 KB: the quarters' partial scores; the parity sums at the end + __shared__ __align__(16) float sp[PT][G]; // probabilities, cell-major + __shared__ long long srow[PT]; + __shared__ float s_alpha[G], s_m[G], s_l[G]; + const int t = threadIdx.x, lane = t & 31, warp = t >> 5; + for (int i = t; i < G * HD; i += THREADS) sq[i / HD][i % HD] = q[i]; + if (t < G) { s_m[t] = -FLT_MAX; s_l[t] = 0.0f; } + const int quarter = warp >> 1, cell = ((warp & 1) << 5) | lane; // scores + const int parity = t >> 7, d0 = 2 * (t & 127); // values + float acc0[G], acc1[G]; +#pragma unroll + for (int h = 0; h < G; ++h) acc0[h] = acc1[h] = 0.0f; + for (int c0 = 0; c0 < n_ids; c0 += PT) { + const int n_here = min(PT, n_ids - c0); + if (t < PT) { + long long r = -1; + if (t < n_here) { + const int id = ids[c0 + t]; + r = ((long long) p.page_table[id / page_size] * n_kv_heads + kvh) * page_size + id % page_size; + } + srow[t] = r; + } + __syncthreads(); + float s[G]; +#pragma unroll + for (int h = 0; h < G; ++h) s[h] = 0.0f; + if (cell < n_here) { + const long long row = srow[cell]; +#pragma unroll + for (int j = 0; j < 64; j += 16) { + const int d = quarter * 64 + j; + float kf[16]; + if constexpr (INT8) { + const uint4 raw = *reinterpret_cast(p.k_q + row * HD + d); + i8x4(raw.x, kf); i8x4(raw.y, kf + 4); i8x4(raw.z, kf + 8); i8x4(raw.w, kf + 12); + } else { + const uint4* src = reinterpret_cast(p.k_pool + row * HD + d); +#pragma unroll + for (int u = 0; u < 2; ++u) { + const uint4 raw = src[u]; + const __half2* h2 = reinterpret_cast(&raw); +#pragma unroll + for (int v = 0; v < 4; ++v) { + const float2 f = __half22float2(h2[v]); + kf[8 * u + 2 * v] = f.x; + kf[8 * u + 2 * v + 1] = f.y; + } + } + } +#pragma unroll + for (int h = 0; h < G; ++h) { + const float4* qh = reinterpret_cast(&sq[h][d]); + float a = s[h]; +#pragma unroll + for (int v = 0; v < 4; ++v) { + const float4 q4 = qh[v]; + a = fmaf(kf[4 * v], q4.x, a); + a = fmaf(kf[4 * v + 1], q4.y, a); + a = fmaf(kf[4 * v + 2], q4.z, a); + a = fmaf(kf[4 * v + 3], q4.w, a); + } + s[h] = a; + } + } + if constexpr (INT8) { + const float ks = __half2float(__ushort_as_half(p.k_scale[row * (HD / KV_Q8_GROUP) + quarter])); +#pragma unroll + for (int h = 0; h < G; ++h) s[h] *= ks; + } + } +#pragma unroll + for (int h = 0; h < G; ++h) spart[quarter][h][cell] = s[h]; + __syncthreads(); + // online softmax: warp w takes heads w and w + 8, a lane cells lane and lane + 32 + for (int h = warp; h < G; h += WARPS) { + float a = -FLT_MAX, b = -FLT_MAX; + if (lane < n_here) a = (spart[0][h][lane] + spart[1][h][lane] + spart[2][h][lane] + spart[3][h][lane]) * scale; + if (lane + 32 < n_here) + b = (spart[0][h][lane + 32] + spart[1][h][lane + 32] + spart[2][h][lane + 32] + spart[3][h][lane + 32]) * scale; + const float m_old = s_m[h]; + const float m_new = fmaxf(m_old, warp_max(fmaxf(a, b))); + const float ea = lane < n_here ? __expf(a - m_new) : 0.0f; + const float eb = lane + 32 < n_here ? __expf(b - m_new) : 0.0f; + sp[lane][h] = ea; + sp[lane + 32][h] = eb; + const float l = warp_sum(ea + eb); + if (lane == 0) { + const float alpha = __expf(m_old - m_new); + s_alpha[h] = alpha; + s_l[h] = fmaf(s_l[h], alpha, l); + s_m[h] = m_new; + } + } + __syncthreads(); +#pragma unroll + for (int h = 0; h < G; ++h) { + acc0[h] *= s_alpha[h]; + acc1[h] *= s_alpha[h]; + } + for (int c = parity; c < n_here; c += 2) { + const long long r = srow[c]; + float v0, v1; + if constexpr (INT8) { + const uint32_t w = *reinterpret_cast(p.v_q + r * HD + d0) ^ 0x8080u; + const float sc = __half2float(__ushort_as_half(p.v_scale[r * (HD / KV_Q8_GROUP) + d0 / KV_Q8_GROUP])); + v0 = (__int_as_float((int) __byte_perm(w, 0x4B000000u, 0x7650u)) - 8388736.0f) * sc; + v1 = (__int_as_float((int) __byte_perm(w, 0x4B000000u, 0x7651u)) - 8388736.0f) * sc; + } else { + const float2 f = __half22float2(*reinterpret_cast(p.v_pool + r * HD + d0)); + v0 = f.x; + v1 = f.y; + } + const float4* pc = reinterpret_cast(sp[c]); + const float4 pa = pc[0], pb = pc[1], pd = pc[2]; + const float pr[G] = {pa.x, pa.y, pa.z, pa.w, pb.x, pb.y, pb.z, pb.w, pd.x, pd.y, pd.z, pd.w}; +#pragma unroll + for (int h = 0; h < G; ++h) { + acc0[h] = fmaf(pr[h], v0, acc0[h]); + acc1[h] = fmaf(pr[h], v1, acc1[h]); + } + } + __syncthreads(); // srow, sp and s_alpha are rewritten by the next tile + } + // the odd cells' sums into shared memory, added to the even ones' + float* odd = &spart[0][0][0]; // G x HD floats + if (parity == 1) { +#pragma unroll + for (int h = 0; h < G; ++h) { + odd[h * HD + d0] = acc0[h]; + odd[h * HD + d0 + 1] = acc1[h]; + } + } + __syncthreads(); + if (parity == 0) { +#pragma unroll + for (int h = 0; h < G; ++h) { + const float l = s_l[h]; + attn[(size_t) h * HD + d0] = l > 0.0f ? (acc0[h] + odd[h * HD + d0]) / l : 0.0f; + attn[(size_t) h * HD + d0 + 1] = l > 0.0f ? (acc1[h] + odd[h * HD + d0 + 1]) / l : 0.0f; + } + } +} + } // namespace +void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, + const QsaShapes& s, float* attn, int64_t n_q, void* stream) { + if (n_q <= 0) return; + const bool int8 = pools.k_q != nullptr; + if (s.head_dim != HD || s.n_head != (int64_t) G * s.n_head_kv || cap <= 0 || !ids || !steps || !pools.page_table || + (int8 ? (!pools.v_q || !pools.k_scale || !pools.v_scale) : (!pools.k_pool || !pools.v_pool))) { + std::fprintf(stderr, "qsa_prefill_attn: unsupported geometry or missing buffers\n"); + std::exit(1); + } + const float scale = 1.0f / sqrtf((float) HD); + cudaStream_t st = (cudaStream_t) stream; + for (int64_t q0 = 0; q0 < n_q; q0 += 65535) { + const int64_t nb = n_q - q0 < 65535 ? n_q - q0 : 65535; + const dim3 grid((unsigned) s.n_head_kv, (unsigned) nb); + const size_t qo = (size_t) q0 * (size_t) s.n_head * HD; + if (int8) + attn_prefill_kernel<<>>(q + qo, pools, ids + (size_t) q0 * (size_t) cap, + steps + (size_t) q0 * kStepCount, (int) s.n_head_kv, + (int) s.page_size, scale, attn + qo, (int) cap); + else + attn_prefill_kernel<<>>(q + qo, pools, ids + (size_t) q0 * (size_t) cap, + steps + (size_t) q0 * kStepCount, (int) s.n_head_kv, + (int) s.page_size, scale, attn + qo, (int) cap); + } + const cudaError_t e = cudaGetLastError(); + if (e != cudaSuccess) { + std::fprintf(stderr, "qsa_prefill_attn: %s\n", cudaGetErrorString(e)); + std::exit(1); + } +} + void qsa_decode_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream) { if (n_q <= 0) return; diff --git a/src/kernels/cuda/qsa_select.cu b/src/kernels/cuda/qsa_select.cu index 6b13b2ff11d..269e348a3f1 100644 --- a/src/kernels/cuda/qsa_select.cu +++ b/src/kernels/cuda/qsa_select.cu @@ -29,24 +29,29 @@ __global__ void __launch_bounds__(SCORE_WARPS * 32) block_scores_kernel(const fl const int64_t qi = blockIdx.y; const int32_t* st = steps + qi * kStepCount; const int64_t n_kv = st[kStepNKv], n_bid = st[kStepNBid]; - const int64_t b = (int64_t) blockIdx.x * SCORE_WARPS + (threadIdx.x >> 5); - if (b > n_bid || b >= max_blocks) return; + const int64_t last = n_bid < max_blocks - 1 ? n_bid : max_blocks - 1; const int lane = threadIdx.x & 31; - const float* key = (b == n_bid) ? dead : pooled + b * IDX_DIM; - const float4 k4 = *reinterpret_cast(key + lane * 4); const float* q = q_idx + qi * IDX_HEADS * IDX_DIM + lane * 4; - float score = 0.0f; + float4 q4[IDX_HEADS]; #pragma unroll - for (int h = 0; h < IDX_HEADS; ++h) { - const float4 q4 = *reinterpret_cast(q + h * IDX_DIM); - float d = k4.x * q4.x + k4.y * q4.y + k4.z * q4.z + k4.w * q4.w; + for (int h = 0; h < IDX_HEADS; ++h) q4[h] = *reinterpret_cast(q + h * IDX_DIM); + // a warp per block; the grid covers the first blocks, and warps go round for the rest + for (int64_t b = (int64_t) blockIdx.x * SCORE_WARPS + (threadIdx.x >> 5); b <= last; + b += (int64_t) gridDim.x * SCORE_WARPS) { + const float* key = (b == n_bid) ? dead : pooled + b * IDX_DIM; + const float4 k4 = *reinterpret_cast(key + lane * 4); + float score = 0.0f; #pragma unroll - for (int o = 16; o > 0; o >>= 1) d += __shfl_xor_sync(0xffffffffu, d, o); - score += d > 0.0f ? d : 0.0f; - } - if (lane == 0) { - if (b == n_bid && n_kv % R != 0) score += 1e9f; - out[qi * max_blocks + b] = score; + for (int h = 0; h < IDX_HEADS; ++h) { + float d = k4.x * q4[h].x + k4.y * q4[h].y + k4.z * q4[h].z + k4.w * q4[h].w; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) d += __shfl_xor_sync(0xffffffffu, d, o); + score += d > 0.0f ? d : 0.0f; + } + if (lane == 0) { + if (b == n_bid && n_kv % R != 0) score += 1e9f; + out[qi * max_blocks + b] = score; + } } } @@ -151,13 +156,14 @@ __global__ void __launch_bounds__(TOPK_T) block_topk_kernel(const float* __restr } // namespace void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx, const int32_t* steps, int64_t nq, - int64_t max_blocks, const QsaShapes& s, float* scores, void* stream) { + int64_t max_blocks, const QsaShapes& s, float* scores, void* stream, int64_t grid_blocks) { if (nq <= 0) return; - if (s.idx_dim != IDX_DIM || s.idx_n_head != IDX_HEADS || s.idx_block != R || nq > 65535) { + if (s.idx_dim != IDX_DIM || s.idx_n_head != IDX_HEADS || s.idx_block != R || nq > 65535 || grid_blocks < 1) { std::fprintf(stderr, "qsa_block_scores: unsupported indexer geometry\n"); std::exit(1); } - const dim3 grid((unsigned) ((max_blocks + SCORE_WARPS - 1) / SCORE_WARPS), (unsigned) nq); + const int64_t covered = grid_blocks < max_blocks ? grid_blocks : max_blocks; + const dim3 grid((unsigned) ((covered + SCORE_WARPS - 1) / SCORE_WARPS), (unsigned) nq); block_scores_kernel<<>>(pooled, dead, q_idx, steps, max_blocks, scores); const cudaError_t e = cudaGetLastError(); diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index ae1aece1116..219351a3d14 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -3,7 +3,8 @@ // A verify window's token t must come out bit for bit as it would in a window of any other size (the drafts are // accepted exactly when greedy decode would have produced them), so every batched kernel is checked BITWISE against // the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. The -// prompt path's batched PLE arithmetic is held to the same standard. +// prompt path's batched PLE arithmetic is held to the same standard; its attention kernel, which sums in another +// order, is checked against the decode kernel within a tolerance. #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/fused_gr.hpp" @@ -12,6 +13,7 @@ #include "strata/kernels/native_ple_postops.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_rope.hpp" +#include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" @@ -505,6 +507,87 @@ int test_ple_tokens(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the prompt path's attention vs the decode kernel (INT8 and FP16 pools; 1 to 2051 selected cells) +int test_prefill_attn(std::mt19937& rng, cudaStream_t s) { + const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); + const int HD = (int) sh.head_dim, NH = (int) sh.n_head, NKV = (int) sh.n_head_kv; + const int cells = 4096, pages = cells / (int) sh.page_size; + const int64_t cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, sh); + const size_t n_codes = (size_t) cells * NKV * HD, n_scales = n_codes / strata::kernels::KV_Q8_GROUP; + std::vector table((size_t) pages); + for (int i = 0; i < pages; ++i) table[(size_t) i] = (i * 3 + 1) % pages; // a permuted page table + int32_t* d_table = dev(table.size()); + up(d_table, table); + auto half_bits = [&](float lo, float hi) { // a random positive FP16 in [lo, hi) + const float f = std::uniform_real_distribution(lo, hi)(rng); + uint32_t b; + std::memcpy(&b, &f, 4); + const int e = (int) ((b >> 23) & 0xFF) - 127 + 15; + return (uint16_t) ((e << 10) | ((b >> 13) & 0x3FF)); + }; + std::vector kq(n_codes), vq(n_codes); + for (auto* v : {&kq, &vq}) + for (auto& c : *v) c = (int8_t) ((int) (rng() % 255) - 127); + std::vector ks(n_scales), vs(n_scales), kh(n_codes), vh(n_codes); + for (auto& x : ks) x = half_bits(0.002f, 0.05f); + for (auto& x : vs) x = half_bits(0.002f, 0.05f); + for (auto& x : kh) x = (uint16_t) (half_bits(0.01f, 1.0f) | ((rng() & 1u) << 15)); + for (auto& x : vh) x = (uint16_t) (half_bits(0.01f, 1.0f) | ((rng() & 1u) << 15)); + int8_t *d_kq = dev(n_codes), *d_vq = dev(n_codes); + uint16_t *d_ks = dev(n_scales), *d_vs = dev(n_scales), *d_kh = dev(n_codes), + *d_vh = dev(n_codes); + up(d_kq, kq); up(d_vq, vq); up(d_ks, ks); up(d_vs, vs); up(d_kh, kh); up(d_vh, vh); + const std::vector widths = {1, 5, 64, 65, 700, 2051}; + const int n_q = (int) widths.size(); + std::vector ids((size_t) n_q * cap, 0), steps((size_t) n_q * strata::kernels::kStepCount, 0); + for (int i = 0; i < n_q; ++i) { + std::vector all(cells); + for (int c = 0; c < cells; ++c) all[(size_t) c] = c; + std::shuffle(all.begin(), all.end(), rng); + std::sort(all.begin(), all.begin() + widths[(size_t) i]); + std::copy(all.begin(), all.begin() + widths[(size_t) i], ids.begin() + (size_t) i * cap); + steps[(size_t) i * strata::kernels::kStepCount + strata::kernels::kStepWidth] = widths[(size_t) i]; + } + std::vector q((size_t) n_q * NH * HD); + for (auto& x : q) x = 3.0f * std::normal_distribution(0.0f, 1.0f)(rng); + int32_t *d_ids = dev(ids.size()), *d_steps = dev(steps.size()); + float *d_q = dev(q.size()), *d_a = dev(q.size()), *d_b = dev(q.size()); + float* d_scratch = dev((size_t) n_q * strata::kernels::qsa_decode_attn_scratch_floats(cap, sh)); + up(d_ids, ids); up(d_steps, steps); up(d_q, q); + int bad = 0; + double worst = 0.0; + for (int int8 = 0; int8 < 2; ++int8) { + strata::kernels::QsaAttnPools pools; + pools.page_table = d_table; + if (int8) { pools.k_q = d_kq; pools.v_q = d_vq; pools.k_scale = d_ks; pools.v_scale = d_vs; } + else { pools.k_pool = d_kh; pools.v_pool = d_vh; } + strata::kernels::qsa_decode_attn_batch(d_q, pools, d_ids, d_steps, cap, sh, d_scratch, d_a, n_q, s); + strata::kernels::qsa_prefill_attn(d_q, pools, d_ids, d_steps, cap, sh, d_b, n_q, s); + check(cudaStreamSynchronize(s), "prefill attn"); + const std::vector a = down(d_a, q.size()), b = down(d_b, q.size()); + for (int r = 0; r < n_q * NH; ++r) { // relative to the largest value of the head's output row + double big = 0.0, diff = 0.0; + for (int d = 0; d < HD; ++d) { + big = std::max(big, (double) std::fabs(a[(size_t) r * HD + d])); + diff = std::max(diff, (double) std::fabs(a[(size_t) r * HD + d] - b[(size_t) r * HD + d])); + } + const double rel = diff / std::max(big, 1e-30); + worst = std::max(worst, rel); + if (!(rel <= 2e-5)) { + if (bad < 5) std::fprintf(stderr, "prefill attn: %s pools, query %d head %d: relative difference %.3g\n", + int8 ? "INT8" : "FP16", r / NH, r % NH, rel); + ++bad; + } + } + } + for (void* p : {(void*) d_table, (void*) d_kq, (void*) d_vq, (void*) d_ks, (void*) d_vs, (void*) d_kh, (void*) d_vh, + (void*) d_ids, (void*) d_steps, (void*) d_q, (void*) d_a, (void*) d_b, (void*) d_scratch}) + cudaFree(p); + std::printf("prefill_attn: %s (INT8 and FP16 pools, 1-2051 cells; largest relative difference %.2g)\n", + bad ? "OUTSIDE TOLERANCE" : "within 2e-5 of the decode kernel", worst); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -526,6 +609,7 @@ int main(int argc, char** argv) { bad += test_kv_append(rng, s); bad += test_indexer_append(rng, s); bad += test_ple_tokens(rng, s); + bad += test_prefill_attn(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index e83dfc21756..ae8f6272e6f 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -91,8 +91,7 @@ struct Prefill::Impl { int32_t* sel_ids = nullptr; float* sel_scores = nullptr; // [sel_batch, max_blocks] int64_t sel_batch = 256, max_blocks = 0; - float* attn_scratch = nullptr; - int64_t attn_batch = 32, cap = 0; + int64_t cap = 0; // MoE float *logits = nullptr, *w = nullptr, *GU = nullptr, *Dm = nullptr, *sgate = nullptr, *sup = nullptr, *shared = nullptr, *sg = nullptr; @@ -172,7 +171,6 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co m.sel_ids = o.take(T * (size_t) m.cap, ok); m.max_blocks = ss.qsa_states[0].max_cells / s.idx_block + 2; m.sel_scores = o.take((size_t) m.sel_batch * (size_t) m.max_blocks, ok); - m.attn_scratch = o.take((size_t) m.attn_batch * strata::kernels::qsa_decode_attn_scratch_floats(m.cap, s), ok); m.logits = o.take(T * NE, ok); m.w = o.take(T * K, ok); m.ids = o.take(T * K, ok); m.slot_dev = o.take(T * K, ok); m.src_dev = o.take(T * K, ok); m.Xs = o.take(T * K * N, ok); m.GU = o.take(T * K * 1280, ok); @@ -213,7 +211,6 @@ uint64_t Prefill::bytes_needed(const core::ModelGeometry& g, const core::Session o.take(T * (size_t) cap, ok); const int64_t max_blocks = ss.qsa_states[0].max_cells / s.idx_block + 2; f(256 * (size_t) max_blocks); - f(32 * strata::kernels::qsa_decode_attn_scratch_floats(cap, s)); f(T * NE); f(T * K); o.take(T * K, ok); o.take(T * K, ok); o.take(T * K, ok); o.take(T * K * N, ok); f(T * K * 1280); o.take(T * K * 640, ok); f(T * K * N); f(T * 640); f(T * 640); o.take(T * 640, ok); f(T * N); f(T); @@ -402,8 +399,11 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e for (int64_t t0 = 0; t0 < T; t0 += m.sel_batch) { const int64_t nb = std::min(m.sel_batch, T - t0); const int32_t* steps0 = m.steps_dev + t0 * strata::kernels::kStepCount; + // the batch's last query has completed the most blocks + const int64_t grid_blocks = + m.steps_host[(size_t) (t0 + nb - 1) * strata::kernels::kStepCount + strata::kernels::kStepNBid] + 1; strata::kernels::qsa_block_scores(st.idx_pooled, st.idx_dead, m.q_idx + t0 * 512, steps0, nb, - m.max_blocks, s, m.sel_scores, m.cs); + m.max_blocks, s, m.sel_scores, m.cs, grid_blocks); strata::kernels::qsa_block_topk(m.sel_scores, steps0, nb, m.max_blocks, m.cap, s, m.sel_ids + t0 * m.cap, m.cs); } @@ -412,12 +412,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } mark(kPsQsaAttn); - for (int64_t t0 = 0; t0 < T; t0 += m.attn_batch) { - const int64_t nb = std::min(m.attn_batch, T - t0); - strata::kernels::qsa_decode_attn_batch(m.q + t0 * ZV, pools, m.sel_ids + t0 * m.cap, - m.steps_dev + t0 * strata::kernels::kStepCount, m.cap, s, - m.attn_scratch, m.attn + t0 * ZV, nb, m.cs); - } + strata::kernels::qsa_prefill_attn(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s, m.attn, T, m.cs); mark(kPsQsaOut); gate_attn(m.attn, m.Qf, m.attn_h, T, m.cs); if (!native_proj(m.gemm, wo, m.attn_h, m.bo, T, v.name("attn_output.weight"), err)) return false; From 40e1a7adbb7537f0bf2ecb494b41a3d884291404 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 01:28:07 +0200 Subject: [PATCH 19/92] Prompt path GDN: the conv over the whole chunk at once, a faster recurrence The conv + L2 norm walked the chunk one token at a time per channel (320 warps); the prompt path now uses the verify window's gdn_conv_l2_multi over all T tokens plus a small history update. The recurrence keeps the decode kernels' arithmetic (four 32-row partial sums per column, added in order) but puts a column's row groups in adjacent lanes, so the sums are shuffles; each thread holds two columns, a head is split over four blocks, and each token's inputs arrive through a cp.async ring 7 tokens ahead: one barrier per token. The output norm is a separate pass. verify_parity checks the prompt path's GDN bitwise against the verify window's kernels (1-37 tokens). 16K-token prompt: recurrence 1.36 -> 0.46 s; IQ3_XXS 15.9 -> 15.2 s, UD-Q4_K_XL 22.4 -> 21.7 s. --- CMakeLists.txt | 2 +- include/strata/kernels/verify_kernels.hpp | 3 +- include/strata/prefill/kernels.hpp | 12 +- include/strata/prefill/prefill.hpp | 2 +- src/kernels/cuda/verify_kernels.cu | 2 +- src/kernels/verify_parity.cpp | 84 ++++++++- src/prefill/kernels.cu | 220 ++++++++++++++-------- src/prefill/prefill.cpp | 11 +- 8 files changed, 248 insertions(+), 88 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 2f838b33507..b769cc8d1ad 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -364,7 +364,7 @@ if(STRATA_ENABLE_CUDA) # The verify window's batched kernels vs the per-token kernel sequences they replace, bitwise (synthetic data). add_executable(verify_parity src/kernels/verify_parity.cpp) target_include_directories(verify_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) - target_link_libraries(verify_parity PRIVATE strata_kernels CUDA::cudart) + target_link_libraries(verify_parity PRIVATE strata_kernels strata_prefill CUDA::cudart) add_test(NAME verify_parity COMMAND verify_parity --selftest) # Plan v0.3 P3: native MMVQ for 1..8 tokens (bitwise selftest + --bench timing); synthetic data, no model. diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index 31e3e813399..fb94397b43a 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -21,7 +21,8 @@ namespace strata::kernels { inline constexpr int kVerifyMaxT = 8; /// For token t of T: conv over [history(3) | qkv_0 .. qkv_t] -> SiLU -> L2 norm of the q/k heads -> h[t]. -/// `history` is NOT written. Bitwise `fused_gdn_conv_l2` per token. +/// `history` is NOT written. Bitwise `fused_gdn_conv_l2` per token. Up to 65535 tokens (the prompt path runs a +/// whole chunk through it). void gdn_conv_l2_multi(const float* history, const float* qkv, const float* conv_w, float* h, int channels, int qk_heads, float eps, int n_tok, void* stream, int t_begin = 0); /// history <- the last 3 entries of [history | qkv_0 .. qkv_{n-1}], n = *n_keep (0 leaves it as it was). diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index b70a2a5629e..5d56e9d5697 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -26,12 +26,14 @@ void gr_broadcast(const float* e, float* R, int64_t T, void* stream); /// gate[t,h] = softplus(ab[t,h] + dt[h]) * ssm_a[h]; beta[t,h] = sigmoid(ab[t, 48 + h]) (ab: [T, 96]) void gdn_gates(const float* ab, const float* dt, const float* ssm_a, float* gate, float* beta, int64_t T, void* stream); /// The 4-tap causal conv + SiLU over the chunk (history [C][3] in, updated to the chunk's last three inputs), then -/// the L2 norm of the q and k heads of every token. h: [T, C]. +/// the L2 norm of the q and k heads of every token: the verify window's gdn_conv_l2_multi over all T tokens at once. +/// h: [T, C]. void gdn_conv(float* history, const float* qkv, const float* conv_w, float* h, int64_t T, float eps, void* stream); -/// The recurrence over the chunk, block per value head, state in registers; y[t] = rmsnorm(o) * gamma * sigmoid(z) -/// (FP32 and FP16 bits: the out projection is quantized). -void gdn_recurrence(float* state, const float* h, const float* gate, const float* beta, const float* z, - const float* gamma, float eps, float* y, uint16_t* y16, int64_t T, void* stream); +/// The recurrence over the chunk (state in registers): y[t] = o (before the norm). With gdn_out_norm, bitwise +/// gdn_step_norm_multi over the tokens. +void gdn_scan(float* state, const float* h, const float* gate, const float* beta, float* y, int64_t T, void* stream); +/// In place: y[t] = rmsnorm(o) * gamma * sigmoid(z) per head, and its FP16 bits (the out projection is quantized). +void gdn_out_norm(float* y, const float* z, const float* gamma, float eps, uint16_t* y16, int64_t T, void* stream); // ---- MoE /// softmax over 512, top-10 (ties to the lower id), weights renormalised over the ten (the native router). diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index 0b3bf8e6e47..39619dba4a2 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -25,7 +25,7 @@ namespace strata::prefill { /// The sections `Prefill::profile` times, in the order a layer runs them. enum PrefillSection { - kPsPle, kPsHcRead, kPsGdn, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, + kPsPle, kPsHcRead, kPsGdn, kPsGdnConv, kPsGdnScan, kPsGdnNorm, kPsGdnOut, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, kPsExperts, kPsCombine, kPsCount }; const char* prefill_section_name(int section); diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index a5216da28c4..2eaa0c07548 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -381,7 +381,7 @@ void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream) { void gdn_conv_l2_multi(const float* history, const float* qkv, const float* conv_w, float* h, int channels, int qk_heads, float eps, int n_tok, void* stream, int t_begin) { - if (!history || !qkv || !conv_w || !h || channels % S != 0 || n_tok < 1 || n_tok > kVerifyMaxT) { + if (!history || !qkv || !conv_w || !h || channels % S != 0 || n_tok < 1 || n_tok > 65535) { std::fprintf(stderr, "gdn_conv_l2_multi: invalid arguments\n"); std::exit(1); } diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 219351a3d14..2560bdf5c15 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -3,8 +3,8 @@ // A verify window's token t must come out bit for bit as it would in a window of any other size (the drafts are // accepted exactly when greedy decode would have produced them), so every batched kernel is checked BITWISE against // the per-token kernels it stands in for, on random inputs that include -0.0, denormals and large values. The -// prompt path's batched PLE arithmetic is held to the same standard; its attention kernel, which sums in another -// order, is checked against the decode kernel within a tolerance. +// prompt path's batched PLE and GDN arithmetic is held to the same standard; its attention kernel, which sums in +// another order, is checked against the decode kernel within a tolerance. #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/fused_gr.hpp" @@ -16,6 +16,8 @@ #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" +#include "strata/kernels/verify_kernels.hpp" +#include "strata/prefill/kernels.hpp" #include @@ -588,6 +590,83 @@ int test_prefill_attn(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the prompt path's GDN conv and recurrence over a chunk vs the verify window's kernels, 8 tokens at a time +int test_prefill_gdn(std::mt19937& rng, cudaStream_t s) { + const int C = 10240, HK = 16, HV = 48, S = 128, ZV = HV * S; + const float eps = 1e-6f; + std::normal_distribution nd(0.0f, 1.0f); + auto uni = [&](float a, float b) { return std::uniform_real_distribution(a, b)(rng); }; + std::vector w((size_t) C * 4), hist((size_t) C * 3), state((size_t) S * HV * S), gamma((size_t) S); + for (auto& x : w) x = 0.5f * nd(rng); + for (auto& x : gamma) x = uni(0.5f, 1.5f); + float *d_w = dev(w.size()), *d_gamma = dev(gamma.size()); + up(d_w, w); + up(d_gamma, gamma); + int bad = 0; + for (int T : {1, 2, 3, 5, 37}) { + std::vector qkv((size_t) T * C), gate((size_t) T * HV), beta((size_t) T * HV), z((size_t) T * ZV); + for (auto& x : qkv) x = nd(rng); + for (auto& x : hist) x = nd(rng); + for (auto& x : state) x = 0.1f * nd(rng); + for (auto& x : gate) x = uni(-2.0f, 0.0f); + for (auto& x : beta) x = uni(0.0f, 1.0f); + for (auto& x : z) x = nd(rng); + float *d_qkv = dev(qkv.size()), *d_gate = dev(gate.size()), *d_beta = dev(beta.size()), + *d_z = dev(z.size()); + up(d_qkv, qkv); up(d_gate, gate); up(d_beta, beta); up(d_z, z); + float* d_hist[2]; + float* d_state[2]; + float* d_h[2]; + float* d_y[2]; + for (int p = 0; p < 2; ++p) { + d_hist[p] = dev(hist.size()); + d_state[p] = dev(state.size()); + d_h[p] = dev(qkv.size()); + d_y[p] = dev(z.size()); + up(d_hist[p], hist); + up(d_state[p], state); + } + uint16_t* d_y16 = dev(z.size()); + // the verify window's kernels, a window of up to 8 tokens after another, each committed + std::vector counts; + for (int t0 = 0; t0 < T; t0 += 8) counts.push_back(std::min(8, T - t0)); + int32_t* d_counts = dev(counts.size()); + up(d_counts, counts); + for (size_t i = 0; i < counts.size(); ++i) { + const size_t t0 = i * 8; + const int n = counts[i]; + strata::kernels::gdn_conv_l2_multi(d_hist[0], d_qkv + t0 * C, d_w, d_h[0] + t0 * C, C, 2 * HK, eps, n, s); + strata::kernels::gdn_conv_commit(d_hist[0], d_qkv + t0 * C, C, d_counts + i, s); + strata::kernels::gdn_step_norm_multi(d_state[0], d_h[0] + t0 * C, C, d_gate + t0 * HV, d_beta + t0 * HV, + d_z + t0 * ZV, d_gamma, eps, d_y[0] + t0 * ZV, HK, HV, n, d_counts + i, s); + } + // the prompt path, all T at once + strata::prefill::gdn_conv(d_hist[1], d_qkv, d_w, d_h[1], T, eps, s); + strata::prefill::gdn_scan(d_state[1], d_h[1], d_gate, d_beta, d_y[1], T, s); + strata::prefill::gdn_out_norm(d_y[1], d_z, d_gamma, eps, d_y16, T, s); + check(cudaStreamSynchronize(s), "prefill gdn"); + const int b = bitwise_diff(down(d_h[0], qkv.size()), down(d_h[1], qkv.size()), "conv output") + + bitwise_diff(down(d_y[0], z.size()), down(d_y[1], z.size()), "y") + + bitwise_diff(down(d_state[0], state.size()), down(d_state[1], state.size()), "state") + + bitwise_diff(down(d_hist[0], hist.size()), down(d_hist[1], hist.size()), "conv history"); + if (b) std::fprintf(stderr, "prefill gdn: T %d differs\n", T); + bad += b; + for (int p = 0; p < 2; ++p) { + cudaFree(d_hist[p]); + cudaFree(d_state[p]); + cudaFree(d_h[p]); + cudaFree(d_y[p]); + } + for (float* p : {d_qkv, d_gate, d_beta, d_z}) cudaFree(p); + cudaFree(d_y16); + cudaFree(d_counts); + } + cudaFree(d_w); + cudaFree(d_gamma); + std::printf("prefill_gdn: %s\n", bad ? "MISMATCH" : "bitwise equal to the verify window's kernels (1-37 tokens)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -610,6 +689,7 @@ int main(int argc, char** argv) { bad += test_indexer_append(rng, s); bad += test_ple_tokens(rng, s); bad += test_prefill_attn(rng, s); + bad += test_prefill_gdn(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index 29b29a85ec9..29113045902 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -1,10 +1,12 @@ // src/prefill/kernels.cu - see include/strata/prefill/kernels.hpp. #include "strata/prefill/kernels.hpp" #include "strata/kernels/mrope.hpp" +#include "strata/kernels/verify_kernels.hpp" #include #include +#include #include #include #include @@ -116,88 +118,145 @@ __global__ void gdn_gates_kernel(const float* __restrict__ ab, const float* __re gate[i] = (v > 20.0f ? v : log1pf(__expf(v))) * ssm_a[h]; beta[i] = sigm(ab[t * 2 * HV + HV + h]); } -// one thread per channel, walks the chunk; then a second kernel normalises -__global__ void gdn_conv_kernel(float* __restrict__ hist, const float* __restrict__ qkv, const float* __restrict__ w, - float* __restrict__ h, int64_t T) { +// The conv history after a chunk: its last three inputs, older ones moved up when the chunk is shorter. +__global__ void gdn_conv_hist_kernel(float* __restrict__ hist, const float* __restrict__ qkv, int64_t T) { const int c = blockIdx.x * blockDim.x + threadIdx.x; if (c >= C) return; - float v0 = hist[c * 3], v1 = hist[c * 3 + 1], v2 = hist[c * 3 + 2]; - const float w0 = w[c * 4], w1 = w[c * 4 + 1], w2 = w[c * 4 + 2], w3 = w[c * 4 + 3]; - for (int64_t t = 0; t < T; ++t) { - const float x = qkv[t * C + c]; - const float s = v0 * w0 + v1 * w1 + v2 * w2 + x * w3; - h[t * C + c] = s / (1.0f + __expf(-s)); - v0 = v1; v1 = v2; v2 = x; + const float old[3] = {hist[c * 3], hist[c * 3 + 1], hist[c * 3 + 2]}; +#pragma unroll + for (int r = 0; r < 3; ++r) { + const int64_t j = T - 3 + r; + hist[c * 3 + r] = j >= 0 ? qkv[j * C + c] : old[r + T]; } - hist[c * 3] = v0; hist[c * 3 + 1] = v1; hist[c * 3 + 2] = v2; -} -__global__ void gdn_l2_kernel(float* __restrict__ h, float eps) { - // block (t, head) over the 32 q/k heads, 128 threads - const int64_t t = blockIdx.y; - const int head = blockIdx.x; - float* x = h + t * C + head * S; - const float v = x[threadIdx.x]; - float sq = warp_sum(v * v); - __shared__ float part[4]; - if ((threadIdx.x & 31) == 0) part[threadIdx.x >> 5] = sq; - __syncthreads(); - const float ss = part[0] + part[1] + part[2] + part[3]; - x[threadIdx.x] = v * rsqrtf(ss + eps); -} -constexpr int RG = 4, RPG = S / RG; -__global__ void __launch_bounds__(S * RG) gdn_rec_kernel(float* __restrict__ state, const float* __restrict__ h, - const float* __restrict__ gate, - const float* __restrict__ beta, const float* __restrict__ z, - const float* __restrict__ gamma, float eps, - float* __restrict__ y, uint16_t* __restrict__ y16, int64_t T) { - __shared__ float sk[S], sq[S], red[RG][S], wsum[16]; - const int head = blockIdx.x, col = threadIdx.x, rg = threadIdx.y, tid = rg * S + col; +} +// cp.async of one 4-byte word into shared memory (sm_80+), and its group bookkeeping. +__device__ __forceinline__ void cp_async4(float* dst, const float* src) { + const unsigned d = (unsigned) __cvta_generic_to_shared(dst); + asm volatile("cp.async.ca.shared.global [%0], [%1], 4;\n" ::"r"(d), "l"(src)); +} +__device__ __forceinline__ void cp_async_commit() { asm volatile("cp.async.commit_group;\n" ::); } +template __device__ __forceinline__ void cp_async_wait() { asm volatile("cp.async.wait_group %0;\n" ::"n"(N)); } + +// The recurrence over the chunk, the arithmetic of gdn_step_norm_multi: a thread per (CPT columns, group of 32 +// state rows), a head's columns split over S / CB blocks. A column's four row groups sit in adjacent lanes, so +// both sums over the row groups are shuffles added in the decode kernels' order (group 0 + 1 + 2 + 3). Each +// token's q, k, v, gate and beta are copied (cp.async) into a ring of DEPTH token slots DEPTH - 1 tokens ahead, so +// the reads from memory overlap the steps before them: one barrier per token. Each thread reads its row group's +// 32 q and 32 k values into registers with 16-byte loads once for its CPT columns; a row group is padded to 36 +// words, so the four groups a warp reads at once sit in different banks. Writes o (before the norm) to `out`; +// gdn_norm_kernel finishes each token. +constexpr int RG = 4, RPG = S / RG, RP = RPG + 4, DEPTH = 8, SLOT = 2 * RG * RP; +template +__global__ void __launch_bounds__(CB / CPT * RG) gdn_scan_kernel(float* __restrict__ state, + const float* __restrict__ h, + const float* __restrict__ gate, + const float* __restrict__ beta, + float* __restrict__ out, int T) { + constexpr int NT = CB / CPT * RG, QL = (2 * S + NT - 1) / NT; // threads; q/k values each thread copies + static_assert(CB % (8 * CPT) == 0 && NT >= CB + 2, "8 column groups a warp; a thread each for v, gate, beta"); + __shared__ __align__(16) float sqk[DEPTH * SLOT]; // per slot: q, then k, [row group][row] + __shared__ float sv[DEPTH * CB], sgb[DEPTH * 2]; + const int head = blockIdx.x, tid = threadIdx.x, lane = tid & 31; + const int cl = ((tid >> 5) * 8 + (lane >> 2)) * CPT, rg = lane & 3, g0 = lane & ~3; // first local column const int qh = head % HK; - float s[RPG]; - float* base = state + ((size_t) (rg * RPG) * HV + head) * S + col; + // what this thread copies for every token: q/k value i = tid + j * NT < 2S (row i & 127 of q or k), then v of + // local column tid (tid < CB), the gate (tid == CB) or beta (tid == CB + 1) + int qk_off[QL], qk_at[QL]; +#pragma unroll + for (int j = 0; j < QL; ++j) { + const int i = tid + j * NT; + qk_off[j] = i < 2 * S ? (i < S ? 0 : HK * S) + qh * S + (i & (S - 1)) : -1; + qk_at[j] = (i >> 7) * RG * RP + ((i & (S - 1)) / RPG) * RP + (i & (RPG - 1)); + } + const int v_off = 2 * HK * S + head * S + blockIdx.y * CB + tid; + const float* gb = (tid == CB ? gate : beta) + head; + float s[CPT][RPG]; + float* base = state + ((size_t) (rg * RPG) * HV + head) * S + blockIdx.y * CB + cl; const size_t rs = (size_t) HV * S; #pragma unroll - for (int r = 0; r < RPG; ++r) s[r] = base[r * rs]; - const float g_col = gamma[col]; - for (int64_t t = 0; t < T; ++t) { - const float* ht = h + t * C; - __syncthreads(); - if (tid < S) { sq[tid] = ht[qh * S + tid]; sk[tid] = ht[HK * S + qh * S + tid]; } - __syncthreads(); - const float g = __expf(gate[t * HV + head]); - float kv = 0.0f; + for (int c = 0; c < CPT; ++c) #pragma unroll - for (int r = 0; r < RPG; ++r) kv = fmaf(s[r], sk[rg * RPG + r], kv); - red[rg][col] = kv; - __syncthreads(); - const float kv_col = red[0][col] + red[1][col] + red[2][col] + red[3][col]; - const float delta = (ht[2 * HK * S + head * S + col] - g * kv_col) * beta[t * HV + head]; - float o = 0.0f; + for (int r = 0; r < RPG; ++r) s[c][r] = base[r * rs + c]; + int next = 0; // the next token to copy + auto issue = [&]() { + if (next < T) { + const int slot = next & (DEPTH - 1); + const float* ht = h + (size_t) next * C; #pragma unroll - for (int r = 0; r < RPG; ++r) { - s[r] = fmaf(g, s[r], sk[rg * RPG + r] * delta); - o = fmaf(s[r], sq[rg * RPG + r], o); + for (int j = 0; j < QL; ++j) + if (qk_off[j] >= 0) cp_async4(&sqk[slot * SLOT + qk_at[j]], ht + qk_off[j]); + if (tid < CB) cp_async4(&sv[slot * CB + tid], ht + v_off); + else if (tid < CB + 2) cp_async4(&sgb[slot * 2 + tid - CB], gb + (size_t) next * HV); } - __syncthreads(); - red[rg][col] = o; - __syncthreads(); - float oc = 0.0f, sp = 0.0f; - if (rg == 0) { - oc = (red[0][col] + red[1][col] + red[2][col] + red[3][col]) * rsqrtf((float) S); - sp = oc * oc; + cp_async_commit(); + ++next; + }; + for (int t = 0; t < DEPTH - 1; ++t) issue(); + for (int t = 0; t < T; ++t) { + cp_async_wait(); + __syncthreads(); // token t's slot is filled; every thread is done with token t - 1's slot + issue(); + const int slot = t & (DEPTH - 1); + float qr[RPG], kr[RPG]; + const float4* q4 = reinterpret_cast(&sqk[slot * SLOT + rg * RP]); + const float4* k4 = reinterpret_cast(&sqk[slot * SLOT + RG * RP + rg * RP]); +#pragma unroll + for (int j = 0; j < RPG / 4; ++j) { + const float4 a = k4[j], b = q4[j]; + kr[4 * j] = a.x; kr[4 * j + 1] = a.y; kr[4 * j + 2] = a.z; kr[4 * j + 3] = a.w; + qr[4 * j] = b.x; qr[4 * j + 1] = b.y; qr[4 * j + 2] = b.z; qr[4 * j + 3] = b.w; } - sp = warp_sum(sp); - if ((tid & 31) == 0) wsum[tid >> 5] = sp; - __syncthreads(); - if (rg == 0) { - const float ss = wsum[0] + wsum[1] + wsum[2] + wsum[3]; - const float v = oc * rsqrtf(ss / (float) S + eps) * g_col * sigm(z[t * HV * S + head * S + col]); - y[t * HV * S + head * S + col] = v; - y16[t * HV * S + head * S + col] = hf(v); + const float g = __expf(sgb[slot * 2]), bt = sgb[slot * 2 + 1]; + float kv[CPT], o[CPT]; +#pragma unroll + for (int c = 0; c < CPT; ++c) kv[c] = o[c] = 0.0f; +#pragma unroll + for (int r = 0; r < RPG; ++r) +#pragma unroll + for (int c = 0; c < CPT; ++c) kv[c] = fmaf(s[c][r], kr[r], kv[c]); + float delta[CPT]; +#pragma unroll + for (int c = 0; c < CPT; ++c) { + const float k0 = __shfl_sync(0xffffffffu, kv[c], g0), k1 = __shfl_sync(0xffffffffu, kv[c], g0 + 1), + k2 = __shfl_sync(0xffffffffu, kv[c], g0 + 2), k3 = __shfl_sync(0xffffffffu, kv[c], g0 + 3); + const float kv_col = k0 + k1 + k2 + k3; + delta[c] = (sv[slot * CB + cl + c] - g * kv_col) * bt; + } +#pragma unroll + for (int r = 0; r < RPG; ++r) +#pragma unroll + for (int c = 0; c < CPT; ++c) { + s[c][r] = fmaf(g, s[c][r], kr[r] * delta[c]); + o[c] = fmaf(s[c][r], qr[r], o[c]); + } + float* ot = out + (size_t) t * HV * S + head * S + blockIdx.y * CB + cl; +#pragma unroll + for (int c = 0; c < CPT; ++c) { + const float o0 = __shfl_sync(0xffffffffu, o[c], g0), o1 = __shfl_sync(0xffffffffu, o[c], g0 + 1), + o2 = __shfl_sync(0xffffffffu, o[c], g0 + 2), o3 = __shfl_sync(0xffffffffu, o[c], g0 + 3); + if (rg == 0) ot[c] = (o0 + o1 + o2 + o3) * rsqrtf((float) S); } } + cp_async_wait<0>(); +#pragma unroll + for (int c = 0; c < CPT; ++c) #pragma unroll - for (int r = 0; r < RPG; ++r) base[r * rs] = s[r]; + for (int r = 0; r < RPG; ++r) base[r * rs + c] = s[c][r]; +} +// y = rmsnorm(o) * gamma * sigmoid(z) per (token, head), in place over o; also the FP16 image +__global__ void __launch_bounds__(S) gdn_norm_kernel(float* __restrict__ y, const float* __restrict__ z, + const float* __restrict__ gamma, float eps, + uint16_t* __restrict__ y16) { + __shared__ float wsum[S / 32]; + const size_t i = ((size_t) blockIdx.y * HV + blockIdx.x) * S + threadIdx.x; + const float oc = y[i]; + const float sp = warp_sum(oc * oc); + if ((threadIdx.x & 31) == 0) wsum[threadIdx.x >> 5] = sp; + __syncthreads(); + const float ss = wsum[0] + wsum[1] + wsum[2] + wsum[3]; + const float v = oc * rsqrtf(ss / (float) S + eps) * gamma[threadIdx.x] * sigm(z[i]); + y[i] = v; + y16[i] = hf(v); } // ---------------------------------------------------------------- MoE @@ -420,14 +479,25 @@ void gdn_gates(const float* ab, const float* dt, const float* ssm_a, float* gate check("gdn_gates"); } void gdn_conv(float* history, const float* qkv, const float* conv_w, float* h, int64_t T, float eps, void* stream) { - gdn_conv_kernel<<>>(history, qkv, conv_w, h, T); - gdn_l2_kernel<<>>(h, eps); + for (int64_t t0 = 0; t0 < T; t0 += 65535) + strata::kernels::gdn_conv_l2_multi(history, qkv, conv_w, h, C, 2 * HK, eps, (int) std::min(65535, T - t0), + stream, (int) t0); + gdn_conv_hist_kernel<<>>(history, qkv, T); check("gdn_conv"); } -void gdn_recurrence(float* state, const float* h, const float* gate, const float* beta, const float* z, - const float* gamma, float eps, float* y, uint16_t* y16, int64_t T, void* stream) { - gdn_rec_kernel<<>>(state, h, gate, beta, z, gamma, eps, y, y16, T); - check("gdn_recurrence"); +void gdn_scan(float* state, const float* h, const float* gate, const float* beta, float* y, int64_t T, void* stream) { + // 32 columns a block, 2 a thread (fastest of 16-128 columns, 1-2 a thread: 0.67 us a token on the 3090) + constexpr int CB = 32, CPT = 2; + gdn_scan_kernel<<>>(state, h, gate, beta, y, (int) T); + check("gdn_scan"); +} +void gdn_out_norm(float* y, const float* z, const float* gamma, float eps, uint16_t* y16, int64_t T, void* stream) { + for (int64_t t0 = 0; t0 < T; t0 += 65535) { + const int64_t n = std::min(65535, T - t0); + gdn_norm_kernel<<>>(y + t0 * HV * S, z + t0 * HV * S, gamma, + eps, y16 + t0 * HV * S); + } + check("gdn_out_norm"); } void route(const float* logits, int32_t* ids, float* weights, int64_t T, void* stream) { route_kernel<<<(unsigned) ((T + 7) / 8), 256, 0, (cudaStream_t) stream>>>(logits, ids, weights, T); diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index ae8f6272e6f..d04c3c51e05 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -356,8 +356,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e if (!bf16_proj(m.gemm, wa, m.mixed_bf, m.ab, T, v.name("ssm_alpha.weight"), err, 2 * HV)) return false; if (!bf16_proj(m.gemm, wb, m.mixed_bf, m.ab + HV, T, v.name("ssm_beta.weight"), err, 2 * HV)) return false; gdn_gates(m.ab, (const float*) wdt->data, (const float*) wsa->data, m.gate, m.beta, T, m.cs); + mark(kPsGdnConv); gdn_conv(conv, m.qkv, (const float*) wc->data, m.hbuf, T, EPS, m.cs); - gdn_recurrence(state, m.hbuf, m.gate, m.beta, m.z, (const float*) wnm->data, EPS, m.y, m.y_h, T, m.cs); + mark(kPsGdnScan); + gdn_scan(state, m.hbuf, m.gate, m.beta, m.y, T, m.cs); + mark(kPsGdnNorm); + gdn_out_norm(m.y, m.z, (const float*) wnm->data, EPS, m.y_h, T, m.cs); + mark(kPsGdnOut); if (!native_proj(m.gemm, wo, m.y_h, m.bo, T, v.name("ssm_out.weight"), err)) return false; ++gdn_index; } else if (half == 0) { @@ -562,7 +567,9 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const char* prefill_section_name(int section) { static const char* const names[kPsCount] = { - "PLE block", "HC read (mixer)", "GDN mixer", "QSA projections, norms, KV", "QSA indexer appends", + "PLE block", "HC read (mixer)", "GDN projections", "GDN conv", "GDN recurrence", "GDN output norm", + "GDN output projection", + "QSA projections, norms, KV", "QSA indexer appends", "QSA block scores + top-k", "QSA attention", "QSA gate + output", "HC write + HC read (FFN)", "router, shared, grouping", "experts", "combine + HC write"}; return section >= 0 && section < kPsCount ? names[section] : "?"; From f8de703c249456b085eea6126f95735aa637b062 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 02:24:53 +0200 Subject: [PATCH 20/92] Prompt path: the experts the main GPU's cache lacks run on the second GPU A prompt chunk uses nearly every expert, and the 3090 streamed all it did not hold over its PCIe 4.0 x4 link. With --second-gpu the prompt path now hands them to the second GPU (x16): per layer the 3090 copies the chunk's MoE input (FP16) to pinned host memory while its router and shared expert run; the second GPU computes its experts from its own cache or streamed from the pinned arena, and returns each token's weighted sum (FP32), which the 3090's combine adds to its cached experts' rows (prefill/offload.cpp). Experts go through in batches of consecutive ids (up to 16 experts and 4096 rows): one staging copy per run of adjacent arena blobs, one dequant launch (iq_dequant_experts_f16), one gather, per-expert GEMMs (cuBLAS does not group FP16 -> FP32), one SwiGLU and one ordered per-token add (moe_gather_add, the same sums as one moe_scatter_add per expert). --no-prompt-offload streams them to the main GPU as before. 16K-token prompt, UD-Q4_K_XL: 21.6 -> 11.9 s. --- CMakeLists.txt | 3 +- include/strata/kernels/iq_kernels.hpp | 4 + include/strata/prefill/gemm.hpp | 7 + include/strata/prefill/kernels.hpp | 11 +- include/strata/prefill/offload.hpp | 61 +++++ include/strata/prefill/prefill.hpp | 5 + src/kernels/cuda/iq_kernels.cu | 32 +++ src/prefill/gemm.cu | 22 ++ src/prefill/kernels.cu | 49 +++- src/prefill/offload.cpp | 360 ++++++++++++++++++++++++++ src/prefill/prefill.cpp | 63 ++++- src/program/generate.cpp | 29 +++ 12 files changed, 634 insertions(+), 12 deletions(-) create mode 100644 include/strata/prefill/offload.hpp create mode 100644 src/prefill/offload.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index b769cc8d1ad..48801a6632c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -249,7 +249,8 @@ if(STRATA_ENABLE_CUDA) # what composes them into a token. No tokenizer - the prompt arrives as ids - because Gate C1 compares logits # against llama.cpp on the SAME ids and a tokenizer on one side of that comparison is a second variable. # plan v0.3 P5: batched prompt processing (tensor-core GEMMs through cuBLAS). - add_library(strata_prefill STATIC src/prefill/gemm.cu src/prefill/kernels.cu src/prefill/prefill.cpp) + add_library(strata_prefill STATIC src/prefill/gemm.cu src/prefill/kernels.cu src/prefill/offload.cpp + src/prefill/prefill.cpp) target_include_directories(strata_prefill PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) target_link_libraries(strata_prefill PUBLIC strata_engine CUDA::cublas CUDA::cudart) diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index 8b1baf6955a..e083771d2bd 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -34,6 +34,10 @@ void iq_embed_rows(int ggml_type, const void* table, size_t row_bytes, const int /// uses: row 2r = gate row r, row 2r+1 = up row r. void iq_dequant_gu_f16(int ggml_type, const void* gate, const void* up, int64_t n_ff, int64_t n_embd, uint16_t* dst, void* stream); +/// iq_dequant_gu_f16 and iq_dequant_f16 (the down matrix) of n experts in one launch: expert i's blob at blobs[i] +/// (a DEVICE array), its gate/up into gu + i * 2 * n_ff * n_embd and its down into dn + i * n_embd * n_ff. +void iq_dequant_experts_f16(int gu_type, int d_type, const uint8_t* const* blobs, int n, size_t up_off, size_t down_off, + int64_t n_ff, int64_t n_embd, uint16_t* gu, uint16_t* dn, void* stream); /// The layout of one native expert blob: [gate rows | up rows | down rows], raw GGUF blocks. struct NativeExpertLayout { diff --git a/include/strata/prefill/gemm.hpp b/include/strata/prefill/gemm.hpp index b0aca482b7c..daa6582bc7e 100644 --- a/include/strata/prefill/gemm.hpp +++ b/include/strata/prefill/gemm.hpp @@ -32,6 +32,12 @@ class Gemm { void f16(const uint16_t* X, const uint16_t* W, float* Y, int64_t T, int64_t N, int64_t K, int64_t ldy = 0, float beta = 0.0f); + /// Y_i[rows[i], N] = X_i[rows[i], K] . W_i[N, K]^T (FP16 in, FP32 out) for n problems of one shape but their + /// own row counts, in one cuBLAS grouped call; the pointer arrays are DEVICE memory, `rows` host memory. Where + /// cuBLAS does not group these types it runs f16() per problem, from the host copies of the pointers. + void f16_grouped(const uint16_t* const* X, const uint16_t* const* W, float* const* Y, const uint16_t* const* X_host, + const uint16_t* const* W_host, float* const* Y_host, const int* rows, int n, int64_t N, int64_t K); + /// W given as native GGUF blocks of `ggml_type`, dequantized to FP16 in the scratch, X in FP16. void native(const uint16_t* X, int ggml_type, const void* W_blocks, float* Y, int64_t T, int64_t N, int64_t K, int64_t ldy = 0, float beta = 0.0f); @@ -47,6 +53,7 @@ class Gemm { int64_t scratch_elems_ = 0; void* workspace_ = nullptr; bool external_ = false; + bool grouped_ = true; // cuBLAS groups FP16 -> FP32 GEMMs (cleared at the first refusal) }; diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index 5d56e9d5697..355f9af58be 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -46,9 +46,16 @@ void swiglu_interleaved(const float* gu, uint16_t* h16, int64_t n, void* stream) void swiglu_pair(const float* g, const float* u, uint16_t* h16, int64_t n, void* stream); /// Gather rows: dst16[i, :] = x16[src[i], :] (n rows of `width` BF16). void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int64_t n, int64_t width, void* stream); -/// bo[t, :] = shared[t, :] * sigmoid(sg[t]) + sum_k w[t, k] * D[slot[t, k], :] +/// bo[t, :] = sum_k w[t, k] * D[slot[t, k], :] + shared[t, :] * sigmoid(sg[t]) (+ partial[t, :]); a slot of -1 is +/// an entry the second GPU computed, whose sum `partial` holds. void moe_combine(const float* D, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo, - int64_t T, void* stream); + int64_t T, void* stream, const float* partial = nullptr); +/// sum[src[r], :] += w[r] * rows[r, :] for n rows (one expert's rows: no token twice). +void moe_scatter_add(float* sum, const float* rows, const float* w, const int32_t* src, int64_t n, void* stream); +/// The same for several experts' rows (row r at rows[r - r0]), where a token may recur: token tok[b] adds its rows +/// list[start[b] .. start[b + 1]) in that order, so it matches moe_scatter_add per expert in the list's order. +void moe_gather_add(float* sum, const float* rows, int64_t r0, const float* w, const int32_t* tok, const int32_t* start, + const int32_t* list, int64_t n_tok, void* stream); // ---- QSA helpers /// In place: x[r, :] = x[r, :] * rsqrt(mean x^2 + eps) * w over rows of `cols` (row stride `ld`). diff --git a/include/strata/prefill/offload.hpp b/include/strata/prefill/offload.hpp new file mode 100644 index 00000000000..6b13004c3de --- /dev/null +++ b/include/strata/prefill/offload.hpp @@ -0,0 +1,61 @@ +// include/strata/prefill/offload.hpp - the prompt path's experts on a second GPU. +// +// A prompt chunk uses nearly every expert of every layer, and the first GPU (the 3090, on a PCIe 4.0 x4 link) has +// to stream all the ones its cache does not hold: ~0.65 GB (IQ3_XXS) to ~1.4 GB (UD-Q4_K_XL) a layer at 6.4 GB/s. +// The second GPU sits on a x16 link. Per layer the first GPU copies the chunk's MoE input (FP16) to pinned host +// memory while its router and shared expert run; the second GPU copies it in, computes each of its experts - +// from its own cache when it holds them, else streamed from the pinned arena through a ring of staging slots - +// and adds each expert's weighted rows into a per-token FP32 sum, in ascending expert order. The sum returns +// through pinned host memory to the first GPU, which computes the experts its own cache holds meanwhile and adds +// the sum in its combine. Neither GPU reads the other's memory: Windows gives GeForce cards no peer access. +#pragma once + +#include "strata/core/expert_cache.hpp" +#include "strata/core/expert_source.hpp" + +#include + +#include +#include +#include +#include + +namespace strata::prefill { + +class Offload { +public: + Offload(); + ~Offload(); + Offload(const Offload&) = delete; + Offload& operator=(const Offload&) = delete; + + /// Buffers for chunks of up to `chunk` tokens on `device`; `main_device` is made current again after every + /// call. `cache`/`res`: the device's expert cache and its residency (n_layers x n_expert slots, -1 when not + /// held), or null. `borrow`/`borrow_bytes`: device memory for the buffers (lent cache slots), else allocated. + bool init(int device, int main_device, core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* res, + int64_t n_expert, int64_t chunk, std::string& err, void* borrow = nullptr, uint64_t borrow_bytes = 0); + /// Device bytes `init` needs for chunks of `chunk` tokens (what a borrowed region must hold). + static uint64_t bytes_needed(int64_t chunk, int64_t n_expert); + + /// Pinned [chunk, n_embd]: the first GPU writes the MoE input (FP16) here, and reads the sums (FP32) here. + uint16_t* host_input() const; + float* host_sum() const; + + /// Layer `layer` of a chunk of T tokens: expert experts[j] serves rows [off[j], off[j + 1]), row r being token + /// src[r] with routing weight w[r]. Queues the work behind `input_ready` (an event of the first GPU, recorded + /// once host_input holds the input) and returns; done() is recorded once host_sum holds the sums. + bool run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, + const std::vector& src, const std::vector& w, cudaEvent_t input_ready, std::string& err); + cudaEvent_t done() const; + + int64_t experts_resident = 0; ///< expert-layer groups served from its cache + int64_t experts_streamed = 0; ///< expert blobs copied from the arena + double ms_host = 0; ///< host time queuing its work + double ms_gpu = 0; ///< its GPU time from the input's arrival (all layers but the last) + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace strata::prefill diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index 39619dba4a2..0ca46551cf2 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -23,6 +23,8 @@ namespace strata::prefill { +class Offload; + /// The sections `Prefill::profile` times, in the order a layer runs them. enum PrefillSection { kPsPle, kPsHcRead, kPsGdn, kPsGdnConv, kPsGdnScan, kPsGdnNorm, kPsGdnOut, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, @@ -78,6 +80,9 @@ class Prefill { /// one after every chunk). bool profile = false; + /// The experts this GPU's cache does not hold, computed on a second GPU instead of streamed here; null: here. + Offload* offload = nullptr; + private: struct Impl; std::unique_ptr impl_; diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index c32ed429246..6bcfd6000cb 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -937,6 +937,24 @@ __global__ void dequant_gu_kernel(int ty, const void* __restrict__ gate, const v dq_dispatch<__half>(ty, parity ? up : gate, i, y + ((2 * r + parity) * per_row + c) * QK_K, threadIdx.x); } +// several experts at once, expert z's blob at blobs[z]: first its gate/up superblocks (as dequant_gu_kernel), then +// its down superblocks (as dequant_flat_kernel) +__global__ void dequant_experts_kernel(int gu_ty, int d_ty, const uint8_t* const* __restrict__ blobs, size_t up_off, + size_t down_off, int64_t per_row, int64_t gu_blocks, __half* __restrict__ gu, + __half* __restrict__ dn, int64_t gu_elems, int64_t dn_elems) { + const uint8_t* b = blobs[blockIdx.z]; + const int64_t i = blockIdx.x; + if (i < 2 * gu_blocks) { + const int parity = (int) (i / gu_blocks); + const int64_t j = i % gu_blocks, r = j / per_row, c = j % per_row; + dq_dispatch<__half>(gu_ty, parity ? b + up_off : b, j, + gu + blockIdx.z * gu_elems + ((2 * r + parity) * per_row + c) * QK_K, threadIdx.x); + } else { + const int64_t j = i - 2 * gu_blocks; + dq_dispatch<__half>(d_ty, b + down_off, j, dn + blockIdx.z * dn_elems + j * QK_K, threadIdx.x); + } +} + // the formats dq_dispatch dequantizes bool is_iq(int t) { return t == 16 || t == 17 || t == 18 || t == 20 || t == 21 || t == 22 || t == 23 || t == 29 || t == 42 || t == 11 || @@ -1029,6 +1047,20 @@ void iq_dequant_gu_f16(int t, const void* gate, const void* up, int64_t n_ff, in check("iq_dequant_gu_f16"); } +void iq_dequant_experts_f16(int gu_type, int d_type, const uint8_t* const* blobs, int n, size_t up_off, size_t down_off, + int64_t n_ff, int64_t n_embd, uint16_t* gu, uint16_t* dn, void* stream) { + if (n <= 0) return; + if (n_embd % 256 != 0 || (n_embd * n_ff) % 256 != 0 || !is_iq(gu_type) || !is_iq(d_type) || n > 65535) { + std::fprintf(stderr, "iq_dequant_experts_f16: bad arguments\n"); + std::exit(1); + } + const int64_t per_row = n_embd / 256, gu_blocks = n_ff * per_row, d_blocks = n_embd * n_ff / 256; + dequant_experts_kernel<<>>( + gu_type, d_type, blobs, up_off, down_off, per_row, gu_blocks, (__half*) gu, (__half*) dn, 2 * n_ff * n_embd, + n_embd * n_ff); + check("iq_dequant_experts_f16"); +} + bool native_expert_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept { const int qg = fmt_qk(gu_type), qd = fmt_qk(d_type); return qg > 0 && qd > 0 && is_iq(gu_type) && is_iq(d_type) && n_embd % qg == 0 && n_ff % qd == 0; diff --git a/src/prefill/gemm.cu b/src/prefill/gemm.cu index 2ecca2c6eaf..ccc4d35c58a 100644 --- a/src/prefill/gemm.cu +++ b/src/prefill/gemm.cu @@ -7,6 +7,7 @@ #include #include +#include namespace strata::prefill { namespace { @@ -86,6 +87,27 @@ void Gemm::f16(const uint16_t* X, const uint16_t* W, float* Y, int64_t T, int64_ "cublasGemmEx f16"); } +void Gemm::f16_grouped(const uint16_t* const* X, const uint16_t* const* W, float* const* Y, const uint16_t* const* X_host, + const uint16_t* const* W_host, float* const* Y_host, const int* rows, int n, int64_t N, int64_t K) { + if (n <= 0) return; + if (grouped_) { + // column-major, as f16(): Y_i^T[N, rows] = W_i (K x N col-major, transposed) . X_i^T[K, rows]; a group per problem + const std::vector ta((size_t) n, CUBLAS_OP_T), tb((size_t) n, CUBLAS_OP_N); + const std::vector m((size_t) n, (int) N), k((size_t) n, (int) K), ld_wx((size_t) n, (int) K), + ld_y((size_t) n, (int) N), size((size_t) n, 1); + const std::vector alpha((size_t) n, 1.0f), beta((size_t) n, 0.0f); + const cublasStatus_t st = cublasGemmGroupedBatchedEx( + (cublasHandle_t) handle_, ta.data(), tb.data(), m.data(), rows, k.data(), alpha.data(), (const void* const*) W, + CUDA_R_16F, ld_wx.data(), (const void* const*) X, CUDA_R_16F, ld_wx.data(), beta.data(), (void* const*) Y, + CUDA_R_32F, ld_y.data(), n, size.data(), CUBLAS_COMPUTE_32F); + if (st == CUBLAS_STATUS_SUCCESS) return; + if (st != CUBLAS_STATUS_NOT_SUPPORTED && st != CUBLAS_STATUS_INVALID_VALUE) ck(st, "cublasGemmGroupedBatchedEx"); + grouped_ = false; + std::fprintf(stderr, "prefill gemm: cuBLAS does not group FP16 -> FP32 GEMMs here; one call per expert\n"); + } + for (int i = 0; i < n; ++i) f16(X_host[i], W_host[i], Y_host[i], rows[i], N, K); +} + void Gemm::native(const uint16_t* X, int ggml_type, const void* W_blocks, float* Y, int64_t T, int64_t N, int64_t K, int64_t ldy, float beta) { if (N * K > scratch_elems_) { diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index 29113045902..1021637d548 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -348,14 +348,27 @@ __global__ void gather_rows16_kernel(const uint16_t* __restrict__ x, const int32 } __global__ void moe_combine_kernel(const float* __restrict__ Dm, const int32_t* __restrict__ slot, const float* __restrict__ w, const float* __restrict__ shared, - const float* __restrict__ sg, float* __restrict__ bo, int64_t T) { + const float* __restrict__ sg, const float* __restrict__ partial, + float* __restrict__ bo, int64_t T) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= T * N) return; const int64_t t = i / N, d = i % N; float s = 0.0f; #pragma unroll - for (int k = 0; k < 10; ++k) s = fmaf(w[t * 10 + k], Dm[(int64_t) slot[t * 10 + k] * N + d], s); - bo[i] = s + shared[i] * sigm(sg[t]); + for (int k = 0; k < 10; ++k) { + const int32_t r = slot[t * 10 + k]; + if (r >= 0) s = fmaf(w[t * 10 + k], Dm[(int64_t) r * N + d], s); + } + const float v = s + shared[i] * sigm(sg[t]); + bo[i] = partial ? v + partial[i] : v; +} +__global__ void moe_scatter_add_kernel(float* __restrict__ sum, const float* __restrict__ rows, + const float* __restrict__ w, const int32_t* __restrict__ src, int64_t n) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (i >= n * N) return; + const int64_t r = i / N, d = i % N; + float* o = sum + (int64_t) src[r] * N + d; + *o = fmaf(w[r], rows[i], *o); } // ---------------------------------------------------------------- QSA helpers @@ -526,10 +539,36 @@ void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int check("gather_rows16"); } void moe_combine(const float* Dm, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo, - int64_t T, void* stream) { - moe_combine_kernel<<>>(Dm, slot, w, shared, sg, bo, T); + int64_t T, void* stream, const float* partial) { + moe_combine_kernel<<>>(Dm, slot, w, shared, sg, partial, bo, T); check("moe_combine"); } +__global__ void moe_gather_add_kernel(float* __restrict__ sum, const float* __restrict__ rows, int64_t r0, + const float* __restrict__ w, const int32_t* __restrict__ tok, + const int32_t* __restrict__ start, const int32_t* __restrict__ list) { + const int64_t d = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (d >= N) return; + const int b = blockIdx.y; + float* o = sum + (int64_t) tok[b] * N + d; + float s = *o; + for (int i = start[b]; i < start[b + 1]; ++i) { + const int r = list[i]; + s = fmaf(w[r], rows[(r - r0) * N + d], s); + } + *o = s; +} +void moe_gather_add(float* sum, const float* rows, int64_t r0, const float* w, const int32_t* tok, const int32_t* start, + const int32_t* list, int64_t n_tok, void* stream) { + if (n_tok <= 0) return; + moe_gather_add_kernel<<>>( + sum, rows, r0, w, tok, start, list); + check("moe_gather_add"); +} +void moe_scatter_add(float* sum, const float* rows, const float* w, const int32_t* src, int64_t n, void* stream) { + if (n <= 0) return; + moe_scatter_add_kernel<<>>(sum, rows, w, src, n); + check("moe_scatter_add"); +} void rms_rows(float* x, const float* w, int64_t rows, int64_t cols, int64_t ld, float eps, void* stream) { if (rows <= 0) return; rms_rows_kernel<<<(unsigned) rows, 256, 0, (cudaStream_t) stream>>>(x, w, cols, ld, eps); diff --git a/src/prefill/offload.cpp b/src/prefill/offload.cpp new file mode 100644 index 00000000000..6b6653fb17f --- /dev/null +++ b/src/prefill/offload.cpp @@ -0,0 +1,360 @@ +// src/prefill/offload.cpp - see include/strata/prefill/offload.hpp. +#include "strata/prefill/offload.hpp" + +#include "strata/kernels/cpu/expert_layout.hpp" +#include "strata/kernels/iq_kernels.hpp" +#include "strata/prefill/gemm.hpp" +#include "strata/prefill/kernels.hpp" + +#include +#include +#include + +namespace strata::prefill { +namespace { + +constexpr int64_t N = 2560, FF2 = 1280, FF = 640; +// Experts go through in batches of consecutive ids, at most G experts and ROWS rows (an expert with more rows is a +// batch of its own, in pieces of ROWS): one staging copy per run of adjacent arena blobs, one dequant launch, one +// gather, two grouped GEMMs, one SwiGLU and one scatter-add per batch. Per-expert launches made the host, not the +// GPU, the limit (~10 calls for each of ~500 experts a layer). +constexpr int G = 16; +constexpr int64_t ROWS = 4096; +constexpr int SLOTS = 2; // batches in staging: the copies of one overlap the work on the other +constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace +constexpr int NPTR = 7; // per-expert device pointers: blob, xs, gate/up, gu, h, down, d + +struct DeviceScope { // the second GPU current for one call, the main one again afterwards + int main; + DeviceScope(int dev, int main_dev) : main(main_dev) { cudaSetDevice(dev); } + ~DeviceScope() { cudaSetDevice(main); } +}; + +// Bump allocation from a borrowed region, else cudaMalloc; with neither it only counts (bytes_needed). +struct Alloc { + uint8_t* base = nullptr; + uint64_t cap = 0, used = 0; + bool count_only = false; + std::vector* owned = nullptr; + template T* take(size_t n, bool& ok) { + const uint64_t bytes = ((uint64_t) n * sizeof(T) + 255) & ~255ull; + if (count_only) { used += bytes; return nullptr; } + if (base != nullptr) { + if (used + bytes > cap) { ok = false; return nullptr; } + T* p = (T*) (base + used); + used += bytes; + return p; + } + void* p = nullptr; + if (cudaMalloc(&p, bytes) != cudaSuccess) { ok = false; return nullptr; } + owned->push_back(p); + used += bytes; + return (T*) p; + } +}; + +} // namespace + +struct Offload::Impl { + int dev = -1, main = 0; + core::ExpertSource* src = nullptr; + const core::ExpertCache* cache = nullptr; + const int32_t* res = nullptr; + int64_t n_expert = 0, chunk = 0; + size_t blob_cap = 0; // staging bytes per expert + cudaStream_t s = nullptr, copy = nullptr; + cudaEvent_t done = nullptr, copied[SLOTS] = {}, used[SLOTS] = {}; + cudaEvent_t t0 = nullptr, t1 = nullptr; // the previous layer's GPU time, read at the next one + bool timed = false, live[SLOTS] = {}; + Gemm gemm; + std::vector owned; + uint16_t *input = nullptr, *xs = nullptr, *hh = nullptr, *dq_gu = nullptr, *dq_d = nullptr; + float *sum = nullptr, *w = nullptr, *gu = nullptr, *d = nullptr; + int32_t* rows_src = nullptr; + int32_t* adds = nullptr; // per batch: its tokens, their row lists' starts, the row lists (moe_gather_add) + std::vector adds_host, mark, first, count; + uint8_t* stage[SLOTS] = {}; + void** ptrs = nullptr; // NPTR arrays of n_expert device pointers + std::vector ptrs_host; + uint16_t* h_input = nullptr; + float* h_sum = nullptr; + + // the same sequence counted (bytes_needed) or carved + bool carve(Alloc& a, int64_t T) { + bool ok = true; + const size_t t = (size_t) T; + input = a.take(t * N, ok); + sum = a.take(t * N, ok); + rows_src = a.take(t * 10, ok); + w = a.take(t * 10, ok); + adds = a.take(t * 10 * 3 + (size_t) n_expert * 2, ok); + xs = a.take((size_t) ROWS * N, ok); + gu = a.take((size_t) ROWS * FF2, ok); + hh = a.take((size_t) ROWS * FF, ok); + d = a.take((size_t) ROWS * N, ok); + dq_gu = a.take((size_t) G * FF2 * N, ok); + dq_d = a.take((size_t) G * N * FF, ok); + for (auto& p : stage) p = a.take((size_t) G * blob_cap, ok); + ptrs = a.take((size_t) NPTR * (size_t) n_expert, ok); + void* ws = a.take(GEMM_WS, ok); + if (ok && !a.count_only) { + std::string err; + ok = gemm.init_external(s, nullptr, 0, ws, GEMM_WS, err); + } + return ok; + } +}; + +Offload::Offload() : impl_(new Impl) {} + +Offload::~Offload() { + Impl& m = *impl_; + if (m.dev < 0) return; + DeviceScope scope(m.dev, m.main); + if (m.s) cudaStreamSynchronize(m.s); + if (m.copy) cudaStreamSynchronize(m.copy); + for (int i = 0; i < SLOTS; ++i) { + if (m.copied[i]) cudaEventDestroy(m.copied[i]); + if (m.used[i]) cudaEventDestroy(m.used[i]); + } + for (cudaEvent_t e : {m.done, m.t0, m.t1}) if (e) cudaEventDestroy(e); + if (m.h_input) cudaFreeHost(m.h_input); + if (m.h_sum) cudaFreeHost(m.h_sum); + for (void* p : m.owned) cudaFree(p); + if (m.copy) cudaStreamDestroy(m.copy); + if (m.s) cudaStreamDestroy(m.s); +} + +uint64_t Offload::bytes_needed(int64_t chunk, int64_t n_expert) { + Impl m; + m.n_expert = n_expert; + m.blob_cap = ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; + Alloc a; + a.count_only = true; + m.carve(a, chunk); + return a.used + (1u << 20); +} + +bool Offload::init(int device, int main_device, core::ExpertSource* src, const core::ExpertCache* cache, + const int32_t* res, int64_t n_expert, int64_t chunk, std::string& err, void* borrow, + uint64_t borrow_bytes) { + Impl& m = *impl_; + if (!strata::kernels::cpu::expert_layout().native) { + err = "prompt offload: needs a native pack"; + return false; + } + m.dev = device; + m.main = main_device; + m.src = src; + m.cache = cache; + m.res = res; + m.n_expert = n_expert; + m.chunk = chunk; + m.blob_cap = ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; + m.ptrs_host.resize((size_t) NPTR * (size_t) n_expert); + m.adds_host.reserve((size_t) chunk * 30); + m.mark.assign((size_t) chunk, -1); + m.first.resize((size_t) chunk); + m.count.resize((size_t) chunk); + DeviceScope scope(device, main_device); + bool ok = cudaStreamCreateWithFlags(&m.s, cudaStreamNonBlocking) == cudaSuccess && + cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) == cudaSuccess && + cudaEventCreateWithFlags(&m.done, cudaEventDisableTiming) == cudaSuccess && + cudaEventCreate(&m.t0) == cudaSuccess && cudaEventCreate(&m.t1) == cudaSuccess; + for (int i = 0; ok && i < SLOTS; ++i) + ok = cudaEventCreateWithFlags(&m.copied[i], cudaEventDisableTiming) == cudaSuccess && + cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) == cudaSuccess; + if (!ok) { err = "prompt offload: streams or events"; return false; } + // the input and the sums: pinned and portable, both GPUs copy them + if (cudaHostAlloc((void**) &m.h_input, (size_t) chunk * N * 2, cudaHostAllocPortable) != cudaSuccess || + cudaHostAlloc((void**) &m.h_sum, (size_t) chunk * N * 4, cudaHostAllocPortable) != cudaSuccess) { + err = "prompt offload: pinned input and sum buffers"; + return false; + } + Alloc a; + a.base = (uint8_t*) borrow; + a.cap = borrow_bytes; + a.owned = &m.owned; + if (!m.carve(a, chunk)) { + err = "prompt offload: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; + return false; + } + return true; +} + +uint16_t* Offload::host_input() const { return impl_->h_input; } +float* Offload::host_sum() const { return impl_->h_sum; } +cudaEvent_t Offload::done() const { return impl_->done; } + +bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, + const std::vector& src, const std::vector& w, cudaEvent_t input_ready, + std::string& err) { + Impl& m = *impl_; + const size_t n_exp = experts.size(), n_rows = src.size(); + if (T > m.chunk || off.size() != n_exp + 1 || w.size() != n_rows || (size_t) off.back() != n_rows || + n_rows > (size_t) T * 10 || n_exp > (size_t) m.n_expert) { + err = "prompt offload: a layer's share is out of range"; + return false; + } + DeviceScope scope(m.dev, m.main); + const auto h0 = std::chrono::steady_clock::now(); + if (m.timed) { // the previous layer is done by now (the main GPU waited for it) + float ms = 0; + cudaEventElapsedTime(&ms, m.t0, m.t1); + ms_gpu += ms; + } + const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; + const size_t bb = (size_t) strata::kernels::cpu::expert_layout().blob_bytes(layer); + const size_t stride = (bb + 255) & ~(size_t) 255; // staging: a run of arena blobs lands as one copy when equal + auto held = [&](int32_t e) -> int32_t { + return m.res && m.cache ? m.res[(size_t) layer * (size_t) m.n_expert + (size_t) e] : -1; + }; + // ---- the batches, and every expert's device pointers (one upload) + std::vector> batches; + for (size_t j0 = 0; j0 < n_exp;) { + size_t j1 = j0 + 1; + int64_t rows = off[j0 + 1] - off[j0]; + if (rows <= ROWS) + while (j1 < n_exp && j1 - j0 < (size_t) G && rows + (off[j1 + 1] - off[j1]) <= ROWS) rows += off[j1 + 1] - off[j1++]; + batches.emplace_back(j0, j1); + j0 = j1; + } + void** P = m.ptrs_host.data(); + const size_t E = (size_t) m.n_expert; // array stride + std::vector rows_of(n_exp); + for (size_t b = 0; b < batches.size(); ++b) { + const auto [j0, j1] = batches[b]; + size_t streamed = 0; + for (size_t j = j0; j < j1; ++j) { + const int64_t r = off[j] - off[j0]; + const int32_t slot = held(experts[j]); + P[0 * E + j] = slot >= 0 ? (void*) m.cache->device_slot(slot) : (void*) (m.stage[b % SLOTS] + streamed++ * stride); + P[1 * E + j] = m.xs + r * N; + P[2 * E + j] = m.dq_gu + (j - j0) * (size_t) (FF2 * N); + P[3 * E + j] = m.gu + r * FF2; + P[4 * E + j] = m.hh + r * FF; + P[5 * E + j] = m.dq_d + (j - j0) * (size_t) (N * FF); + P[6 * E + j] = m.d + r * N; + rows_of[j] = off[j + 1] - off[j]; + } + } + // per batch, its tokens and each one's rows in order (a token may be routed to several of the batch's experts): + // [tokens | starts (n + 1) | rows] at adds_at[b] + std::vector adds_at(batches.size()); + std::vector ntok(batches.size()); + m.adds_host.clear(); + for (size_t b = 0; b < batches.size(); ++b) { + const int32_t r0 = off[batches[b].first], r1 = off[batches[b].second]; + std::vector toks; + for (int32_t r = r0; r < r1; ++r) { + const int32_t t = src[(size_t) r]; + if (m.mark[(size_t) t] != (int32_t) b) { + m.mark[(size_t) t] = (int32_t) b; + m.first[(size_t) t] = (int32_t) toks.size(); + m.count[(size_t) toks.size()] = 0; + toks.push_back(t); + } + ++m.count[(size_t) m.first[(size_t) t]]; + } + const size_t nt = toks.size(), at = m.adds_host.size(); + adds_at[b] = at; + ntok[b] = (int32_t) nt; + m.adds_host.resize(at + nt + (nt + 1) + (size_t) (r1 - r0)); + int32_t* h_tok = m.adds_host.data() + at; + int32_t* h_start = h_tok + nt; + int32_t* h_list = h_start + nt + 1; + h_start[0] = 0; + for (size_t i = 0; i < nt; ++i) { + h_tok[i] = toks[i]; + h_start[i + 1] = h_start[i] + m.count[i]; + m.count[i] = h_start[i]; // the fill cursor + } + for (int32_t r = r0; r < r1; ++r) h_list[m.count[(size_t) m.first[(size_t) src[(size_t) r]]]++] = r; + for (int32_t t : toks) m.mark[(size_t) t] = -1; + } + cudaStreamWaitEvent(m.s, input_ready, 0); + cudaEventRecord(m.t0, m.s); + if (!m.adds_host.empty()) + cudaMemcpyAsync(m.adds, m.adds_host.data(), m.adds_host.size() * 4, cudaMemcpyHostToDevice, m.s); + // pageable sources: the copies are staged before they return, so the caller may reuse the vectors + cudaMemcpyAsync(m.ptrs, P, NPTR * E * sizeof(void*), cudaMemcpyHostToDevice, m.s); + cudaMemcpyAsync(m.input, m.h_input, (size_t) T * N * 2, cudaMemcpyHostToDevice, m.s); + if (n_rows > 0) { + cudaMemcpyAsync(m.rows_src, src.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); + cudaMemcpyAsync(m.w, w.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); + } + cudaMemsetAsync(m.sum, 0, (size_t) T * N * 4, m.s); + // ---- staging: the streamed experts of batch b into slot b % SLOTS, a copy per run of adjacent arena blobs + auto stage_batch = [&](size_t b) { + const auto [j0, j1] = batches[b]; + const int sl = (int) (b % SLOTS); + bool any = false; + for (size_t j = j0; j < j1;) { + const int32_t e = experts[j]; + if (held(e) >= 0) { ++j; continue; } + if (!any && m.live[sl]) cudaStreamWaitEvent(m.copy, m.used[sl], 0); // its previous batch is dequantized + any = true; + const uint8_t* b0 = m.src->blob(layer, e); + uint8_t* dst = (uint8_t*) P[0 * E + j]; + size_t k = j + 1; // the run: the next experts' blobs follow this one in the arena and in staging + if (m.src->pinned(layer, e) && stride == bb) + while (k < j1 && held(experts[k]) < 0 && m.src->pinned(layer, experts[k]) && + m.src->blob(layer, experts[k]) == b0 + (k - j) * bb && + (uint8_t*) P[0 * E + k] == dst + (k - j) * bb) ++k; + cudaMemcpyAsync(dst, b0, (k - j) * bb, cudaMemcpyHostToDevice, m.copy); // pageable blobs: staged + experts_streamed += (int64_t) (k - j); + j = k; + } + if (any) { + cudaEventRecord(m.copied[sl], m.copy); + m.live[sl] = true; + } + return any; + }; + std::vector has_stream(batches.size(), 0); + if (!batches.empty()) has_stream[0] = stage_batch(0); + auto** dp = (const uint16_t**) m.ptrs; // device arrays by index + for (size_t b = 0; b < batches.size(); ++b) { + if (b + 1 < batches.size()) has_stream[b + 1] = stage_batch(b + 1); // the next batch copies meanwhile + const auto [j0, j1] = batches[b]; + const int n = (int) (j1 - j0); + const int sl = (int) (b % SLOTS); + if (has_stream[b]) cudaStreamWaitEvent(m.s, m.copied[sl], 0); + strata::kernels::iq_dequant_experts_f16(f.gu_type, f.d_type, (const uint8_t* const*) (m.ptrs + 0 * E + j0), n, + f.up_off, f.down_off, f.n_ff, f.n_embd, m.dq_gu, m.dq_d, m.s); + if (has_stream[b]) cudaEventRecord(m.used[sl], m.s); + for (size_t j = j0; j < j1; ++j) if (held(experts[j]) >= 0) ++experts_resident; + const int64_t r0 = off[j0], rows = off[j1] - r0; + if (rows > ROWS) { // one expert: its rows in pieces + for (int64_t p = 0; p < rows; p += ROWS) { + const int64_t nr = std::min(ROWS, rows - p); + gather_rows16(m.input, m.rows_src + r0 + p, m.xs, nr, N, m.s); + m.gemm.f16(m.xs, m.dq_gu, m.gu, nr, FF2, N); + swiglu_interleaved(m.gu, m.hh, nr, m.s); + m.gemm.f16(m.hh, m.dq_d, m.d, nr, N, FF); + moe_scatter_add(m.sum, m.d, m.w + r0 + p, m.rows_src + r0 + p, nr, m.s); + } + continue; + } + gather_rows16(m.input, m.rows_src + r0, m.xs, rows, N, m.s); + m.gemm.f16_grouped(dp + 1 * E + j0, dp + 2 * E + j0, (float* const*) (m.ptrs + 3 * E + j0), + (const uint16_t* const*) (P + 1 * E + j0), (const uint16_t* const*) (P + 2 * E + j0), + (float* const*) (P + 3 * E + j0), rows_of.data() + j0, n, FF2, N); + swiglu_interleaved(m.gu, m.hh, rows, m.s); + m.gemm.f16_grouped(dp + 4 * E + j0, dp + 5 * E + j0, (float* const*) (m.ptrs + 6 * E + j0), + (const uint16_t* const*) (P + 4 * E + j0), (const uint16_t* const*) (P + 5 * E + j0), + (float* const*) (P + 6 * E + j0), rows_of.data() + j0, n, N, FF); + const int32_t* a = m.adds + adds_at[b]; + moe_gather_add(m.sum, m.d, r0, m.w, a, a + ntok[b], a + 2 * ntok[b] + 1, ntok[b], m.s); + } + cudaMemcpyAsync(m.h_sum, m.sum, (size_t) T * N * 4, cudaMemcpyDeviceToHost, m.s); + cudaEventRecord(m.done, m.s); + cudaEventRecord(m.t1, m.s); + m.timed = true; + ms_host += std::chrono::duration(std::chrono::steady_clock::now() - h0).count(); + const cudaError_t e = cudaGetLastError(); + if (e != cudaSuccess) { err = std::string("prompt offload: ") + cudaGetErrorString(e); return false; } + return true; +} + +} // namespace strata::prefill diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index d04c3c51e05..8c288d23848 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -13,6 +13,7 @@ #include "strata/kernels/qsa_select.hpp" #include "strata/prefill/gemm.hpp" #include "strata/prefill/kernels.hpp" +#include "strata/prefill/offload.hpp" #include @@ -98,6 +99,11 @@ struct Prefill::Impl { int32_t *ids = nullptr, *slot_dev = nullptr, *src_dev = nullptr; uint16_t *Xs = nullptr, *Hh = nullptr, *sh_h = nullptr; std::vector ids_host, slot_host, src_host, cnt, off; + // the second GPU's share (Prefill::offload): the input goes out and the sums come back on `xfer` + cudaStream_t xfer = nullptr; + cudaEvent_t ev_mixed = nullptr, ev_input = nullptr, ev_sum = nullptr; + std::vector entry_of, experts2, off2, src2; + std::vector w_host, w2; uint16_t* dq_gu[DQ] = {}; uint16_t* dq_d[DQ] = {}; uint8_t* stage_dev[STAGE] = {}; @@ -121,6 +127,9 @@ Prefill::~Prefill() { if (impl_->stage_host[i]) cudaFreeHost(impl_->stage_host[i]); } if (impl_->copy) cudaStreamDestroy(impl_->copy); + if (impl_->xfer) cudaStreamSynchronize(impl_->xfer); + for (cudaEvent_t e : {impl_->ev_mixed, impl_->ev_input, impl_->ev_sum}) if (e) cudaEventDestroy(e); + if (impl_->xfer) cudaStreamDestroy(impl_->xfer); for (void* p : impl_->owned) cudaFree(p); } @@ -138,7 +147,14 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co if (g.n_embd != N || g.hc != HC || g.hc_lr != LR || g.n_expert != NE || ss.k != K) { err = "prefill: geometry differs from the artifact's"; return false; } - if (cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) != cudaSuccess) { err = "prefill: copy stream"; return false; } + if (cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&m.ev_mixed, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&m.ev_input, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&m.ev_sum, cudaEventDisableTiming) != cudaSuccess) { + err = "prefill: streams and events"; + return false; + } const size_t T = (size_t) chunk; bool ok = true; Alloc o; @@ -186,6 +202,7 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co } m.ple_emb = o.take(T * N, ok); m.ids_host.resize(T * K); m.slot_host.resize(T * K); m.src_host.resize(T * K); m.cnt.resize(NE); m.off.resize(NE + 1); + m.entry_of.resize(T * K); m.w_host.resize(T * K); m.ple_emb_host.resize(T * N); m.ple_rows.resize(T * strata::kernels::PLE_N_HEADS); if (!ok) { err = "prefill: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; return false; } return true; @@ -430,6 +447,12 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); if (!wr || !wgi || !wsg || !wsu || !wsd) return false; + if (offload) { // the input's trip to the second GPU starts while the router and shared expert run + cudaEventRecord(m.ev_mixed, m.cs); + cudaStreamWaitEvent(m.xfer, m.ev_mixed, 0); + cudaMemcpyAsync(offload->host_input(), m.mixed_h, (size_t) T * N * 2, cudaMemcpyDeviceToHost, m.xfer); + cudaEventRecord(m.ev_input, m.xfer); + } mark(kPsRouter); if (!bf16_proj(m.gemm, wr, m.mixed_bf, m.logits, T, v.name("ffn_gate_inp.weight"), err)) return false; route(m.logits, m.ids, m.w, T, m.cs); @@ -442,6 +465,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e m.gemm.bf16(m.mixed_bf, (const uint16_t*) wgi->data, m.sg, T, 1, N); // group the (token, k) pairs by expert on the host cudaMemcpyAsync(m.ids_host.data(), m.ids, (size_t) T * K * 4, cudaMemcpyDeviceToHost, m.cs); + if (offload) cudaMemcpyAsync(m.w_host.data(), m.w, (size_t) T * K * 4, cudaMemcpyDeviceToHost, m.cs); cudaStreamSynchronize(m.cs); std::fill(m.cnt.begin(), m.cnt.end(), 0); for (int64_t i = 0; i < T * K; ++i) { @@ -457,14 +481,38 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const int32_t p = fill[(size_t) e]++; m.slot_host[(size_t) i] = p; m.src_host[(size_t) p] = (int32_t) (i / K); + m.entry_of[(size_t) p] = (int32_t) i; + } + auto resident = [&](int32_t e) { + return m.host_res && m.cache && m.host_res[(size_t) l * NE + e] >= 0; + }; + if (offload) { // the experts this GPU's cache does not hold go to the second GPU + m.experts2.clear(); + m.off2.assign(1, 0); + m.src2.clear(); + m.w2.clear(); + for (int32_t e = 0; e < NE; ++e) { + if (m.cnt[(size_t) e] == 0 || resident(e)) continue; + m.experts2.push_back(e); + for (int32_t p = m.off[(size_t) e]; p < m.off[(size_t) e + 1]; ++p) { + const int32_t i = m.entry_of[(size_t) p]; + m.src2.push_back(i / (int32_t) K); + m.w2.push_back(m.w_host[(size_t) i]); + m.slot_host[(size_t) i] = -1; // the combine takes its row from the returned sum + } + m.off2.push_back((int32_t) m.src2.size()); + } + if (!offload->run_layer(l, T, m.experts2, m.off2, m.src2, m.w2, m.ev_input, err)) return false; } cudaMemcpyAsync(m.slot_dev, m.slot_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); cudaMemcpyAsync(m.src_dev, m.src_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); gather_rows16(m.mixed_h, m.src_dev, m.Xs, T * K, N, m.cs); mark(kPsExperts); - // the experts, in id order: resident ones from VRAM, the others through the staging ring + // the experts, in id order: resident ones from VRAM, the others through the staging ring (or on + // the second GPU) std::vector order; - for (int32_t e = 0; e < NE; ++e) if (m.cnt[(size_t) e] > 0) order.push_back(e); + for (int32_t e = 0; e < NE; ++e) + if (m.cnt[(size_t) e] > 0 && (!offload || resident(e))) order.push_back(e); const strata::kernels::cpu::ExpertLayout& lay = strata::kernels::cpu::expert_layout(); // Stage ahead: the copy stream moves blobs host -> device while the compute stream works. int stage_next = 0; @@ -527,8 +575,15 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e swiglu_interleaved(m.GU + o0 * 1280, m.Hh + o0 * 640, ne, m.cs); m.gemm.f16(m.Hh + o0 * 640, m.dq_d[q], m.Dm + o0 * N, ne, N, 640); } + float* sums = m.emb; // the chunk's embeddings are in R by now + if (offload) { + cudaStreamWaitEvent(m.xfer, offload->done(), 0); + cudaMemcpyAsync(sums, offload->host_sum(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.xfer); + cudaEventRecord(m.ev_sum, m.xfer); + cudaStreamWaitEvent(m.cs, m.ev_sum, 0); + } mark(kPsCombine); - moe_combine(m.Dm, m.slot_dev, m.w, m.shared, m.sg, m.bo, T, m.cs); + moe_combine(m.Dm, m.slot_dev, m.w, m.shared, m.sg, m.bo, T, m.cs, offload ? sums : nullptr); } // ---- the hyper-connection write of this half if (half == 0) mark(kPsHcFfn); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 41deb1f8695..7fc1deee327 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -44,6 +44,7 @@ #include "strata/core/verify.hpp" #include "strata/core/mtp.hpp" #include "strata/core/prompt_cache.hpp" +#include "strata/prefill/offload.hpp" #include "strata/prefill/prefill.hpp" #include "strata/core/native_dense.hpp" #include "strata/program/logits_selection.hpp" @@ -203,6 +204,9 @@ struct Options { /// Plan v0.3 P5: the prompt path borrows the top expert-cache slots for its buffers and refills them after /// the prompt (default); `--no-prefill-borrow` reserves the buffers' VRAM for the whole session instead. bool no_prefill_borrow = false; + /// With --second-gpu, the prompt path's experts the main GPU's cache does not hold run on the second GPU + /// (default); `--no-prompt-offload` streams them to the main GPU instead. + bool no_prompt_offload = false; /// Plan v0.3 P5 validation: batch only positions [0, P) and run the rest of the prompt through the token path /// (teacher-forced), so the logits of positions >= P - which depend on the batched state - can be scored /// against the oracle at many positions. 0 = the whole prompt but the last position. @@ -344,6 +348,8 @@ void usage() { " --second-gpu-gib G its VRAM for experts (default: all but --second-gpu-reserve-mib, 2048)\n" " --second-gpu-min-mb M a layer's missed experts from which it takes its share (default 4; below\n" " it the CPU is quicker than its ~100 us round trip)\n" + " --no-prompt-offload with --second-gpu: the prompt path streams the experts the main GPU's cache\n" + " lacks to the main GPU instead of computing them on the second\n" " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" " accept the drafts that match it, so speculation settings compare on the\n" " same text (not --serve)\n" @@ -584,6 +590,7 @@ int main(int argc, char** argv) { else if (a == "--prefill") o.prefill_chunk = std::atoll(next("--prefill")); else if (a == "--no-split-rows") o.no_split_rows = true; else if (a == "--no-prefill-borrow") o.no_prefill_borrow = true; + else if (a == "--no-prompt-offload") o.no_prompt_offload = true; else if (a == "--prefill-until") o.prefill_until = std::atoll(next("--prefill-until")); else if (a == "--dump-final-r") o.dump_final_r = next("--dump-final-r"); else if (a == "--spec") o.spec = std::atoi(next("--spec")); @@ -1995,6 +2002,15 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; } + strata::prefill::Offload offload; + if (o.second_gpu >= 0 && !o.no_prompt_offload) { + if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), + g.n_expert, o.prefill_chunk, err)) { + std::fprintf(stderr, "strata serve: %s\n", err.c_str()); + return 1; + } + prefill.offload = &offload; + } strata::core::Verifier ver; strata::core::VerifyHits vh; vh.d_res = thits.d_res; @@ -2327,6 +2343,7 @@ int main(int argc, char** argv) { int64_t pos_start = 0; int64_t spec_pos = 0; // plan v0.3 P6: where the speculative loop starts (0 = not used) strata::prefill::Prefill prefill; + strata::prefill::Offload offload; double prefill_batched_ms = 0; std::FILE* final_r = o.dump_final_r.empty() ? nullptr : std::fopen(o.dump_final_r.c_str(), "wb"); std::vector final_r_host(final_r ? (size_t) (g.hc * g.n_embd) : 0); @@ -2366,6 +2383,14 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } + if (o.second_gpu >= 0 && !o.no_prompt_offload) { + if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), + g.n_expert, o.prefill_chunk, err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } + prefill.offload = &offload; + } if (!o.mtp.empty()) { if (!mtp.bind(wt, &native_head, nullptr, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); @@ -2414,6 +2439,10 @@ int main(int argc, char** argv) { (long long) ps.tokens, (long long) ps.chunks, ps.ms_total, ps.ms_total > 0 ? 1000.0 * (double) ps.tokens / ps.ms_total : 0.0, (long long) ps.experts_streamed, (long long) ps.experts_dma, ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); + if (prefill.offload != nullptr) + std::fprintf(stderr, "strata generate: prefill on the second GPU: experts streamed %lld, resident %lld; " + "%.1f ms queuing, %.1f ms GPU\n", (long long) offload.experts_streamed, + (long long) offload.experts_resident, offload.ms_host, offload.ms_gpu); if (o.prefill_profile) { std::fprintf(stderr, "strata generate: prefill GPU ms by section:"); for (int i = 0; i < strata::prefill::kPsCount; ++i) From 6e3864842f2b95d9836857053293795a828e3bef Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 03:17:21 +0200 Subject: [PATCH 21/92] Prompt path: its buffers on the second GPU borrow that GPU's cache slots too; one SlotLoan for both GPUs and both modes --- src/program/generate.cpp | 210 ++++++++++++++++++++++----------------- 1 file changed, 119 insertions(+), 91 deletions(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 7fc1deee327..d37f31bd419 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -201,8 +201,8 @@ struct Options { /// Plan v0.3 P5: batched prompt processing in chunks of this many tokens (0 = the token path). int64_t prefill_chunk = 0; bool no_split_rows = false; ///< plan v0.3 P4 A/B: one whole expert per pool thread - /// Plan v0.3 P5: the prompt path borrows the top expert-cache slots for its buffers and refills them after - /// the prompt (default); `--no-prefill-borrow` reserves the buffers' VRAM for the whole session instead. + /// Plan v0.3 P5: the prompt path borrows the last expert-cache slots of each GPU for its buffers and refills + /// them after the prompt (default); `--no-prefill-borrow` reserves the buffers' VRAM for the whole session. bool no_prefill_borrow = false; /// With --second-gpu, the prompt path's experts the main GPU's cache does not hold run on the second GPU /// (default); `--no-prompt-offload` streams them to the main GPU instead. @@ -482,6 +482,72 @@ int argmax(const std::vector& v) { return best; } +/// Expert-cache slots lent to the batched prompt path for its buffers: the cache's last slots, as many as hold the +/// buffers, while at least 128 stay. `lend` takes their experts out of the residency table (the prompt path +/// streams them instead), `give_back` copies them in again from the arena once the prompt is done. +struct SlotLoan { + strata::core::ExpertCache* cache = nullptr; + std::vector* res = nullptr; + int32_t* d_res = nullptr; // the table's device copy, or null + int device = 0; // the cache's GPU; `main_device` is current again after `give_back` + int main_device = 0; + int32_t first = -1; // the first lent slot, -1 when the cache cannot spare the bytes + std::vector> lent; // (residency index, slot) while lent + + uint64_t offset(int64_t slot) const { + return cache->slot_offsets() ? cache->slot_offsets()[slot] + : (uint64_t) slot * (uint64_t) (cache->bytes() / cache->slots()); + } + /// Plans a loan of `need` bytes; false when the cache cannot spare them. + bool plan(strata::core::ExpertCache& c, std::vector& r, int32_t* dr, uint64_t need, int dev, int main_dev) { + cache = &c; + res = &r; + d_res = dr; + device = dev; + main_device = main_dev; + int64_t k = 0; + while (k < c.slots() && (uint64_t) c.bytes() - offset(c.slots() - k) < need) ++k; + first = k + 128 <= c.slots() ? (int32_t) (c.slots() - k) : -1; + return first >= 0; + } + void* base() const { return first >= 0 ? (void*) cache->device_slot(first) : nullptr; } + uint64_t bytes() const { return first >= 0 ? (uint64_t) cache->bytes() - offset(first) : 0; } + int64_t slots() const { return first >= 0 ? cache->slots() - first : 0; } + void lend() { + if (first < 0) return; + std::vector& r = *res; + for (size_t i = 0; i < r.size(); ++i) + if (r[i] >= first) { + lent.emplace_back((int32_t) i, r[i]); + r[i] = strata::core::kNotResident; + } + if (d_res != nullptr) cudaMemcpy(d_res, r.data(), r.size() * sizeof(int32_t), cudaMemcpyHostToDevice); + } + /// After the borrower's last use: its GPU work is waited for first. + bool give_back(strata::core::ExpertSource& src, int64_t n_expert, std::string& err) { + if (lent.empty()) return true; + cudaSetDevice(device); + cudaDeviceSynchronize(); + bool ok = true; + const auto& lay = strata::kernels::cpu::expert_layout(); + for (const auto& [i, slot] : lent) { + const int64_t l = i / n_expert; + const uint8_t* b = src.blob(l, i % n_expert); + if (b == nullptr || !cache->fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(l))) { + if (b == nullptr) err = "no arena blob for a lent slot"; + ok = false; + break; + } + (*res)[(size_t) i] = slot; + } + cudaSetDevice(main_device); + if (ok && d_res != nullptr) + cudaMemcpy(d_res, res->data(), res->size() * sizeof(int32_t), cudaMemcpyHostToDevice); + lent.clear(); + return ok; + } +}; + } // namespace int main(int argc, char** argv) { @@ -1868,7 +1934,10 @@ int main(int argc, char** argv) { cudaSetDevice(o.second_gpu); size_t free_b = 0, total_b = 0; cudaMemGetInfo(&free_b, &total_b); - const uint64_t reserve = (uint64_t) o.second_gpu_reserve_mib << 20; + // the prompt path's buffers there borrow cache slots, else they are reserved here + const bool offload_own = o.prefill_chunk > 0 && !o.no_prompt_offload && o.no_prefill_borrow; + const uint64_t reserve = ((uint64_t) o.second_gpu_reserve_mib << 20) + + (offload_own ? strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert) : 0); const uint64_t budget = o.second_gpu_gib > 0 ? (uint64_t) (o.second_gpu_gib * 1073741824.0) : (free_b > reserve ? (uint64_t) free_b - reserve : 0); const auto& lay = strata::kernels::cpu::expert_layout(); @@ -1910,7 +1979,8 @@ int main(int argc, char** argv) { if (ok && adaptive) { ok = tier2.init(gpu2.cache(), *srcp, host_res2, nullptr, g.n_layers, g.n_expert, o.adapt_swaps, err, o.second_gpu, main_dev); - for (size_t i = 0; ok && i < empty2.size(); ++i) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); + // lowest slots first: the last ones, which the prompt path borrows, stay empty longest + for (size_t i = empty2.size(); ok && i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); tier2.set_upper(&tier); } if (!ok) { @@ -1979,33 +2049,25 @@ int main(int argc, char** argv) { return 2; } strata::prefill::Prefill prefill; - void* borrow = nullptr; - uint64_t borrow_bytes = 0; - int32_t lend_first = -1; - if (!o.no_prefill_borrow && d_res != nullptr) { - const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk); - const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob; - int64_t k = (int64_t) ((need + (uint64_t) blob - 1) / (uint64_t) blob); - if (xcache.slot_offsets() != nullptr) { - k = 0; - while (k < xcache.slots() && - (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[xcache.slots() - k]) < need) ++k; - } - if (k + 128 <= xcache.slots()) { - lend_first = (int32_t) (xcache.slots() - k); - borrow = xcache.device_slot(lend_first); - borrow_bytes = xcache.slot_offsets() ? (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[lend_first]) - : (uint64_t) k * (uint64_t) blob; - } - } - if (!prefill.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, borrow, borrow_bytes)) { + SlotLoan loan, loan2; // the prompt path's buffers on each GPU, lent for every batched prompt + if (!o.no_prefill_borrow && loan.plan(xcache, host_res, d_res, strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk), + o.main_gpu, o.main_gpu)) + std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB)\n", + (long long) loan.slots(), (double) loan.bytes() / 1073741824.0); + if (!prefill.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, loan.base(), + loan.bytes())) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; } strata::prefill::Offload offload; if (o.second_gpu >= 0 && !o.no_prompt_offload) { + if (!o.no_prefill_borrow && !host_res2.empty() && + loan2.plan(gpu2.cache(), host_res2, nullptr, strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert), + o.second_gpu, o.main_gpu)) + std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB) on the second GPU\n", + (long long) loan2.slots(), (double) loan2.bytes() / 1073741824.0); if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), - g.n_expert, o.prefill_chunk, err)) { + g.n_expert, o.prefill_chunk, err, loan2.base(), loan2.bytes())) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; } @@ -2194,16 +2256,9 @@ int main(int argc, char** argv) { if (last > reuse && batched_end - last <= o.feed_max) batched_end = last; } if (batched_end > reuse) { - std::vector> lent_now; - apply_pending(true); - if (lend_first >= 0) { - for (size_t i = 0; i < host_res.size(); ++i) - if (host_res[i] >= lend_first) { - lent_now.emplace_back((int32_t) i, host_res[i]); - host_res[i] = strata::core::kNotResident; - } - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - } + apply_pending(true); // no copy may still land in a lent slot + loan.lend(); + loan2.lend(); for (int64_t a0 = reuse; a0 < batched_end;) { const int64_t b0 = track ? std::min(batched_end, a0 + step) : batched_end; if (!prefill.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } @@ -2213,17 +2268,10 @@ int main(int argc, char** argv) { } a0 = b0; } - for (const auto& [i, slot] : lent_now) { - const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert); - if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err, - (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) { - std::printf("ERR refilling a lent slot failed: %s\n", err.c_str()); - return 1; - } - host_res[(size_t) i] = slot; + if (!loan.give_back(*srcp, g.n_expert, err) || !loan2.give_back(*srcp, g.n_expert, err)) { + report("refilling a lent slot failed: " + err); + return 1; } - if (!lent_now.empty()) - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); } // prompt tokens through verify windows: each window is committed whole, and the draft layer catches up // on the same cells (its drafts are not used) @@ -2347,45 +2395,31 @@ int main(int argc, char** argv) { double prefill_batched_ms = 0; std::FILE* final_r = o.dump_final_r.empty() ? nullptr : std::fopen(o.dump_final_r.c_str(), "wb"); std::vector final_r_host(final_r ? (size_t) (g.hc * g.n_embd) : 0); - std::vector> lent; // (residency index, slot) lent to the prompt path + SlotLoan loan, loan2; // cache slots for the prompt path's buffers on each GPU, refilled after the prompt if (o.prefill_chunk > 0 && n_prompt > 1) { - void* borrow = nullptr; - uint64_t borrow_bytes = 0; - if (!o.no_prefill_borrow && !host_res.empty() && d_res != nullptr) { - const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk); - const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob; - int64_t k = (int64_t) ((need + (uint64_t) blob - 1) / (uint64_t) blob); - if (xcache.slot_offsets() != nullptr) { // sized slots: take slots from the end until they hold `need` - k = 0; - while (k < xcache.slots() && - (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[xcache.slots() - k]) < need) ++k; - } - if (k + 128 <= xcache.slots()) { // the lent slots are refilled after the prompt - const int32_t first = (int32_t) (xcache.slots() - k); - for (size_t i = 0; i < host_res.size(); ++i) - if (host_res[i] >= first) { - lent.emplace_back((int32_t) i, host_res[i]); - host_res[i] = strata::core::kNotResident; - } - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - borrow = xcache.device_slot(first); - borrow_bytes = xcache.slot_offsets() ? (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[first]) - : (uint64_t) k * (uint64_t) blob; - std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB)\n", (long long) k, - (double) borrow_bytes / 1073741824.0); - } - } - if (borrow == nullptr) + if (!o.no_prefill_borrow && !host_res.empty() && d_res != nullptr && + loan.plan(xcache, host_res, d_res, strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk), o.main_gpu, + o.main_gpu)) + std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB)\n", + (long long) loan.slots(), (double) loan.bytes() / 1073741824.0); + else std::fprintf(stderr, "strata generate: prompt path allocates its own buffers (no cache slots to borrow)\n"); + loan.lend(); if (!prefill.init(wt, g, ss, srcp, o.expert_cache > 0 ? &xcache : nullptr, - host_res.empty() ? nullptr : host_res.data(), o.prefill_chunk, main_cs, err, borrow, - borrow_bytes)) { + host_res.empty() ? nullptr : host_res.data(), o.prefill_chunk, main_cs, err, loan.base(), + loan.bytes())) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } if (o.second_gpu >= 0 && !o.no_prompt_offload) { + if (!o.no_prefill_borrow && !host_res2.empty() && + loan2.plan(gpu2.cache(), host_res2, nullptr, strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert), + o.second_gpu, o.main_gpu)) + std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB) on the second GPU\n", + (long long) loan2.slots(), (double) loan2.bytes() / 1073741824.0); + loan2.lend(); if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), - g.n_expert, o.prefill_chunk, err)) { + g.n_expert, o.prefill_chunk, err, loan2.base(), loan2.bytes())) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -2410,22 +2444,16 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } - // refill the lent slots from the arena and give them back to the decode tier - if (!lent.empty()) { - const Clock::time_point tr = Clock::now(); - for (const auto& [i, slot] : lent) { - const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert); - if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err, - (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) { - std::fprintf(stderr, "strata generate: refilling a lent slot failed: %s\n", err.c_str()); - return 1; - } - host_res[(size_t) i] = slot; - } - cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); - std::fprintf(stderr, "strata generate: %zu lent slots refilled in %.1f ms\n", lent.size(), - std::chrono::duration(Clock::now() - tr).count()); + // refill the lent slots from the arena and give them back to the decode tiers + const size_t n_lent[2] = {loan.lent.size(), loan2.lent.size()}; + const Clock::time_point tr = Clock::now(); + if (!loan.give_back(*srcp, g.n_expert, err) || !loan2.give_back(*srcp, g.n_expert, err)) { + std::fprintf(stderr, "strata generate: refilling a lent slot failed: %s\n", err.c_str()); + return 1; } + if (n_lent[0] + n_lent[1] > 0) + std::fprintf(stderr, "strata generate: lent slots refilled (%zu, %zu on the second GPU) in %.1f ms\n", + n_lent[0], n_lent[1], std::chrono::duration(Clock::now() - tr).count()); prefill_batched_ms = std::chrono::duration(Clock::now() - tp0).count(); prefill_ms += prefill_batched_ms; pos_start = n_batched; From 83349c72eb2a7c89e737ced7d74b631b841beca6 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 03:52:54 +0200 Subject: [PATCH 22/92] Prompt path: dense work in 2048-token sub-chunks over shared scratch, a layer's weights dequantized once; the routed experts of both GPUs through one batched runner; buffers sized to each prompt (16K: 1.4 GiB of cache slots lent instead of 10.9) --- CMakeLists.txt | 2 +- include/strata/prefill/experts.hpp | 74 ++ include/strata/prefill/gemm.hpp | 14 +- include/strata/prefill/kernels.hpp | 13 +- include/strata/prefill/offload.hpp | 61 -- include/strata/prefill/prefill.hpp | 53 +- src/prefill/{offload.cpp => experts.cpp} | 274 ++++--- src/prefill/gemm.cu | 32 +- src/prefill/kernels.cu | 33 +- src/prefill/prefill.cpp | 889 +++++++++++------------ src/program/generate.cpp | 171 +++-- 11 files changed, 825 insertions(+), 791 deletions(-) create mode 100644 include/strata/prefill/experts.hpp delete mode 100644 include/strata/prefill/offload.hpp rename src/prefill/{offload.cpp => experts.cpp} (54%) diff --git a/CMakeLists.txt b/CMakeLists.txt index 48801a6632c..d65c77ecbe6 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -249,7 +249,7 @@ if(STRATA_ENABLE_CUDA) # what composes them into a token. No tokenizer - the prompt arrives as ids - because Gate C1 compares logits # against llama.cpp on the SAME ids and a tokenizer on one side of that comparison is a second variable. # plan v0.3 P5: batched prompt processing (tensor-core GEMMs through cuBLAS). - add_library(strata_prefill STATIC src/prefill/gemm.cu src/prefill/kernels.cu src/prefill/offload.cpp + add_library(strata_prefill STATIC src/prefill/gemm.cu src/prefill/kernels.cu src/prefill/experts.cpp src/prefill/prefill.cpp) target_include_directories(strata_prefill PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) target_link_libraries(strata_prefill PUBLIC strata_engine CUDA::cublas CUDA::cudart) diff --git a/include/strata/prefill/experts.hpp b/include/strata/prefill/experts.hpp new file mode 100644 index 00000000000..6c286ef113d --- /dev/null +++ b/include/strata/prefill/experts.hpp @@ -0,0 +1,74 @@ +// include/strata/prefill/experts.hpp - the prompt path's routed experts on one GPU. +// +// A prompt chunk uses nearly every expert of every layer. Its experts go through in batches of consecutive ids: per +// batch one staging copy for each run of adjacent arena blobs among the experts the GPU's cache does not hold, one +// dequantization launch, one gather of the batch's rows, the two GEMMs of each expert with a SwiGLU between them, and +// one ordered add of every row, times its routing weight, into its token's sum. A token's experts add up in ascending +// id order, whatever the batches. +// +// Local: on the prompt path's own GPU and stream, reading the chunk's MoE input and adding into its MoE output there. +// Remote: on a second GPU. The first GPU (the 3090) sits on a PCIe 4.0 x4 link, over which an expert its cache lacks +// streams at 6.4 GB/s (~0.65 GB (IQ3_XXS) to ~1.4 GB (UD-Q4_K_XL) a layer); the second GPU sits on x16. The first GPU +// copies the chunk's MoE input (FP16) to pinned host memory; the second copies it in, computes the experts the first +// one's cache lacks - from its own cache when it holds them - and returns their sums (FP32) the same way, while the +// first computes the experts its cache holds. Neither GPU reads the other's memory: Windows gives GeForce cards no +// peer access. +#pragma once + +#include "strata/core/expert_cache.hpp" +#include "strata/core/expert_source.hpp" + +#include + +#include +#include +#include +#include + +namespace strata::prefill { + +class ExpertRunner { +public: + ExpertRunner(); + ~ExpertRunner(); + ExpertRunner(const ExpertRunner&) = delete; + ExpertRunner& operator=(const ExpertRunner&) = delete; + + /// For chunks of up to `max_chunk` tokens on `device`. `stream`: the prompt path's stream on that device (local), + /// or null (remote: its own streams; `main_device` is made current again after every call). `cache`/`res`: the + /// device's expert cache and its residency (n_layers x n_expert slots, -1 when not held), or null. `streams`: + /// whether experts its cache does not hold come its way (staging buffers). + bool init(int device, int main_device, void* stream, core::ExpertSource* src, const core::ExpertCache* cache, + const int32_t* res, int64_t n_expert, int64_t max_chunk, bool streams, std::string& err); + /// Device bytes of the buffers for chunks of `chunk` tokens. + static uint64_t bytes_needed(int64_t chunk, int64_t n_expert, bool remote, bool streams); + /// The buffers for chunks of up to `chunk` tokens, carved from `region` (`bytes` long, e.g. lent cache slots), or + /// with a null region allocated for `max_chunk` tokens (once). Before the first layer of every prompt that + /// uses a region. + bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err); + + /// Remote: pinned [max_chunk, n_embd]: the MoE input (FP16) goes in here, the sums (FP32) come out here. + uint16_t* host_input() const; + float* host_sum() const; + + /// Layer `layer` of a chunk of T tokens: expert experts[j] serves rows [off[j], off[j + 1]), row r being token + /// src[r] with routing weight w[r]; each row times its weight adds to its token's sum. Local: reads `input` + /// [T, n_embd] (FP16) and adds into `sum` [T, n_embd], queued on the prompt path's stream. Remote: `input` and + /// `sum` are null; its work waits for `input_ready` (an event of the first GPU, recorded once host_input holds + /// the input), and done() is recorded once host_sum holds the sums. Returns once the work is queued. + bool run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, + const std::vector& src, const std::vector& w, cudaEvent_t input_ready, + const uint16_t* input, float* sum, std::string& err); + cudaEvent_t done() const; + + int64_t experts_resident = 0; ///< expert-layer groups served from its cache + int64_t experts_streamed = 0; ///< expert blobs copied from the arena + double ms_host = 0; ///< host time queuing its work + double ms_gpu = 0; ///< remote: its GPU time from the input's arrival (all layers but the last) + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace strata::prefill diff --git a/include/strata/prefill/gemm.hpp b/include/strata/prefill/gemm.hpp index daa6582bc7e..976fed5904e 100644 --- a/include/strata/prefill/gemm.hpp +++ b/include/strata/prefill/gemm.hpp @@ -18,11 +18,11 @@ class Gemm { Gemm(const Gemm&) = delete; Gemm& operator=(const Gemm&) = delete; - /// `scratch_elems`: BF16 elements of the dequantization scratch (the largest weight dequantized at once). - bool init(void* stream, int64_t scratch_elems, std::string& err); - /// The same with caller-owned device buffers (the prompt path borrowing expert-cache slots). - bool init_external(void* stream, uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes, - std::string& err); + /// The cuBLAS handle on `stream`; set_buffers() before the first product. + bool init(void* stream, std::string& err); + /// Caller-owned device buffers (e.g. lent expert-cache slots): the dequantization scratch (`scratch_elems` FP16 + /// elements: the largest weight dequantized at once; may be null for no native() calls) and the cuBLAS workspace. + void set_buffers(uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes); /// Y[T, N] (fp32, row stride ldy) = X[T, K] (bf16, row-major) . W[N, K]^T (bf16, row-major). `beta` = 1 adds. void bf16(const uint16_t* X, const uint16_t* W, float* Y, int64_t T, int64_t N, int64_t K, int64_t ldy = 0, @@ -51,11 +51,7 @@ class Gemm { void* stream_ = nullptr; uint16_t* scratch_ = nullptr; int64_t scratch_elems_ = 0; - void* workspace_ = nullptr; - bool external_ = false; bool grouped_ = true; // cuBLAS groups FP16 -> FP32 GEMMs (cleared at the first refusal) }; - - } // namespace strata::prefill diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index 355f9af58be..888531f77b1 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -14,11 +14,12 @@ namespace strata::prefill { void gr_norm(const float* R, const float* w_norm, float eps, float* xn, uint16_t* xn16, int64_t T, void* stream); /// lo16[t, k] = bf16(silu(lo[t, k] / hc)) void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream); -/// mixed[t, d] = mean_c xn[t, c, d] * sigmoid(gated[t, c, d]); FP32, BF16 and FP16 (either image may be null). +/// mixed[t, d] = mean_c xn[t, c, d] * sigmoid(gated[t, c, d]); FP32, BF16 and FP16 (any of them may be null). void gr_mix(const float* xn, const float* gated, float* mixed, uint16_t* mixed16, int64_t T, void* stream, uint16_t* mixed_h = nullptr); -/// R[t, c, d] += bo[t, d] * 2 sigmoid(inj[t, c] / hc) (inj has row stride inj_ld) -void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream); +/// R[t, c, d] += (bo[t, d] (+ partial[t, d])) * 2 sigmoid(inj[t, c] / hc) (inj has row stride inj_ld) +void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream, + const float* partial = nullptr); /// R[t, c, :] = e[t, :] for all four streams (the embedding broadcast). void gr_broadcast(const float* e, float* R, int64_t T, void* stream); @@ -46,10 +47,8 @@ void swiglu_interleaved(const float* gu, uint16_t* h16, int64_t n, void* stream) void swiglu_pair(const float* g, const float* u, uint16_t* h16, int64_t n, void* stream); /// Gather rows: dst16[i, :] = x16[src[i], :] (n rows of `width` BF16). void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int64_t n, int64_t width, void* stream); -/// bo[t, :] = sum_k w[t, k] * D[slot[t, k], :] + shared[t, :] * sigmoid(sg[t]) (+ partial[t, :]); a slot of -1 is -/// an entry the second GPU computed, whose sum `partial` holds. -void moe_combine(const float* D, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo, - int64_t T, void* stream, const float* partial = nullptr); +/// In place: x[t, :] *= sigmoid(sg[t]) (the shared expert's scalar gate) +void moe_shared_gate(float* x, const float* sg, int64_t T, void* stream); /// sum[src[r], :] += w[r] * rows[r, :] for n rows (one expert's rows: no token twice). void moe_scatter_add(float* sum, const float* rows, const float* w, const int32_t* src, int64_t n, void* stream); /// The same for several experts' rows (row r at rows[r - r0]), where a token may recur: token tok[b] adds its rows diff --git a/include/strata/prefill/offload.hpp b/include/strata/prefill/offload.hpp deleted file mode 100644 index 6b13004c3de..00000000000 --- a/include/strata/prefill/offload.hpp +++ /dev/null @@ -1,61 +0,0 @@ -// include/strata/prefill/offload.hpp - the prompt path's experts on a second GPU. -// -// A prompt chunk uses nearly every expert of every layer, and the first GPU (the 3090, on a PCIe 4.0 x4 link) has -// to stream all the ones its cache does not hold: ~0.65 GB (IQ3_XXS) to ~1.4 GB (UD-Q4_K_XL) a layer at 6.4 GB/s. -// The second GPU sits on a x16 link. Per layer the first GPU copies the chunk's MoE input (FP16) to pinned host -// memory while its router and shared expert run; the second GPU copies it in, computes each of its experts - -// from its own cache when it holds them, else streamed from the pinned arena through a ring of staging slots - -// and adds each expert's weighted rows into a per-token FP32 sum, in ascending expert order. The sum returns -// through pinned host memory to the first GPU, which computes the experts its own cache holds meanwhile and adds -// the sum in its combine. Neither GPU reads the other's memory: Windows gives GeForce cards no peer access. -#pragma once - -#include "strata/core/expert_cache.hpp" -#include "strata/core/expert_source.hpp" - -#include - -#include -#include -#include -#include - -namespace strata::prefill { - -class Offload { -public: - Offload(); - ~Offload(); - Offload(const Offload&) = delete; - Offload& operator=(const Offload&) = delete; - - /// Buffers for chunks of up to `chunk` tokens on `device`; `main_device` is made current again after every - /// call. `cache`/`res`: the device's expert cache and its residency (n_layers x n_expert slots, -1 when not - /// held), or null. `borrow`/`borrow_bytes`: device memory for the buffers (lent cache slots), else allocated. - bool init(int device, int main_device, core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* res, - int64_t n_expert, int64_t chunk, std::string& err, void* borrow = nullptr, uint64_t borrow_bytes = 0); - /// Device bytes `init` needs for chunks of `chunk` tokens (what a borrowed region must hold). - static uint64_t bytes_needed(int64_t chunk, int64_t n_expert); - - /// Pinned [chunk, n_embd]: the first GPU writes the MoE input (FP16) here, and reads the sums (FP32) here. - uint16_t* host_input() const; - float* host_sum() const; - - /// Layer `layer` of a chunk of T tokens: expert experts[j] serves rows [off[j], off[j + 1]), row r being token - /// src[r] with routing weight w[r]. Queues the work behind `input_ready` (an event of the first GPU, recorded - /// once host_input holds the input) and returns; done() is recorded once host_sum holds the sums. - bool run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, - const std::vector& src, const std::vector& w, cudaEvent_t input_ready, std::string& err); - cudaEvent_t done() const; - - int64_t experts_resident = 0; ///< expert-layer groups served from its cache - int64_t experts_streamed = 0; ///< expert blobs copied from the arena - double ms_host = 0; ///< host time queuing its work - double ms_gpu = 0; ///< its GPU time from the input's arrival (all layers but the last) - -private: - struct Impl; - std::unique_ptr impl_; -}; - -} // namespace strata::prefill diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index 0ca46551cf2..8254b7a9afc 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -1,11 +1,12 @@ // include/strata/prefill/prefill.hpp - plan v0.3 P5: batched prompt processing. // -// The prompt's positions [pos0, pos0 + n) are processed in chunks of `chunk` tokens through all 48 layers, leaving -// the session state (GDN recurrence and conv state, QSA KV pools and indexer, PLE history) where the token path -// would have left it; the decode loop then continues with the next token. Per layer: the projections are -// tensor-core GEMMs (quantized weights dequantized to FP16 on the fly, BF16 weights as they are), the recurrences -// walk the chunk inside one kernel, and the routed experts are grouped by expert: resident ones are read from the -// VRAM tier, the others streamed from the host arena through a pinned ring on a copy stream. +// The prompt's positions [pos0, pos0 + n) are processed in chunks through all 48 layers, leaving the session state +// (GDN recurrence and conv state, QSA KV pools and indexer, PLE history) where the token path would have left it; the +// decode loop then continues with the next token. Per layer: the projections are tensor-core GEMMs (quantized +// weights dequantized to FP16 on the fly, BF16 weights as they are) and the recurrences walk the tokens inside one +// kernel, in sub-chunks of up to 2048 tokens whose buffers are reused; the routed experts then run over the whole +// chunk (strata/prefill/experts.hpp), so an expert the cache does not hold is streamed once per chunk. Long chunks +// cost little VRAM besides the residual stream (~55 KB a token). // // Requires the native weights (`--native`): every quantized projection must carry its GGUF blocks. #pragma once @@ -15,6 +16,7 @@ #include "strata/core/layer.hpp" #include "strata/core/session.hpp" #include "strata/core/weights.hpp" +#include "strata/prefill/experts.hpp" #include #include @@ -23,8 +25,6 @@ namespace strata::prefill { -class Offload; - /// The sections `Prefill::profile` times, in the order a layer runs them. enum PrefillSection { kPsPle, kPsHcRead, kPsGdn, kPsGdnConv, kPsGdnScan, kPsGdnNorm, kPsGdnOut, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, @@ -36,10 +36,9 @@ struct PrefillStats { int64_t tokens = 0; int64_t chunks = 0; double ms_total = 0; - double ms_experts_host = 0; ///< host time staging non-resident experts - int64_t experts_streamed = 0; ///< expert blobs copied host -> device - int64_t experts_dma = 0; ///< ...of which straight from the pinned arena (no CPU copy) - int64_t experts_resident = 0; ///< expert-layer groups served from the VRAM tier + double ms_experts_host = 0; ///< host time queuing this GPU's experts + int64_t experts_streamed = 0; ///< expert blobs copied host -> this GPU + int64_t experts_resident = 0; ///< expert-layer groups served from its VRAM tier double ms_ple = 0; double ms_section[kPsCount] = {}; ///< GPU ms per section, with `Prefill::profile` }; @@ -51,18 +50,23 @@ class Prefill { Prefill(const Prefill&) = delete; Prefill& operator=(const Prefill&) = delete; - /// `host_res`: the static residency table (n_layers x n_expert, slot or -1) or null; `cache` its slots. - /// `borrow`/`borrow_bytes`: device memory to carve every buffer from (the top slots of the expert cache, - /// lent for the prompt and refilled after it); null = allocate normally. + /// For chunks of up to `max_chunk` tokens. `host_res`: the static residency table (n_layers x n_expert, slot + /// or -1) or null; `cache` its slots. `offload`: a runner on a second GPU (initialized, bound by the caller) + /// that computes the experts this GPU's cache does not hold; null: they stream here. bool init(const core::WeightTable& wt, const core::ModelGeometry& g, core::SessionState& ss, - core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* host_res, int64_t chunk, - void* stream, std::string& err, void* borrow = nullptr, uint64_t borrow_bytes = 0); - - /// Device bytes `init` needs for a chunk of `chunk` tokens (what a borrowed region must hold). - static uint64_t bytes_needed(const core::ModelGeometry& g, const core::SessionState& ss, int64_t chunk); - - /// Positions [pos0, pos0 + n) holding `tokens`; `ss.ple_prev` must be the two tokens before pos0 (oldest - /// first, -1 for none) and is advanced to the last two of these. + core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* host_res, int64_t max_chunk, + void* stream, ExpertRunner* offload, std::string& err); + + /// Device bytes of the buffers for chunks of `chunk` tokens. + static uint64_t bytes_needed(const core::ModelGeometry& g, const core::SessionState& ss, int64_t chunk, + bool offload); + /// The buffers for chunks of up to `chunk` tokens, carved from `region` (`bytes` long: lent expert-cache + /// slots), or with a null region allocated for the longest chunk (once). Before the first chunk of every prompt + /// that uses a region. + bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err); + + /// Positions [pos0, pos0 + n) holding `tokens`, in chunks of the bound length; `ss.ple_prev` must be the two + /// tokens before pos0 (oldest first, -1 for none) and is advanced to the last two of these. bool run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& err); const PrefillStats& stats() const { return stats_; } @@ -80,9 +84,6 @@ class Prefill { /// one after every chunk). bool profile = false; - /// The experts this GPU's cache does not hold, computed on a second GPU instead of streamed here; null: here. - Offload* offload = nullptr; - private: struct Impl; std::unique_ptr impl_; diff --git a/src/prefill/offload.cpp b/src/prefill/experts.cpp similarity index 54% rename from src/prefill/offload.cpp rename to src/prefill/experts.cpp index 6b6653fb17f..dcf71fef5f6 100644 --- a/src/prefill/offload.cpp +++ b/src/prefill/experts.cpp @@ -1,5 +1,5 @@ -// src/prefill/offload.cpp - see include/strata/prefill/offload.hpp. -#include "strata/prefill/offload.hpp" +// src/prefill/experts.cpp - see include/strata/prefill/experts.hpp. +#include "strata/prefill/experts.hpp" #include "strata/kernels/cpu/expert_layout.hpp" #include "strata/kernels/iq_kernels.hpp" @@ -8,66 +8,57 @@ #include #include -#include namespace strata::prefill { namespace { -constexpr int64_t N = 2560, FF2 = 1280, FF = 640; -// Experts go through in batches of consecutive ids, at most G experts and ROWS rows (an expert with more rows is a -// batch of its own, in pieces of ROWS): one staging copy per run of adjacent arena blobs, one dequant launch, one -// gather, two grouped GEMMs, one SwiGLU and one scatter-add per batch. Per-expert launches made the host, not the -// GPU, the limit (~10 calls for each of ~500 experts a layer). +constexpr int64_t N = 2560, FF2 = 1280, FF = 640, K = 10; +// Batches of consecutive ids, at most G experts and ROWS rows (an expert with more rows is a batch of its own, in +// pieces of ROWS). Per-expert launches made the host, not the GPU, the limit (~10 calls for each of ~500 experts a +// layer). constexpr int G = 16; constexpr int64_t ROWS = 4096; constexpr int SLOTS = 2; // batches in staging: the copies of one overlap the work on the other constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace constexpr int NPTR = 7; // per-expert device pointers: blob, xs, gate/up, gu, h, down, d -struct DeviceScope { // the second GPU current for one call, the main one again afterwards +struct OnDevice { // a remote runner's GPU current for one call, the main one again afterwards int main; - DeviceScope(int dev, int main_dev) : main(main_dev) { cudaSetDevice(dev); } - ~DeviceScope() { cudaSetDevice(main); } + bool set; + OnDevice(int dev, int main_dev, bool remote) : main(main_dev), set(remote) { if (set) cudaSetDevice(dev); } + ~OnDevice() { if (set) cudaSetDevice(main); } }; -// Bump allocation from a borrowed region, else cudaMalloc; with neither it only counts (bytes_needed). +// Bump allocation from a region; without one it only counts (bytes_needed). struct Alloc { uint8_t* base = nullptr; uint64_t cap = 0, used = 0; - bool count_only = false; - std::vector* owned = nullptr; template T* take(size_t n, bool& ok) { const uint64_t bytes = ((uint64_t) n * sizeof(T) + 255) & ~255ull; - if (count_only) { used += bytes; return nullptr; } - if (base != nullptr) { - if (used + bytes > cap) { ok = false; return nullptr; } - T* p = (T*) (base + used); - used += bytes; - return p; - } - void* p = nullptr; - if (cudaMalloc(&p, bytes) != cudaSuccess) { ok = false; return nullptr; } - owned->push_back(p); + T* p = base != nullptr ? (T*) (base + used) : nullptr; used += bytes; - return (T*) p; + if (base != nullptr && used > cap) ok = false; + return p; } }; +size_t blob_cap() { return ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; } + } // namespace -struct Offload::Impl { +struct ExpertRunner::Impl { int dev = -1, main = 0; + bool remote = false, streams = false; core::ExpertSource* src = nullptr; const core::ExpertCache* cache = nullptr; const int32_t* res = nullptr; - int64_t n_expert = 0, chunk = 0; - size_t blob_cap = 0; // staging bytes per expert + int64_t n_expert = 0, max_chunk = 0, T = 0; // T: the chunk the buffers are bound for cudaStream_t s = nullptr, copy = nullptr; cudaEvent_t done = nullptr, copied[SLOTS] = {}, used[SLOTS] = {}; - cudaEvent_t t0 = nullptr, t1 = nullptr; // the previous layer's GPU time, read at the next one + cudaEvent_t t0 = nullptr, t1 = nullptr; // the previous layer's GPU time (remote), read at the next one bool timed = false, live[SLOTS] = {}; Gemm gemm; - std::vector owned; + void* owned = nullptr; // the buffers' own allocation (no region) uint16_t *input = nullptr, *xs = nullptr, *hh = nullptr, *dq_gu = nullptr, *dq_d = nullptr; float *sum = nullptr, *w = nullptr, *gu = nullptr, *d = nullptr; int32_t* rows_src = nullptr; @@ -80,37 +71,38 @@ struct Offload::Impl { float* h_sum = nullptr; // the same sequence counted (bytes_needed) or carved - bool carve(Alloc& a, int64_t T) { - bool ok = true; + static void carve(Alloc& a, int64_t T, int64_t n_expert, bool remote, bool streams, Impl* m, void** ws, bool& ok) { const size_t t = (size_t) T; - input = a.take(t * N, ok); - sum = a.take(t * N, ok); - rows_src = a.take(t * 10, ok); - w = a.take(t * 10, ok); - adds = a.take(t * 10 * 3 + (size_t) n_expert * 2, ok); - xs = a.take((size_t) ROWS * N, ok); - gu = a.take((size_t) ROWS * FF2, ok); - hh = a.take((size_t) ROWS * FF, ok); - d = a.take((size_t) ROWS * N, ok); - dq_gu = a.take((size_t) G * FF2 * N, ok); - dq_d = a.take((size_t) G * N * FF, ok); - for (auto& p : stage) p = a.take((size_t) G * blob_cap, ok); - ptrs = a.take((size_t) NPTR * (size_t) n_expert, ok); - void* ws = a.take(GEMM_WS, ok); - if (ok && !a.count_only) { - std::string err; - ok = gemm.init_external(s, nullptr, 0, ws, GEMM_WS, err); - } - return ok; + uint16_t* input = remote ? a.take(t * N, ok) : nullptr; + float* sum = remote ? a.take(t * N, ok) : nullptr; + int32_t* rows_src = a.take(t * K, ok); + float* w = a.take(t * K, ok); + int32_t* adds = a.take(t * K * 3 + (size_t) n_expert * 2, ok); + uint16_t* xs = a.take((size_t) ROWS * N, ok); + float* gu = a.take((size_t) ROWS * FF2, ok); + uint16_t* hh = a.take((size_t) ROWS * FF, ok); + float* d = a.take((size_t) ROWS * N, ok); + uint16_t* dq_gu = a.take((size_t) G * FF2 * N, ok); + uint16_t* dq_d = a.take((size_t) G * N * FF, ok); + uint8_t* stage[SLOTS] = {}; + if (streams) + for (auto& p : stage) p = a.take((size_t) G * blob_cap(), ok); + void** ptrs = a.take((size_t) NPTR * (size_t) n_expert, ok); + *ws = a.take(GEMM_WS, ok); + if (m == nullptr) return; + m->input = input; m->sum = sum; m->rows_src = rows_src; m->w = w; m->adds = adds; + m->xs = xs; m->gu = gu; m->hh = hh; m->d = d; m->dq_gu = dq_gu; m->dq_d = dq_d; + for (int i = 0; i < SLOTS; ++i) m->stage[i] = stage[i]; + m->ptrs = ptrs; } }; -Offload::Offload() : impl_(new Impl) {} +ExpertRunner::ExpertRunner() : impl_(new Impl) {} -Offload::~Offload() { +ExpertRunner::~ExpertRunner() { Impl& m = *impl_; if (m.dev < 0) return; - DeviceScope scope(m.dev, m.main); + OnDevice on(m.dev, m.main, m.remote); if (m.s) cudaStreamSynchronize(m.s); if (m.copy) cudaStreamSynchronize(m.copy); for (int i = 0; i < SLOTS; ++i) { @@ -120,91 +112,123 @@ Offload::~Offload() { for (cudaEvent_t e : {m.done, m.t0, m.t1}) if (e) cudaEventDestroy(e); if (m.h_input) cudaFreeHost(m.h_input); if (m.h_sum) cudaFreeHost(m.h_sum); - for (void* p : m.owned) cudaFree(p); + if (m.owned) cudaFree(m.owned); if (m.copy) cudaStreamDestroy(m.copy); - if (m.s) cudaStreamDestroy(m.s); + if (m.remote && m.s) cudaStreamDestroy(m.s); } -uint64_t Offload::bytes_needed(int64_t chunk, int64_t n_expert) { - Impl m; - m.n_expert = n_expert; - m.blob_cap = ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; +uint64_t ExpertRunner::bytes_needed(int64_t chunk, int64_t n_expert, bool remote, bool streams) { Alloc a; - a.count_only = true; - m.carve(a, chunk); - return a.used + (1u << 20); + bool ok = true; + void* ws = nullptr; + Impl::carve(a, chunk, n_expert, remote, streams, nullptr, &ws, ok); + return a.used; } -bool Offload::init(int device, int main_device, core::ExpertSource* src, const core::ExpertCache* cache, - const int32_t* res, int64_t n_expert, int64_t chunk, std::string& err, void* borrow, - uint64_t borrow_bytes) { +bool ExpertRunner::init(int device, int main_device, void* stream, core::ExpertSource* src, const core::ExpertCache* cache, + const int32_t* res, int64_t n_expert, int64_t max_chunk, bool streams, std::string& err) { Impl& m = *impl_; - if (!strata::kernels::cpu::expert_layout().native) { - err = "prompt offload: needs a native pack"; + const auto& lay = strata::kernels::cpu::expert_layout(); + if (stream == nullptr && !lay.native) { + err = "prompt experts on a second GPU: need a native pack"; return false; } m.dev = device; m.main = main_device; + m.remote = stream == nullptr; + m.streams = streams; m.src = src; m.cache = cache; m.res = res; m.n_expert = n_expert; - m.chunk = chunk; - m.blob_cap = ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; + m.max_chunk = max_chunk; m.ptrs_host.resize((size_t) NPTR * (size_t) n_expert); - m.adds_host.reserve((size_t) chunk * 30); - m.mark.assign((size_t) chunk, -1); - m.first.resize((size_t) chunk); - m.count.resize((size_t) chunk); - DeviceScope scope(device, main_device); - bool ok = cudaStreamCreateWithFlags(&m.s, cudaStreamNonBlocking) == cudaSuccess && - cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) == cudaSuccess && - cudaEventCreateWithFlags(&m.done, cudaEventDisableTiming) == cudaSuccess && + m.adds_host.reserve((size_t) max_chunk * 30); + m.mark.assign((size_t) max_chunk, -1); + m.first.resize((size_t) max_chunk); + m.count.resize((size_t) max_chunk); + OnDevice on(device, main_device, m.remote); + bool ok = cudaEventCreateWithFlags(&m.done, cudaEventDisableTiming) == cudaSuccess && cudaEventCreate(&m.t0) == cudaSuccess && cudaEventCreate(&m.t1) == cudaSuccess; - for (int i = 0; ok && i < SLOTS; ++i) - ok = cudaEventCreateWithFlags(&m.copied[i], cudaEventDisableTiming) == cudaSuccess && - cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) == cudaSuccess; - if (!ok) { err = "prompt offload: streams or events"; return false; } + if (m.remote) ok = ok && cudaStreamCreateWithFlags(&m.s, cudaStreamNonBlocking) == cudaSuccess; + else m.s = (cudaStream_t) stream; + if (streams) { + ok = ok && cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) == cudaSuccess; + for (int i = 0; ok && i < SLOTS; ++i) + ok = cudaEventCreateWithFlags(&m.copied[i], cudaEventDisableTiming) == cudaSuccess && + cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) == cudaSuccess; + } + if (!ok || !m.gemm.init(m.s, err)) { + if (err.empty()) err = "prompt experts: streams or events"; + return false; + } // the input and the sums: pinned and portable, both GPUs copy them - if (cudaHostAlloc((void**) &m.h_input, (size_t) chunk * N * 2, cudaHostAllocPortable) != cudaSuccess || - cudaHostAlloc((void**) &m.h_sum, (size_t) chunk * N * 4, cudaHostAllocPortable) != cudaSuccess) { - err = "prompt offload: pinned input and sum buffers"; + if (m.remote && (cudaHostAlloc((void**) &m.h_input, (size_t) max_chunk * N * 2, cudaHostAllocPortable) != cudaSuccess || + cudaHostAlloc((void**) &m.h_sum, (size_t) max_chunk * N * 4, cudaHostAllocPortable) != cudaSuccess)) { + err = "prompt experts: pinned input and sum buffers"; + return false; + } + return true; +} + +bool ExpertRunner::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err) { + Impl& m = *impl_; + OnDevice on(m.dev, m.main, m.remote); + if (region == nullptr) { + if (m.owned != nullptr) return true; + chunk = m.max_chunk; + bytes = bytes_needed(chunk, m.n_expert, m.remote, m.streams); + if (cudaMalloc(&m.owned, bytes) != cudaSuccess) { + m.owned = nullptr; + err = "prompt experts: " + std::to_string(bytes >> 20) + " MiB of device buffers"; + return false; + } + region = m.owned; + } + if (chunk > m.max_chunk) { + err = "prompt experts: a chunk longer than the one they were set up for"; return false; } Alloc a; - a.base = (uint8_t*) borrow; - a.cap = borrow_bytes; - a.owned = &m.owned; - if (!m.carve(a, chunk)) { - err = "prompt offload: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; + a.base = (uint8_t*) region; + a.cap = bytes; + bool ok = true; + void* ws = nullptr; + Impl::carve(a, chunk, m.n_expert, m.remote, m.streams, &m, &ws, ok); + if (!ok) { + err = "prompt experts: their buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; return false; } + m.gemm.set_buffers(nullptr, 0, ws, GEMM_WS); + m.T = chunk; return true; } -uint16_t* Offload::host_input() const { return impl_->h_input; } -float* Offload::host_sum() const { return impl_->h_sum; } -cudaEvent_t Offload::done() const { return impl_->done; } +uint16_t* ExpertRunner::host_input() const { return impl_->h_input; } +float* ExpertRunner::host_sum() const { return impl_->h_sum; } +cudaEvent_t ExpertRunner::done() const { return impl_->done; } -bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, - const std::vector& src, const std::vector& w, cudaEvent_t input_ready, - std::string& err) { +bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, + const std::vector& src, const std::vector& w, cudaEvent_t input_ready, + const uint16_t* input, float* sum, std::string& err) { Impl& m = *impl_; const size_t n_exp = experts.size(), n_rows = src.size(); - if (T > m.chunk || off.size() != n_exp + 1 || w.size() != n_rows || (size_t) off.back() != n_rows || - n_rows > (size_t) T * 10 || n_exp > (size_t) m.n_expert) { - err = "prompt offload: a layer's share is out of range"; + if (T > m.T || off.size() != n_exp + 1 || w.size() != n_rows || (size_t) off.back() != n_rows || + n_rows > (size_t) T * K || n_exp > (size_t) m.n_expert || (!m.remote && (input == nullptr || sum == nullptr))) { + err = "prompt experts: a layer's share is out of range"; return false; } - DeviceScope scope(m.dev, m.main); + OnDevice on(m.dev, m.main, m.remote); const auto h0 = std::chrono::steady_clock::now(); - if (m.timed) { // the previous layer is done by now (the main GPU waited for it) + if (m.timed && cudaEventQuery(m.t1) == cudaSuccess) { // the previous layer (the main GPU waited for it) float ms = 0; cudaEventElapsedTime(&ms, m.t0, m.t1); ms_gpu += ms; } - const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; - const size_t bb = (size_t) strata::kernels::cpu::expert_layout().blob_bytes(layer); + m.timed = false; + const auto& lay = strata::kernels::cpu::expert_layout(); + const auto& f = lay.fmt.empty() ? strata::kernels::cpu::NativeFmt{} : lay.fmt[(size_t) layer]; + const size_t bb = (size_t) lay.blob_bytes(layer); const size_t stride = (bb + 255) & ~(size_t) 255; // staging: a run of arena blobs lands as one copy when equal auto held = [&](int32_t e) -> int32_t { return m.res && m.cache ? m.res[(size_t) layer * (size_t) m.n_expert + (size_t) e] : -1; @@ -228,6 +252,10 @@ bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& ex for (size_t j = j0; j < j1; ++j) { const int64_t r = off[j] - off[j0]; const int32_t slot = held(experts[j]); + if (slot < 0 && !m.streams) { + err = "prompt experts: an expert its cache does not hold, and no staging"; + return false; + } P[0 * E + j] = slot >= 0 ? (void*) m.cache->device_slot(slot) : (void*) (m.stage[b % SLOTS] + streamed++ * stride); P[1 * E + j] = m.xs + r * N; P[2 * E + j] = m.dq_gu + (j - j0) * (size_t) (FF2 * N); @@ -272,18 +300,22 @@ bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& ex for (int32_t r = r0; r < r1; ++r) h_list[m.count[(size_t) m.first[(size_t) src[(size_t) r]]]++] = r; for (int32_t t : toks) m.mark[(size_t) t] = -1; } - cudaStreamWaitEvent(m.s, input_ready, 0); - cudaEventRecord(m.t0, m.s); + if (m.remote) { + cudaStreamWaitEvent(m.s, input_ready, 0); + cudaEventRecord(m.t0, m.s); + } + const uint16_t* in = m.remote ? m.input : input; + float* out = m.remote ? m.sum : sum; if (!m.adds_host.empty()) cudaMemcpyAsync(m.adds, m.adds_host.data(), m.adds_host.size() * 4, cudaMemcpyHostToDevice, m.s); // pageable sources: the copies are staged before they return, so the caller may reuse the vectors cudaMemcpyAsync(m.ptrs, P, NPTR * E * sizeof(void*), cudaMemcpyHostToDevice, m.s); - cudaMemcpyAsync(m.input, m.h_input, (size_t) T * N * 2, cudaMemcpyHostToDevice, m.s); + if (m.remote) cudaMemcpyAsync(m.input, m.h_input, (size_t) T * N * 2, cudaMemcpyHostToDevice, m.s); if (n_rows > 0) { cudaMemcpyAsync(m.rows_src, src.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); cudaMemcpyAsync(m.w, w.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); } - cudaMemsetAsync(m.sum, 0, (size_t) T * N * 4, m.s); + if (m.remote) cudaMemsetAsync(m.sum, 0, (size_t) T * N * 4, m.s); // ---- staging: the streamed experts of batch b into slot b % SLOTS, a copy per run of adjacent arena blobs auto stage_batch = [&](size_t b) { const auto [j0, j1] = batches[b]; @@ -320,23 +352,27 @@ bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& ex const int n = (int) (j1 - j0); const int sl = (int) (b % SLOTS); if (has_stream[b]) cudaStreamWaitEvent(m.s, m.copied[sl], 0); - strata::kernels::iq_dequant_experts_f16(f.gu_type, f.d_type, (const uint8_t* const*) (m.ptrs + 0 * E + j0), n, - f.up_off, f.down_off, f.n_ff, f.n_embd, m.dq_gu, m.dq_d, m.s); + if (lay.native) + strata::kernels::iq_dequant_experts_f16(f.gu_type, f.d_type, (const uint8_t* const*) (m.ptrs + 0 * E + j0), n, + f.up_off, f.down_off, f.n_ff, f.n_embd, m.dq_gu, m.dq_d, m.s); + else + for (size_t j = j0; j < j1; ++j) + blob_dequant_f16((const uint8_t*) P[0 * E + j], (uint16_t*) P[2 * E + j], (uint16_t*) P[5 * E + j], m.s); if (has_stream[b]) cudaEventRecord(m.used[sl], m.s); for (size_t j = j0; j < j1; ++j) if (held(experts[j]) >= 0) ++experts_resident; const int64_t r0 = off[j0], rows = off[j1] - r0; if (rows > ROWS) { // one expert: its rows in pieces for (int64_t p = 0; p < rows; p += ROWS) { const int64_t nr = std::min(ROWS, rows - p); - gather_rows16(m.input, m.rows_src + r0 + p, m.xs, nr, N, m.s); + gather_rows16(in, m.rows_src + r0 + p, m.xs, nr, N, m.s); m.gemm.f16(m.xs, m.dq_gu, m.gu, nr, FF2, N); swiglu_interleaved(m.gu, m.hh, nr, m.s); m.gemm.f16(m.hh, m.dq_d, m.d, nr, N, FF); - moe_scatter_add(m.sum, m.d, m.w + r0 + p, m.rows_src + r0 + p, nr, m.s); + moe_scatter_add(out, m.d, m.w + r0 + p, m.rows_src + r0 + p, nr, m.s); } continue; } - gather_rows16(m.input, m.rows_src + r0, m.xs, rows, N, m.s); + gather_rows16(in, m.rows_src + r0, m.xs, rows, N, m.s); m.gemm.f16_grouped(dp + 1 * E + j0, dp + 2 * E + j0, (float* const*) (m.ptrs + 3 * E + j0), (const uint16_t* const*) (P + 1 * E + j0), (const uint16_t* const*) (P + 2 * E + j0), (float* const*) (P + 3 * E + j0), rows_of.data() + j0, n, FF2, N); @@ -345,15 +381,17 @@ bool Offload::run_layer(int64_t layer, int64_t T, const std::vector& ex (const uint16_t* const*) (P + 4 * E + j0), (const uint16_t* const*) (P + 5 * E + j0), (float* const*) (P + 6 * E + j0), rows_of.data() + j0, n, N, FF); const int32_t* a = m.adds + adds_at[b]; - moe_gather_add(m.sum, m.d, r0, m.w, a, a + ntok[b], a + 2 * ntok[b] + 1, ntok[b], m.s); + moe_gather_add(out, m.d, r0, m.w, a, a + ntok[b], a + 2 * ntok[b] + 1, ntok[b], m.s); + } + if (m.remote) { + cudaMemcpyAsync(m.h_sum, m.sum, (size_t) T * N * 4, cudaMemcpyDeviceToHost, m.s); + cudaEventRecord(m.done, m.s); + cudaEventRecord(m.t1, m.s); + m.timed = true; } - cudaMemcpyAsync(m.h_sum, m.sum, (size_t) T * N * 4, cudaMemcpyDeviceToHost, m.s); - cudaEventRecord(m.done, m.s); - cudaEventRecord(m.t1, m.s); - m.timed = true; ms_host += std::chrono::duration(std::chrono::steady_clock::now() - h0).count(); const cudaError_t e = cudaGetLastError(); - if (e != cudaSuccess) { err = std::string("prompt offload: ") + cudaGetErrorString(e); return false; } + if (e != cudaSuccess) { err = std::string("prompt experts: ") + cudaGetErrorString(e); return false; } return true; } diff --git a/src/prefill/gemm.cu b/src/prefill/gemm.cu index ccc4d35c58a..c85446a4e31 100644 --- a/src/prefill/gemm.cu +++ b/src/prefill/gemm.cu @@ -23,45 +23,23 @@ void ck(cublasStatus_t s, const char* what) { Gemm::~Gemm() { if (handle_) cublasDestroy((cublasHandle_t) handle_); - if (!external_) { - if (scratch_) cudaFree(scratch_); - if (workspace_) cudaFree(workspace_); - } } -bool Gemm::init_external(void* stream, uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes, - std::string& err) { +bool Gemm::init(void* stream, std::string& err) { cublasHandle_t h = nullptr; if (cublasCreate(&h) != CUBLAS_STATUS_SUCCESS) { err = "prefill gemm: cublasCreate failed"; return false; } handle_ = h; stream_ = stream; - external_ = true; cublasSetStream(h, (cudaStream_t) stream); - workspace_ = workspace; - cublasSetWorkspace(h, workspace_, ws_bytes); cublasSetMathMode(h, CUBLAS_DEFAULT_MATH); - scratch_ = scratch; - scratch_elems_ = scratch_elems; return true; } -bool Gemm::init(void* stream, int64_t scratch_elems, std::string& err) { - cublasHandle_t h = nullptr; - if (cublasCreate(&h) != CUBLAS_STATUS_SUCCESS) { err = "prefill gemm: cublasCreate failed"; return false; } - handle_ = h; - stream_ = stream; - cublasSetStream(h, (cudaStream_t) stream); - // A fixed workspace so the handle never allocates on the way (and graphs could capture it later). - const size_t ws = 32u << 20; - if (cudaMalloc(&workspace_, ws) != cudaSuccess) { err = "prefill gemm: workspace"; return false; } - cublasSetWorkspace(h, workspace_, ws); - cublasSetMathMode(h, CUBLAS_DEFAULT_MATH); - if (scratch_elems > 0 && cudaMalloc((void**) &scratch_, (size_t) scratch_elems * 2) != cudaSuccess) { - err = "prefill gemm: dequant scratch of " + std::to_string(scratch_elems * 2 >> 20) + " MiB"; - return false; - } +void Gemm::set_buffers(uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes) { + // a fixed workspace, so the handle never allocates on the way + cublasSetWorkspace((cublasHandle_t) handle_, workspace, ws_bytes); + scratch_ = scratch; scratch_elems_ = scratch_elems; - return true; } void Gemm::bf16(const uint16_t* X, const uint16_t* W, float* Y, int64_t T, int64_t N, int64_t K, int64_t ldy, diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index 1021637d548..724771bb512 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -89,16 +89,17 @@ __global__ void gr_mix_kernel(const float* __restrict__ xn, const float* __restr s = fmaf(xn[j], sigm(g[j]), s); } s /= (float) HC; - mixed[i] = s; + if (mixed) mixed[i] = s; if (mixed16) mixed16[i] = bf(s); if (mixed_h) mixed_h[i] = hf(s); } __global__ void gr_write_kernel(float* __restrict__ R, const float* __restrict__ bo, const float* __restrict__ inj, - int64_t inj_ld, int64_t T) { + int64_t inj_ld, int64_t T, const float* __restrict__ partial) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= T * D) return; const int64_t t = i / D, c = (i % D) / N, d = i % N; - R[i] = fmaf(bo[t * N + d], 2.0f * sigm(inj[t * inj_ld + c] / (float) HC), R[i]); + const float b = partial ? bo[t * N + d] + partial[t * N + d] : bo[t * N + d]; + R[i] = fmaf(b, 2.0f * sigm(inj[t * inj_ld + c] / (float) HC), R[i]); } __global__ void gr_broadcast_kernel(const float* __restrict__ e, float* __restrict__ R, int64_t T) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; @@ -346,21 +347,10 @@ __global__ void gather_rows16_kernel(const uint16_t* __restrict__ x, const int32 const int64_t r = i / per, j = i % per; reinterpret_cast(dst)[r * per + j] = reinterpret_cast(x)[(int64_t) src[r] * per + j]; } -__global__ void moe_combine_kernel(const float* __restrict__ Dm, const int32_t* __restrict__ slot, - const float* __restrict__ w, const float* __restrict__ shared, - const float* __restrict__ sg, const float* __restrict__ partial, - float* __restrict__ bo, int64_t T) { +__global__ void moe_shared_gate_kernel(float* __restrict__ x, const float* __restrict__ sg, int64_t T) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= T * N) return; - const int64_t t = i / N, d = i % N; - float s = 0.0f; -#pragma unroll - for (int k = 0; k < 10; ++k) { - const int32_t r = slot[t * 10 + k]; - if (r >= 0) s = fmaf(w[t * 10 + k], Dm[(int64_t) r * N + d], s); - } - const float v = s + shared[i] * sigm(sg[t]); - bo[i] = partial ? v + partial[i] : v; + x[i] = x[i] * sigm(sg[i / N]); } __global__ void moe_scatter_add_kernel(float* __restrict__ sum, const float* __restrict__ rows, const float* __restrict__ w, const int32_t* __restrict__ src, int64_t n) { @@ -479,8 +469,8 @@ void gr_mix(const float* xn, const float* gated, float* mixed, uint16_t* mixed16 gr_mix_kernel<<>>(xn, gated, mixed, mixed16, T, mixed_h); check("gr_mix"); } -void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream) { - gr_write_kernel<<>>(R, bo, inj, inj_ld, T); +void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream, const float* partial) { + gr_write_kernel<<>>(R, bo, inj, inj_ld, T, partial); check("gr_write"); } void gr_broadcast(const float* e, float* R, int64_t T, void* stream) { @@ -538,10 +528,9 @@ void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int gather_rows16_kernel<<>>(x16, src, dst16, n, width); check("gather_rows16"); } -void moe_combine(const float* Dm, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo, - int64_t T, void* stream, const float* partial) { - moe_combine_kernel<<>>(Dm, slot, w, shared, sg, partial, bo, T); - check("moe_combine"); +void moe_shared_gate(float* x, const float* sg, int64_t T, void* stream) { + moe_shared_gate_kernel<<>>(x, sg, T); + check("moe_shared_gate"); } __global__ void moe_gather_add_kernel(float* __restrict__ sum, const float* __restrict__ rows, int64_t r0, const float* __restrict__ w, const int32_t* __restrict__ tok, diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 8c288d23848..a1193591dd1 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -2,26 +2,21 @@ #include "strata/prefill/prefill.hpp" #include "strata/core/layout.hpp" -#include "strata/kernels/cpu/expert.hpp" +#include "strata/kernels/dequant_bf16.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_ple_postops.hpp" #include "strata/kernels/ngram.hpp" -#include "strata/kernels/iq_kernels.hpp" -#include "strata/kernels/cpu/expert_layout.hpp" #include "strata/kernels/qsa.hpp" #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/qsa_select.hpp" #include "strata/prefill/gemm.hpp" #include "strata/prefill/kernels.hpp" -#include "strata/prefill/offload.hpp" #include #include #include #include -#include -#include #include namespace strata::prefill { @@ -31,210 +26,229 @@ using Clock = std::chrono::steady_clock; constexpr float EPS = 1e-6f; constexpr int64_t N = 2560, HC = 4, D = N * HC, LR = 320, K = 10, NE = 512; constexpr int64_t C = 10240, ZV = 6144, HV = 48; -// plan v0.3 P6: staging holds the largest blob of the pack (a native pack's blobs differ per layer) -inline int64_t MAXBLOB() { return (int64_t) strata::kernels::cpu::expert_layout().max_blob; } -constexpr int STAGE = 8; // host->device expert staging ring -constexpr int DQ = 2; // dequantized-expert ring (FP16 gate/up + down) +constexpr int64_t SUB = 2048; // tokens of a sub-chunk: everything but the routed experts +constexpr int64_t SEL_BATCH = 256; // queries per block-score launch +// A layer's quantized projections, dequantized to FP16 once for all its sub-chunks: the PLE key, or the mixer's +// (GDN: qkv, gate, out; QSA: q, k, v, out) and the shared expert's +constexpr int64_t W16 = std::max(D * N, std::max(C + ZV + ZV, 12288 + 512 + 512 + ZV) * N + 3 * 640 * N); +constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace double ms_since(Clock::time_point t) { return std::chrono::duration(Clock::now() - t).count(); } -// Either cudaMalloc (owned, freed with the object) or a bump allocation from a borrowed region; with no base and -// no region it only counts, which is how `bytes_needed` sizes the region. +// Bump allocation from a region; without one it only counts (bytes_needed). struct Alloc { uint8_t* base = nullptr; uint64_t cap = 0, used = 0; - bool count_only = false; - std::vector* owned = nullptr; - template T* take(size_t n, bool& ok) { + bool ok = true; + template T* take(size_t n) { const uint64_t bytes = ((uint64_t) n * sizeof(T) + 256 + 255) & ~255ull; - if (count_only) { used += bytes; return nullptr; } - if (base != nullptr) { - if (used + bytes > cap) { ok = false; return nullptr; } - T* p = (T*) (base + used); - used += bytes; - return p; - } - void* p = nullptr; - if (cudaMalloc(&p, bytes) != cudaSuccess) { ok = false; return nullptr; } - owned->push_back(p); + T* p = base != nullptr ? (T*) (base + used) : nullptr; used += bytes; - return (T*) p; + if (base != nullptr && used > cap) ok = false; + return p; } }; +// The device buffers of one binding. +struct Bufs { + uint16_t* w16 = nullptr; + void* gemm_ws = nullptr; + // the chunk's: residual stream, the MoE's input (FP16) and output, the FFN half's injection, routing, steps + float *R = nullptr, *bo_moe = nullptr, *inj_f = nullptr, *w = nullptr; + uint16_t* mixed_h = nullptr; + int32_t *ids = nullptr, *steps = nullptr; + // a sub-chunk's: the mixer's BF16 input, its output, the mixer half's injection + uint16_t* mixed_bf = nullptr; + float *bo = nullptr, *inj_a = nullptr; + // the union: what one step needs at a time + float* emb = nullptr; + float *xn = nullptr, *lo = nullptr, *gated = nullptr; + uint16_t *xn16 = nullptr, *lo16 = nullptr; + float *ple_emb = nullptr, *ple_kp = nullptr, *ple_key = nullptr, *ple_q = nullptr, *ple_gated = nullptr; + float *qkv = nullptr, *z = nullptr, *ab = nullptr, *gate = nullptr, *beta = nullptr, *hbuf = nullptr, *y = nullptr; + uint16_t* y_h = nullptr; + float *Kc = nullptr, *Vc = nullptr, *Qf = nullptr, *q = nullptr, *idx_raw = nullptr, *q_idx = nullptr, + *attn = nullptr, *sel_scores = nullptr; + uint16_t* attn_h = nullptr; + int32_t* sel_ids = nullptr; + float *logits = nullptr, *sgate = nullptr, *sup = nullptr, *sg = nullptr; + uint16_t* sh_h = nullptr; + float* sums = nullptr; // the second GPU's sums + uint8_t* local = nullptr; // this GPU's expert runner + uint64_t local_bytes = 0; +}; + +// Chunks of T tokens: the chunk-long buffers, the sub-chunk ones, then the union of the steps a layer takes one after +// the other on the stream. (Staging copies run ahead on a copy stream only while the routed experts run, after the +// host has waited for the stream, and the stream waits for them before the next step.) +Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) { + const size_t t = (size_t) T, p = (size_t) std::min(T, SUB); + Bufs b; + b.w16 = a.take((size_t) W16); + b.gemm_ws = a.take(GEMM_WS); + b.R = a.take(t * D); + b.mixed_h = a.take(t * N); + b.bo_moe = a.take(t * N); + b.inj_f = a.take(t * HC); + b.ids = a.take(t * K); + b.w = a.take(t * K); + b.steps = a.take(t * strata::kernels::kStepCount); + b.mixed_bf = a.take(p * N); + b.bo = a.take(p * N); + b.inj_a = a.take(p * HC); + const uint64_t u0 = a.used; + uint64_t u1 = u0; + auto next = [&] { u1 = std::max(u1, a.used); a.used = u0; }; + b.emb = a.take(p * N); + next(); // the hyper-connection read + b.xn = a.take(p * D); b.xn16 = a.take(p * D); b.lo = a.take(p * LR); + b.lo16 = a.take(p * LR); b.gated = a.take(p * D); + next(); // the PLE block + b.ple_emb = a.take(p * N); b.ple_kp = a.take(p * D); b.ple_key = a.take(p * D); + b.ple_q = a.take(p * D); b.ple_gated = a.take(p * D); + next(); // GDN + b.qkv = a.take(p * C); b.z = a.take(p * ZV); b.ab = a.take(p * 2 * HV); + b.gate = a.take(p * HV); b.beta = a.take(p * HV); b.hbuf = a.take(p * C); + b.y = a.take(p * ZV); b.y_h = a.take(p * ZV); + next(); // QSA + b.Kc = a.take(p * 512); b.Vc = a.take(p * 512); b.Qf = a.take(p * 12288); + b.q = a.take(p * ZV); b.idx_raw = a.take(p * 128); b.q_idx = a.take(p * 512); + b.attn = a.take(p * ZV); b.attn_h = a.take(p * ZV); b.sel_ids = a.take(p * (size_t) cap); + b.sel_scores = a.take((size_t) SEL_BATCH * (size_t) max_blocks); + next(); // the router and the shared expert + b.logits = a.take(p * NE); b.sgate = a.take(p * 640); b.sup = a.take(p * 640); + b.sh_h = a.take(p * 640); b.sg = a.take(p); + next(); // the routed experts + b.sums = offload ? a.take(t * N) : nullptr; + b.local_bytes = ExpertRunner::bytes_needed(T, NE, false, !offload); + b.local = a.take(b.local_bytes); + next(); + a.used = u1; + return b; +} + +strata::kernels::QsaShapes shapes_of(const core::ModelGeometry& g) { + strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes(); + s.n_head = g.n_head; s.n_head_kv = g.n_head_kv; s.head_dim = g.head_dim; s.idx_n_head = g.idx_q_heads; + s.idx_dim = g.idx_key_dim; + return s; +} + } // namespace struct Prefill::Impl { const core::WeightTable* wt = nullptr; const core::ModelGeometry* g = nullptr; core::SessionState* ss = nullptr; - core::ExpertSource* src = nullptr; const core::ExpertCache* cache = nullptr; const int32_t* host_res = nullptr; - int64_t T = 0; - cudaStream_t cs = nullptr, copy = nullptr; - Gemm gemm; - std::vector owned; - // chunk buffers - float *emb = nullptr, *R = nullptr, *xn = nullptr, *lo = nullptr, *gated = nullptr, *inj = nullptr; - uint16_t *xn16 = nullptr, *lo16 = nullptr; - float* mixed = nullptr; - uint16_t *mixed_bf = nullptr, *mixed_h = nullptr; - float* bo = nullptr; - // GDN - float *qkv = nullptr, *z = nullptr, *ab = nullptr, *gate = nullptr, *beta = nullptr, *hbuf = nullptr, *y = nullptr; - uint16_t* y_h = nullptr; - // QSA - float *Kc = nullptr, *Vc = nullptr, *Qf = nullptr, *q = nullptr, *idx_raw = nullptr, *q_idx = nullptr, *attn = nullptr; - uint16_t* attn_h = nullptr; - int32_t* steps_dev = nullptr; - std::vector steps_host; - int32_t* sel_ids = nullptr; - float* sel_scores = nullptr; // [sel_batch, max_blocks] - int64_t sel_batch = 256, max_blocks = 0; - int64_t cap = 0; - // MoE - float *logits = nullptr, *w = nullptr, *GU = nullptr, *Dm = nullptr, *sgate = nullptr, *sup = nullptr, - *shared = nullptr, *sg = nullptr; - int32_t *ids = nullptr, *slot_dev = nullptr, *src_dev = nullptr; - uint16_t *Xs = nullptr, *Hh = nullptr, *sh_h = nullptr; - std::vector ids_host, slot_host, src_host, cnt, off; - // the second GPU's share (Prefill::offload): the input goes out and the sums come back on `xfer` + ExpertRunner* offload = nullptr; + ExpertRunner local; + int64_t max_chunk = 0, T = 0; // T: the chunk the buffers are bound for + strata::kernels::QsaShapes s; + int64_t cap = 0, max_blocks = 0; + cudaStream_t cs = nullptr; + // the second GPU's share: the input goes out and the sums come back on `xfer` cudaStream_t xfer = nullptr; cudaEvent_t ev_mixed = nullptr, ev_input = nullptr, ev_sum = nullptr; - std::vector entry_of, experts2, off2, src2; - std::vector w_host, w2; - uint16_t* dq_gu[DQ] = {}; - uint16_t* dq_d[DQ] = {}; - uint8_t* stage_dev[STAGE] = {}; - uint8_t* stage_host[STAGE] = {}; - cudaEvent_t copied[STAGE] = {}, used[STAGE] = {}; - bool stage_live[STAGE] = {}; - // PLE - float* ple_emb = nullptr; + Gemm gemm; + void* owned = nullptr; // the buffers' own allocation (no region) + Bufs b; + std::vector> w16_have; // this layer's dequantized weights + int64_t w16_used = 0; + int32_t* ids_host = nullptr; // pinned [max_chunk, K] + float* w_host = nullptr; + std::vector steps_host, cnt, off, entry_of; + std::vector ex1, off1, src1, ex2, off2, src2; // this GPU's experts, the second GPU's + std::vector w1, w2; std::vector ple_emb_host; std::vector ple_rows; - PrefillStats* stats = nullptr; }; Prefill::Prefill() : impl_(new Impl) {} Prefill::~Prefill() { if (!impl_) return; - if (impl_->cs) cudaStreamSynchronize(impl_->cs); - for (int i = 0; i < STAGE; ++i) { - if (impl_->copied[i]) cudaEventDestroy(impl_->copied[i]); - if (impl_->used[i]) cudaEventDestroy(impl_->used[i]); - if (impl_->stage_host[i]) cudaFreeHost(impl_->stage_host[i]); - } - if (impl_->copy) cudaStreamDestroy(impl_->copy); - if (impl_->xfer) cudaStreamSynchronize(impl_->xfer); - for (cudaEvent_t e : {impl_->ev_mixed, impl_->ev_input, impl_->ev_sum}) if (e) cudaEventDestroy(e); - if (impl_->xfer) cudaStreamDestroy(impl_->xfer); - for (void* p : impl_->owned) cudaFree(p); -} - -namespace { -constexpr int64_t GEMM_SCRATCH = 32ll << 20; // FP16 elements for the largest dequantized dense weight -constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace + Impl& m = *impl_; + if (m.cs) cudaStreamSynchronize(m.cs); + if (m.xfer) cudaStreamSynchronize(m.xfer); + for (cudaEvent_t e : {m.ev_mixed, m.ev_input, m.ev_sum}) if (e) cudaEventDestroy(e); + if (m.xfer) cudaStreamDestroy(m.xfer); + if (m.ids_host) cudaFreeHost(m.ids_host); + if (m.w_host) cudaFreeHost(m.w_host); + if (m.owned) cudaFree(m.owned); } bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, core::SessionState& ss, - core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* host_res, int64_t chunk, - void* stream, std::string& err, void* borrow, uint64_t borrow_bytes) { + core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* host_res, int64_t max_chunk, + void* stream, ExpertRunner* offload, std::string& err) { Impl& m = *impl_; - m.wt = &wt; m.g = &g; m.ss = &ss; m.src = src; m.cache = cache; m.host_res = host_res; - m.T = chunk; m.cs = (cudaStream_t) stream; m.stats = &stats_; + m.wt = &wt; m.g = &g; m.ss = &ss; m.cache = cache; m.host_res = host_res; m.offload = offload; + m.max_chunk = max_chunk; m.cs = (cudaStream_t) stream; if (g.n_embd != N || g.hc != HC || g.hc_lr != LR || g.n_expert != NE || ss.k != K) { err = "prefill: geometry differs from the artifact's"; return false; } - if (cudaStreamCreateWithFlags(&m.copy, cudaStreamNonBlocking) != cudaSuccess || - cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || + m.s = shapes_of(g); + m.cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, m.s); + m.max_blocks = ss.qsa_states[0].max_cells / m.s.idx_block + 2; + if (cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&m.ev_mixed, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&m.ev_input, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&m.ev_sum, cudaEventDisableTiming) != cudaSuccess) { - err = "prefill: streams and events"; + cudaEventCreateWithFlags(&m.ev_sum, cudaEventDisableTiming) != cudaSuccess || + cudaHostAlloc((void**) &m.ids_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess || + cudaHostAlloc((void**) &m.w_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess) { + err = "prefill: streams, events and pinned routing buffers"; return false; } - const size_t T = (size_t) chunk; - bool ok = true; - Alloc o; - o.base = (uint8_t*) borrow; - o.cap = borrow_bytes; - o.owned = &m.owned; - { - uint16_t* gs = o.take((size_t) GEMM_SCRATCH, ok); - void* ws = o.take(GEMM_WS, ok); - if (!ok) { err = "prefill: GEMM scratch does not fit"; return false; } - if (!m.gemm.init_external(stream, gs, GEMM_SCRATCH, ws, GEMM_WS, err)) return false; - } - m.emb = o.take(T * N, ok); m.R = o.take(T * D, ok); m.xn = o.take(T * D, ok); - m.xn16 = o.take(T * D, ok); m.lo = o.take(T * LR, ok); m.lo16 = o.take(T * LR, ok); - m.gated = o.take(T * D, ok); m.inj = o.take(T * HC, ok); - m.mixed = o.take(T * N, ok); m.mixed_bf = o.take(T * N, ok); - m.mixed_h = o.take(T * N, ok); m.bo = o.take(T * N, ok); - m.qkv = o.take(T * C, ok); m.z = o.take(T * ZV, ok); m.ab = o.take(T * 2 * HV, ok); - m.gate = o.take(T * HV, ok); m.beta = o.take(T * HV, ok); m.hbuf = o.take(T * C, ok); - m.y = o.take(T * ZV, ok); m.y_h = o.take(T * ZV, ok); - m.Kc = o.take(T * 512, ok); m.Vc = o.take(T * 512, ok); m.Qf = o.take(T * 12288, ok); - m.q = o.take(T * ZV, ok); m.idx_raw = o.take(T * 128, ok); m.q_idx = o.take(T * 512, ok); - m.attn = o.take(T * ZV, ok); m.attn_h = o.take(T * ZV, ok); - m.steps_dev = o.take(T * strata::kernels::kStepCount, ok); + const size_t T = (size_t) max_chunk; m.steps_host.resize(T * strata::kernels::kStepCount); - strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes(); - s.n_head = g.n_head; s.n_head_kv = g.n_head_kv; s.head_dim = g.head_dim; s.idx_n_head = g.idx_q_heads; - s.idx_dim = g.idx_key_dim; - m.cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, s); - m.sel_ids = o.take(T * (size_t) m.cap, ok); - m.max_blocks = ss.qsa_states[0].max_cells / s.idx_block + 2; - m.sel_scores = o.take((size_t) m.sel_batch * (size_t) m.max_blocks, ok); - m.logits = o.take(T * NE, ok); m.w = o.take(T * K, ok); m.ids = o.take(T * K, ok); - m.slot_dev = o.take(T * K, ok); m.src_dev = o.take(T * K, ok); - m.Xs = o.take(T * K * N, ok); m.GU = o.take(T * K * 1280, ok); - m.Hh = o.take(T * K * 640, ok); m.Dm = o.take(T * K * N, ok); - m.sgate = o.take(T * 640, ok); m.sup = o.take(T * 640, ok); m.sh_h = o.take(T * 640, ok); - m.shared = o.take(T * N, ok); m.sg = o.take(T, ok); - for (int i = 0; i < DQ; ++i) { m.dq_gu[i] = o.take(1280 * 2560, ok); m.dq_d[i] = o.take(2560 * 640, ok); } - for (int i = 0; i < STAGE; ++i) { - m.stage_dev[i] = o.take((size_t) MAXBLOB(), ok); - if (cudaHostAlloc((void**) &m.stage_host[i], (size_t) MAXBLOB(), cudaHostAllocDefault) != cudaSuccess) ok = false; - if (cudaEventCreateWithFlags(&m.copied[i], cudaEventDisableTiming) != cudaSuccess) ok = false; - if (cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) != cudaSuccess) ok = false; - } - m.ple_emb = o.take(T * N, ok); - m.ids_host.resize(T * K); m.slot_host.resize(T * K); m.src_host.resize(T * K); m.cnt.resize(NE); m.off.resize(NE + 1); - m.entry_of.resize(T * K); m.w_host.resize(T * K); - m.ple_emb_host.resize(T * N); m.ple_rows.resize(T * strata::kernels::PLE_N_HEADS); - if (!ok) { err = "prefill: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; return false; } - return true; + m.cnt.resize(NE); + m.off.resize(NE + 1); + m.entry_of.resize(T * K); + m.ple_emb_host.resize(T * N); + m.ple_rows.resize(T * strata::kernels::PLE_N_HEADS); + int dev = 0; + cudaGetDevice(&dev); + return m.gemm.init(stream, err) && + m.local.init(dev, dev, stream, src, cache, host_res, NE, max_chunk, offload == nullptr, err); } -uint64_t Prefill::bytes_needed(const core::ModelGeometry& g, const core::SessionState& ss, int64_t chunk) { - // the same allocation sequence as `init`, counted - (void) g; - const size_t T = (size_t) chunk; - bool ok = true; - Alloc o; - o.count_only = true; - o.take((size_t) GEMM_SCRATCH, ok); - o.take(GEMM_WS, ok); - auto f = [&](size_t n) { o.take(n, ok); }; - f(T * N); f(T * D); f(T * D); o.take(T * D, ok); f(T * LR); o.take(T * LR, ok); - f(T * D); f(T * HC); f(T * N); o.take(T * N, ok); o.take(T * N, ok); f(T * N); - f(T * C); f(T * ZV); f(T * 2 * HV); f(T * HV); f(T * HV); f(T * C); f(T * ZV); o.take(T * ZV, ok); - f(T * 512); f(T * 512); f(T * 12288); f(T * ZV); f(T * 128); f(T * 512); f(T * ZV); o.take(T * ZV, ok); - o.take(T * strata::kernels::kStepCount, ok); - strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes(); - const int64_t cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, s); - o.take(T * (size_t) cap, ok); - const int64_t max_blocks = ss.qsa_states[0].max_cells / s.idx_block + 2; - f(256 * (size_t) max_blocks); - f(T * NE); f(T * K); o.take(T * K, ok); o.take(T * K, ok); o.take(T * K, ok); - o.take(T * K * N, ok); f(T * K * 1280); o.take(T * K * 640, ok); f(T * K * N); - f(T * 640); f(T * 640); o.take(T * 640, ok); f(T * N); f(T); - for (int i = 0; i < DQ; ++i) { o.take(1280 * 2560, ok); o.take(2560 * 640, ok); } - for (int i = 0; i < STAGE; ++i) o.take((size_t) MAXBLOB(), ok); - f(T * N); - return o.used + (8u << 20); // alignment slack +uint64_t Prefill::bytes_needed(const core::ModelGeometry& g, const core::SessionState& ss, int64_t chunk, bool offload) { + const strata::kernels::QsaShapes s = shapes_of(g); + Alloc a; + layout(a, chunk, strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, s), + ss.qsa_states[0].max_cells / s.idx_block + 2, offload); + return a.used; +} + +bool Prefill::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err) { + Impl& m = *impl_; + if (region == nullptr) { + if (m.owned != nullptr) return true; // allocated once, for the longest chunk + chunk = m.max_chunk; + bytes = bytes_needed(*m.g, *m.ss, chunk, m.offload != nullptr); + if (cudaMalloc(&m.owned, bytes) != cudaSuccess) { + m.owned = nullptr; + err = "prefill: " + std::to_string(bytes >> 20) + " MiB of device buffers"; + return false; + } + region = m.owned; + } + if (chunk <= 0 || chunk > m.max_chunk) { + err = "prefill: a chunk of " + std::to_string(chunk) + " tokens (set up for " + std::to_string(m.max_chunk) + ")"; + return false; + } + Alloc a; + a.base = (uint8_t*) region; + a.cap = bytes; + m.b = layout(a, chunk, m.cap, m.max_blocks, m.offload != nullptr); + if (!a.ok) { + err = "prefill: device buffers for a chunk of " + std::to_string(chunk) + " tokens do not fit"; + return false; + } + m.gemm.set_buffers(nullptr, 0, m.b.gemm_ws, GEMM_WS); + if (!m.local.bind(m.b.local, m.b.local_bytes, chunk, err)) return false; + m.T = chunk; + return true; } namespace { @@ -244,12 +258,6 @@ const core::WeightRef* need(const core::LayerView& v, const char* suffix, std::s if (!r) err = v.name(suffix) + " is missing"; return r; } -bool native_proj(Gemm& gm, const core::WeightRef* w, const uint16_t* X, float* Y, int64_t T, const std::string& name, - std::string& err, int64_t ldy = 0) { - if (!w->native_data) { err = "prefill: " + name + " has no native GGUF blocks (run with --native)"; return false; } - gm.native(X, w->native_type, w->native_data, Y, T, w->ne1, w->ne0, ldy); - return true; -} bool bf16_proj(Gemm& gm, const core::WeightRef* w, const uint16_t* X, float* Y, int64_t T, const std::string& name, std::string& err, int64_t ldy = 0) { if (w->kind != core::WeightKind::Bf16InF32 || !w->data) { err = "prefill: " + name + " is not a resident BF16 tensor"; return false; } @@ -261,15 +269,42 @@ bool bf16_proj(Gemm& gm, const core::WeightRef* w, const uint16_t* X, float* Y, bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& err) { Impl& m = *impl_; + const Bufs& b = m.b; const core::ModelGeometry& g = *m.g; core::SessionState& ss = *m.ss; + const strata::kernels::QsaShapes& s = m.s; + constexpr int64_t SC = strata::kernels::kStepCount; + if (m.T <= 0) { err = "prefill: no buffers bound"; return false; } const auto t_start = Clock::now(); - strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes(); - s.n_head = g.n_head; s.n_head_kv = g.n_head_kv; s.head_dim = g.head_dim; s.idx_n_head = g.idx_q_heads; - s.idx_dim = g.idx_key_dim; const uint64_t gdn_floats = (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size + (uint64_t) g.ssm_conv_channels * (g.ssm_d_conv - 1); int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]}; + // Y[T, rows] = X . W^T for a quantized W [rows, cols]: dequantized at its first use in the layer + auto w16 = [&](int type, const void* blocks, int64_t rows, int64_t cols) -> const uint16_t* { + for (const auto& [k, p] : m.w16_have) + if (k == blocks) return p; + if (m.w16_used + rows * cols > W16) return nullptr; + uint16_t* p = b.w16 + m.w16_used; + strata::kernels::dequant_f16(type, blocks, 0, rows, cols, p, m.cs); + m.w16_used += (rows * cols + 127) / 128 * 128; + m.w16_have.emplace_back(blocks, p); + return p; + }; + auto native_proj = [&](const core::WeightRef* w, const uint16_t* X, float* Y, int64_t T, const std::string& name, + std::string& e) { + const uint16_t* W = w->native_data ? w16(w->native_type, w->native_data, w->ne1, w->ne0) : nullptr; + if (W == nullptr) { + e = "prefill: " + name + (w->native_data ? " does not fit the dequantization buffer" + : " has no native GGUF blocks (run with --native)"); + return false; + } + m.gemm.f16(X, W, Y, T, w->ne1, w->ne0); + return true; + }; + auto new_layer = [&] { + m.w16_have.clear(); + m.w16_used = 0; + }; // --prefill-profile: an event at the start of each section; the time to the next event is charged to it std::vector> ev; auto mark = [&](int section) { @@ -283,19 +318,22 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e for (int64_t c0 = 0; c0 < n; c0 += m.T) { const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0; ++stats_.chunks; - // ---- embeddings, broadcast to the four streams - for (int64_t t = 0; t < T; ++t) { - const float* row = embd_rows ? embd_rows[p0 + t] : nullptr; - if (row) { - if (cudaMemcpyAsync(m.emb + t * N, row, (size_t) N * 4, cudaMemcpyHostToDevice, m.cs) != cudaSuccess) { - err = "prefill: the image embedding upload failed"; + // ---- the embeddings, broadcast to the four streams + for (int64_t t0 = 0; t0 < T; t0 += SUB) { + const int64_t P = std::min(SUB, T - t0); + for (int64_t t = t0; t < t0 + P; ++t) { + const float* row = embd_rows ? embd_rows[p0 + t] : nullptr; + if (row) { + if (cudaMemcpyAsync(b.emb + (t - t0) * N, row, (size_t) N * 4, cudaMemcpyHostToDevice, m.cs) != cudaSuccess) { + err = "prefill: the image embedding upload failed"; + return false; + } + } else if (!core::embed_row(*m.wt, g, tokens[c0 + t], b.emb + (t - t0) * N, m.cs, err)) { return false; } - } else if (!core::embed_row(*m.wt, g, tokens[c0 + t], m.emb + t * N, m.cs, err)) { - return false; } + gr_broadcast(b.emb, b.R + t0 * D, P, m.cs); } - gr_broadcast(m.emb, m.R, T, m.cs); // ---- the PLE rows of the whole chunk, one batched SSD request const bool ple_on = ss.ple.ready(); if (ple_on) { @@ -307,288 +345,234 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e prev[1] = tok; } if (!ss.ple.table->gather_batch(m.ple_rows.data(), (size_t) T, m.ple_emb_host.data(), err)) return false; - cudaMemcpyAsync(m.ple_emb, m.ple_emb_host.data(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.cs); stats_.ms_ple += ms_since(tp); } else { for (int64_t t = 0; t < T; ++t) { prev[0] = prev[1]; prev[1] = (int32_t) tokens[c0 + t]; } } // ---- the QSA step records of every position in the chunk - for (int64_t t = 0; t < T; ++t) strata::kernels::qsa_step_fill(m.steps_host.data() + t * strata::kernels::kStepCount, p0 + t, s); - cudaMemcpyAsync(m.steps_dev, m.steps_host.data(), (size_t) T * strata::kernels::kStepCount * 4, - cudaMemcpyHostToDevice, m.cs); + for (int64_t t = 0; t < T; ++t) strata::kernels::qsa_step_fill(m.steps_host.data() + t * SC, p0 + t, s); + cudaMemcpyAsync(b.steps, m.steps_host.data(), (size_t) T * SC * 4, cudaMemcpyHostToDevice, m.cs); int64_t qsa_index = 0, gdn_index = 0; for (int64_t l = 0; l < g.n_layers; ++l) { const core::LayerView v(*m.wt, l); + const bool qsa = core::is_qsa_layer(g, l); + new_layer(); mark(kPsPle); - // ---- the PLE block at layer 1: the key and value projections of the chunk, then the decode path's - // per-token arithmetic (the conv reads the previous tokens' normalized rows) + // ---- the PLE block at layer 1: the key and value projections, then the decode path's per-token + // arithmetic (the conv reads the previous tokens' normalized rows) if (l == 1 && ple_on) { const strata::kernels::PleWeights& pw = ss.ple.w; - to_bf16(m.ple_emb, m.mixed_bf, T * N, m.cs); - if (pw.key_bf16 != nullptr) { - m.gemm.bf16(m.mixed_bf, pw.key_bf16, m.xn, T, D, N); - } else if (pw.key_native_data != nullptr) { - to_f16(m.ple_emb, m.mixed_h, T * N, m.cs); - m.gemm.native(m.mixed_h, pw.key_native_type, pw.key_native_data, m.xn, T, D, N); - } else { - err = "prefill: the PLE key has neither a BF16 nor a native GGUF form (run with --native)"; - return false; + for (int64_t t0 = 0; t0 < T; t0 += SUB) { + const int64_t P = std::min(SUB, T - t0); + cudaMemcpyAsync(b.ple_emb, m.ple_emb_host.data() + t0 * N, (size_t) P * N * 4, cudaMemcpyHostToDevice, m.cs); + to_bf16(b.ple_emb, b.mixed_bf, P * N, m.cs); + if (pw.key_bf16 != nullptr) { + m.gemm.bf16(b.mixed_bf, pw.key_bf16, b.ple_kp, P, D, N); + } else if (const uint16_t* W = pw.key_native_data ? w16(pw.key_native_type, pw.key_native_data, D, N) + : nullptr) { + to_f16(b.ple_emb, b.mixed_h, P * N, m.cs); + m.gemm.f16(b.mixed_h, W, b.ple_kp, P, D, N); + } else { + err = "prefill: the PLE key has neither a BF16 nor a native GGUF form (run with --native)"; + return false; + } + m.gemm.bf16(b.mixed_bf, pw.value_bf16, b.bo, P, N, N); + try { + strata::kernels::native_ple_postops_tokens( + b.ple_kp, b.R + t0 * D, b.bo, ss.ple.hist, pw, + {b.ple_key, b.ple_q, b.inj_a, b.ple_gated, b.ple_q, b.R + t0 * D}, (int) P, m.cs); + } catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; } } - m.gemm.bf16(m.mixed_bf, pw.value_bf16, m.bo, T, N, N); - try { - strata::kernels::native_ple_postops_tokens(m.xn, m.R, m.bo, ss.ple.hist, pw, - {m.qkv, m.hbuf, m.inj, m.gated, m.hbuf, m.R}, (int) T, m.cs); - } catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; } + new_layer(); } - for (int half = 0; half < 2; ++half) { - if (half == 0) mark(kPsHcRead); - // ---- the hyper-connection read of this half - const char* pre = half == 0 ? "hc_attn_" : "hc_ffn_"; - const std::string sn = std::string(pre) + "norm.weight", sd = std::string(pre) + "down.weight", - su = std::string(pre) + "up.weight", si = std::string(pre) + "inject.weight"; - const core::WeightRef *wn = need(v, sn.c_str(), err), *wd = need(v, sd.c_str(), err), - *wu = need(v, su.c_str(), err), *wi = need(v, si.c_str(), err); - if (!wn || !wd || !wu || !wi) return false; - gr_norm(m.R, (const float*) wn->data, EPS, m.xn, m.xn16, T, m.cs); - if (!bf16_proj(m.gemm, wd, m.xn16, m.lo, T, sd, err)) return false; - gr_silu(m.lo, m.lo16, T, m.cs); - if (!bf16_proj(m.gemm, wu, m.lo16, m.gated, T, su, err)) return false; - if (!bf16_proj(m.gemm, wi, m.xn16, m.inj, T, si, err)) return false; - gr_mix(m.xn, m.gated, m.mixed, m.mixed_bf, T, m.cs, m.mixed_h); + const core::WeightRef *wr = nullptr, *wgi = nullptr, *wsg = nullptr, *wsu = nullptr, *wsd = nullptr; + if (!(wr = need(v, "ffn_gate_inp.weight", err)) || !(wgi = need(v, "ffn_gate_inp_shexp.weight", err)) || + !(wsg = need(v, "ffn_gate_shexp.weight", err)) || !(wsu = need(v, "ffn_up_shexp.weight", err)) || + !(wsd = need(v, "ffn_down_shexp.weight", err))) + return false; + if (wgi->kind != core::WeightKind::Bf16InF32) { err = "prefill: shared gate is not BF16"; return false; } + // ================ per sub-chunk: the mixer half, then the FFN half up to the routing ================ + for (int64_t t0 = 0; t0 < T; t0 += SUB) { + const int64_t P = std::min(SUB, T - t0), pp = p0 + t0; + uint16_t* mixed_h = b.mixed_h + t0 * N; + for (int half = 0; half < 2; ++half) { + mark(half == 0 ? kPsHcRead : kPsHcFfn); + // ---- the hyper-connection read of this half + const char* pre = half == 0 ? "hc_attn_" : "hc_ffn_"; + const std::string sn = std::string(pre) + "norm.weight", sd = std::string(pre) + "down.weight", + su = std::string(pre) + "up.weight", si = std::string(pre) + "inject.weight"; + const core::WeightRef *wn = need(v, sn.c_str(), err), *wd = need(v, sd.c_str(), err), + *wu = need(v, su.c_str(), err), *wi = need(v, si.c_str(), err); + if (!wn || !wd || !wu || !wi) return false; + float* inj = half == 0 ? b.inj_a : b.inj_f + t0 * HC; + gr_norm(b.R + t0 * D, (const float*) wn->data, EPS, b.xn, b.xn16, P, m.cs); + if (!bf16_proj(m.gemm, wd, b.xn16, b.lo, P, sd, err)) return false; + gr_silu(b.lo, b.lo16, P, m.cs); + if (!bf16_proj(m.gemm, wu, b.lo16, b.gated, P, su, err)) return false; + if (!bf16_proj(m.gemm, wi, b.xn16, inj, P, si, err)) return false; + gr_mix(b.xn, b.gated, nullptr, b.mixed_bf, P, m.cs, mixed_h); + if (half == 1) break; - if (half == 0 && !core::is_qsa_layer(g, l)) { - // ======================= GDN ======================= - const core::WeightRef *wqkv = need(v, "attn_qkv.weight", err), *wg = need(v, "attn_gate.weight", err), - *wo = need(v, "ssm_out.weight", err), *wa = need(v, "ssm_alpha.weight", err), - *wb = need(v, "ssm_beta.weight", err), *wc = need(v, "ssm_conv1d.weight", err), - *wnm = need(v, "ssm_norm.weight", err), *wdt = need(v, "ssm_dt.bias", err), - *wsa = need(v, "ssm_a", err); - if (!wqkv || !wg || !wo || !wa || !wb || !wc || !wnm || !wdt || !wsa) return false; - mark(kPsGdn); - float* state = ss.gdn_state + (size_t) gdn_index * gdn_floats; - float* conv = state + (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size; - if (!native_proj(m.gemm, wqkv, m.mixed_h, m.qkv, T, v.name("attn_qkv.weight"), err)) return false; - if (!native_proj(m.gemm, wg, m.mixed_h, m.z, T, v.name("attn_gate.weight"), err)) return false; - if (!bf16_proj(m.gemm, wa, m.mixed_bf, m.ab, T, v.name("ssm_alpha.weight"), err, 2 * HV)) return false; - if (!bf16_proj(m.gemm, wb, m.mixed_bf, m.ab + HV, T, v.name("ssm_beta.weight"), err, 2 * HV)) return false; - gdn_gates(m.ab, (const float*) wdt->data, (const float*) wsa->data, m.gate, m.beta, T, m.cs); - mark(kPsGdnConv); - gdn_conv(conv, m.qkv, (const float*) wc->data, m.hbuf, T, EPS, m.cs); - mark(kPsGdnScan); - gdn_scan(state, m.hbuf, m.gate, m.beta, m.y, T, m.cs); - mark(kPsGdnNorm); - gdn_out_norm(m.y, m.z, (const float*) wnm->data, EPS, m.y_h, T, m.cs); - mark(kPsGdnOut); - if (!native_proj(m.gemm, wo, m.y_h, m.bo, T, v.name("ssm_out.weight"), err)) return false; - ++gdn_index; - } else if (half == 0) { - // ======================= QSA ======================= - const core::QsaState& st = ss.qsa_states[qsa_index]; - const core::WeightRef *wq = need(v, "attn_q.weight", err), *wk = need(v, "attn_k.weight", err), - *wv = need(v, "attn_v.weight", err), *wo = need(v, "attn_output.weight", err), - *wik = need(v, "indexer.k_proj.weight", err), - *wiq = need(v, "indexer.q_proj.weight", err), - *wqn = need(v, "attn_q_norm.weight", err), *wkn = need(v, "attn_k_norm.weight", err), - *wiqn = need(v, "indexer.q_norm.weight", err), - *wikn = need(v, "indexer.k_norm.weight", err); - if (!wq || !wk || !wv || !wo || !wik || !wiq || !wqn || !wkn || !wiqn || !wikn) return false; - mark(kPsQsaProj); - if (!native_proj(m.gemm, wk, m.mixed_h, m.Kc, T, v.name("attn_k.weight"), err)) return false; - if (!native_proj(m.gemm, wv, m.mixed_h, m.Vc, T, v.name("attn_v.weight"), err)) return false; - if (!native_proj(m.gemm, wq, m.mixed_h, m.Qf, T, v.name("attn_q.weight"), err)) return false; - if (!bf16_proj(m.gemm, wik, m.mixed_bf, m.idx_raw, T, v.name("indexer.k_proj.weight"), err)) return false; - if (!bf16_proj(m.gemm, wiq, m.mixed_bf, m.q_idx, T, v.name("indexer.q_proj.weight"), err)) return false; - rms_rows(m.Kc, (const float*) wkn->data, T * 2, 256, 256, EPS, m.cs); - rope(m.Kc, T, 2, 256, 512, p0, (float) strata::kernels::qsa_freq_base(), m.cs); - kv_append(m.Kc, m.Vc, T, p0, st.page_table, s.page_size, st.kv_int8 ? nullptr : st.k_pool, - st.kv_int8 ? nullptr : st.v_pool, st.k_q, st.v_q, st.k_scale, st.v_scale, m.cs); - split_q(m.Qf, m.q, T, m.cs); - rms_rows(m.q, (const float*) wqn->data, T * 24, 256, 256, EPS, m.cs); - rope(m.q, T, 24, 256, 6144, p0, (float) strata::kernels::qsa_freq_base(), m.cs); - rms_rows(m.q_idx, (const float*) wiqn->data, T * 4, 128, 128, EPS, m.cs); - rope(m.q_idx, T, 4, 128, 512, p0, (float) strata::kernels::qsa_freq_base(), m.cs); - // the indexer appends of the chunk; then scores + selection for many queries at once: - // a query reads completed blocks (final once completed) and `dead` for its own tail block - const strata::kernels::QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; - mark(kPsQsaIndexer); - try { - strata::kernels::native_qsa_indexer_append_multi( - m.idx_raw, m.steps_dev + strata::kernels::kStepPos, (int) strata::kernels::kStepCount, (int) T, 0, - (const float*) wikn->data, EPS, ib, s, st.max_cells, (float) strata::kernels::qsa_freq_base(), m.cs); - } catch (const std::exception& e) { err = std::string("prefill indexer: ") + e.what(); return false; } - mark(kPsQsaScores); - for (int64_t t0 = 0; t0 < T; t0 += m.sel_batch) { - const int64_t nb = std::min(m.sel_batch, T - t0); - const int32_t* steps0 = m.steps_dev + t0 * strata::kernels::kStepCount; - // the batch's last query has completed the most blocks - const int64_t grid_blocks = - m.steps_host[(size_t) (t0 + nb - 1) * strata::kernels::kStepCount + strata::kernels::kStepNBid] + 1; - strata::kernels::qsa_block_scores(st.idx_pooled, st.idx_dead, m.q_idx + t0 * 512, steps0, nb, - m.max_blocks, s, m.sel_scores, m.cs, grid_blocks); - strata::kernels::qsa_block_topk(m.sel_scores, steps0, nb, m.max_blocks, m.cap, s, - m.sel_ids + t0 * m.cap, m.cs); - } - strata::kernels::QsaAttnPools pools; - pools.page_table = st.page_table; - if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } - else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } - mark(kPsQsaAttn); - strata::kernels::qsa_prefill_attn(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s, m.attn, T, m.cs); - mark(kPsQsaOut); - gate_attn(m.attn, m.Qf, m.attn_h, T, m.cs); - if (!native_proj(m.gemm, wo, m.attn_h, m.bo, T, v.name("attn_output.weight"), err)) return false; - ++qsa_index; - } else { - // ======================= MoE ======================= - const core::WeightRef *wr = need(v, "ffn_gate_inp.weight", err), - *wgi = need(v, "ffn_gate_inp_shexp.weight", err), - *wsg = need(v, "ffn_gate_shexp.weight", err), - *wsu = need(v, "ffn_up_shexp.weight", err), - *wsd = need(v, "ffn_down_shexp.weight", err); - if (!wr || !wgi || !wsg || !wsu || !wsd) return false; - if (offload) { // the input's trip to the second GPU starts while the router and shared expert run - cudaEventRecord(m.ev_mixed, m.cs); - cudaStreamWaitEvent(m.xfer, m.ev_mixed, 0); - cudaMemcpyAsync(offload->host_input(), m.mixed_h, (size_t) T * N * 2, cudaMemcpyDeviceToHost, m.xfer); - cudaEventRecord(m.ev_input, m.xfer); - } - mark(kPsRouter); - if (!bf16_proj(m.gemm, wr, m.mixed_bf, m.logits, T, v.name("ffn_gate_inp.weight"), err)) return false; - route(m.logits, m.ids, m.w, T, m.cs); - // the shared expert and its scalar gate - if (!native_proj(m.gemm, wsg, m.mixed_h, m.sgate, T, v.name("ffn_gate_shexp.weight"), err)) return false; - if (!native_proj(m.gemm, wsu, m.mixed_h, m.sup, T, v.name("ffn_up_shexp.weight"), err)) return false; - swiglu_pair(m.sgate, m.sup, m.sh_h, T, m.cs); - if (!native_proj(m.gemm, wsd, m.sh_h, m.shared, T, v.name("ffn_down_shexp.weight"), err)) return false; - if (wgi->kind != core::WeightKind::Bf16InF32) { err = "prefill: shared gate is not BF16"; return false; } - m.gemm.bf16(m.mixed_bf, (const uint16_t*) wgi->data, m.sg, T, 1, N); - // group the (token, k) pairs by expert on the host - cudaMemcpyAsync(m.ids_host.data(), m.ids, (size_t) T * K * 4, cudaMemcpyDeviceToHost, m.cs); - if (offload) cudaMemcpyAsync(m.w_host.data(), m.w, (size_t) T * K * 4, cudaMemcpyDeviceToHost, m.cs); - cudaStreamSynchronize(m.cs); - std::fill(m.cnt.begin(), m.cnt.end(), 0); - for (int64_t i = 0; i < T * K; ++i) { - const int32_t e = m.ids_host[(size_t) i]; - if (e < 0 || e >= NE) { err = "prefill: routed id out of range"; return false; } - ++m.cnt[(size_t) e]; - } - m.off[0] = 0; - for (int64_t e = 0; e < NE; ++e) m.off[(size_t) e + 1] = m.off[(size_t) e] + m.cnt[(size_t) e]; - std::vector fill(m.off.begin(), m.off.end() - 1); - for (int64_t i = 0; i < T * K; ++i) { - const int32_t e = m.ids_host[(size_t) i]; - const int32_t p = fill[(size_t) e]++; - m.slot_host[(size_t) i] = p; - m.src_host[(size_t) p] = (int32_t) (i / K); - m.entry_of[(size_t) p] = (int32_t) i; - } - auto resident = [&](int32_t e) { - return m.host_res && m.cache && m.host_res[(size_t) l * NE + e] >= 0; - }; - if (offload) { // the experts this GPU's cache does not hold go to the second GPU - m.experts2.clear(); - m.off2.assign(1, 0); - m.src2.clear(); - m.w2.clear(); - for (int32_t e = 0; e < NE; ++e) { - if (m.cnt[(size_t) e] == 0 || resident(e)) continue; - m.experts2.push_back(e); - for (int32_t p = m.off[(size_t) e]; p < m.off[(size_t) e + 1]; ++p) { - const int32_t i = m.entry_of[(size_t) p]; - m.src2.push_back(i / (int32_t) K); - m.w2.push_back(m.w_host[(size_t) i]); - m.slot_host[(size_t) i] = -1; // the combine takes its row from the returned sum - } - m.off2.push_back((int32_t) m.src2.size()); + if (!qsa) { + // ======================= GDN ======================= + const core::WeightRef *wqkv = need(v, "attn_qkv.weight", err), *wg = need(v, "attn_gate.weight", err), + *wo = need(v, "ssm_out.weight", err), *wa = need(v, "ssm_alpha.weight", err), + *wb = need(v, "ssm_beta.weight", err), *wc = need(v, "ssm_conv1d.weight", err), + *wnm = need(v, "ssm_norm.weight", err), *wdt = need(v, "ssm_dt.bias", err), + *wsa = need(v, "ssm_a", err); + if (!wqkv || !wg || !wo || !wa || !wb || !wc || !wnm || !wdt || !wsa) return false; + mark(kPsGdn); + float* state = ss.gdn_state + (size_t) gdn_index * gdn_floats; + float* conv = state + (uint64_t) g.ssm_state_size * g.ssm_v_heads * g.ssm_state_size; + if (!native_proj(wqkv, mixed_h, b.qkv, P, v.name("attn_qkv.weight"), err)) return false; + if (!native_proj(wg, mixed_h, b.z, P, v.name("attn_gate.weight"), err)) return false; + if (!bf16_proj(m.gemm, wa, b.mixed_bf, b.ab, P, v.name("ssm_alpha.weight"), err, 2 * HV)) return false; + if (!bf16_proj(m.gemm, wb, b.mixed_bf, b.ab + HV, P, v.name("ssm_beta.weight"), err, 2 * HV)) return false; + gdn_gates(b.ab, (const float*) wdt->data, (const float*) wsa->data, b.gate, b.beta, P, m.cs); + mark(kPsGdnConv); + gdn_conv(conv, b.qkv, (const float*) wc->data, b.hbuf, P, EPS, m.cs); + mark(kPsGdnScan); + gdn_scan(state, b.hbuf, b.gate, b.beta, b.y, P, m.cs); + mark(kPsGdnNorm); + gdn_out_norm(b.y, b.z, (const float*) wnm->data, EPS, b.y_h, P, m.cs); + mark(kPsGdnOut); + if (!native_proj(wo, b.y_h, b.bo, P, v.name("ssm_out.weight"), err)) return false; + } else { + // ======================= QSA ======================= + const core::QsaState& st = ss.qsa_states[qsa_index]; + const core::WeightRef *wq = need(v, "attn_q.weight", err), *wk = need(v, "attn_k.weight", err), + *wv = need(v, "attn_v.weight", err), *wo = need(v, "attn_output.weight", err), + *wik = need(v, "indexer.k_proj.weight", err), + *wiq = need(v, "indexer.q_proj.weight", err), + *wqn = need(v, "attn_q_norm.weight", err), *wkn = need(v, "attn_k_norm.weight", err), + *wiqn = need(v, "indexer.q_norm.weight", err), + *wikn = need(v, "indexer.k_norm.weight", err); + if (!wq || !wk || !wv || !wo || !wik || !wiq || !wqn || !wkn || !wiqn || !wikn) return false; + const int32_t* steps = b.steps + t0 * SC; + mark(kPsQsaProj); + if (!native_proj(wk, mixed_h, b.Kc, P, v.name("attn_k.weight"), err)) return false; + if (!native_proj(wv, mixed_h, b.Vc, P, v.name("attn_v.weight"), err)) return false; + if (!native_proj(wq, mixed_h, b.Qf, P, v.name("attn_q.weight"), err)) return false; + if (!bf16_proj(m.gemm, wik, b.mixed_bf, b.idx_raw, P, v.name("indexer.k_proj.weight"), err)) return false; + if (!bf16_proj(m.gemm, wiq, b.mixed_bf, b.q_idx, P, v.name("indexer.q_proj.weight"), err)) return false; + rms_rows(b.Kc, (const float*) wkn->data, P * 2, 256, 256, EPS, m.cs); + rope(b.Kc, P, 2, 256, 512, pp, (float) strata::kernels::qsa_freq_base(), m.cs); + kv_append(b.Kc, b.Vc, P, pp, st.page_table, s.page_size, st.kv_int8 ? nullptr : st.k_pool, + st.kv_int8 ? nullptr : st.v_pool, st.k_q, st.v_q, st.k_scale, st.v_scale, m.cs); + split_q(b.Qf, b.q, P, m.cs); + rms_rows(b.q, (const float*) wqn->data, P * 24, 256, 256, EPS, m.cs); + rope(b.q, P, 24, 256, 6144, pp, (float) strata::kernels::qsa_freq_base(), m.cs); + rms_rows(b.q_idx, (const float*) wiqn->data, P * 4, 128, 128, EPS, m.cs); + rope(b.q_idx, P, 4, 128, 512, pp, (float) strata::kernels::qsa_freq_base(), m.cs); + // the indexer appends of the sub-chunk; then scores + selection for many queries at once: + // a query reads completed blocks (final once completed) and `dead` for its own tail block + const strata::kernels::QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; + mark(kPsQsaIndexer); + try { + strata::kernels::native_qsa_indexer_append_multi( + b.idx_raw, steps + strata::kernels::kStepPos, (int) SC, (int) P, 0, + (const float*) wikn->data, EPS, ib, s, st.max_cells, (float) strata::kernels::qsa_freq_base(), m.cs); + } catch (const std::exception& e) { err = std::string("prefill indexer: ") + e.what(); return false; } + mark(kPsQsaScores); + for (int64_t q0 = 0; q0 < P; q0 += SEL_BATCH) { + const int64_t nb = std::min(SEL_BATCH, P - q0); + // the batch's last query has completed the most blocks + const int64_t grid_blocks = + m.steps_host[(size_t) (t0 + q0 + nb - 1) * SC + strata::kernels::kStepNBid] + 1; + strata::kernels::qsa_block_scores(st.idx_pooled, st.idx_dead, b.q_idx + q0 * 512, steps + q0 * SC, + nb, m.max_blocks, s, b.sel_scores, m.cs, grid_blocks); + strata::kernels::qsa_block_topk(b.sel_scores, steps + q0 * SC, nb, m.max_blocks, m.cap, s, + b.sel_ids + q0 * m.cap, m.cs); } - if (!offload->run_layer(l, T, m.experts2, m.off2, m.src2, m.w2, m.ev_input, err)) return false; + strata::kernels::QsaAttnPools pools; + pools.page_table = st.page_table; + if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } + else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } + mark(kPsQsaAttn); + strata::kernels::qsa_prefill_attn(b.q, pools, b.sel_ids, steps, m.cap, s, b.attn, P, m.cs); + mark(kPsQsaOut); + gate_attn(b.attn, b.Qf, b.attn_h, P, m.cs); + if (!native_proj(wo, b.attn_h, b.bo, P, v.name("attn_output.weight"), err)) return false; } - cudaMemcpyAsync(m.slot_dev, m.slot_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); - cudaMemcpyAsync(m.src_dev, m.src_host.data(), (size_t) T * K * 4, cudaMemcpyHostToDevice, m.cs); - gather_rows16(m.mixed_h, m.src_dev, m.Xs, T * K, N, m.cs); - mark(kPsExperts); - // the experts, in id order: resident ones from VRAM, the others through the staging ring (or on - // the second GPU) - std::vector order; - for (int32_t e = 0; e < NE; ++e) - if (m.cnt[(size_t) e] > 0 && (!offload || resident(e))) order.push_back(e); - const strata::kernels::cpu::ExpertLayout& lay = strata::kernels::cpu::expert_layout(); - // Stage ahead: the copy stream moves blobs host -> device while the compute stream works. - int stage_next = 0; - std::vector stage_of(order.size(), -1); - auto stage_one = [&](size_t j) -> bool { - const int32_t e = order[j]; - const bool resident = m.host_res && m.cache && m.host_res[(size_t) l * NE + e] >= 0; - if (resident) return true; - const int sl = stage_next; - stage_next = (stage_next + 1) % STAGE; - const auto th = Clock::now(); - const uint8_t* b = m.src->blob(l, e); - if (!b) { err = "prefill: expert source has no blob"; return false; } - if (m.src->pinned(l, e)) { - // DMA straight from the page-locked arena: the copy stream only waits for the slot - if (m.stage_live[sl]) cudaStreamWaitEvent(m.copy, m.used[sl], 0); - cudaMemcpyAsync(m.stage_dev[sl], b, (size_t) lay.blob_bytes(l), cudaMemcpyHostToDevice, m.copy); - ++stats_.experts_dma; - } else { - if (m.stage_live[sl]) cudaEventSynchronize(m.used[sl]); // its previous blob was dequantized - std::memcpy(m.stage_host[sl], b, (size_t) lay.blob_bytes(l)); - cudaMemcpyAsync(m.stage_dev[sl], m.stage_host[sl], (size_t) lay.blob_bytes(l), cudaMemcpyHostToDevice, m.copy); - } - cudaEventRecord(m.copied[sl], m.copy); - m.stage_live[sl] = true; - stage_of[j] = sl; - stats_.ms_experts_host += ms_since(th); - ++stats_.experts_streamed; - return true; - }; - size_t staged = 0; - const size_t lookahead = STAGE - 1; - for (size_t j = 0; j < order.size(); ++j) { - while (staged < order.size() && staged <= j + lookahead) { - if (!stage_one(staged)) return false; - ++staged; - } - const int32_t e = order[j]; - const uint8_t* blob_dev = nullptr; - if (stage_of[j] < 0) { - blob_dev = m.cache->device_slot(m.host_res[(size_t) l * NE + e]); - ++stats_.experts_resident; - } else { - cudaStreamWaitEvent(m.cs, m.copied[stage_of[j]], 0); - blob_dev = m.stage_dev[stage_of[j]]; - } - const int q = (int) (j % DQ); - if (lay.native) { - // plan v0.3 P6: a native pack's layer, dequantized by llama.cpp's own formulas - const auto& f = lay.fmt[(size_t) l]; - strata::kernels::iq_dequant_gu_f16(f.gu_type, blob_dev, blob_dev + f.up_off, f.n_ff, f.n_embd, - m.dq_gu[q], m.cs); - strata::kernels::iq_dequant_f16(f.d_type, blob_dev + f.down_off, f.n_embd * f.n_ff, m.dq_d[q], m.cs); - } else { - blob_dequant_f16(blob_dev, m.dq_gu[q], m.dq_d[q], m.cs); - } - if (stage_of[j] >= 0) cudaEventRecord(m.used[stage_of[j]], m.cs); - const int64_t o0 = m.off[(size_t) e], ne = m.cnt[(size_t) e]; - m.gemm.f16(m.Xs + o0 * N, m.dq_gu[q], m.GU + o0 * 1280, ne, 1280, N); - swiglu_interleaved(m.GU + o0 * 1280, m.Hh + o0 * 640, ne, m.cs); - m.gemm.f16(m.Hh + o0 * 640, m.dq_d[q], m.Dm + o0 * N, ne, N, 640); - } - float* sums = m.emb; // the chunk's embeddings are in R by now - if (offload) { - cudaStreamWaitEvent(m.xfer, offload->done(), 0); - cudaMemcpyAsync(sums, offload->host_sum(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.xfer); - cudaEventRecord(m.ev_sum, m.xfer); - cudaStreamWaitEvent(m.cs, m.ev_sum, 0); - } - mark(kPsCombine); - moe_combine(m.Dm, m.slot_dev, m.w, m.shared, m.sg, m.bo, T, m.cs, offload ? sums : nullptr); + // ---- the hyper-connection write of the mixer half + mark(kPsHcFfn); + gr_write(b.R + t0 * D, b.bo, b.inj_a, HC, P, m.cs); + } + // ---- the FFN half's input is ready: its trip to the second GPU starts here + if (m.offload) { + cudaEventRecord(m.ev_mixed, m.cs); + cudaStreamWaitEvent(m.xfer, m.ev_mixed, 0); + cudaMemcpyAsync(m.offload->host_input() + t0 * N, mixed_h, (size_t) P * N * 2, cudaMemcpyDeviceToHost, m.xfer); } - // ---- the hyper-connection write of this half - if (half == 0) mark(kPsHcFfn); - gr_write(m.R, m.bo, m.inj, HC, T, m.cs); + // ---- the router, and the shared expert (its gated output starts the MoE sum) + mark(kPsRouter); + if (!bf16_proj(m.gemm, wr, b.mixed_bf, b.logits, P, v.name("ffn_gate_inp.weight"), err)) return false; + route(b.logits, b.ids + t0 * K, b.w + t0 * K, P, m.cs); + if (!native_proj(wsg, mixed_h, b.sgate, P, v.name("ffn_gate_shexp.weight"), err)) return false; + if (!native_proj(wsu, mixed_h, b.sup, P, v.name("ffn_up_shexp.weight"), err)) return false; + swiglu_pair(b.sgate, b.sup, b.sh_h, P, m.cs); + if (!native_proj(wsd, b.sh_h, b.bo_moe + t0 * N, P, v.name("ffn_down_shexp.weight"), err)) return false; + m.gemm.bf16(b.mixed_bf, (const uint16_t*) wgi->data, b.sg, P, 1, N); + moe_shared_gate(b.bo_moe + t0 * N, b.sg, P, m.cs); + cudaMemcpyAsync(m.ids_host + t0 * K, b.ids + t0 * K, (size_t) P * K * 4, cudaMemcpyDeviceToHost, m.cs); + cudaMemcpyAsync(m.w_host + t0 * K, b.w + t0 * K, (size_t) P * K * 4, cudaMemcpyDeviceToHost, m.cs); + } + if (!qsa) ++gdn_index; + else ++qsa_index; + // ================ the routed experts of the whole chunk ================ + if (m.offload) cudaEventRecord(m.ev_input, m.xfer); + if (cudaStreamSynchronize(m.cs) != cudaSuccess) { + err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + mark(kPsExperts); + // group the (token, k) entries by expert: this GPU takes the experts its cache holds (all of them without + // a second GPU), the second GPU the others + std::fill(m.cnt.begin(), m.cnt.end(), 0); + for (int64_t i = 0; i < T * K; ++i) { + const int32_t e = m.ids_host[i]; + if (e < 0 || e >= NE) { err = "prefill: routed id out of range"; return false; } + ++m.cnt[(size_t) e]; + } + m.off[0] = 0; + for (int64_t e = 0; e < NE; ++e) m.off[(size_t) e + 1] = m.off[(size_t) e] + m.cnt[(size_t) e]; + { + std::vector fill(m.off.begin(), m.off.end() - 1); + for (int64_t i = 0; i < T * K; ++i) m.entry_of[(size_t) fill[(size_t) m.ids_host[i]]++] = (int32_t) i; + } + for (auto* v2 : {&m.ex1, &m.src1, &m.ex2, &m.src2}) v2->clear(); + m.w1.clear(); + m.w2.clear(); + m.off1.assign(1, 0); + m.off2.assign(1, 0); + for (int32_t e = 0; e < NE; ++e) { + if (m.cnt[(size_t) e] == 0) continue; + const bool here = !m.offload || (m.host_res && m.cache && m.host_res[(size_t) l * NE + e] >= 0); + std::vector& ex = here ? m.ex1 : m.ex2; + std::vector& of = here ? m.off1 : m.off2; + std::vector& sr = here ? m.src1 : m.src2; + std::vector& wv = here ? m.w1 : m.w2; + ex.push_back(e); + for (int32_t p = m.off[(size_t) e]; p < m.off[(size_t) e + 1]; ++p) { + const int32_t i = m.entry_of[(size_t) p]; + sr.push_back(i / (int32_t) K); + wv.push_back(m.w_host[i]); + } + of.push_back((int32_t) sr.size()); + } + if (m.offload && !m.offload->run_layer(l, T, m.ex2, m.off2, m.src2, m.w2, m.ev_input, nullptr, nullptr, err)) + return false; + if (!m.local.run_layer(l, T, m.ex1, m.off1, m.src1, m.w1, nullptr, b.mixed_h, b.bo_moe, err)) return false; + // ---- the MoE output (+ the second GPU's sums) into the residual stream + mark(kPsCombine); + if (m.offload) { + cudaStreamWaitEvent(m.xfer, m.offload->done(), 0); + cudaMemcpyAsync(b.sums, m.offload->host_sum(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.xfer); + cudaEventRecord(m.ev_sum, m.xfer); + cudaStreamWaitEvent(m.cs, m.ev_sum, 0); } + gr_write(b.R, b.bo_moe, b.inj_f, HC, T, m.cs, m.offload ? b.sums : nullptr); } if (profile) { mark(kPsPle); @@ -607,7 +591,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError()); return false; } - if (!on_chunk(m.R, T, p0, err)) return false; + if (!on_chunk(b.R, T, p0, err)) return false; } } ss.ple_prev[0] = prev[0]; @@ -616,6 +600,9 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError()); return false; } + stats_.experts_streamed = m.local.experts_streamed; + stats_.experts_resident = m.local.experts_resident; + stats_.ms_experts_host = m.local.ms_host; stats_.ms_total += ms_since(t_start); return true; } @@ -626,7 +613,7 @@ const char* prefill_section_name(int section) { "GDN output projection", "QSA projections, norms, KV", "QSA indexer appends", "QSA block scores + top-k", "QSA attention", "QSA gate + output", "HC write + HC read (FFN)", - "router, shared, grouping", "experts", "combine + HC write"}; + "router, shared expert", "grouping, experts", "combine + HC write"}; return section >= 0 && section < kPsCount ? names[section] : "?"; } diff --git a/src/program/generate.cpp b/src/program/generate.cpp index d37f31bd419..c5a052dff9d 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -44,7 +44,7 @@ #include "strata/core/verify.hpp" #include "strata/core/mtp.hpp" #include "strata/core/prompt_cache.hpp" -#include "strata/prefill/offload.hpp" +#include "strata/prefill/experts.hpp" #include "strata/prefill/prefill.hpp" #include "strata/core/native_dense.hpp" #include "strata/program/logits_selection.hpp" @@ -498,16 +498,11 @@ struct SlotLoan { return cache->slot_offsets() ? cache->slot_offsets()[slot] : (uint64_t) slot * (uint64_t) (cache->bytes() / cache->slots()); } - /// Plans a loan of `need` bytes; false when the cache cannot spare them. - bool plan(strata::core::ExpertCache& c, std::vector& r, int32_t* dr, uint64_t need, int dev, int main_dev) { - cache = &c; - res = &r; - d_res = dr; - device = dev; - main_device = main_dev; + /// The last slots that hold `need` bytes; false when the cache cannot spare them. + bool plan(uint64_t need) { int64_t k = 0; - while (k < c.slots() && (uint64_t) c.bytes() - offset(c.slots() - k) < need) ++k; - first = k + 128 <= c.slots() ? (int32_t) (c.slots() - k) : -1; + while (k < cache->slots() && (uint64_t) cache->bytes() - offset(cache->slots() - k) < need) ++k; + first = k + 128 <= cache->slots() ? (int32_t) (cache->slots() - k) : -1; return first >= 0; } void* base() const { return first >= 0 ? (void*) cache->device_slot(first) : nullptr; } @@ -548,6 +543,45 @@ struct SlotLoan { } }; +/// The batched prompt path's device buffers on each GPU: expert-cache slots lent for every prompt, as many as its +/// chunk needs, else allocated once for the longest chunk. +struct PromptBuffers { + strata::prefill::Prefill* prefill = nullptr; + strata::prefill::ExpertRunner* offload = nullptr; // the second GPU's share, or null + const strata::core::ModelGeometry* g = nullptr; + const strata::core::SessionState* ss = nullptr; + SlotLoan loan, loan2; // each GPU's cache (a null cache lends nothing) + bool lend = false, lend2 = false; + + /// After the prompt path and its second GPU's runner have been initialized. + bool init(int64_t max_chunk, bool borrow, std::string& err) { + lend = borrow && loan.cache != nullptr && loan.res != nullptr && !loan.res->empty() && + loan.plan(strata::prefill::Prefill::bytes_needed(*g, *ss, max_chunk, offload != nullptr)); + lend2 = offload != nullptr && borrow && loan2.cache != nullptr && loan2.res != nullptr && !loan2.res->empty() && + loan2.plan(strata::prefill::ExpertRunner::bytes_needed(max_chunk, g->n_expert, true, true)); + return (lend || prefill->bind(nullptr, 0, max_chunk, err)) && + (offload == nullptr || lend2 || offload->bind(nullptr, 0, max_chunk, err)); + } + /// Before a prompt whose chunks hold up to `chunk` tokens. + bool take(int64_t chunk, std::string& err) { + if (lend) { + loan.plan(strata::prefill::Prefill::bytes_needed(*g, *ss, chunk, offload != nullptr)); + loan.lend(); + if (!prefill->bind(loan.base(), loan.bytes(), chunk, err)) return false; + } + if (lend2) { + loan2.plan(strata::prefill::ExpertRunner::bytes_needed(chunk, g->n_expert, true, true)); + loan2.lend(); + if (!offload->bind(loan2.base(), loan2.bytes(), chunk, err)) return false; + } + return true; + } + /// After it: the lent slots refilled. + bool give_back(strata::core::ExpertSource& src, std::string& err) { + return loan.give_back(src, g->n_expert, err) && loan2.give_back(src, g->n_expert, err); + } +}; + } // namespace int main(int argc, char** argv) { @@ -1199,7 +1233,10 @@ int main(int argc, char** argv) { // under WDDM an over-subscribed allocation does not fail, it pages to system memory and crawls. // (with borrowing - the default with a profile - the prompt path lends cache slots instead) const bool borrow = !o.no_prefill_borrow && !o.expert_profile.empty(); - const int64_t prefill_mib = (o.prefill_chunk > 0 && !borrow) ? 160 + (o.prefill_chunk * 680) / 1024 : 0; + const int64_t prefill_mib = (o.prefill_chunk > 0 && !borrow) + ? (int64_t) (strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk, + o.second_gpu >= 0 && !o.no_prompt_offload) >> 20) + : 0; const int64_t reserve = ((int64_t) o.vram_reserve_mib + prefill_mib) << 20; int64_t slots = ((int64_t) free_b - reserve) / (int64_t) strata::kernels::cpu::expert_layout().max_blob; if (!profile.empty()) slots = std::min(slots, (int64_t) profile.size()); @@ -1937,7 +1974,7 @@ int main(int argc, char** argv) { // the prompt path's buffers there borrow cache slots, else they are reserved here const bool offload_own = o.prefill_chunk > 0 && !o.no_prompt_offload && o.no_prefill_borrow; const uint64_t reserve = ((uint64_t) o.second_gpu_reserve_mib << 20) + - (offload_own ? strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert) : 0); + (offload_own ? strata::prefill::ExpertRunner::bytes_needed(o.prefill_chunk, g.n_expert, true, true) : 0); const uint64_t budget = o.second_gpu_gib > 0 ? (uint64_t) (o.second_gpu_gib * 1073741824.0) : (free_b > reserve ? (uint64_t) free_b - reserve : 0); const auto& lay = strata::kernels::cpu::expert_layout(); @@ -2013,6 +2050,35 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: token graph captured (48 layers, one launch per token)\n"); } + // ---- the batched prompt path, and the second GPU's share of its experts (not --no-prompt-offload) + auto setup_prompt_path = [&](strata::prefill::Prefill& prefill, strata::prefill::ExpertRunner& offload, + PromptBuffers& pbuf, std::string& e) -> bool { + const bool off = o.second_gpu >= 0 && !o.no_prompt_offload; + if (off && !offload.init(o.second_gpu, o.main_gpu, nullptr, srcp, &gpu2.cache(), + host_res2.empty() ? nullptr : host_res2.data(), g.n_expert, o.prefill_chunk, true, e)) + return false; + if (!prefill.init(wt, g, ss, srcp, o.expert_cache > 0 ? &xcache : nullptr, + host_res.empty() ? nullptr : host_res.data(), o.prefill_chunk, main_cs, off ? &offload : nullptr, e)) + return false; + pbuf.prefill = &prefill; + pbuf.offload = off ? &offload : nullptr; + pbuf.g = &g; + pbuf.ss = &ss; + if (d_res != nullptr) { + pbuf.loan.cache = &xcache; + pbuf.loan.res = &host_res; + pbuf.loan.d_res = d_res; + pbuf.loan.device = pbuf.loan.main_device = o.main_gpu; + } + if (off) { + pbuf.loan2.cache = &gpu2.cache(); + pbuf.loan2.res = &host_res2; + pbuf.loan2.device = o.second_gpu; + pbuf.loan2.main_device = o.main_gpu; + } + return pbuf.init(o.prefill_chunk, !o.no_prefill_borrow, e); + }; + // ================================ WHERE THE HOST TERM GOES, PER TOKEN ================================ // // **`--gpu-only-full` MEASURES THE 48 LAYER GRAPHS AND THE LM HEAD AND NOTHING ELSE.** It never enters @@ -2049,30 +2115,16 @@ int main(int argc, char** argv) { return 2; } strata::prefill::Prefill prefill; - SlotLoan loan, loan2; // the prompt path's buffers on each GPU, lent for every batched prompt - if (!o.no_prefill_borrow && loan.plan(xcache, host_res, d_res, strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk), - o.main_gpu, o.main_gpu)) - std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB)\n", - (long long) loan.slots(), (double) loan.bytes() / 1073741824.0); - if (!prefill.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, loan.base(), - loan.bytes())) { + strata::prefill::ExpertRunner offload; + PromptBuffers pbuf; + if (!setup_prompt_path(prefill, offload, pbuf, err)) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; } - strata::prefill::Offload offload; - if (o.second_gpu >= 0 && !o.no_prompt_offload) { - if (!o.no_prefill_borrow && !host_res2.empty() && - loan2.plan(gpu2.cache(), host_res2, nullptr, strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert), - o.second_gpu, o.main_gpu)) - std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB) on the second GPU\n", - (long long) loan2.slots(), (double) loan2.bytes() / 1073741824.0); - if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), - g.n_expert, o.prefill_chunk, err, loan2.base(), loan2.bytes())) { - std::fprintf(stderr, "strata serve: %s\n", err.c_str()); - return 1; - } - prefill.offload = &offload; - } + if (pbuf.lend || pbuf.lend2) + std::fprintf(stderr, "strata serve: the prompt path borrows up to %lld cache slots (%.2f GiB), %lld (%.2f GiB) " + "on the second GPU\n", (long long) pbuf.loan.slots(), (double) pbuf.loan.bytes() / 1073741824.0, + (long long) pbuf.loan2.slots(), (double) pbuf.loan2.bytes() / 1073741824.0); strata::core::Verifier ver; strata::core::VerifyHits vh; vh.d_res = thits.d_res; @@ -2257,8 +2309,7 @@ int main(int argc, char** argv) { } if (batched_end > reuse) { apply_pending(true); // no copy may still land in a lent slot - loan.lend(); - loan2.lend(); + if (!pbuf.take(std::min(o.prefill_chunk, batched_end - reuse), err)) { report(err); return 1; } for (int64_t a0 = reuse; a0 < batched_end;) { const int64_t b0 = track ? std::min(batched_end, a0 + step) : batched_end; if (!prefill.run(ids.data() + a0, b0 - a0, a0, err)) { report(err); return 1; } @@ -2268,7 +2319,7 @@ int main(int argc, char** argv) { } a0 = b0; } - if (!loan.give_back(*srcp, g.n_expert, err) || !loan2.give_back(*srcp, g.n_expert, err)) { + if (!pbuf.give_back(*srcp, err)) { report("refilling a lent slot failed: " + err); return 1; } @@ -2391,40 +2442,23 @@ int main(int argc, char** argv) { int64_t pos_start = 0; int64_t spec_pos = 0; // plan v0.3 P6: where the speculative loop starts (0 = not used) strata::prefill::Prefill prefill; - strata::prefill::Offload offload; + strata::prefill::ExpertRunner offload; + PromptBuffers pbuf; double prefill_batched_ms = 0; std::FILE* final_r = o.dump_final_r.empty() ? nullptr : std::fopen(o.dump_final_r.c_str(), "wb"); std::vector final_r_host(final_r ? (size_t) (g.hc * g.n_embd) : 0); - SlotLoan loan, loan2; // cache slots for the prompt path's buffers on each GPU, refilled after the prompt if (o.prefill_chunk > 0 && n_prompt > 1) { - if (!o.no_prefill_borrow && !host_res.empty() && d_res != nullptr && - loan.plan(xcache, host_res, d_res, strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk), o.main_gpu, - o.main_gpu)) - std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB)\n", - (long long) loan.slots(), (double) loan.bytes() / 1073741824.0); - else - std::fprintf(stderr, "strata generate: prompt path allocates its own buffers (no cache slots to borrow)\n"); - loan.lend(); - if (!prefill.init(wt, g, ss, srcp, o.expert_cache > 0 ? &xcache : nullptr, - host_res.empty() ? nullptr : host_res.data(), o.prefill_chunk, main_cs, err, loan.base(), - loan.bytes())) { + const int64_t n_batched = (o.prefill_until > 0 && o.prefill_until < n_prompt - 1) ? o.prefill_until : n_prompt - 1; + if (!setup_prompt_path(prefill, offload, pbuf, err) || !pbuf.take(std::min(o.prefill_chunk, n_batched), err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } - if (o.second_gpu >= 0 && !o.no_prompt_offload) { - if (!o.no_prefill_borrow && !host_res2.empty() && - loan2.plan(gpu2.cache(), host_res2, nullptr, strata::prefill::Offload::bytes_needed(o.prefill_chunk, g.n_expert), - o.second_gpu, o.main_gpu)) - std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB) on the second GPU\n", - (long long) loan2.slots(), (double) loan2.bytes() / 1073741824.0); - loan2.lend(); - if (!offload.init(o.second_gpu, o.main_gpu, srcp, &gpu2.cache(), host_res2.empty() ? nullptr : host_res2.data(), - g.n_expert, o.prefill_chunk, err, loan2.base(), loan2.bytes())) { - std::fprintf(stderr, "strata generate: %s\n", err.c_str()); - return 1; - } - prefill.offload = &offload; - } + if (pbuf.lend || pbuf.lend2) + std::fprintf(stderr, "strata generate: prompt path borrows %lld cache slots (%.2f GiB), %lld (%.2f GiB) on the " + "second GPU\n", (long long) pbuf.loan.slots(), (double) pbuf.loan.bytes() / 1073741824.0, + (long long) pbuf.loan2.slots(), (double) pbuf.loan2.bytes() / 1073741824.0); + else + std::fprintf(stderr, "strata generate: prompt path allocates its own buffers (no cache slots to borrow)\n"); if (!o.mtp.empty()) { if (!mtp.bind(wt, &native_head, nullptr, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); @@ -2439,15 +2473,14 @@ int main(int argc, char** argv) { } prefill.profile = o.prefill_profile; const Clock::time_point tp0 = Clock::now(); - const int64_t n_batched = (o.prefill_until > 0 && o.prefill_until < n_prompt - 1) ? o.prefill_until : n_prompt - 1; if (!prefill.run(o.tokens.data(), n_batched, 0, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } // refill the lent slots from the arena and give them back to the decode tiers - const size_t n_lent[2] = {loan.lent.size(), loan2.lent.size()}; + const size_t n_lent[2] = {pbuf.loan.lent.size(), pbuf.loan2.lent.size()}; const Clock::time_point tr = Clock::now(); - if (!loan.give_back(*srcp, g.n_expert, err) || !loan2.give_back(*srcp, g.n_expert, err)) { + if (!pbuf.give_back(*srcp, err)) { std::fprintf(stderr, "strata generate: refilling a lent slot failed: %s\n", err.c_str()); return 1; } @@ -2463,11 +2496,11 @@ int main(int argc, char** argv) { ss.ple_prev[1] = pos_start >= 1 ? (int32_t) o.tokens[(size_t) (pos_start - 1)] : -1; const strata::prefill::PrefillStats& ps = prefill.stats(); std::fprintf(stderr, "strata generate: prefill %lld tokens in %lld chunks, %.1f ms (%.1f tok/s); experts " - "streamed %lld (%lld by DMA, host %.1f ms), resident %lld; PLE %.1f ms\n", + "streamed %lld (host %.1f ms), resident %lld; PLE %.1f ms\n", (long long) ps.tokens, (long long) ps.chunks, ps.ms_total, ps.ms_total > 0 ? 1000.0 * (double) ps.tokens / ps.ms_total : 0.0, (long long) ps.experts_streamed, - (long long) ps.experts_dma, ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); - if (prefill.offload != nullptr) + ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); + if (pbuf.offload != nullptr) std::fprintf(stderr, "strata generate: prefill on the second GPU: experts streamed %lld, resident %lld; " "%.1f ms queuing, %.1f ms GPU\n", (long long) offload.experts_streamed, (long long) offload.experts_resident, offload.ms_host, offload.ms_gpu); From 2acfac4ba1e95a54857c5433bad3b441c333807d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 04:58:26 +0200 Subject: [PATCH 23/92] Prompt path: expert blobs of long chunks copied ahead while the dense steps run; per-expert GEMMs over side streams; the second GPU's sums return as FP16; pinned uploads --- include/strata/prefill/experts.hpp | 31 +++-- include/strata/prefill/gemm.hpp | 14 ++- include/strata/prefill/kernels.hpp | 6 +- include/strata/prefill/prefill.hpp | 7 +- src/prefill/experts.cpp | 175 ++++++++++++++++++++++++----- src/prefill/gemm.cu | 56 ++++++++- src/prefill/kernels.cu | 15 ++- src/prefill/prefill.cpp | 44 ++++++-- src/program/generate.cpp | 18 +-- 9 files changed, 300 insertions(+), 66 deletions(-) diff --git a/include/strata/prefill/experts.hpp b/include/strata/prefill/experts.hpp index 6c286ef113d..8e32d4f7306 100644 --- a/include/strata/prefill/experts.hpp +++ b/include/strata/prefill/experts.hpp @@ -4,14 +4,16 @@ // batch one staging copy for each run of adjacent arena blobs among the experts the GPU's cache does not hold, one // dequantization launch, one gather of the batch's rows, the two GEMMs of each expert with a SwiGLU between them, and // one ordered add of every row, times its routing weight, into its token's sum. A token's experts add up in ascending -// id order, whatever the batches. +// id order, whatever the batches. A chunk of a few hundred tokens or more uses nearly every expert, so the blobs a +// layer will stream can be copied ahead (`prefetch`), while the prompt path's dense work runs and before the routing +// is known; experts that do not fit there stream batch by batch. // // Local: on the prompt path's own GPU and stream, reading the chunk's MoE input and adding into its MoE output there. // Remote: on a second GPU. The first GPU (the 3090) sits on a PCIe 4.0 x4 link, over which an expert its cache lacks // streams at 6.4 GB/s (~0.65 GB (IQ3_XXS) to ~1.4 GB (UD-Q4_K_XL) a layer); the second GPU sits on x16. The first GPU // copies the chunk's MoE input (FP16) to pinned host memory; the second copies it in, computes the experts the first -// one's cache lacks - from its own cache when it holds them - and returns their sums (FP32) the same way, while the -// first computes the experts its cache holds. Neither GPU reads the other's memory: Windows gives GeForce cards no +// one's cache lacks - from its own cache when it holds them - and returns their sums (FP16: they stay below ~20) the +// same way, while the first computes the experts its cache holds. Neither GPU reads the other's memory: Windows gives GeForce cards no // peer access. #pragma once @@ -40,16 +42,30 @@ class ExpertRunner { /// whether experts its cache does not hold come its way (staging buffers). bool init(int device, int main_device, void* stream, core::ExpertSource* src, const core::ExpertCache* cache, const int32_t* res, int64_t n_expert, int64_t max_chunk, bool streams, std::string& err); + /// Chunks of at least this many tokens prefetch: they use nearly every expert, and their dense steps last long + /// enough to hide the copies (a 521-token chunk prefetching was 2.2 s slower on IQ3_XXS: routing is skewed, so + /// many of the experts no cache holds go unused in a short chunk). + static constexpr int64_t kPrefetchMin = 4096; + /// Device bytes of the buffers for chunks of `chunk` tokens. static uint64_t bytes_needed(int64_t chunk, int64_t n_expert, bool remote, bool streams); + /// The same with its prefetch area, for the residency `skip` (see prefetch_need) and its own as they are now. + uint64_t bytes_for(int64_t chunk, const int32_t* skip) const; /// The buffers for chunks of up to `chunk` tokens, carved from `region` (`bytes` long, e.g. lent cache slots), or /// with a null region allocated for `max_chunk` tokens (once). Before the first layer of every prompt that - /// uses a region. - bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err); + /// uses a region. `area`: device memory for the prefetched blobs; null: what the region has left. + bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err, void* area = nullptr, + uint64_t area_bytes = 0); + /// Bytes of prefetch area that hold a layer's streamed experts: the largest layer's experts that neither its + /// cache nor `skip` (another GPU's residency, n_layers x n_expert, or null) holds, and a margin. + uint64_t prefetch_need(const int32_t* skip) const; + /// Before a layer's routing: copies the blobs of `experts` (ascending ids) its cache does not hold into the + /// prefetch area, as many as fit, on its copy stream; run_layer of that layer reads them there. + bool prefetch(int64_t layer, const std::vector& experts, std::string& err); - /// Remote: pinned [max_chunk, n_embd]: the MoE input (FP16) goes in here, the sums (FP32) come out here. + /// Remote: pinned [max_chunk, n_embd]: the MoE input goes in here, the sums come out here (both FP16). uint16_t* host_input() const; - float* host_sum() const; + uint16_t* host_sum() const; /// Layer `layer` of a chunk of T tokens: expert experts[j] serves rows [off[j], off[j + 1]), row r being token /// src[r] with routing weight w[r]; each row times its weight adds to its token's sum. Local: reads `input` @@ -63,6 +79,7 @@ class ExpertRunner { int64_t experts_resident = 0; ///< expert-layer groups served from its cache int64_t experts_streamed = 0; ///< expert blobs copied from the arena + int64_t experts_prefetched = 0; ///< ...of which ahead of the routing double ms_host = 0; ///< host time queuing its work double ms_gpu = 0; ///< remote: its GPU time from the input's arrival (all layers but the last) diff --git a/include/strata/prefill/gemm.hpp b/include/strata/prefill/gemm.hpp index 976fed5904e..70054e8af0f 100644 --- a/include/strata/prefill/gemm.hpp +++ b/include/strata/prefill/gemm.hpp @@ -18,8 +18,9 @@ class Gemm { Gemm(const Gemm&) = delete; Gemm& operator=(const Gemm&) = delete; - /// The cuBLAS handle on `stream`; set_buffers() before the first product. - bool init(void* stream, std::string& err); + /// The cuBLAS handle on `stream`, and with `side_streams` the side streams f16_grouped() spreads over; + /// set_buffers() before the first product. + bool init(void* stream, std::string& err, bool side_streams = false); /// Caller-owned device buffers (e.g. lent expert-cache slots): the dequantization scratch (`scratch_elems` FP16 /// elements: the largest weight dequantized at once; may be null for no native() calls) and the cuBLAS workspace. void set_buffers(uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes); @@ -34,7 +35,9 @@ class Gemm { /// Y_i[rows[i], N] = X_i[rows[i], K] . W_i[N, K]^T (FP16 in, FP32 out) for n problems of one shape but their /// own row counts, in one cuBLAS grouped call; the pointer arrays are DEVICE memory, `rows` host memory. Where - /// cuBLAS does not group these types it runs f16() per problem, from the host copies of the pointers. + /// cuBLAS does not group these types it runs f16() per problem, from the host copies of the pointers, spread + /// over the side streams when the problems average 128 rows or more (a GEMM of a few hundred rows leaves most + /// of the GPU idle; below that the launches, not the GPU, set the pace). void f16_grouped(const uint16_t* const* X, const uint16_t* const* W, float* const* Y, const uint16_t* const* X_host, const uint16_t* const* W_host, float* const* Y_host, const int* rows, int n, int64_t N, int64_t K); @@ -47,8 +50,13 @@ class Gemm { void* stream() const { return stream_; } private: + static constexpr int AUX = 3; // side streams for independent products void* handle_ = nullptr; void* stream_ = nullptr; + void* aux_handle_[AUX] = {}; + void* aux_stream_[AUX] = {}; + void* fork_ = nullptr; + void* join_[AUX] = {}; uint16_t* scratch_ = nullptr; int64_t scratch_elems_ = 0; bool grouped_ = true; // cuBLAS groups FP16 -> FP32 GEMMs (cleared at the first refusal) diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index 888531f77b1..bcf078cb97b 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -17,9 +17,9 @@ void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream); /// mixed[t, d] = mean_c xn[t, c, d] * sigmoid(gated[t, c, d]); FP32, BF16 and FP16 (any of them may be null). void gr_mix(const float* xn, const float* gated, float* mixed, uint16_t* mixed16, int64_t T, void* stream, uint16_t* mixed_h = nullptr); -/// R[t, c, d] += (bo[t, d] (+ partial[t, d])) * 2 sigmoid(inj[t, c] / hc) (inj has row stride inj_ld) +/// R[t, c, d] += (bo[t, d] (+ partial[t, d], FP16)) * 2 sigmoid(inj[t, c] / hc) (inj has row stride inj_ld) void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream, - const float* partial = nullptr); + const uint16_t* partial = nullptr); /// R[t, c, :] = e[t, :] for all four streams (the embedding broadcast). void gr_broadcast(const float* e, float* R, int64_t T, void* stream); @@ -47,6 +47,8 @@ void swiglu_interleaved(const float* gu, uint16_t* h16, int64_t n, void* stream) void swiglu_pair(const float* g, const float* u, uint16_t* h16, int64_t n, void* stream); /// Gather rows: dst16[i, :] = x16[src[i], :] (n rows of `width` BF16). void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int64_t n, int64_t width, void* stream); +/// y = FP16 bits of x, saturated at +-65504 (the second GPU's expert sums on their way back: below ~20 measured) +void sums_to_f16(const float* x, uint16_t* y, int64_t n, void* stream); /// In place: x[t, :] *= sigmoid(sg[t]) (the shared expert's scalar gate) void moe_shared_gate(float* x, const float* sg, int64_t T, void* stream); /// sum[src[r], :] += w[r] * rows[r, :] for n rows (one expert's rows: no token twice). diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index 8254b7a9afc..3e3e424a383 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -38,6 +38,7 @@ struct PrefillStats { double ms_total = 0; double ms_experts_host = 0; ///< host time queuing this GPU's experts int64_t experts_streamed = 0; ///< expert blobs copied host -> this GPU + int64_t experts_prefetched = 0; ///< ...of which ahead of the routing int64_t experts_resident = 0; ///< expert-layer groups served from its VRAM tier double ms_ple = 0; double ms_section[kPsCount] = {}; ///< GPU ms per section, with `Prefill::profile` @@ -60,9 +61,11 @@ class Prefill { /// Device bytes of the buffers for chunks of `chunk` tokens. static uint64_t bytes_needed(const core::ModelGeometry& g, const core::SessionState& ss, int64_t chunk, bool offload); + /// The same with the area its experts are prefetched into (without a second GPU), for the residency as it is now. + uint64_t bytes_for(int64_t chunk) const; /// The buffers for chunks of up to `chunk` tokens, carved from `region` (`bytes` long: lent expert-cache - /// slots), or with a null region allocated for the longest chunk (once). Before the first chunk of every prompt - /// that uses a region. + /// slots; what is left over takes prefetched experts), or with a null region allocated for the longest chunk + /// (once). Before the first chunk of every prompt that uses a region. bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err); /// Positions [pos0, pos0 + n) holding `tokens`, in chunks of the bound length; `ss.ple_prev` must be the two diff --git a/src/prefill/experts.cpp b/src/prefill/experts.cpp index dcf71fef5f6..d778bfb1f13 100644 --- a/src/prefill/experts.cpp +++ b/src/prefill/experts.cpp @@ -21,6 +21,7 @@ constexpr int64_t ROWS = 4096; constexpr int SLOTS = 2; // batches in staging: the copies of one overlap the work on the other constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace constexpr int NPTR = 7; // per-expert device pointers: blob, xs, gate/up, gu, h, down, d +constexpr int PRE_GROUP = 16; // prefetched blobs per copy event struct OnDevice { // a remote runner's GPU current for one call, the main one again afterwards int main; @@ -44,6 +45,12 @@ struct Alloc { size_t blob_cap() { return ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; } +// the pinned upload block: NPTR x n_expert pointers, then the adds (at most 3 per row, 2 per expert), rows, weights +size_t adds_cap(int64_t chunk, int64_t n_expert) { return (size_t) (chunk * K * 3 + n_expert * 2); } +size_t up_bytes(int64_t chunk, int64_t n_expert) { + return (size_t) NPTR * (size_t) n_expert * sizeof(void*) + (adds_cap(chunk, n_expert) + (size_t) chunk * K * 2) * 4; +} + } // namespace struct ExpertRunner::Impl { @@ -63,12 +70,24 @@ struct ExpertRunner::Impl { float *sum = nullptr, *w = nullptr, *gu = nullptr, *d = nullptr; int32_t* rows_src = nullptr; int32_t* adds = nullptr; // per batch: its tokens, their row lists' starts, the row lists (moe_gather_add) - std::vector adds_host, mark, first, count; + std::vector mark, first, count; uint8_t* stage[SLOTS] = {}; void** ptrs = nullptr; // NPTR arrays of n_expert device pointers - std::vector ptrs_host; - uint16_t* h_input = nullptr; - float* h_sum = nullptr; + // a layer's uploads, staged in pinned memory (a pageable copy blocks the host behind queued expert copies): + // [pointers | adds | rows | weights], for chunks of up to max_chunk tokens + uint8_t* h_up = nullptr; + cudaEvent_t up_done = nullptr; // the previous layer's uploads have been read + bool up_live = false; + uint16_t *h_input = nullptr, *h_sum = nullptr; + // a layer's blobs copied ahead: at pre_off[e] in `pre` (-1: not there), covered by event pre_ev[pre_grp[e]] + uint8_t* pre = nullptr; + uint64_t pre_cap = 0; + int64_t pre_layer = -1; + std::vector pre_off; + std::vector pre_grp; + std::vector pre_ev; + cudaEvent_t pre_used = nullptr; // the last read of the area + bool pre_live = false; // the same sequence counted (bytes_needed) or carved static void carve(Alloc& a, int64_t T, int64_t n_expert, bool remote, bool streams, Impl* m, void** ws, bool& ok) { @@ -109,9 +128,11 @@ ExpertRunner::~ExpertRunner() { if (m.copied[i]) cudaEventDestroy(m.copied[i]); if (m.used[i]) cudaEventDestroy(m.used[i]); } - for (cudaEvent_t e : {m.done, m.t0, m.t1}) if (e) cudaEventDestroy(e); + for (cudaEvent_t e : {m.done, m.t0, m.t1, m.pre_used, m.up_done}) if (e) cudaEventDestroy(e); + for (cudaEvent_t e : m.pre_ev) if (e) cudaEventDestroy(e); if (m.h_input) cudaFreeHost(m.h_input); if (m.h_sum) cudaFreeHost(m.h_sum); + if (m.h_up) cudaFreeHost(m.h_up); if (m.owned) cudaFree(m.owned); if (m.copy) cudaStreamDestroy(m.copy); if (m.remote && m.s) cudaStreamDestroy(m.s); @@ -142,14 +163,14 @@ bool ExpertRunner::init(int device, int main_device, void* stream, core::ExpertS m.res = res; m.n_expert = n_expert; m.max_chunk = max_chunk; - m.ptrs_host.resize((size_t) NPTR * (size_t) n_expert); - m.adds_host.reserve((size_t) max_chunk * 30); m.mark.assign((size_t) max_chunk, -1); m.first.resize((size_t) max_chunk); m.count.resize((size_t) max_chunk); OnDevice on(device, main_device, m.remote); bool ok = cudaEventCreateWithFlags(&m.done, cudaEventDisableTiming) == cudaSuccess && - cudaEventCreate(&m.t0) == cudaSuccess && cudaEventCreate(&m.t1) == cudaSuccess; + cudaEventCreateWithFlags(&m.up_done, cudaEventDisableTiming) == cudaSuccess && + cudaEventCreate(&m.t0) == cudaSuccess && cudaEventCreate(&m.t1) == cudaSuccess && + cudaHostAlloc((void**) &m.h_up, up_bytes(max_chunk, n_expert), cudaHostAllocDefault) == cudaSuccess; if (m.remote) ok = ok && cudaStreamCreateWithFlags(&m.s, cudaStreamNonBlocking) == cudaSuccess; else m.s = (cudaStream_t) stream; if (streams) { @@ -157,23 +178,29 @@ bool ExpertRunner::init(int device, int main_device, void* stream, core::ExpertS for (int i = 0; ok && i < SLOTS; ++i) ok = cudaEventCreateWithFlags(&m.copied[i], cudaEventDisableTiming) == cudaSuccess && cudaEventCreateWithFlags(&m.used[i], cudaEventDisableTiming) == cudaSuccess; + m.pre_ev.assign((size_t) ((n_expert + PRE_GROUP - 1) / PRE_GROUP), nullptr); + for (auto& e : m.pre_ev) ok = ok && cudaEventCreateWithFlags(&e, cudaEventDisableTiming) == cudaSuccess; + ok = ok && cudaEventCreateWithFlags(&m.pre_used, cudaEventDisableTiming) == cudaSuccess; + m.pre_off.assign((size_t) n_expert, -1); + m.pre_grp.assign((size_t) n_expert, 0); } - if (!ok || !m.gemm.init(m.s, err)) { + if (!ok || !m.gemm.init(m.s, err, true)) { if (err.empty()) err = "prompt experts: streams or events"; return false; } // the input and the sums: pinned and portable, both GPUs copy them if (m.remote && (cudaHostAlloc((void**) &m.h_input, (size_t) max_chunk * N * 2, cudaHostAllocPortable) != cudaSuccess || - cudaHostAlloc((void**) &m.h_sum, (size_t) max_chunk * N * 4, cudaHostAllocPortable) != cudaSuccess)) { + cudaHostAlloc((void**) &m.h_sum, (size_t) max_chunk * N * 2, cudaHostAllocPortable) != cudaSuccess)) { err = "prompt experts: pinned input and sum buffers"; return false; } return true; } -bool ExpertRunner::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err) { +bool ExpertRunner::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err, void* area, uint64_t area_bytes) { Impl& m = *impl_; OnDevice on(m.dev, m.main, m.remote); + m.pre_layer = -1; if (region == nullptr) { if (m.owned != nullptr) return true; chunk = m.max_chunk; @@ -201,11 +228,87 @@ bool ExpertRunner::bind(void* region, uint64_t bytes, int64_t chunk, std::string } m.gemm.set_buffers(nullptr, 0, ws, GEMM_WS); m.T = chunk; + if (area == nullptr && region != m.owned && bytes > a.used) { + area = (uint8_t*) region + a.used; + area_bytes = bytes - a.used; + } + m.pre = m.streams ? (uint8_t*) area : nullptr; + m.pre_cap = m.pre != nullptr ? area_bytes : 0; + return true; +} + +uint64_t ExpertRunner::bytes_for(int64_t chunk, const int32_t* skip) const { + const Impl& m = *impl_; + return bytes_needed(chunk, m.n_expert, m.remote, m.streams) + (chunk >= kPrefetchMin ? prefetch_need(skip) : 0); +} + +uint64_t ExpertRunner::prefetch_need(const int32_t* skip) const { + const Impl& m = *impl_; + if (!m.streams) return 0; + const auto& lay = strata::kernels::cpu::expert_layout(); + uint64_t most = 0; + for (int64_t l = 0; l < lay.n_layers; ++l) { + const size_t stride = ((size_t) lay.blob_bytes(l) + 255) & ~(size_t) 255; + uint64_t sum = 0; + for (int64_t e = 0; e < m.n_expert; ++e) { + const size_t i = (size_t) (l * m.n_expert + e); + if ((m.res == nullptr || m.res[i] < 0) && (skip == nullptr || skip[i] < 0)) sum += stride; + } + most = std::max(most, sum); + } + return most + most / 10; // the slots lent for the prompt make a few more experts stream +} + +bool ExpertRunner::prefetch(int64_t layer, const std::vector& experts, std::string& err) { + Impl& m = *impl_; + if (m.pre_cap == 0) return true; + OnDevice on(m.dev, m.main, m.remote); + const auto& lay = strata::kernels::cpu::expert_layout(); + const size_t bb = (size_t) lay.blob_bytes(layer); + const size_t stride = (bb + 255) & ~(size_t) 255; + std::fill(m.pre_off.begin(), m.pre_off.end(), -1); + m.pre_layer = layer; + if (m.pre_live) cudaStreamWaitEvent(m.copy, m.pre_used, 0); // the previous layer has read the area + uint64_t at = 0; + int32_t grp = 0, in_grp = 0; + const size_t n = experts.size(); + for (size_t j = 0; j < n;) { + const int32_t e = experts[j]; + const bool held = m.res && m.cache && m.res[(size_t) layer * (size_t) m.n_expert + (size_t) e] >= 0; + if (held) { ++j; continue; } + if (at + stride > m.pre_cap) break; + // a run: the next experts' blobs follow this one in the arena, and fit + const uint8_t* b0 = m.src->blob(layer, e); + if (b0 == nullptr) { err = "prompt experts: the expert source has no blob"; return false; } + size_t k = j + 1; + if (m.src->pinned(layer, e) && stride == bb) + while (k < n && experts[(size_t) k] == e + (int32_t) (k - j) && at + (k - j + 1) * stride <= m.pre_cap && + !(m.res && m.cache && m.res[(size_t) layer * (size_t) m.n_expert + (size_t) experts[k]] >= 0) && + m.src->pinned(layer, experts[k]) && m.src->blob(layer, experts[k]) == b0 + (k - j) * bb && + in_grp + (int32_t) (k - j) < PRE_GROUP) ++k; + cudaMemcpyAsync(m.pre + at, b0, (k - j) * bb, cudaMemcpyHostToDevice, m.copy); // pageable blobs: staged + for (size_t i = j; i < k; ++i) { + m.pre_off[(size_t) experts[i]] = (int64_t) (at + (i - j) * stride); + m.pre_grp[(size_t) experts[i]] = grp; + } + at += (k - j) * stride; + experts_prefetched += (int64_t) (k - j); + experts_streamed += (int64_t) (k - j); + in_grp += (int32_t) (k - j); + j = k; + if (in_grp >= PRE_GROUP) { + cudaEventRecord(m.pre_ev[(size_t) grp++], m.copy); + in_grp = 0; + } + } + if (in_grp > 0) cudaEventRecord(m.pre_ev[(size_t) grp], m.copy); + const cudaError_t e = cudaGetLastError(); + if (e != cudaSuccess) { err = std::string("prompt experts: prefetch: ") + cudaGetErrorString(e); return false; } return true; } uint16_t* ExpertRunner::host_input() const { return impl_->h_input; } -float* ExpertRunner::host_sum() const { return impl_->h_sum; } +uint16_t* ExpertRunner::host_sum() const { return impl_->h_sum; } cudaEvent_t ExpertRunner::done() const { return impl_->done; } bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, @@ -233,6 +336,7 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector int32_t { return m.res && m.cache ? m.res[(size_t) layer * (size_t) m.n_expert + (size_t) e] : -1; }; + auto pre = [&](int32_t e) -> int64_t { return m.pre_layer == layer ? m.pre_off[(size_t) e] : -1; }; // ---- the batches, and every expert's device pointers (one upload) std::vector> batches; for (size_t j0 = 0; j0 < n_exp;) { @@ -243,8 +347,12 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector rows_of(n_exp); for (size_t b = 0; b < batches.size(); ++b) { const auto [j0, j1] = batches[b]; @@ -256,7 +364,9 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0 ? (void*) m.cache->device_slot(slot) : (void*) (m.stage[b % SLOTS] + streamed++ * stride); + P[0 * E + j] = slot >= 0 ? (void*) m.cache->device_slot(slot) + : pre(experts[j]) >= 0 ? (void*) (m.pre + pre(experts[j])) + : (void*) (m.stage[b % SLOTS] + streamed++ * stride); P[1 * E + j] = m.xs + r * N; P[2 * E + j] = m.dq_gu + (j - j0) * (size_t) (FF2 * N); P[3 * E + j] = m.gu + r * FF2; @@ -270,7 +380,7 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector adds_at(batches.size()); std::vector ntok(batches.size()); - m.adds_host.clear(); + size_t n_adds = 0; for (size_t b = 0; b < batches.size(); ++b) { const int32_t r0 = off[batches[b].first], r1 = off[batches[b].second]; std::vector toks; @@ -284,11 +394,11 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector 0) cudaMemcpyAsync(m.adds, h_adds, n_adds * 4, cudaMemcpyHostToDevice, m.s); cudaMemcpyAsync(m.ptrs, P, NPTR * E * sizeof(void*), cudaMemcpyHostToDevice, m.s); if (m.remote) cudaMemcpyAsync(m.input, m.h_input, (size_t) T * N * 2, cudaMemcpyHostToDevice, m.s); if (n_rows > 0) { - cudaMemcpyAsync(m.rows_src, src.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); - cudaMemcpyAsync(m.w, w.data(), n_rows * 4, cudaMemcpyHostToDevice, m.s); + cudaMemcpyAsync(m.rows_src, h_rows, n_rows * 4, cudaMemcpyHostToDevice, m.s); + cudaMemcpyAsync(m.w, h_w, n_rows * 4, cudaMemcpyHostToDevice, m.s); } + cudaEventRecord(m.up_done, m.s); + m.up_live = true; if (m.remote) cudaMemsetAsync(m.sum, 0, (size_t) T * N * 4, m.s); // ---- staging: the streamed experts of batch b into slot b % SLOTS, a copy per run of adjacent arena blobs auto stage_batch = [&](size_t b) { @@ -323,14 +435,14 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0) { ++j; continue; } + if (held(e) >= 0 || pre(e) >= 0) { ++j; continue; } if (!any && m.live[sl]) cudaStreamWaitEvent(m.copy, m.used[sl], 0); // its previous batch is dequantized any = true; const uint8_t* b0 = m.src->blob(layer, e); uint8_t* dst = (uint8_t*) P[0 * E + j]; size_t k = j + 1; // the run: the next experts' blobs follow this one in the arena and in staging if (m.src->pinned(layer, e) && stride == bb) - while (k < j1 && held(experts[k]) < 0 && m.src->pinned(layer, experts[k]) && + while (k < j1 && held(experts[k]) < 0 && pre(experts[k]) < 0 && m.src->pinned(layer, experts[k]) && m.src->blob(layer, experts[k]) == b0 + (k - j) * bb && (uint8_t*) P[0 * E + k] == dst + (k - j) * bb) ++k; cudaMemcpyAsync(dst, b0, (k - j) * bb, cudaMemcpyHostToDevice, m.copy); // pageable blobs: staged @@ -352,6 +464,10 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0) last_grp = std::max(last_grp, m.pre_grp[(size_t) experts[j]]); + if (last_grp >= 0) cudaStreamWaitEvent(m.s, m.pre_ev[(size_t) last_grp], 0); if (lay.native) strata::kernels::iq_dequant_experts_f16(f.gu_type, f.d_type, (const uint8_t* const*) (m.ptrs + 0 * E + j0), n, f.up_off, f.down_off, f.n_ff, f.n_embd, m.dq_gu, m.dq_d, m.s); @@ -359,6 +475,10 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0) { + cudaEventRecord(m.pre_used, m.s); + m.pre_live = true; + } for (size_t j = j0; j < j1; ++j) if (held(experts[j]) >= 0) ++experts_resident; const int64_t r0 = off[j0], rows = off[j1] - r0; if (rows > ROWS) { // one expert: its rows in pieces @@ -383,8 +503,9 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector #include +#include #include #include #include @@ -23,21 +24,51 @@ void ck(cublasStatus_t s, const char* what) { Gemm::~Gemm() { if (handle_) cublasDestroy((cublasHandle_t) handle_); + for (int i = 0; i < AUX; ++i) { + if (aux_handle_[i]) cublasDestroy((cublasHandle_t) aux_handle_[i]); + if (join_[i]) cudaEventDestroy((cudaEvent_t) join_[i]); + if (aux_stream_[i]) { + cudaStreamSynchronize((cudaStream_t) aux_stream_[i]); + cudaStreamDestroy((cudaStream_t) aux_stream_[i]); + } + } + if (fork_) cudaEventDestroy((cudaEvent_t) fork_); } -bool Gemm::init(void* stream, std::string& err) { +bool Gemm::init(void* stream, std::string& err, bool side_streams) { cublasHandle_t h = nullptr; if (cublasCreate(&h) != CUBLAS_STATUS_SUCCESS) { err = "prefill gemm: cublasCreate failed"; return false; } handle_ = h; stream_ = stream; cublasSetStream(h, (cudaStream_t) stream); cublasSetMathMode(h, CUBLAS_DEFAULT_MATH); + if (!side_streams) return true; + if (cudaEventCreateWithFlags((cudaEvent_t*) &fork_, cudaEventDisableTiming) != cudaSuccess) { + err = "prefill gemm: events"; + return false; + } + for (int i = 0; i < AUX; ++i) { + cublasHandle_t a = nullptr; + if (cudaStreamCreateWithFlags((cudaStream_t*) &aux_stream_[i], cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags((cudaEvent_t*) &join_[i], cudaEventDisableTiming) != cudaSuccess || + cublasCreate(&a) != CUBLAS_STATUS_SUCCESS) { + err = "prefill gemm: side streams"; + return false; + } + aux_handle_[i] = a; + cublasSetStream(a, (cudaStream_t) aux_stream_[i]); + cublasSetMathMode(a, CUBLAS_DEFAULT_MATH); + } return true; } void Gemm::set_buffers(uint16_t* scratch, int64_t scratch_elems, void* workspace, size_t ws_bytes) { - // a fixed workspace, so the handle never allocates on the way - cublasSetWorkspace((cublasHandle_t) handle_, workspace, ws_bytes); + // a fixed workspace, so the handles never allocate on the way: a slice for each stream + const int ns = aux_handle_[0] ? AUX + 1 : 1; + const size_t slice = ws_bytes / (size_t) ns / 256 * 256; + cublasSetWorkspace((cublasHandle_t) handle_, workspace, slice); + for (int i = 0; i + 1 < ns; ++i) + cublasSetWorkspace((cublasHandle_t) aux_handle_[i], (uint8_t*) workspace + (size_t) (i + 1) * slice, slice); scratch_ = scratch; scratch_elems_ = scratch_elems; } @@ -83,7 +114,24 @@ void Gemm::f16_grouped(const uint16_t* const* X, const uint16_t* const* W, float grouped_ = false; std::fprintf(stderr, "prefill gemm: cuBLAS does not group FP16 -> FP32 GEMMs here; one call per expert\n"); } - for (int i = 0; i < n; ++i) f16(X_host[i], W_host[i], Y_host[i], rows[i], N, K); + // round robin over the stream and its side streams, which join it again + int64_t sum_rows = 0; + for (int i = 0; i < n; ++i) sum_rows += rows[i]; + const int ns = aux_handle_[0] && sum_rows >= 128 * (int64_t) n ? std::min(n, AUX + 1) : 1; + if (ns > 1) { + cudaEventRecord((cudaEvent_t) fork_, (cudaStream_t) stream_); + for (int s = 1; s < ns; ++s) cudaStreamWaitEvent((cudaStream_t) aux_stream_[s - 1], (cudaEvent_t) fork_, 0); + } + void* const main = handle_; + for (int i = 0; i < n; ++i) { + handle_ = i % ns == 0 ? main : aux_handle_[i % ns - 1]; + f16(X_host[i], W_host[i], Y_host[i], rows[i], N, K); + } + handle_ = main; + for (int s = 1; s < ns; ++s) { + cudaEventRecord((cudaEvent_t) join_[s - 1], (cudaStream_t) aux_stream_[s - 1]); + cudaStreamWaitEvent((cudaStream_t) stream_, (cudaEvent_t) join_[s - 1], 0); + } } void Gemm::native(const uint16_t* X, int ggml_type, const void* W_blocks, float* Y, int64_t T, int64_t N, int64_t K, diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index 724771bb512..c086a3a1da5 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -94,11 +94,11 @@ __global__ void gr_mix_kernel(const float* __restrict__ xn, const float* __restr if (mixed_h) mixed_h[i] = hf(s); } __global__ void gr_write_kernel(float* __restrict__ R, const float* __restrict__ bo, const float* __restrict__ inj, - int64_t inj_ld, int64_t T, const float* __restrict__ partial) { + int64_t inj_ld, int64_t T, const uint16_t* __restrict__ partial) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= T * D) return; const int64_t t = i / D, c = (i % D) / N, d = i % N; - const float b = partial ? bo[t * N + d] + partial[t * N + d] : bo[t * N + d]; + const float b = partial ? bo[t * N + d] + __half2float(__ushort_as_half(partial[t * N + d])) : bo[t * N + d]; R[i] = fmaf(b, 2.0f * sigm(inj[t * inj_ld + c] / (float) HC), R[i]); } __global__ void gr_broadcast_kernel(const float* __restrict__ e, float* __restrict__ R, int64_t T) { @@ -469,7 +469,8 @@ void gr_mix(const float* xn, const float* gated, float* mixed, uint16_t* mixed16 gr_mix_kernel<<>>(xn, gated, mixed, mixed16, T, mixed_h); check("gr_mix"); } -void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream, const float* partial) { +void gr_write(float* R, const float* bo, const float* inj, int64_t inj_ld, int64_t T, void* stream, + const uint16_t* partial) { gr_write_kernel<<>>(R, bo, inj, inj_ld, T, partial); check("gr_write"); } @@ -528,6 +529,14 @@ void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int gather_rows16_kernel<<>>(x16, src, dst16, n, width); check("gather_rows16"); } +__global__ void sums_to_f16_kernel(const float* __restrict__ x, uint16_t* __restrict__ y, int64_t n) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = hf(fminf(fmaxf(x[i], -65504.0f), 65504.0f)); +} +void sums_to_f16(const float* x, uint16_t* y, int64_t n, void* stream) { + sums_to_f16_kernel<<>>(x, y, n); + check("sums_to_f16"); +} void moe_shared_gate(float* x, const float* sg, int64_t T, void* stream) { moe_shared_gate_kernel<<>>(x, sg, T); check("moe_shared_gate"); diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index a1193591dd1..7823fc41243 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -53,8 +53,8 @@ struct Alloc { struct Bufs { uint16_t* w16 = nullptr; void* gemm_ws = nullptr; - // the chunk's: residual stream, the MoE's input (FP16) and output, the FFN half's injection, routing, steps - float *R = nullptr, *bo_moe = nullptr, *inj_f = nullptr, *w = nullptr; + // the chunk's: residual stream, the MoE's input (FP16) and output, the FFN half's injection, routing, steps, PLE rows + float *R = nullptr, *bo_moe = nullptr, *inj_f = nullptr, *w = nullptr, *ple_emb = nullptr; uint16_t* mixed_h = nullptr; int32_t *ids = nullptr, *steps = nullptr; // a sub-chunk's: the mixer's BF16 input, its output, the mixer half's injection @@ -64,7 +64,7 @@ struct Bufs { float* emb = nullptr; float *xn = nullptr, *lo = nullptr, *gated = nullptr; uint16_t *xn16 = nullptr, *lo16 = nullptr; - float *ple_emb = nullptr, *ple_kp = nullptr, *ple_key = nullptr, *ple_q = nullptr, *ple_gated = nullptr; + float *ple_kp = nullptr, *ple_key = nullptr, *ple_q = nullptr, *ple_gated = nullptr; float *qkv = nullptr, *z = nullptr, *ab = nullptr, *gate = nullptr, *beta = nullptr, *hbuf = nullptr, *y = nullptr; uint16_t* y_h = nullptr; float *Kc = nullptr, *Vc = nullptr, *Qf = nullptr, *q = nullptr, *idx_raw = nullptr, *q_idx = nullptr, @@ -73,7 +73,7 @@ struct Bufs { int32_t* sel_ids = nullptr; float *logits = nullptr, *sgate = nullptr, *sup = nullptr, *sg = nullptr; uint16_t* sh_h = nullptr; - float* sums = nullptr; // the second GPU's sums + uint16_t* sums = nullptr; // the second GPU's sums (FP16) uint8_t* local = nullptr; // this GPU's expert runner uint64_t local_bytes = 0; }; @@ -93,6 +93,7 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.ids = a.take(t * K); b.w = a.take(t * K); b.steps = a.take(t * strata::kernels::kStepCount); + b.ple_emb = a.take(t * N); b.mixed_bf = a.take(p * N); b.bo = a.take(p * N); b.inj_a = a.take(p * HC); @@ -104,7 +105,7 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.xn = a.take(p * D); b.xn16 = a.take(p * D); b.lo = a.take(p * LR); b.lo16 = a.take(p * LR); b.gated = a.take(p * D); next(); // the PLE block - b.ple_emb = a.take(p * N); b.ple_kp = a.take(p * D); b.ple_key = a.take(p * D); + b.ple_kp = a.take(p * D); b.ple_key = a.take(p * D); b.ple_q = a.take(p * D); b.ple_gated = a.take(p * D); next(); // GDN b.qkv = a.take(p * C); b.z = a.take(p * ZV); b.ab = a.take(p * 2 * HV); @@ -119,7 +120,7 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.logits = a.take(p * NE); b.sgate = a.take(p * 640); b.sup = a.take(p * 640); b.sh_h = a.take(p * 640); b.sg = a.take(p); next(); // the routed experts - b.sums = offload ? a.take(t * N) : nullptr; + b.sums = offload ? a.take(t * N) : nullptr; b.local_bytes = ExpertRunner::bytes_needed(T, NE, false, !offload); b.local = a.take(b.local_bytes); next(); @@ -220,6 +221,12 @@ uint64_t Prefill::bytes_needed(const core::ModelGeometry& g, const core::Session return a.used; } +uint64_t Prefill::bytes_for(int64_t chunk) const { + const Impl& m = *impl_; + return bytes_needed(*m.g, *m.ss, chunk, m.offload != nullptr) + + (m.offload == nullptr && chunk >= ExpertRunner::kPrefetchMin ? m.local.prefetch_need(nullptr) : 0); +} + bool Prefill::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err) { Impl& m = *impl_; if (region == nullptr) { @@ -246,7 +253,11 @@ bool Prefill::bind(void* region, uint64_t bytes, int64_t chunk, std::string& err return false; } m.gemm.set_buffers(nullptr, 0, m.b.gemm_ws, GEMM_WS); - if (!m.local.bind(m.b.local, m.b.local_bytes, chunk, err)) return false; + // the rest of a lent region takes this GPU's prefetched experts (outside the union: copies run beside the steps) + const bool area = region != m.owned && bytes > a.used; + if (!m.local.bind(m.b.local, m.b.local_bytes, chunk, err, area ? (uint8_t*) region + a.used : nullptr, + area ? bytes - a.used : 0)) + return false; m.T = chunk; return true; } @@ -345,6 +356,8 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e prev[1] = tok; } if (!ss.ple.table->gather_batch(m.ple_rows.data(), (size_t) T, m.ple_emb_host.data(), err)) return false; + // before any expert copy of the chunk takes the link + cudaMemcpyAsync(b.ple_emb, m.ple_emb_host.data(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.cs); stats_.ms_ple += ms_since(tp); } else { for (int64_t t = 0; t < T; ++t) { prev[0] = prev[1]; prev[1] = (int32_t) tokens[c0 + t]; } @@ -358,6 +371,14 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const core::LayerView v(*m.wt, l); const bool qsa = core::is_qsa_layer(g, l); new_layer(); + // a long chunk uses nearly every expert: the ones this GPU's cache lacks are copied ahead, here while + // the dense steps run, or on the second GPU + if (T >= ExpertRunner::kPrefetchMin) { + m.ex1.clear(); + for (int32_t e = 0; e < NE; ++e) + if (!(m.host_res && m.cache && m.host_res[(size_t) l * NE + e] >= 0)) m.ex1.push_back(e); + if (!(m.offload ? m.offload : &m.local)->prefetch(l, m.ex1, err)) return false; + } mark(kPsPle); // ---- the PLE block at layer 1: the key and value projections, then the decode path's per-token // arithmetic (the conv reads the previous tokens' normalized rows) @@ -365,13 +386,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const strata::kernels::PleWeights& pw = ss.ple.w; for (int64_t t0 = 0; t0 < T; t0 += SUB) { const int64_t P = std::min(SUB, T - t0); - cudaMemcpyAsync(b.ple_emb, m.ple_emb_host.data() + t0 * N, (size_t) P * N * 4, cudaMemcpyHostToDevice, m.cs); - to_bf16(b.ple_emb, b.mixed_bf, P * N, m.cs); + const float* ple_emb = b.ple_emb + t0 * N; + to_bf16(ple_emb, b.mixed_bf, P * N, m.cs); if (pw.key_bf16 != nullptr) { m.gemm.bf16(b.mixed_bf, pw.key_bf16, b.ple_kp, P, D, N); } else if (const uint16_t* W = pw.key_native_data ? w16(pw.key_native_type, pw.key_native_data, D, N) : nullptr) { - to_f16(b.ple_emb, b.mixed_h, P * N, m.cs); + to_f16(ple_emb, b.mixed_h, P * N, m.cs); m.gemm.f16(b.mixed_h, W, b.ple_kp, P, D, N); } else { err = "prefill: the PLE key has neither a BF16 nor a native GGUF form (run with --native)"; @@ -568,7 +589,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e mark(kPsCombine); if (m.offload) { cudaStreamWaitEvent(m.xfer, m.offload->done(), 0); - cudaMemcpyAsync(b.sums, m.offload->host_sum(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.xfer); + cudaMemcpyAsync(b.sums, m.offload->host_sum(), (size_t) T * N * 2, cudaMemcpyHostToDevice, m.xfer); cudaEventRecord(m.ev_sum, m.xfer); cudaStreamWaitEvent(m.cs, m.ev_sum, 0); } @@ -601,6 +622,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e return false; } stats_.experts_streamed = m.local.experts_streamed; + stats_.experts_prefetched = m.local.experts_prefetched; stats_.experts_resident = m.local.experts_resident; stats_.ms_experts_host = m.local.ms_host; stats_.ms_total += ms_since(t_start); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index c5a052dff9d..e3f7949734c 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -564,13 +564,16 @@ struct PromptBuffers { } /// Before a prompt whose chunks hold up to `chunk` tokens. bool take(int64_t chunk, std::string& err) { + // sized before either loan, from the residency both GPUs' prefetch areas are sized for + const uint64_t need = lend ? prefill->bytes_for(chunk) : 0; + const uint64_t need2 = lend2 ? offload->bytes_for(chunk, loan.res ? loan.res->data() : nullptr) : 0; if (lend) { - loan.plan(strata::prefill::Prefill::bytes_needed(*g, *ss, chunk, offload != nullptr)); + loan.plan(need); loan.lend(); if (!prefill->bind(loan.base(), loan.bytes(), chunk, err)) return false; } if (lend2) { - loan2.plan(strata::prefill::ExpertRunner::bytes_needed(chunk, g->n_expert, true, true)); + loan2.plan(need2); loan2.lend(); if (!offload->bind(loan2.base(), loan2.bytes(), chunk, err)) return false; } @@ -2496,14 +2499,15 @@ int main(int argc, char** argv) { ss.ple_prev[1] = pos_start >= 1 ? (int32_t) o.tokens[(size_t) (pos_start - 1)] : -1; const strata::prefill::PrefillStats& ps = prefill.stats(); std::fprintf(stderr, "strata generate: prefill %lld tokens in %lld chunks, %.1f ms (%.1f tok/s); experts " - "streamed %lld (host %.1f ms), resident %lld; PLE %.1f ms\n", + "streamed %lld (%lld ahead), host %.1f ms, resident %lld; PLE %.1f ms\n", (long long) ps.tokens, (long long) ps.chunks, ps.ms_total, ps.ms_total > 0 ? 1000.0 * (double) ps.tokens / ps.ms_total : 0.0, (long long) ps.experts_streamed, - ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); + (long long) ps.experts_prefetched, ps.ms_experts_host, (long long) ps.experts_resident, ps.ms_ple); if (pbuf.offload != nullptr) - std::fprintf(stderr, "strata generate: prefill on the second GPU: experts streamed %lld, resident %lld; " - "%.1f ms queuing, %.1f ms GPU\n", (long long) offload.experts_streamed, - (long long) offload.experts_resident, offload.ms_host, offload.ms_gpu); + std::fprintf(stderr, "strata generate: prefill on the second GPU: experts streamed %lld (%lld ahead), " + "resident %lld; %.1f ms queuing, %.1f ms GPU\n", (long long) offload.experts_streamed, + (long long) offload.experts_prefetched, (long long) offload.experts_resident, offload.ms_host, + offload.ms_gpu); if (o.prefill_profile) { std::fprintf(stderr, "strata generate: prefill GPU ms by section:"); for (int i = 0; i < strata::prefill::kPsCount; ++i) From becaa0b9dba8ecafdb77cfcb2cc98d9f722cc2d3 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 04:59:18 +0200 Subject: [PATCH 24/92] --feed-max default 128: the batched prompt path now breaks even with verify windows at ~185 (IQ3_XXS) and ~45 tokens (UD-Q4_K_XL) --- src/program/generate.cpp | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index e3f7949734c..50fee2f0493 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -262,10 +262,11 @@ struct Options { /// Checkpoint interval inside long prompts, in whole prompt chunks (counted from the request's first new cell). int64_t cache_every = 16384; /// Prompt parts of up to this many tokens go through verify windows instead of the batched prompt path: a new - /// part that short, and a last prompt chunk that short. A batched call costs a fixed ~5-7 s on the 3090 (it - /// lends and refills expert-cache slots and streams every expert its tokens use); verify windows cost 15-22 ms - /// per token. They break even at ~480 (IQ3_XXS) and ~530 tokens (UD-Q4_K_XL; bench/feed_test.py). - int64_t feed_max = 512; + /// part that short, and a last prompt chunk that short. A batched call has a fixed cost (it lends and refills + /// expert-cache slots and streams every expert its tokens use): ~1.5 s with IQ3_XXS on the 3090 alone, ~0.7 s + /// with UD-Q4_K_XL and the second GPU; verify windows cost 12-17 ms per token. They break even at ~185 and + /// ~45 tokens (bench/feed_test.py). + int64_t feed_max = 128; int adapt_swaps = 96; }; @@ -334,7 +335,7 @@ void usage() { " --cache-ram MIB --serve: host memory for the K/V cells of sequences a request replaced,\n" " so switching back to them continues where they were (default 16384)\n" " --feed-max N --serve: new prompt parts and last prompt chunks of up to N tokens run\n" - " through verify windows instead of the batched prompt path (default 512)\n" + " through verify windows instead of the batched prompt path (default 128)\n" " --spec T verify windows of up to T tokens: the last one and T-1 MTP drafts (2..8)\n" " --spec-min-p P a draft enters the window only while it and the drafts before it have at\n" " least probability P under the draft layer (default 0: always T-1 drafts)\n" From f4ee9b58cb7955063c9a8df4cbe5fe09f86f0224 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 06:16:00 +0200 Subject: [PATCH 25/92] Second GPU: the next layer's likeliest experts copied to it while the RAM is idle (--second-gpu-prefetch 4), its rows out in full PCIe transactions, the second tier's moves paced behind the prefetch Each layer's router also runs on the previous layer's FFN input on a side branch of the verify window's graph; once the CPU's rows of a layer are in, the host copies the next layer's likeliest experts that no GPU holds into prefetch slots on the second GPU (~0.3 ms of idle RAM per layer), and that layer's share takes them from there. The second GPU's down kernel wrote its rows one float at a time into mapped host memory: 0.25 ms a layer, and the CPU pool read RAM at 38 instead of 48 GB/s. UD-Q4_K_XL code, fixed text: 71.6 -> 77.4 t/s (rows) -> 86.1 (prefetch 4); thinking 70.2 -> 73.6. --- include/strata/core/adaptive_tier.hpp | 14 ++++++- include/strata/core/expert_source.hpp | 6 ++- include/strata/core/second_gpu.hpp | 37 +++++++++++++++-- include/strata/core/verify.hpp | 20 ++++++++- include/strata/kernels/iq_kernels.hpp | 5 +++ src/core/adaptive_tier.cpp | 53 +++++++++++++++++------- src/core/expert_source.cpp | 44 ++++++++++++++++++-- src/core/second_gpu.cpp | 59 +++++++++++++++++++++++++-- src/core/verify.cpp | 57 ++++++++++++++++++++++++-- src/kernels/cuda/iq_kernels.cu | 15 +++++++ src/program/generate.cpp | 36 ++++++++++++++-- 11 files changed, 311 insertions(+), 35 deletions(-) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp index a91c7cf46bd..aa731985035 100644 --- a/include/strata/core/adaptive_tier.hpp +++ b/include/strata/core/adaptive_tier.hpp @@ -43,7 +43,12 @@ class AdaptiveTier { /// tier ranked before another on the same counts). Nothing to do while the previous moves are in flight. /// False when a copy could not be submitted. bool adapt(std::vector& usage, std::string& err, bool decay = true); - /// Admits the landed moves into `host_res` (and `d_res`); `wait` blocks until they have landed. + /// Paced: adapt() only queues its moves, and each pump() submits up to `per_pump` of them (a copy queue that + /// never runs far ahead, so copies submitted after a pump do not wait behind a whole update). + void set_pace(int per_pump) { pace_ = per_pump; } + bool pump(std::string& err); + /// Admits the landed moves into `host_res` (and `d_res`) once all are submitted; `wait` submits the queued ones + /// and blocks until they have landed. void apply_pending(bool wait); int64_t swaps = 0, fills = 0; ///< experts moved into an occupied / an empty slot @@ -61,6 +66,13 @@ class AdaptiveTier { std::vector upper_has_; // per (layer, expert), rebuilt each call std::vector> free_; // per layer std::vector> pending_; // (residency index, slot) once the copies have landed + struct Move { int32_t layer, in, out, slot; }; // out: the evicted expert, < 0 for an empty slot + std::vector queued_; // this update's moves; [next_, end) not submitted yet + size_t next_ = 0; + int pace_ = 0; + std::string failed_; // a copy apply_pending could not submit + bool submit(const Move& m, std::string& err); + bool pump_n(size_t n, std::string& err); cudaStream_t stream_ = nullptr; cudaEvent_t ev_ = nullptr; }; diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index e2f97d7f23f..5724ed914a3 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -220,7 +220,7 @@ struct ExpertDispatch { /// the entries a cache of the N most-routed experts would serve. std::vector routed; /// A second GPU's expert tier (verify windows) and its residency table (n_layers x n_expert, slot or -1): the - /// experts it holds and the first GPU's cache does not are computed there, beside the CPU pool. + /// experts it holds or has prefetched and the first GPU's cache does not are computed there, beside the CPU pool. SecondGpu* gpu2 = nullptr; const int32_t* host_res2 = nullptr; int64_t gpu2_entries = 0; ///< routed entries the second GPU served @@ -251,6 +251,10 @@ void expert_pool_dispatch(void* user, const float* x_f, const int32_t* ids, cons /// resident experts' rows are zeroed (the GPU adds them). Requires `host_res` (the token-graph residency). void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out); +/// Copies `layer`'s likeliest experts that neither GPU holds to the second GPU's prefetch slots, where that layer's +/// dispatch finds them; `ids`/`w` (n_tok, k) are the layer's predicted routing (`Verifier::set_predict`). +void expert_prefetch_multi(ExpertDispatch& d, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, + int64_t k); /// **THE HITS, LAUNCHED AFTER THE MISSES ARE STAGED AND BEFORE `post[l]`.** Same shape as `PoolFn` and for the /// same reason: `session_loop` owns the ORDER and this owns the work, so the loop needs to know nothing about diff --git a/include/strata/core/second_gpu.hpp b/include/strata/core/second_gpu.hpp index cb500429d12..eb71662cc31 100644 --- a/include/strata/core/second_gpu.hpp +++ b/include/strata/core/second_gpu.hpp @@ -3,10 +3,15 @@ // It holds experts the first GPU's cache does not (the profile's next ranks, then the conversation's, through its // own AdaptiveTier) and computes them for each layer of a verify window while the CPU pool computes the misses. // It never spins on host memory: per layer the host launches one captured graph (copy the plan in, quantize the -// activations, the grouped experts; activations, plan and rows in mapped host memory) before the CPU pool and -// collects the rows after it, so it keeps its latency on a GPU that also drives a display (a graph spinning there -// stalls whenever the desktop draws). One graph per format pair and group-count bucket: the grouped kernels' -// grids are sized for the bucket. Native packs only: the kernels are the first GPU's hit path. +// activations, the grouped experts, their rows out; activations, plan and rows in mapped host memory) before the CPU +// pool and collects the rows after it, so it keeps its latency on a GPU that also drives a display (a graph spinning +// there stalls whenever the desktop draws). The rows cross PCIe in 16-byte stores: written one float at a time they +// took ~0.2 ms a layer and slowed the CPU pool's RAM reads. One graph per format pair and group-count bucket: the +// grouped kernels' grids are sized for the bucket. Native packs only: the kernels are the first GPU's hit path. +// +// Prefetch: between a layer's CPU pool and the next layer's ring the RAM is idle (the first GPU's serial part, ~0.3 ms +// a layer), and this GPU's x16 link copies ~4 experts in that time. The next layer's likeliest experts that neither +// GPU holds go into a few slots of their own, and that layer's share takes them from there. #pragma once #include "strata/core/expert_cache.hpp" @@ -32,6 +37,19 @@ class SecondGpu { ExpertCache& cache() { return cache_; } int device() const { return dev_; } + /// Room for `slots` prefetched experts of up to `blob_bytes`. After init(). + bool init_prefetch(int slots, uint64_t blob_bytes, std::string& err); + int prefetch_slots() const { return pre_max_; } + /// Starts copying `layer`'s experts ids[0..n) (n <= prefetch_slots(), `bytes` each from the pinned blobs `src`) into + /// the prefetch slots, once the last submitted layer is done with them. + bool prefetch(int64_t layer, const int32_t* ids, const uint8_t* const* src, int n, uint64_t bytes, std::string& err); + /// The prefetch slot holding (`layer`, `expert`), or -1; submit() takes prefetch slot p as slot -2 - p. + int prefetched(int64_t layer, int32_t expert) const { + if (layer != pre_layer_) return -1; + for (int p = 0; p < pre_n_; ++p) if (pre_ids_[p] == expert) return p; + return -1; + } + /// One layer's share: group g is the expert in slot `slots[g]`, serving the routed entries /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations. bool submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, @@ -40,6 +58,7 @@ class SecondGpu { bool finish(float* out, std::string& err); int64_t layers = 0, experts = 0, entries_done = 0; + int64_t prefetch_copied = 0, prefetch_used = 0; ///< experts copied ahead / of those, computed here double ms_wait = 0; ///< host time waiting for it after the CPU pool private: @@ -54,9 +73,19 @@ class SecondGpu { float *h_x_ = nullptr, *m_x_ = nullptr, *h_out_ = nullptr, *m_out_ = nullptr; uint8_t *h_plan_ = nullptr, *m_plan_ = nullptr; uint8_t *d_xq_ = nullptr, *d_plan_ = nullptr, *d_scratch_ = nullptr; + float* d_rows_ = nullptr; // the rows, then copied out to the mapped ones in full transactions struct Graph { int gu, d, groups; cudaGraphExec_t exec; }; std::vector graphs_; // per (gate/up, down) format pair and group-count bucket std::vector pending_; // the submitted layer's entries, in row order + // prefetch slots, their copy stream, and the event of the last copies + static constexpr int kPrefetchMax = 16; + uint8_t* d_pre_ = nullptr; + uint64_t pre_cap_ = 0; + int pre_max_ = 0, pre_n_ = 0; + int64_t pre_layer_ = -1; + int32_t pre_ids_[kPrefetchMax] = {}; + cudaStream_t pre_s_ = nullptr; + cudaEvent_t pre_ev_ = nullptr; }; } // namespace strata::core diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index a6a6c26215c..3e473493d73 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -39,6 +39,9 @@ class NativeHead; /// second GPU's rows, the main GPU's zeroed unless `GpuPlanSink::host_rows_only`. using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out, int64_t layer); +/// Layer `layer`'s likely experts, its router applied to the previous layer's FFN input: ids and weights (n_tok, k), +/// host memory. +using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); struct VerifyHits { const int32_t* d_res = nullptr; ///< device [n_layers * n_expert] slot or -1 @@ -86,8 +89,12 @@ class Verifier { /// launch, ~2 us). Set before the first `run`; `print_profile` writes the per-stage table to stdout. void set_profile(bool on) { profile_ = on; } void print_profile() const; + /// Each layer's router also runs on the previous layer's FFN input, on a side branch of the graph; once the CPU's + /// rows of a layer are in, `fn` gets the next layer's prediction (the RAM is idle until the next ring). Windows + /// of one token group. Set before the first `run`. + void set_predict(PredictFn fn, void* user) { predict_ = fn; predict_user_ = user; } - double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0; + double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0, ms_predict = 0; int64_t windows = 0; private: @@ -138,6 +145,15 @@ class Verifier { bool pcie_share_ = true; int groups_[9] = {}; float* h_ymiss_ = nullptr; float* m_ymiss_ = nullptr; // T * k * n_embd + // the next layer's predicted experts: two halves by layer parity, so a prediction stays put until the host has + // taken it; the counter holds the predictions published in this window + PredictFn predict_ = nullptr; + void* predict_user_ = nullptr; + int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k + float* h_pw_ = nullptr; float* m_pw_ = nullptr; // 2 x T * k + uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; + cudaStream_t side_ = nullptr; // the prediction's branch of the graph + cudaEvent_t fork_ = nullptr, join_ = nullptr; // device void* arena_ = nullptr; @@ -153,6 +169,8 @@ class Verifier { int32_t* sel_ = nullptr; float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *hit_out_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr; + float *plogits_ = nullptr, *pw_ = nullptr; // the next layer's router (prediction) + int32_t* pids_ = nullptr; int32_t* plan_ = nullptr; // device copy of the plan block uint8_t* staging_ = nullptr; // VRAM slots for the PCIe share of the misses static constexpr int64_t kStagingBlobs = 16; diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index e083771d2bd..19faf396807 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -61,5 +61,10 @@ size_t native_expert_scratch_bytes(int64_t cap_entries, int64_t n_ff); void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long* grp_ptr, const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_dst, const int32_t* ent_tok, int64_t cap_groups, int64_t cap_entries, const void* x_q8_1, void* scratch, float* out, void* stream); +/// Copies the first grp_start[*n_groups] rows of `rows` (n_embd floats each, n_embd % 4 == 0) to `out`, at most +/// `max_rows`, in 16-byte stores: rows bound for mapped host memory cross PCIe in full transactions (the grouped +/// kernels' one-float stores there cost ~0.2 ms a layer). Counts are read on the device. +void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, int64_t n_embd, + int64_t max_rows, float* out, void* stream); } // namespace strata::kernels diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 81a8e29ed08..3fc3876516e 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -50,16 +50,40 @@ int64_t AdaptiveTier::free_slots() const { return n; } +bool AdaptiveTier::submit(const Move& m, std::string& err) { + const uint8_t* b = src_->blob(m.layer, m.in); + if (b == nullptr) { err = "adaptive tier: a refill copy failed"; return false; } + if (m.out >= 0) (*res_)[(size_t) (m.layer * n_expert_ + m.out)] = kNotResident; // evicted now: a miss meanwhile + if (cudaMemcpyAsync(cache_->device_slot(m.slot), b, (size_t) strata::kernels::cpu::expert_layout().blob_bytes(m.layer), + cudaMemcpyHostToDevice, stream_) != cudaSuccess) { + err = "adaptive tier: a refill copy failed"; + return false; + } + return true; +} + +bool AdaptiveTier::pump_n(size_t n, std::string& err) { + if (next_ >= queued_.size()) return true; + OnDevice on(dev_, main_); + for (size_t i = 0; i < n && next_ < queued_.size(); ++i) + if (!submit(queued_[next_++], err)) return false; + if (cudaEventRecord(ev_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + return true; +} + +bool AdaptiveTier::pump(std::string& err) { return pump_n((size_t) pace_, err); } + bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay) { const auto t0 = std::chrono::steady_clock::now(); + if (!failed_.empty()) { err = failed_; return false; } if (!pending_.empty()) return true; // the previous moves are still in flight if (upper_ != nullptr) { upper_has_.assign((size_t) (n_layers_ * n_expert_), 0); for (size_t i = 0; i < upper_has_.size(); ++i) upper_has_[i] = (*upper_->res_)[i] >= 0; for (const auto& pr : upper_->pending_) upper_has_[(size_t) pr.first] = 1; } - struct Move { float gain; int32_t layer, in, out, slot; }; // out < 0: an empty slot - std::vector moves; + struct Ranked { float gain; int32_t layer, in, out, slot; }; // out < 0: an empty slot + std::vector moves; std::vector> cand, vict; std::vector& res = *res_; for (int64_t l = 0; l < n_layers_; ++l) { @@ -87,29 +111,23 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay r[vict[i].second]}); } } - std::sort(moves.begin(), moves.end(), [](const Move& a, const Move& b) { return a.gain > b.gain; }); + std::sort(moves.begin(), moves.end(), [](const Ranked& a, const Ranked& b) { return a.gain > b.gain; }); if ((int) moves.size() > max_moves_) moves.resize((size_t) max_moves_); - const auto& lay = strata::kernels::cpu::expert_layout(); - OnDevice on(dev_, main_); - for (const Move& m : moves) { - const uint8_t* b = src_->blob(m.layer, m.in); - // asynchronous: the copies run while the MTP drafts; the next window waits for them - if (b == nullptr || cudaMemcpyAsync(cache_->device_slot(m.slot), b, (size_t) lay.blob_bytes(m.layer), - cudaMemcpyHostToDevice, stream_) != cudaSuccess) { - err = "adaptive tier: a refill copy failed"; - return false; - } + // the copies run beside the next windows; a move counts once admitted (apply_pending) + queued_.clear(); + next_ = 0; + for (const Ranked& m : moves) { if (m.out >= 0) { - res[(size_t) (m.layer * n_expert_ + m.out)] = kNotResident; // evicted now: a miss meanwhile ++swaps; } else { std::vector& fr = free_[(size_t) m.layer]; fr.erase(std::find(fr.begin(), fr.end(), m.slot)); ++fills; } + queued_.push_back({m.layer, m.in, m.out, m.slot}); pending_.emplace_back((int32_t) (m.layer * n_expert_ + m.in), m.slot); // resident once the copy has landed } - if (!moves.empty()) cudaEventRecord(ev_, stream_); + if (pace_ <= 0 && !pump_n(queued_.size(), err)) return false; if (decay) for (float& v : usage) v *= 0.7f; ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); @@ -118,6 +136,11 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay void AdaptiveTier::apply_pending(bool wait) { if (pending_.empty()) return; + if (next_ < queued_.size()) { // paced moves not submitted yet + if (!wait) return; + std::string err; + if (!pump_n(queued_.size(), err)) { failed_ = err; return; } + } if (wait) cudaEventSynchronize(ev_); else if (cudaEventQuery(ev_) != cudaSuccess) return; std::vector& res = *res_; diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index b891dded379..d6644e67bd6 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -281,9 +281,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 // the second GPU's share: group g = the expert in its slot g2_slot[g], entries g2_ent[g2_start[g] ..) int n2g = 0, n2e = 0; int32_t g2_slot[128], g2_start[129], g2_ent[128]; - auto on_gpu2 = [&](int32_t e) { - return d.gpu2 != nullptr && d.host_res2[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0; + auto slot2 = [&](int32_t e) -> int32_t { // its slot there, -2 - p for prefetch slot p, or -1 + if (d.gpu2 == nullptr) return -1; + const int32_t s = d.host_res2[(size_t) d.layers * (size_t) d.n_expert + (size_t) e]; + if (s >= 0) return s; + const int p = d.gpu2->prefetched(d.layers, e); + return p >= 0 ? -2 - p : -1; }; + auto on_gpu2 = [&](int32_t e) { return slot2(e) != -1; }; // The second GPU takes this layer's share only when the layer's misses are big enough that the CPU would need // longer for all of them than its round trip (~100 us: ~4 MB of experts at the pool's ~45 GB/s); otherwise it // would only add its latency. @@ -336,7 +341,7 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 : (size_t) slot * (size_t) d.cache_blob)); } else if (on_gpu2_now(e)) { kd = 2; - g2_slot[n2g] = d.host_res2[(size_t) d.layers * (size_t) d.n_expert + (size_t) e]; + g2_slot[n2g] = slot2(e); g2_start[n2g++] = n2e; for (int64_t i = i0; i < n; ++i) if (first_of[i] == i0) g2_ent[n2e++] = (int32_t) i; @@ -479,6 +484,39 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 d.experts += n_tok * k; } +void expert_prefetch_multi(ExpertDispatch& d, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, + int64_t k) { + if (d.failed || d.gpu2 == nullptr || d.gpu2->prefetch_slots() <= 0) return; + // the predicted experts neither GPU holds, ranked by their summed routing weight over the window's tokens + int32_t cand[128]; + float score[128]; + int nc = 0; + for (int64_t i = 0; i < n_tok * k && i < 128; ++i) { + const int32_t e = ids[i]; + if (e < 0 || e >= d.n_expert) continue; + const size_t at = (size_t) layer * (size_t) d.n_expert + (size_t) e; + if (d.host_res[at] >= 0 || d.host_res2[at] >= 0 || !d.src->pinned(layer, e)) continue; + int c = 0; + while (c < nc && cand[c] != e) ++c; + if (c == nc) { cand[nc] = e; score[nc++] = 0.0f; } + score[c] += w[i]; + } + const int m = (std::min)(nc, d.gpu2->prefetch_slots()); + const uint8_t* src[128]; + for (int q = 0; q < m; ++q) { + int best = q; + for (int c = q + 1; c < nc; ++c) if (score[c] > score[best]) best = c; + std::swap(cand[q], cand[best]); + std::swap(score[q], score[best]); + src[q] = d.src->blob(layer, cand[q]); + } + if (!d.gpu2->prefetch(layer, cand, src, m, strata::kernels::cpu::expert_layout().blob_bytes(layer), d.gpu2_err)) { + d.failed = true; + d.fail = d.gpu2_err.c_str(); + d.fail_layer = layer; + } +} + void expert_hit_run(void* user, void* stream, HitPhase phase, const int32_t* ids, int64_t k) { ExpertDispatch& d = *(ExpertDispatch*) user; if (d.failed) return; diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp index f38baf2b225..924f7a82084 100644 --- a/src/core/second_gpu.cpp +++ b/src/core/second_gpu.cpp @@ -36,14 +36,17 @@ SecondGpu::~SecondGpu() { if (dev_ < 0) return; DeviceScope scope(dev_, main_); if (s_) cudaStreamSynchronize(s_); + if (pre_s_) cudaStreamSynchronize(pre_s_); for (auto& gr : graphs_) cudaGraphExecDestroy(gr.exec); cache_.close(); - void* dev[] = {d_xq_, d_plan_, d_scratch_}; + void* dev[] = {d_xq_, d_plan_, d_scratch_, d_rows_, d_pre_}; for (void* p : dev) if (p) cudaFree(p); void* host[] = {h_x_, h_out_, h_plan_}; for (void* p : host) if (p) cudaFreeHost(p); if (ev_) cudaEventDestroy(ev_); + if (pre_ev_) cudaEventDestroy(pre_ev_); if (s_) cudaStreamDestroy(s_); + if (pre_s_) cudaStreamDestroy(pre_s_); } bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, int max_t, int64_t k, std::string& err) { @@ -71,6 +74,7 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, run("device buffers", cudaMalloc((void**) &d_xq_, (size_t) max_t * (size_t) (n_embd / 32) * 36)); run("device buffers", cudaMalloc((void**) &d_plan_, plan_bytes(cap_))); run("device buffers", cudaMalloc((void**) &d_scratch_, strata::kernels::native_expert_scratch_bytes(cap_, n_ff))); + run("device buffers", cudaMalloc((void**) &d_rows_, ob)); if (e != cudaSuccess) { err = std::string("second GPU: ") + step + ": " + cudaGetErrorString(e); return false; @@ -81,6 +85,42 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, return true; } +bool SecondGpu::init_prefetch(int slots, uint64_t blob_bytes, std::string& err) { + if (slots <= 0) return true; + if (slots > kPrefetchMax) { err = "second GPU: at most " + std::to_string(kPrefetchMax) + " prefetch slots"; return false; } + DeviceScope scope(dev_, main_); + pre_cap_ = (blob_bytes + 255) / 256 * 256; + if (cudaStreamCreateWithFlags(&pre_s_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&pre_ev_, cudaEventDisableTiming) != cudaSuccess || + cudaMalloc((void**) &d_pre_, (size_t) slots * pre_cap_) != cudaSuccess) { + err = std::string("second GPU: prefetch slots: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + pre_max_ = slots; + return true; +} + +bool SecondGpu::prefetch(int64_t layer, const int32_t* ids, const uint8_t* const* src, int n, uint64_t bytes, + std::string& err) { + if (n <= 0) return true; + if (n > pre_max_ || bytes > pre_cap_) { err = "second GPU: a prefetch does not fit its slots"; return false; } + DeviceScope scope(dev_, main_); + bool ok = cudaStreamWaitEvent(pre_s_, ev_, 0) == cudaSuccess; // the last submitted layer may read the slots + for (int i = 0; ok && i < n; ++i) + ok = cudaMemcpyAsync(d_pre_ + (size_t) i * pre_cap_, src[i], (size_t) bytes, cudaMemcpyHostToDevice, pre_s_) == + cudaSuccess; + if (!ok || cudaEventRecord(pre_ev_, pre_s_) != cudaSuccess) { + err = std::string("second GPU: prefetch: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + (void) cudaStreamQuery(pre_s_); // submit now, not at the next driver call + pre_layer_ = layer; + pre_n_ = n; + for (int i = 0; i < n; ++i) pre_ids_[i] = ids[i]; + prefetch_copied += n; + return true; +} + bool SecondGpu::graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& exec, std::string& err) { for (const auto& gr : graphs_) if (gr.gu == gu_type && gr.d == d_type && gr.groups == groups) { exec = gr.exec; return true; } @@ -98,7 +138,8 @@ bool SecondGpu::graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& strata::kernels::quantize_q8_1_rows(m_x_, max_t_, n_embd_, d_xq_, s_); strata::kernels::native_expert_grouped(L, (const unsigned long long*) (pi + ptr_off(cap_)), pi + 4, pi, pi + 4 + cap_ + 1, pi + 4 + 2 * cap_ + 1, groups, cap_, d_xq_, d_scratch_, - m_out_, s_); + d_rows_, s_); + strata::kernels::native_expert_rows_out(d_rows_, pi, pi + 4, n_embd_, cap_, m_out_, s_); } catch (const std::exception& ex) { cudaStreamEndCapture(s_, &graph); if (graph) cudaGraphDestroy(graph); @@ -135,13 +176,23 @@ bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, cons dst[j] = j; tok[j] = (int32_t) (entries[j] / k); } - for (int g = 0; g < n_groups; ++g) ptr[g] = (unsigned long long) cache_.device_slot(slots[g]); + bool pre = false; + for (int g = 0; g < n_groups; ++g) { + if (slots[g] <= -2) { // a prefetch slot + ptr[g] = (unsigned long long) (d_pre_ + (size_t) (-2 - slots[g]) * pre_cap_); + pre = true; + ++prefetch_used; + } else { + ptr[g] = (unsigned long long) cache_.device_slot(slots[g]); + } + } std::memcpy(h_x_, x, (size_t) n_tok * (size_t) n_embd_ * sizeof(float)); const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; DeviceScope scope(dev_, main_); cudaGraphExec_t exec = nullptr; if (!graph_for(f.gu_type, f.d_type, bucket(n_groups), exec, err)) return false; - if (cudaGraphLaunch(exec, s_) != cudaSuccess || cudaEventRecord(ev_, s_) != cudaSuccess) { + if ((pre && cudaStreamWaitEvent(s_, pre_ev_, 0) != cudaSuccess) || cudaGraphLaunch(exec, s_) != cudaSuccess || + cudaEventRecord(ev_, s_) != cudaSuccess) { err = std::string("second GPU: ") + cudaGetErrorString(cudaGetLastError()); return false; } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 29e65aef86c..043e94ac632 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -98,10 +98,13 @@ Verifier::~Verifier() { if (commit_exec_) cudaGraphExecDestroy(commit_exec_); if (cs_) cudaStreamDestroy(cs_); if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } + if (side_) cudaStreamDestroy(side_); + if (fork_) cudaEventDestroy(fork_); + if (join_) cudaEventDestroy(join_); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, - h_flagA_, h_plan_, h_flagB_}; + h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -164,7 +167,10 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(64, (void**) &h_flag_, (void**) &m_flag_) && mapped(64, (void**) &h_flagA_, (void**) &m_flagA_) && mapped(64, (void**) &h_flagB_, (void**) &m_flagB_) && - mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_); + mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_) && + mapped(2 * T * K * 4, (void**) &h_pids_, (void**) &m_pids_) && + mapped(2 * T * K * 4, (void**) &h_pw_, (void**) &m_pw_) && + mapped(64, (void**) &h_pseq_, (void**) &m_pseq_); if (!ok) { err = "verify: mapped staging allocation failed"; return false; } // the GPU plan: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | ptr2(cap u64) | start2(cap+1) { @@ -210,6 +216,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& attn_scratch_ = b.take(T * (uint64_t) attn_scratch_floats_); tail_snap_ = b.take(nQ * TS); logits_ = b.take(T * (uint64_t) g.n_expert); w_ = b.take(T * K); ids_ = b.take(T * K); + plogits_ = b.take(T * (uint64_t) g.n_expert); pw_ = b.take(T * K); pids_ = b.take(T * K); shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); plan_ = b.take(2 * ((uint64_t) plan_i32_ + 16)); @@ -242,7 +249,10 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& err = "verify: copy stream create failed"; return false; } - if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess) { + if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&side_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess) { err = "verify: stream create failed"; return false; } @@ -315,6 +325,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { broadcast_streams(emb_, R_, N, (int) HC, T, cs); } + // set_predict: the next layer's router on this layer's FFN input, on a side branch forked after the doorbell and + // joined before the next layer rewrites that input + auto predicts = [&](int64_t l) { + return predict_ != nullptr && G == 1 && l + 1 < g.n_layers && native_router_enabled(); + }; + // per-layer state indices (GDN and QSA layers are numbered separately) std::vector gdn_idx((size_t) g.n_layers, -1), qsa_idx((size_t) g.n_layers, -1); { @@ -510,6 +526,28 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, m_ids_ + tb * K, m_w_ + tb * K, m_seq_, cs); stamp(l, 5); + if (predicts(l)) { + const WeightRef* wr1 = need(LayerView(wt, l + 1), "ffn_gate_inp.weight", err); + if (!wr1) return false; + if (cudaEventRecord(fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, fork_, 0) != cudaSuccess) { + err = "verify: the prediction branch could not fork"; + return false; + } + const size_t half = (size_t) ((l + 1) & 1) * (size_t) (MT * K); + try { + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wr1->data, plogits_, N, NE, n, side_); + native_router_top10_multi(plogits_, pids_, pw_, n, side_); + doorbell_publish(nullptr, pids_, pw_, 0, (int64_t) n * K, nullptr, m_pids_ + half, m_pw_ + half, m_pseq_, + side_); + } catch (const std::exception& e) { + err = "verify layer " + std::to_string(l) + " prediction: " + e.what(); + return false; + } + if (cudaEventRecord(join_, side_) != cudaSuccess) { + err = "verify: the prediction branch could not join"; + return false; + } + } { const WeightRef *wgi = need(v, "ffn_gate_inp_shexp.weight", err), *wsg = need(v, "ffn_gate_shexp.weight", err), *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); @@ -605,6 +643,10 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } } stamp(l, 11); + if (predicts(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { + err = "verify: the prediction branch could not join"; + return false; + } if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); stamp(l, 12); @@ -787,6 +829,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *(volatile uint32_t*) h_flag_ = 0; *(volatile uint32_t*) h_flagA_ = 0; *(volatile uint32_t*) h_flagB_ = 0; + *(volatile uint32_t*) h_pseq_ = 0; std::atomic_thread_fence(std::memory_order_seq_cst); last_t_ = T; last_pos0_ = pos0; @@ -847,6 +890,14 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *flag = want; ms_wait += std::chrono::duration(b - a).count(); ms_pool += ms_since(b); + // the next layer's prediction, published during this layer's pool + if (predict_ != nullptr && G == 1 && l + 1 < g.n_layers && *(volatile uint32_t*) h_pseq_ >= (uint32_t) (l + 1)) { + std::atomic_thread_fence(std::memory_order_acquire); + const Clock::time_point c = Clock::now(); + const size_t half = (size_t) ((l + 1) & 1) * (size_t) max_t_ * (size_t) ss.k; + predict_(predict_user_, l + 1, h_pids_ + half, h_pw_ + half, T, ss.k); + ms_predict += ms_since(c); + } } const cudaError_t se = cudaStreamSynchronize(cs_); if (se != cudaSuccess) { err = std::string("verify: ") + cudaGetErrorString(se); return false; } diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index 6bcfd6000cb..d8f793bbb40 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -626,6 +626,13 @@ __global__ void __launch_bounds__(256) native_down_kernel(const unsigned long lo } } +__global__ void rows_out_kernel(const float4* __restrict__ src, const int32_t* __restrict__ n_groups, + const int32_t* __restrict__ grp_start, long long n4, float4* __restrict__ dst) { + const long long total = (long long) grp_start[*n_groups] * n4; + for (long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; i < total; i += (long long) gridDim.x * blockDim.x) + dst[i] = src[i]; +} + // ---------------------------------------------------------------- q8_1 (quantize.cu) __global__ void quantize_q8_1_kernel(const float* __restrict__ x, block_q8_1* __restrict__ y, long long n) { const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; @@ -1117,4 +1124,12 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long check("native_expert_grouped/down"); } +void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, int64_t n_embd, + int64_t max_rows, float* out, void* stream) { + const long long n4 = n_embd / 4, most = max_rows * n4; + const unsigned blocks = (unsigned) std::min((most + 255) / 256, 256); + rows_out_kernel<<>>((const float4*) rows, n_groups, grp_start, n4, (float4*) out); + check("native_expert_rows_out"); +} + } // namespace strata::kernels diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 50fee2f0493..d1b76d2c050 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -233,6 +233,7 @@ struct Options { double second_gpu_gib = 0.0; int second_gpu_reserve_mib = 2048; double second_gpu_min_mb = 4.0; ///< a layer's misses from which it takes its share (smaller: the CPU is quicker) + int second_gpu_prefetch = 4; ///< per layer, the likeliest experts no GPU holds copied to it ahead (0 = off) /// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the /// pinned arena while the CPU computes the rest (verify windows). double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs) @@ -349,6 +350,9 @@ void usage() { " --second-gpu-gib G its VRAM for experts (default: all but --second-gpu-reserve-mib, 2048)\n" " --second-gpu-min-mb M a layer's missed experts from which it takes its share (default 4; below\n" " it the CPU is quicker than its ~100 us round trip)\n" + " --second-gpu-prefetch N per layer, copy its N likeliest experts that no GPU holds (the next\n" + " layer's router on this layer's input) to it while the RAM is idle (default 4;\n" + " 0 = off)\n" " --no-prompt-offload with --second-gpu: the prompt path streams the experts the main GPU's cache\n" " lacks to the main GPU instead of computing them on the second\n" " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" @@ -434,6 +438,9 @@ struct Drive { /// needs no locking. `d.layers` is the CURRENT layer on entry (the adapter increments it as it walks the /// blob), which is why the layer index comes from there rather than from a counter of our own. std::FILE* routing = nullptr; + /// The second GPU's adaptive tier when its moves are paced behind the prefetch copies. + strata::core::AdaptiveTier* tier2 = nullptr; + std::string tier_err; }; void drive_pool(void* user, const float* x_f, const int32_t* ids, const float* weights, int64_t n_embd, int64_t k, @@ -465,6 +472,17 @@ void drive_pool(void* user, const float* x_f, const int32_t* ids, const float* w } } +/// The second GPU's prefetch from a layer's predicted routing (Verifier::set_predict). +void drive_predict(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k) { + Drive* t = (Drive*) user; + strata::core::expert_prefetch_multi(t->d, layer, ids, w, n_tok, k); + // a few of the second tier's moves, queued behind the prefetch rather than in front of the next ones + if (t->tier2 != nullptr && !t->d.failed && !t->tier2->pump(t->tier_err)) { + t->d.failed = true; + t->d.fail = t->tier_err.c_str(); + } +} + /// Plan v0.3 P6: the pool for a verify window. void drive_pool_multi(void* user, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out, int64_t layer) { @@ -708,6 +726,7 @@ int main(int argc, char** argv) { else if (a == "--second-gpu-gib") o.second_gpu_gib = std::atof(next("--second-gpu-gib")); else if (a == "--second-gpu-reserve-mib") o.second_gpu_reserve_mib = std::atoi(next("--second-gpu-reserve-mib")); else if (a == "--second-gpu-min-mb") o.second_gpu_min_mb = std::atof(next("--second-gpu-min-mb")); + else if (a == "--second-gpu-prefetch") o.second_gpu_prefetch = std::atoi(next("--second-gpu-prefetch")); else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac")); else if (a == "--adapt-every") o.adapt_every = std::atoi(next("--adapt-every")); else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); @@ -1968,7 +1987,8 @@ int main(int argc, char** argv) { const int main_dev = o.main_gpu; cudaDeviceProp prop{}; cudaGetDeviceProperties(&prop, o.second_gpu); - if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err)) { + if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err) || + !gpu2.init_prefetch(o.second_gpu_prefetch, strata::kernels::cpu::expert_layout().max_blob, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -2023,6 +2043,12 @@ int main(int argc, char** argv) { // lowest slots first: the last ones, which the prompt path borrows, stay empty longest for (size_t i = empty2.size(); ok && i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); tier2.set_upper(&tier); + // with the prefetch, the tier's copies go four per layer behind it, an update within ~24 layers: queued in + // front of the prefetch copies, an update's ~300 MB would hold them up for ~7 ms + if (gpu2.prefetch_slots() > 0) { + tier2.set_pace(4); + drive.tier2 = &tier2; + } } if (!ok) { std::fprintf(stderr, "strata generate: second GPU: %s\n", sizes.empty() ? "no VRAM for experts" : err.c_str()); @@ -2143,6 +2169,7 @@ int main(int argc, char** argv) { ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 + if (gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); std::vector cur; prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { std::vector nxt((size_t) T); @@ -2780,6 +2807,7 @@ int main(int argc, char** argv) { ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 + if (gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); drive.d.plan = ver.plan_sink(); drive.d.pcie_num = (int) (o.pcie_frac * 256.0 + 0.5); if (drive.d.pcie_num < 0) drive.d.pcie_num = 0; @@ -2955,11 +2983,13 @@ int main(int argc, char** argv) { if (rounds > 0 && drive.d.gpu2 != nullptr) std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " "%.3f ms/round waiting for it after the CPU pool; %lld experts swapped in and %lld into empty " - "slots (%lld still empty)\n", "second GPU", + "slots (%lld still empty); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round " + "of host time)\n", "second GPU", (double) gpu2.entries_done / (double) rounds, (double) gpu2.experts / (double) rounds, (double) gpu2.layers / (double) rounds, (double) drive.d.gpu2_skipped / (double) rounds, gpu2.ms_wait / (double) rounds, (long long) tier2.swaps, (long long) tier2.fills, - (long long) tier2.free_slots()); + (long long) tier2.free_slots(), (double) gpu2.prefetch_copied / (double) rounds, + (double) gpu2.prefetch_used / (double) rounds, ver.ms_predict / (double) rounds); if (rounds > 0 && !drive.d.routed.empty()) { // the share of the routed entries the N most-routed experts of this run take std::vector c = drive.d.routed; From c2a2bb6471d6fe01839e3b71554240583dc95db4 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 06:49:47 +0200 Subject: [PATCH 26/92] CPU pool: the second GPU's share is launched once the workers have started on the CPU's (its launch took the host ~15 us before them); UD-Q4_K_XL code +0.5% --- include/strata/kernels/cpu/pool.hpp | 9 +++++--- src/core/expert_source.cpp | 35 ++++++++++++++++++++++------- src/kernels/cpu/pool.cpp | 13 +++++++---- 3 files changed, 42 insertions(+), 15 deletions(-) diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp index c138b2b231c..d10ebe486a8 100644 --- a/include/strata/kernels/cpu/pool.hpp +++ b/include/strata/kernels/cpu/pool.hpp @@ -115,8 +115,11 @@ class ExpertPool { /// Plan v0.3 P6: `run_split` for multi-token jobs (at most `kMaxSplitMulti`); the rows of each expert are /// read once for all of its tokens. void run_split_multi(ExpertJobMulti* jobs, int n); - /// Plan v0.3 P6: the same for a native pack's layer (ggml-cpu arithmetic, `nact` activations). - void run_split_multi_native(const NativeFmt& f, ExpertJobMulti* jobs, int n); + /// Plan v0.3 P6: the same for a native pack's layer (ggml-cpu arithmetic, `nact` activations). `first`, when + /// given, runs once on the host: after the workers have started on the first phase, before it joins them (or + /// alone when there are no jobs). + void run_split_multi_native(const NativeFmt& f, ExpertJobMulti* jobs, int n, void (*first)(void*) = nullptr, + void* ctx = nullptr); static constexpr int kMaxSplitMulti = 96; /// run_split_multi's phases, accumulated ms: gate/up rows, the intermediate quantization, down rows. double ms_multi_gu = 0, ms_multi_q = 0, ms_multi_down = 0; @@ -143,7 +146,7 @@ class ExpertPool { private: void worker(int id); void drain(int id, ExpertScratch& scratch); - void run_phase(int mode, int n_tasks); + void run_phase(int mode, int n_tasks, void (*first)(void*) = nullptr, void* ctx = nullptr); int n_ = 0; bool host_works_ = true; diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index d6644e67bd6..ba2b5f63e9c 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -394,12 +394,6 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (P.publish) P.publish(P.ctx); if (P.fetch) P.fetch(P.ctx, dma_src, P.pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work g2_start[n2g] = n2e; - if (n2g > 0 && !d.gpu2->submit(d.layers, x_f, (int) n_tok, k, g2_slot, g2_start, g2_ent, n2g, d.gpu2_err)) { - d.failed = true; - d.fail = d.gpu2_err.c_str(); - d.fail_layer = d.layers; - return; - } } else { for (int64_t i = 0; i < n; ++i) { const int32_t e = ids[i]; @@ -461,8 +455,33 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 ++d.multi_entries; } const auto c3 = std::chrono::steady_clock::now(); - if (native) d.pool->run_split_multi_native(lay.fmt[(size_t) d.layers], d.jobs_multi.data(), njobs); - else d.pool->run_split_multi(d.jobs_multi.data(), njobs); + // the second GPU's share goes out once the CPU's workers have started on theirs: its launch takes the host + // tens of microseconds + struct Submit2 { + ExpertDispatch* d; + const float* x; + int n_tok, n_groups; + int64_t k; + const int32_t *slots, *starts, *entries; + bool ok; + } s2{&d, x_f, (int) n_tok, n2g, k, g2_slot, g2_start, g2_ent, true}; + void (*submit2)(void*) = [](void* p) { + Submit2& s = *(Submit2*) p; + s.ok = s.d->gpu2->submit(s.d->layers, s.x, s.n_tok, s.k, s.slots, s.starts, s.entries, s.n_groups, s.d->gpu2_err); + }; + if (n2g <= 0) submit2 = nullptr; + if (native) { + d.pool->run_split_multi_native(lay.fmt[(size_t) d.layers], d.jobs_multi.data(), njobs, submit2, &s2); + } else { + if (submit2 != nullptr) submit2(&s2); + d.pool->run_split_multi(d.jobs_multi.data(), njobs); + } + if (!s2.ok) { + d.failed = true; + d.fail = d.gpu2_err.c_str(); + d.fail_layer = d.layers; + return; + } if (n2g > 0 && !d.gpu2->finish(out, d.gpu2_err)) { // the second GPU's rows, over the zeroed ones d.failed = true; d.fail = d.gpu2_err.c_str(); diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cpp index 2b963ad0974..e4537f0aa26 100644 --- a/src/kernels/cpu/pool.cpp +++ b/src/kernels/cpu/pool.cpp @@ -239,13 +239,14 @@ void ExpertPool::drain(int id, ExpertScratch& scratch) { } } -void ExpertPool::run_phase(int mode, int n_tasks) { +void ExpertPool::run_phase(int mode, int n_tasks, void (*first)(void*), void* ctx) { while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); mode_ = mode; njobs_ = n_tasks; head_.store(0, std::memory_order_relaxed); done_.store(0, std::memory_order_relaxed); epoch_.fetch_add(1, std::memory_order_release); + if (first != nullptr) first(ctx); if (host_works_) drain(-1, host_scratch_); while (done_.load(std::memory_order_acquire) != (uint32_t) n_tasks) _mm_pause(); while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); @@ -306,8 +307,12 @@ void ExpertPool::run_split_multi(ExpertJobMulti* jobs, int n) { ms_drain_ += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); } -void ExpertPool::run_split_multi_native(const NativeFmt& f, ExpertJobMulti* jobs, int n) { - if (n <= 0) return; +void ExpertPool::run_split_multi_native(const NativeFmt& f, ExpertJobMulti* jobs, int n, void (*first)(void*), + void* ctx) { + if (n <= 0) { + if (first != nullptr) first(ctx); + return; + } const auto t0 = std::chrono::steady_clock::now(); // more distinct experts than buffers: run them in batches for (int b0 = 0; b0 < n; b0 += kMaxSplitMulti) { @@ -318,7 +323,7 @@ void ExpertPool::run_split_multi_native(const NativeFmt& f, ExpertJobMulti* jobs mtasks_ = 3 * threads; mrows_ = (int64_t) nb * FF; const auto a = std::chrono::steady_clock::now(); - run_phase(5, mtasks_); + run_phase(5, mtasks_, b0 == 0 ? first : nullptr, ctx); const auto b = std::chrono::steady_clock::now(); for (int e = 0; e < nb; ++e) for (int t = 0; t < mjobs_[e].nt; ++t) From 282334701acd4953e672be0b7f8f287bb0d82426 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 06:54:27 +0200 Subject: [PATCH 27/92] Verify window: the commit graph runs beside the MTP draft (it was launched and waited for alone, 0.67 ms a round); UD-Q4_K_XL code +0.9% --- include/strata/core/verify.hpp | 8 ++++++-- src/core/verify.cpp | 16 +++++++++++++--- src/program/generate.cpp | 23 +++++++++++++++-------- 3 files changed, 34 insertions(+), 13 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 3e473493d73..d917351b408 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -64,8 +64,11 @@ class Verifier { /// The PLE rows are gathered here from `ss.ple_prev` and the tokens. Captures the T-token graph on first use. bool run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, void* user, int32_t* out, std::string& err); - /// Keep the first `n_keep` (1..T) tokens of the last window; advances `ss.ple_prev` by them. - bool commit(int n_keep, std::string& err); + /// Keep the first `n_keep` (1..T) tokens of the last window; advances `ss.ple_prev` by them. With `wait` false + /// the commit graph is only launched (the MTP draft, on its own stream, reads nothing it writes): `wait_commit` + /// before anything outside the next `run` reads the sequence state. + bool commit(int n_keep, std::string& err, bool wait = true); + bool wait_commit(std::string& err); /// Token t's residual after the last layer, (hc, n_embd) on the device, valid until the next `run`. const float* final_R(int t) const; @@ -115,6 +118,7 @@ class Verifier { cudaStream_t cs_ = nullptr; cudaGraphExec_t exec_[9] = {}; cudaGraphExec_t commit_exec_ = nullptr; + bool commit_pending_ = false; // mapped staging (host pointer, device alias) int32_t* h_tok_ = nullptr; int32_t* m_tok_ = nullptr; // T diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 043e94ac632..e93216814bb 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -1004,7 +1004,7 @@ void Verifier::publish_plan(void* ctx) { *(volatile uint32_t*) v->h_flagA_ = v->cur_layer_ + 1; } -bool Verifier::commit(int n_keep, std::string& err) { +bool Verifier::commit(int n_keep, std::string& err, bool wait) { if (n_keep < 1 || n_keep > last_t_) { err = "verify: commit count out of range"; return false; } const Clock::time_point t0 = Clock::now(); h_commit_[0] = n_keep; @@ -1013,13 +1013,23 @@ bool Verifier::commit(int n_keep, std::string& err) { std::atomic_thread_fence(std::memory_order_seq_cst); const cudaError_t le = cudaGraphLaunch(commit_exec_, cs_); if (le != cudaSuccess) { err = std::string("verify: commit launch: ") + cudaGetErrorString(le); return false; } - const cudaError_t se = cudaStreamSynchronize(cs_); - if (se != cudaSuccess) { err = std::string("verify: commit: ") + cudaGetErrorString(se); return false; } + (void) cudaStreamQuery(cs_); // submit now, not at the next driver call + commit_pending_ = true; for (int t = 0; t < n_keep; ++t) { ss_->ple_prev[0] = ss_->ple_prev[1]; ss_->ple_prev[1] = last_tokens_[t]; } ms_commit += ms_since(t0); + return !wait || wait_commit(err); +} + +bool Verifier::wait_commit(std::string& err) { + if (!commit_pending_) return true; + const Clock::time_point t0 = Clock::now(); + commit_pending_ = false; + const cudaError_t se = cudaStreamSynchronize(cs_); + ms_commit += ms_since(t0); + if (se != cudaSuccess) { err = std::string("verify: commit: ") + cudaGetErrorString(se); return false; } return true; } diff --git a/src/program/generate.cpp b/src/program/generate.cpp index d1b76d2c050..5a31672e431 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2369,9 +2369,10 @@ int main(int argc, char** argv) { if (drive.d.failed && drive.d.fail) err = drive.d.fail; return false; } - if (!ver.commit(T, err)) return false; + if (!ver.commit(T, err, false)) return false; // beside the draft for (int i = 0; i < T; ++i) outv[(size_t) i] = (int32_t) ids[(size_t) (j + i + 1)]; - if (!mtp.draft(T, outv.data(), j, T - 1, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) + if (!mtp.draft(T, outv.data(), j, T - 1, drafts.data(), err, dprob.data(), (float) o.spec_min_p) || + !ver.wait_commit(err)) return false; if (track) pcache.set(j, ids.data() + j, T); j += T; @@ -2424,7 +2425,7 @@ int main(int argc, char** argv) { bool adapt_ok = true; if (!drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0) adapt_thr = std::thread([&] { adapt_ok = adapt(); }); - if (!ver.commit(a + 1, err)) { + if (!ver.commit(a + 1, err, false)) { // beside the draft if (adapt_thr.joinable()) adapt_thr.join(); std::printf("ERR %s\n", err.c_str()); return 1; @@ -2439,8 +2440,9 @@ int main(int argc, char** argv) { } std::fflush(stdout); ++rounds; - const bool drafted = eos || produced_n >= max_new || - mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p); + const bool drafted = (eos || produced_n >= max_new || + mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && + ver.wait_commit(err); if (adapt_thr.joinable()) adapt_thr.join(); if (!adapt_ok) { std::printf("ERR an adaptive refill failed\n"); @@ -2905,7 +2907,7 @@ int main(int argc, char** argv) { bool adapt_ok = true; if (!drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0) adapt_thr = std::thread([&] { adapt_ok = adapt(); }); - if (!ver.commit(a + 1, err)) { + if (!ver.commit(a + 1, err, false)) { // beside the draft if (adapt_thr.joinable()) adapt_thr.join(); std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; @@ -2922,11 +2924,16 @@ int main(int argc, char** argv) { } if (eos) { if (adapt_thr.joinable()) adapt_thr.join(); + if (!ver.wait_commit(err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } total_ms += std::chrono::duration(Clock::now() - t0).count(); break; } - const bool drafted = !use_mtp || (int64_t) produced.size() >= max_new || - mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p); + const bool drafted = (!use_mtp || (int64_t) produced.size() >= max_new || + mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && + ver.wait_commit(err); if (adapt_thr.joinable()) adapt_thr.join(); if (!adapt_ok) { std::fprintf(stderr, "strata generate: %s\n", adapt_err.c_str()); From b01a3fbac35a6974d82cce6961f56707ffa14569 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 07:05:38 +0200 Subject: [PATCH 28/92] --ple-io ram: the PLE table read into RAM once the experts are loaded (28.8 GB in ~7 s), direct reads until then; a 16K-token prompt of new text 0.5 s faster --- include/strata/kernels/ngram.hpp | 11 ++- src/kernels/ngram.cpp | 133 +++++++++++++++++++++++++++++-- src/program/generate.cpp | 14 +++- 3 files changed, 144 insertions(+), 14 deletions(-) diff --git a/include/strata/kernels/ngram.hpp b/include/strata/kernels/ngram.hpp index b1e90d24a05..7a88fd9ad77 100644 --- a/include/strata/kernels/ngram.hpp +++ b/include/strata/kernels/ngram.hpp @@ -102,12 +102,14 @@ void ple_dequant_row(int type, const uint8_t* row, float* out160); /// How the table's rows are read (plan v0.3 P2). `Direct` is the default: unbuffered 4 KiB reads from the SSD, /// so the table never occupies RAM or the OS file cache. `Mmap` is the earlier memory-mapped path, kept as the -/// A/B arm; it returns the same bytes. +/// A/B arm; it returns the same bytes. `Ram`: Direct until `start_ram_load` has read the whole table into RAM +/// (unbuffered, so the file cache holds no second copy), then from RAM: a 16K-token prompt chunk's 262K random +/// row reads take ~0.6 s from the SSD. /// /// NEVER KEEP THE SHARD MAPPED WHILE READING IT DIRECT: a live section on the same file serializes the unbuffered /// reads (311 -> 1,575 us per token, bench/results/2026-09-23-p2-ssd-direct). Direct mode drops its own mapping /// after the header parse; nothing else in the process may hold one. -enum class PleIo { Direct, Mmap }; +enum class PleIo { Direct, Mmap, Ram }; struct PleIoOptions { PleIo mode = PleIo::Direct; @@ -143,6 +145,11 @@ class PleTable { /// Fault injection (Direct mode): every row read completes no earlier than `us` after issue. void set_injected_delay_us(double us); + /// Ram mode: reads the table into (pageable) RAM on a thread of its own; the rows come from the SSD until it is + /// done. Reports on stderr when done. + bool start_ram_load(std::string& err); + bool ram_ready() const; + /// Maps the ORIGINAL second GGUF shard read-only. The tensor's data does NOT start at file offset 0: /// the manifest's `shard2_tensor.offset` is relative to the GGUF's DATA SECTION, and the header before it /// is 192 bytes. Reading at 0 would silently decode the header plus 192 bytes of shifted rows - still diff --git a/src/kernels/ngram.cpp b/src/kernels/ngram.cpp index 582940ae5bf..e1f585325d0 100644 --- a/src/kernels/ngram.cpp +++ b/src/kernels/ngram.cpp @@ -5,6 +5,12 @@ #include "strata/artifact/gguf_reader.hpp" #include "strata/kernels/f16_bits.hpp" #include "strata/ngram/ple_reader.hpp" +#include "strata/platform/direct_file.hpp" + +#include +#include +#include +#include #include #include @@ -147,8 +153,73 @@ struct PleTable::Impl { bool pending = false; uint32_t rows[PLE_N_HEADS] = {}; uint8_t raw[PLE_N_HEADS * PLE_ROW_BYTES_Q8_0] = {}; + // Ram mode: the table read into `ram_base` by `loader`; `ram` (the table's first row) valid once `ram_ready` + std::string path; + uint64_t table_offset = 0; + uint8_t* ram_base = nullptr; + uint64_t ram_bytes = 0; + const uint8_t* ram = nullptr; + std::atomic ram_ready{false}, cancel{false}; + std::thread loader; + std::string ram_note; + bool pending_ram = false; // the pending token's rows come from RAM + + bool from_ram() const { return mode == PleIo::Ram && ram_ready.load(std::memory_order_acquire); } + bool direct() const { return (mode == PleIo::Direct || mode == PleIo::Ram) && !from_ram(); } + void load(); }; +// The whole table in 8 MiB unbuffered reads, four in flight. Pageable: locking it resident first (VirtualLock of +// 28.8 GB) faults in its pages under the process's working-set lock and stalled the startup's other threads ~8 s. +void PleTable::Impl::load() { + using strata::platform::DirectFile; + const auto t0 = std::chrono::steady_clock::now(); + const uint64_t a0 = table_offset & ~(uint64_t) (DirectFile::alignment() - 1); + const uint64_t need = table_offset + n_rows * (uint64_t) row_bytes - a0; + DirectFile f; + std::string err; + if (!f.open(path, err)) { ram_note = "PLE table in RAM: " + err; std::fprintf(stderr, "strata: %s\n", ram_note.c_str()); return; } + constexpr uint32_t kChunk = 8u << 20; + constexpr int kDepth = 4; + uint64_t next = 0, got = 0; + int inflight = 0; + bool ok = true; + while (ok && !cancel.load() && (next < need || inflight > 0)) { + while (ok && inflight < kDepth && next < need) { + const uint32_t len = (uint32_t) std::min(kChunk, ram_bytes - next); + ok = f.submit(a0 + next, ram_base + next, len, next, err); + next += len; + ++inflight; + } + strata::platform::Completion c[kDepth]; + const int k = ok ? f.wait(c, kDepth, -1) : 0; + for (int i = 0; i < k; ++i) { + if (c[i].tag == DirectFile::WAKE_TAG) continue; + --inflight; + ok = ok && c[i].ok; + got += c[i].bytes; + } + } + while (inflight > 0) { // a failed or cancelled load still collects what it queued + strata::platform::Completion c[kDepth]; + const int k = f.wait(c, kDepth, -1); + for (int i = 0; i < k; ++i) if (c[i].tag != DirectFile::WAKE_TAG) --inflight; + } + if (cancel.load()) return; + if (!ok || got < need) { + ram_note = "PLE table in RAM: the load failed" + (err.empty() ? std::string() : ": " + err) + "; rows stay on the SSD"; + std::fprintf(stderr, "strata: %s\n", ram_note.c_str()); + return; + } + ram = ram_base + (table_offset - a0); + const double s = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + char buf[120]; + std::snprintf(buf, sizeof buf, "PLE table in RAM: %.1f GB in %.1f s", (double) need / 1e9, s); + ram_note = buf; + ram_ready.store(true, std::memory_order_release); + std::fprintf(stderr, "strata: %s\n", ram_note.c_str()); +} + PleTable::PleTable() : impl_(new Impl) {} PleTable::~PleTable() { close(); delete impl_; } @@ -210,10 +281,12 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO close(); return false; } - if (io.mode == PleIo::Direct) { + if (io.mode == PleIo::Direct || io.mode == PleIo::Ram) { // The parse above is the validated source of the offset; the mapping itself is not kept, so no page of // the table can enter this process's working set or the file cache through it. const uint64_t table_offset = impl_->file->data_start() + t->offset; + impl_->path = gguf_path; + impl_->table_offset = table_offset; const uint64_t n_rows = impl_->n_rows; delete impl_->file; impl_->file = nullptr; @@ -230,6 +303,17 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO } void PleTable::close() { + if (impl_->loader.joinable()) { + impl_->cancel = true; + impl_->loader.join(); + } + impl_->ram_ready = false; + impl_->cancel = false; + impl_->ram = nullptr; + if (impl_->ram_base != nullptr) { + strata::platform::DirectFile::free_aligned(impl_->ram_base); + impl_->ram_base = nullptr; + } impl_->reader.close(); impl_->pending = false; impl_->mode = PleIo::Mmap; @@ -240,12 +324,35 @@ void PleTable::close() { } bool PleTable::is_open() const { return impl_->data != nullptr || impl_->reader.is_open(); } + +bool PleTable::start_ram_load(std::string& err) { + using strata::platform::DirectFile; + if (impl_->mode != PleIo::Ram || impl_->ram_base != nullptr) return true; + const uint64_t a0 = impl_->table_offset & ~(uint64_t) (DirectFile::alignment() - 1); + const uint64_t end = impl_->table_offset + impl_->n_rows * (uint64_t) impl_->row_bytes; + impl_->ram_bytes = (end - a0 + DirectFile::alignment() - 1) / DirectFile::alignment() * DirectFile::alignment(); + impl_->ram_base = (uint8_t*) DirectFile::alloc_aligned((size_t) impl_->ram_bytes); + if (impl_->ram_base == nullptr) { + err = "PLE table in RAM: " + std::to_string(impl_->ram_bytes >> 20) + " MiB could not be allocated"; + return false; + } + impl_->loader = std::thread([m = impl_] { m->load(); }); + return true; +} + +bool PleTable::ram_ready() const { return impl_->from_ram(); } PleIo PleTable::mode() const { return impl_->mode; } uint64_t PleTable::rows() const { return impl_->n_rows; } uint64_t PleTable::bytes_read() const { return impl_->bytes_read; } void PleTable::read_row(uint32_t row, float* out160) const { - if (impl_->mode == PleIo::Direct && impl_->reader.is_open()) { + if (impl_->from_ram()) { + if (row >= impl_->n_rows) std::memset(out160, 0, (size_t) PLE_HEAD_DIM * sizeof(float)); + else ple_dequant_row(impl_->type, impl_->ram + (size_t) row * impl_->row_bytes, out160); + impl_->bytes_read += impl_->row_bytes; + return; + } + if (impl_->direct() && impl_->reader.is_open()) { uint8_t raw[PLE_ROW_BYTES_Q8_0]; std::string err; const auto t = impl_->reader.issue(&row, 1, raw); @@ -267,9 +374,10 @@ void PleTable::read_row(uint32_t row, float* out160) const { bool PleTable::issue(const uint32_t* rows16) { std::memcpy(impl_->rows, rows16, sizeof impl_->rows); - if (impl_->mode == PleIo::Direct) { + if (impl_->mode != PleIo::Mmap) { if (impl_->pending) return false; // one token in flight per table - impl_->ticket = impl_->reader.issue(impl_->rows, PLE_N_HEADS, impl_->raw); + impl_->pending_ram = impl_->from_ram(); + if (!impl_->pending_ram) impl_->ticket = impl_->reader.issue(impl_->rows, PLE_N_HEADS, impl_->raw); impl_->pending = true; return true; } @@ -293,7 +401,11 @@ bool PleTable::issue(const uint32_t* rows16) { bool PleTable::collect(float* out2560, std::string& err) { if (!impl_->pending) { err = "PleTable::collect without issue"; return false; } impl_->pending = false; - if (impl_->mode == PleIo::Direct) { + if (impl_->pending_ram) { + for (int h = 0; h < PLE_N_HEADS; ++h) read_row(impl_->rows[h], out2560 + (size_t) h * PLE_HEAD_DIM); + return true; + } + if (impl_->mode != PleIo::Mmap) { if (!impl_->reader.collect(impl_->ticket, err)) return false; for (int h = 0; h < PLE_N_HEADS; ++h) ple_dequant_row(impl_->type, impl_->raw + (size_t) h * impl_->row_bytes, out2560 + (size_t) h * PLE_HEAD_DIM); @@ -307,7 +419,7 @@ bool PleTable::collect(float* out2560, std::string& err) { bool PleTable::gather_batch(const uint32_t* rows, size_t n_tokens, float* out, std::string& err) { if (impl_->pending) { err = "PleTable::gather_batch while a token is in flight"; return false; } const size_t n = n_tokens * (size_t) PLE_N_HEADS; - if (impl_->mode == PleIo::Direct) { + if (impl_->direct()) { std::vector raw(n * impl_->row_bytes); const auto ticket = impl_->reader.issue(rows, n, raw.data()); if (!impl_->reader.collect(ticket, err)) return false; @@ -323,7 +435,8 @@ bool PleTable::gather_batch(const uint32_t* rows, size_t n_tokens, float* out, s void PleTable::set_injected_delay_us(double us) { impl_->reader.set_injected_delay_us(us); } std::string PleTable::io_report() const { - if (impl_->mode != PleIo::Direct || !impl_->reader.is_open()) return {}; + if (impl_->from_ram()) return "ple io: " + impl_->ram_note; + if (impl_->mode == PleIo::Mmap || !impl_->reader.is_open()) return {}; const strata::ngram::ReaderStats& s = impl_->reader.stats(); char buf[320]; std::snprintf(buf, sizeof buf, @@ -337,7 +450,11 @@ std::string PleTable::io_report() const { } void PleTable::gather(const uint32_t* rows16, float* out2560) const { - if (impl_->mode == PleIo::Direct) { + if (impl_->from_ram()) { + for (int h = 0; h < PLE_N_HEADS; ++h) read_row(rows16[h], out2560 + (size_t) h * PLE_HEAD_DIM); + return; + } + if (impl_->direct()) { // `gather` stays const for its existing callers; the reader's state is the table's I/O state. PleTable* self = const_cast(this); std::string err; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 5a31672e431..927bed88015 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -281,8 +281,9 @@ void usage() { " --ple-gguf PATH the PLE table: the model shard holding per_layer_token_embd (with --native,\n" " the --native model's shard that holds it by default)\n" " --no-ple explicit diagnostic ablation of the PLE layer\n" - " --ple-io direct|mmap n-gram table reads (plan v0.3 P2). direct (default): unbuffered SSD\n" - " reads, the table never enters RAM or the file cache; mmap: A/B arm\n" + " --ple-io direct|mmap|ram n-gram table reads (plan v0.3 P2). direct (default): unbuffered SSD\n" + " reads, the table never enters RAM or the file cache; mmap: A/B arm; ram:\n" + " read into RAM once the experts are loaded (28.8 GB), direct reads until then\n" " --ple-row-cache N bounded cache of fetched rows, 90 B each (default 1048576; 0 = off)\n" " --ple-inflight N outstanding SSD reads (default 64)\n" " --ple-delay-us U fault injection: each row read completes no earlier than U us\n" @@ -785,7 +786,7 @@ int main(int argc, char** argv) { return 2; } - if ((o.ple_io != "direct" && o.ple_io != "mmap") || o.ple_row_cache < 0 || o.ple_inflight < 1 || + if ((o.ple_io != "direct" && o.ple_io != "mmap" && o.ple_io != "ram") || o.ple_row_cache < 0 || o.ple_inflight < 1 || o.ple_inflight > 1024 || !(o.ple_delay_us >= 0)) { std::fprintf(stderr, "strata generate: invalid --ple-io/--ple-row-cache/--ple-inflight/--ple-delay-us\n"); return 2; @@ -1101,7 +1102,9 @@ int main(int argc, char** argv) { float* ple_scratch = nullptr; if (!o.ple_gguf.empty()) { strata::kernels::PleIoOptions pio; - pio.mode = o.ple_io == "mmap" ? strata::kernels::PleIo::Mmap : strata::kernels::PleIo::Direct; + pio.mode = o.ple_io == "mmap" ? strata::kernels::PleIo::Mmap + : o.ple_io == "ram" ? strata::kernels::PleIo::Ram + : strata::kernels::PleIo::Direct; pio.max_inflight = (uint32_t) o.ple_inflight; pio.cache_rows = (uint64_t) o.ple_row_cache; pio.io_thread = !o.ple_sync_submit; @@ -1229,6 +1232,9 @@ int main(int argc, char** argv) { if (!o.mtp.empty()) mtp.set_prompt_len((int64_t) o.tokens.size()); if (!o.mtp.empty() && !mtp.load(o.mtp, g, ss, o.spec, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } } + // --ple-io ram: the table goes into RAM once the SSD has delivered the experts and the draft layer + if (ple_table.is_open() && !ple_table.start_ram_load(err)) + std::fprintf(stderr, "strata generate: %s; its rows stay on the SSD\n", err.c_str()); strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker); if (o.no_ple_prefetch) strata::kernels::ple_prefetch_enable(false); // ---- R4's slot storage. Allocated AFTER the weights and the session, so `cudaMemGetInfo` inside `open` From e34cea03d6e75990ff348033c42348d4f896f4a0 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 07:14:48 +0200 Subject: [PATCH 29/92] --second-gpu-prefetch-mb (default 12) instead of an expert count: the idle RAM window holds 4 UD-Q4_K_XL or ~7 IQ3_XXS experts (IQ3_XXS code 98.9 -> 108.3 t/s with the second GPU) --- include/strata/core/expert_source.hpp | 1 + include/strata/core/second_gpu.hpp | 9 +++++---- src/core/expert_source.cpp | 3 ++- src/core/second_gpu.cpp | 2 +- src/program/generate.cpp | 18 ++++++++++++------ 5 files changed, 21 insertions(+), 12 deletions(-) diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index 5724ed914a3..6f63f70482a 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -227,6 +227,7 @@ struct ExpertDispatch { int64_t gpu2_skipped = 0; ///< layers the CPU took the second GPU's share of (it was quicker) bool gpu2_used = false; ///< this layer's decision uint64_t gpu2_min_bytes = 0; ///< a layer's miss bytes from which the second GPU takes its share + uint64_t gpu2_prefetch_bytes = 0; ///< per layer, its likeliest experts no GPU holds, copied there ahead std::string gpu2_err; ///< `fail` points here when the second GPU failed /// Set when `dispatch` could not produce an answer. The loop itself has no error channel, so this is /// where a source failure surfaces: the driver checks it after `session_loop` returns rather than the diff --git a/include/strata/core/second_gpu.hpp b/include/strata/core/second_gpu.hpp index eb71662cc31..e12a5b96cd3 100644 --- a/include/strata/core/second_gpu.hpp +++ b/include/strata/core/second_gpu.hpp @@ -9,9 +9,10 @@ // took ~0.2 ms a layer and slowed the CPU pool's RAM reads. One graph per format pair and group-count bucket: the // grouped kernels' grids are sized for the bucket. Native packs only: the kernels are the first GPU's hit path. // -// Prefetch: between a layer's CPU pool and the next layer's ring the RAM is idle (the first GPU's serial part, ~0.3 ms -// a layer), and this GPU's x16 link copies ~4 experts in that time. The next layer's likeliest experts that neither -// GPU holds go into a few slots of their own, and that layer's share takes them from there. +// Prefetch: between a layer's CPU pool and the next layer's ring the RAM is idle (the first GPU's serial part, ~0.28 ms +// a layer), and this GPU's x16 link copies ~12 MiB in that time: 4 UD-Q4_K_XL or ~7 IQ3_XXS experts. The next layer's +// likeliest experts that neither GPU holds go into a few slots of their own, and that layer's share takes them from +// there. #pragma once #include "strata/core/expert_cache.hpp" @@ -37,7 +38,7 @@ class SecondGpu { ExpertCache& cache() { return cache_; } int device() const { return dev_; } - /// Room for `slots` prefetched experts of up to `blob_bytes`. After init(). + /// Room for `slots` prefetched experts (at most 16) of up to `blob_bytes`. After init(). bool init_prefetch(int slots, uint64_t blob_bytes, std::string& err); int prefetch_slots() const { return pre_max_; } /// Starts copying `layer`'s experts ids[0..n) (n <= prefetch_slots(), `bytes` each from the pinned blobs `src`) into diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index ba2b5f63e9c..30a66c55b96 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -520,7 +520,8 @@ void expert_prefetch_multi(ExpertDispatch& d, int64_t layer, const int32_t* ids, if (c == nc) { cand[nc] = e; score[nc++] = 0.0f; } score[c] += w[i]; } - const int m = (std::min)(nc, d.gpu2->prefetch_slots()); + const int m = (int) (std::min)((uint64_t) (std::min)(nc, d.gpu2->prefetch_slots()), + d.gpu2_prefetch_bytes / strata::kernels::cpu::expert_layout().blob_bytes(layer)); const uint8_t* src[128]; for (int q = 0; q < m; ++q) { int best = q; diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp index 924f7a82084..68adfae3a9a 100644 --- a/src/core/second_gpu.cpp +++ b/src/core/second_gpu.cpp @@ -86,8 +86,8 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, } bool SecondGpu::init_prefetch(int slots, uint64_t blob_bytes, std::string& err) { + slots = (std::min)(slots, kPrefetchMax); if (slots <= 0) return true; - if (slots > kPrefetchMax) { err = "second GPU: at most " + std::to_string(kPrefetchMax) + " prefetch slots"; return false; } DeviceScope scope(dev_, main_); pre_cap_ = (blob_bytes + 255) / 256 * 256; if (cudaStreamCreateWithFlags(&pre_s_, cudaStreamNonBlocking) != cudaSuccess || diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 927bed88015..d23b6c73f36 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -233,7 +233,7 @@ struct Options { double second_gpu_gib = 0.0; int second_gpu_reserve_mib = 2048; double second_gpu_min_mb = 4.0; ///< a layer's misses from which it takes its share (smaller: the CPU is quicker) - int second_gpu_prefetch = 4; ///< per layer, the likeliest experts no GPU holds copied to it ahead (0 = off) + double second_gpu_prefetch_mb = 12.0; ///< per layer, the likeliest experts no GPU holds copied to it ahead (0 = off) /// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the /// pinned arena while the CPU computes the rest (verify windows). double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs) @@ -351,9 +351,9 @@ void usage() { " --second-gpu-gib G its VRAM for experts (default: all but --second-gpu-reserve-mib, 2048)\n" " --second-gpu-min-mb M a layer's missed experts from which it takes its share (default 4; below\n" " it the CPU is quicker than its ~100 us round trip)\n" - " --second-gpu-prefetch N per layer, copy its N likeliest experts that no GPU holds (the next\n" - " layer's router on this layer's input) to it while the RAM is idle (default 4;\n" - " 0 = off)\n" + " --second-gpu-prefetch-mb M per layer, copy its likeliest experts that no GPU holds (the next\n" + " layer's router on this layer's input), up to M MiB, to it while the RAM is idle\n" + " (default 12: ~0.28 ms at its ~48 GB/s; 0 = off)\n" " --no-prompt-offload with --second-gpu: the prompt path streams the experts the main GPU's cache\n" " lacks to the main GPU instead of computing them on the second\n" " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" @@ -727,7 +727,7 @@ int main(int argc, char** argv) { else if (a == "--second-gpu-gib") o.second_gpu_gib = std::atof(next("--second-gpu-gib")); else if (a == "--second-gpu-reserve-mib") o.second_gpu_reserve_mib = std::atoi(next("--second-gpu-reserve-mib")); else if (a == "--second-gpu-min-mb") o.second_gpu_min_mb = std::atof(next("--second-gpu-min-mb")); - else if (a == "--second-gpu-prefetch") o.second_gpu_prefetch = std::atoi(next("--second-gpu-prefetch")); + else if (a == "--second-gpu-prefetch-mb") o.second_gpu_prefetch_mb = std::atof(next("--second-gpu-prefetch-mb")); else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac")); else if (a == "--adapt-every") o.adapt_every = std::atoi(next("--adapt-every")); else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); @@ -1993,8 +1993,14 @@ int main(int argc, char** argv) { const int main_dev = o.main_gpu; cudaDeviceProp prop{}; cudaGetDeviceProperties(&prop, o.second_gpu); + // prefetch slots for the most of the smallest experts the budget holds + const auto& lay0 = strata::kernels::cpu::expert_layout(); + uint64_t min_blob = lay0.max_blob; + for (int64_t l = 0; l < g.n_layers; ++l) min_blob = std::min(min_blob, lay0.blob_bytes(l)); + const uint64_t pf_bytes = (uint64_t) (std::max(0.0, o.second_gpu_prefetch_mb) * 1048576.0); + drive.d.gpu2_prefetch_bytes = pf_bytes; if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err) || - !gpu2.init_prefetch(o.second_gpu_prefetch, strata::kernels::cpu::expert_layout().max_blob, err)) { + !gpu2.init_prefetch((int) std::min(pf_bytes / min_blob, 64), lay0.max_blob, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } From 00974d25b2499110c3b9305d4a530a5ebfe5b0e8 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 07:44:44 +0200 Subject: [PATCH 30/92] Second GPU: without free VRAM there the engine runs on the main GPU alone (it failed to start); its budget always fills, the rest as empty slots (without the adaptive tier IQ3_XXS had nothing there and lost the prompt offload) --- src/core/adaptive_tier.cpp | 1 + src/program/generate.cpp | 124 +++++++++++++++++++------------------ 2 files changed, 66 insertions(+), 59 deletions(-) diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 3fc3876516e..01c55f3e87d 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -38,6 +38,7 @@ bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector host_res2; strata::core::AdaptiveTier tier2; @@ -1999,51 +2000,51 @@ int main(int argc, char** argv) { for (int64_t l = 0; l < g.n_layers; ++l) min_blob = std::min(min_blob, lay0.blob_bytes(l)); const uint64_t pf_bytes = (uint64_t) (std::max(0.0, o.second_gpu_prefetch_mb) * 1048576.0); drive.d.gpu2_prefetch_bytes = pf_bytes; - if (!gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err) || - !gpu2.init_prefetch((int) std::min(pf_bytes / min_blob, 64), lay0.max_blob, err)) { - std::fprintf(stderr, "strata generate: %s\n", err.c_str()); - return 1; - } - cudaSetDevice(o.second_gpu); - size_t free_b = 0, total_b = 0; - cudaMemGetInfo(&free_b, &total_b); - // the prompt path's buffers there borrow cache slots, else they are reserved here - const bool offload_own = o.prefill_chunk > 0 && !o.no_prompt_offload && o.no_prefill_borrow; - const uint64_t reserve = ((uint64_t) o.second_gpu_reserve_mib << 20) + - (offload_own ? strata::prefill::ExpertRunner::bytes_needed(o.prefill_chunk, g.n_expert, true, true) : 0); - const uint64_t budget = o.second_gpu_gib > 0 ? (uint64_t) (o.second_gpu_gib * 1073741824.0) - : (free_b > reserve ? (uint64_t) free_b - reserve : 0); + bool ok = gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err) && + gpu2.init_prefetch((int) std::min(pf_bytes / min_blob, 64), lay0.max_blob, err); const auto& lay = strata::kernels::cpu::expert_layout(); std::vector sizes; std::vector> pre; std::vector empty2; - uint64_t used = 0; - for (size_t i = (size_t) prefilled; i < profile.size(); ++i) { - const auto& pr = profile[i]; - if (host_res[(size_t) (pr.first * g.n_expert + pr.second)] >= 0) continue; // the first GPU holds it - const uint64_t b = (lay.blob_bytes(pr.first) + 255) / 256 * 256; - if (used + b > budget) break; - used += b; - sizes.push_back((int64_t) lay.blob_bytes(pr.first)); - pre.push_back(pr); - } - if (adaptive) { - uint64_t round = 0; - for (int64_t l = 0; l < g.n_layers; ++l) round += (lay.blob_bytes(l) + 255) / 256 * 256; - for (uint64_t q = (budget - used) / round; q > 0; --q) - for (int64_t l = 0; l < g.n_layers; ++l) { - sizes.push_back((int64_t) lay.blob_bytes(l)); - empty2.push_back((int32_t) l); - } - } - bool ok = !sizes.empty() && gpu2.cache().open_sized(sizes, g.n_layers, g.n_expert, err); - for (size_t i = 0; ok && i < pre.size(); ++i) { - const int32_t slot = gpu2.cache().admit(pre[i].first, pre[i].second); - const uint8_t* b = srcp->blob(pre[i].first, pre[i].second); - ok = slot != strata::core::kNotResident && b != nullptr && - gpu2.cache().fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pre[i].first)); + if (ok) { + cudaSetDevice(o.second_gpu); + size_t free_b = 0, total_b = 0; + cudaMemGetInfo(&free_b, &total_b); + // the prompt path's buffers there borrow cache slots, else they are reserved here + const bool offload_own = o.prefill_chunk > 0 && !o.no_prompt_offload && o.no_prefill_borrow; + const uint64_t reserve = ((uint64_t) o.second_gpu_reserve_mib << 20) + + (offload_own ? strata::prefill::ExpertRunner::bytes_needed(o.prefill_chunk, g.n_expert, true, true) : 0); + const uint64_t budget = o.second_gpu_gib > 0 ? (uint64_t) (o.second_gpu_gib * 1073741824.0) + : (free_b > reserve ? (uint64_t) free_b - reserve : 0); + uint64_t used = 0; + for (size_t i = (size_t) prefilled; i < profile.size(); ++i) { + const auto& pr = profile[i]; + if (host_res[(size_t) (pr.first * g.n_expert + pr.second)] >= 0) continue; // the first GPU holds it + const uint64_t b = (lay.blob_bytes(pr.first) + 255) / 256 * 256; + if (used + b > budget) break; + used += b; + sizes.push_back((int64_t) lay.blob_bytes(pr.first)); + pre.push_back(pr); + } + { // the rest in empty slots (without the adaptive tier only the prompt path's loans use them) + uint64_t round = 0; + for (int64_t l = 0; l < g.n_layers; ++l) round += (lay.blob_bytes(l) + 255) / 256 * 256; + for (uint64_t q = (budget - used) / round; q > 0; --q) + for (int64_t l = 0; l < g.n_layers; ++l) { + sizes.push_back((int64_t) lay.blob_bytes(l)); + empty2.push_back((int32_t) l); + } + } + if (sizes.empty()) err = "no VRAM for experts (" + std::to_string(free_b >> 20) + " MiB free)"; + ok = !sizes.empty() && gpu2.cache().open_sized(sizes, g.n_layers, g.n_expert, err); + for (size_t i = 0; ok && i < pre.size(); ++i) { + const int32_t slot = gpu2.cache().admit(pre[i].first, pre[i].second); + const uint8_t* b = srcp->blob(pre[i].first, pre[i].second); + ok = slot != strata::core::kNotResident && b != nullptr && + gpu2.cache().fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pre[i].first)); + } + cudaSetDevice(main_dev); } - cudaSetDevice(main_dev); if (ok) { host_res2.assign((size_t) (g.n_layers * g.n_expert), strata::core::kNotResident); for (int64_t l = 0; l < g.n_layers; ++l) @@ -2052,25 +2053,30 @@ int main(int argc, char** argv) { if (ok && adaptive) { ok = tier2.init(gpu2.cache(), *srcp, host_res2, nullptr, g.n_layers, g.n_expert, o.adapt_swaps, err, o.second_gpu, main_dev); - // lowest slots first: the last ones, which the prompt path borrows, stay empty longest - for (size_t i = empty2.size(); ok && i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); - tier2.set_upper(&tier); - // with the prefetch, the tier's copies go four per layer behind it, an update within ~24 layers: queued in - // front of the prefetch copies, an update's ~300 MB would hold them up for ~7 ms - if (gpu2.prefetch_slots() > 0) { - tier2.set_pace(4); - drive.tier2 = &tier2; + if (ok) { + // lowest slots first: the last ones, which the prompt path borrows, stay empty longest + for (size_t i = empty2.size(); i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); + tier2.set_upper(&tier); + // with the prefetch, the tier's copies go four per layer behind it, an update within ~24 layers: + // queued in front of the prefetch copies, an update's ~300 MB would hold them up for ~7 ms + if (gpu2.prefetch_slots() > 0) { + tier2.set_pace(4); + drive.tier2 = &tier2; + } } } - if (!ok) { - std::fprintf(stderr, "strata generate: second GPU: %s\n", sizes.empty() ? "no VRAM for experts" : err.c_str()); - return 1; + if (ok) { + drive.d.gpu2 = &gpu2; + drive.d.host_res2 = host_res2.data(); + drive.d.gpu2_min_bytes = (uint64_t) (o.second_gpu_min_mb * 1048576.0); + std::fprintf(stderr, "strata generate: second GPU %d (%s): %zu experts from the profile and %zu empty slots, " + "%.2f GiB\n", o.second_gpu, prop.name, pre.size(), empty2.size(), gpu2.cache().gib()); + } else { + // another process may hold its VRAM (a model, a desktop app): the engine runs on the main GPU alone + std::fprintf(stderr, "strata generate: second GPU %d unused: %s\n", o.second_gpu, err.c_str()); + o.second_gpu = -1; + host_res2.clear(); } - drive.d.gpu2 = &gpu2; - drive.d.host_res2 = host_res2.data(); - drive.d.gpu2_min_bytes = (uint64_t) (o.second_gpu_min_mb * 1048576.0); - std::fprintf(stderr, "strata generate: second GPU %d (%s): %zu experts from the profile and %zu empty slots, " - "%.2f GiB\n", o.second_gpu, prop.name, pre.size(), empty2.size(), gpu2.cache().gib()); } // the second tier ranks after the first on the same counts, and decays them std::string adapt_err; @@ -2181,7 +2187,7 @@ int main(int argc, char** argv) { ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 - if (gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); std::vector cur; prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { std::vector nxt((size_t) T); @@ -2821,7 +2827,7 @@ int main(int argc, char** argv) { ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 - if (gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); drive.d.plan = ver.plan_sink(); drive.d.pcie_num = (int) (o.pcie_frac * 256.0 + 0.5); if (drive.d.pcie_num < 0) drive.d.pcie_num = 0; From a085c36dc68951bb07f00ca036aa486e84d93418 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 08:25:07 +0200 Subject: [PATCH 31/92] README: speeds with the second GPU for prompts and generation; --feed-max comment with the current break-even --- README.md | 12 +++++++----- src/program/generate.cpp | 6 +++--- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 68dca8a0956..5e6a11c574b 100644 --- a/README.md +++ b/README.md @@ -94,15 +94,17 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 572-608 tok/s, generation 52-60 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 945-1043 tok/s, generation 57-66 tok/s. ### A second GPU With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0) holds more experts and computes its share of -each layer while the CPU computes the rest; it may be the card that drives the display. Build for both GPUs' -architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). UD-Q4_K_XL on an RTX 3090 plus an RTX 5070 Ti (12.6 GB of -its VRAM): generation 59-69 tok/s, +9-19%. It pays when the CPU's share of a token is the larger one: with -IQ3_XXS, whose experts mostly fit the first GPU, it gains nothing. +each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM +is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for +both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its +VRAM), 16K-131K prompts: UD-Q4_K_XL 1435-1724 tok/s instead of 945-1043, generation 79-93 tok/s instead of 57-66; +IQ3_XXS 1468-1791 instead of 1241-1458, generation 82-110 instead of 79-99. When the card has no free VRAM, the +engine runs on the main GPU alone. ## Before you start diff --git a/src/program/generate.cpp b/src/program/generate.cpp index ea3756e6375..45e53ad50cd 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -264,9 +264,9 @@ struct Options { int64_t cache_every = 16384; /// Prompt parts of up to this many tokens go through verify windows instead of the batched prompt path: a new /// part that short, and a last prompt chunk that short. A batched call has a fixed cost (it lends and refills - /// expert-cache slots and streams every expert its tokens use): ~1.5 s with IQ3_XXS on the 3090 alone, ~0.7 s - /// with UD-Q4_K_XL and the second GPU; verify windows cost 12-17 ms per token. They break even at ~185 and - /// ~45 tokens (bench/feed_test.py). + /// expert-cache slots and reads every expert its tokens use): ~0.6 s with a second GPU, ~1.5 s with IQ3_XXS on + /// the 3090 alone; verify windows cost 10-15 ms per token. They break even at ~50-65 tokens with a second GPU, + /// ~185 without (bench/feed_test.py). int64_t feed_max = 128; int adapt_swaps = 96; }; From 00aa7bbcf5447e692fc9d2bbc86e396131ec189c Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 14:01:27 +0200 Subject: [PATCH 32/92] The output head is loaded before the expert cache is sized: UD-Q4_K_XL's Q8_0 head (644 MiB) came after it and left the 3090 over-committed (0 MiB free with everything loaded, 462 MiB now; its cache 4537 slots instead of 4752, generation -0.4 to -0.9%, prompts unchanged) Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- src/program/generate.cpp | 48 ++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index badb82f6bd4..dd7ca5b8898 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -1250,10 +1250,30 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s; its rows stay on the SSD\n", err.c_str()); strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker); if (o.no_ple_prefetch) strata::kernels::ple_prefetch_enable(false); - // ---- R4's slot storage. Allocated AFTER the weights and the session, so `cudaMemGetInfo` inside `open` - // sees the memory this process actually has left rather than the card's idle figure - and refuses with both - // numbers if the slots do not fit, instead of handing back a cache smaller than it was asked for. - mem_mark("the weights, the session and the drafter"); + std::fprintf(stderr, "strata generate: session is up; locating the head\n"); + const strata::core::WeightRef* wo = wt.find("output.weight"); + if (wo == nullptr) { std::fprintf(stderr, "strata generate: output.weight is missing\n"); return 1; } + const int64_t n_vocab = wo->ne1; + strata::core::NativeHead native_head; + if (!o.native_head_gguf.empty()) { + std::vector head_shards; + try { + head_shards = strata::gguf_split_paths(o.native_head_gguf); + } catch (const std::exception& e) { + std::fprintf(stderr, "strata generate: native head: %s\n", e.what()); + return 1; + } + if (!native_head.load(head_shards, g.n_embd, n_vocab, err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } + std::fprintf(stderr, "strata generate: native head, type %d, %llu bytes\n", native_head.type(), + (unsigned long long) native_head.weight_bytes()); + } + // ---- R4's slot storage. Allocated AFTER the weights, the session and the head, so `cudaMemGetInfo` inside + // `open` sees the memory this process actually has left rather than the card's idle figure - and refuses with + // both numbers if the slots do not fit, instead of handing back a cache smaller than it was asked for. + mem_mark("the weights, the session, the drafter and the head"); strata::core::ExpertCache xcache; std::vector> profile; if (!o.expert_profile.empty()) { @@ -1583,26 +1603,6 @@ int main(int argc, char** argv) { } mem_mark("the expert cache and the graphs"); - std::fprintf(stderr, "strata generate: session is up; locating the head\n"); - const strata::core::WeightRef* wo = wt.find("output.weight"); - if (wo == nullptr) { std::fprintf(stderr, "strata generate: output.weight is missing\n"); return 1; } - const int64_t n_vocab = wo->ne1; - strata::core::NativeHead native_head; - if (!o.native_head_gguf.empty()) { - std::vector head_shards; - try { - head_shards = strata::gguf_split_paths(o.native_head_gguf); - } catch (const std::exception& e) { - std::fprintf(stderr, "strata generate: native head: %s\n", e.what()); - return 1; - } - if (!native_head.load(head_shards, g.n_embd, n_vocab, err)) { - std::fprintf(stderr, "strata generate: %s\n", err.c_str()); - return 1; - } - std::fprintf(stderr, "strata generate: native head, type %d, %llu bytes\n", native_head.type(), - (unsigned long long) native_head.weight_bytes()); - } std::vector logits((size_t) n_vocab); float* d_logits = nullptr; if (cudaMalloc(&d_logits, (size_t) n_vocab * 4) != cudaSuccess) { From 8b783bbfa5434ba3c2a3881c0d391d0db3f380c7 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 14:16:11 +0200 Subject: [PATCH 33/92] Prefetch: a prediction that comes after the CPU's rows is waited for (microseconds: its branch is joined before the next layer) instead of skipped, so which experts the 5070 Ti takes, and the output, no longer depend on timing (with --adapt-every 0 two engine processes produce the same tokens); IQ3_XXS +1.3%, UD-Q4_K_XL thinking +0.5%, code unchanged; --stats counts the waits Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 6 +++--- src/core/verify.cpp | 18 +++++++++++++----- src/program/generate.cpp | 5 +++-- 3 files changed, 19 insertions(+), 10 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index d917351b408..5669b5a5506 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -93,12 +93,12 @@ class Verifier { void set_profile(bool on) { profile_ = on; } void print_profile() const; /// Each layer's router also runs on the previous layer's FFN input, on a side branch of the graph; once the CPU's - /// rows of a layer are in, `fn` gets the next layer's prediction (the RAM is idle until the next ring). Windows - /// of one token group. Set before the first `run`. + /// rows of a layer are in, `fn` gets the next layer's prediction (the RAM is idle until the next ring), waited + /// for when it comes later (`predict_late`). Windows of one token group. Set before the first `run`. void set_predict(PredictFn fn, void* user) { predict_ = fn; predict_user_ = user; } double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0, ms_predict = 0; - int64_t windows = 0; + int64_t windows = 0, predict_late = 0; private: bool capture(int T, std::string& err); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index e93216814bb..9366fc3c65a 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -890,12 +890,20 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *flag = want; ms_wait += std::chrono::duration(b - a).count(); ms_pool += ms_since(b); - // the next layer's prediction, published during this layer's pool - if (predict_ != nullptr && G == 1 && l + 1 < g.n_layers && *(volatile uint32_t*) h_pseq_ >= (uint32_t) (l + 1)) { - std::atomic_thread_fence(std::memory_order_acquire); + // the next layer's prediction, published during this layer's pool, else microseconds later (its branch is + // joined before the next layer): waited for, so the experts the second GPU takes never depend on timing + if (predict_ != nullptr && G == 1 && l + 1 < g.n_layers && native_router_enabled()) { const Clock::time_point c = Clock::now(); - const size_t half = (size_t) ((l + 1) & 1) * (size_t) max_t_ * (size_t) ss.k; - predict_(predict_user_, l + 1, h_pids_ + half, h_pw_ + half, T, ss.k); + if (*(volatile uint32_t*) h_pseq_ < (uint32_t) (l + 1)) { + ++predict_late; + while (*(volatile uint32_t*) h_pseq_ < (uint32_t) (l + 1) && Clock::now() - c < std::chrono::milliseconds(50)) + _mm_pause(); + } + if (*(volatile uint32_t*) h_pseq_ >= (uint32_t) (l + 1)) { + std::atomic_thread_fence(std::memory_order_acquire); + const size_t half = (size_t) ((l + 1) & 1) * (size_t) max_t_ * (size_t) ss.k; + predict_(predict_user_, l + 1, h_pids_ + half, h_pw_ + half, T, ss.k); + } ms_predict += ms_since(c); } } diff --git a/src/program/generate.cpp b/src/program/generate.cpp index dd7ca5b8898..63f5df6d369 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -3168,12 +3168,13 @@ int main(int argc, char** argv) { std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " "%.3f ms/round waiting for it after the CPU pool; %lld experts swapped in and %lld into empty " "slots (%lld still empty); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round " - "of host time)\n", "second GPU", + "of host time, %lld predictions waited for)\n", "second GPU", (double) gpu2.entries_done / (double) rounds, (double) gpu2.experts / (double) rounds, (double) gpu2.layers / (double) rounds, (double) drive.d.gpu2_skipped / (double) rounds, gpu2.ms_wait / (double) rounds, (long long) tier2.swaps, (long long) tier2.fills, (long long) tier2.free_slots(), (double) gpu2.prefetch_copied / (double) rounds, - (double) gpu2.prefetch_used / (double) rounds, ver.ms_predict / (double) rounds); + (double) gpu2.prefetch_used / (double) rounds, ver.ms_predict / (double) rounds, + (long long) ver.predict_late); if (rounds > 0 && !drive.d.routed.empty()) { // the share of the routed entries the N most-routed experts of this run take std::vector c = drive.d.routed; From d538e3f2b7d2780ebe7a313bd4eb76ef37b0b966 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 15:57:03 +0200 Subject: [PATCH 34/92] QSA block selection: scores read each key once for all queries, the top-k keeps its keys in registers The block scores ran one warp per (query, block) and read the pooled keys once per query; the top-k ran one 256-thread block per query whose passes each re-read the scores, with shared atomics on a few hot bins and thread 0's serial scans. Now 8-lane groups score two key blocks each for every query (the queries and their step values in shared memory), summing each score in the tree the warp butterfly used, so the scores are bit for bit the same; the top-k holds the keys in registers (1024 threads, rows of 1024 blocks, 131K cells), counts in per-warp histograms, and places the cells from ballots and one scan of the (row, warp) totals. qsa_select_parity keeps the replaced kernels and checks the scores bitwise and the selections (71 cases, decode windows and prompt batches, 0 to 200K cells, ties, zero scores, every tail length). IQ3_XXS with --adapt-every 0: window hashes and tokens identical to the previous build after 65K and 131K-token prompts; a 131K-token prompt 90.5 -> 70.6 s, generation at 131K 79.2 -> 85.1 t/s; 65K 38.3 -> 35.3 s, 87.6 -> 90.2 t/s. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- CMakeLists.txt | 5 + include/strata/kernels/qsa_select.hpp | 22 +- src/kernels/cuda/qsa_select.cu | 360 +++++++++++++++++------- src/kernels/qsa_select_parity.cu | 382 ++++++++++++++++++++++++++ 4 files changed, 665 insertions(+), 104 deletions(-) create mode 100644 src/kernels/qsa_select_parity.cu diff --git a/CMakeLists.txt b/CMakeLists.txt index d65c77ecbe6..33b9eddecb5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -368,6 +368,11 @@ if(STRATA_ENABLE_CUDA) target_link_libraries(verify_parity PRIVATE strata_kernels strata_prefill CUDA::cudart) add_test(NAME verify_parity COMMAND verify_parity --selftest) + # The QSA block scores and top-k against the kernels they replaced, kept in the test: bitwise (synthetic data). + add_executable(qsa_select_parity src/kernels/qsa_select_parity.cu) + target_link_libraries(qsa_select_parity PRIVATE strata_kernels CUDA::cudart) + add_test(NAME qsa_select_parity COMMAND qsa_select_parity) + # Plan v0.3 P3: native MMVQ for 1..8 tokens (bitwise selftest + --bench timing); synthetic data, no model. if(STRATA_BUILD_TESTS) add_executable(native_mmvq_multi bench/micro/native_mmvq_multi.cpp) diff --git a/include/strata/kernels/qsa_select.hpp b/include/strata/kernels/qsa_select.hpp index 51708359779..231324febc9 100644 --- a/include/strata/kernels/qsa_select.hpp +++ b/include/strata/kernels/qsa_select.hpp @@ -5,12 +5,13 @@ // single-block kernel that makes 32 bit-serial passes over every CELL (topk_kernel): ~0.1-0.2 ms per query at 32K, // once per QSA layer and token - 1-2 ms of a decode token and most of a 32K prompt's time. Here: // -// qsa_block_scores : one warp per (query, block), FP32; relu per indexer head, summed; block n_bid (the -// incomplete tail) scores the `dead` key and gets +1e9 when it has cells - which is exactly -// what the per-token path reads there, because pooled[n_bid] holds `dead` at that time (in -// a chunk it may already hold a block completed later, so it is not read). -// qsa_block_topk : one block per query; a 4-pass radix select over the query's n_bid + 1 blocks, each -// weighted by its cell count, then the cells emitted in ascending order with ties to the +// qsa_block_scores : FP32; relu per indexer head, summed; block n_bid (the incomplete tail) scores the `dead` key +// and gets +1e9 when it has cells - which is exactly what the per-token path reads there, +// because pooled[n_bid] holds `dead` at that time (in a chunk it may already hold a block +// completed later, so it is not read). A 4-lane group per key block reads the key once for +// all queries and sums each score in the order one warp per (query, block) did. +// qsa_block_topk : one 1024-thread block per query; a 4-pass radix select over the query's n_bid + 1 blocks, +// each weighted by its cell count, then the cells emitted in ascending order with ties to the // lowest index - the same selection as topk_kernel, over a quarter of the elements. // // Queries carry their own step record (pos, n_kv, n_bid, width) as everywhere else in QSA. @@ -23,13 +24,14 @@ namespace strata::kernels { /// scores [nq, max_blocks]; q_idx [nq, idx_n_head, idx_dim] (normed and rotated); steps [nq, kStepCount]. -/// The grid holds a warp for each of the first `grid_blocks` blocks of every query and its warps go round for the -/// rest: a captured graph passes a fixed size (qsa_score_grid_blocks), the prompt path the chunk's largest n_bid + 1. +/// The grid covers the first `grid_blocks` blocks and goes round for the rest: a captured graph passes a fixed size +/// (qsa_score_grid_blocks), the prompt path the batch's largest n_bid + 1. void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx, const int32_t* steps, int64_t nq, int64_t max_blocks, const QsaShapes& s, float* scores, void* stream, int64_t grid_blocks); -/// The block-score grid of the decode paths, whatever the context: 1024 warps per query. -constexpr int64_t qsa_score_grid_blocks = 1024; +/// The block-score grid of the decode paths, whatever the context: 164 thread blocks of 64 key blocks, one wave on +/// the RTX 3090 (two per SM); longer contexts go round. +constexpr int64_t qsa_score_grid_blocks = 10496; /// ids [nq, cap] (cells, ascending); `cap` >= the largest selection width. void qsa_block_topk(const float* scores, const int32_t* steps, int64_t nq, int64_t max_blocks, int64_t cap, diff --git a/src/kernels/cuda/qsa_select.cu b/src/kernels/cuda/qsa_select.cu index 269e348a3f1..1a4864f3b1c 100644 --- a/src/kernels/cuda/qsa_select.cu +++ b/src/kernels/cuda/qsa_select.cu @@ -11,8 +11,14 @@ namespace strata::kernels { namespace { constexpr int IDX_DIM = 128, IDX_HEADS = 4, R = 4; +constexpr unsigned FULL = 0xffffffffu; constexpr int SCORE_WARPS = 8; -constexpr int TOPK_T = 256; +constexpr int SCORE_KB = 2; // key blocks per 8-lane group +constexpr int SCORE_TILE = SCORE_WARPS * 4 * SCORE_KB; // key blocks per thread block and round +constexpr int SCORE_QC = 16; // queries staged in shared memory at a time (32 KB) +constexpr int TOPK_T = 1024, TOPK_W = TOPK_T / 32; +constexpr int TOPK_ROWS = 32; // rows of TOPK_T blocks held in registers +constexpr int TOPK_MAX_ROWS = 64; // 65536 blocks: 262144 cells __device__ __forceinline__ uint32_t order_key(float s) { const float v = s + 0.0f; @@ -21,136 +27,302 @@ __device__ __forceinline__ uint32_t order_key(float s) { return (b & 0x80000000u) ? ~b : (b | 0x80000000u); } +// One query's scores for the group's SCORE_KB key blocks. Lane j of the 8-lane group holds each key's dims 4l..4l+3 +// for l = j, j+8, j+16, j+24; per head its four 4-term dot products d_l are summed in the lane as +// (d_j + d_j+16) + (d_j+8 + d_j+24), then across the group by xor 4, 2 and 1: the tree in which a warp-wide xor +// butterfly (16, 8, 4, 2, 1) over lanes l sums the same d_l, so each score is bit for bit that of one warp per +// (query, block). TAIL: key `tail_kb` (-1: none) is the query's tail block and takes the `dead` key instead. +template +__device__ __forceinline__ void group_scores(const float4 (&k)[SCORE_KB][4], const float* __restrict__ dead, + int tail_kb, const float* q, int j, float (&score)[SCORE_KB]) { +#pragma unroll + for (int kb = 0; kb < SCORE_KB; ++kb) score[kb] = 0.0f; +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) { + float4 q4[4]; +#pragma unroll + for (int i = 0; i < 4; ++i) q4[i] = *reinterpret_cast(q + h * IDX_DIM + 4 * (j + 8 * i)); +#pragma unroll + for (int kb = 0; kb < SCORE_KB; ++kb) { + float d[4]; +#pragma unroll + for (int i = 0; i < 4; ++i) { + float4 k4 = k[kb][i]; + if (TAIL && kb == tail_kb) k4 = *reinterpret_cast(dead + 4 * (j + 8 * i)); + d[i] = k4.x * q4[i].x + k4.y * q4[i].y + k4.z * q4[i].z + k4.w * q4[i].w; + } + float c = (d[0] + d[2]) + (d[1] + d[3]); + c += __shfl_xor_sync(FULL, c, 4); + c += __shfl_xor_sync(FULL, c, 2); + c += __shfl_xor_sync(FULL, c, 1); + score[kb] += c > 0.0f ? c : 0.0f; + } + } +} + +// 8-lane groups of SCORE_KB consecutive key blocks each, read once for all the queries, which go through shared +// memory SCORE_QC at a time. Block n_bid of a query is its incomplete tail: the `dead` key, +1e9 when it has cells. __global__ void __launch_bounds__(SCORE_WARPS * 32) block_scores_kernel(const float* __restrict__ pooled, const float* __restrict__ dead, const float* __restrict__ q_idx, - const int32_t* __restrict__ steps, + const int32_t* __restrict__ steps, int nq, int64_t max_blocks, float* __restrict__ out) { - const int64_t qi = blockIdx.y; - const int32_t* st = steps + qi * kStepCount; - const int64_t n_kv = st[kStepNKv], n_bid = st[kStepNBid]; - const int64_t last = n_bid < max_blocks - 1 ? n_bid : max_blocks - 1; - const int lane = threadIdx.x & 31; - const float* q = q_idx + qi * IDX_HEADS * IDX_DIM + lane * 4; - float4 q4[IDX_HEADS]; -#pragma unroll - for (int h = 0; h < IDX_HEADS; ++h) q4[h] = *reinterpret_cast(q + h * IDX_DIM); - // a warp per block; the grid covers the first blocks, and warps go round for the rest - for (int64_t b = (int64_t) blockIdx.x * SCORE_WARPS + (threadIdx.x >> 5); b <= last; - b += (int64_t) gridDim.x * SCORE_WARPS) { - const float* key = (b == n_bid) ? dead : pooled + b * IDX_DIM; - const float4 k4 = *reinterpret_cast(key + lane * 4); - float score = 0.0f; -#pragma unroll - for (int h = 0; h < IDX_HEADS; ++h) { - float d = k4.x * q4[h].x + k4.y * q4[h].y + k4.z * q4[h].z + k4.w * q4[h].w; -#pragma unroll - for (int o = 16; o > 0; o >>= 1) d += __shfl_xor_sync(0xffffffffu, d, o); - score += d > 0.0f ? d : 0.0f; + __shared__ __align__(16) float qs[SCORE_QC * IDX_HEADS * IDX_DIM]; + __shared__ int s_nkv[SCORE_QC], s_nbid[SCORE_QC]; + const int t = threadIdx.x, lane = t & 31, j = lane & 7; + const int slot = ((t >> 5) * 4 + (lane >> 3)) * SCORE_KB; // the group's first block within a tile + int64_t top = -1; // the queries' last block: the rounds end there + for (int qi = 0; qi < nq; ++qi) { + const int64_t n_bid = steps[(int64_t) qi * kStepCount + kStepNBid]; + const int64_t last = n_bid < max_blocks - 1 ? n_bid : max_blocks - 1; + top = last > top ? last : top; + } + for (int64_t b0 = (int64_t) blockIdx.x * SCORE_TILE; b0 <= top; b0 += (int64_t) gridDim.x * SCORE_TILE) { + const int64_t bg = b0 + slot; + float4 k[SCORE_KB][4]; +#pragma unroll + for (int kb = 0; kb < SCORE_KB; ++kb) { + const float* key = pooled + (bg + kb <= top ? bg + kb : 0) * IDX_DIM; +#pragma unroll + for (int i = 0; i < 4; ++i) k[kb][i] = __ldg(reinterpret_cast(key + 4 * (j + 8 * i))); } - if (lane == 0) { - if (b == n_bid && n_kv % R != 0) score += 1e9f; - out[qi * max_blocks + b] = score; + for (int q0 = 0; q0 < nq; q0 += SCORE_QC) { + const int qn = nq - q0 < SCORE_QC ? nq - q0 : SCORE_QC; + __syncthreads(); + const float4* src = reinterpret_cast(q_idx + (int64_t) q0 * IDX_HEADS * IDX_DIM); + for (int i = t; i < qn * IDX_HEADS * IDX_DIM / 4; i += SCORE_WARPS * 32) + reinterpret_cast(qs)[i] = src[i]; + if (t < qn) { + s_nkv[t] = steps[(int64_t) (q0 + t) * kStepCount + kStepNKv]; + s_nbid[t] = steps[(int64_t) (q0 + t) * kStepCount + kStepNBid]; + } + __syncthreads(); + for (int qq = 0; qq < qn; ++qq) { + const int64_t n_kv = s_nkv[qq], n_bid = s_nbid[qq]; + const int64_t last = n_bid < max_blocks - 1 ? n_bid : max_blocks - 1; + const int tail_kb = n_bid >= bg && n_bid < bg + SCORE_KB ? (int) (n_bid - bg) : -1; + const float* q = qs + qq * IDX_HEADS * IDX_DIM; + float score[SCORE_KB]; + // warp-uniform branches: every lane of the warp runs the group shuffles + if (__any_sync(FULL, tail_kb >= 0)) group_scores(k, dead, tail_kb, q, j, score); + else group_scores(k, dead, -1, q, j, score); + if (j == 0) { + float* o = out + (int64_t) (q0 + qq) * max_blocks; +#pragma unroll + for (int kb = 0; kb < SCORE_KB; ++kb) { + if (bg + kb > last) break; + o[bg + kb] = kb == tail_kb && n_kv % R != 0 ? score[kb] + 1e9f : score[kb]; + } + } + } + } + } +} + +__device__ __forceinline__ int warp_sum(int v) { +#pragma unroll + for (int o = 16; o > 0; o >>= 1) v += __shfl_xor_sync(FULL, v, o); + return v; +} + +// The exclusive prefixes of a and b over the block's threads in thread order, and their totals. +__device__ __forceinline__ void block_exclusive2(int a, int b, int& ea, int& eb, int& ta, int& tb, int* s_w) { + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + int ia = a, ib = b; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) { + const int ya = __shfl_up_sync(FULL, ia, o), yb = __shfl_up_sync(FULL, ib, o); + if (lane >= o) { ia += ya; ib += yb; } + } + if (lane == 31) { s_w[warp] = ia; s_w[TOPK_W + warp] = ib; } + __syncthreads(); + if (warp == 0) { + const int wa = s_w[lane], wb = s_w[TOPK_W + lane]; + int xa = wa, xb = wb; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) { + const int ya = __shfl_up_sync(FULL, xa, o), yb = __shfl_up_sync(FULL, xb, o); + if (lane >= o) { xa += ya; xb += yb; } } + s_w[lane] = xa - wa; + s_w[TOPK_W + lane] = xb - wb; + if (lane == 31) { s_w[2 * TOPK_W] = xa; s_w[2 * TOPK_W + 1] = xb; } } + __syncthreads(); + ea = s_w[warp] + ia - a; + eb = s_w[TOPK_W + warp] + ib - b; + ta = s_w[2 * TOPK_W]; + tb = s_w[2 * TOPK_W + 1]; + __syncthreads(); } +// The same selection as the per-cell topk_kernel: the `width` cells of the largest block scores (a block's score +// counts for each of its cells), ties to the lowest cell, emitted ascending. A radix select, 8 bits at a time, over +// the order keys, with a histogram per warp. The complete blocks go in rows of TOPK_T: thread t has block +// row * TOPK_T + t, the first TOPK_ROWS rows in registers (read once, 131K cells), the others from memory in each +// pass. A block's cells then land at (cells above the threshold before it) + min(cells at it before it, budget): +// ballots within a (row, warp), one scan over the (row, warp) totals. The tail block (n_bid) is thread 0's. __global__ void __launch_bounds__(TOPK_T) block_topk_kernel(const float* __restrict__ scores, const int32_t* __restrict__ steps, int64_t max_blocks, int64_t cap, int32_t* __restrict__ ids) { + __shared__ int sm[TOPK_W * 256]; // the per-warp histograms; then the scan table and the output __shared__ int hist[256]; - __shared__ int s_a[TOPK_T], s_b[TOPK_T]; + __shared__ int s_w[2 * TOPK_W + 2]; __shared__ int s_digit, s_above; const int64_t qi = blockIdx.x; const int32_t* st = steps + qi * kStepCount; const int64_t n_kv = st[kStepNKv], n_bid = st[kStepNBid], width = st[kStepWidth]; int32_t* out = ids + qi * cap; - const int t = threadIdx.x; + const int t = threadIdx.x, lane = t & 31, warp = t >> 5; if (n_kv <= width) { // everything is selected: the identity, ascending for (int64_t j = t; j < n_kv; j += TOPK_T) out[j] = (int32_t) j; return; } const float* sc = scores + qi * max_blocks; - const int64_t nb = n_bid + 1; // blocks 0..n_bid, the last possibly empty - const int64_t per = (nb + TOPK_T - 1) / TOPK_T; - const int64_t b0 = (int64_t) t * per, b1 = (b0 + per < nb) ? b0 + per : nb; - auto weight = [&](int64_t b) -> int { return b < n_bid ? R : (int) (n_kv - n_bid * R); }; - // ---- radix select: the largest key thr with (cells with key >= thr) >= width, 8 bits at a time + const int tail_w = (int) (n_kv - n_bid * R); // the cells of block n_bid, 0..3 + const int nrows = (int) ((n_bid + TOPK_T - 1) / TOPK_T); + uint32_t key[TOPK_ROWS]; +#pragma unroll + for (int i = 0; i < TOPK_ROWS; ++i) { + const int64_t b = (int64_t) i * TOPK_T + t; + key[i] = b < n_bid ? order_key(sc[b]) : 0u; + } + // ---- radix select: the largest key thr with (cells with key >= thr) >= width uint32_t prefix = 0; int above = 0; // cells strictly above the digits fixed so far + int* whist = sm + warp * 256; for (int shift = 24; shift >= 0; shift -= 8) { - for (int i = t; i < 256; i += TOPK_T) hist[i] = 0; - __syncthreads(); + for (int i = lane; i < 256; i += 32) whist[i] = 0; + __syncwarp(); const uint32_t hi_mask = shift == 24 ? 0u : (0xffffffffu << (shift + 8)); - for (int64_t b = b0; b < b1; ++b) { - const int w = weight(b); - if (w == 0) continue; - const uint32_t k = order_key(sc[b]); - if ((k & hi_mask) == (prefix & hi_mask)) atomicAdd(&hist[(k >> shift) & 255], w); + int cd = -1, cc = 0; // a run of one digit is added at once + auto count = [&](uint32_t k) { + if ((k & hi_mask) != (prefix & hi_mask)) return; + const int d = (int) ((k >> shift) & 255); + if (d != cd) { + if (cc != 0) atomicAdd(&whist[cd], cc); + cd = d; + cc = 0; + } + cc += R; + }; +#pragma unroll + for (int i = 0; i < TOPK_ROWS; ++i) + if ((int64_t) i * TOPK_T + t < n_bid) count(key[i]); + for (int i = TOPK_ROWS; i < nrows; ++i) { + const int64_t b = (int64_t) i * TOPK_T + t; + if (b < n_bid) count(order_key(sc[b])); + } + if (cc != 0) atomicAdd(&whist[cd], cc); + if (t == 0 && tail_w > 0) { + const uint32_t k = order_key(sc[n_bid]); + if ((k & hi_mask) == (prefix & hi_mask)) atomicAdd(&whist[(k >> shift) & 255], tail_w); + } + __syncthreads(); + if (t < 256) { + int s = 0; +#pragma unroll 8 + for (int w = 0; w < TOPK_W; ++w) s += sm[w * 256 + t]; + hist[t] = s; } __syncthreads(); - if (t == 0) { - int cum = above, d = 255; - for (; d > 0; --d) { - if (cum + hist[d] >= width) break; - cum += hist[d]; + if (warp == 0) { + // the digit: the largest d >= 1 with above + (cells with a digit >= d) >= width, else 0; lane l holds the + // bins 255 - 8l down to 248 - 8l + int v[8], s = 0; +#pragma unroll + for (int i = 0; i < 8; ++i) { v[i] = hist[255 - (lane * 8 + i)]; s += v[i]; } + int incl = s; +#pragma unroll + for (int o = 1; o < 32; o <<= 1) { + const int y = __shfl_up_sync(FULL, incl, o); + if (lane >= o) incl += y; + } + int run = incl - s, below = 0; +#pragma unroll + for (int i = 0; i < 8; ++i) { + run += v[i]; + if (255 - (lane * 8 + i) >= 1 && above + run < width) ++below; + } + const int digit = 255 - warp_sum(below); + int gt = 0; +#pragma unroll + for (int i = 0; i < 8; ++i) + if (255 - (lane * 8 + i) > digit) gt += v[i]; + gt = warp_sum(gt); + if (lane == 0) { + s_digit = digit; + s_above = above + gt; } - s_digit = d; - s_above = cum; } __syncthreads(); prefix |= (uint32_t) s_digit << shift; above = s_above; - __syncthreads(); } const uint32_t thr = prefix; - const int64_t eq_budget = width - above; // cells equal to thr that fit, lowest index first - // ---- per-thread counts of cells above and at the threshold, then their exclusive prefixes - int gt = 0, eq = 0; - for (int64_t b = b0; b < b1; ++b) { - const int w = weight(b); - if (w == 0) continue; - const uint32_t k = order_key(sc[b]); - if (k > thr) gt += w; - else if (k == thr) eq += w; - } - s_a[t] = gt; - s_b[t] = eq; - __syncthreads(); - if (t == 0) { - int ag = 0, ae = 0; - for (int i = 0; i < TOPK_T; ++i) { - const int g = s_a[i], e = s_b[i]; - s_a[i] = ag; s_b[i] = ae; - ag += g; ae += e; - } + const int budget = (int) (width - above); // cells equal to thr that fit, lowest index first + // ---- the cells above and at thr per (row, warp), their exclusive prefixes in block order + int* tab_gt = sm; + int* tab_eq = sm + TOPK_MAX_ROWS * 32; + int* obuf = sm + 2 * TOPK_MAX_ROWS * 32; // the selection, copied out coalesced + __syncthreads(); // the histograms are no longer read + auto row_key = [&](int i) { return i < TOPK_ROWS ? 0u : order_key(sc[(int64_t) i * TOPK_T + t]); }; +#pragma unroll + for (int i = 0; i < TOPK_ROWS; ++i) { + if (i >= nrows) break; + const bool in = (int64_t) i * TOPK_T + t < n_bid; + const unsigned mg = __ballot_sync(FULL, in && key[i] > thr), me = __ballot_sync(FULL, in && key[i] == thr); + if (lane == 0) { tab_gt[i * 32 + warp] = R * __popc(mg); tab_eq[i * 32 + warp] = R * __popc(me); } + } + for (int i = TOPK_ROWS; i < nrows; ++i) { + const bool in = (int64_t) i * TOPK_T + t < n_bid; + const uint32_t k = in ? row_key(i) : 0u; + const unsigned mg = __ballot_sync(FULL, in && k > thr), me = __ballot_sync(FULL, in && k == thr); + if (lane == 0) { tab_gt[i * 32 + warp] = R * __popc(mg); tab_eq[i * 32 + warp] = R * __popc(me); } } __syncthreads(); - const int64_t eq_before = s_b[t]; - int64_t my_eq = eq_budget - eq_before; - if (my_eq < 0) my_eq = 0; - if (my_eq > eq) my_eq = eq; - const int sel = gt + (int) my_eq; - __syncthreads(); - s_a[t] = sel; - __syncthreads(); - if (t == 0) { - int a = 0; - for (int i = 0; i < TOPK_T; ++i) { const int c = s_a[i]; s_a[i] = a; a += c; } + int gt_all, eq_all; + { + const int e = nrows * 32, k0 = 2 * t; // entries (row, warp) in block order, two per thread + const int g0 = k0 < e ? tab_gt[k0] : 0, e0 = k0 < e ? tab_eq[k0] : 0; + const int g1 = k0 + 1 < e ? tab_gt[k0 + 1] : 0, e1 = k0 + 1 < e ? tab_eq[k0 + 1] : 0; + int gx, ex; + block_exclusive2(g0 + g1, e0 + e1, gx, ex, gt_all, eq_all, s_w); + if (k0 < e) { tab_gt[k0] = gx; tab_eq[k0] = ex; } + if (k0 + 1 < e) { tab_gt[k0 + 1] = gx + g0; tab_eq[k0 + 1] = ex + e0; } } __syncthreads(); - int64_t wpos = s_a[t]; - int64_t eq_left = my_eq; - for (int64_t b = b0; b < b1; ++b) { - const int w = weight(b); - if (w == 0) continue; - const uint32_t k = order_key(sc[b]); - if (k > thr) { - for (int c = 0; c < w; ++c) out[wpos++] = (int32_t) (b * R + c); - } else if (k == thr) { - for (int c = 0; c < w && eq_left > 0; ++c, --eq_left) out[wpos++] = (int32_t) (b * R + c); - } + // ---- the cells, into shared memory + const unsigned lt = (1u << lane) - 1u; + auto emit = [&](int i, uint32_t k, bool in) { + const bool g = in && k > thr, q = in && k == thr; + const unsigned mg = __ballot_sync(FULL, g), me = __ballot_sync(FULL, q); + if (!g && !q) return; + const int gb = tab_gt[i * 32 + warp] + R * __popc(mg & lt); + const int eb = tab_eq[i * 32 + warp] + R * __popc(me & lt); + const int pos = gb + (eb < budget ? eb : budget); + const int take = g ? R : (budget - eb < 0 ? 0 : (budget - eb > R ? R : budget - eb)); + const int64_t b = (int64_t) i * TOPK_T + t; + for (int c = 0; c < take; ++c) obuf[pos + c] = (int32_t) (b * R + c); + }; +#pragma unroll + for (int i = 0; i < TOPK_ROWS; ++i) { + if (i >= nrows) break; + emit(i, key[i], (int64_t) i * TOPK_T + t < n_bid); } + for (int i = TOPK_ROWS; i < nrows; ++i) { + const bool in = (int64_t) i * TOPK_T + t < n_bid; + emit(i, in ? row_key(i) : 0u, in); + } + if (t == 0 && tail_w > 0) { // the tail block comes last + const uint32_t k = order_key(sc[n_bid]); + const int pos = gt_all + (eq_all < budget ? eq_all : budget); + const int take = k > thr ? tail_w : (k == thr ? (budget - eq_all < 0 ? 0 : (budget - eq_all > tail_w ? tail_w + : budget - eq_all)) : 0); + for (int c = 0; c < take; ++c) obuf[pos + c] = (int32_t) (n_bid * R + c); + } + __syncthreads(); + for (int64_t j = t; j < width; j += TOPK_T) out[j] = obuf[j]; } } // namespace @@ -163,9 +335,9 @@ void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx std::exit(1); } const int64_t covered = grid_blocks < max_blocks ? grid_blocks : max_blocks; - const dim3 grid((unsigned) ((covered + SCORE_WARPS - 1) / SCORE_WARPS), (unsigned) nq); - block_scores_kernel<<>>(pooled, dead, q_idx, steps, max_blocks, - scores); + const unsigned grid = (unsigned) ((covered + SCORE_TILE - 1) / SCORE_TILE); + block_scores_kernel<<>>(pooled, dead, q_idx, steps, (int) nq, + max_blocks, scores); const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { std::fprintf(stderr, "qsa_block_scores: %s\n", cudaGetErrorString(e)); std::exit(1); } } @@ -173,7 +345,7 @@ void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx void qsa_block_topk(const float* scores, const int32_t* steps, int64_t nq, int64_t max_blocks, int64_t cap, const QsaShapes& s, int32_t* ids, void* stream) { if (nq <= 0) return; - if (s.idx_block != R || cap < qsa_selection_width(kTopkMaxCells, s)) { + if (s.idx_block != R || cap < qsa_selection_width(kTopkMaxCells, s) || max_blocks > (int64_t) TOPK_MAX_ROWS * TOPK_T) { std::fprintf(stderr, "qsa_block_topk: unsupported geometry or cap\n"); std::exit(1); } diff --git a/src/kernels/qsa_select_parity.cu b/src/kernels/qsa_select_parity.cu new file mode 100644 index 00000000000..358ef1ea898 --- /dev/null +++ b/src/kernels/qsa_select_parity.cu @@ -0,0 +1,382 @@ +// src/kernels/qsa_select_parity.cu - the QSA block scores and top-k (qsa_select.cu) against the kernels they +// replaced, kept here as the reference: scores bitwise, selections identical, for decode windows (1-8 queries, +// the captured graph's fixed grid) and the prompt path's batches (256 queries), from the identity case to 200K +// cells, with exact score ties, zero scores and every tail length. --bench times both. +#include "strata/kernels/qsa.hpp" +#include "strata/kernels/qsa_select.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ref { + +using namespace strata::kernels; +constexpr int IDX_DIM = 128, IDX_HEADS = 4, R = 4; +constexpr int SCORE_WARPS = 8; +constexpr int TOPK_T = 256; + +__device__ __forceinline__ uint32_t order_key(float s) { + const float v = s + 0.0f; + if (!(v == v)) return 0u; + const uint32_t b = __float_as_uint(v); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} + +__global__ void __launch_bounds__(SCORE_WARPS * 32) block_scores_kernel(const float* __restrict__ pooled, + const float* __restrict__ dead, + const float* __restrict__ q_idx, + const int32_t* __restrict__ steps, + int64_t max_blocks, float* __restrict__ out) { + const int64_t qi = blockIdx.y; + const int32_t* st = steps + qi * kStepCount; + const int64_t n_kv = st[kStepNKv], n_bid = st[kStepNBid]; + const int64_t last = n_bid < max_blocks - 1 ? n_bid : max_blocks - 1; + const int lane = threadIdx.x & 31; + const float* q = q_idx + qi * IDX_HEADS * IDX_DIM + lane * 4; + float4 q4[IDX_HEADS]; +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) q4[h] = *reinterpret_cast(q + h * IDX_DIM); + for (int64_t b = (int64_t) blockIdx.x * SCORE_WARPS + (threadIdx.x >> 5); b <= last; + b += (int64_t) gridDim.x * SCORE_WARPS) { + const float* key = (b == n_bid) ? dead : pooled + b * IDX_DIM; + const float4 k4 = *reinterpret_cast(key + lane * 4); + float score = 0.0f; +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) { + float d = k4.x * q4[h].x + k4.y * q4[h].y + k4.z * q4[h].z + k4.w * q4[h].w; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) d += __shfl_xor_sync(0xffffffffu, d, o); + score += d > 0.0f ? d : 0.0f; + } + if (lane == 0) { + if (b == n_bid && n_kv % R != 0) score += 1e9f; + out[qi * max_blocks + b] = score; + } + } +} + +__global__ void __launch_bounds__(TOPK_T) block_topk_kernel(const float* __restrict__ scores, + const int32_t* __restrict__ steps, int64_t max_blocks, + int64_t cap, int32_t* __restrict__ ids) { + __shared__ int hist[256]; + __shared__ int s_a[TOPK_T], s_b[TOPK_T]; + __shared__ int s_digit, s_above; + const int64_t qi = blockIdx.x; + const int32_t* st = steps + qi * kStepCount; + const int64_t n_kv = st[kStepNKv], n_bid = st[kStepNBid], width = st[kStepWidth]; + int32_t* out = ids + qi * cap; + const int t = threadIdx.x; + if (n_kv <= width) { + for (int64_t j = t; j < n_kv; j += TOPK_T) out[j] = (int32_t) j; + return; + } + const float* sc = scores + qi * max_blocks; + const int64_t nb = n_bid + 1; + const int64_t per = (nb + TOPK_T - 1) / TOPK_T; + const int64_t b0 = (int64_t) t * per, b1 = (b0 + per < nb) ? b0 + per : nb; + auto weight = [&](int64_t b) -> int { return b < n_bid ? R : (int) (n_kv - n_bid * R); }; + uint32_t prefix = 0; + int above = 0; + for (int shift = 24; shift >= 0; shift -= 8) { + for (int i = t; i < 256; i += TOPK_T) hist[i] = 0; + __syncthreads(); + const uint32_t hi_mask = shift == 24 ? 0u : (0xffffffffu << (shift + 8)); + for (int64_t b = b0; b < b1; ++b) { + const int w = weight(b); + if (w == 0) continue; + const uint32_t k = order_key(sc[b]); + if ((k & hi_mask) == (prefix & hi_mask)) atomicAdd(&hist[(k >> shift) & 255], w); + } + __syncthreads(); + if (t == 0) { + int cum = above, d = 255; + for (; d > 0; --d) { + if (cum + hist[d] >= width) break; + cum += hist[d]; + } + s_digit = d; + s_above = cum; + } + __syncthreads(); + prefix |= (uint32_t) s_digit << shift; + above = s_above; + __syncthreads(); + } + const uint32_t thr = prefix; + const int64_t eq_budget = width - above; + int gt = 0, eq = 0; + for (int64_t b = b0; b < b1; ++b) { + const int w = weight(b); + if (w == 0) continue; + const uint32_t k = order_key(sc[b]); + if (k > thr) gt += w; + else if (k == thr) eq += w; + } + s_a[t] = gt; + s_b[t] = eq; + __syncthreads(); + if (t == 0) { + int ag = 0, ae = 0; + for (int i = 0; i < TOPK_T; ++i) { + const int g = s_a[i], e = s_b[i]; + s_a[i] = ag; s_b[i] = ae; + ag += g; ae += e; + } + } + __syncthreads(); + const int64_t eq_before = s_b[t]; + int64_t my_eq = eq_budget - eq_before; + if (my_eq < 0) my_eq = 0; + if (my_eq > eq) my_eq = eq; + const int sel = gt + (int) my_eq; + __syncthreads(); + s_a[t] = sel; + __syncthreads(); + if (t == 0) { + int a = 0; + for (int i = 0; i < TOPK_T; ++i) { const int c = s_a[i]; s_a[i] = a; a += c; } + } + __syncthreads(); + int64_t wpos = s_a[t]; + int64_t eq_left = my_eq; + for (int64_t b = b0; b < b1; ++b) { + const int w = weight(b); + if (w == 0) continue; + const uint32_t k = order_key(sc[b]); + if (k > thr) { + for (int c = 0; c < w; ++c) out[wpos++] = (int32_t) (b * R + c); + } else if (k == thr) { + for (int c = 0; c < w && eq_left > 0; ++c, --eq_left) out[wpos++] = (int32_t) (b * R + c); + } + } +} + +void scores(const float* pooled, const float* dead, const float* q_idx, const int32_t* steps, int64_t nq, + int64_t max_blocks, float* out, cudaStream_t s, int64_t grid_blocks) { + const int64_t covered = grid_blocks < max_blocks ? grid_blocks : max_blocks; + const dim3 grid((unsigned) ((covered + SCORE_WARPS - 1) / SCORE_WARPS), (unsigned) nq); + block_scores_kernel<<>>(pooled, dead, q_idx, steps, max_blocks, out); +} + +void topk(const float* scores, const int32_t* steps, int64_t nq, int64_t max_blocks, int64_t cap, int32_t* ids, + cudaStream_t s) { + block_topk_kernel<<<(unsigned) nq, TOPK_T, 0, s>>>(scores, steps, max_blocks, cap, ids); +} + +} // namespace ref + +namespace { + +using namespace strata::kernels; + +void check(cudaError_t e, const char* what) { + if (e != cudaSuccess) { + std::fprintf(stderr, "%s: %s\n", what, cudaGetErrorString(e)); + std::exit(1); + } +} + +template T* dev(size_t n) { + T* p = nullptr; + check(cudaMalloc(&p, n * sizeof(T)), "cudaMalloc"); + return p; +} + +constexpr int64_t kMaxCells = 200192; // the configs' --max-context +constexpr int QD = 512; // a query's floats: 4 heads x 128 + +struct Data { + QsaShapes s = qsa_real_shapes(); + int64_t max_blocks = kMaxCells / 4 + 2; + int64_t cap = qsa_selection_width(kTopkMaxCells, qsa_real_shapes()); + float *pooled = nullptr, *dead = nullptr, *q = nullptr, *sc_ref = nullptr, *sc_new = nullptr; + int32_t *steps = nullptr, *ids_ref = nullptr, *ids_new = nullptr; + int64_t max_q = 256; +}; + +// Keys like the indexer's: most N(0,1); runs of copies (exact score ties, so the threshold often falls inside a +// run of equal keys and blocks are taken in part), all-zero keys (scores exactly 0) and -0.0 components. +void fill(Data& d, std::mt19937& rng) { + std::normal_distribution n01(0.0f, 1.0f); + std::vector pooled((size_t) d.max_blocks * 128); + for (int64_t b = 0; b < d.max_blocks; ++b) { + float* row = pooled.data() + (size_t) b * 128; + const int kind = (int) (rng() % 64); + if (b > 0 && kind < 6) std::memcpy(row, row - 128, 128 * sizeof(float)); // a copy of the previous block + else if (kind == 6) std::fill(row, row + 128, 0.0f); + else + for (int i = 0; i < 128; ++i) row[i] = (rng() % 97 == 0) ? -0.0f : n01(rng); + } + std::vector dead(128); + for (auto& x : dead) x = n01(rng); + std::vector q((size_t) d.max_q * QD); + for (auto& x : q) x = 0.3f * n01(rng); + d.pooled = dev(pooled.size()); + d.dead = dev(128); + d.q = dev(q.size()); + check(cudaMemcpy(d.pooled, pooled.data(), pooled.size() * 4, cudaMemcpyHostToDevice), "up pooled"); + check(cudaMemcpy(d.dead, dead.data(), 128 * 4, cudaMemcpyHostToDevice), "up dead"); + check(cudaMemcpy(d.q, q.data(), q.size() * 4, cudaMemcpyHostToDevice), "up q"); + d.sc_ref = dev((size_t) d.max_q * d.max_blocks); + d.sc_new = dev((size_t) d.max_q * d.max_blocks); + d.steps = dev((size_t) d.max_q * kStepCount); + d.ids_ref = dev((size_t) d.max_q * d.cap); + d.ids_new = dev((size_t) d.max_q * d.cap); +} + +std::vector steps_from(const Data& d, int64_t pos0, int64_t nq) { + std::vector st((size_t) nq * kStepCount); + for (int64_t i = 0; i < nq; ++i) qsa_step_fill(st.data() + i * kStepCount, pos0 + i, d.s); + check(cudaMemcpy(d.steps, st.data(), st.size() * 4, cudaMemcpyHostToDevice), "up steps"); + return st; +} + +// both implementations on queries at positions pos0.., then scores bitwise and selections equal +int compare(Data& d, int64_t pos0, int64_t nq, int64_t grid_blocks, cudaStream_t s, const char* what) { + const std::vector st = steps_from(d, pos0, nq); + check(cudaMemset(d.sc_ref, 0xff, (size_t) nq * d.max_blocks * 4), "clear"); + check(cudaMemset(d.sc_new, 0xff, (size_t) nq * d.max_blocks * 4), "clear"); + check(cudaMemset(d.ids_ref, 0xff, (size_t) nq * d.cap * 4), "clear"); + check(cudaMemset(d.ids_new, 0xff, (size_t) nq * d.cap * 4), "clear"); + ref::scores(d.pooled, d.dead, d.q, d.steps, nq, d.max_blocks, d.sc_ref, s, grid_blocks); + ref::topk(d.sc_ref, d.steps, nq, d.max_blocks, d.cap, d.ids_ref, s); + qsa_block_scores(d.pooled, d.dead, d.q, d.steps, nq, d.max_blocks, d.s, d.sc_new, s, grid_blocks); + qsa_block_topk(d.sc_new, d.steps, nq, d.max_blocks, d.cap, d.s, d.ids_new, s); + check(cudaStreamSynchronize(s), what); + std::vector a((size_t) nq * d.max_blocks), b(a.size()); + std::vector ia((size_t) nq * d.cap), ib(ia.size()); + check(cudaMemcpy(a.data(), d.sc_ref, a.size() * 4, cudaMemcpyDeviceToHost), "down"); + check(cudaMemcpy(b.data(), d.sc_new, b.size() * 4, cudaMemcpyDeviceToHost), "down"); + check(cudaMemcpy(ia.data(), d.ids_ref, ia.size() * 4, cudaMemcpyDeviceToHost), "down"); + check(cudaMemcpy(ib.data(), d.ids_new, ib.size() * 4, cudaMemcpyDeviceToHost), "down"); + int bad = 0; + for (int64_t i = 0; i < nq; ++i) { + const int64_t n_bid = st[(size_t) (i * kStepCount + kStepNBid)], width = st[(size_t) (i * kStepCount + kStepWidth)]; + const int64_t last = std::min(n_bid, d.max_blocks - 1); + for (int64_t k = 0; k <= last; ++k) { + const size_t o = (size_t) (i * d.max_blocks + k); + if (a[o] != b[o]) { + if (bad < 5) { + float fa, fb; + std::memcpy(&fa, &a[o], 4); + std::memcpy(&fb, &b[o], 4); + std::fprintf(stderr, "%s: pos %lld, block %lld: score %.9g (0x%08x) vs %.9g (0x%08x)\n", what, + (long long) (pos0 + i), (long long) k, fa, a[o], fb, b[o]); + } + ++bad; + } + } + for (int64_t k = 0; k < width; ++k) { + const size_t o = (size_t) (i * d.cap + k); + if (ia[o] != ib[o]) { + if (bad < 5) + std::fprintf(stderr, "%s: pos %lld, selection entry %lld: cell %d vs %d\n", what, + (long long) (pos0 + i), (long long) k, ia[o], ib[o]); + ++bad; + } + } + } + return bad; +} + +int selftest(Data& d, cudaStream_t s) { + int bad = 0, cases = 0; + // decode windows: the captured graph's fixed grid, 1-8 queries at consecutive positions + const int64_t pos_dec[] = {0, 3, 2046, 2047, 2050, 2051, 2052, 8190, 16381, 32766, 65533, 131070, 200180}; + for (int64_t p : pos_dec) + for (int64_t nq : {1, 2, 3, 4, 8}) { + if (p + nq > kMaxCells) continue; + bad += compare(d, p, nq, qsa_score_grid_blocks, s, "decode"); + ++cases; + } + // prompt batches: 256 queries, the grid covering the batch's last block + const int64_t pos_pre[] = {0, 1900, 16384, 65536 - 100, 131072 - 256, 199936 - 1}; + for (int64_t p : pos_pre) { + const int64_t nq = 256, last = p + nq - 1; + bad += compare(d, p, nq, (last + 1) / 4 + 1, s, "prompt"); + ++cases; + } + std::printf("qsa_select: %s (%d cases: scores bitwise, selections equal)\n", bad ? "MISMATCH" : "PASS", cases); + return bad; +} + +// GPU time per call: `reps` calls captured in a graph, as the engine runs them (a launch outside a graph costs more +// than these kernels on Windows) +template float time_ms(cudaStream_t s, int reps, const F& fn) { + cudaGraph_t g = nullptr; + cudaGraphExec_t ge = nullptr; + check(cudaStreamBeginCapture(s, cudaStreamCaptureModeThreadLocal), "capture"); + for (int i = 0; i < reps; ++i) fn(); + check(cudaStreamEndCapture(s, &g), "capture"); + check(cudaGraphInstantiate(&ge, g, 0), "instantiate"); + cudaEvent_t a, b; + cudaEventCreate(&a); + cudaEventCreate(&b); + check(cudaGraphLaunch(ge, s), "launch"); + cudaEventRecord(a, s); + for (int i = 0; i < 3; ++i) check(cudaGraphLaunch(ge, s), "launch"); + cudaEventRecord(b, s); + check(cudaEventSynchronize(b), "bench"); + float ms = 0; + cudaEventElapsedTime(&ms, a, b); + cudaEventDestroy(a); + cudaEventDestroy(b); + cudaGraphExecDestroy(ge); + cudaGraphDestroy(g); + return ms / (float) (3 * reps); +} + +void bench(Data& d, cudaStream_t s) { + std::printf("%-26s %12s %12s %12s %12s\n", "case", "ref scores", "new scores", "ref top-k", "new top-k"); + struct Case { const char* name; int64_t pos; int64_t nq; bool prompt; }; + const Case cs[] = {{"decode 4 q, 1K cells", 1024, 4, false}, {"decode 4 q, 2K cells", 2048, 4, false}, + {"decode 4 q, 16K cells", 16384, 4, false}, + {"decode 4 q, 64K cells", 65536, 4, false}, {"decode 4 q, 131K cells", 131072, 4, false}, + {"decode 1 q, 131K cells", 131072, 1, false}, {"prompt 256 q at 16K", 16384 - 256, 256, true}, + {"prompt 256 q at 64K", 65536 - 256, 256, true}, {"prompt 256 q at 131K", 131072 - 256, 256, true}}; + for (const Case& c : cs) { + steps_from(d, c.pos, c.nq); + const int64_t grid = c.prompt ? (c.pos + c.nq) / 4 + 1 : qsa_score_grid_blocks; + const int reps = c.prompt ? 5 : 50; + const float rs = time_ms(s, reps, [&] { + ref::scores(d.pooled, d.dead, d.q, d.steps, c.nq, d.max_blocks, d.sc_ref, s, grid); + }); + const float ns = time_ms(s, reps, [&] { + qsa_block_scores(d.pooled, d.dead, d.q, d.steps, c.nq, d.max_blocks, d.s, d.sc_new, s, grid); + }); + const float rt = time_ms(s, reps, [&] { ref::topk(d.sc_ref, d.steps, c.nq, d.max_blocks, d.cap, d.ids_ref, s); }); + const float nt = time_ms(s, reps, [&] { + qsa_block_topk(d.sc_new, d.steps, c.nq, d.max_blocks, d.cap, d.s, d.ids_new, s); + }); + std::printf("%-26s %9.1f us %9.1f us %9.1f us %9.1f us\n", c.name, rs * 1e3f, ns * 1e3f, rt * 1e3f, nt * 1e3f); + } +} + +} // namespace + +int main(int argc, char** argv) { + bool do_bench = false; + for (int i = 1; i < argc; ++i) { + if (std::string(argv[i]) == "--bench") do_bench = true; + else { + std::fprintf(stderr, "usage: qsa_select_parity [--bench]\n"); + return 2; + } + } + std::mt19937 rng(20260926); + cudaStream_t s = nullptr; + check(cudaStreamCreate(&s), "stream"); + Data d; + fill(d, rng); + const int bad = selftest(d, s); + if (do_bench) bench(d, s); + return bad ? 1 : 0; +} From ae3f3c61a700708aedd1386db794fff3fc265668 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 16:14:48 +0200 Subject: [PATCH 35/92] The draft layer's prompt K/V in the batched prompt path The MTP draft layer read a prompt 4 cells per captured graph with a host sync after each (0.74 s of a 16K-token prompt, 1.5 s from 32K on). The prompt path now computes them after every chunk in its own arithmetic, as for the model's layers: the input norms, fc_embedding and fc_hidden, the attention hyper-connection read, K (normed, rotated) and V as GEMMs over 2048-token sub-chunks in the chunk's buffers, from the window's first cell on (41 ms of GPU time for 16K tokens). embed_rows embeds token ids from device memory in one launch (the draft layer's gather, shared). With the previous commit: a 16K-token prompt 9.51 -> 8.90 s (IQ3_XXS) and 10.24 -> 9.39 s (UD-Q4_K_XL), 65K 38.7 -> 34.0 s. Drafts accepted on the fixed text after 16K: 0.703 -> 0.704 (IQ3_XXS), 0.725 -> 0.718 (UD-Q4_K_XL); the tokens after a 65K-token prompt are unchanged (--adapt-every 0). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/layer.hpp | 3 ++ include/strata/core/mtp.hpp | 29 +++++++++---- include/strata/prefill/prefill.hpp | 13 +++--- src/core/layer.cpp | 67 ++++++++++++++++++++++-------- src/core/mtp.cpp | 67 +++++++----------------------- src/prefill/prefill.cpp | 62 +++++++++++++++++++++++---- src/program/generate.cpp | 20 ++------- 7 files changed, 153 insertions(+), 108 deletions(-) diff --git a/include/strata/core/layer.hpp b/include/strata/core/layer.hpp index 031a076a869..dfb9d9dcbc0 100644 --- a/include/strata/core/layer.hpp +++ b/include/strata/core/layer.hpp @@ -434,6 +434,9 @@ uint64_t block_buffers_init(const ModelGeometry& g, void* base, BlockBuffers& b) /// from the kernel's would make the prompt embed differently from the tokens generated after it. bool embed_row(const WeightTable& tables, const ModelGeometry& g, int64_t token, float* out_dev, void* stream, std::string& err); +/// embed_row for `n` token ids in DEVICE memory (checked by the caller) into out_dev [n, n_embd], one launch. +bool embed_rows(const WeightTable& tables, const ModelGeometry& g, const int32_t* tokens_dev, int64_t n, float* out_dev, + void* stream, std::string& err); /// `ref/model.py` L520: `fin = GR.gr_read(R, output_hc_*..., inject=None, rms_eps); logits = output @ fin`. /// diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index 400d79d3f52..5f4cdb8745a 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -16,7 +16,9 @@ // * the attention is DENSE over every cell the layer has seen - identical to the model's sparse selection // below 2,051 cells - so speculative cells (draft steps, rejected window rows) never touch indexer state and // are simply overwritten when their positions are processed again; -// * all 512 routed experts live in VRAM (708 MB) and run through the grouped hit kernels. +// * all 512 routed experts live in VRAM (708 MB) and run through the grouped hit kernels; +// * the K/V of a prompt's cells come from the batched prompt path (`prompt_kv`), in its GEMM arithmetic, as the +// model's own layers there. #pragma once #include "strata/core/layer.hpp" @@ -32,6 +34,19 @@ namespace strata::core { class NativeHead; +/// What the batched prompt path needs for the draft layer's K/V over a prompt (strata/prefill/prefill.hpp): the two +/// input norms and projections, the attention hyper-connection read, the K and V projections and the K norm. +struct MtpPromptKv { + const float *norm_emb = nullptr, *norm_hidden = nullptr; ///< [n_embd], [hc * n_embd] + const void *fc_emb = nullptr, *fc_hidden = nullptr; ///< Q8_0 [n_embd, n_embd] + const float* hc_norm = nullptr; ///< [hc * n_embd] + const uint16_t *hc_down = nullptr, *hc_up = nullptr; ///< BF16 [hc_lr, hc * n_embd], [hc * n_embd, hc_lr] + const void *k_proj = nullptr, *v_proj = nullptr; ///< Q8_0 [n_head_kv * head_dim, n_embd] + const float* k_norm = nullptr; ///< [head_dim] + QsaState* kv = nullptr; ///< the layer's K/V cells + int64_t first_cell = 0; ///< cells before it the window never reaches again +}; + class MtpDrafter { public: MtpDrafter() = default; @@ -43,18 +58,16 @@ class MtpDrafter { /// Call before the VRAM expert tier is sized: this takes ~0.9 GB. bool load(const std::string& rt_dir, const ModelGeometry& g, SessionState& ss, int max_t, std::string& err, int64_t window = 32768); - /// The prompt's length: prefill() skips the cells the attention window can never reach again. + /// The prompt's length: the prompt's K/V skip the cells the attention window can never reach again. void set_prompt_len(int64_t n) { prompt_len_ = n; } uint64_t vram_bytes() const { return vram_; } /// The layer's own K/V cells (the prompt cache stashes them with the model's). QsaState& kv_state() { return st_; } + /// The weights and cells the batched prompt path computes the prompt's K/V with (after load and set_prompt_len). + MtpPromptKv prompt_kv(); /// The main model's embedding and head, and the verify window's final residuals (T rows, hc*n_embd each). bool bind(const WeightTable& wt, const NativeHead* head, const float* window_R, std::string& err); - /// Prompt cells [cell0, cell0 + n): residual rows `R_rows` (device, hc*n_embd each) and `next_tokens` (host, - /// the token at position cell+1). Runs in batches of up to max_t rows. - bool prefill(const float* R_rows, const int32_t* next_tokens, int64_t n, int64_t cell0, std::string& err); - /// One round: catch-up over T cells from `p` (rows = the window's final residuals, `tokens` = the window's /// argmaxes: row t pairs R_{p+t} with the token at p+t+1), then the draft chain from row `a` (the last /// accepted row) for T-1 drafts at cells p+a+1 ... `drafts` gets T-1 tokens. @@ -65,12 +78,11 @@ class MtpDrafter { bool draft_first(int T, const float* R_row, int32_t token, int64_t cell, int32_t* drafts, std::string& err, float* probs = nullptr, float min_p = 0.0f, int* n_drafts = nullptr); - double ms_draft = 0, ms_prefill = 0; + double ms_draft = 0; int64_t rounds = 0; private: bool record_forward(int T, int step_row0, cudaStream_t cs, std::string& err); - bool capture_prefill(int T, std::string& err); bool capture_round(int T, std::string& err); bool capture_step(int j, std::string& err); cudaGraphExec_t step_exec_[9] = {}; @@ -87,7 +99,6 @@ class MtpDrafter { int64_t n_vocab_ = 0; uint64_t vram_ = 0; cudaStream_t cs_ = nullptr; - cudaGraphExec_t prefill_exec_[9] = {}; cudaGraphExec_t round_exec_[9] = {}; struct Tensor { std::string name, kind; int64_t rows = 0, cols = 0; uint64_t off = 0, bytes = 0; }; diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp index b2e5f0e7d7b..4879f44a099 100644 --- a/include/strata/prefill/prefill.hpp +++ b/include/strata/prefill/prefill.hpp @@ -14,6 +14,7 @@ #include "strata/core/expert_cache.hpp" #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" +#include "strata/core/mtp.hpp" #include "strata/core/session.hpp" #include "strata/core/weights.hpp" #include "strata/prefill/experts.hpp" @@ -28,7 +29,7 @@ namespace strata::prefill { /// The sections `Prefill::profile` times, in the order a layer runs them. enum PrefillSection { kPsPle, kPsHcRead, kPsGdn, kPsGdnConv, kPsGdnScan, kPsGdnNorm, kPsGdnOut, kPsQsaProj, kPsQsaIndexer, kPsQsaScores, kPsQsaAttn, kPsQsaOut, kPsHcFfn, kPsRouter, - kPsExperts, kPsCombine, kPsCount + kPsExperts, kPsCombine, kPsDraft, kPsCount }; const char* prefill_section_name(int section); @@ -69,17 +70,17 @@ class Prefill { bool bind(void* region, uint64_t bytes, int64_t chunk, std::string& err); /// Positions [pos0, pos0 + n) holding `tokens`, in chunks of the bound length; `ss.ple_prev` must be the two - /// tokens before pos0 (oldest first, -1 for none) and is advanced to the last two of these. + /// tokens before pos0 (oldest first, -1 for none) and is advanced to the last two of these. With a `draft` + /// layer tokens[n], the token after the last position, is read too. bool run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& err); /// Positions the last `run` completed: n, or fewer after `should_stop` (the session state holds them). int64_t processed() const { return processed_; } const PrefillStats& stats() const { return stats_; } - /// Plan v0.3 P6: called after every chunk with the chunk's final multi-stream residual rows (device, - /// T x hc*n_embd, valid until the next chunk) and the chunk's first position; the MTP draft layer builds its - /// K/V from them. The prefill stream is synchronized before the call. - std::function on_chunk; + /// The MTP draft layer (or null): after every chunk, its K/V over the chunk's cells in the chunk's batched + /// arithmetic (cell t pairs the chunk's final residual at t with the token at t + 1). + core::MtpDrafter* draft = nullptr; /// Checked before every chunk: true stops the prompt there (`run` returns false with err "cancelled", and the /// state and `ss.ple_prev` hold the chunks before it, `processed()` positions). diff --git a/src/core/layer.cpp b/src/core/layer.cpp index da241879c38..b2a275ff836 100644 --- a/src/core/layer.cpp +++ b/src/core/layer.cpp @@ -27,6 +27,7 @@ #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/fused_gdn.hpp" #include "strata/kernels/qsa_select.hpp" +#include "strata/kernels/verify_kernels.hpp" #include #include #include @@ -771,34 +772,25 @@ void doorbell_reset(const Doorbell& db) { if (db.h_flag) *(volatile uint32_t*) db.h_flag = 0; } // ================================ THE TWO ENDS OF A TOKEN ================================ -bool embed_row(const WeightTable& tables, const ModelGeometry& g, int64_t token, float* out_dev, - void* stream, std::string& err) { - if (const NativeEmbed* ne = native_embed()) { // plan v0.3 P6: the GGUF-form table (IQ model files) - if (token < 0 || out_dev == nullptr) { err = "embed_row: invalid token or output"; return false; } - ne->gather_one(token, out_dev, stream); - return true; - } +namespace { +// The pack's S2/S4/S8 embedding table, checked against its planes; null with `err` when it cannot be read. +const WeightRef* s_embedding(const WeightTable& tables, const ModelGeometry& g, std::string& err) { const WeightRef* w = tables.find(EMBEDDING_NAME); - if (w == nullptr) { err = "token_embd.weight is missing"; return false; } + if (w == nullptr) { err = "token_embd.weight is missing"; return nullptr; } if (w->code_bits != 2 && w->code_bits != 4 && w->code_bits != 8) { err = "embed_row: token_embd.weight is not an S2/S4/S8 tensor"; - return false; + return nullptr; } if (w->codebook_iq4nl) { err = "embed_row: an IQ4NL embedding is not supported"; - return false; + return nullptr; } - if (w->data == nullptr || out_dev == nullptr || w->ne0 <= 0 || w->ne1 <= 0 || + if (w->data == nullptr || w->ne0 <= 0 || w->ne1 <= 0 || w->ne0 != g.n_embd || w->group_elems <= 0 || w->ne0 % w->group_elems != 0 || w->ne0 % (8 / w->code_bits) != 0) { err = "embed_row: invalid embedding pointers, dimensions or group size"; - return false; + return nullptr; } - if (token < 0 || token >= w->ne1) { - err = "embed_row: token " + std::to_string(token) + " is outside 0.." + std::to_string(w->ne1 - 1); - return false; - } - const uint64_t row_codes = (uint64_t) (w->ne0 / (8 / w->code_bits)); const uint64_t row_groups = (uint64_t) (w->ne0 / w->group_elems); const uint64_t rows = (uint64_t) w->ne1; @@ -813,9 +805,48 @@ bool embed_row(const WeightTable& tables, const ModelGeometry& g, int64_t token, w->has_offset != (w->offset_bytes != 0) || (w->has_offset && w->offset_bytes != w->scales_bytes)) { err = "embed_row: embedding planes do not match the loaded tensor shape"; - return false; + return nullptr; } + return w; +} +} // namespace +bool embed_rows(const WeightTable& tables, const ModelGeometry& g, const int32_t* tokens_dev, int64_t n, float* out_dev, + void* stream, std::string& err) { + if (n <= 0) return true; + if (tokens_dev == nullptr || out_dev == nullptr) { err = "embed_rows: invalid tokens or output"; return false; } + if (const NativeEmbed* ne = native_embed()) { + ne->gather_dev(tokens_dev, n, out_dev, stream); + return true; + } + const WeightRef* w = s_embedding(tables, g, err); + if (w == nullptr) return false; + const auto* codes = static_cast(w->data); + const auto* scales = reinterpret_cast(codes + w->codes_bytes); + const auto* offsets = w->has_offset ? reinterpret_cast(codes + w->codes_bytes + w->scales_bytes) + : nullptr; + strata::kernels::embedding_gather_dev(codes, scales, offsets, tokens_dev, (int) n, w->ne0, w->code_bits, w->code_bias, + w->group_elems, (uint64_t) (w->ne0 / (8 / w->code_bits)), + (uint64_t) (w->ne0 / w->group_elems), out_dev, stream); + return true; +} + +bool embed_row(const WeightTable& tables, const ModelGeometry& g, int64_t token, float* out_dev, + void* stream, std::string& err) { + if (const NativeEmbed* ne = native_embed()) { // plan v0.3 P6: the GGUF-form table (IQ model files) + if (token < 0 || out_dev == nullptr) { err = "embed_row: invalid token or output"; return false; } + ne->gather_one(token, out_dev, stream); + return true; + } + const WeightRef* w = s_embedding(tables, g, err); + if (w == nullptr) return false; + if (out_dev == nullptr) { err = "embed_row: invalid output"; return false; } + if (token < 0 || token >= w->ne1) { + err = "embed_row: token " + std::to_string(token) + " is outside 0.." + std::to_string(w->ne1 - 1); + return false; + } + const uint64_t row_codes = (uint64_t) (w->ne0 / (8 / w->code_bits)); + const uint64_t row_groups = (uint64_t) (w->ne0 / w->group_elems); const auto* codes = static_cast(w->data); const auto* scales = reinterpret_cast(codes + w->codes_bytes); const auto* offsets = w->has_offset ? reinterpret_cast(codes + w->codes_bytes + w->scales_bytes) diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 8c6536d7483..2441982e666 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -80,7 +80,6 @@ bool read_file(const std::string& path, std::vector& out) { MtpDrafter::~MtpDrafter() { if (cs_) cudaStreamSynchronize(cs_); - for (auto& e : prefill_exec_) if (e) cudaGraphExecDestroy(e); for (auto& e : round_exec_) if (e) cudaGraphExecDestroy(e); for (auto& e : step_exec_) if (e) cudaGraphExecDestroy(e); if (cs_) cudaStreamDestroy(cs_); @@ -278,17 +277,7 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri const int32_t* pos = pos_ + row0 * NH; try { // ---- the two input branches - const WeightRef* we = wt_->find("token_embd.weight"); - if (!we) { err = "mtp: token_embd.weight is missing"; return false; } - if (const NativeEmbed* ne = native_embed()) { // plan v0.3 P6: the GGUF-form table - ne->gather_dev(tok_, T, emb_, cs); - } else { - const auto* codes = (const uint8_t*) we->data; - const auto* scales = (const float*) (codes + we->codes_bytes); - const auto* offsets = we->has_offset ? (const float*) (codes + we->codes_bytes + we->scales_bytes) : nullptr; - embedding_gather_dev(codes, scales, offsets, tok_, T, we->ne0, we->code_bits, we->code_bias, we->group_elems, - (uint64_t) (we->ne0 / (8 / we->code_bits)), (uint64_t) (we->ne0 / we->group_elems), emb_, cs); - } + if (!embed_rows(*wt_, g, tok_, T, emb_, cs, err)) return false; native_qsa_rms_norm_weighted(emb_, f32("pre_fc_norm_embedding.weight"), en_, (int) N, T, EPS, cs); native_quantize_q8_1(en_, xq_, (int) N, T, cs); native_mmvq(GGML_Q8_0, q8("fc_embedding.weight"), xq_, e2_, (int) N, (int) N, T, cs); @@ -439,17 +428,6 @@ bool finish_capture(cudaStream_t cs, bool ok, cudaGraphExec_t& exec, const char* } } // namespace -bool MtpDrafter::capture_prefill(int T, std::string& err) { - if (prefill_exec_[T]) return true; - using namespace strata::kernels; - if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { err = "mtp: begin capture"; return false; } - copy_i32_from_mapped(tok_, m_tok_, T, cs_); - copy_i32_from_mapped(step_, m_step_, (int64_t) T * 4, cs_); - copy_i32_from_mapped(pos_, m_pos_, (int64_t) T * g_->n_head, cs_); - const bool ok = record_forward(T, -1, cs_, err); // K/V only, rows [0, T) - return finish_capture(cs_, ok, prefill_exec_[T], "prefill", err); -} - bool MtpDrafter::capture_round(int T, std::string& err) { if (round_exec_[T]) return true; using namespace strata::kernels; @@ -491,34 +469,21 @@ bool MtpDrafter::capture_step(int j, std::string& err) { return finish_capture(cs_, ok, step_exec_[j], "step", err); } -bool MtpDrafter::prefill(const float* R_rows, const int32_t* next_tokens, int64_t n, int64_t cell0, std::string& err) { - const Clock::time_point t0 = Clock::now(); - const int64_t HCN = g_->hc * g_->n_embd; - // cells the window can never reach again need no K/V - const int64_t first_needed = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; - for (int64_t c = 0; c < n; c += max_t_) { - const int T = (int) std::min(max_t_, n - c); - if (cell0 + c + T <= first_needed) continue; - if (!capture_prefill(T, err)) return false; - for (int t = 0; t < T; ++t) { - const int64_t cell = cell0 + c + t; - h_tok_[t] = next_tokens[c + t]; - h_step_[t * 4 + 0] = (int32_t) cell; - h_step_[t * 4 + 1] = (int32_t) (cell + 1); - h_step_[t * 4 + 2] = (int32_t) ((cell + 1) / 4); - h_step_[t * 4 + 3] = (int32_t) (cell + 1); - for (int64_t h = 0; h < g_->n_head; ++h) h_pos_[t * g_->n_head + h] = (int32_t) cell; - } - if (cudaMemcpyAsync(Rin_, R_rows + (size_t) c * HCN, (size_t) T * HCN * sizeof(float), cudaMemcpyDeviceToDevice, - cs_) != cudaSuccess || - cudaGraphLaunch(prefill_exec_[T], cs_) != cudaSuccess || - cudaStreamSynchronize(cs_) != cudaSuccess) { - err = std::string("mtp prefill: ") + cudaGetErrorString(cudaGetLastError()); - return false; - } - } - ms_prefill += ms_since(t0); - return true; +MtpPromptKv MtpDrafter::prompt_kv() { + MtpPromptKv k; + k.norm_emb = f32("pre_fc_norm_embedding.weight"); + k.norm_hidden = f32("pre_fc_norm_hidden.weight"); + k.fc_emb = q8("fc_embedding.weight"); + k.fc_hidden = q8("fc_hidden.weight"); + k.hc_norm = f32("attn_hyper_connection.hc_norm.weight"); + k.hc_down = bf16("attn_hyper_connection.input_mix_weight_down.weight"); + k.hc_up = bf16("attn_hyper_connection.input_mix_weight_up.weight"); + k.k_proj = q8("self_attn.k_proj.weight"); + k.v_proj = q8("self_attn.v_proj.weight"); + k.k_norm = f32("self_attn.k_norm.weight"); + k.kv = &st_; + k.first_cell = (window_ > 0 && prompt_len_ > 0) ? prompt_len_ - window_ - 64 : 0; + return k; } bool MtpDrafter::draft(int T, const int32_t* tokens, int64_t p, int a, int32_t* drafts, std::string& err, diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 5c8ec97514a..8651ce3e038 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -9,6 +9,7 @@ #include "strata/kernels/qsa.hpp" #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/qsa_select.hpp" +#include "strata/kernels/verify_kernels.hpp" #include "strata/prefill/gemm.hpp" #include "strata/prefill/kernels.hpp" @@ -33,6 +34,7 @@ constexpr int64_t SEL_BATCH = 256; // queries per block-score launch // (GDN: qkv, gate, out; QSA: q, k, v, out) and the shared expert's constexpr int64_t W16 = std::max(D * N, std::max(C + ZV + ZV, 12288 + 512 + 512 + ZV) * N + 3 * 640 * N); constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace +constexpr int GGML_Q8_0 = 8; double ms_since(Clock::time_point t) { return std::chrono::duration(Clock::now() - t).count(); } @@ -327,6 +329,56 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e ev.push_back({section, e}); }; + // The MTP draft layer's K/V over a chunk's cells: its input norms and projections (e = fc_embedding(rms(the + // embedding of the token at t + 1)), h = fc_hidden(rms(R_t)) per stream, h + e on every stream), the attention + // hyper-connection read, then K (normed and rotated) and V into its cells, from the window's first cell on. The + // chunk's buffers are free once its last layer is done; its residual rows are normed in place. + const core::MtpPromptKv dk = draft != nullptr ? draft->prompt_kv() : core::MtpPromptKv{}; + if (draft != nullptr && (!dk.norm_emb || !dk.norm_hidden || !dk.fc_emb || !dk.fc_hidden || !dk.hc_norm || + !dk.hc_down || !dk.hc_up || !dk.k_proj || !dk.v_proj || !dk.k_norm)) { + err = "prefill: a draft layer weight is missing"; + return false; + } + auto draft_pass = [&](const int64_t* toks, int64_t T, int64_t p0) -> bool { + const int64_t from = std::clamp(dk.first_cell - p0, 0, T); + if (from == T) return true; + mark(kPsDraft); + new_layer(); + const uint16_t *We = w16(GGML_Q8_0, dk.fc_emb, N, N), *Wh = w16(GGML_Q8_0, dk.fc_hidden, N, N), + *Wk = w16(GGML_Q8_0, dk.k_proj, 512, N), *Wv = w16(GGML_Q8_0, dk.v_proj, 512, N); + if (!We || !Wh || !Wk || !Wv) { + err = "prefill: the draft layer's projections do not fit the dequantization buffer"; + return false; + } + for (int64_t t = from; t < T; ++t) m.ids_host[t - from] = (int32_t) toks[t + 1]; + cudaMemcpyAsync(b.ids, m.ids_host, (size_t) (T - from) * 4, cudaMemcpyHostToDevice, m.cs); + const core::QsaState& st = *dk.kv; + for (int64_t t0 = from; t0 < T; t0 += SUB) { + const int64_t P = std::min(SUB, T - t0), pp = p0 + t0; + float* R = b.R + t0 * D; + if (!core::embed_rows(*m.wt, g, b.ids + (t0 - from), P, b.emb, m.cs, err)) return false; + rms_rows(b.emb, dk.norm_emb, P, N, N, EPS, m.cs); + to_f16(b.emb, b.mixed_h, P * N, m.cs); + m.gemm.f16(b.mixed_h, We, b.bo, P, N, N); + rms_rows(R, dk.norm_hidden, P, D, D, EPS, m.cs); + to_f16(R, b.xn16, P * D, m.cs); + m.gemm.f16(b.xn16, Wh, b.gated, P * HC, N, N); + strata::kernels::add_streams_broadcast(b.gated, b.bo, b.gated, N, (int) HC, (int) P, m.cs); + gr_norm(b.gated, dk.hc_norm, EPS, b.xn, b.xn16, P, m.cs); + m.gemm.bf16(b.xn16, dk.hc_down, b.lo, P, LR, D); + gr_silu(b.lo, b.lo16, P, m.cs); + m.gemm.bf16(b.lo16, dk.hc_up, b.gated, P, D, LR); + gr_mix(b.xn, b.gated, nullptr, nullptr, P, m.cs, b.mixed_h); + m.gemm.f16(b.mixed_h, Wk, b.Kc, P, 512, N); + m.gemm.f16(b.mixed_h, Wv, b.Vc, P, 512, N); + rms_rows(b.Kc, dk.k_norm, P * 2, 256, 256, EPS, m.cs); + rope(b.Kc, P, 2, 256, 512, pp, (float) strata::kernels::qsa_freq_base(), m.cs); + kv_append(b.Kc, b.Vc, P, pp, st.page_table, s.page_size, st.kv_int8 ? nullptr : st.k_pool, + st.kv_int8 ? nullptr : st.v_pool, st.k_q, st.v_q, st.k_scale, st.v_scale, m.cs); + } + return true; + }; + static const bool trace = std::getenv("STRATA_TRACE") != nullptr; processed_ = 0; int64_t c0 = 0; @@ -601,6 +653,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e } gr_write(b.R, b.bo_moe, b.inj_f, HC, T, m.cs, m.offload ? b.sums : nullptr); } + if (draft != nullptr && !draft_pass(tokens + c0, T, p0)) return false; if (profile) { mark(kPsPle); cudaEventSynchronize(ev.back().second); @@ -613,13 +666,6 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e ev.clear(); } stats_.tokens += T; - if (on_chunk) { - if (cudaStreamSynchronize(m.cs) != cudaSuccess) { - err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError()); - return false; - } - if (!on_chunk(b.R, T, p0, err)) return false; - } } ss.ple_prev[0] = prev[0]; ss.ple_prev[1] = prev[1]; @@ -646,7 +692,7 @@ const char* prefill_section_name(int section) { "GDN output projection", "QSA projections, norms, KV", "QSA indexer appends", "QSA block scores + top-k", "QSA attention", "QSA gate + output", "HC write + HC read (FFN)", - "router, shared expert", "grouping, experts", "combine + HC write"}; + "router, shared expert", "grouping, experts", "combine + HC write", "draft layer K/V"}; return section >= 0 && section < kPsCount ? names[section] : "?"; } diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 63f5df6d369..e68513ecd11 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2279,12 +2279,7 @@ int main(int argc, char** argv) { ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); - std::vector cur; - prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { - std::vector nxt((size_t) T); - for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)]; - return mtp.prefill(R_rows, nxt.data(), T, p0, e); - }; + prefill.draft = &mtp; drive.d.plan = ver.plan_sink(); drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5))); ver.set_pcie_share(drive.d.pcie_num > 0); @@ -2470,7 +2465,6 @@ int main(int argc, char** argv) { bool bad = false; for (int64_t t : ids) bad = bad || t < 0 || t >= n_vocab; if (bad) { std::printf("ERR a token id is outside the vocabulary\n"); continue; } - cur = ids; const Clock::time_point r0 = Clock::now(); auto report = [&](const std::string& e) { std::fprintf(stderr, "strata serve: %s\n", e.c_str()); @@ -2672,12 +2666,7 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } - prefill.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool { - // cell i pairs R_i with the token at i + 1 (every such token is in the prompt) - std::vector nxt((size_t) T); - for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) o.tokens[(size_t) (p0 + t + 1)]; - return mtp.prefill(R_rows, nxt.data(), T, p0, e); - }; + prefill.draft = &mtp; // cell i pairs R_i with the token at i + 1 (every such token is in the prompt) } prefill.profile = o.prefill_profile; const Clock::time_point tp0 = Clock::now(); @@ -3196,9 +3185,8 @@ int main(int argc, char** argv) { drive.d.pcie_num); (void) pool_ms0; if (use_mtp && rounds > 0) - std::printf("%-24s %.3f ms/round drafting (%lld rounds), MTP prompt %.1f ms, %.0f MiB of VRAM\n", "mtp", - mtp.ms_draft / (double) mtp.rounds, (long long) mtp.rounds, mtp.ms_prefill, - (double) mtp.vram_bytes() / 1048576.0); + std::printf("%-24s %.3f ms/round drafting (%lld rounds), %.0f MiB of VRAM\n", "mtp", + mtp.ms_draft / (double) mtp.rounds, (long long) mtp.rounds, (double) mtp.vram_bytes() / 1048576.0); } if (dump != nullptr && std::fclose(dump) != 0) { From 21ec11f84e58301ecfafd35907fa1aae3d847b36 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 16:42:27 +0200 Subject: [PATCH 36/92] PLE rows of a verify window read while layer 0 runs, the drafts' while the draft layer runs The verify window's staging waited 0.4-0.7 ms a round for the SSD reads of its tokens' PLE rows. The graph now waits for them (a host flag) before layer 1, the first to read them: the host starts the reads before the launch and collects them while layer 0 runs. The next window's first token, and each draft likely to be verified (probability >= --spec-min-p), start their reads as soon as they are known (Verifier::ple_ahead, MtpDrafter::on_draft), in read-ahead slots of the PLE table beside its single-token issue/collect. Window hashes unchanged (fixed text, --adapt-every 0, both models). Host time per round 0.41-0.71 -> 0.03-0.19 ms; generation IQ3_XXS code +2.5%, thinking +1.5%; UD-Q4_K_XL code +1.3-1.6%, thinking +1.1%. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/mtp.hpp | 5 ++++ include/strata/core/verify.hpp | 8 +++++- include/strata/kernels/ngram.hpp | 8 ++++++ src/core/mtp.cpp | 2 ++ src/core/verify.cpp | 40 ++++++++++++++++++++++----- src/kernels/ngram.cpp | 46 ++++++++++++++++++++++++++++++++ src/program/generate.cpp | 10 +++++++ 7 files changed, 111 insertions(+), 8 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index 5f4cdb8745a..68af2074a0e 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -27,6 +27,7 @@ #include #include +#include #include #include @@ -78,6 +79,10 @@ class MtpDrafter { bool draft_first(int T, const float* R_row, int32_t token, int64_t cell, int32_t* drafts, std::string& err, float* probs = nullptr, float min_p = 0.0f, int* n_drafts = nullptr); + /// Called with each draft as soon as it is known (j = 0.., its probability under the draft layer), while the + /// chain goes on: the next window's PLE rows start there. + std::function on_draft; + double ms_draft = 0; int64_t rounds = 0; diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 5669b5a5506..82630bc1abf 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -61,8 +61,12 @@ class Verifier { const NativeHead* head, int max_t, std::string& err); /// One window: `tokens[0..T)` at positions pos0.., the pool served per layer; `out[t]` = argmax after token t. - /// The PLE rows are gathered here from `ss.ple_prev` and the tokens. Captures the T-token graph on first use. + /// The PLE rows follow `ss.ple_prev` and the tokens: read while layer 0 runs (the graph waits for them before + /// layer 1), unless `ple_ahead` already started them. Captures the T-token graph on first use. bool run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, void* user, int32_t* out, std::string& err); + /// Starts reading the PLE rows of the next window's token t (0, 1, ... in order, after `commit`): `run` takes + /// them if its token t, and the ones before it, are these. For the drafts, while the draft layer runs. + void ple_ahead(int t, int32_t token); /// Keep the first `n_keep` (1..T) tokens of the last window; advances `ss.ple_prev` by them. With `wait` false /// the commit graph is only launched (the MTP draft, on its own stream, reads nothing it writes): `wait_commit` @@ -134,6 +138,8 @@ class Verifier { uint32_t* h_flag_ = nullptr; uint32_t* m_flag_ = nullptr; uint32_t* h_flagA_ = nullptr; uint32_t* m_flagA_ = nullptr; // the GPU plan is in place uint32_t* h_flagB_ = nullptr; uint32_t* m_flagB_ = nullptr; // the PCIe share's DMA copies have landed + uint32_t* h_pleflag_ = nullptr; uint32_t* m_pleflag_ = nullptr; // the PLE rows are in h_ple_ + int32_t ahead_tok_[8] = {}; // the tokens ple_ahead read for cudaStream_t copy_ = nullptr; // the copy engine's stream (DMA of missed experts) struct FlagSet { uint32_t* flag; uint32_t value; }; FlagSet flag_sets_[2 * 64 * 2] = {}; // host-function arguments, one per (layer, group) diff --git a/include/strata/kernels/ngram.hpp b/include/strata/kernels/ngram.hpp index 7a88fd9ad77..ef6fc180176 100644 --- a/include/strata/kernels/ngram.hpp +++ b/include/strata/kernels/ngram.hpp @@ -142,6 +142,14 @@ class PleTable { /// while a single-token `issue` is pending. The mapped mode gathers row by row. bool gather_batch(const uint32_t* rows, size_t n_tokens, float* out, std::string& err); + /// Tokens read ahead, each in a slot of its own beside issue/collect: `ahead` starts reading a token's 16 rows + /// (a slot still holding other rows waits for them first; outside Direct mode the rows are only recorded), + /// `ahead_holds` tells whether a slot holds exactly these rows, `ahead_collect` waits and gives the 2560 floats. + static constexpr int kAheadSlots = 8; + void ahead(int slot, const uint32_t* rows16); + bool ahead_holds(int slot, const uint32_t* rows16) const; + bool ahead_collect(int slot, float* out2560, std::string& err); + /// Fault injection (Direct mode): every row read completes no earlier than `us` after issue. void set_injected_delay_us(double us); diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 2441982e666..a9a44ff97c3 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -514,6 +514,7 @@ bool MtpDrafter::draft(int T, const int32_t* tokens, int64_t p, int a, int32_t* drafts[0] = ((volatile int32_t*) h_out_)[0]; float pj = ((volatile float*) h_prob_)[0]; if (probs) probs[0] = pj; + if (on_draft) on_draft(0, drafts[0], pj); int n = 1; // the chain continues while the last draft is likely enough to be verified for (int j = 1; j < max_t_ - 1 && pj >= min_p; ++j) { @@ -527,6 +528,7 @@ bool MtpDrafter::draft(int T, const int32_t* tokens, int64_t p, int a, int32_t* drafts[j] = ((volatile int32_t*) h_out_)[j]; pj = ((volatile float*) h_prob_)[j]; if (probs) probs[j] = pj; + if (on_draft) on_draft(j, drafts[j], pj); ++n; } for (int j = n; j < max_t_ - 1; ++j) { drafts[j] = 0; if (probs) probs[j] = 0.0f; } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 9366fc3c65a..52c7651f55a 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -104,7 +104,7 @@ Verifier::~Verifier() { if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, - h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_}; + h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -170,7 +170,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_) && mapped(2 * T * K * 4, (void**) &h_pids_, (void**) &m_pids_) && mapped(2 * T * K * 4, (void**) &h_pw_, (void**) &m_pw_) && - mapped(64, (void**) &h_pseq_, (void**) &m_pseq_); + mapped(64, (void**) &h_pseq_, (void**) &m_pseq_) && + mapped(64, (void**) &h_pleflag_, (void**) &m_pleflag_); if (!ok) { err = "verify: mapped staging allocation failed"; return false; } // the GPU plan: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | ptr2(cap u64) | start2(cap+1) { @@ -303,7 +304,6 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { copy_i32_from_mapped(tok_, m_tok_, T, cs); copy_i32_from_mapped(step_, m_step_, (int64_t) T * kStepCount, cs); copy_i32_from_mapped(pos_, m_pos_, (int64_t) T * NH, cs); - if (ple_on) copy_from_mapped(ple_, m_ple_, (int64_t) T * N, cs); // ---- the embeddings, broadcast to the hc streams if (const NativeEmbed* ne = native_embed()) { // plan v0.3 P6: the GGUF-form table @@ -357,6 +357,10 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } bool pending = l > 0; // the previous layer's FFN write, folded into this layer's first read if (l == 1 && ple_on) { + if (grp == 0) { // the host reads the rows while layer 0 runs + wait_flag_ge(m_pleflag_, 1, cs); + copy_from_mapped(ple_, m_ple_, (int64_t) T * N, cs); + } float* normalized = (float*) ((uint8_t*) ss.ple.scratch + ple_block_scratch_bytes()); for (int t = tb; t < te; ++t) { gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); @@ -815,21 +819,23 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, qsa_step_fill(h_step_ + t * kStepCount, pos0 + t, s); for (int64_t h = 0; h < g.n_head; ++h) h_pos_[t * g.n_head + h] = (int32_t) (pos0 + t); } - if (ss.ple.ready()) { - uint32_t rows[kVerifyMaxT * PLE_N_HEADS]; + const bool ple_on = ss.ple.ready(); + if (ple_on) { // the reads start here unless ple_ahead started them int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]}; for (int t = 0; t < T; ++t) { - ngram_rows(&tokens[t], prev, 1, ss.ple.consts, rows + t * PLE_N_HEADS); + uint32_t rows[PLE_N_HEADS]; + ngram_rows(&tokens[t], prev, 1, ss.ple.consts, rows); + if (!ss.ple.table->ahead_holds(t, rows)) ss.ple.table->ahead(t, rows); prev[0] = prev[1]; prev[1] = tokens[t]; } - if (!ss.ple.table->gather_batch(rows, (size_t) T, h_ple_, err)) return false; } *(volatile uint32_t*) h_seq_ = 0; *(volatile uint32_t*) h_flag_ = 0; *(volatile uint32_t*) h_flagA_ = 0; *(volatile uint32_t*) h_flagB_ = 0; *(volatile uint32_t*) h_pseq_ = 0; + *(volatile uint32_t*) h_pleflag_ = 0; std::atomic_thread_fence(std::memory_order_seq_cst); last_t_ = T; last_pos0_ = pos0; @@ -840,6 +846,15 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, if (le != cudaSuccess) { err = std::string("verify: launch: ") + cudaGetErrorString(le); return false; } (void) cudaStreamQuery(cs_); VDBG("launched\n"); + if (ple_on) { // before layer 1: the graph waits for the flag + const Clock::time_point tp = Clock::now(); + for (int t = 0; t < T; ++t) + if (!ss.ple.table->ahead_collect(t, h_ple_ + (size_t) t * g.n_embd, err)) return false; + std::atomic_thread_fence(std::memory_order_seq_cst); + _mm_sfence(); + *(volatile uint32_t*) h_pleflag_ = 1; + ms_host += ms_since(tp); + } volatile uint32_t* const seq = h_seq_; volatile uint32_t* const flag = h_flag_; const int G = groups_[T] > 0 ? groups_[T] : 1; @@ -957,6 +972,17 @@ void Verifier::print_profile() const { } } +void Verifier::ple_ahead(int t, int32_t token) { + using namespace strata::kernels; + SessionState& ss = *ss_; + if (!ss.ple.ready() || t < 0 || t >= max_t_) return; + ahead_tok_[t] = token; + int32_t prev[2] = {t >= 2 ? ahead_tok_[t - 2] : ss.ple_prev[t], t >= 1 ? ahead_tok_[t - 1] : ss.ple_prev[1]}; + uint32_t rows[PLE_N_HEADS]; + ngram_rows(&token, prev, 1, ss.ple.consts, rows); + ss.ple.table->ahead(t, rows); +} + void Verifier::set_plan_slot(int grp) { const int64_t cap = sink_.cap; int32_t* base = h_plan_ + (size_t) grp * (size_t) plan_i32_; diff --git a/src/kernels/ngram.cpp b/src/kernels/ngram.cpp index e1f585325d0..8db9298852a 100644 --- a/src/kernels/ngram.cpp +++ b/src/kernels/ngram.cpp @@ -163,6 +163,14 @@ struct PleTable::Impl { std::thread loader; std::string ram_note; bool pending_ram = false; // the pending token's rows come from RAM + // the read-ahead slots: a token's rows in flight from the SSD (`via_reader`), else read at collect + struct Ahead { + bool pending = false, via_reader = false; + uint32_t rows[PLE_N_HEADS] = {}; + uint8_t raw[PLE_N_HEADS * PLE_ROW_BYTES_Q8_0] = {}; + strata::ngram::PleReader::Ticket ticket; + }; + Ahead aheads[PleTable::kAheadSlots]; bool from_ram() const { return mode == PleIo::Ram && ram_ready.load(std::memory_order_acquire); } bool direct() const { return (mode == PleIo::Direct || mode == PleIo::Ram) && !from_ram(); } @@ -315,6 +323,7 @@ void PleTable::close() { impl_->ram_base = nullptr; } impl_->reader.close(); + for (auto& a : impl_->aheads) a.pending = false; impl_->pending = false; impl_->mode = PleIo::Mmap; delete impl_->file; @@ -432,6 +441,43 @@ bool PleTable::gather_batch(const uint32_t* rows, size_t n_tokens, float* out, s return true; } +void PleTable::ahead(int slot, const uint32_t* rows16) { + if (slot < 0 || slot >= kAheadSlots) return; + Impl::Ahead& a = impl_->aheads[slot]; + if (a.pending && a.via_reader) { // the slot's earlier rows (a draft the window left out): long since read + std::string err; + (void) impl_->reader.collect(a.ticket, err); + } + std::memcpy(a.rows, rows16, sizeof a.rows); + a.via_reader = impl_->direct() && impl_->reader.is_open(); + if (a.via_reader) a.ticket = impl_->reader.issue(a.rows, PLE_N_HEADS, a.raw); + a.pending = true; +} + +bool PleTable::ahead_holds(int slot, const uint32_t* rows16) const { + if (slot < 0 || slot >= kAheadSlots) return false; + const Impl::Ahead& a = impl_->aheads[slot]; + return a.pending && std::memcmp(a.rows, rows16, sizeof a.rows) == 0; +} + +bool PleTable::ahead_collect(int slot, float* out2560, std::string& err) { + if (slot < 0 || slot >= kAheadSlots || !impl_->aheads[slot].pending) { + err = "PleTable::ahead_collect: slot " + std::to_string(slot) + " holds no rows"; + return false; + } + Impl::Ahead& a = impl_->aheads[slot]; + a.pending = false; + if (!a.via_reader) { + for (int h = 0; h < PLE_N_HEADS; ++h) read_row(a.rows[h], out2560 + (size_t) h * PLE_HEAD_DIM); + return true; + } + if (!impl_->reader.collect(a.ticket, err)) return false; + for (int h = 0; h < PLE_N_HEADS; ++h) + ple_dequant_row(impl_->type, a.raw + (size_t) h * impl_->row_bytes, out2560 + (size_t) h * PLE_HEAD_DIM); + impl_->bytes_read += (uint64_t) PLE_N_HEADS * impl_->row_bytes; + return true; +} + void PleTable::set_injected_delay_us(double us) { impl_->reader.set_injected_delay_us(us); } std::string PleTable::io_report() const { diff --git a/src/program/generate.cpp b/src/program/generate.cpp index e68513ecd11..db917b82964 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2559,6 +2559,9 @@ int main(int argc, char** argv) { int64_t produced_n = 0; const char* finish = "length"; const Clock::time_point d0 = Clock::now(); + mtp.on_draft = [&](int j, int32_t tok, float prob) { // the drafts the next window will verify + if (prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); + }; while (produced_n < max_new) { if (stop_req.load()) { finish = "cancel"; break; } // the client went away int T = S; @@ -2600,6 +2603,7 @@ int main(int argc, char** argv) { return 1; } if (track) pcache.set(p, window.data(), a + 1); // the committed cells p .. p + a + ver.ple_ahead(0, outv[(size_t) a]); first_window = false; bool eos = false; for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) { @@ -2625,6 +2629,7 @@ int main(int argc, char** argv) { x = outv[(size_t) a]; p += a + 1; } + mtp.on_draft = nullptr; const double decode_ms = std::chrono::duration(Clock::now() - d0).count(); std::printf("DONE %lld %lld %.1f %.1f %s %lld\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms, finish, (long long) reuse); @@ -3009,6 +3014,10 @@ int main(int argc, char** argv) { const double pool_ms0 = drive.cpu_ms; const int64_t misses0 = drive.d.multi_misses, entries0 = drive.d.multi_entries; if (o.stats) drive.d.routed.assign((size_t) (g.n_layers * g.n_expert), 0); + if (use_mtp) + mtp.on_draft = [&](int j, int32_t tok, float prob) { // the drafts the next window will verify + if (prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); + }; while ((int64_t) produced.size() < max_new) { const Clock::time_point t0 = Clock::now(); int T = o.spec; @@ -3078,6 +3087,7 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } + ver.ple_ahead(0, outv[(size_t) a]); ++rounds; drafts_total += T - 1; drafts_ok += a; From 8d8af3bfe8a9f2850eba853d590c13d3d6aacfa9 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 16:45:47 +0200 Subject: [PATCH 37/92] --window-hashes reads the final rows from mapped memory the graph writes The hash's per-window cudaMemcpy of the final residual rows queued behind the adaptive tier's expert copies on the copy engine and slowed generation (UD-Q4_K_XL code 89 -> 77 t/s with the tier on). With --window-hashes the verify graph now copies them into mapped host memory with a kernel after the head. Same bytes hashed (identical files to the previous build); UD-Q4_K_XL code with hashes 88.3 t/s. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 6 ++++++ src/core/verify.cpp | 6 ++++-- src/program/generate.cpp | 6 ++---- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 82630bc1abf..4782d7ffe77 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -77,6 +77,10 @@ class Verifier { /// Token t's residual after the last layer, (hc, n_embd) on the device, valid until the next `run`. const float* final_R(int t) const; const float* final_R_all() const { return R_; } + /// With `set_host_rows(true)` (before the first `run`), the same rows in host memory once `run` returns: the + /// graph writes them there itself, so reading them waits for no copy engine. + void set_host_rows(bool on) { host_rows_ = on; } + const float* final_R_host() const { return h_rows_; } /// The GPU plan the pool writes each layer (VRAM hits + the PCIe share of the misses); give it to the /// dispatch (`ExpertDispatch::plan`) before the first `run`. @@ -139,6 +143,8 @@ class Verifier { uint32_t* h_flagA_ = nullptr; uint32_t* m_flagA_ = nullptr; // the GPU plan is in place uint32_t* h_flagB_ = nullptr; uint32_t* m_flagB_ = nullptr; // the PCIe share's DMA copies have landed uint32_t* h_pleflag_ = nullptr; uint32_t* m_pleflag_ = nullptr; // the PLE rows are in h_ple_ + bool host_rows_ = false; + float* h_rows_ = nullptr; float* m_rows_ = nullptr; // T * hc * n_embd: the final rows (set_host_rows) int32_t ahead_tok_[8] = {}; // the tokens ple_ahead read for cudaStream_t copy_ = nullptr; // the copy engine's stream (DMA of missed experts) struct FlagSet { uint32_t* flag; uint32_t value; }; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 52c7651f55a..329d93f2389 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -104,7 +104,7 @@ Verifier::~Verifier() { if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, - h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_}; + h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -171,7 +171,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(2 * T * K * 4, (void**) &h_pids_, (void**) &m_pids_) && mapped(2 * T * K * 4, (void**) &h_pw_, (void**) &m_pw_) && mapped(64, (void**) &h_pseq_, (void**) &m_pseq_) && - mapped(64, (void**) &h_pleflag_, (void**) &m_pleflag_); + mapped(64, (void**) &h_pleflag_, (void**) &m_pleflag_) && + mapped(T * HC * N * 4, (void**) &h_rows_, (void**) &m_rows_); if (!ok) { err = "verify: mapped staging allocation failed"; return false; } // the GPU plan: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | ptr2(cap u64) | start2(cap+1) { @@ -696,6 +697,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { sp.temperature = 0.0f; sample_tokens(head_logits_, T, (int) n_vocab_, nullptr, 0, sp, m_out_, cs); } + if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // the kernel copies either way stamp(g.n_layers, 3); stamp(g.n_layers, 4); // right after the last: one stamp's own cost return true; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index db917b82964..8b656dba111 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2978,6 +2978,7 @@ int main(int argc, char** argv) { mem_mark("the verifier and the drafter's binding"); ver.set_split(o.spec_split); ver.set_profile(o.window_profile); + ver.set_host_rows(!o.window_hashes.empty()); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); @@ -3008,7 +3009,6 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_hashes.c_str()); return 1; } - std::vector hash_rows; std::vector accepted_hist((size_t) o.spec, 0); int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; @@ -3054,10 +3054,8 @@ int main(int argc, char** argv) { } if (hashes != nullptr) { // FNV-1a over the window's final residual rows const size_t n_floats = (size_t) T * (size_t) (g.hc * g.n_embd); - hash_rows.resize(n_floats); - cudaMemcpy(hash_rows.data(), ver.final_R_all(), n_floats * sizeof(float), cudaMemcpyDeviceToHost); uint64_t h = 1469598103934665603ull; - const auto* bytes = (const uint8_t*) hash_rows.data(); + const auto* bytes = (const uint8_t*) ver.final_R_host(); for (size_t i = 0; i < n_floats * sizeof(float); ++i) h = (h ^ bytes[i]) * 1099511628211ull; std::fprintf(hashes, "%lld %lld %d %016llx", (long long) rounds, (long long) p, T, (unsigned long long) h); for (int i = 0; i < T; ++i) std::fprintf(hashes, " %d", (int) outv[(size_t) i]); From 92d1abb6c2f6a014cfcc66a3b02659fd7f3ff078 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 16:58:24 +0200 Subject: [PATCH 38/92] Verify window: the PLE flag is raised even when a row read fails The graph waits for it before layer 1; returning on a read error without it left the graph spinning. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- src/core/verify.cpp | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 329d93f2389..743f003667f 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -697,7 +697,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { sp.temperature = 0.0f; sample_tokens(head_logits_, T, (int) n_vocab_, nullptr, 0, sp, m_out_, cs); } - if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // the kernel copies either way + if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // a plain copy kernel, here to host stamp(g.n_layers, 3); stamp(g.n_layers, 4); // right after the last: one stamp's own cost return true; @@ -848,14 +848,15 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, if (le != cudaSuccess) { err = std::string("verify: launch: ") + cudaGetErrorString(le); return false; } (void) cudaStreamQuery(cs_); VDBG("launched\n"); - if (ple_on) { // before layer 1: the graph waits for the flag + if (ple_on) { // before layer 1: the graph waits for the flag, raised even when a read failed const Clock::time_point tp = Clock::now(); - for (int t = 0; t < T; ++t) - if (!ss.ple.table->ahead_collect(t, h_ple_ + (size_t) t * g.n_embd, err)) return false; + bool ok = true; + for (int t = 0; t < T && ok; ++t) ok = ss.ple.table->ahead_collect(t, h_ple_ + (size_t) t * g.n_embd, err); std::atomic_thread_fence(std::memory_order_seq_cst); _mm_sfence(); *(volatile uint32_t*) h_pleflag_ = 1; ms_host += ms_since(tp); + if (!ok) return false; } volatile uint32_t* const seq = h_seq_; volatile uint32_t* const flag = h_flag_; From 81d271e478b963daecc4ae5e3e1e70fceb52b809 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 17:09:54 +0200 Subject: [PATCH 39/92] DETAILS.md: the RTX 3090 speeds with the new QSA selection, draft pass and PLE reads Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- docs/DETAILS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index e4dd0249d56..2c78d65efd1 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -103,7 +103,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 945-1043 tok/s, generation 57-66 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 996-1112 tok/s, generation 57-67 tok/s. ### A second GPU @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1435-1724 tok/s instead of 945-1043, generation 79-93 tok/s instead of 57-66; -IQ3_XXS 1468-1791 instead of 1241-1458, generation 82-110 instead of 79-99. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1721-1909 tok/s instead of 996-1112, generation 80-94 tok/s instead of 57-67; +IQ3_XXS 1833-2017 instead of 1446-1602, generation 88-113 instead of 88-100. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From 3f3c26deb14004535eda72d83d18b90f9edba0fa Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 19:34:35 +0200 Subject: [PATCH 40/92] Verify window: the 3090 decides its cache hits in the graph, the shared expert beside them The window copies a snapshot of the residency table (Verifier::residency, which the pool reads too) and plans the main GPU's hits on the device (verify_hit_plan), so its cached experts start right after the doorbell instead of waiting for the host's plan; the host plan now carries only the PCIe share, and the combine takes both lists. The shared expert runs on a graph branch of its own, forked at the doorbell and joined before the combine, and no longer converts the activations to BF16 when the native gate reads them in FP32. Window hashes unchanged on both models (also with a PCIe share, --spec-split and --second-gpu -1); verify_parity checks the plan against the host's grouping and the combine with two lists. Fixed text, both GPUs: IQ3_XXS code 105.4 -> 108.9 t/s, thinking 89.1 -> 93.2; UD-Q4_K_XL code 86.9 -> 88.0, thinking 74.7 -> 77.5. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/expert_source.hpp | 33 ++- include/strata/core/verify.hpp | 45 +++- include/strata/kernels/native_moe.hpp | 9 +- include/strata/kernels/shared_expert.hpp | 1 + include/strata/kernels/verify_kernels.hpp | 7 + src/core/expert_source.cpp | 164 ++++++------ src/core/verify.cpp | 289 ++++++++++++++-------- src/kernels/cuda/native_moe.cu | 17 +- src/kernels/cuda/shared_expert.cu | 1 + src/kernels/cuda/verify_kernels.cu | 66 +++++ src/kernels/verify_parity.cpp | 85 ++++++- src/program/generate.cpp | 11 +- 12 files changed, 491 insertions(+), 237 deletions(-) diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index 6f63f70482a..c4539081aac 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -63,23 +63,22 @@ class ExpertSource { virtual const uint8_t* device_alias(int64_t layer, int64_t expert) const { (void) layer; (void) expert; return nullptr; } }; -/// Plan v0.3 P6: what the GPU computes in a verify window's layer, written by the pool (mapped host memory) right -/// after the ring and published before the CPU starts its own share. Groups of entries that share a blob; the -/// blob is a VRAM slot or a pinned host blob read over PCIe. +/// Plan v0.3 P6: the PCIe share of a verify window's layer - missed experts the main GPU reads over PCIe - written by +/// the pool (mapped host memory) right after the ring and published before the CPU starts its own share. (The +/// experts in its VRAM tier the window decides itself, from the residency the pool reads too.) Groups of entries +/// that share a blob: `ptr2[q]` is staging slot q (DMA) or the arena's device alias, its entries dst/tok +/// [start2[q], start2[q + 1]). counts[1] = entries, counts[2] = groups. struct GpuPlanSink { - int32_t* counts = nullptr; ///< [0] groups, [1] entries - int32_t* start = nullptr; ///< cap + 1 + int32_t* counts = nullptr; int32_t* dst = nullptr; ///< cap: the entry's row of parts (token * k + j) int32_t* tok = nullptr; ///< cap: the entry's token - unsigned long long* ptr = nullptr; ///< cap: the group's blob, device address - /// Plan v0.3 P6 (DMA): the PCIe share is a second list of groups - `ptr2[q]` is staging slot q, `start2` indexes - /// the same dst/tok entries - computed by the GPU after `fetch` has copied their blobs into staging with the - /// copy engine. counts[0] = VRAM groups, counts[1] = all entries, counts[2] = PCIe groups. - unsigned long long* ptr2 = nullptr; - int32_t* start2 = nullptr; + unsigned long long* ptr2 = nullptr; ///< cap + int32_t* start2 = nullptr; ///< cap + 1 unsigned long long staging = 0; int64_t staging_cap = 0; int64_t cap = 0; + /// Whether the window takes a PCIe share (`--pcie-frac` > 0): the pool publishes one every layer, else none. + bool pcie = false; void (*publish)(void* ctx) = nullptr; /// Starts the DMA copies of `n` host blobs (pinned) into staging slots 0..n-1 and signals the GPU when they land. void (*fetch)(void* ctx, const uint8_t* const* src, int n, size_t bytes) = nullptr; @@ -156,9 +155,9 @@ struct ExpertDispatch { /// so the decision is made exactly once, on this layer's ids, and neither side can re-decide it. std::vector is_hit; bool decided = false; - /// Plan v0.3 P4 token graph: the STATIC residency table (`n_layers x n_expert`, slot or -1), the host's copy - /// of what the device hit path reads. When set, the pool leaves a resident expert's row at zero (the GPU - /// computes it) without any `Launch` callback. + /// Plan v0.3 P4 token graph: the residency table (`n_layers x n_expert`, slot or -1), the host's copy of what the + /// device hit path reads (in verify windows `Verifier::residency`). When set, the pool leaves a resident + /// expert's row to the GPU without any `Launch` callback. const int32_t* host_res = nullptr; /// Plan v0.3 P4: split every expert by rows across the pool's threads (default on; A/B `--no-split-rows`). bool split_rows = true; @@ -205,8 +204,8 @@ struct ExpertDispatch { std::vector nact_multi; std::vector jobs_multi; std::vector job_of; - /// Plan v0.3 P6: the verify window's GPU plan (VRAM hits + the PCIe share of the misses); `pcie_num`/256 of - /// each layer's distinct missed experts (the last ones in routing order) are read by the GPU over PCIe. + /// Plan v0.3 P6: the verify window's PCIe share; `pcie_num`/256 of each layer's distinct missed experts (the last + /// ones in routing order) are read by the GPU over PCIe. GpuPlanSink* plan = nullptr; int pcie_num = 0; int64_t pcie_experts = 0; ///< distinct experts the GPU read over PCIe in verify windows @@ -249,7 +248,7 @@ void expert_pool_dispatch(void* user, const float* x_f, const int32_t* ids, cons /// Plan v0.3 P6: the pool for a verify window of `n_tok` tokens. `x_f` is (n_tok, n_embd), `ids` (n_tok, k) and /// `out` (n_tok * k, n_embd). Each distinct missed expert is computed once for all the tokens routed to it; -/// resident experts' rows are zeroed (the GPU adds them). Requires `host_res` (the token-graph residency). +/// resident experts' rows are the main GPU's (zeroed, or left unwritten with `GpuPlanSink::host_rows_only`). void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out); /// Copies `layer`'s likeliest experts that neither GPU holds to the second GPU's prefetch slots, where that layer's diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 4782d7ffe77..1ced279195c 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -17,8 +17,13 @@ // and the PLE history is set to its snapshot after token n_keep-1. K/V cells past the accepted prefix are simply // overwritten when those positions are processed again, before any query can read them. // +// EXPERTS. The main GPU computes the routed experts its VRAM tier holds, decided in the graph from a snapshot of the +// residency table that the pool reads too (`residency`), so neither waits for the other; its shared expert runs on a +// branch of its own beside them. The pool computes the rest on the CPU and the second GPU, and with a PCIe share +// publishes the missed experts the GPU reads over PCIe. +// // Requires the default native decode configuration (native projections, fused GR, fused GDN, fast attention and -// selection, native indexer) and a profile-filled VRAM expert tier with its residency table on the device. +// selection, native indexer) and a profile-filled VRAM expert tier. #pragma once #include "strata/core/expert_source.hpp" @@ -43,9 +48,12 @@ using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, i /// host memory. using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); +/// The main GPU's VRAM expert tier. struct VerifyHits { - const int32_t* d_res = nullptr; ///< device [n_layers * n_expert] slot or -1 - const uint8_t* cache_base = nullptr; ///< slot 0 of the VRAM expert arena + const int32_t* res = nullptr; ///< host [n_layers * n_expert]: slot or -1, the live residency table + const uint8_t* cache_base = nullptr; ///< slot 0 of the VRAM expert arena + const uint64_t* slot_off = nullptr; ///< [slots]: each slot's byte offset from `cache_base`; null: slot * blob + int64_t slots = 0; int64_t blob = 0; }; @@ -74,6 +82,10 @@ class Verifier { bool commit(int n_keep, std::string& err, bool wait = true); bool wait_commit(std::string& err); + /// The residency the window decides the main GPU's hits by: `VerifyHits::res` as `init` and the last `run` found + /// it. The pool reads this one (`ExpertDispatch::host_res`), so it leaves exactly the GPU's rows to the GPU. + const int32_t* residency() const { return h_res_; } + /// Token t's residual after the last layer, (hc, n_embd) on the device, valid until the next `run`. const float* final_R(int t) const; const float* final_R_all() const { return R_; } @@ -82,8 +94,8 @@ class Verifier { void set_host_rows(bool on) { host_rows_ = on; } const float* final_R_host() const { return h_rows_; } - /// The GPU plan the pool writes each layer (VRAM hits + the PCIe share of the misses); give it to the - /// dispatch (`ExpertDispatch::plan`) before the first `run`. + /// Where the pool publishes each layer's PCIe share of the misses; give it to the dispatch + /// (`ExpertDispatch::plan`) before the first `run`. GpuPlanSink* plan_sink() { return &sink_; } /// Plan v0.3 P6: split the window into two token groups and pipeline the CPU experts of one with the GPU work /// of the other (default on). Set before the first `run`. @@ -95,7 +107,7 @@ class Verifier { void set_pcie_mode(int mode) { sink_.pcie_mode = mode; } /// Whether the pool may give the GPU a PCIe share of the misses (`--pcie-frac` > 0); without one the window /// has no PCIe stage. Set before the first `run`. - void set_pcie_share(bool on) { pcie_share_ = on; } + void set_pcie_share(bool on) { pcie_share_ = on; sink_.pcie = on; } /// Diagnostics: GPU timestamps between the stages of every layer (`--window-profile`; each costs a kernel /// launch, ~2 us). Set before the first `run`; `print_profile` writes the per-stage table to stdout. void set_profile(bool on) { profile_ = on; } @@ -140,7 +152,7 @@ class Verifier { float* h_w_ = nullptr; float* m_w_ = nullptr; // T * k uint32_t* h_seq_ = nullptr; uint32_t* m_seq_ = nullptr; uint32_t* h_flag_ = nullptr; uint32_t* m_flag_ = nullptr; - uint32_t* h_flagA_ = nullptr; uint32_t* m_flagA_ = nullptr; // the GPU plan is in place + uint32_t* h_flagA_ = nullptr; uint32_t* m_flagA_ = nullptr; // the PCIe share's plan is in place uint32_t* h_flagB_ = nullptr; uint32_t* m_flagB_ = nullptr; // the PCIe share's DMA copies have landed uint32_t* h_pleflag_ = nullptr; uint32_t* m_pleflag_ = nullptr; // the PLE rows are in h_ple_ bool host_rows_ = false; @@ -151,8 +163,11 @@ class Verifier { FlagSet flag_sets_[2 * 64 * 2] = {}; // host-function arguments, one per (layer, group) static void fetch_dma(void* ctx, const uint8_t* const* src, int n, size_t bytes); static void raise_flag(uint32_t* flag, uint32_t value); - int32_t* h_plan_ = nullptr; int32_t* m_plan_ = nullptr; // counts | start | dst | tok | ptr (as int32 pairs) - int64_t plan_i32_ = 0; // int32 words in the plan block + int32_t* h_plan_ = nullptr; int32_t* m_plan_ = nullptr; // the PCIe share's plan block per token group + int64_t plan_i32_ = 0, plan_ptr_off_ = 0, plan_stride_ = 0; // a plan block's int32 words, its ptr field, the + // device blocks' stride (see `init`) + int32_t* h_res_ = nullptr; int32_t* m_res_ = nullptr; // the residency snapshot (`residency`) + int64_t res_words_ = 0; GpuPlanSink sink_; uint32_t cur_layer_ = 0; static void publish_plan(void* ctx); @@ -170,6 +185,8 @@ class Verifier { uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; cudaStream_t side_ = nullptr; // the prediction's branch of the graph cudaEvent_t fork_ = nullptr, join_ = nullptr; + cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) + cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; // device void* arena_ = nullptr; @@ -187,7 +204,10 @@ class Verifier { int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr; float *plogits_ = nullptr, *pw_ = nullptr; // the next layer's router (prediction) int32_t* pids_ = nullptr; - int32_t* plan_ = nullptr; // device copy of the plan block + int32_t* plan_ = nullptr; // device copies of the PCIe share's plan blocks + int32_t* dplan_ = nullptr; // the main GPU's hit plans, decided on the device + int32_t* res_ = nullptr; // the residency snapshot, copied as a window starts + unsigned long long* slot_ptr_ = nullptr; // each VRAM slot's address uint8_t* staging_ = nullptr; // VRAM slots for the PCIe share of the misses static constexpr int64_t kStagingBlobs = 16; uint8_t* hit_xq_ = nullptr; @@ -200,8 +220,9 @@ class Verifier { float* hist_snap_ = nullptr; // T * NG_HIST * NG_HC_DIM int64_t cap_ = 0, max_blocks_ = 0, attn_scratch_floats_ = 0; - // --window-profile: stamps [window size][layer, then one row for the window][stage] - static constexpr int kStamps = 13; + // --window-profile: stamps [window size][layer, then one row for the window][stage]: a layer's main-stream stages, + // then the end of its shared expert's branch + static constexpr int kLayerStamps = 13, kStamps = 14; bool profile_ = false; unsigned long long* stamps_ = nullptr; std::vector h_stamps_; diff --git a/include/strata/kernels/native_moe.hpp b/include/strata/kernels/native_moe.hpp index 793231770dd..42735f9e131 100644 --- a/include/strata/kernels/native_moe.hpp +++ b/include/strata/kernels/native_moe.hpp @@ -18,9 +18,10 @@ void native_moe_combine(const float* parts, const float* weights, const float* s // The verify window's combine for n_tok tokens, bitwise `native_moe_combine` per token over the rows the window // used to assemble: row t*k + j is the GPU's (`gpu_rows`, row-indexed, as the host's zeroed row plus the hit) when -// an entry of dst[0..*count) names it, and otherwise the host's (`host_rows`, mapped memory: only these rows cross -// PCIe). weights (n_tok, k), shared and output (n_tok, n_embd). Graph-capturable. +// an entry of dst[0..*count) or dst2[0..*count2) names it (`dst2` may be null), and otherwise the host's +// (`host_rows`, mapped memory: only these rows cross PCIe). weights (n_tok, k), shared and output (n_tok, n_embd). +// Graph-capturable. void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, - const int32_t* count, const float* weights, const float* shared, float* output, - int64_t n_embd, int64_t k, int n_tok, void* stream); + const int32_t* count, const int32_t* dst2, const int32_t* count2, const float* weights, + const float* shared, float* output, int64_t n_embd, int64_t k, int n_tok, void* stream); } diff --git a/include/strata/kernels/shared_expert.hpp b/include/strata/kernels/shared_expert.hpp index 8909ed9e607..948fd896e7a 100644 --- a/include/strata/kernels/shared_expert.hpp +++ b/include/strata/kernels/shared_expert.hpp @@ -24,6 +24,7 @@ namespace strata::kernels { /// Configure before session capture; captured graphs retain their selected kernels. /// In this mode shared_expert requires its optional unrounded x_f32 input. void shared_expert_set_native_bf16(bool enabled); +bool shared_expert_native_bf16(); /// Optional native GGUF projections. Each supported type with nonnull data /// replaces only that canonical projection; absent or unsupported entries fall diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index fb94397b43a..8902ccf51d5 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -57,6 +57,13 @@ void copy_indexed(float* dst, const float* src, int64_t stride, const int32_t* i void fetch_blobs(const unsigned long long* src, const int32_t* n, uint8_t* dst, int64_t blob_bytes, int cap, void* stream); /// ptr[k] = base + k * blob_bytes for k < *n (the staged copies `fetch_blobs` made). void rebase_ptrs(unsigned long long* ptr, const int32_t* n, uint8_t* base, int64_t blob_bytes, void* stream); +/// The main GPU's share of a verify window's layer, decided on the device: the entries of `ids` (n = tokens * k, +/// n <= 128) whose expert its VRAM tier holds (`res[e]` = slot, else < 0; `slot_ptr[slot]` = the slot's address). One +/// group per distinct resident expert, in the order of their first entries, its entries in routing order; written in +/// the plan block's layout (`Verifier`): counts [groups, entries, 0, 0] | start (cap + 1) | dst (cap) | tok (cap) | +/// ptr (cap u64, at int32 offset `ptr_off`). +void verify_hit_plan(const int32_t* ids, int n, int k, const int32_t* res, int n_expert, + const unsigned long long* slot_ptr, int32_t* plan, int64_t cap, int64_t ptr_off, void* stream); // ---- the MTP draft layer (src/core/mtp.cpp) /// R[t][c][:] = h[t][c][:] + e[t][:] (the embedding branch added to every stream). diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index ec231289ac3..4dfc00bafbb 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -281,9 +281,19 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (!d.routed.empty()) for (int64_t i = 0; i < n_tok * k; ++i) if (ids[i] >= 0 && ids[i] < d.n_expert) ++d.routed[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]]; - // ---- plan v0.3 P6: the GPU's share, decided and published FIRST so the GPU starts while the CPU works. - // Distinct experts in routing order; resident ones and the last pcie_num/256 of the missed ones go to the GPU. + // ---- plan v0.3 P6: who computes each routed entry. Distinct experts in routing order: the ones the main GPU's + // VRAM tier holds are its own (the window decides them itself, from the same residency), the second GPU's when it + // takes this layer, the last pcie_num/256 of the rest the PCIe share (published FIRST, so the GPU reads them while + // the CPU works), the others the CPU's. const int64_t n = n_tok * k; + if (n > 128) { + d.failed = true; + d.fail = "a verify window routes more than 128 entries"; + d.fail_layer = d.layers; + return; + } + const int32_t* res = d.host_res != nullptr ? d.host_res + (size_t) d.layers * (size_t) d.n_expert : nullptr; + auto in_vram = [&](int32_t e) { return res != nullptr && res[e] >= 0; }; // e in range int32_t kind[128]; // per entry: -1 CPU, 0 VRAM, 1 PCIe, 2 the second GPU // the second GPU's share: group g = the expert in its slot g2_slot[g], entries g2_ent[g2_start[g] ..) int n2g = 0, n2e = 0; @@ -296,17 +306,22 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 return p >= 0 ? -2 - p : -1; }; auto on_gpu2 = [&](int32_t e) { return slot2(e) != -1; }; + int64_t distinct[128], first_of[128]; + int nd = 0; + for (int64_t i = 0; i < n; ++i) { + first_of[i] = i; + for (int64_t j = 0; j < i; ++j) + if (ids[j] == ids[i]) { first_of[i] = first_of[j]; break; } + if (first_of[i] == i) distinct[nd++] = i; + } // The second GPU takes this layer's share only when the layer's misses are big enough that the CPU would need // longer for all of them than its round trip (~100 us: ~4 MB of experts at the pool's ~45 GB/s); otherwise it // would only add its latency. if (d.gpu2 != nullptr) { uint64_t miss_bytes = 0, gpu2_bytes = 0; - for (int64_t i = 0; i < n_tok * k; ++i) { - const int32_t e = ids[i]; - if (e < 0 || e >= d.n_expert || d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) continue; - bool first = true; - for (int64_t j = 0; j < i && first; ++j) first = ids[j] != e; - if (!first) continue; + for (int q = 0; q < nd; ++q) { + const int32_t e = ids[distinct[q]]; + if (e < 0 || e >= d.n_expert || in_vram(e)) continue; miss_bytes += lay.blob_bytes(d.layers); if (on_gpu2(e)) gpu2_bytes += lay.blob_bytes(d.layers); } @@ -314,103 +329,74 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (gpu2_bytes > 0 && !d.gpu2_used) ++d.gpu2_skipped; } auto on_gpu2_now = [&](int32_t e) { return d.gpu2_used && on_gpu2(e); }; - if (d.plan != nullptr && n <= 128 && n <= d.plan->cap) { - int64_t distinct[128], first_of[128]; - int nd = 0, nmiss = 0; - for (int64_t i = 0; i < n; ++i) { - first_of[i] = i; - for (int64_t j = 0; j < i; ++j) - if (ids[j] == ids[i]) { first_of[i] = first_of[j]; break; } - if (first_of[i] == i) { - distinct[nd++] = i; - const int32_t e = ids[i]; - if (e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] < 0 && - !on_gpu2_now(e)) - ++nmiss; - } - } - const bool pcie_ok = d.pcie_num > 0 && d.src->device_alias(d.layers, 0) != nullptr; - const int m = pcie_ok ? (nmiss * d.pcie_num) >> 8 : 0; - int miss_rank = 0, groups = 0, entries = 0, fetches = 0; - GpuPlanSink& P = *d.plan; - const uint8_t* dma_src[64]; - int64_t pcie_i0[64]; - for (int q = 0; q < nd; ++q) { - const int64_t i0 = distinct[q]; - const int32_t e = ids[i0]; - int kd = -1; - unsigned long long ptr = 0; - if (e >= 0 && e < d.n_expert) { - const int32_t slot = d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e]; - if (slot >= 0) { - kd = 0; - ptr = (unsigned long long) (d.cache_base + (d.cache_slot_off ? (size_t) d.cache_slot_off[slot] - : (size_t) slot * (size_t) d.cache_blob)); - } else if (on_gpu2_now(e)) { - kd = 2; - g2_slot[n2g] = slot2(e); - g2_start[n2g++] = n2e; - for (int64_t i = i0; i < n; ++i) - if (first_of[i] == i0) g2_ent[n2e++] = (int32_t) i; - } else { - if (miss_rank >= nmiss - m && fetches < P.staging_cap && fetches < 64) { - const uint8_t* src = d.src->blob(d.layers, e); - if (src != nullptr && d.src->pinned(d.layers, e)) { - kd = 1; - dma_src[fetches] = src; - pcie_i0[fetches] = i0; - ++fetches; - } + GpuPlanSink* P = d.plan != nullptr && d.plan->pcie && n <= d.plan->cap ? d.plan : nullptr; + int nmiss = 0; + for (int q = 0; q < nd; ++q) { + const int32_t e = ids[distinct[q]]; + if (e >= 0 && e < d.n_expert && !in_vram(e) && !on_gpu2_now(e)) ++nmiss; + } + const bool pcie_ok = P != nullptr && d.pcie_num > 0 && d.src->device_alias(d.layers, 0) != nullptr; + const int m = pcie_ok ? (nmiss * d.pcie_num) >> 8 : 0; + int miss_rank = 0, fetches = 0; + const uint8_t* dma_src[64]; + int64_t pcie_i0[64]; + for (int q = 0; q < nd; ++q) { + const int64_t i0 = distinct[q]; + const int32_t e = ids[i0]; + int kd = -1; + if (e >= 0 && e < d.n_expert) { + if (in_vram(e)) { + kd = 0; + } else if (on_gpu2_now(e)) { + kd = 2; + g2_slot[n2g] = slot2(e); + g2_start[n2g++] = n2e; + for (int64_t i = i0; i < n; ++i) + if (first_of[i] == i0) g2_ent[n2e++] = (int32_t) i; + } else { + if (pcie_ok && miss_rank >= nmiss - m && fetches < P->staging_cap && fetches < 64) { + const uint8_t* src = d.src->blob(d.layers, e); + if (src != nullptr && d.src->pinned(d.layers, e)) { + kd = 1; + dma_src[fetches] = src; + pcie_i0[fetches] = i0; + ++fetches; } - ++miss_rank; } + ++miss_rank; } - for (int64_t i = i0; i < n; ++i) - if (first_of[i] == i0) kind[i] = kd; - if (kd != 0) continue; // the VRAM groups first; the PCIe groups below - P.ptr[groups] = ptr; - P.start[groups] = entries; - for (int64_t i = i0; i < n; ++i) - if (first_of[i] == i0) { - P.dst[entries] = (int32_t) i; - P.tok[entries] = (int32_t) (i / k); - ++entries; - } - ++groups; } - P.start[groups] = entries; + for (int64_t i = i0; i < n; ++i) + if (first_of[i] == i0) kind[i] = kd; + } + g2_start[n2g] = n2e; + if (P != nullptr) { // the PCIe groups: staging slot q, their entries in routing order const uint64_t bb = lay.blob_bytes(d.layers); - for (int q = 0; q < fetches; ++q) { // the PCIe groups: staging slot q, entries after the VRAM ones + int entries = 0; + for (int q = 0; q < fetches; ++q) { const int64_t i0 = pcie_i0[q]; - P.ptr2[q] = P.pcie_mode != 0 ? (unsigned long long) d.src->device_alias(d.layers, ids[i0]) - : P.staging + (unsigned long long) q * (unsigned long long) bb; - P.start2[q] = entries; + P->ptr2[q] = P->pcie_mode != 0 ? (unsigned long long) d.src->device_alias(d.layers, ids[i0]) + : P->staging + (unsigned long long) q * (unsigned long long) bb; + P->start2[q] = entries; for (int64_t i = i0; i < n; ++i) if (first_of[i] == i0) { - P.dst[entries] = (int32_t) i; - P.tok[entries] = (int32_t) (i / k); + P->dst[entries] = (int32_t) i; + P->tok[entries] = (int32_t) (i / k); ++entries; } ++d.pcie_experts; } - P.start2[fetches] = entries; - P.counts[0] = groups; - P.counts[1] = entries; - P.counts[2] = fetches; + P->start2[fetches] = entries; + P->counts[0] = 0; + P->counts[1] = entries; + P->counts[2] = fetches; std::atomic_thread_fence(std::memory_order_seq_cst); pt("publish", fetches); - if (P.publish) P.publish(P.ctx); + if (P->publish) P->publish(P->ctx); pt("fetch", fetches); - if (P.fetch) P.fetch(P.ctx, dma_src, P.pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work - g2_start[n2g] = n2e; - } else { - for (int64_t i = 0; i < n; ++i) { - const int32_t e = ids[i]; - kind[i] = (e >= 0 && e < d.n_expert && d.host_res != nullptr && - d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) ? 0 : -1; - } + if (P->fetch) P->fetch(P->ctx, dma_src, P->pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work } - const bool skip_gpu_rows = d.plan != nullptr && d.plan->host_rows_only && n <= 128 && n <= d.plan->cap; + const bool skip_gpu_rows = d.plan != nullptr && d.plan->host_rows_only; const auto c1 = std::chrono::steady_clock::now(); if (native && lay.fmt[(size_t) d.layers].gu_type == 42) // a native Q2_0 pack: the Q2_0 kernels' activations for (int64_t t = 0; t < n_tok; ++t) act_quant_any(x_f + (size_t) t * H, H, d.act_multi[(size_t) t]); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 743f003667f..c322e7173d1 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -99,12 +99,13 @@ Verifier::~Verifier() { if (cs_) cudaStreamDestroy(cs_); if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (side_) cudaStreamDestroy(side_); - if (fork_) cudaEventDestroy(fork_); - if (join_) cudaEventDestroy(join_); + if (shs_) cudaStreamDestroy(shs_); + for (cudaEvent_t e : {fork_, join_, shfork_, shjoin_, res_ready_}) + if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, - h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_}; + h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_, h_res_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -121,7 +122,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& err = "verify: the window must hold 2.." + std::to_string(strata::kernels::kVerifyMaxT) + " tokens"; return false; } - if (hits.d_res == nullptr || hits.cache_base == nullptr || hits.blob <= 0) { + if (hits.res == nullptr || hits.cache_base == nullptr || hits.blob <= 0 || hits.slots <= 0) { err = "verify: needs the profile-filled VRAM expert tier (--expert-profile and --expert-cache)"; return false; } @@ -174,29 +175,37 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(64, (void**) &h_pleflag_, (void**) &m_pleflag_) && mapped(T * HC * N * 4, (void**) &h_rows_, (void**) &m_rows_); if (!ok) { err = "verify: mapped staging allocation failed"; return false; } - // the GPU plan: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | ptr2(cap u64) | start2(cap+1) + // A plan block, one per token group: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | + // ptr2(cap u64) | start2(cap+1). The main GPU's hits (`verify_hit_plan`, device) fill counts [groups, entries], + // start, dst, tok and ptr; the pool's PCIe share (mapped, `sink_`) counts [-, entries, groups], dst, tok, ptr2 and + // start2. { const int64_t cap = (int64_t) (T * K); - const int64_t i32 = 4 + (cap + 1) + cap + cap; - const int64_t ptr_off = (i32 + 1) & ~1ll; - plan_i32_ = ptr_off + 4 * cap + (cap + 1) + 1; + plan_ptr_off_ = (4 + (cap + 1) + cap + cap + 1) & ~1ll; + plan_i32_ = (plan_ptr_off_ + 4 * cap + (cap + 1) + 2) & ~1ll; + plan_stride_ = plan_i32_ + 16; if (!mapped((size_t) plan_i32_ * 4 * 2 + 64, (void**) &h_plan_, (void**) &m_plan_)) { err = "verify: mapped plan allocation failed"; return false; } sink_.counts = h_plan_; - sink_.start = h_plan_ + 4; - sink_.dst = sink_.start + cap + 1; + sink_.dst = h_plan_ + 4 + cap + 1; sink_.tok = sink_.dst + cap; - sink_.ptr = (unsigned long long*) (h_plan_ + ptr_off); - sink_.ptr2 = sink_.ptr + cap; - sink_.start2 = h_plan_ + ptr_off + 4 * cap; + sink_.ptr2 = (unsigned long long*) (h_plan_ + plan_ptr_off_) + cap; + sink_.start2 = h_plan_ + plan_ptr_off_ + 4 * cap; sink_.cap = cap; sink_.publish = &Verifier::publish_plan; sink_.host_rows_only = strata::kernels::native_moe_combine_enabled(); // the gather-combine (post) sink_.fetch = &Verifier::fetch_dma; sink_.ctx = this; } + // the residency snapshot, rounded up to whole 16-byte copies + res_words_ = (g.n_layers * g.n_expert + 3) & ~3ll; + if (!mapped((size_t) res_words_ * 4, (void**) &h_res_, (void**) &m_res_)) { + err = "verify: mapped residency allocation failed"; + return false; + } + std::memcpy(h_res_, hits.res, (size_t) (g.n_layers * g.n_expert) * sizeof(int32_t)); // ---- the device arena: the same sequence counted, then carved auto carve = [&](Bump& b) { @@ -221,7 +230,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& plogits_ = b.take(T * (uint64_t) g.n_expert); pw_ = b.take(T * K); pids_ = b.take(T * K); shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); - plan_ = b.take(2 * ((uint64_t) plan_i32_ + 16)); + plan_ = b.take(2 * (uint64_t) plan_stride_); dplan_ = b.take(2 * (uint64_t) plan_stride_); + res_ = b.take((uint64_t) res_words_); slot_ptr_ = b.take((uint64_t) hits.slots); staging_ = b.take((uint64_t) kStagingBlobs * strata::kernels::cpu::expert_layout().max_blob); hit_xq_ = b.take(T * (N / 32) * 34); hit_xs_ = b.take(T * (N / 32)); nat_xq_ = b.take(T * (N / 32) * 36); @@ -247,14 +257,28 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sink_.staging = (unsigned long long) staging_; sink_.staging_cap = kStagingBlobs; (void) TS; + { + std::vector sp((size_t) hits.slots); + for (int64_t i = 0; i < hits.slots; ++i) + sp[(size_t) i] = (unsigned long long) (hits.cache_base + (hits.slot_off != nullptr ? hits.slot_off[i] + : (uint64_t) i * (uint64_t) hits.blob)); + if (cudaMemcpy(slot_ptr_, sp.data(), sp.size() * sizeof(unsigned long long), cudaMemcpyHostToDevice) != cudaSuccess) { + err = "verify: the slot table could not be uploaded"; + return false; + } + } if (cudaStreamCreateWithFlags(©_, cudaStreamNonBlocking) != cudaSuccess) { err = "verify: copy stream create failed"; return false; } if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&side_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess) { + cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&shfork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&shjoin_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&res_ready_, cudaEventDisableTiming) != cudaSuccess) { err = "verify: stream create failed"; return false; } @@ -295,16 +319,26 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { const int tb_[2] = {0, (T + 1) / 2}, te_[2] = {G == 2 ? (T + 1) / 2 : T, T}; groups_[T] = G; // --window-profile: row l < n_layers holds layer l's stages, row n_layers the window's own stamps - auto stamp = [&](int64_t row, int i) { + auto stamp = [&](int64_t row, int i, cudaStream_t st) { if (stamps_ != nullptr && G == 1) - gpu_stamp(stamps_ + ((size_t) T * (size_t) (g.n_layers + 1) + (size_t) row) * kStamps + i, cs); + gpu_stamp(stamps_ + ((size_t) T * (size_t) (g.n_layers + 1) + (size_t) row) * kStamps + i, st); }; - stamp(g.n_layers, 0); + stamp(g.n_layers, 0, cs); - // ---- the window's inputs, from mapped staging + // ---- the window's inputs, from mapped staging; the residency snapshot on the shared expert's branch (the first + // hit plan waits for it) copy_i32_from_mapped(tok_, m_tok_, T, cs); copy_i32_from_mapped(step_, m_step_, (int64_t) T * kStepCount, cs); copy_i32_from_mapped(pos_, m_pos_, (int64_t) T * NH, cs); + if (cudaEventRecord(shfork_, cs) != cudaSuccess || cudaStreamWaitEvent(shs_, shfork_, 0) != cudaSuccess) { + err = "verify: the residency copy could not fork"; + return false; + } + copy_from_mapped((float*) res_, (const float*) m_res_, res_words_, shs_); // int32 words, copied as they are + if (cudaEventRecord(res_ready_, shs_) != cudaSuccess) { + err = "verify: the residency copy could not join"; + return false; + } // ---- the embeddings, broadcast to the hc streams if (const NativeEmbed* ne = native_embed()) { // plan v0.3 P6: the GGUF-form table @@ -342,10 +376,55 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } } + // the shared expert: in a window of one token group on a branch of its own that forks at the doorbell, beside the + // routed experts (joined before the combine); in a split window inline + const bool beside = G == 1; + auto shared = [&](int64_t l, int grp) -> bool { + const int tb = tb_[grp], te = te_[grp], n = te - tb; + const LayerView v(wt, l); + const WeightRef *wgi = need(v, "ffn_gate_inp_shexp.weight", err), *wsg = need(v, "ffn_gate_shexp.weight", err), + *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); + if (!wgi || !wsg || !wsu || !wsd) return false; + if (!native_of(wsg, v.name("ffn_gate_shexp.weight"), err) || !native_of(wsu, v.name("ffn_up_shexp.weight"), err) || + !native_of(wsd, v.name("ffn_down_shexp.weight"), err)) + return false; + NativeSharedWeights nsw; + nsw.gate_type = wsg->native_type; nsw.gate_data = wsg->native_data; + nsw.up_type = wsu->native_type; nsw.up_data = wsu->native_data; + nsw.down_type = wsd->native_type; nsw.down_data = wsd->native_data; + nsw.q8_1 = xq_; // the main stream's next use is the next layer's mixer + cudaStream_t ws = cs; + if (beside) { + if (cudaEventRecord(shfork_, cs) != cudaSuccess || cudaStreamWaitEvent(shs_, shfork_, 0) != cudaSuccess) { + err = "verify: the shared expert's branch could not fork"; + return false; + } + ws = shs_; + } + if (!shared_expert_native_bf16()) // the scalar gate's BF16 input (the native gate reads x) + for (int t = tb; t < te; ++t) f32_to_bf16_bulk(mixed_ + t * N, sh_bf16_ + t * N, N, ws); + try { + shared_expert_multi(n, mixed_ + tb * N, sh_bf16_ + tb * N, nsw, (const uint16_t*) wgi->data, + sh_gate_ + (size_t) tb * g.n_ff, sh_up_ + (size_t) tb * g.n_ff, sh_g_ + tb, shared_ + tb * N, + N, g.n_ff, ws); + } catch (const std::exception& e) { + err = std::string("verify shared expert: ") + e.what(); + return false; + } + if (beside) { + stamp(l, kLayerStamps, shs_); + if (cudaEventRecord(shjoin_, shs_) != cudaSuccess) { + err = "verify: the shared expert's branch could not join"; + return false; + } + } + return true; + }; + // ---------------------------------------------------------------- pre(l, group): up to the ring auto pre = [&](int64_t l, int grp) -> bool { const int tb = tb_[grp], te = te_[grp], n = te - tb; - stamp(l, 0); + stamp(l, 0, cs); const LayerView v(wt, l); const char* pfx[2] = {"hc_attn_", "hc_ffn_"}; const WeightRef *wn[2], *wd[2], *wu[2], *wi[2]; @@ -393,7 +472,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { fused_gr_read_multi(fa, n, xn_ + (size_t) tb * HC * N, cs); }; gr_read_group(0, pending, inj2_, inj_); - stamp(l, 1); + stamp(l, 1, cs); float* xm = mixed_ + tb * N; try { if (!is_qsa_layer(g, l)) { @@ -507,9 +586,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { err = "verify layer " + std::to_string(l) + ": " + e.what(); return false; } - stamp(l, 2); + stamp(l, 2, cs); gr_read_group(1, true, inj_, inj2_); - stamp(l, 3); + stamp(l, 3, cs); if (native_router_enabled()) { // one weight read and one launch each for the window's tokens const WeightRef* wr = need(v, "ffn_gate_inp.weight", err); if (!wr) return false; @@ -527,10 +606,25 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false; } } - stamp(l, 4); + stamp(l, 4, cs); doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, m_ids_ + tb * K, m_w_ + tb * K, m_seq_, cs); - stamp(l, 5); + stamp(l, 5, cs); + // after the doorbell: the shared expert's branch first (the longest), the main GPU's routed experts - the ones + // its VRAM tier holds, decided here from the residency snapshot the pool reads too (it leaves their rows to + // the GPU) - and once they are planned, the next layer's prediction + if (beside && !shared(l, grp)) return false; + if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { + err = "verify: the residency copy could not join"; + return false; + } + verify_hit_plan(ids_ + tb * K, (int) (n * K), (int) K, res_ + (size_t) l * NE, (int) NE, slot_ptr_, + dplan_ + (size_t) grp * (size_t) plan_stride_, MT * K, plan_ptr_off_, cs); + if (strata::kernels::cpu::expert_layout().native) + quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, cs); + else + quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, cs); + stamp(l, 6, cs); if (predicts(l)) { const WeightRef* wr1 = need(LayerView(wt, l + 1), "ffn_gate_inp.weight", err); if (!wr1) return false; @@ -553,112 +647,105 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } } - { - const WeightRef *wgi = need(v, "ffn_gate_inp_shexp.weight", err), *wsg = need(v, "ffn_gate_shexp.weight", err), - *wsu = need(v, "ffn_up_shexp.weight", err), *wsd = need(v, "ffn_down_shexp.weight", err); - if (!wgi || !wsg || !wsu || !wsd) return false; - if (!native_of(wsg, v.name("ffn_gate_shexp.weight"), err) || !native_of(wsu, v.name("ffn_up_shexp.weight"), err) || - !native_of(wsd, v.name("ffn_down_shexp.weight"), err)) - return false; - NativeSharedWeights nsw; - nsw.gate_type = wsg->native_type; nsw.gate_data = wsg->native_data; - nsw.up_type = wsu->native_type; nsw.up_data = wsu->native_data; - nsw.down_type = wsd->native_type; nsw.down_data = wsd->native_data; - nsw.q8_1 = xq_; - for (int t = tb; t < te; ++t) f32_to_bf16_bulk(mixed_ + t * N, sh_bf16_ + t * N, N, cs); - try { - shared_expert_multi(n, xm, sh_bf16_ + tb * N, nsw, (const uint16_t*) wgi->data, sh_gate_ + (size_t) tb * g.n_ff, - sh_up_ + (size_t) tb * g.n_ff, sh_g_ + tb, shared_ + tb * N, N, g.n_ff, cs); - } catch (const std::exception& e) { - err = std::string("verify shared expert: ") + e.what(); - return false; - } - } - if (strata::kernels::cpu::expert_layout().native) - quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, cs); - else - quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, cs); - stamp(l, 6); - return true; + return beside || shared(l, grp); }; // ---------------------------------------------------------------- post(l, group): experts, combine auto post = [&](int64_t l, int grp) -> bool { const int tb = tb_[grp], te = te_[grp], n = te - tb; const uint32_t ring = (uint32_t) (l * G + grp + 1); - wait_flag_ge(m_flagA_, ring, cs); // the pool published this group's GPU plan - stamp(l, 7); const int64_t cap = (int64_t) n * K, capx = (int64_t) max_t_ * K; - int32_t* pl = plan_ + (size_t) grp * (size_t) (plan_i32_ + 16); - copy_i32_from_mapped(pl, m_plan_ + (size_t) grp * (size_t) plan_i32_, plan_i32_, cs); - const int32_t* p_counts = pl; - const int32_t* p_start = pl + 4; - const int32_t* p_dst = p_start + capx + 1; - const int32_t* p_tok = p_dst + capx; - const int64_t ptr_off = ((4 + (capx + 1) + 2 * capx) + 1) & ~1ll; - const unsigned long long* p_ptr = (const unsigned long long*) (pl + ptr_off); - const unsigned long long* p_ptr2 = p_ptr + capx; - const int32_t* p_start2 = pl + ptr_off + 4 * capx; + struct Plan { // a plan block's fields (`init`) + const int32_t *counts = nullptr, *start = nullptr, *dst = nullptr, *tok = nullptr, *start2 = nullptr; + const unsigned long long *ptr = nullptr, *ptr2 = nullptr; + }; + auto plan_at = [&](const int32_t* pl) { + Plan p; + p.counts = pl; + p.start = pl + 4; + p.dst = p.start + capx + 1; + p.tok = p.dst + capx; + p.ptr = (const unsigned long long*) (pl + plan_ptr_off_); + p.ptr2 = p.ptr + capx; + p.start2 = pl + plan_ptr_off_ + 4 * capx; + return p; + }; float* hit_out = hit_out_ + (size_t) tb * K * N; const auto& lay = strata::kernels::cpu::expert_layout(); - // plan v0.3 P6: the VRAM groups now; the PCIe groups once the copy engine has landed them in staging - auto grouped = [&](const unsigned long long* gp, const int32_t* gs, const int32_t* gn) { + auto grouped = [&](const Plan& p, const unsigned long long* gp, const int32_t* gs, const int32_t* gn) { if (lay.native) { // the layer's GGUF formats (i-quant gate/up, Q2_0 / IQ4_NL down) const auto& f = lay.fmt[(size_t) l]; const NativeExpertLayout L = native_expert_layout(f.gu_type, f.d_type, f.n_embd, f.n_ff); - native_expert_grouped(L, gp, gs, gn, p_dst, p_tok, cap, cap, + native_expert_grouped(L, gp, gs, gn, p.dst, p.tok, cap, cap, nat_xq_ + (size_t) tb * (N / 32) * 36, hit_scratch_, hit_out, cs); } else { - moe_grouped_s2(gp, gs, gn, p_dst, p_tok, cap, cap, hit_xq_ + (size_t) tb * (N / 32) * 34, + moe_grouped_s2(gp, gs, gn, p.dst, p.tok, cap, cap, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), hit_scratch_, hit_out, cs); } }; - grouped(p_ptr, p_start, p_counts); - stamp(l, 8); + // plan v0.3 P6: the VRAM groups at once; the PCIe groups once the pool has published them and the copy engine + // has landed them in staging + const Plan vram = plan_at(dplan_ + (size_t) grp * (size_t) plan_stride_); + grouped(vram, vram.ptr, vram.start, vram.counts); + stamp(l, 7, cs); + Plan pcie; if (pcie_share_) { - wait_flag_ge(m_flagB_, ring, cs); // the PCIe share is in staging (DMA) or mapped + wait_flag_ge(m_flagA_, ring, cs); // the pool published this group's PCIe share + stamp(l, 8, cs); + int32_t* pl = plan_ + (size_t) grp * (size_t) plan_stride_; + copy_i32_from_mapped(pl, m_plan_ + (size_t) grp * (size_t) plan_i32_, plan_i32_, cs); + pcie = plan_at(pl); + wait_flag_ge(m_flagB_, ring, cs); // it is in staging (DMA) or mapped if (sink_.pcie_mode == 2) { // stage it with a copy kernel, then point at staging const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; uint8_t* stage = staging_ + (size_t) (grp * per) * lay.max_blob; - fetch_blobs(p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs); - rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs); + fetch_blobs(pcie.ptr2, pcie.counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs); + rebase_ptrs((unsigned long long*) pcie.ptr2, pcie.counts + 2, stage, (int64_t) lay.blob_bytes(l), cs); } - grouped(p_ptr2, p_start2, p_counts + 2); + grouped(pcie, pcie.ptr2, pcie.start2, pcie.counts + 2); + } else { + stamp(l, 8, cs); } - stamp(l, 9); + stamp(l, 9, cs); wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows - stamp(l, 10); + stamp(l, 10, cs); + if (beside && cudaStreamWaitEvent(cs, shjoin_, 0) != cudaSuccess) { + err = "verify: the shared expert's branch could not join"; + return false; + } if (native_moe_combine_enabled()) { // only the host's rows cross PCIe; the GPU's come from `hit_out` (the pool leaves their rows unwritten) try { - native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, p_dst, p_counts + 1, w_ + tb * K, - shared_ + tb * N, bo_ + tb * N, N, K, n, cs); + native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, vram.dst, vram.counts + 1, pcie.dst, + pcie.counts != nullptr ? pcie.counts + 1 : nullptr, w_ + tb * K, shared_ + tb * N, + bo_ + tb * N, N, K, n, cs); } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + " combine: " + e.what(); return false; } } else { copy_from_mapped(parts_ + (size_t) tb * K * N, m_ymiss_ + (size_t) tb * K * N, (int64_t) n * K * N, cs); - moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, p_dst, p_counts + 1, cap, N, cs); + moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, vram.dst, vram.counts + 1, cap, N, cs); + if (pcie_share_) moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, pcie.dst, pcie.counts + 1, cap, N, cs); for (int t = tb; t < te; ++t) { MoEBuffers mb = ss.moe; mb.weights = w_ + t * K; mb.shared = shared_ + t * N; if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; } } - stamp(l, 11); + stamp(l, 11, cs); if (predicts(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { err = "verify: the prediction branch could not join"; return false; } if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); - stamp(l, 12); + stamp(l, 12, cs); return true; }; - stamp(g.n_layers, 1); + stamp(g.n_layers, 1, cs); for (int grp = 0; grp < G; ++grp) if (!pre(0, grp)) return false; for (int64_t l = 0; l < g.n_layers; ++l) @@ -666,7 +753,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!post(l, grp)) return false; if (l + 1 < g.n_layers && !pre(l + 1, grp)) return false; } - stamp(g.n_layers, 2); + stamp(g.n_layers, 2, cs); // ---- the head, T columns, and the argmax of each { @@ -698,8 +785,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { sample_tokens(head_logits_, T, (int) n_vocab_, nullptr, 0, sp, m_out_, cs); } if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // a plain copy kernel, here to host - stamp(g.n_layers, 3); - stamp(g.n_layers, 4); // right after the last: one stamp's own cost + stamp(g.n_layers, 3, cs); + stamp(g.n_layers, 4, cs); // right after the last: one stamp's own cost return true; } @@ -832,6 +919,8 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, prev[1] = tokens[t]; } } + // the residency this window decides the main GPU's hits by, and the pool reads (`residency`) + std::memcpy(h_res_, hits_.res, (size_t) (g.n_layers * g.n_expert) * sizeof(int32_t)); *(volatile uint32_t*) h_seq_ = 0; *(volatile uint32_t*) h_flag_ = 0; *(volatile uint32_t*) h_flagA_ = 0; @@ -895,11 +984,10 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, VDBG("layer %lld served\n", (long long) l); std::atomic_thread_fence(std::memory_order_seq_cst); _mm_sfence(); - if (*(volatile uint32_t*) h_flagA_ != want) { // the pool did not publish a plan: an empty one + if (pcie_share_ && *(volatile uint32_t*) h_flagA_ != want) { // no PCIe share published: an empty one sink_.counts[0] = 0; sink_.counts[1] = 0; sink_.counts[2] = 0; - sink_.start[0] = 0; sink_.start2[0] = 0; std::atomic_thread_fence(std::memory_order_seq_cst); *(volatile uint32_t*) h_flagA_ = want; @@ -939,9 +1027,11 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, for (int64_t l = 0; l < NL; ++l) { const unsigned long long* st = h_stamps_.data() + (size_t) l * kStamps; const int kind = is_qsa_layer(g, l) ? 1 : 0; - const unsigned long long before = l == 0 ? w[1] : (st - kStamps)[kStamps - 1]; + const unsigned long long before = l == 0 ? w[1] : (st - kStamps)[kLayerStamps - 1]; prof_ns_[(size_t) kind * kStamps] += (double) (st[0] - before); - for (int i = 1; i < kStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); + for (int i = 1; i < kLayerStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); + // the shared expert's branch, from the doorbell (it forks there) to its end + prof_ns_[(size_t) kind * kStamps + kLayerStamps] += (double) (st[kLayerStamps] - st[5]); ++prof_layers_[kind]; } prof_window_ns_[0] += (double) (w[1] - w[0]); @@ -957,13 +1047,14 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, void Verifier::print_profile() const { if (prof_windows_ == 0) return; - static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", - "router", "doorbell", "shared expert", "wait for plan", - "VRAM experts", "PCIe experts", "wait for CPU", "rows + combine", - "last HC write"}; + static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", + "router", "doorbell", "hit plan", "VRAM experts", + "wait for PCIe plan", "PCIe experts", "wait for CPU", "rows + combine", + "last HC write", "shared expert, beside"}; const double nw = (double) prof_windows_; std::printf("%-24s GPU time over %lld windows, ms per window: inputs %.3f, head %.3f, whole window %.3f; " - "one stamp %.2f us (each stage below includes one)\n", "window profile", (long long) prof_windows_, + "one stamp %.2f us (each stage below includes one; the last runs from the doorbell beside the " + "four before \"rows + combine\")\n", "window profile", (long long) prof_windows_, prof_window_ns_[0] / nw / 1e6, prof_window_ns_[1] / nw / 1e6, prof_window_ns_[2] / nw / 1e6, prof_window_ns_[3] / nw / 1e3); std::printf(" %-22s %12s %12s %14s\n", "stage", "GDN us/layer", "QSA us/layer", "ms per window"); @@ -989,15 +1080,11 @@ void Verifier::ple_ahead(int t, int32_t token) { void Verifier::set_plan_slot(int grp) { const int64_t cap = sink_.cap; int32_t* base = h_plan_ + (size_t) grp * (size_t) plan_i32_; - const int64_t i32 = 4 + (cap + 1) + cap + cap; - const int64_t ptr_off = (i32 + 1) & ~1ll; sink_.counts = base; - sink_.start = base + 4; - sink_.dst = sink_.start + cap + 1; + sink_.dst = base + 4 + cap + 1; sink_.tok = sink_.dst + cap; - sink_.ptr = (unsigned long long*) (base + ptr_off); - sink_.ptr2 = sink_.ptr + cap; - sink_.start2 = base + ptr_off + 4 * cap; + sink_.ptr2 = (unsigned long long*) (base + plan_ptr_off_) + cap; + sink_.start2 = base + plan_ptr_off_ + 4 * cap; const int G = groups_[last_t_] > 0 ? groups_[last_t_] : 1; const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; sink_.staging = (unsigned long long) (staging_ + (size_t) (grp * per) * strata::kernels::cpu::expert_layout().max_blob); diff --git a/src/kernels/cuda/native_moe.cu b/src/kernels/cuda/native_moe.cu index ebc912d97c3..f9144ba9a29 100644 --- a/src/kernels/cuda/native_moe.cu +++ b/src/kernels/cuda/native_moe.cu @@ -48,15 +48,16 @@ constexpr int kMaxK = 15; // loads its k values (the host's over PCIe, all in flight at once) and sums them as `combine` does. __global__ void gather_combine(const float* __restrict__ gpu_rows, const float* host_rows, const int32_t* __restrict__ dst, const int32_t* __restrict__ count, + const int32_t* __restrict__ dst2, const int32_t* __restrict__ count2, const float* __restrict__ weights, const float* __restrict__ shared, float* __restrict__ output, int64_t n_embd, int k) { __shared__ unsigned on_gpu; // bit j: row t*k + j is the GPU's const int t = blockIdx.y; if (threadIdx.x == 0) on_gpu = 0u; __syncthreads(); - const int c = *count; - for (int i = threadIdx.x; i < c; i += blockDim.x) { - const int r = dst[i] - t * k; + const int c = *count, c2 = dst2 != nullptr ? *count2 : 0; + for (int i = threadIdx.x; i < c + c2; i += blockDim.x) { + const int r = (i < c ? dst[i] : dst2[i - c]) - t * k; if (r >= 0 && r < k) atomicOr(&on_gpu, 1u << r); } __syncthreads(); @@ -114,15 +115,15 @@ void native_moe_combine(const float* parts, const float* weights, const float* s if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, - const int32_t* count, const float* weights, const float* shared, float* output, - int64_t n_embd, int64_t k, int n_tok, void* stream) { + const int32_t* count, const int32_t* dst2, const int32_t* count2, const float* weights, + const float* shared, float* output, int64_t n_embd, int64_t k, int n_tok, void* stream) { if (!stream || n_embd <= 0 || n_embd > std::numeric_limits::max() || k < 1 || k > kMaxK || n_tok < 1 || - !gpu_rows || !host_rows || !dst || !count || !weights || !output) + !gpu_rows || !host_rows || !dst || !count || (dst2 != nullptr && count2 == nullptr) || !weights || !output) throw std::invalid_argument("native MoE gather-combine requires a stream, positive width, 1..15 experts " "and its buffers"); const dim3 grid(unsigned((n_embd + 255) / 256), unsigned(n_tok)); - gather_combine<<(stream)>>>(gpu_rows, host_rows, dst, count, weights, - shared, output, n_embd, int(k)); + gather_combine<<(stream)>>>(gpu_rows, host_rows, dst, count, dst2, count2, + weights, shared, output, n_embd, int(k)); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/cuda/shared_expert.cu b/src/kernels/cuda/shared_expert.cu index 52ce6c044e8..d911a45fb15 100644 --- a/src/kernels/cuda/shared_expert.cu +++ b/src/kernels/cuda/shared_expert.cu @@ -150,6 +150,7 @@ __global__ void moe_combine_kernel(const float* __restrict__ parts, const float* } // namespace void shared_expert_set_native_bf16(bool enabled) { native_bf16 = enabled; } +bool shared_expert_native_bf16() { return native_bf16; } namespace { __global__ void scale_rows_kernel(float* __restrict__ out, const float* __restrict__ g, int n) { diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index 2eaa0c07548..2a303ff6e05 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -229,6 +229,61 @@ __global__ void rebase_ptrs_kernel(unsigned long long* ptr, const int32_t* n, un if (k < *n) ptr[k] = base + (unsigned long long) k * (unsigned long long) bytes; } +// Thread i = routed entry i. A group per distinct resident expert, in the order of their first entries; a group's +// entries in routing order. +constexpr int kHitPlanMax = 128; +__global__ void hit_plan_kernel(const int32_t* __restrict__ ids, int n, int k, const int32_t* __restrict__ res, + int n_expert, const unsigned long long* __restrict__ slot_ptr, int32_t* __restrict__ plan, + int cap, int ptr_off) { + __shared__ int32_t s_id[kHitPlanMax], s_size[kHitPlanMax], s_start[kHitPlanMax]; + const int i = threadIdx.x; + int32_t e = -1, slot = -1; + if (i < n) { + e = ids[i]; + if (e >= 0 && e < n_expert) slot = res[e]; + } + s_id[i] = e; + __syncthreads(); + const bool hit = slot >= 0; + int first = i; + if (hit) + for (int j = 0; j < i; ++j) + if (s_id[j] == e) { first = j; break; } + const bool lead = hit && first == i; + int size = 0; + if (lead) + for (int j = i; j < n; ++j) size += s_id[j] == e; + s_size[i] = size; // > 0 at a group's first entry only + const int groups = __syncthreads_count(lead); + const int entries = __syncthreads_count(hit); + int32_t* start = plan + 4; + int32_t* dst = start + cap + 1; + int32_t* tok = dst + cap; + auto* ptr = (unsigned long long*) (plan + ptr_off); + if (lead) { + int grp = 0, at = 0; + for (int j = 0; j < i; ++j) + if (s_size[j] > 0) { ++grp; at += s_size[j]; } + s_start[i] = at; + start[grp] = at; + ptr[grp] = slot_ptr[slot]; + } + __syncthreads(); + if (hit) { + int rank = 0; + for (int j = first; j < i; ++j) rank += s_id[j] == e; + dst[s_start[first] + rank] = i; + tok[s_start[first] + rank] = i / k; + } + if (i == 0) { + plan[0] = groups; + plan[1] = entries; + plan[2] = 0; + plan[3] = 0; + start[groups] = entries; + } +} + __global__ void add_streams_broadcast_kernel(const float* __restrict__ h, const float* __restrict__ e, float* __restrict__ R, int64_t n, int hc) { const int t = blockIdx.y; @@ -321,6 +376,17 @@ void add_streams_broadcast(const float* h, const float* e, float* R, int64_t n_e check("add_streams_broadcast"); } +void verify_hit_plan(const int32_t* ids, int n, int k, const int32_t* res, int n_expert, + const unsigned long long* slot_ptr, int32_t* plan, int64_t cap, int64_t ptr_off, void* stream) { + if (n < 1 || n > kHitPlanMax || n > cap || k < 1 || (ptr_off & 1) != 0) { + std::fprintf(stderr, "verify_hit_plan: invalid arguments\n"); + std::exit(1); + } + hit_plan_kernel<<<1, kHitPlanMax, 0, (cudaStream_t) stream>>>(ids, n, k, res, n_expert, slot_ptr, plan, (int) cap, + (int) ptr_off); + check("verify_hit_plan"); +} + void ident_hits(const int32_t* ids, int n, int32_t* slot, int32_t* dst, int32_t* count, void* stream) { if (n < 1 || n > 1024) { std::fprintf(stderr, "ident_hits: n out of range\n"); std::exit(1); } ident_hits_kernel<<<1, 1024, 0, (cudaStream_t) stream>>>(ids, n, slot, dst, count); diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 2560bdf5c15..9018d250b21 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -107,12 +107,14 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { float *d_hit = dev(hit.size()), *d_parts = dev(hit.size()), *d_w = dev(w.size()), *d_sh = dev(shared.size()), *d_out_old = dev(shared.size()), *d_out_new = dev(shared.size()); - int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(1); + int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(3); up(d_hit, hit); up(d_w, w); up(d_sh, shared); if (!dst.empty()) up(d_dst, dst); - up(d_count, std::vector{count}); + // the new path takes them as two lists (the window's VRAM share and the pool's PCIe share), split at random + const int32_t split = rep % 2 == 0 ? count : (int32_t) (rng() % (dst.size() + 1)); + up(d_count, std::vector{count, split, count - split}); float* h_map = nullptr; float* m_map = nullptr; check(cudaHostAlloc((void**) &h_map, host.size() * sizeof(float), cudaHostAllocMapped), "cudaHostAlloc"); @@ -128,7 +130,8 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { check(cudaStreamSynchronize(s), "old path"); // new std::memcpy(h_map, host_poison.data(), host.size() * sizeof(float)); - strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count, d_w, d_sh, d_out_new, N, K, n_tok, s); + strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count + 1, rep % 2 == 0 ? nullptr : d_dst + split, + d_count + 2, d_w, d_sh, d_out_new, N, K, n_tok, s); check(cudaStreamSynchronize(s), "new path"); const int b = bitwise_diff(down(d_out_old, shared.size()), down(d_out_new, shared.size()), "gather_combine"); if (b) std::fprintf(stderr, "gather_combine: n_tok %d rep %d: %d of %zu differ\n", n_tok, rep, b, shared.size()); @@ -139,7 +142,80 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { cudaFree(p); } } - std::printf("gather_combine: %s\n", bad ? "MISMATCH" : "bitwise equal (n_tok 1-4, 8; 8 plans each)"); + std::printf("gather_combine: %s\n", bad ? "MISMATCH" : "bitwise equal (n_tok 1-4, 8; 8 plans each, one or two lists)"); + return bad; +} + +// ---- the main GPU's hit plan decided on the device vs the host's rule: distinct experts in routing order, the +// resident ones as groups of their entries in routing order +int test_hit_plan(std::mt19937& rng, cudaStream_t s) { + const int NE = 512, K = 10, slots = 300; + const int64_t cap = 8 * K, ptr_off = ((4 + (cap + 1) + 2 * cap) + 1) & ~1ll, words = ptr_off + 2 * cap; + std::vector slot_ptr(slots); + for (int i = 0; i < slots; ++i) slot_ptr[(size_t) i] = 0x700000000ull + (unsigned long long) i * 1382400ull; + unsigned long long* d_sp = dev(slots); + up(d_sp, slot_ptr); + int32_t *d_ids = dev(cap), *d_res = dev(NE), *d_plan = dev(words); + int bad = 0; + for (int n_tok : {1, 2, 3, 4, 8}) { + for (int rep = 0; rep < 24; ++rep) { + const int n = n_tok * K; + // each token's k experts distinct (rep 3: repeats within a token too), drawn from a small set so tokens share + std::vector set((size_t) (4 + rng() % 40)), ids((size_t) n), res((size_t) NE, -1); + for (auto& e : set) e = (int32_t) (rng() % NE); + for (int t = 0; t < n_tok; ++t) + for (int j = 0; j < K; ++j) { + int32_t e; + bool again; + do { + e = rng() % 4 == 0 ? (int32_t) (rng() % NE) : set[rng() % set.size()]; + again = false; + for (int m = 0; m < j && rep != 3; ++m) again = again || ids[(size_t) (t * K + m)] == e; + } while (again); + ids[(size_t) (t * K + j)] = e; + } + for (int e = 0; e < NE; ++e) // resident: none, all, or at random + if (rep == 1 || (rep != 0 && rng() % 3 != 0)) res[(size_t) e] = (int32_t) (rng() % slots); + std::vector first((size_t) n), start, dst, tok; + std::vector ptr; + for (int i = 0; i < n; ++i) { + first[(size_t) i] = i; + for (int j = 0; j < i; ++j) + if (ids[(size_t) j] == ids[(size_t) i]) { first[(size_t) i] = j; break; } + } + for (int i = 0; i < n; ++i) { + if (first[(size_t) i] != i || res[(size_t) ids[(size_t) i]] < 0) continue; + ptr.push_back(slot_ptr[(size_t) res[(size_t) ids[(size_t) i]]]); + start.push_back((int32_t) dst.size()); + for (int j = i; j < n; ++j) + if (first[(size_t) j] == i) { dst.push_back(j); tok.push_back(j / K); } + } + start.push_back((int32_t) dst.size()); + up(d_ids, ids); + up(d_res, res); + check(cudaMemset(d_plan, 0xff, (size_t) words * 4), "memset"); + strata::kernels::verify_hit_plan(d_ids, n, K, d_res, NE, d_sp, d_plan, cap, ptr_off, s); + check(cudaStreamSynchronize(s), "hit plan"); + const std::vector pl = down(d_plan, (size_t) words); + const int groups = (int) ptr.size(), entries = (int) dst.size(); + bool ok = pl[0] == groups && pl[1] == entries; + for (int q = 0; ok && q <= groups; ++q) ok = pl[(size_t) (4 + q)] == start[(size_t) q]; + for (int q = 0; ok && q < groups; ++q) { + unsigned long long p; + std::memcpy(&p, &pl[(size_t) (ptr_off + 2 * q)], 8); + ok = p == ptr[(size_t) q]; + } + for (int q = 0; ok && q < entries; ++q) + ok = pl[(size_t) (4 + cap + 1 + q)] == dst[(size_t) q] && pl[(size_t) (4 + 2 * cap + 1 + q)] == tok[(size_t) q]; + if (!ok) { + std::fprintf(stderr, "hit_plan: n_tok %d rep %d: groups %d/%d entries %d/%d\n", n_tok, rep, pl[0], groups, + pl[1], entries); + ++bad; + } + } + } + for (void* p : {(void*) d_sp, (void*) d_ids, (void*) d_res, (void*) d_plan}) cudaFree(p); + std::printf("hit_plan: %s\n", bad ? "MISMATCH" : "the host's groups (n_tok 1-4, 8; 24 routings each)"); return bad; } @@ -681,6 +757,7 @@ int main(int argc, char** argv) { check(cudaStreamCreate(&s), "stream"); int bad = 0; bad += test_gather_combine(rng, s); + bad += test_hit_plan(rng, s); bad += test_mmvf_multi(rng, s); bad += test_router_multi(rng, s); bad += test_gr_read(rng, s); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 8b656dba111..d7ab6419eb6 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2265,8 +2265,10 @@ int main(int argc, char** argv) { mem_mark("the head and the prompt path"); strata::core::Verifier ver; strata::core::VerifyHits vh; - vh.d_res = thits.d_res; + vh.res = host_res.data(); vh.cache_base = thits.cache_base; + vh.slot_off = xcache.slot_offsets(); + vh.slots = xcache.slots(); vh.blob = thits.blob; if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.spec, err) || !mtp.bind(wt, &native_head, ver.final_R_all(), err)) { @@ -2281,6 +2283,7 @@ int main(int argc, char** argv) { if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); prefill.draft = &mtp; drive.d.plan = ver.plan_sink(); + drive.d.host_res = ver.residency(); // the snapshot the windows decide their GPU hits by drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5))); ver.set_pcie_share(drive.d.pcie_num > 0); if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); @@ -2963,8 +2966,10 @@ int main(int argc, char** argv) { mem_mark("the head and the prompt path"); strata::core::Verifier ver; strata::core::VerifyHits vh; - vh.d_res = thits.d_res; + vh.res = host_res.data(); vh.cache_base = thits.cache_base; + vh.slot_off = xcache.slot_offsets(); + vh.slots = xcache.slots(); vh.blob = thits.blob; if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.spec, err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); @@ -2983,6 +2988,7 @@ int main(int argc, char** argv) { : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); drive.d.plan = ver.plan_sink(); + drive.d.host_res = ver.residency(); // the snapshot the windows decide their GPU hits by drive.d.pcie_num = (int) (o.pcie_frac * 256.0 + 0.5); if (drive.d.pcie_num < 0) drive.d.pcie_num = 0; if (drive.d.pcie_num > 256) drive.d.pcie_num = 256; @@ -3195,6 +3201,7 @@ int main(int argc, char** argv) { if (use_mtp && rounds > 0) std::printf("%-24s %.3f ms/round drafting (%lld rounds), %.0f MiB of VRAM\n", "mtp", mtp.ms_draft / (double) mtp.rounds, (long long) mtp.rounds, (double) mtp.vram_bytes() / 1048576.0); + drive.d.host_res = host_res.data(); // the verifier's snapshot goes with it } if (dump != nullptr && std::fclose(dump) != 0) { From cf307ea616073bac53a1dbe459c4111182719fd6 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 19:34:41 +0200 Subject: [PATCH 41/92] Grouped native experts in two kernels: gate/up with SwiGLU and q8_1, then down native_expert_grouped (the 3090's cached experts in verify windows, the 5070 Ti's share) ran four kernels: one warp per gate/up row, SwiGLU, the q8_1 quantization of h, one warp per down row. Now a block computes one group's gate and up rows for a 32-row chunk of n_ff (two rows a warp), SwiGLU and that chunk's q8_1 block, and a warp writes four down rows; every value's sums and their order are the old ones, so the results are bitwise equal (expert_grouped_parity keeps the old kernels: all 15 gate/up and 4 down formats, plans of 1-8 tokens). The K-quants' gate/up get one block per SM, without which they spill. The format dot products move to iq_dot.cuh, shared with the test. 3-21% faster per call on the two model files' formats (--bench). Fixed text, both GPUs, after the previous commit: IQ3_XXS code 108.9 -> 111.5 t/s, thinking 93.2 -> 96.2; UD-Q4_K_XL code 88.0 -> 89.4, thinking 77.5 -> 78.3. Window hashes unchanged. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- CMakeLists.txt | 5 + src/kernels/cuda/iq_dot.cuh | 550 +++++++++++++++++++++ src/kernels/cuda/iq_kernels.cu | 700 ++++----------------------- src/kernels/expert_grouped_parity.cu | 354 ++++++++++++++ 4 files changed, 1015 insertions(+), 594 deletions(-) create mode 100644 src/kernels/cuda/iq_dot.cuh create mode 100644 src/kernels/expert_grouped_parity.cu diff --git a/CMakeLists.txt b/CMakeLists.txt index 33b9eddecb5..2249b9cbde3 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -516,6 +516,11 @@ if(STRATA_NATIVE_EXPERTS) add_executable(native_expert_parity src/kernels/native_expert_parity.cpp) target_include_directories(native_expert_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) target_link_libraries(native_expert_parity PRIVATE strata_kernels strata_kernels_cpu strata_artifact CUDA::cudart) + # The grouped native expert kernels against the ones they replaced, kept in the test: bitwise (synthetic experts). + add_executable(expert_grouped_parity src/kernels/expert_grouped_parity.cu) + target_include_directories(expert_grouped_parity PRIVATE third_party/ggml src/kernels/cuda) + target_link_libraries(expert_grouped_parity PRIVATE strata_kernels strata_kernels_cpu CUDA::cudart) + add_test(NAME expert_grouped_parity COMMAND expert_grouped_parity) endif() endif() diff --git a/src/kernels/cuda/iq_dot.cuh b/src/kernels/cuda/iq_dot.cuh new file mode 100644 index 00000000000..43ef02a87f2 --- /dev/null +++ b/src/kernels/cuda/iq_dot.cuh @@ -0,0 +1,550 @@ +// src/kernels/cuda/iq_dot.cuh - the GGUF formats' dot products against q8_1 activations (one call covers a slice of a +// block), for iq_kernels.cu and the parity test that keeps its earlier expert kernels. +// +// Transcribed from llama.cpp (ggml/src/ggml-cuda/vecdotq.cuh at the commit in third_party/ggml/VERSION.txt; MIT +// license, third_party/ggml/LICENSE). The block structs and codebook grids come from its ggml-common.h, included +// unchanged. +#pragma once + +#include +#include + +#define GGML_COMMON_DECL_CUDA +#define GGML_COMMON_IMPL_CUDA +#include "ggml-common.h" + +namespace strata::kernels { +namespace { + +// ---------------------------------------------------------------- llama.cpp helpers (vecdotq.cuh) +__device__ __forceinline__ int get_int_b2(const void* x, const int& i32) { + const uint16_t* x16 = (const uint16_t*) x; + int x32 = x16[2 * i32 + 0] << 0; + x32 |= x16[2 * i32 + 1] << 16; + return x32; +} +__device__ __forceinline__ int get_int_b4(const void* x, const int& i32) { return ((const int*) x)[i32]; } +__device__ __forceinline__ uint32_t unpack_ksigns(const uint8_t v) { + const uint32_t p = __popc(v) & 1; + const uint32_t s = v ^ p << 7; + return s * 0x01010101; +} +__device__ __forceinline__ int2 get_int_from_table_16(const int& q4, const int8_t* table) { + const uint32_t* table32 = (const uint32_t*) table; + uint32_t tmp[2]; + const uint32_t low_high_selection_indices = (0x32103210 | ((q4 & 0x88888888) >> 1)); +#pragma unroll + for (uint32_t i = 0; i < 2; ++i) { + const uint32_t shift = 16 * i; + const uint32_t low = __byte_perm(table32[0], table32[1], q4 >> shift); + const uint32_t high = __byte_perm(table32[2], table32[3], q4 >> shift); + tmp[i] = __byte_perm(low, high, low_high_selection_indices >> shift); + } + return make_int2(__byte_perm(tmp[0], tmp[1], 0x6420), __byte_perm(tmp[0], tmp[1], 0x7531)); +} +#define ggml_cuda_dp4a(a, b, c) __dp4a((a), (b), (c)) + +// ---------------------------------------------------------------- the dot products (vecdotq.cuh) +__device__ __forceinline__ float vec_dot_q2_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q2_0* bq2_0 = (const block_q2_0*) vbq + kbx; + const float d2 = bq2_0->d; + const int16_t* qs = (const int16_t*) bq2_0->qs + iqs * 4; + const block_q8_1* bq8_1_chunk = bq8_1 + iqs; + int sumi = 0; +#pragma unroll + for (int j = 0; j < 4; ++j) { + const int q = qs[j]; + const int u = get_int_b4(bq8_1_chunk->qs, j * 2 + 0); + const int v = get_int_b4(bq8_1_chunk->qs, j * 2 + 1); + const int qe = __byte_perm(0x020100FF, 0x020100FF, q >> 0); + const int qo = __byte_perm(0x020100FF, 0x020100FF, q >> 2); + const int qx = __byte_perm(qe, qo, 0x5140); + const int qy = __byte_perm(qe, qo, 0x7362); + sumi = ggml_cuda_dp4a(u, qx, sumi); + sumi = ggml_cuda_dp4a(v, qy, sumi); + } + const float d8 = __low2float(bq8_1_chunk->ds); + return d2 * d8 * sumi; +} + +__device__ __forceinline__ float vec_dot_iq2_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_xxs* bq2 = (const block_iq2_xxs*) vbq + kbx; + const int q2 = get_int_b2(bq2->qs, iqs); + const uint8_t* aux8 = (const uint8_t*) &q2; + const uint32_t aux32 = get_int_b2(bq2->qs, iqs + 1); + int sumi = 0; +#pragma unroll + for (int k0 = 0; k0 < 8; k0 += 2) { + const uint2 grid_pos = ((const uint2*) iq2xxs_grid)[aux8[k0 / 2]]; + const uint32_t signs = unpack_ksigns(aux32 >> (7 * k0 / 2)); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid0 = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 0); + sumi = ggml_cuda_dp4a(grid0, u0, sumi); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid1 = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 1); + sumi = ggml_cuda_dp4a(grid1, u1, sumi); + } + const int ls = aux32 >> 27 | 1; + sumi = sumi * ls / 8; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_iq2_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_xs* bq2 = (const block_iq2_xs*) vbq + kbx; + const int2 q2_packed = make_int2(get_int_b2(bq2->qs, iqs + 0), get_int_b2(bq2->qs, iqs + 1)); + const uint16_t* q2 = (const uint16_t*) &q2_packed; + const int ls0 = bq2->scales[iqs / 2] & 0x0F; + const int ls1 = bq2->scales[iqs / 2] >> 4; + int sumi0 = 0, sumi1 = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const uint2 grid_pos = ((const uint2*) iq2xs_grid)[q2[l0 / 2] & 0x1FF]; + const uint32_t signs = unpack_ksigns(q2[l0 / 2] >> 9); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + if (l0 < 4) { + sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); + sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); + } else { + sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); + sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); + } + } + const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_iq2_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_s* bq2 = (const block_iq2_s*) vbq + kbx; + const int qs_packed = get_int_b2(bq2->qs, iqs / 2); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq2->qh[iqs / 2]; + const int signs_packed_32 = get_int_b2(bq2->qs, QK_K / 32 + iqs / 2); + const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; + const int ls0 = bq2->scales[iqs / 2] & 0x0F; + const int ls1 = bq2->scales[iqs / 2] >> 4; + int sumi0 = 0, sumi1 = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int* grid_pos = (const int*) (iq2s_grid + (qs[l0 / 2] | ((qh << (8 - l0)) & 0x300))); + const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); + const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); + const int grid_l = __vsub4(grid_pos[0] ^ signs0, signs0); + const int grid_h = __vsub4(grid_pos[1] ^ signs1, signs1); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + if (l0 < 4) { + sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); + sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); + } else { + sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); + sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); + } + } + const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_iq3_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq3_xxs* bq3 = (const block_iq3_xxs*) vbq + kbx; + const int2 q3_packed = make_int2(get_int_b2(bq3->qs, iqs), get_int_b2(bq3->qs, iqs + 1)); + const uint8_t* q3 = (const uint8_t*) &q3_packed; + const uint32_t aux32 = get_int_b2(bq3->qs, QK_K / 16 + iqs / 2); + int sumi = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int2 grid_pos = make_int2(iq3xxs_grid[q3[l0 + 0]], iq3xxs_grid[q3[l0 + 1]]); + const uint32_t signs = unpack_ksigns(aux32 >> (7 * l0 / 2)); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid_l, u0, sumi); + sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + } + const int ls = aux32 >> 28; + sumi = (ls * sumi + sumi / 2) / 2; + const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_iq3_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq3_s* bq3 = (const block_iq3_s*) vbq + kbx; + const int2 qs_packed = make_int2(get_int_b2(bq3->qs, iqs + 0), get_int_b2(bq3->qs, iqs + 1)); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq3->qh[iqs / 2]; + const int signs_packed_32 = get_int_b2(bq3->signs, iqs / 2); + const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; + int sumi = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int2 grid_pos = make_int2(iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)], + iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]); + const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); + const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid_l, u0, sumi); + sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + } + sumi *= 1 + 2 * ((bq3->scales[iqs / 4] >> ((iqs << 1) & 0x04)) & 0x0F); + const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_iq1_m_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq1_m* bq1 = (const block_iq1_m*) vbq + kbx; + const int qs_packed = get_int_b4(bq1->qs, iqs); + const uint8_t* qs = (const uint8_t*) &qs_packed; + int sumi[2] = {0, 0}; + float sumf[2] = {0.0f, 0.0f}; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int qhl = bq1->qh[2 * iqs + l0 / 4] >> (4 * ((l0 / 2) % 2)); + const int grid = iq1s_grid_gpu[qs[l0 / 2] | ((qhl & 0x07) << 8)]; + const int grid0 = (grid >> 0) & 0x0F0F0F0F; + const int grid1 = (grid >> 4) & 0x0F0F0F0F; + const int u0 = get_int_b4(bq8_1[iqs].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs].qs, l0 + 1); + sumi[l0 / 4] = ggml_cuda_dp4a(grid0, u0, sumi[l0 / 4]); + sumi[l0 / 4] = ggml_cuda_dp4a(grid1, u1, sumi[l0 / 4]); + const float delta = -1.0f + IQ1M_DELTA - (qhl & 0x08) * (2.0f * IQ1M_DELTA / 0x08); + int sumy = 0; + sumy = ggml_cuda_dp4a(u0, 0x01010101, sumy); + sumy = ggml_cuda_dp4a(u1, 0x01010101, sumy); + sumf[l0 / 4] += delta * sumy; + } + const uint16_t* sc = (const uint16_t*) bq1->scales; + iq1m_scale_t scale; + scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00F0) | ((sc[2] >> 4) & 0x0F00) | (sc[3] & 0xF000); + const float d = __half2float(scale.f16) * __low2float(bq8_1[iqs].ds); + const int tmp = sc[iqs / 2] >> (6 * (iqs % 2)); + const int sc0 = 2 * ((tmp >> 0) & 0x07) + 1; + const int sc1 = 2 * ((tmp >> 3) & 0x07) + 1; + return d * ((sumi[0] + sumf[0]) * sc0 + (sumi[1] + sumf[1]) * sc1); +} + +__device__ __forceinline__ float vec_dot_iq4_nl_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq4_nl* bq4 = (const block_iq4_nl*) vbq + kbx; + const int* q8 = (const int*) bq8_1->qs + iqs; + int sumi = 0; +#pragma unroll + for (int l = 0; l < 2; ++l) { + const int aux_q4 = get_int_b2(bq4->qs, iqs + l); + const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl); + sumi = ggml_cuda_dp4a(v.x, q8[l + 0], sumi); + sumi = ggml_cuda_dp4a(v.y, q8[l + 4], sumi); + } + const float d = __half2float(bq4->d) * __low2float(bq8_1->ds); + return d * sumi; +} + +// Q4_K / Q5_K gate/up and Q5_1 / Q8_0 down (the unsloth UD-Q4_K_XL file's experts) +__device__ __forceinline__ float vec_dot_q4_K_q8_1_impl_vmmq(const int* __restrict__ v, const int* __restrict__ u, + const uint8_t* __restrict__ sc, const uint8_t* __restrict__ m, + const half2& dm4, const float* __restrict__ d8) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < QR4_K; ++i) { + const int v0i = (v[0] >> (4 * i)) & 0x0F0F0F0F; + const int v1i = (v[1] >> (4 * i)) & 0x0F0F0F0F; + const int dot1 = ggml_cuda_dp4a(v1i, u[2 * i + 1], ggml_cuda_dp4a(v0i, u[2 * i + 0], 0)); + const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 1], ggml_cuda_dp4a(0x01010101, u[2 * i + 0], 0)); + sumf_d += d8[i] * (dot1 * sc[i]); + sumf_m += d8[i] * (dot2 * m[i]); + } + const float2 dm4f = __half22float2(dm4); + return dm4f.x * sumf_d - dm4f.y * sumf_m; +} + +// the 6-bit scale and min of the 32-value group pair `bq8_offset / 2` (vecdotq.cuh, shared by Q4_K and Q5_K) +__device__ __forceinline__ void k_scales(const uint8_t* scales8, int bq8_offset, uint16_t aux[2]) { + const uint16_t* scales = (const uint16_t*) scales8; + const int j = bq8_offset / 2; + const int jm = j & 1; + const uint32_t s0 = scales[jm + 0]; + const uint32_t s2 = scales[jm + 2]; + const uint32_t s4 = scales[jm + 4]; + const uint32_t hi = (uint32_t) -(int32_t) (j >= 2); + aux[0] = (uint16_t) (((s0 & 0x3f3f) & ~hi) | ((((s4 >> 0) & 0x0f0f) | ((s0 & 0xc0c0) >> 2)) & hi)); + aux[1] = (uint16_t) (((s2 & 0x3f3f) & ~hi) | ((((s4 >> 4) & 0x0f0f) | ((s2 & 0xc0c0) >> 2)) & hi)); +} + +__device__ __forceinline__ float vec_dot_q4_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q4_K* bq4_K = (const block_q4_K*) vbq + kbx; + int v[2]; + int u[2 * QR4_K]; + float d8[QR4_K]; + const int bq8_offset = QR4_K * ((iqs / 2) / (QI8_1 / 2)); + const int* q4 = (const int*) (bq4_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); + v[0] = q4[0]; + v[1] = q4[4]; + uint16_t aux[2]; + k_scales(bq4_K->scales, bq8_offset, aux); + const uint8_t* sc = (const uint8_t*) aux; + const uint8_t* m = sc + 2; + for (int i = 0; i < QR4_K; ++i) { + const block_q8_1* bq8i = bq8_1 + bq8_offset + i; + d8[i] = __low2float(bq8i->ds); + const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); + u[2 * i + 0] = q8[0]; + u[2 * i + 1] = q8[4]; + } + return vec_dot_q4_K_q8_1_impl_vmmq(v, u, sc, m, bq4_K->dm, d8); +} + +__device__ __forceinline__ float vec_dot_q5_K_q8_1_impl_vmmq(const int* __restrict__ vl, const int* __restrict__ vh, + const int* __restrict__ u, const uint8_t* __restrict__ sc, + const uint8_t* __restrict__ m, const half2& dm5, + const float* __restrict__ d8) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < QR5_K; ++i) { + const int vl0i = (vl[0] >> (4 * i)) & 0x0F0F0F0F; + const int vl1i = (vl[1] >> (4 * i)) & 0x0F0F0F0F; + const int vh0i = ((vh[0] >> i) << 4) & 0x10101010; + const int vh1i = ((vh[1] >> i) << 4) & 0x10101010; + const int v0i = vl0i | vh0i; + const int v1i = vl1i | vh1i; + const int dot1 = ggml_cuda_dp4a(v0i, u[2 * i + 0], ggml_cuda_dp4a(v1i, u[2 * i + 1], 0)); + const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 0], ggml_cuda_dp4a(0x01010101, u[2 * i + 1], 0)); + sumf_d += d8[i] * (dot1 * sc[i]); + sumf_m += d8[i] * (dot2 * m[i]); + } + const float2 dm5f = __half22float2(dm5); + return dm5f.x * sumf_d - dm5f.y * sumf_m; +} + +__device__ __forceinline__ float vec_dot_q5_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q5_K* bq5_K = (const block_q5_K*) vbq + kbx; + int vl[2]; + int vh[2]; + int u[2 * QR5_K]; + float d8[QR5_K]; + const int bq8_offset = QR5_K * ((iqs / 2) / (QI8_1 / 2)); + const int* ql = (const int*) (bq5_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); + const int* qh = (const int*) (bq5_K->qh + 4 * ((iqs / 2) % 4)); + vl[0] = ql[0]; + vl[1] = ql[4]; + vh[0] = qh[0] >> bq8_offset; + vh[1] = qh[4] >> bq8_offset; + uint16_t aux[2]; + k_scales(bq5_K->scales, bq8_offset, aux); + const uint8_t* sc = (const uint8_t*) aux; + const uint8_t* m = sc + 2; +#pragma unroll + for (int i = 0; i < QR5_K; ++i) { + const block_q8_1* bq8i = bq8_1 + bq8_offset + i; + d8[i] = __low2float(bq8i->ds); + const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); + u[2 * i + 0] = q8[0]; + u[2 * i + 1] = q8[4]; + } + return vec_dot_q5_K_q8_1_impl_vmmq(vl, vh, u, sc, m, bq5_K->dm, d8); +} + +constexpr int VDR_Q5_1 = 2, VDR_Q8_0 = 2; // VDR_Q5_1_Q8_1_MMVQ, VDR_Q8_0_Q8_1_MMVQ + +// The weights are llama.cpp's; the min term is not. llama.cpp multiplies the Q5_1 min by the q8_1 block's sum of +// the ORIGINAL activations while the scaled term uses the quantized ones, which adds min * (sum x - sum x_q) to +// every block: 1.9% instead of 1.2% relative error per expert (native_expert_parity). Like ggml-cpu, and like +// llama.cpp's own Q4_K / Q5_K dots, the min multiplies the quantized sum here. +__device__ __forceinline__ float vec_dot_q5_1_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q5_1* bq5_1 = (const block_q5_1*) vbq + kbx; + int sumi = 0, sumu = 0; +#pragma unroll + for (int i = 0; i < VDR_Q5_1; ++i) { + const int vl = get_int_b4(bq5_1->qs, iqs + i); + const int vh = get_int_b4(bq5_1->qh, 0) >> (4 * (iqs + i)); + const int u0 = get_int_b4(bq8_1->qs, iqs + i), u1 = get_int_b4(bq8_1->qs, iqs + i + QI5_1); + int vi0 = (vl >> 0) & 0x0F0F0F0F; + vi0 |= (vh << 4) & 0x00000010; + vi0 |= (vh << 11) & 0x00001000; + vi0 |= (vh << 18) & 0x00100000; + vi0 |= (vh << 25) & 0x10000000; + sumi = ggml_cuda_dp4a(vi0, u0, sumi); + int vi1 = (vl >> 4) & 0x0F0F0F0F; + vi1 |= (vh >> 12) & 0x00000010; + vi1 |= (vh >> 5) & 0x00001000; + vi1 |= (vh << 2) & 0x00100000; + vi1 |= (vh << 9) & 0x10000000; + sumi = ggml_cuda_dp4a(vi1, u1, sumi); + sumu = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumu)); + } + const float2 dm5 = __half22float2(bq5_1->dm); + const float d8 = __low2float(bq8_1->ds); + return sumi * (dm5.x * d8) + sumu * (dm5.y * d8); +} + +// Q6_K gate/up (UD-Q5_K_XL, UD-Q6_K_XL), VDR_Q6_K_Q8_1_MMVQ = 1 +__device__ __forceinline__ float vec_dot_q6_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q6_K* bq6_K = (const block_q6_K*) vbq + kbx; + const int bq8_offset = 2 * QR6_K * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 4); + const int scale_offset = (QI6_K / 4) * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 8); + const int vh_shift = 2 * ((iqs % (QI6_K / 2)) / (QI6_K / 4)); + const int vl = get_int_b2(bq6_K->ql, iqs); + const int vh = get_int_b2(bq6_K->qh, (QI6_K / 4) * (iqs / (QI6_K / 2)) + iqs % (QI6_K / 4)) >> vh_shift; + const int8_t* scales = bq6_K->scales + scale_offset; + float sumf = 0.0f; +#pragma unroll + for (int i = 0; i < QR6_K; ++i) { + const int u = get_int_b4(bq8_1[bq8_offset + 2 * i].qs, iqs % QI8_1); + const float d8 = __low2float(bq8_1[bq8_offset + 2 * i].ds); + const int sc = scales[4 * i]; + const int vil = (vl >> (4 * i)) & 0x0F0F0F0F; + const int vih = ((vh >> (4 * i)) << 4) & 0x30303030; + const int vi = __vsubss4((vil | vih), 0x20202020); // (vil | vih) - 32 + sumf += d8 * (ggml_cuda_dp4a(vi, u, 0) * sc); + } + return __half2float(bq6_K->d) * sumf; +} + +// IQ1_S gate/up (UD-IQ1_S), VDR 1. As for Q5_1, the delta term takes the quantized activation sum (ggml-cpu's +// bsums) rather than llama.cpp's sum of the original values. +__device__ __forceinline__ float vec_dot_iq1_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq1_s* bq1 = (const block_iq1_s*) vbq + kbx; + const int qs_packed = get_int_b2(bq1->qs, iqs); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq1->qh[iqs]; + int sumi = 0, sumy = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int grid = iq1s_grid_gpu[qs[l0 / 2] | (((qh >> 3 * (l0 / 2)) & 0x07) << 8)]; + const int grid0 = (grid >> 0) & 0x0F0F0F0F; + const int grid1 = (grid >> 4) & 0x0F0F0F0F; + const int u0 = get_int_b4(bq8_1[iqs].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid0, u0, sumi); + sumi = ggml_cuda_dp4a(grid1, u1, sumi); + sumy = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumy)); + } + const float d1q = __half2float(bq1->d) * (((qh >> 11) & 0x0E) + 1); + const float delta = -1.0f + IQ1S_DELTA - (qh & 0x8000) * (2.0f * IQ1S_DELTA / 0x8000); + return d1q * __low2float(bq8_1[iqs].ds) * (sumi + sumy * delta); +} + +// IQ4_XS gate/up (UD-IQ4_XS, UD-Q3_K_XL), VDR 4 +__device__ __forceinline__ float vec_dot_iq4_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq4_xs* bq4 = (const block_iq4_xs*) vbq + kbx; + int sumi = 0; +#pragma unroll + for (int j = 0; j < 4; ++j) { + const int aux_q4 = get_int_b4(bq4->qs, iqs + j); + const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl); + const int u0 = get_int_b4(bq8_1[iqs / 4].qs, j + 0); + const int u1 = get_int_b4(bq8_1[iqs / 4].qs, j + 4); + sumi = ggml_cuda_dp4a(v.x, u0, sumi); + sumi = ggml_cuda_dp4a(v.y, u1, sumi); + } + const int ls = ((bq4->scales_l[iqs / 8] >> (iqs & 0x04)) & 0x0F) | (((bq4->scales_h >> (iqs / 2)) & 0x03) << 4); + sumi *= ls - 32; + const float d = __half2float(bq4->d) * __low2float(bq8_1[iqs / 4].ds); + return d * sumi; +} + +__device__ __forceinline__ float vec_dot_q8_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_q8_0* bq8_0 = (const block_q8_0*) vbq + kbx; + int sumi = 0; +#pragma unroll + for (int i = 0; i < VDR_Q8_0; ++i) + sumi = ggml_cuda_dp4a(get_int_b2(bq8_0->qs, iqs + i), get_int_b4(bq8_1->qs, iqs + i), sumi); + const float d8_0 = __half2float(bq8_0->d), d8_1 = __low2float(bq8_1->ds); + return d8_0 * d8_1 * ((float) sumi); +} + +// ---------------------------------------------------------------- the formats +// qk = values per block, ipb = dot calls per block (qi / vdr), step = the iqs stride between calls. +template struct Fmt; +template<> struct Fmt<16> { static constexpr int qk = 256, ipb = 8, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_xxs_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<17> { static constexpr int qk = 256, ipb = 8, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_xs_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<18> { static constexpr int qk = 256, ipb = 8, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq3_xxs_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<20> { static constexpr int qk = 32, ipb = 2, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_nl_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<21> { static constexpr int qk = 256, ipb = 8, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq3_s_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<22> { static constexpr int qk = 256, ipb = 8, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_s_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<29> { static constexpr int qk = 256, ipb = 8, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_m_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<42> { static constexpr int qk = 64, ipb = 2, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q2_0_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<12> { static constexpr int qk = 256, ipb = QI4_K / 2, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q4_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<13> { static constexpr int qk = 256, ipb = QI5_K / 2, step = 2; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<7> { static constexpr int qk = 32, ipb = QI5_1 / VDR_Q5_1, step = VDR_Q5_1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_1_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<8> { static constexpr int qk = 32, ipb = QI8_0 / VDR_Q8_0, step = VDR_Q8_0; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q8_0_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<14> { static constexpr int qk = 256, ipb = QI6_K, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q6_K_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<19> { static constexpr int qk = 256, ipb = QI1_S, step = 1; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_s_q8_1(v, y, kbx, iqs); } }; +template<> struct Fmt<23> { static constexpr int qk = 256, ipb = QI4_XS / 4, step = 4; + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_xs_q8_1(v, y, kbx, iqs); } }; + +// Every format above, for the dispatch switches: X(type) once per Fmt. +#define STRATA_FMTS(X) X(7) X(8) X(12) X(13) X(14) X(16) X(17) X(18) X(19) X(20) X(21) X(22) X(23) X(29) X(42) + +inline int fmt_qk(int t) { + switch (t) { +#define STRATA_QK(T) case T: return Fmt::qk; + STRATA_FMTS(STRATA_QK) +#undef STRATA_QK + default: return 0; + } +} + +__device__ __forceinline__ float warp_sum(float v) { +#pragma unroll + for (int o = 16; o > 0; o >>= 1) v += __shfl_xor_sync(0xffffffffu, v, o); + return v; +} + +// One row against one q8_1 activation, the whole warp: call k = (block, part) is lane-strided. +template +__device__ __forceinline__ float row_dot(const uint8_t* row, const block_q8_1* x, int nb, int lane) { + using F = Fmt; + float s = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); + s += F::dot(row, x + kbx * (F::qk / 32), kbx, iqs); + } + return warp_sum(s); +} + +} // namespace +} // namespace strata::kernels diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index d8f793bbb40..70b0d9bec37 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -1,17 +1,10 @@ // src/kernels/cuda/iq_kernels.cu - see include/strata/kernels/iq_kernels.hpp. // -// The dot products (vec_dot_*_q8_1), the dequantizers and the q8_1 quantizer are transcribed from llama.cpp -// (ggml/src/ggml-cuda/vecdotq.cuh, dequantize.cuh, quantize.cu at the commit in third_party/ggml/VERSION.txt; -// MIT license, third_party/ggml/LICENSE). The block structs and codebook grids come from its ggml-common.h, -// included unchanged. +// The dequantizers and the q8_1 quantizer are transcribed from llama.cpp (ggml/src/ggml-cuda/dequantize.cuh, +// quantize.cu at the commit in third_party/ggml/VERSION.txt; MIT license, third_party/ggml/LICENSE), the dot products +// in iq_dot.cuh. #include "strata/kernels/iq_kernels.hpp" - -#include -#include - -#define GGML_COMMON_DECL_CUDA -#define GGML_COMMON_IMPL_CUDA -#include "ggml-common.h" +#include "iq_dot.cuh" #include #include @@ -24,536 +17,6 @@ void check(const char* what) { if (e != cudaSuccess) { std::fprintf(stderr, "%s: %s\n", what, cudaGetErrorString(e)); std::exit(1); } } -// ---------------------------------------------------------------- llama.cpp helpers (vecdotq.cuh) -__device__ __forceinline__ int get_int_b2(const void* x, const int& i32) { - const uint16_t* x16 = (const uint16_t*) x; - int x32 = x16[2 * i32 + 0] << 0; - x32 |= x16[2 * i32 + 1] << 16; - return x32; -} -__device__ __forceinline__ int get_int_b4(const void* x, const int& i32) { return ((const int*) x)[i32]; } -__device__ __forceinline__ uint32_t unpack_ksigns(const uint8_t v) { - const uint32_t p = __popc(v) & 1; - const uint32_t s = v ^ p << 7; - return s * 0x01010101; -} -__device__ __forceinline__ int2 get_int_from_table_16(const int& q4, const int8_t* table) { - const uint32_t* table32 = (const uint32_t*) table; - uint32_t tmp[2]; - const uint32_t low_high_selection_indices = (0x32103210 | ((q4 & 0x88888888) >> 1)); -#pragma unroll - for (uint32_t i = 0; i < 2; ++i) { - const uint32_t shift = 16 * i; - const uint32_t low = __byte_perm(table32[0], table32[1], q4 >> shift); - const uint32_t high = __byte_perm(table32[2], table32[3], q4 >> shift); - tmp[i] = __byte_perm(low, high, low_high_selection_indices >> shift); - } - return make_int2(__byte_perm(tmp[0], tmp[1], 0x6420), __byte_perm(tmp[0], tmp[1], 0x7531)); -} -#define ggml_cuda_dp4a(a, b, c) __dp4a((a), (b), (c)) - -// ---------------------------------------------------------------- the dot products (vecdotq.cuh) -__device__ __forceinline__ float vec_dot_q2_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q2_0* bq2_0 = (const block_q2_0*) vbq + kbx; - const float d2 = bq2_0->d; - const int16_t* qs = (const int16_t*) bq2_0->qs + iqs * 4; - const block_q8_1* bq8_1_chunk = bq8_1 + iqs; - int sumi = 0; -#pragma unroll - for (int j = 0; j < 4; ++j) { - const int q = qs[j]; - const int u = get_int_b4(bq8_1_chunk->qs, j * 2 + 0); - const int v = get_int_b4(bq8_1_chunk->qs, j * 2 + 1); - const int qe = __byte_perm(0x020100FF, 0x020100FF, q >> 0); - const int qo = __byte_perm(0x020100FF, 0x020100FF, q >> 2); - const int qx = __byte_perm(qe, qo, 0x5140); - const int qy = __byte_perm(qe, qo, 0x7362); - sumi = ggml_cuda_dp4a(u, qx, sumi); - sumi = ggml_cuda_dp4a(v, qy, sumi); - } - const float d8 = __low2float(bq8_1_chunk->ds); - return d2 * d8 * sumi; -} - -__device__ __forceinline__ float vec_dot_iq2_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq2_xxs* bq2 = (const block_iq2_xxs*) vbq + kbx; - const int q2 = get_int_b2(bq2->qs, iqs); - const uint8_t* aux8 = (const uint8_t*) &q2; - const uint32_t aux32 = get_int_b2(bq2->qs, iqs + 1); - int sumi = 0; -#pragma unroll - for (int k0 = 0; k0 < 8; k0 += 2) { - const uint2 grid_pos = ((const uint2*) iq2xxs_grid)[aux8[k0 / 2]]; - const uint32_t signs = unpack_ksigns(aux32 >> (7 * k0 / 2)); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid0 = __vsub4(grid_pos.x ^ signs0, signs0); - const int u0 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 0); - sumi = ggml_cuda_dp4a(grid0, u0, sumi); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid1 = __vsub4(grid_pos.y ^ signs1, signs1); - const int u1 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 1); - sumi = ggml_cuda_dp4a(grid1, u1, sumi); - } - const int ls = aux32 >> 27 | 1; - sumi = sumi * ls / 8; - const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_iq2_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq2_xs* bq2 = (const block_iq2_xs*) vbq + kbx; - const int2 q2_packed = make_int2(get_int_b2(bq2->qs, iqs + 0), get_int_b2(bq2->qs, iqs + 1)); - const uint16_t* q2 = (const uint16_t*) &q2_packed; - const int ls0 = bq2->scales[iqs / 2] & 0x0F; - const int ls1 = bq2->scales[iqs / 2] >> 4; - int sumi0 = 0, sumi1 = 0; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const uint2 grid_pos = ((const uint2*) iq2xs_grid)[q2[l0 / 2] & 0x1FF]; - const uint32_t signs = unpack_ksigns(q2[l0 / 2] >> 9); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); - const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); - const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - if (l0 < 4) { - sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); - sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); - } else { - sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); - sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); - } - } - const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; - const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_iq2_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq2_s* bq2 = (const block_iq2_s*) vbq + kbx; - const int qs_packed = get_int_b2(bq2->qs, iqs / 2); - const uint8_t* qs = (const uint8_t*) &qs_packed; - const int qh = bq2->qh[iqs / 2]; - const int signs_packed_32 = get_int_b2(bq2->qs, QK_K / 32 + iqs / 2); - const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; - const int ls0 = bq2->scales[iqs / 2] & 0x0F; - const int ls1 = bq2->scales[iqs / 2] >> 4; - int sumi0 = 0, sumi1 = 0; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const int* grid_pos = (const int*) (iq2s_grid + (qs[l0 / 2] | ((qh << (8 - l0)) & 0x300))); - const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); - const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); - const int grid_l = __vsub4(grid_pos[0] ^ signs0, signs0); - const int grid_h = __vsub4(grid_pos[1] ^ signs1, signs1); - const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - if (l0 < 4) { - sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); - sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); - } else { - sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); - sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); - } - } - const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; - const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_iq3_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq3_xxs* bq3 = (const block_iq3_xxs*) vbq + kbx; - const int2 q3_packed = make_int2(get_int_b2(bq3->qs, iqs), get_int_b2(bq3->qs, iqs + 1)); - const uint8_t* q3 = (const uint8_t*) &q3_packed; - const uint32_t aux32 = get_int_b2(bq3->qs, QK_K / 16 + iqs / 2); - int sumi = 0; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const int2 grid_pos = make_int2(iq3xxs_grid[q3[l0 + 0]], iq3xxs_grid[q3[l0 + 1]]); - const uint32_t signs = unpack_ksigns(aux32 >> (7 * l0 / 2)); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); - const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); - const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - sumi = ggml_cuda_dp4a(grid_l, u0, sumi); - sumi = ggml_cuda_dp4a(grid_h, u1, sumi); - } - const int ls = aux32 >> 28; - sumi = (ls * sumi + sumi / 2) / 2; - const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_iq3_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq3_s* bq3 = (const block_iq3_s*) vbq + kbx; - const int2 qs_packed = make_int2(get_int_b2(bq3->qs, iqs + 0), get_int_b2(bq3->qs, iqs + 1)); - const uint8_t* qs = (const uint8_t*) &qs_packed; - const int qh = bq3->qh[iqs / 2]; - const int signs_packed_32 = get_int_b2(bq3->signs, iqs / 2); - const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; - int sumi = 0; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const int2 grid_pos = make_int2(iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)], - iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]); - const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); - const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); - const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - sumi = ggml_cuda_dp4a(grid_l, u0, sumi); - sumi = ggml_cuda_dp4a(grid_h, u1, sumi); - } - sumi *= 1 + 2 * ((bq3->scales[iqs / 4] >> ((iqs << 1) & 0x04)) & 0x0F); - const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_iq1_m_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq1_m* bq1 = (const block_iq1_m*) vbq + kbx; - const int qs_packed = get_int_b4(bq1->qs, iqs); - const uint8_t* qs = (const uint8_t*) &qs_packed; - int sumi[2] = {0, 0}; - float sumf[2] = {0.0f, 0.0f}; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const int qhl = bq1->qh[2 * iqs + l0 / 4] >> (4 * ((l0 / 2) % 2)); - const int grid = iq1s_grid_gpu[qs[l0 / 2] | ((qhl & 0x07) << 8)]; - const int grid0 = (grid >> 0) & 0x0F0F0F0F; - const int grid1 = (grid >> 4) & 0x0F0F0F0F; - const int u0 = get_int_b4(bq8_1[iqs].qs, l0 + 0); - const int u1 = get_int_b4(bq8_1[iqs].qs, l0 + 1); - sumi[l0 / 4] = ggml_cuda_dp4a(grid0, u0, sumi[l0 / 4]); - sumi[l0 / 4] = ggml_cuda_dp4a(grid1, u1, sumi[l0 / 4]); - const float delta = -1.0f + IQ1M_DELTA - (qhl & 0x08) * (2.0f * IQ1M_DELTA / 0x08); - int sumy = 0; - sumy = ggml_cuda_dp4a(u0, 0x01010101, sumy); - sumy = ggml_cuda_dp4a(u1, 0x01010101, sumy); - sumf[l0 / 4] += delta * sumy; - } - const uint16_t* sc = (const uint16_t*) bq1->scales; - iq1m_scale_t scale; - scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00F0) | ((sc[2] >> 4) & 0x0F00) | (sc[3] & 0xF000); - const float d = __half2float(scale.f16) * __low2float(bq8_1[iqs].ds); - const int tmp = sc[iqs / 2] >> (6 * (iqs % 2)); - const int sc0 = 2 * ((tmp >> 0) & 0x07) + 1; - const int sc1 = 2 * ((tmp >> 3) & 0x07) + 1; - return d * ((sumi[0] + sumf[0]) * sc0 + (sumi[1] + sumf[1]) * sc1); -} - -__device__ __forceinline__ float vec_dot_iq4_nl_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq4_nl* bq4 = (const block_iq4_nl*) vbq + kbx; - const int* q8 = (const int*) bq8_1->qs + iqs; - int sumi = 0; -#pragma unroll - for (int l = 0; l < 2; ++l) { - const int aux_q4 = get_int_b2(bq4->qs, iqs + l); - const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl); - sumi = ggml_cuda_dp4a(v.x, q8[l + 0], sumi); - sumi = ggml_cuda_dp4a(v.y, q8[l + 4], sumi); - } - const float d = __half2float(bq4->d) * __low2float(bq8_1->ds); - return d * sumi; -} - -// Q4_K / Q5_K gate/up and Q5_1 / Q8_0 down (the unsloth UD-Q4_K_XL file's experts) -__device__ __forceinline__ float vec_dot_q4_K_q8_1_impl_vmmq(const int* __restrict__ v, const int* __restrict__ u, - const uint8_t* __restrict__ sc, const uint8_t* __restrict__ m, - const half2& dm4, const float* __restrict__ d8) { - float sumf_d = 0.0f; - float sumf_m = 0.0f; -#pragma unroll - for (int i = 0; i < QR4_K; ++i) { - const int v0i = (v[0] >> (4 * i)) & 0x0F0F0F0F; - const int v1i = (v[1] >> (4 * i)) & 0x0F0F0F0F; - const int dot1 = ggml_cuda_dp4a(v1i, u[2 * i + 1], ggml_cuda_dp4a(v0i, u[2 * i + 0], 0)); - const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 1], ggml_cuda_dp4a(0x01010101, u[2 * i + 0], 0)); - sumf_d += d8[i] * (dot1 * sc[i]); - sumf_m += d8[i] * (dot2 * m[i]); - } - const float2 dm4f = __half22float2(dm4); - return dm4f.x * sumf_d - dm4f.y * sumf_m; -} - -// the 6-bit scale and min of the 32-value group pair `bq8_offset / 2` (vecdotq.cuh, shared by Q4_K and Q5_K) -__device__ __forceinline__ void k_scales(const uint8_t* scales8, int bq8_offset, uint16_t aux[2]) { - const uint16_t* scales = (const uint16_t*) scales8; - const int j = bq8_offset / 2; - const int jm = j & 1; - const uint32_t s0 = scales[jm + 0]; - const uint32_t s2 = scales[jm + 2]; - const uint32_t s4 = scales[jm + 4]; - const uint32_t hi = (uint32_t) -(int32_t) (j >= 2); - aux[0] = (uint16_t) (((s0 & 0x3f3f) & ~hi) | ((((s4 >> 0) & 0x0f0f) | ((s0 & 0xc0c0) >> 2)) & hi)); - aux[1] = (uint16_t) (((s2 & 0x3f3f) & ~hi) | ((((s4 >> 4) & 0x0f0f) | ((s2 & 0xc0c0) >> 2)) & hi)); -} - -__device__ __forceinline__ float vec_dot_q4_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q4_K* bq4_K = (const block_q4_K*) vbq + kbx; - int v[2]; - int u[2 * QR4_K]; - float d8[QR4_K]; - const int bq8_offset = QR4_K * ((iqs / 2) / (QI8_1 / 2)); - const int* q4 = (const int*) (bq4_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); - v[0] = q4[0]; - v[1] = q4[4]; - uint16_t aux[2]; - k_scales(bq4_K->scales, bq8_offset, aux); - const uint8_t* sc = (const uint8_t*) aux; - const uint8_t* m = sc + 2; - for (int i = 0; i < QR4_K; ++i) { - const block_q8_1* bq8i = bq8_1 + bq8_offset + i; - d8[i] = __low2float(bq8i->ds); - const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); - u[2 * i + 0] = q8[0]; - u[2 * i + 1] = q8[4]; - } - return vec_dot_q4_K_q8_1_impl_vmmq(v, u, sc, m, bq4_K->dm, d8); -} - -__device__ __forceinline__ float vec_dot_q5_K_q8_1_impl_vmmq(const int* __restrict__ vl, const int* __restrict__ vh, - const int* __restrict__ u, const uint8_t* __restrict__ sc, - const uint8_t* __restrict__ m, const half2& dm5, - const float* __restrict__ d8) { - float sumf_d = 0.0f; - float sumf_m = 0.0f; -#pragma unroll - for (int i = 0; i < QR5_K; ++i) { - const int vl0i = (vl[0] >> (4 * i)) & 0x0F0F0F0F; - const int vl1i = (vl[1] >> (4 * i)) & 0x0F0F0F0F; - const int vh0i = ((vh[0] >> i) << 4) & 0x10101010; - const int vh1i = ((vh[1] >> i) << 4) & 0x10101010; - const int v0i = vl0i | vh0i; - const int v1i = vl1i | vh1i; - const int dot1 = ggml_cuda_dp4a(v0i, u[2 * i + 0], ggml_cuda_dp4a(v1i, u[2 * i + 1], 0)); - const int dot2 = ggml_cuda_dp4a(0x01010101, u[2 * i + 0], ggml_cuda_dp4a(0x01010101, u[2 * i + 1], 0)); - sumf_d += d8[i] * (dot1 * sc[i]); - sumf_m += d8[i] * (dot2 * m[i]); - } - const float2 dm5f = __half22float2(dm5); - return dm5f.x * sumf_d - dm5f.y * sumf_m; -} - -__device__ __forceinline__ float vec_dot_q5_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q5_K* bq5_K = (const block_q5_K*) vbq + kbx; - int vl[2]; - int vh[2]; - int u[2 * QR5_K]; - float d8[QR5_K]; - const int bq8_offset = QR5_K * ((iqs / 2) / (QI8_1 / 2)); - const int* ql = (const int*) (bq5_K->qs + 16 * bq8_offset + 4 * ((iqs / 2) % 4)); - const int* qh = (const int*) (bq5_K->qh + 4 * ((iqs / 2) % 4)); - vl[0] = ql[0]; - vl[1] = ql[4]; - vh[0] = qh[0] >> bq8_offset; - vh[1] = qh[4] >> bq8_offset; - uint16_t aux[2]; - k_scales(bq5_K->scales, bq8_offset, aux); - const uint8_t* sc = (const uint8_t*) aux; - const uint8_t* m = sc + 2; -#pragma unroll - for (int i = 0; i < QR5_K; ++i) { - const block_q8_1* bq8i = bq8_1 + bq8_offset + i; - d8[i] = __low2float(bq8i->ds); - const int* q8 = (const int*) bq8i->qs + ((iqs / 2) % 4); - u[2 * i + 0] = q8[0]; - u[2 * i + 1] = q8[4]; - } - return vec_dot_q5_K_q8_1_impl_vmmq(vl, vh, u, sc, m, bq5_K->dm, d8); -} - -constexpr int VDR_Q5_1 = 2, VDR_Q8_0 = 2; // VDR_Q5_1_Q8_1_MMVQ, VDR_Q8_0_Q8_1_MMVQ - -// The weights are llama.cpp's; the min term is not. llama.cpp multiplies the Q5_1 min by the q8_1 block's sum of -// the ORIGINAL activations while the scaled term uses the quantized ones, which adds min * (sum x - sum x_q) to -// every block: 1.9% instead of 1.2% relative error per expert (native_expert_parity). Like ggml-cpu, and like -// llama.cpp's own Q4_K / Q5_K dots, the min multiplies the quantized sum here. -__device__ __forceinline__ float vec_dot_q5_1_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q5_1* bq5_1 = (const block_q5_1*) vbq + kbx; - int sumi = 0, sumu = 0; -#pragma unroll - for (int i = 0; i < VDR_Q5_1; ++i) { - const int vl = get_int_b4(bq5_1->qs, iqs + i); - const int vh = get_int_b4(bq5_1->qh, 0) >> (4 * (iqs + i)); - const int u0 = get_int_b4(bq8_1->qs, iqs + i), u1 = get_int_b4(bq8_1->qs, iqs + i + QI5_1); - int vi0 = (vl >> 0) & 0x0F0F0F0F; - vi0 |= (vh << 4) & 0x00000010; - vi0 |= (vh << 11) & 0x00001000; - vi0 |= (vh << 18) & 0x00100000; - vi0 |= (vh << 25) & 0x10000000; - sumi = ggml_cuda_dp4a(vi0, u0, sumi); - int vi1 = (vl >> 4) & 0x0F0F0F0F; - vi1 |= (vh >> 12) & 0x00000010; - vi1 |= (vh >> 5) & 0x00001000; - vi1 |= (vh << 2) & 0x00100000; - vi1 |= (vh << 9) & 0x10000000; - sumi = ggml_cuda_dp4a(vi1, u1, sumi); - sumu = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumu)); - } - const float2 dm5 = __half22float2(bq5_1->dm); - const float d8 = __low2float(bq8_1->ds); - return sumi * (dm5.x * d8) + sumu * (dm5.y * d8); -} - -// Q6_K gate/up (UD-Q5_K_XL, UD-Q6_K_XL), VDR_Q6_K_Q8_1_MMVQ = 1 -__device__ __forceinline__ float vec_dot_q6_K_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q6_K* bq6_K = (const block_q6_K*) vbq + kbx; - const int bq8_offset = 2 * QR6_K * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 4); - const int scale_offset = (QI6_K / 4) * (iqs / (QI6_K / 2)) + (iqs % (QI6_K / 2)) / (QI6_K / 8); - const int vh_shift = 2 * ((iqs % (QI6_K / 2)) / (QI6_K / 4)); - const int vl = get_int_b2(bq6_K->ql, iqs); - const int vh = get_int_b2(bq6_K->qh, (QI6_K / 4) * (iqs / (QI6_K / 2)) + iqs % (QI6_K / 4)) >> vh_shift; - const int8_t* scales = bq6_K->scales + scale_offset; - float sumf = 0.0f; -#pragma unroll - for (int i = 0; i < QR6_K; ++i) { - const int u = get_int_b4(bq8_1[bq8_offset + 2 * i].qs, iqs % QI8_1); - const float d8 = __low2float(bq8_1[bq8_offset + 2 * i].ds); - const int sc = scales[4 * i]; - const int vil = (vl >> (4 * i)) & 0x0F0F0F0F; - const int vih = ((vh >> (4 * i)) << 4) & 0x30303030; - const int vi = __vsubss4((vil | vih), 0x20202020); // (vil | vih) - 32 - sumf += d8 * (ggml_cuda_dp4a(vi, u, 0) * sc); - } - return __half2float(bq6_K->d) * sumf; -} - -// IQ1_S gate/up (UD-IQ1_S), VDR 1. As for Q5_1, the delta term takes the quantized activation sum (ggml-cpu's -// bsums) rather than llama.cpp's sum of the original values. -__device__ __forceinline__ float vec_dot_iq1_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq1_s* bq1 = (const block_iq1_s*) vbq + kbx; - const int qs_packed = get_int_b2(bq1->qs, iqs); - const uint8_t* qs = (const uint8_t*) &qs_packed; - const int qh = bq1->qh[iqs]; - int sumi = 0, sumy = 0; -#pragma unroll - for (int l0 = 0; l0 < 8; l0 += 2) { - const int grid = iq1s_grid_gpu[qs[l0 / 2] | (((qh >> 3 * (l0 / 2)) & 0x07) << 8)]; - const int grid0 = (grid >> 0) & 0x0F0F0F0F; - const int grid1 = (grid >> 4) & 0x0F0F0F0F; - const int u0 = get_int_b4(bq8_1[iqs].qs, l0 + 0); - const int u1 = get_int_b4(bq8_1[iqs].qs, l0 + 1); - sumi = ggml_cuda_dp4a(grid0, u0, sumi); - sumi = ggml_cuda_dp4a(grid1, u1, sumi); - sumy = ggml_cuda_dp4a(0x01010101, u1, ggml_cuda_dp4a(0x01010101, u0, sumy)); - } - const float d1q = __half2float(bq1->d) * (((qh >> 11) & 0x0E) + 1); - const float delta = -1.0f + IQ1S_DELTA - (qh & 0x8000) * (2.0f * IQ1S_DELTA / 0x8000); - return d1q * __low2float(bq8_1[iqs].ds) * (sumi + sumy * delta); -} - -// IQ4_XS gate/up (UD-IQ4_XS, UD-Q3_K_XL), VDR 4 -__device__ __forceinline__ float vec_dot_iq4_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_iq4_xs* bq4 = (const block_iq4_xs*) vbq + kbx; - int sumi = 0; -#pragma unroll - for (int j = 0; j < 4; ++j) { - const int aux_q4 = get_int_b4(bq4->qs, iqs + j); - const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl); - const int u0 = get_int_b4(bq8_1[iqs / 4].qs, j + 0); - const int u1 = get_int_b4(bq8_1[iqs / 4].qs, j + 4); - sumi = ggml_cuda_dp4a(v.x, u0, sumi); - sumi = ggml_cuda_dp4a(v.y, u1, sumi); - } - const int ls = ((bq4->scales_l[iqs / 8] >> (iqs & 0x04)) & 0x0F) | (((bq4->scales_h >> (iqs / 2)) & 0x03) << 4); - sumi *= ls - 32; - const float d = __half2float(bq4->d) * __low2float(bq8_1[iqs / 4].ds); - return d * sumi; -} - -__device__ __forceinline__ float vec_dot_q8_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, - const int& kbx, const int& iqs) { - const block_q8_0* bq8_0 = (const block_q8_0*) vbq + kbx; - int sumi = 0; -#pragma unroll - for (int i = 0; i < VDR_Q8_0; ++i) - sumi = ggml_cuda_dp4a(get_int_b2(bq8_0->qs, iqs + i), get_int_b4(bq8_1->qs, iqs + i), sumi); - const float d8_0 = __half2float(bq8_0->d), d8_1 = __low2float(bq8_1->ds); - return d8_0 * d8_1 * ((float) sumi); -} - -// ---------------------------------------------------------------- the formats -// qk = values per block, ipb = dot calls per block (qi / vdr), step = the iqs stride between calls. -template struct Fmt; -template<> struct Fmt<16> { static constexpr int qk = 256, ipb = 8, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_xxs_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<17> { static constexpr int qk = 256, ipb = 8, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_xs_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<18> { static constexpr int qk = 256, ipb = 8, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq3_xxs_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<20> { static constexpr int qk = 32, ipb = 2, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_nl_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<21> { static constexpr int qk = 256, ipb = 8, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq3_s_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<22> { static constexpr int qk = 256, ipb = 8, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_s_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<29> { static constexpr int qk = 256, ipb = 8, step = 1; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_m_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<42> { static constexpr int qk = 64, ipb = 2, step = 1; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q2_0_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<12> { static constexpr int qk = 256, ipb = QI4_K / 2, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q4_K_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<13> { static constexpr int qk = 256, ipb = QI5_K / 2, step = 2; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_K_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<7> { static constexpr int qk = 32, ipb = QI5_1 / VDR_Q5_1, step = VDR_Q5_1; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q5_1_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<8> { static constexpr int qk = 32, ipb = QI8_0 / VDR_Q8_0, step = VDR_Q8_0; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q8_0_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<14> { static constexpr int qk = 256, ipb = QI6_K, step = 1; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_q6_K_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<19> { static constexpr int qk = 256, ipb = QI1_S, step = 1; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq1_s_q8_1(v, y, kbx, iqs); } }; -template<> struct Fmt<23> { static constexpr int qk = 256, ipb = QI4_XS / 4, step = 4; - __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_xs_q8_1(v, y, kbx, iqs); } }; - -// Every format above, for the dispatch switches: X(type) once per Fmt. -#define STRATA_FMTS(X) X(7) X(8) X(12) X(13) X(14) X(16) X(17) X(18) X(19) X(20) X(21) X(22) X(23) X(29) X(42) - -int fmt_qk(int t) { - switch (t) { -#define STRATA_QK(T) case T: return Fmt::qk; - STRATA_FMTS(STRATA_QK) -#undef STRATA_QK - default: return 0; - } -} - -__device__ __forceinline__ float warp_sum(float v) { -#pragma unroll - for (int o = 16; o > 0; o >>= 1) v += __shfl_xor_sync(0xffffffffu, v, o); - return v; -} - -// One row against one q8_1 activation, the whole warp: call k = (block, part) is lane-strided. -template -__device__ __forceinline__ float row_dot(const uint8_t* row, const block_q8_1* x, int nb, int lane) { - using F = Fmt; - float s = 0.0f; - for (int k = lane; k < nb * F::ipb; k += 32) { - const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); - s += F::dot(row, x + kbx * (F::qk / 32), kbx, iqs); - } - return warp_sum(s); -} - template __global__ void __launch_bounds__(128) mmvq_kernel(const uint8_t* __restrict__ w, size_t row_bytes, const block_q8_1* __restrict__ x, float* __restrict__ y, int n_in, @@ -570,59 +33,111 @@ __global__ void __launch_bounds__(128) mmvq_kernel(const uint8_t* __restrict__ w } // ---------------------------------------------------------------- grouped native experts -constexpr int GU_ROWS = 8; // rows per block (one warp each) +// Two kernels a call. The first computes a group's gate and up rows for one chunk of 32 rows of n_ff - one q8_1 block +// of h - SwiGLU and that block for each entry; the second the down rows. A warp takes several rows at once, one +// lane-strided accumulator each, so every value's sums and their order are `row_dot`'s. +constexpr int GUS_ROWS = 2, GUS_WARPS = 16; // gate and up rows per warp; 8 warps x 4 rows = one q8_1 block +constexpr int GUS_BATCH = 8; // entries whose block of h is staged at once +constexpr int DOWN_ROWS = 4, DOWN_WARPS = 8; + +// The K-quants' gate/up rows need more registers than the i-quants': with one block per SM they do not spill (Q4_K, Q5_K +// 15-17% faster at 2-4 tokens), while the i-quants run best at the occupancy the compiler picks (0: no bound). +template constexpr int gus_min_blocks() { return T == 12 || T == 13 || T == 14 ? 1 : 0; } template -__global__ void __launch_bounds__(256) native_gu_kernel(const unsigned long long* __restrict__ grp_ptr, - const int32_t* __restrict__ grp_start, - const int32_t* __restrict__ n_groups, - const int32_t* __restrict__ ent_tok, - const block_q8_1* __restrict__ xq, NativeExpertLayout L, - float* __restrict__ gate, float* __restrict__ up) { +__global__ void __launch_bounds__(GUS_WARPS * 32, gus_min_blocks()) native_gus_kernel(const unsigned long long* __restrict__ grp_ptr, + const int32_t* __restrict__ grp_start, + const int32_t* __restrict__ n_groups, + const int32_t* __restrict__ ent_tok, + const block_q8_1* __restrict__ xq, + NativeExpertLayout L, block_q8_1* __restrict__ hq) { + using F = Fmt; const int g = blockIdx.y; if (g >= *n_groups) return; - const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31; - const int row = blockIdx.x * GU_ROWS + warp; // 0 .. 2*n_ff - if (row >= 2 * L.n_ff) return; - const bool is_up = row >= L.n_ff; - const int r = is_up ? row - (int) L.n_ff : row; - const uint8_t* blob = (const uint8_t*) grp_ptr[g]; - const uint8_t* wr = blob + (is_up ? L.up_off : 0) + (size_t) r * L.gu_row; - const int nb = (int) (L.n_embd / Fmt::qk), xb = (int) (L.n_embd / 32); + __shared__ float sh[GUS_BATCH][32]; + const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31, c = blockIdx.x; + const uint8_t* wg = (const uint8_t*) grp_ptr[g] + (size_t) (32 * c + GUS_ROWS * warp) * L.gu_row; + const uint8_t* wu = wg + L.up_off; + const int nb = (int) (L.n_embd / F::qk), xb = (int) (L.n_embd / 32), hb = (int) (L.n_ff / 32); const int e0 = grp_start[g], e1 = grp_start[g + 1]; - for (int e = e0; e < e1; ++e) { - const float s = row_dot(wr, xq + (size_t) ent_tok[e] * xb, nb, lane); - if (lane == 0) (is_up ? up : gate)[(size_t) e * L.n_ff + r] = s; + for (int b0 = e0; b0 < e1; b0 += GUS_BATCH) { + const int b1 = min(e1, b0 + GUS_BATCH); + for (int e = b0; e < b1; ++e) { + const block_q8_1* x = xq + (size_t) ent_tok[e] * xb; + float sg[GUS_ROWS], su[GUS_ROWS]; +#pragma unroll + for (int i = 0; i < GUS_ROWS; ++i) sg[i] = su[i] = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); + const block_q8_1* xk = x + kbx * (F::qk / 32); +#pragma unroll + for (int i = 0; i < GUS_ROWS; ++i) { + sg[i] += F::dot(wg + (size_t) i * L.gu_row, xk, kbx, iqs); + su[i] += F::dot(wu + (size_t) i * L.gu_row, xk, kbx, iqs); + } + } +#pragma unroll + for (int i = 0; i < GUS_ROWS; ++i) { + const float gv = warp_sum(sg[i]), uv = warp_sum(su[i]); + if (lane == 0) sh[e - b0][GUS_ROWS * warp + i] = (gv / (1.0f + __expf(-gv))) * uv; + } + } + __syncthreads(); + for (int e = b0 + warp; e < b1; e += GUS_WARPS) { // an entry's block of h, as quantize_q8_1_kernel + const float xi = sh[e - b0][lane]; + float amax = fabsf(xi), sum = xi; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, o)); + sum += __shfl_xor_sync(0xffffffffu, sum, o); + } + const float d = amax / 127.0f; + const int8_t q = amax == 0.0f ? 0 : roundf(xi / d); + block_q8_1* y = hq + (size_t) e * hb + c; + y->qs[lane] = q; + if (lane == 0) y->ds = make_half2(d, sum); + } + __syncthreads(); } } -__global__ void swiglu_entries_kernel(const float* __restrict__ gate, const float* __restrict__ up, float* __restrict__ h, - long long n) { - const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; - if (i >= n) return; - const float g = gate[i]; - h[i] = (g / (1.0f + __expf(-g))) * up[i]; -} - template -__global__ void __launch_bounds__(256) native_down_kernel(const unsigned long long* __restrict__ grp_ptr, - const int32_t* __restrict__ grp_start, - const int32_t* __restrict__ n_groups, - const int32_t* __restrict__ ent_dst, - const block_q8_1* __restrict__ hq, NativeExpertLayout L, - float* __restrict__ out) { +__global__ void __launch_bounds__(DOWN_WARPS * 32) native_down_kernel(const unsigned long long* __restrict__ grp_ptr, + const int32_t* __restrict__ grp_start, + const int32_t* __restrict__ n_groups, + const int32_t* __restrict__ ent_dst, + const block_q8_1* __restrict__ hq, + NativeExpertLayout L, float* __restrict__ out) { + using F = Fmt; const int g = blockIdx.y; if (g >= *n_groups) return; const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31; - const int r = blockIdx.x * 8 + warp; - if (r >= L.n_embd) return; - const uint8_t* blob = (const uint8_t*) grp_ptr[g]; - const uint8_t* wr = blob + L.down_off + (size_t) r * L.d_row; - const int nb = (int) (L.n_ff / Fmt::qk), hb = (int) (L.n_ff / 32); + const int r0 = (blockIdx.x * DOWN_WARPS + warp) * DOWN_ROWS; + const uint8_t* wr = (const uint8_t*) grp_ptr[g] + L.down_off + (size_t) r0 * L.d_row; + const int nb = (int) (L.n_ff / F::qk), hb = (int) (L.n_ff / 32); const int e0 = grp_start[g], e1 = grp_start[g + 1]; for (int e = e0; e < e1; ++e) { - const float s = row_dot(wr, hq + (size_t) e * hb, nb, lane); - if (lane == 0) out[(size_t) ent_dst[e] * L.n_embd + r] = s; + const block_q8_1* x = hq + (size_t) e * hb; + float sd[DOWN_ROWS]; +#pragma unroll + for (int i = 0; i < DOWN_ROWS; ++i) sd[i] = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); + const block_q8_1* xk = x + kbx * (F::qk / 32); +#pragma unroll + for (int i = 0; i < DOWN_ROWS; ++i) sd[i] += F::dot(wr + (size_t) i * L.d_row, xk, kbx, iqs); + } + float v[DOWN_ROWS]; +#pragma unroll + for (int i = 0; i < DOWN_ROWS; ++i) v[i] = warp_sum(sd[i]); + if (lane == 0) { + float* o = out + (size_t) ent_dst[e] * L.n_embd + r0; + if constexpr (DOWN_ROWS == 4) *(float4*) o = make_float4(v[0], v[1], v[2], v[3]); + else if constexpr (DOWN_ROWS == 2) *(float2*) o = make_float2(v[0], v[1]); + else +#pragma unroll + for (int i = 0; i < DOWN_ROWS; ++i) o[i] = v[i]; + } } } @@ -1088,35 +603,32 @@ NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd, } size_t native_expert_scratch_bytes(int64_t cap, int64_t n_ff) { - const size_t f = (size_t) cap * (size_t) n_ff * sizeof(float); - return 3 * ((f + 255) & ~(size_t) 255) + (((size_t) cap * (size_t) (n_ff / 32) * sizeof(block_q8_1) + 255) & ~(size_t) 255); + return ((size_t) cap * (size_t) (n_ff / 32) * sizeof(block_q8_1) + 255) & ~(size_t) 255; } void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long* grp_ptr, const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_dst, const int32_t* ent_tok, int64_t cap_groups, int64_t cap_entries, const void* x_q8_1, void* scratch, float* out, void* stream) { if (cap_groups <= 0 || cap_entries <= 0) return; + if (L.n_ff % 32 != 0 || L.n_embd % (DOWN_WARPS * DOWN_ROWS) != 0) { + std::fprintf(stderr, "native_expert_grouped: n_ff %lld / n_embd %lld do not divide into the kernels' rows\n", + (long long) L.n_ff, (long long) L.n_embd); + std::exit(1); + } cudaStream_t s = (cudaStream_t) stream; - const size_t f = (size_t) cap_entries * (size_t) L.n_ff * sizeof(float), fa = (f + 255) & ~(size_t) 255; - float* gate = (float*) scratch; - float* up = (float*) ((uint8_t*) scratch + fa); - float* h = (float*) ((uint8_t*) scratch + 2 * fa); - block_q8_1* hq = (block_q8_1*) ((uint8_t*) scratch + 3 * fa); + auto* hq = (block_q8_1*) scratch; const auto* X = (const block_q8_1*) x_q8_1; - const dim3 ggu((unsigned) ((2 * L.n_ff + GU_ROWS - 1) / GU_ROWS), (unsigned) cap_groups); + const dim3 ggu((unsigned) (L.n_ff / 32), (unsigned) cap_groups); switch (L.gu_type) { -#define STRATA_GU(T) case T: native_gu_kernel<<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; +#define STRATA_GU(T) case T: native_gus_kernel<<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, hq); break; STRATA_FMTS(STRATA_GU) #undef STRATA_GU default: std::fprintf(stderr, "native_expert_grouped: gate/up type %d\n", L.gu_type); std::exit(1); } check("native_expert_grouped/gu"); - const long long nh = (long long) cap_entries * L.n_ff; - swiglu_entries_kernel<<<(unsigned) ((nh + 255) / 256), 256, 0, s>>>(gate, up, h, nh); - quantize_q8_1_kernel<<<(unsigned) ((nh + 255) / 256), 256, 0, s>>>(h, hq, nh); - const dim3 gd((unsigned) ((L.n_embd + 7) / 8), (unsigned) cap_groups); + const dim3 gd((unsigned) (L.n_embd / (DOWN_WARPS * DOWN_ROWS)), (unsigned) cap_groups); switch (L.d_type) { -#define STRATA_DOWN(T) case T: native_down_kernel<<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; +#define STRATA_DOWN(T) case T: native_down_kernel<<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; STRATA_FMTS(STRATA_DOWN) #undef STRATA_DOWN default: std::fprintf(stderr, "native_expert_grouped: down type %d\n", L.d_type); std::exit(1); diff --git a/src/kernels/expert_grouped_parity.cu b/src/kernels/expert_grouped_parity.cu new file mode 100644 index 00000000000..2b391ce7684 --- /dev/null +++ b/src/kernels/expert_grouped_parity.cu @@ -0,0 +1,354 @@ +// src/kernels/expert_grouped_parity.cu - `native_expert_grouped` (the verify window's and the second GPU's grouped +// experts) against the kernels it replaced, kept here verbatim: bitwise over every GPU gate/up and down format, on +// synthetic experts (random weights quantized by ggml) and plans of 1-8 tokens. +// +// expert_grouped_parity [--bench] +// +// --bench times both inside CUDA graphs (a kernel launched outside one pays WDDM's ~30-40 us) on the model files' +// format pairs, with routings shaped like the verify windows' (tokens share about a quarter of their experts). +#include "strata/kernels/iq_kernels.hpp" +#include "iq_dot.cuh" + +#include "ggml.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr int64_t H = 2560, FF = 640, K = 10, MAXT = 8, SLOTS = 48, DISTINCT = 4; + +void check(cudaError_t e, const char* what) { + if (e != cudaSuccess) { + std::fprintf(stderr, "%s: %s\n", what, cudaGetErrorString(e)); + std::exit(1); + } +} + +} // namespace + +// ---- the kernels native_expert_grouped launched before, verbatim +namespace strata::kernels::ref { + +constexpr int GU_ROWS = 8; // rows per block (one warp each) + +template +__global__ void __launch_bounds__(256) native_gu_kernel(const unsigned long long* __restrict__ grp_ptr, + const int32_t* __restrict__ grp_start, + const int32_t* __restrict__ n_groups, + const int32_t* __restrict__ ent_tok, + const block_q8_1* __restrict__ xq, NativeExpertLayout L, + float* __restrict__ gate, float* __restrict__ up) { + const int g = blockIdx.y; + if (g >= *n_groups) return; + const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31; + const int row = blockIdx.x * GU_ROWS + warp; // 0 .. 2*n_ff + if (row >= 2 * L.n_ff) return; + const bool is_up = row >= L.n_ff; + const int r = is_up ? row - (int) L.n_ff : row; + const uint8_t* blob = (const uint8_t*) grp_ptr[g]; + const uint8_t* wr = blob + (is_up ? L.up_off : 0) + (size_t) r * L.gu_row; + const int nb = (int) (L.n_embd / Fmt::qk), xb = (int) (L.n_embd / 32); + const int e0 = grp_start[g], e1 = grp_start[g + 1]; + for (int e = e0; e < e1; ++e) { + const float s = row_dot(wr, xq + (size_t) ent_tok[e] * xb, nb, lane); + if (lane == 0) (is_up ? up : gate)[(size_t) e * L.n_ff + r] = s; + } +} + +__global__ void swiglu_entries_kernel(const float* __restrict__ gate, const float* __restrict__ up, float* __restrict__ h, + long long n) { + const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; + if (i >= n) return; + const float g = gate[i]; + h[i] = (g / (1.0f + __expf(-g))) * up[i]; +} + +template +__global__ void __launch_bounds__(256) native_down_kernel(const unsigned long long* __restrict__ grp_ptr, + const int32_t* __restrict__ grp_start, + const int32_t* __restrict__ n_groups, + const int32_t* __restrict__ ent_dst, + const block_q8_1* __restrict__ hq, NativeExpertLayout L, + float* __restrict__ out) { + const int g = blockIdx.y; + if (g >= *n_groups) return; + const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31; + const int r = blockIdx.x * 8 + warp; + if (r >= L.n_embd) return; + const uint8_t* blob = (const uint8_t*) grp_ptr[g]; + const uint8_t* wr = blob + L.down_off + (size_t) r * L.d_row; + const int nb = (int) (L.n_ff / Fmt::qk), hb = (int) (L.n_ff / 32); + const int e0 = grp_start[g], e1 = grp_start[g + 1]; + for (int e = e0; e < e1; ++e) { + const float s = row_dot(wr, hq + (size_t) e * hb, nb, lane); + if (lane == 0) out[(size_t) ent_dst[e] * L.n_embd + r] = s; + } +} + +__global__ void quantize_q8_1_kernel(const float* __restrict__ x, block_q8_1* __restrict__ y, long long n) { + const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; + if (i >= n) return; + const float xi = x[i]; + float amax = fabsf(xi), sum = xi; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, o)); + sum += __shfl_xor_sync(0xffffffffu, sum, o); + } + const float d = amax / 127.0f; + const int8_t q = amax == 0.0f ? 0 : roundf(xi / d); + const long long ib = i / 32, iqs = i % 32; + y[ib].qs[iqs] = q; + if (iqs == 0) y[ib].ds = make_half2(d, sum); +} + +size_t scratch_bytes(int64_t cap, int64_t n_ff) { + const size_t f = (size_t) cap * (size_t) n_ff * sizeof(float); + return 3 * ((f + 255) & ~(size_t) 255) + (((size_t) cap * (size_t) (n_ff / 32) * sizeof(block_q8_1) + 255) & ~(size_t) 255); +} + +void grouped(const NativeExpertLayout& L, const unsigned long long* grp_ptr, const int32_t* grp_start, + const int32_t* n_groups, const int32_t* ent_dst, const int32_t* ent_tok, int64_t cap_groups, + int64_t cap_entries, const void* x_q8_1, void* scratch, float* out, cudaStream_t s) { + const size_t f = (size_t) cap_entries * (size_t) L.n_ff * sizeof(float), fa = (f + 255) & ~(size_t) 255; + float* gate = (float*) scratch; + float* up = (float*) ((uint8_t*) scratch + fa); + float* h = (float*) ((uint8_t*) scratch + 2 * fa); + block_q8_1* hq = (block_q8_1*) ((uint8_t*) scratch + 3 * fa); + const auto* X = (const block_q8_1*) x_q8_1; + const dim3 ggu((unsigned) ((2 * L.n_ff + GU_ROWS - 1) / GU_ROWS), (unsigned) cap_groups); + switch (L.gu_type) { +#define STRATA_GU(T) case T: native_gu_kernel<<>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break; + STRATA_FMTS(STRATA_GU) +#undef STRATA_GU + default: std::fprintf(stderr, "ref: gate/up type %d\n", L.gu_type); std::exit(1); + } + const long long nh = (long long) cap_entries * L.n_ff; + swiglu_entries_kernel<<<(unsigned) ((nh + 255) / 256), 256, 0, s>>>(gate, up, h, nh); + quantize_q8_1_kernel<<<(unsigned) ((nh + 255) / 256), 256, 0, s>>>(h, hq, nh); + const dim3 gd((unsigned) ((L.n_embd + 7) / 8), (unsigned) cap_groups); + switch (L.d_type) { +#define STRATA_DOWN(T) case T: native_down_kernel<<>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break; + STRATA_FMTS(STRATA_DOWN) +#undef STRATA_DOWN + default: std::fprintf(stderr, "ref: down type %d\n", L.d_type); std::exit(1); + } +} + +} // namespace strata::kernels::ref + +namespace { + +// DISTINCT experts of random weights (rows of different magnitude) quantized by ggml, repeated over SLOTS slots of one +// device arena so the kernels stream them from memory as they do the VRAM tier +uint8_t* make_arena(const strata::kernels::NativeExpertLayout& L, size_t stride, int seed) { + std::vector blobs((size_t) DISTINCT * L.bytes); + std::mt19937 rng(seed); + std::normal_distribution nd(0.f, 1.f); + auto quant = [&](int type, int64_t rows, int64_t cols, uint8_t* dst) { + std::vector w((size_t) (rows * cols)); + for (int64_t r = 0; r < rows; ++r) { + const float scale = 0.02f * (0.5f + (float) (r % 7) / 7.0f); + for (int64_t c = 0; c < cols; ++c) w[(size_t) (r * cols + c)] = scale * nd(rng); + } + const std::vector imatrix((size_t) cols, 1.0f); + ggml_quantize_chunk((ggml_type) type, w.data(), dst, 0, rows, cols, + ggml_quantize_requires_imatrix((ggml_type) type) ? imatrix.data() : nullptr); + }; + for (int b = 0; b < DISTINCT; ++b) { + uint8_t* blob = blobs.data() + (size_t) b * L.bytes; + quant(L.gu_type, FF, H, blob); + quant(L.gu_type, FF, H, blob + L.up_off); + quant(L.d_type, H, FF, blob + L.down_off); + } + uint8_t* arena = nullptr; + check(cudaMalloc(&arena, (size_t) SLOTS * stride), "arena"); + for (int i = 0; i < SLOTS; ++i) + check(cudaMemcpy(arena + (size_t) i * stride, blobs.data() + (size_t) (i % DISTINCT) * L.bytes, L.bytes, + cudaMemcpyHostToDevice), "arena upload"); + return arena; +} + +// A window's routing: n_tok tokens of K distinct experts each, a quarter drawn from the earlier tokens' ones (the +// verify windows' overlap); `hit` of the experts resident. Grouped as the plan is: distinct resident experts in +// routing order, each with its entries in routing order. +struct Plan { + std::vector ptr; + std::vector start, dst, tok; +}; +Plan make_plan(std::mt19937& rng, int n_tok, double hit, const uint8_t* arena, size_t stride) { + std::vector ids; + std::vector resident(SLOTS); + for (auto& r : resident) r = std::uniform_real_distribution(0, 1)(rng) < hit; + for (int t = 0; t < n_tok; ++t) + for (int j = 0; j < K; ++j) { + int32_t e; + bool again; + do { + const bool reuse = t > 0 && rng() % 4 == 0; + e = reuse ? ids[rng() % (size_t) (t * K)] : (int32_t) (rng() % SLOTS); + again = false; + for (int m = 0; m < j; ++m) again = again || ids[(size_t) (t * K + m)] == e; + } while (again); + ids.push_back(e); + } + Plan p; + const int n = n_tok * (int) K; + for (int i = 0; i < n; ++i) { + bool first = resident[(size_t) ids[(size_t) i]] != 0; + for (int j = 0; j < i && first; ++j) first = ids[(size_t) j] != ids[(size_t) i]; + if (!first) continue; + p.ptr.push_back((unsigned long long) (arena + (size_t) ids[(size_t) i] * stride)); + p.start.push_back((int32_t) p.dst.size()); + for (int j = i; j < n; ++j) + if (ids[(size_t) j] == ids[(size_t) i]) { p.dst.push_back(j); p.tok.push_back(j / (int) K); } + } + p.start.push_back((int32_t) p.dst.size()); + return p; +} + +struct Dev { + unsigned long long* ptr = nullptr; + int32_t *start = nullptr, *n = nullptr, *dst = nullptr, *tok = nullptr; + void *xq = nullptr, *scratch_ref = nullptr, *scratch_new = nullptr; + float *out_ref = nullptr, *out_new = nullptr; +}; + +void upload(const Plan& p, const Dev& d) { + const int32_t groups = (int32_t) p.ptr.size(); + if (groups > 0) { + check(cudaMemcpy(d.ptr, p.ptr.data(), p.ptr.size() * 8, cudaMemcpyHostToDevice), "plan"); + check(cudaMemcpy(d.dst, p.dst.data(), p.dst.size() * 4, cudaMemcpyHostToDevice), "plan"); + check(cudaMemcpy(d.tok, p.tok.data(), p.tok.size() * 4, cudaMemcpyHostToDevice), "plan"); + } + check(cudaMemcpy(d.start, p.start.data(), p.start.size() * 4, cudaMemcpyHostToDevice), "plan"); + check(cudaMemcpy(d.n, &groups, 4, cudaMemcpyHostToDevice), "plan"); +} + +void run(bool reference, const strata::kernels::NativeExpertLayout& L, const Dev& d, int64_t cap, cudaStream_t s) { + if (reference) + strata::kernels::ref::grouped(L, d.ptr, d.start, d.n, d.dst, d.tok, cap, cap, d.xq, d.scratch_ref, d.out_ref, s); + else + strata::kernels::native_expert_grouped(L, d.ptr, d.start, d.n, d.dst, d.tok, cap, cap, d.xq, d.scratch_new, + d.out_new, s); +} + +// microseconds per call, inside a graph of `reps` calls +double time_graph(bool reference, const strata::kernels::NativeExpertLayout& L, const Dev& d, int64_t cap, + cudaStream_t s) { + const int reps = 50; + cudaGraph_t graph = nullptr; + cudaGraphExec_t exec = nullptr; + check(cudaStreamBeginCapture(s, cudaStreamCaptureModeThreadLocal), "capture"); + for (int i = 0; i < reps; ++i) run(reference, L, d, cap, s); + check(cudaStreamEndCapture(s, &graph), "capture"); + check(cudaGraphInstantiate(&exec, graph, 0), "instantiate"); + const auto w0 = std::chrono::steady_clock::now(); // until the GPU has clocked up + while (std::chrono::duration(std::chrono::steady_clock::now() - w0).count() < 0.3) { + check(cudaGraphLaunch(exec, s), "launch"); + check(cudaStreamSynchronize(s), "warm"); + } + double best = 1e30; + for (int k = 0; k < 10; ++k) { + const auto t0 = std::chrono::steady_clock::now(); + check(cudaGraphLaunch(exec, s), "launch"); + check(cudaStreamSynchronize(s), "sync"); + best = std::min(best, std::chrono::duration(std::chrono::steady_clock::now() - t0).count() / reps); + } + cudaGraphExecDestroy(exec); + cudaGraphDestroy(graph); + return best; +} + +} // namespace + +int main(int argc, char** argv) { + const bool bench = argc > 1 && std::string(argv[1]) == "--bench"; + if (argc > 2 || (argc == 2 && !bench)) { + std::fprintf(stderr, "usage: expert_grouped_parity [--bench]\n"); + return 2; + } + cudaStream_t s; + check(cudaStreamCreateWithFlags(&s, cudaStreamNonBlocking), "stream"); + const int64_t cap = MAXT * K; + Dev d; + check(cudaMalloc(&d.ptr, (size_t) cap * 8), "dev"); + check(cudaMalloc(&d.start, (size_t) (cap + 1) * 4), "dev"); + check(cudaMalloc(&d.n, 4), "dev"); + check(cudaMalloc(&d.dst, (size_t) cap * 4), "dev"); + check(cudaMalloc(&d.tok, (size_t) cap * 4), "dev"); + check(cudaMalloc(&d.xq, (size_t) MAXT * (H / 32) * sizeof(block_q8_1)), "dev"); + check(cudaMalloc(&d.scratch_ref, strata::kernels::ref::scratch_bytes(cap, FF)), "dev"); + check(cudaMalloc(&d.scratch_new, strata::kernels::native_expert_scratch_bytes(cap, FF)), "dev"); + check(cudaMalloc(&d.out_ref, (size_t) cap * H * 4), "dev"); + check(cudaMalloc(&d.out_new, (size_t) cap * H * 4), "dev"); + { // the tokens' activations + std::mt19937 rng(7); + std::normal_distribution nd(0.f, 1.f); + std::vector x((size_t) (MAXT * H)); + for (auto& v : x) v = nd(rng); + float* dx = nullptr; + check(cudaMalloc(&dx, x.size() * 4), "dev"); + check(cudaMemcpy(dx, x.data(), x.size() * 4, cudaMemcpyHostToDevice), "x"); + strata::kernels::quantize_q8_1_rows(dx, MAXT, H, d.xq, s); + check(cudaStreamSynchronize(s), "x"); + cudaFree(dx); + } + // parity: every gate/up format with a down format in turn (down rows are n_ff = 640 long: 32- and 64-blocks); + // bench: the model files' pairs (IQ2_XXS .. IQ3_S with IQ4_NL / Q2_0: the IQ3_XXS file; Q4_K, Q5_K with Q5_1, + // Q8_0: UD-Q4_K_XL) + const int gus[] = {7, 8, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 42}, downs[] = {7, 8, 20, 42}; + std::vector> pairs; + if (bench) pairs = {{16, 42}, {17, 20}, {18, 20}, {21, 42}, {22, 20}, {12, 7}, {13, 8}}; + else + for (size_t i = 0; i < sizeof gus / sizeof gus[0]; ++i) pairs.emplace_back(gus[i], downs[i % 4]); + int bad = 0; + std::mt19937 rng(11); + for (const auto& [gu, dn] : pairs) { + const strata::kernels::NativeExpertLayout L = strata::kernels::native_expert_layout(gu, dn, H, FF); + const size_t stride = (L.bytes + 255) & ~(size_t) 255; + uint8_t* arena = make_arena(L, stride, gu * 100 + dn); + const char* names = ggml_type_name((ggml_type) gu); + if (!bench) { + int cases = 0, diff = 0; + for (int n_tok : {1, 2, 3, 4, 8}) + for (int rep = 0; rep < 4; ++rep) { + const Plan p = make_plan(rng, n_tok, rep == 0 ? 1.0 : 0.8, arena, stride); + upload(p, d); + check(cudaMemset(d.out_ref, 0xff, (size_t) cap * H * 4), "memset"); + check(cudaMemset(d.out_new, 0xff, (size_t) cap * H * 4), "memset"); + run(true, L, d, n_tok * K, s); + run(false, L, d, n_tok * K, s); + check(cudaStreamSynchronize(s), "run"); + std::vector a((size_t) cap * H), b((size_t) cap * H); + check(cudaMemcpy(a.data(), d.out_ref, a.size() * 4, cudaMemcpyDeviceToHost), "down"); + check(cudaMemcpy(b.data(), d.out_new, b.size() * 4, cudaMemcpyDeviceToHost), "down"); + diff += a != b; + ++cases; + } + std::printf("%-8s/%-7s %d plans: %s\n", names, ggml_type_name((ggml_type) dn), cases, + diff ? "MISMATCH" : "bitwise equal"); + bad += diff; + } else { + std::printf("%-8s/%-7s", names, ggml_type_name((ggml_type) dn)); + for (int n_tok : {1, 2, 4}) { + const Plan p = make_plan(rng, n_tok, 0.92, arena, stride); + upload(p, d); + const double tr = time_graph(true, L, d, n_tok * K, s), tn = time_graph(false, L, d, n_tok * K, s); + std::printf(" T%d %2zu experts %6.1f -> %5.1f us (%4.0f GB/s)", n_tok, p.ptr.size(), tr, tn, + (double) (p.ptr.size() * L.bytes) / tn * 1e-3); + } + std::printf("\n"); + } + cudaFree(arena); + } + if (!bench) std::printf("expert_grouped_parity: %s\n", bad ? "FAIL" : "PASS"); + return bad ? 1 : 0; +} From a3f14e4c1a60b4ab35a3e70685cd1cc96504e263 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 20:13:29 +0200 Subject: [PATCH 42/92] --window-profile: the wait for the CPU's rows as a histogram The average wait hides which side a layer waits on: the histogram shows the layers whose rows were in when the GPU got there (the flag's poll alone takes 5-8 us). On IQ3_XXS thinking text that is 74% of the layers; on code, and with UD-Q4_K_XL, the CPU side is the later one in most. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 3 +++ src/core/verify.cpp | 13 +++++++++++++ 2 files changed, 16 insertions(+) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 1ced279195c..c705dc2e365 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -229,6 +229,9 @@ class Verifier { std::vector prof_ns_; // [mixer kind (GDN, QSA)][stage], summed double prof_window_ns_[4] = {}; // inputs + embeddings, head, whole window, stamp gap int64_t prof_windows_ = 0, prof_layers_[2] = {}; + static constexpr int kWaitBuckets = 7; // the wait for the CPU's rows, by bucket (us) + static constexpr double kWaitEdges[kWaitBuckets - 1] = {6, 8, 12, 20, 50, 100}; + int64_t prof_wait_hist_[2][kWaitBuckets] = {}; }; } // namespace strata::core diff --git a/src/core/verify.cpp b/src/core/verify.cpp index c322e7173d1..8ee2bbb421a 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -1032,6 +1032,12 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, for (int i = 1; i < kLayerStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); // the shared expert's branch, from the doorbell (it forks there) to its end prof_ns_[(size_t) kind * kStamps + kLayerStamps] += (double) (st[kLayerStamps] - st[5]); + { // the wait for the CPU's rows, by bucket + const double us = (double) (st[10] - st[9]) / 1e3; + int b = 0; + while (b < kWaitBuckets - 1 && us >= kWaitEdges[b]) ++b; + ++prof_wait_hist_[kind][b]; + } ++prof_layers_[kind]; } prof_window_ns_[0] += (double) (w[1] - w[0]); @@ -1064,6 +1070,13 @@ void Verifier::print_profile() const { prof_layers_[0] ? a / (double) prof_layers_[0] / 1e3 : 0.0, prof_layers_[1] ? b / (double) prof_layers_[1] / 1e3 : 0.0, (a + b) / nw / 1e6); } + std::printf(" %-22s", "wait for CPU, layers"); + for (int b = 0; b < kWaitBuckets; ++b) { + const double f = 100.0 * (double) (prof_wait_hist_[0][b] + prof_wait_hist_[1][b]) / + (double) std::max(1, prof_layers_[0] + prof_layers_[1]); + if (b < kWaitBuckets - 1) std::printf(" <%.0f us %.0f%%,", kWaitEdges[b], f); + else std::printf(" more %.0f%%\n", f); + } } void Verifier::ple_ahead(int t, int32_t token) { From 8208f3f7f4fd88e78397cb7b1d5a4c43d51a07da Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 20:13:29 +0200 Subject: [PATCH 43/92] I-quant dot products: the signs as byte masks IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS and IQ3_S applied their signs to the grid with __vcmpne4 and __vsub4, which this GPU emulates in several instructions each. The masks now come from prmt's sign-replicate mode (inline PTX; __byte_perm drops that selector bit) and the sums from sum(g u) - 2 sum((g & m) u): the same integers, so bitwise the same results (expert_grouped_parity keeps the old dot products). The grouped experts of these formats are 10-29% faster per call than before the previous commit; IQ3_XXS fixed text, both GPUs: thinking 95.9 -> 97.2 t/s, code 113.2 -> 113.6. Window hashes unchanged. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- src/kernels/cuda/iq_dot.cuh | 113 +++++++++-------- src/kernels/expert_grouped_parity.cu | 182 +++++++++++++++++++++++++-- 2 files changed, 234 insertions(+), 61 deletions(-) diff --git a/src/kernels/cuda/iq_dot.cuh b/src/kernels/cuda/iq_dot.cuh index 43ef02a87f2..b5aa432c85a 100644 --- a/src/kernels/cuda/iq_dot.cuh +++ b/src/kernels/cuda/iq_dot.cuh @@ -2,8 +2,8 @@ // block), for iq_kernels.cu and the parity test that keeps its earlier expert kernels. // // Transcribed from llama.cpp (ggml/src/ggml-cuda/vecdotq.cuh at the commit in third_party/ggml/VERSION.txt; MIT -// license, third_party/ggml/LICENSE). The block structs and codebook grids come from its ggml-common.h, included -// unchanged. +// license, third_party/ggml/LICENSE); the i-quants apply their signs another way (`sign_masks`), with the same +// integer sums. The block structs and codebook grids come from its ggml-common.h, included unchanged. #pragma once #include @@ -44,6 +44,22 @@ __device__ __forceinline__ int2 get_int_from_table_16(const int& q4, const int8_ } #define ggml_cuda_dp4a(a, b, c) __dp4a((a), (b), (c)) +// The i-quants' signs as byte masks (0xff where a value is negative), four values a word: the multiply puts sign bit +// i on bit 7 of byte i, and prmt's sign-replicate mode (bit 3 of a selector nibble, which __byte_perm drops) fills +// the byte with it. Their dot products take sum(s g u) = sum(g u) - 2 sum((g & m) u) over the grid's unsigned +// values: the integers llama.cpp's sign-applied grid gives, in fewer instructions than __vcmpne4 and __vsub4. +__device__ __forceinline__ uint32_t msb_bytes(uint32_t x) { + uint32_t d; + asm("prmt.b32 %0, %1, 0, 0xBA98;" : "=r"(d) : "r"(x)); + return d; +} +__device__ __forceinline__ uint2 sign_masks(uint32_t s8) { // bit i of s8 -> byte i of (x, y) + return make_uint2(msb_bytes((s8 & 0x0F) * 0x10204080u), msb_bytes((s8 >> 4 & 0x0F) * 0x10204080u)); +} +__device__ __forceinline__ uint2 ksign_masks(uint32_t v7) { // 7 sign bits, the eighth their parity (ksigns_iq2xs) + return sign_masks(v7 | (__popc(v7) & 1) << 7); +} + // ---------------------------------------------------------------- the dot products (vecdotq.cuh) __device__ __forceinline__ float vec_dot_q2_0_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, const int& kbx, const int& iqs) { @@ -74,20 +90,19 @@ __device__ __forceinline__ float vec_dot_iq2_xxs_q8_1(const void* __restrict__ v const int q2 = get_int_b2(bq2->qs, iqs); const uint8_t* aux8 = (const uint8_t*) &q2; const uint32_t aux32 = get_int_b2(bq2->qs, iqs + 1); - int sumi = 0; + int sp = 0, sn = 0; // sum(g u), and over the negative values #pragma unroll for (int k0 = 0; k0 < 8; k0 += 2) { const uint2 grid_pos = ((const uint2*) iq2xxs_grid)[aux8[k0 / 2]]; - const uint32_t signs = unpack_ksigns(aux32 >> (7 * k0 / 2)); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid0 = __vsub4(grid_pos.x ^ signs0, signs0); + const uint2 m = ksign_masks((aux32 >> (7 * k0 / 2)) & 0x7F); const int u0 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 0); - sumi = ggml_cuda_dp4a(grid0, u0, sumi); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid1 = __vsub4(grid_pos.y ^ signs1, signs1); const int u1 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 1); - sumi = ggml_cuda_dp4a(grid1, u1, sumi); + sp = ggml_cuda_dp4a((int) grid_pos.x, u0, sp); + sn = ggml_cuda_dp4a((int) (grid_pos.x & m.x), u0, sn); + sp = ggml_cuda_dp4a((int) grid_pos.y, u1, sp); + sn = ggml_cuda_dp4a((int) (grid_pos.y & m.y), u1, sn); } + int sumi = sp - 2 * sn; const int ls = aux32 >> 27 | 1; sumi = sumi * ls / 8; const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); @@ -101,25 +116,20 @@ __device__ __forceinline__ float vec_dot_iq2_xs_q8_1(const void* __restrict__ vb const uint16_t* q2 = (const uint16_t*) &q2_packed; const int ls0 = bq2->scales[iqs / 2] & 0x0F; const int ls1 = bq2->scales[iqs / 2] >> 4; - int sumi0 = 0, sumi1 = 0; + int sp[2] = {0, 0}, sn[2] = {0, 0}; #pragma unroll for (int l0 = 0; l0 < 8; l0 += 2) { const uint2 grid_pos = ((const uint2*) iq2xs_grid)[q2[l0 / 2] & 0x1FF]; - const uint32_t signs = unpack_ksigns(q2[l0 / 2] >> 9); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const uint2 m = ksign_masks(q2[l0 / 2] >> 9); const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - if (l0 < 4) { - sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); - sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); - } else { - sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); - sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); - } + const int h = l0 / 4; + sp[h] = ggml_cuda_dp4a((int) grid_pos.x, u0, sp[h]); + sn[h] = ggml_cuda_dp4a((int) (grid_pos.x & m.x), u0, sn[h]); + sp[h] = ggml_cuda_dp4a((int) grid_pos.y, u1, sp[h]); + sn[h] = ggml_cuda_dp4a((int) (grid_pos.y & m.y), u1, sn[h]); } + const int sumi0 = sp[0] - 2 * sn[0], sumi1 = sp[1] - 2 * sn[1]; const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); return d * sumi; @@ -135,24 +145,20 @@ __device__ __forceinline__ float vec_dot_iq2_s_q8_1(const void* __restrict__ vbq const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; const int ls0 = bq2->scales[iqs / 2] & 0x0F; const int ls1 = bq2->scales[iqs / 2] >> 4; - int sumi0 = 0, sumi1 = 0; + int sp[2] = {0, 0}, sn[2] = {0, 0}; #pragma unroll for (int l0 = 0; l0 < 8; l0 += 2) { - const int* grid_pos = (const int*) (iq2s_grid + (qs[l0 / 2] | ((qh << (8 - l0)) & 0x300))); - const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); - const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); - const int grid_l = __vsub4(grid_pos[0] ^ signs0, signs0); - const int grid_h = __vsub4(grid_pos[1] ^ signs1, signs1); + const uint2 grid_pos = ((const uint2*) iq2s_grid)[qs[l0 / 2] | ((qh << (8 - l0)) & 0x300)]; + const uint2 m = sign_masks(signs_packed_8[l0 / 2]); const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - if (l0 < 4) { - sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); - sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); - } else { - sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); - sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); - } + const int h = l0 / 4; + sp[h] = ggml_cuda_dp4a((int) grid_pos.x, u0, sp[h]); + sn[h] = ggml_cuda_dp4a((int) (grid_pos.x & m.x), u0, sn[h]); + sp[h] = ggml_cuda_dp4a((int) grid_pos.y, u1, sp[h]); + sn[h] = ggml_cuda_dp4a((int) (grid_pos.y & m.y), u1, sn[h]); } + const int sumi0 = sp[0] - 2 * sn[0], sumi1 = sp[1] - 2 * sn[1]; const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); return d * sumi; @@ -164,20 +170,19 @@ __device__ __forceinline__ float vec_dot_iq3_xxs_q8_1(const void* __restrict__ v const int2 q3_packed = make_int2(get_int_b2(bq3->qs, iqs), get_int_b2(bq3->qs, iqs + 1)); const uint8_t* q3 = (const uint8_t*) &q3_packed; const uint32_t aux32 = get_int_b2(bq3->qs, QK_K / 16 + iqs / 2); - int sumi = 0; + int sp = 0, sn = 0; #pragma unroll for (int l0 = 0; l0 < 8; l0 += 2) { - const int2 grid_pos = make_int2(iq3xxs_grid[q3[l0 + 0]], iq3xxs_grid[q3[l0 + 1]]); - const uint32_t signs = unpack_ksigns(aux32 >> (7 * l0 / 2)); - const int signs0 = __vcmpne4(signs & 0x08040201, 0); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const uint32_t gx = iq3xxs_grid[q3[l0 + 0]], gy = iq3xxs_grid[q3[l0 + 1]]; + const uint2 m = ksign_masks((aux32 >> (7 * l0 / 2)) & 0x7F); const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); - const int signs1 = __vcmpne4(signs & 0x80402010, 0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - sumi = ggml_cuda_dp4a(grid_l, u0, sumi); - sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + sp = ggml_cuda_dp4a((int) gx, u0, sp); + sn = ggml_cuda_dp4a((int) (gx & m.x), u0, sn); + sp = ggml_cuda_dp4a((int) gy, u1, sp); + sn = ggml_cuda_dp4a((int) (gy & m.y), u1, sn); } + int sumi = sp - 2 * sn; const int ls = aux32 >> 28; sumi = (ls * sumi + sumi / 2) / 2; const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); @@ -192,20 +197,20 @@ __device__ __forceinline__ float vec_dot_iq3_s_q8_1(const void* __restrict__ vbq const int qh = bq3->qh[iqs / 2]; const int signs_packed_32 = get_int_b2(bq3->signs, iqs / 2); const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; - int sumi = 0; + int sp = 0, sn = 0; #pragma unroll for (int l0 = 0; l0 < 8; l0 += 2) { - const int2 grid_pos = make_int2(iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)], - iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]); - const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); - const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); - const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); - const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const uint32_t gx = iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)]; + const uint32_t gy = iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]; + const uint2 m = sign_masks(signs_packed_8[l0 / 2]); const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); - sumi = ggml_cuda_dp4a(grid_l, u0, sumi); - sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + sp = ggml_cuda_dp4a((int) gx, u0, sp); + sn = ggml_cuda_dp4a((int) (gx & m.x), u0, sn); + sp = ggml_cuda_dp4a((int) gy, u1, sp); + sn = ggml_cuda_dp4a((int) (gy & m.y), u1, sn); } + int sumi = sp - 2 * sn; sumi *= 1 + 2 * ((bq3->scales[iqs / 4] >> ((iqs << 1) & 0x04)) & 0x0F); const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); return d * sumi; diff --git a/src/kernels/expert_grouped_parity.cu b/src/kernels/expert_grouped_parity.cu index 2b391ce7684..110185b7cce 100644 --- a/src/kernels/expert_grouped_parity.cu +++ b/src/kernels/expert_grouped_parity.cu @@ -1,6 +1,7 @@ // src/kernels/expert_grouped_parity.cu - `native_expert_grouped` (the verify window's and the second GPU's grouped -// experts) against the kernels it replaced, kept here verbatim: bitwise over every GPU gate/up and down format, on -// synthetic experts (random weights quantized by ggml) and plans of 1-8 tokens. +// experts) against the kernels it replaced and the i-quant dot products as they were, kept here verbatim: bitwise +// over every GPU gate/up and down format, on synthetic experts (random weights quantized by ggml) and plans of 1-8 +// tokens. // // expert_grouped_parity [--bench] // @@ -33,9 +34,176 @@ void check(cudaError_t e, const char* what) { } // namespace -// ---- the kernels native_expert_grouped launched before, verbatim +// ---- the kernels native_expert_grouped launched before, verbatim, and the i-quant dot products as they were +// (sign masks through __vcmpne4 / __vsub4) namespace strata::kernels::ref { +__device__ __forceinline__ float ref_vec_dot_iq2_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_xxs* bq2 = (const block_iq2_xxs*) vbq + kbx; + const int q2 = get_int_b2(bq2->qs, iqs); + const uint8_t* aux8 = (const uint8_t*) &q2; + const uint32_t aux32 = get_int_b2(bq2->qs, iqs + 1); + int sumi = 0; +#pragma unroll + for (int k0 = 0; k0 < 8; k0 += 2) { + const uint2 grid_pos = ((const uint2*) iq2xxs_grid)[aux8[k0 / 2]]; + const uint32_t signs = unpack_ksigns(aux32 >> (7 * k0 / 2)); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid0 = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 0); + sumi = ggml_cuda_dp4a(grid0, u0, sumi); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid1 = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, k0 + 1); + sumi = ggml_cuda_dp4a(grid1, u1, sumi); + } + const int ls = aux32 >> 27 | 1; + sumi = sumi * ls / 8; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float ref_vec_dot_iq2_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_xs* bq2 = (const block_iq2_xs*) vbq + kbx; + const int2 q2_packed = make_int2(get_int_b2(bq2->qs, iqs + 0), get_int_b2(bq2->qs, iqs + 1)); + const uint16_t* q2 = (const uint16_t*) &q2_packed; + const int ls0 = bq2->scales[iqs / 2] & 0x0F; + const int ls1 = bq2->scales[iqs / 2] >> 4; + int sumi0 = 0, sumi1 = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const uint2 grid_pos = ((const uint2*) iq2xs_grid)[q2[l0 / 2] & 0x1FF]; + const uint32_t signs = unpack_ksigns(q2[l0 / 2] >> 9); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + if (l0 < 4) { + sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); + sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); + } else { + sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); + sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); + } + } + const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float ref_vec_dot_iq2_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq2_s* bq2 = (const block_iq2_s*) vbq + kbx; + const int qs_packed = get_int_b2(bq2->qs, iqs / 2); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq2->qh[iqs / 2]; + const int signs_packed_32 = get_int_b2(bq2->qs, QK_K / 32 + iqs / 2); + const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; + const int ls0 = bq2->scales[iqs / 2] & 0x0F; + const int ls1 = bq2->scales[iqs / 2] >> 4; + int sumi0 = 0, sumi1 = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int* grid_pos = (const int*) (iq2s_grid + (qs[l0 / 2] | ((qh << (8 - l0)) & 0x300))); + const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); + const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); + const int grid_l = __vsub4(grid_pos[0] ^ signs0, signs0); + const int grid_h = __vsub4(grid_pos[1] ^ signs1, signs1); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + if (l0 < 4) { + sumi0 = ggml_cuda_dp4a(grid_l, u0, sumi0); + sumi0 = ggml_cuda_dp4a(grid_h, u1, sumi0); + } else { + sumi1 = ggml_cuda_dp4a(grid_l, u0, sumi1); + sumi1 = ggml_cuda_dp4a(grid_h, u1, sumi1); + } + } + const int sumi = (sumi0 * ls0 + sumi1 * ls1 + (sumi0 + sumi1) / 2) / 4; + const float d = __half2float(bq2->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float ref_vec_dot_iq3_xxs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq3_xxs* bq3 = (const block_iq3_xxs*) vbq + kbx; + const int2 q3_packed = make_int2(get_int_b2(bq3->qs, iqs), get_int_b2(bq3->qs, iqs + 1)); + const uint8_t* q3 = (const uint8_t*) &q3_packed; + const uint32_t aux32 = get_int_b2(bq3->qs, QK_K / 16 + iqs / 2); + int sumi = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int2 grid_pos = make_int2(iq3xxs_grid[q3[l0 + 0]], iq3xxs_grid[q3[l0 + 1]]); + const uint32_t signs = unpack_ksigns(aux32 >> (7 * l0 / 2)); + const int signs0 = __vcmpne4(signs & 0x08040201, 0); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int signs1 = __vcmpne4(signs & 0x80402010, 0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid_l, u0, sumi); + sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + } + const int ls = aux32 >> 28; + sumi = (ls * sumi + sumi / 2) / 2; + const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +__device__ __forceinline__ float ref_vec_dot_iq3_s_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, + const int& kbx, const int& iqs) { + const block_iq3_s* bq3 = (const block_iq3_s*) vbq + kbx; + const int2 qs_packed = make_int2(get_int_b2(bq3->qs, iqs + 0), get_int_b2(bq3->qs, iqs + 1)); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq3->qh[iqs / 2]; + const int signs_packed_32 = get_int_b2(bq3->signs, iqs / 2); + const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; + int sumi = 0; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const int2 grid_pos = make_int2(iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)], + iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]); + const int signs0 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x03) << 7) | ((signs_packed_8[l0 / 2] & 0x0C) << 21), 0x00000000); + const int signs1 = __vcmpne4(((signs_packed_8[l0 / 2] & 0x30) << 3) | ((signs_packed_8[l0 / 2] & 0xC0) << 17), 0x00000000); + const int grid_l = __vsub4(grid_pos.x ^ signs0, signs0); + const int grid_h = __vsub4(grid_pos.y ^ signs1, signs1); + const int u0 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 0); + const int u1 = get_int_b4(bq8_1[iqs / 2].qs, l0 + 1); + sumi = ggml_cuda_dp4a(grid_l, u0, sumi); + sumi = ggml_cuda_dp4a(grid_h, u1, sumi); + } + sumi *= 1 + 2 * ((bq3->scales[iqs / 4] >> ((iqs << 1) & 0x04)) & 0x0F); + const float d = __half2float(bq3->d) * __low2float(bq8_1[iqs / 2].ds); + return d * sumi; +} + +template struct RefFmt : Fmt {}; +template<> struct RefFmt<16> : Fmt<16> { + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return ref_vec_dot_iq2_xxs_q8_1(v, y, kbx, iqs); } }; +template<> struct RefFmt<17> : Fmt<17> { + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return ref_vec_dot_iq2_xs_q8_1(v, y, kbx, iqs); } }; +template<> struct RefFmt<18> : Fmt<18> { + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return ref_vec_dot_iq3_xxs_q8_1(v, y, kbx, iqs); } }; +template<> struct RefFmt<21> : Fmt<21> { + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return ref_vec_dot_iq3_s_q8_1(v, y, kbx, iqs); } }; +template<> struct RefFmt<22> : Fmt<22> { + __device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return ref_vec_dot_iq2_s_q8_1(v, y, kbx, iqs); } }; + +template +__device__ __forceinline__ float ref_row_dot(const uint8_t* row, const block_q8_1* x, int nb, int lane) { + using F = RefFmt; + float s = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); + s += F::dot(row, x + kbx * (F::qk / 32), kbx, iqs); + } + return warp_sum(s); +} + constexpr int GU_ROWS = 8; // rows per block (one warp each) template @@ -54,10 +222,10 @@ __global__ void __launch_bounds__(256) native_gu_kernel(const unsigned long long const int r = is_up ? row - (int) L.n_ff : row; const uint8_t* blob = (const uint8_t*) grp_ptr[g]; const uint8_t* wr = blob + (is_up ? L.up_off : 0) + (size_t) r * L.gu_row; - const int nb = (int) (L.n_embd / Fmt::qk), xb = (int) (L.n_embd / 32); + const int nb = (int) (L.n_embd / RefFmt::qk), xb = (int) (L.n_embd / 32); const int e0 = grp_start[g], e1 = grp_start[g + 1]; for (int e = e0; e < e1; ++e) { - const float s = row_dot(wr, xq + (size_t) ent_tok[e] * xb, nb, lane); + const float s = ref_row_dot(wr, xq + (size_t) ent_tok[e] * xb, nb, lane); if (lane == 0) (is_up ? up : gate)[(size_t) e * L.n_ff + r] = s; } } @@ -84,10 +252,10 @@ __global__ void __launch_bounds__(256) native_down_kernel(const unsigned long lo if (r >= L.n_embd) return; const uint8_t* blob = (const uint8_t*) grp_ptr[g]; const uint8_t* wr = blob + L.down_off + (size_t) r * L.d_row; - const int nb = (int) (L.n_ff / Fmt::qk), hb = (int) (L.n_ff / 32); + const int nb = (int) (L.n_ff / RefFmt::qk), hb = (int) (L.n_ff / 32); const int e0 = grp_start[g], e1 = grp_start[g + 1]; for (int e = e0; e < e1; ++e) { - const float s = row_dot(wr, hq + (size_t) e * hb, nb, lane); + const float s = ref_row_dot(wr, hq + (size_t) e * hb, nb, lane); if (lane == 0) out[(size_t) ent_dst[e] * L.n_embd + r] = s; } } From 70e73db4bc57655829990d9ea4fa5222b9c5325e Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 20:33:37 +0200 Subject: [PATCH 44/92] DETAILS.md: the speeds with the 3090's expert phase reworked Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- docs/DETAILS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 2c78d65efd1..58ad8d421ec 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -103,7 +103,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 996-1112 tok/s, generation 57-67 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 996-1111 tok/s, generation 58-67 tok/s. ### A second GPU @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1721-1909 tok/s instead of 996-1112, generation 80-94 tok/s instead of 57-67; -IQ3_XXS 1833-2017 instead of 1446-1602, generation 88-113 instead of 88-100. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1717-1913 tok/s instead of 996-1111, generation 81-96 tok/s instead of 58-67; +IQ3_XXS 1842-2016 instead of 1449-1601, generation 96-117 instead of 92-104. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From 738e16602daf04b13afecc0dc891cbb16c5466e6 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 23:16:12 +0200 Subject: [PATCH 45/92] Verify window: the dense projections of 2+ tokens read an interleaved copy of their activations The quantizer also writes the tokens' q8_1 values interleaved (a position of every token in one 16-byte word, block-major for the K-quants and Q8_0, position-major for IQ4_XS and IQ3_S). A warp then takes 2-4 rows, decodes each weight block once and applies it to every token with the single-token dot's arithmetic, walking the chunks of the exact layout's four warps with a sum per warp (IQ3_S: row_dot's one sum) added in the same order before the xor tree, so every value is bitwise the multi-column kernel's. Matrices of fewer than 2048 rows and single tokens keep the old kernels. Used for the mixer's projections and the head. Per call, 4 tokens (mmvq_il_parity --bench): IQ4_XS 2560x10240 40.7 -> 29.3 us, Q6_K 68.7 -> 47.4, Q8_0 60.9 -> 43.1, IQ3_S 2560x6144 48.8 -> 21.3; 2 tokens 10-35% faster. Fixed text, 2 alternating rounds: IQ3_XXS code +3.8%, thinking +2.1%; UD-Q4_K_XL within noise. Window hashes identical to the previous build (both models, thinking and code; --spec-split, --pcie-frac 0.3, --second-gpu -1). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- CMakeLists.txt | 4 + include/strata/core/verify.hpp | 1 + include/strata/kernels/iq_kernels.hpp | 3 + include/strata/kernels/native_mmvq.hpp | 13 + src/core/verify.cpp | 43 ++-- src/kernels/cuda/iq_dot.cuh | 54 ++++ src/kernels/cuda/iq_kernels.cu | 61 +++++ src/kernels/cuda/native_mmvq.cu | 327 +++++++++++++++++++++++++ src/kernels/cuda/q8_1_il.cuh | 78 ++++++ src/kernels/mmvq_il_parity.cu | 196 +++++++++++++++ 10 files changed, 766 insertions(+), 14 deletions(-) create mode 100644 src/kernels/cuda/q8_1_il.cuh create mode 100644 src/kernels/mmvq_il_parity.cu diff --git a/CMakeLists.txt b/CMakeLists.txt index 2249b9cbde3..afcb9b258a5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -372,6 +372,10 @@ if(STRATA_ENABLE_CUDA) add_executable(qsa_select_parity src/kernels/qsa_select_parity.cu) target_link_libraries(qsa_select_parity PRIVATE strata_kernels CUDA::cudart) add_test(NAME qsa_select_parity COMMAND qsa_select_parity) + # The MMVQ of 2..8 tokens from interleaved activations against native_mmvq: bitwise (synthetic matrices). + add_executable(mmvq_il_parity src/kernels/mmvq_il_parity.cu) + target_link_libraries(mmvq_il_parity PRIVATE strata_kernels CUDA::cudart) + add_test(NAME mmvq_il_parity COMMAND mmvq_il_parity) # Plan v0.3 P3: native MMVQ for 1..8 tokens (bitwise selftest + --bench timing); synthetic data, no model. if(STRATA_BUILD_TESTS) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index c705dc2e365..6c50afba3db 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -194,6 +194,7 @@ class Verifier { float *ple_ = nullptr, *emb_ = nullptr, *R_ = nullptr, *mixed_ = nullptr, *bo_ = nullptr; float *inj_ = nullptr, *inj2_ = nullptr, *lo_ = nullptr, *rs_ = nullptr, *xn_ = nullptr; uint8_t* xq_ = nullptr; // T columns of q8_1 + uint8_t* xil_ = nullptr; // their interleaved copy (2+ tokens) float *qkv_L_ = nullptr, *h_L_ = nullptr, *gate_L_ = nullptr, *beta_L_ = nullptr; // per GDN layer float *z_ = nullptr, *y_ = nullptr, *y_dummy_ = nullptr; float *qfull_ = nullptr, *qcur_ = nullptr, *kcur_ = nullptr, *vcur_ = nullptr, *idx_raw_L_ = nullptr; diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index 19faf396807..c98c7454717 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -22,6 +22,9 @@ void quantize_q8_1_rows(const float* x, int64_t n_rows, int64_t n_cols, void* y, /// y[c][r] = W[r] . x[c] for `ncols` columns of q8_1 activations (x stride n_in/32 blocks per column). void iq_mmvq(int ggml_type, const void* w, const void* x_q8_1, float* y, int n_in, int n_out, int ncols, void* stream); +/// iq_mmvq for 2..8 columns from their interleaved copy (native_quantize_q8_1_il), bitwise: IQ3_S. +bool iq_mmvq_il_supported(int ggml_type) noexcept; +void iq_mmvq_il(int ggml_type, const void* w, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream); /// Dequantize `n` contiguous values (n a multiple of 256) to fp16 / fp32. void iq_dequant_f16(int ggml_type, const void* src, int64_t n, uint16_t* dst, void* stream); diff --git a/include/strata/kernels/native_mmvq.hpp b/include/strata/kernels/native_mmvq.hpp index 9f69dd940cf..2b8c14e8e8f 100644 --- a/include/strata/kernels/native_mmvq.hpp +++ b/include/strata/kernels/native_mmvq.hpp @@ -109,4 +109,17 @@ std::size_t native_mmvq_weight_bytes(int ggml_type, int n_in, int n_out); void native_mmvq(int ggml_type, const void* weights, const void* x_q8_1, float* y, int n_in, int n_out, int ncols, void* stream); +// Plan v0.3 P6: 2..8 columns also INTERLEAVED, so one 16-byte load reads a position of every column: CP = 2 (2 +// columns), 4 (3-4) or 8 (5-8), zero-padded; nb = n_in / 32 blocks; int32 bm[nb][8][CP] (block-major), int32 +// pm[8][nb][CP] (position-major), float d[nb][CP] (each block's scale as the dots read it, __low2float of ds). +constexpr int native_q8_1_il_cp(int ncols) { return ncols <= 2 ? 2 : ncols <= 4 ? 4 : 8; } +std::size_t native_q8_1_il_bytes(int n_in, int ncols); +// native_quantize_q8_1 that also writes the interleaved copy to x_il (native_q8_1_il_bytes); 2 <= ncols <= 8. +void native_quantize_q8_1_il(const float* x, void* x_q8_1, void* x_il, int n_in, int ncols, void* stream); +// native_mmvq for 2..8 columns quantized by native_quantize_q8_1_il, bitwise native_mmvq's output: kernels that read +// the interleaved copy for IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0 and IQ3_S matrices of at least 2048 rows, native_mmvq +// otherwise. +void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, + int n_out, int ncols, void* stream); + } // namespace strata::kernels diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 8ee2bbb421a..a232a052ab0 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -216,6 +216,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& inj_ = b.take(T * HC); inj2_ = b.take(T * HC); lo_ = b.take(T * (uint64_t) g.hc_lr); rs_ = b.take(T * HC); xn_ = b.take(T * HC * N); xq_ = b.take(strata::kernels::native_q8_1_bytes(max_in, (int) T)); + xil_ = b.take(strata::kernels::native_q8_1_il_bytes(max_in, (int) T)); qkv_L_ = b.take(nG * T * C); h_L_ = b.take(nG * T * C); gate_L_ = b.take(nG * T * HV); beta_L_ = b.take(nG * T * HV); z_ = b.take(T * ZV); y_ = b.take(T * ZV); y_dummy_ = b.take(T * ZV); @@ -474,6 +475,16 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { gr_read_group(0, pending, inj2_, inj_); stamp(l, 1, cs); float* xm = mixed_ + tb * N; + // the mixer's projections: a group of 2+ tokens is quantized with the interleaved copy the multi-token kernels + // read + auto quant = [&](const float* x, int64_t n_in) { + if (n >= 2) native_quantize_q8_1_il(x, xq_, xil_, (int) n_in, n, cs); + else native_quantize_q8_1(x, xq_, (int) n_in, n, cs); + }; + auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out) { + if (n >= 2) native_mmvq_il(w->native_type, w->native_data, xq_, xil_, y, (int) n_in, (int) n_out, n, cs); + else native_mmvq(w->native_type, w->native_data, xq_, y, (int) n_in, (int) n_out, n, cs); + }; try { if (!is_qsa_layer(g, l)) { // ======================= GDN ======================= @@ -493,18 +504,18 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { float* hb = h_L_ + (size_t) gi * MT * C; float* gate = gate_L_ + (size_t) gi * MT * HV; float* beta = beta_L_ + (size_t) gi * MT * HV; - native_quantize_q8_1(xm, xq_, (int) N, n, cs); - native_mmvq(wqkv->native_type, wqkv->native_data, xq_, qkv + (size_t) tb * C, (int) N, (int) C, n, cs); + quant(xm, N); + proj(wqkv, qkv + (size_t) tb * C, N, C); gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb); gdn_ab_multi(xm, (const uint16_t*) wa->data, (const uint16_t*) wb->data, (const float*) wdt->data, (const float*) wsa->data, gate + (size_t) tb * HV, beta + (size_t) tb * HV, (int) N, (int) HV, n, cs); - native_mmvq(wg->native_type, wg->native_data, xq_, z_ + (size_t) tb * ZV, (int) N, (int) ZV, n, cs); + proj(wg, z_ + (size_t) tb * ZV, N, ZV); // the recurrence from the untouched state over tokens [0, te); outputs only for this group's gdn_step_norm_multi(state, hb, (int) C, gate, beta, z_, (const float*) wnm->data, EPS, y_, (int) HK, (int) HV, te, nullptr, cs, tb); - native_quantize_q8_1(y_ + (size_t) tb * ZV, xq_, (int) ZV, n, cs); - native_mmvq(wout->native_type, wout->native_data, xq_, bo_ + tb * N, (int) ZV, (int) N, n, cs); + quant(y_ + (size_t) tb * ZV, ZV); + proj(wout, bo_ + tb * N, ZV, N); } else { // ======================= QSA ======================= const int64_t qi = qsa_idx[(size_t) l]; @@ -532,10 +543,10 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { (int) NH, cs); }; float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; - native_quantize_q8_1(xm, xq_, (int) N, n, cs); + quant(xm, N); bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, cs); - native_mmvq(wk->native_type, wk->native_data, xq_, kcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); - native_mmvq(wv->native_type, wv->native_data, xq_, vcur_ + tb * NKV * HD, (int) N, (int) (NKV * HD), n, cs); + proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD); + proj(wv, vcur_ + tb * NKV * HD, N, NKV * HD); if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH); else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH); if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, cs); @@ -550,8 +561,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { native_qsa_indexer_append_multi(idx_raw + tb * ID, step_ + tb * kStepCount + kStepPos, (int) kStepCount, n, 0, (const float*) wikn->data, EPS, ib, s, st.max_cells, (float) qsa_freq_base(), cs); - native_mmvq(wq->native_type, wq->native_data, xq_, qfull_ + tb * NH * 2 * HD, (int) N, (int) (NH * 2 * HD), - n, cs); + proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD); // the query halves of every token's heads (rows of 2 * HD: query, then gate) if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, (size_t) HD * 4, (size_t) (n * NH), cudaMemcpyDeviceToDevice, cs) != cudaSuccess) { @@ -579,8 +589,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { else for (int t = tb; t < te; ++t) qsa_gate_apply_f32(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, s, attn32_ + t * NH * HD, cs); - native_quantize_q8_1(attn32_ + tb * NH * HD, xq_, (int) (NH * HD), n, cs); - native_mmvq(wo->native_type, wo->native_data, xq_, bo_ + tb * N, (int) (NH * HD), (int) N, n, cs); + quant(attn32_ + tb * NH * HD, NH * HD); + proj(wo, bo_ + tb * N, NH * HD, N); } } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + ": " + e.what(); @@ -772,8 +782,13 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } if (head_ != nullptr && head_->loaded()) { try { - native_quantize_q8_1(head_mixed_, xq_, (int) N, T, cs); - native_mmvq(head_->type(), head_->weights(), xq_, head_logits_, (int) N, (int) n_vocab_, T, cs); + if (T >= 2) { + native_quantize_q8_1_il(head_mixed_, xq_, xil_, (int) N, T, cs); + native_mmvq_il(head_->type(), head_->weights(), xq_, xil_, head_logits_, (int) N, (int) n_vocab_, T, cs); + } else { + native_quantize_q8_1(head_mixed_, xq_, (int) N, T, cs); + native_mmvq(head_->type(), head_->weights(), xq_, head_logits_, (int) N, (int) n_vocab_, T, cs); + } } catch (const std::exception& e) { err = std::string("verify head: ") + e.what(); return false; diff --git a/src/kernels/cuda/iq_dot.cuh b/src/kernels/cuda/iq_dot.cuh index b5aa432c85a..e53da235423 100644 --- a/src/kernels/cuda/iq_dot.cuh +++ b/src/kernels/cuda/iq_dot.cuh @@ -216,6 +216,60 @@ __device__ __forceinline__ float vec_dot_iq3_s_q8_1(const void* __restrict__ vbq return d * sumi; } +// vec_dot_iq3_s_q8_1 for several activation columns: the grid values, masks and scale decoded once (`load`), each +// column's integer sums and float expression as above (`apply`; X::u(b, p, u) gives int p of q8_1 block b of every +// column, X::scales(b, d) their __low2float scales). +struct IQ3SCols { + uint32_t g[8], gm[8]; // grid words, and their negative values' bytes + int sc; + float dw; + __device__ __forceinline__ void load(const void* __restrict__ vbq, const int& kbx, const int& iqs) { + const block_iq3_s* bq3 = (const block_iq3_s*) vbq + kbx; + const int2 qs_packed = make_int2(get_int_b2(bq3->qs, iqs + 0), get_int_b2(bq3->qs, iqs + 1)); + const uint8_t* qs = (const uint8_t*) &qs_packed; + const int qh = bq3->qh[iqs / 2]; + const int signs_packed_32 = get_int_b2(bq3->signs, iqs / 2); + const uint8_t* signs_packed_8 = (const uint8_t*) &signs_packed_32; +#pragma unroll + for (int l0 = 0; l0 < 8; l0 += 2) { + const uint32_t gx = iq3s_grid[qs[l0 + 0] | ((qh << (8 - l0)) & 0x100)]; + const uint32_t gy = iq3s_grid[qs[l0 + 1] | ((qh << (7 - l0)) & 0x100)]; + const uint2 m = sign_masks(signs_packed_8[l0 / 2]); + g[l0] = gx; + gm[l0] = gx & m.x; + g[l0 + 1] = gy; + gm[l0 + 1] = gy & m.y; + } + sc = 1 + 2 * ((bq3->scales[iqs / 4] >> ((iqs << 1) & 0x04)) & 0x0F); + dw = __half2float(bq3->d); + } + template + __device__ __forceinline__ void apply(const X& x, const int& b, float (&out)[NC]) const { + int sp[NC], sn[NC]; +#pragma unroll + for (int c = 0; c < NC; ++c) sp[c] = sn[c] = 0; +#pragma unroll + for (int l = 0; l < 8; ++l) { + int u[NC]; + x.u(b, l, u); +#pragma unroll + for (int c = 0; c < NC; ++c) { + sp[c] = ggml_cuda_dp4a((int) g[l], u[c], sp[c]); + sn[c] = ggml_cuda_dp4a((int) gm[l], u[c], sn[c]); + } + } + float d8[NC]; + x.scales(b, d8); +#pragma unroll + for (int c = 0; c < NC; ++c) { + int sumi = sp[c] - 2 * sn[c]; + sumi *= sc; + const float d = dw * d8[c]; + out[c] = d * sumi; + } + } +}; + __device__ __forceinline__ float vec_dot_iq1_m_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1, const int& kbx, const int& iqs) { const block_iq1_m* bq1 = (const block_iq1_m*) vbq + kbx; diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index 70b0d9bec37..0013b1c132c 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -5,6 +5,7 @@ // in iq_dot.cuh. #include "strata/kernels/iq_kernels.hpp" #include "iq_dot.cuh" +#include "q8_1_il.cuh" #include #include @@ -32,6 +33,45 @@ __global__ void __launch_bounds__(128) mmvq_kernel(const uint8_t* __restrict__ w } } +// mmvq_kernel for NC columns from their interleaved (position-major) copy: a warp takes R rows, decodes each call's +// weights once and keeps row_dot's lane-strided sum per (row, column), so every value is bitwise mmvq_kernel's. +template +__global__ void __launch_bounds__(128) mmvq_il_iq3_s_kernel(const uint8_t* __restrict__ w, size_t row_bytes, + const int* __restrict__ xq, const float* __restrict__ xd, + float* __restrict__ y, int n_in, int n_out) { + using F = Fmt<21>; + const int lane = threadIdx.x & 31; + const int row0 = (blockIdx.x * 4 + (threadIdx.x >> 5)) * R; + if (row0 >= n_out) return; + const Q81Il x{xq, xd, n_in / 32}; + const int nb = n_in / F::qk; + float acc[R][NC]; +#pragma unroll + for (int i = 0; i < R; ++i) +#pragma unroll + for (int c = 0; c < NC; ++c) acc[i][c] = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); +#pragma unroll + for (int i = 0; i < R; ++i) { + const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row + IQ3SCols wv; + wv.load(w + (size_t) row * row_bytes, kbx, iqs); + float o[NC]; + wv.apply(x, kbx * (F::qk / 32) + iqs / 2, o); +#pragma unroll + for (int c = 0; c < NC; ++c) acc[i][c] += o[c]; + } + } +#pragma unroll + for (int i = 0; i < R; ++i) +#pragma unroll + for (int c = 0; c < NC; ++c) { + const float s = warp_sum(acc[i][c]); + if (lane == 0 && row0 + i < n_out) y[(size_t) c * n_out + row0 + i] = s; + } +} + // ---------------------------------------------------------------- grouped native experts // Two kernels a call. The first computes a group's gate and up rows for one chunk of 32 rows of n_ff - one q8_1 block // of h - SwiGLU and that block for each entry; the second the down rows. A warp takes several rows at once, one @@ -531,6 +571,27 @@ void iq_mmvq(int t, const void* w, const void* x_q8_1, float* y, int n_in, int n check("iq_mmvq"); } +bool iq_mmvq_il_supported(int t) noexcept { return t == 21; } + +void iq_mmvq_il(int t, const void* w, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream) { + if (t != 21 || ncols < 2 || ncols > 8 || n_in % 256 != 0) { + std::fprintf(stderr, "iq_mmvq_il: type %d, %d columns are not supported\n", t, ncols); + std::exit(1); + } + constexpr int R = 2; + const Q81IlParts parts = q8_1_il_parts(x_il, n_in, ncols); + const size_t rb = iq_row_bytes(t, n_in); + const auto* W = (const uint8_t*) w; + const unsigned blocks = (unsigned) (((n_out + R - 1) / R + 3) / 4); + cudaStream_t s = (cudaStream_t) stream; + switch (ncols) { +#define STRATA_IL(N) case N: mmvq_il_iq3_s_kernel<<>>(W, rb, parts.pm, parts.d, y, n_in, n_out); break; + STRATA_IL(2) STRATA_IL(3) STRATA_IL(4) STRATA_IL(5) STRATA_IL(6) STRATA_IL(7) STRATA_IL(8) +#undef STRATA_IL + } + check("iq_mmvq_il"); +} + void iq_dequant_f16(int t, const void* src, int64_t n, uint16_t* dst, void* stream) { if (n % 256 != 0 || !is_iq(t)) { std::fprintf(stderr, "iq_dequant_f16: bad arguments\n"); std::exit(1); } dequant_flat_kernel<__half><<<(unsigned) (n / 256), 32, 0, (cudaStream_t) stream>>>(t, src, (__half*) dst); diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu index a7801fa7250..7495662564d 100644 --- a/src/kernels/cuda/native_mmvq.cu +++ b/src/kernels/cuda/native_mmvq.cu @@ -25,6 +25,7 @@ #include "strata/kernels/native_mmvq.hpp" #include "strata/kernels/iq_kernels.hpp" +#include "q8_1_il.cuh" #include #include @@ -1076,6 +1077,274 @@ void launch_multi(const void* weights, const void* x_q8_1, float* y, int n_in, i } } +// ============================ plan v0.3 P6: ncols = 2..8 from interleaved activations ============================ +// +// `native_quantize_q8_1_il` also writes the columns interleaved (native_mmvq.hpp), so one load reads the same int of +// every column. A warp takes R rows and walks their blocks in the chunks of the exact layout above - its lane l's chunk +// m is what thread 32 (m % 4) + l does in iteration m / 4 - with a sum per virtual warp, added in warp order before the +// xor tree; each weight block is decoded once (the traits' `load`) and every column takes the single-column dot's +// arithmetic, so each value is bitwise the multi-column kernel's. Formats whose lanes read consecutive ints of one q8_1 +// block read the block-major copy; IQ4_XS, whose lanes read the same int of consecutive blocks, the position-major one. +// +// Per format: the traits' blocks and `load`, and `apply` for NC columns (kby = the columns' first q8_1 block of the +// weight block): the single-column `apply`'s expression per column. +struct IlIQ4XS { + using F = IQ4XSTraits; + static constexpr bool PM = true; + template + __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + const int b = kby + iqs / 4; + int sumi[NC]; +#pragma unroll + for (int c = 0; c < NC; ++c) sumi[c] = 0; +#pragma unroll + for (int j = 0; j < 4; ++j) { + int u0[NC], u1[NC]; + x.u(b, j, u0); + x.u(b, j + 4, u1); +#pragma unroll + for (int c = 0; c < NC; ++c) { + sumi[c] = __dp4a(r.v[j].x, u0[c], sumi[c]); + sumi[c] = __dp4a(r.v[j].y, u1[c], sumi[c]); + } + } + float d8[NC]; + x.scales(b, d8); +#pragma unroll + for (int c = 0; c < NC; ++c) { + const int s = sumi[c] * (r.ls - 32); + const float d = r.dw * d8[c]; + out[c] = d * s; + } + } +}; +struct IlQ4K { + using F = Q4KTraits; + static constexpr bool PM = false; + template + __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + int u[4][NC]; + float d8[2][NC]; +#pragma unroll + for (int i = 0; i < 2; ++i) { + const int b = kby + r.bq8_offset + i; + x.u(b, (iqs / 2) % 4, u[2 * i]); + x.u(b, (iqs / 2) % 4 + 4, u[2 * i + 1]); + x.scales(b, d8[i]); + } + const uint8_t* sc = reinterpret_cast(r.aux); + const uint8_t* m = sc + 2; + const float2 dm4f = __half22float2(r.dm); +#pragma unroll + for (int c = 0; c < NC; ++c) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < 2; ++i) { + const int v0i = (r.v[0] >> (4 * i)) & 0x0f0f0f0f; + const int v1i = (r.v[1] >> (4 * i)) & 0x0f0f0f0f; + const int dot1 = __dp4a(v1i, u[2 * i + 1][c], __dp4a(v0i, u[2 * i][c], 0)); + const int dot2 = __dp4a(0x01010101, u[2 * i + 1][c], __dp4a(0x01010101, u[2 * i][c], 0)); + sumf_d += d8[i][c] * (dot1 * sc[i]); + sumf_m += d8[i][c] * (dot2 * m[i]); + } + out[c] = dm4f.x * sumf_d - dm4f.y * sumf_m; + } + } +}; +struct IlQ5K { + using F = Q5KTraits; + static constexpr bool PM = false; + template + __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + int u[4][NC]; + float d8[2][NC]; +#pragma unroll + for (int i = 0; i < 2; ++i) { + const int b = kby + r.bq8_offset + i; + x.u(b, (iqs / 2) % 4, u[2 * i]); + x.u(b, (iqs / 2) % 4 + 4, u[2 * i + 1]); + x.scales(b, d8[i]); + } + const uint8_t* sc = reinterpret_cast(r.aux); + const uint8_t* m = sc + 2; + const float2 dm5f = __half22float2(r.dm); +#pragma unroll + for (int c = 0; c < NC; ++c) { + float sumf_d = 0.0f; + float sumf_m = 0.0f; +#pragma unroll + for (int i = 0; i < 2; ++i) { + const int vl0i = (r.vl[0] >> (4 * i)) & 0x0f0f0f0f; + const int vl1i = (r.vl[1] >> (4 * i)) & 0x0f0f0f0f; + const int vh0i = ((r.vh[0] >> i) << 4) & 0x10101010; + const int vh1i = ((r.vh[1] >> i) << 4) & 0x10101010; + const int v0i = vl0i | vh0i; + const int v1i = vl1i | vh1i; + const int dot1 = __dp4a(v0i, u[2 * i][c], __dp4a(v1i, u[2 * i + 1][c], 0)); + const int dot2 = __dp4a(0x01010101, u[2 * i][c], __dp4a(0x01010101, u[2 * i + 1][c], 0)); + sumf_d += d8[i][c] * (dot1 * sc[i]); + sumf_m += d8[i][c] * (dot2 * m[i]); + } + out[c] = dm5f.x * sumf_d - dm5f.y * sumf_m; + } + } +}; +struct IlQ6K { + using F = Q6KTraits; + static constexpr bool PM = false; + template + __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + int u[2][NC]; + float d8[2][NC]; +#pragma unroll + for (int i = 0; i < 2; ++i) { + x.u(kby + r.bq8_offset + 2 * i, iqs % 8, u[i]); + x.scales(kby + r.bq8_offset + 2 * i, d8[i]); + } + int vi[2], sc[2]; +#pragma unroll + for (int i = 0; i < 2; ++i) { + sc[i] = r.scales[4 * i]; + const int vil = (r.vl >> (4 * i)) & 0x0f0f0f0f; + const int vih = ((r.vh >> (4 * i)) << 4) & 0x30303030; + vi[i] = __vsubss4(vil | vih, 0x20202020); + } +#pragma unroll + for (int c = 0; c < NC; ++c) { + float sumf = 0.0f; +#pragma unroll + for (int i = 0; i < 2; ++i) sumf += d8[i][c] * (__dp4a(vi[i], u[i][c], 0) * sc[i]); + out[c] = r.d * sumf; + } + } +}; +// Q8_0: its traits keep the block pointer (`small_q8_dot` loads it per column); here the codes and scale load once. +struct IlQ80 { + using F = SmallTraits; + static constexpr bool PM = false; + struct W { int v[2]; float d0; }; + __device__ static W load(const Q80Block* __restrict__ w, int iqs) { + W r; + r.v[0] = load_int_b2(w->qs, iqs); + r.v[1] = load_int_b2(w->qs, iqs + 1); + r.d0 = w->d; + return r; + } + template + __device__ static void apply(const W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + int u0[NC], u1[NC]; + x.u(kby, iqs, u0); + x.u(kby, iqs + 1, u1); + float d8[NC]; + x.scales(kby, d8); +#pragma unroll + for (int c = 0; c < NC; ++c) { + int sumi = 0; + sumi = __dp4a(r.v[0], u0[c], sumi); + sumi = __dp4a(r.v[1], u1[c], sumi); + const float d0 = r.d0; + const float d1 = d8[c]; + out[c] = d0 * d1 * float(sumi); + } + } +}; +template struct IlLoad { + using W = typename I::F::W; + __device__ static W load(const typename I::F::Block* w, int kqs) { return I::F::load(w, kqs); } +}; +template <> struct IlLoad { + using W = IlQ80::W; + __device__ static W load(const Q80Block* w, int kqs) { return IlQ80::load(w, kqs); } +}; + +template +__launch_bounds__(4 * WARP) +__global__ void native_mmvq_il_kernel(const typename I::F::Block* __restrict__ w, const int* __restrict__ xq, + const float* __restrict__ xd, float* __restrict__ y, int n_in, int n_out) { + using F = typename I::F; + constexpr int SUB = WARP / F::T; // weight blocks a chunk covers + const int lane = int(threadIdx.x) & (WARP - 1); + const int row0 = (int(blockIdx.x) * 4 + (int(threadIdx.x) >> 5)) * R; + if (row0 >= n_out) return; + const Q81Il x{xq, xd, n_in / Q8K}; + const int bpr = n_in / F::DIV; + const int kqs = F::kqs(lane); + const int nm = (bpr + SUB - 1) / SUB; + float acc[4][R][NC]; +#pragma unroll + for (int v = 0; v < 4; ++v) +#pragma unroll + for (int i = 0; i < R; ++i) +#pragma unroll + for (int c = 0; c < NC; ++c) acc[v][i][c] = 0.0f; + for (int m0 = 0; m0 < nm; m0 += 4) { +#pragma unroll + for (int v = 0; v < 4; ++v) { + const int kbx = SUB * (m0 + v) + lane / F::T; + if (kbx < bpr) { +#pragma unroll + for (int i = 0; i < R; ++i) { + const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row + const typename IlLoad::W wv = IlLoad::load(w + std::size_t(row) * bpr + kbx, kqs); + float o[NC]; + I::template apply(wv, x, kbx * F::KBY, kqs, o); +#pragma unroll + for (int c = 0; c < NC; ++c) acc[v][i][c] += o[c]; + } + } + } + } +#pragma unroll + for (int i = 0; i < R; ++i) +#pragma unroll + for (int c = 0; c < NC; ++c) { + float s = acc[0][i][c]; + s += acc[1][i][c]; + s += acc[2][i][c]; + s += acc[3][i][c]; + s = warp_sum(s); + if (lane == 0 && row0 + i < n_out) y[std::size_t(c) * n_out + row0 + i] = s; + } +} + +template +void launch_il(const void* weights, const void* x_il, float* y, int n_in, int n_out, int ncols, cudaStream_t s) { + const Q81IlParts parts = q8_1_il_parts(x_il, n_in, ncols); + const int* xq = I::PM ? parts.pm : parts.bm; + const float* xd = parts.d; + const auto* w = static_cast(weights); + const unsigned blocks = unsigned(((n_out + R - 1) / R + 3) / 4); + switch (ncols) { +#define STRATA_IL_CASE(N) case N: native_mmvq_il_kernel<<>>(w, xq, xd, y, n_in, n_out); break; + STRATA_IL_CASE(2) STRATA_IL_CASE(3) STRATA_IL_CASE(4) STRATA_IL_CASE(5) STRATA_IL_CASE(6) STRATA_IL_CASE(7) + STRATA_IL_CASE(8) +#undef STRATA_IL_CASE + default: throw std::invalid_argument("native_mmvq_il requires 2 <= ncols <= 8"); + } +} + +// native_quantize_q8_1_kernel plus the interleaved copy (native_mmvq.hpp): the same values in the same bytes +__launch_bounds__(QUANT_THREADS, 1) +__global__ void native_quantize_q8_1_il_kernel(const float* __restrict__ x, Q81Block* __restrict__ y, + int8_t* __restrict__ bm, int8_t* __restrict__ pm, float* __restrict__ dl, + int n_in, int n_total, int cp) { + const int i = int(blockIdx.x) * QUANT_THREADS + int(threadIdx.x); + if (i >= n_total) return; // n_in is a multiple of 32: only whole warps return. + const float xi = x[i]; + const float amax = warp_max(fabsf(xi)); + const float sum = warp_sum(xi); + const float d = amax / 127.0f; + const int8_t q = amax == 0.0f ? 0 : roundf(xi / d); + const half2 ds = make_half2(d, sum); + y[i / Q8K].qs[i % Q8K] = q; + if (i % Q8K == 0) y[i / Q8K].ds = ds; + const int c = i / n_in, e = i - c * n_in, b = e / Q8K, p = (e % Q8K) / 4, nb = n_in / Q8K; + bm[(std::size_t(b * 8 + p) * cp + c) * 4 + e % 4] = q; + pm[(std::size_t(p * nb + b) * cp + c) * 4 + e % 4] = q; + if (e % Q8K == 0) dl[std::size_t(b) * cp + c] = __low2float(ds); +} + void validate_shape(int n_in, int ncols, int block_elems = Q8K) { if (n_in <= 0 || n_in % block_elems != 0) { throw std::invalid_argument("native MMVQ requires n_in > 0 and divisible by its block element count"); @@ -1489,4 +1758,62 @@ void native_mmvq(int ggml_type, const void* weights, const void* x_q8_1, float* } } +std::size_t native_q8_1_il_bytes(int n_in, int ncols) { + validate_shape(n_in, ncols); + return std::size_t(n_in / Q8K) * std::size_t(native_q8_1_il_cp(ncols)) * (16 * sizeof(int) + sizeof(float)); +} + +void native_quantize_q8_1_il(const float* x, void* x_q8_1, void* x_il, int n_in, int ncols, void* stream) { + validate_shape(n_in, ncols); + if (ncols < 2) throw std::invalid_argument("native_quantize_q8_1_il requires 2 <= ncols <= 8"); + validate_pointer(x); + validate_pointer(x_q8_1); + validate_pointer(x_il); + validate_stream(stream); + const int nb = n_in / Q8K, cp = native_q8_1_il_cp(ncols); + auto* bm = static_cast(x_il); + auto* pm = bm + std::size_t(nb) * 8 * cp * 4; + auto* dl = reinterpret_cast(pm + std::size_t(nb) * 8 * cp * 4); + const int n_total = n_in * ncols; + const unsigned blocks = unsigned((std::size_t(n_total) + QUANT_THREADS - 1) / QUANT_THREADS); + native_quantize_q8_1_il_kernel<<(stream)>>>( + x, static_cast(x_q8_1), bm, pm, dl, n_in, n_total, cp); + launch_check(); +} + +void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, + int n_out, int ncols, void* stream) { + // below ~2048 rows the warps are too few for one row each: the multi-column kernel spreads a row over four (and + // for Q8_0 2 columns of fewer than 4096 rows) + const bool il = ggml_type == 23 || ggml_type == 12 || ggml_type == 13 || ggml_type == 14 || ggml_type == 8; + if (ncols < 2 || n_out < 2048 || (ggml_type == 8 && ncols == 2 && n_out < 4096) || + (!il && !iq_mmvq_il_supported(ggml_type))) { + native_mmvq(ggml_type, weights, x_q8_1, y, n_in, n_out, ncols, stream); + return; + } + if (!il) { + iq_mmvq_il(ggml_type, weights, x_il, y, n_in, n_out, ncols, stream); + return; + } + validate_shape(n_in, ncols, ggml_type == 8 ? Q8K : QK); + validate_pointer(weights); + validate_pointer(x_il); + validate_pointer(y); + validate_stream(stream); + const auto s = static_cast(stream); + // rows a warp: 2, and 4 for Q4_K and Q6_K from 8192 rows (the faster on the model files' shapes, mmvq_il_parity + // --bench) + const bool wide = n_out >= 8192; + switch (ggml_type) { + case 23: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + case 12: wide ? launch_il(weights, x_il, y, n_in, n_out, ncols, s) + : launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + case 13: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + case 14: wide ? launch_il(weights, x_il, y, n_in, n_out, ncols, s) + : launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + case 8: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + } + launch_check(); +} + } // namespace strata::kernels diff --git a/src/kernels/cuda/q8_1_il.cuh b/src/kernels/cuda/q8_1_il.cuh new file mode 100644 index 00000000000..0034893a6b5 --- /dev/null +++ b/src/kernels/cuda/q8_1_il.cuh @@ -0,0 +1,78 @@ +// src/kernels/cuda/q8_1_il.cuh - the interleaved copy of 2..8 q8_1 activation columns (native_mmvq.hpp) as the +// multi-column dot products read it, for native_mmvq.cu and iq_kernels.cu. +#pragma once + +#include "strata/kernels/native_mmvq.hpp" + +#include + +#include + +namespace strata::kernels { +namespace { + +// NC columns of the block-major (PM false) or position-major copy: u(b, p, o) = int p of q8_1 block b of every column, +// scales(b, o) = their scales +template struct Q81Il { + static constexpr int CP = native_q8_1_il_cp(NC); + const int* __restrict__ q; + const float* __restrict__ d; + int nb; // blocks a column + __device__ __forceinline__ void u(int b, int p, int (&o)[NC]) const { + const int* s = q + std::size_t(PM ? p * nb + b : b * 8 + p) * CP; + if constexpr (CP == 2) { + const int2 a = *reinterpret_cast(s); + o[0] = a.x; + o[1] = a.y; + } else { + const int4 a = *reinterpret_cast(s); + o[0] = a.x; + if constexpr (NC > 1) o[1] = a.y; + if constexpr (NC > 2) o[2] = a.z; + if constexpr (NC > 3) o[3] = a.w; + if constexpr (NC > 4) { + const int4 e = *reinterpret_cast(s + 4); + o[4] = e.x; + if constexpr (NC > 5) o[5] = e.y; + if constexpr (NC > 6) o[6] = e.z; + if constexpr (NC > 7) o[7] = e.w; + } + } + } + __device__ __forceinline__ void scales(int b, float (&o)[NC]) const { + const float* s = d + std::size_t(b) * CP; + if constexpr (CP == 2) { + const float2 a = *reinterpret_cast(s); + o[0] = a.x; + o[1] = a.y; + } else { + const float4 a = *reinterpret_cast(s); + o[0] = a.x; + if constexpr (NC > 1) o[1] = a.y; + if constexpr (NC > 2) o[2] = a.z; + if constexpr (NC > 3) o[3] = a.w; + if constexpr (NC > 4) { + const float4 e = *reinterpret_cast(s + 4); + o[4] = e.x; + if constexpr (NC > 5) o[5] = e.y; + if constexpr (NC > 6) o[6] = e.z; + if constexpr (NC > 7) o[7] = e.w; + } + } + } +}; + +// The copy's parts inside the buffer native_q8_1_il_bytes sizes: bm, pm (int32) and d (float). +struct Q81IlParts { + const int* bm; + const int* pm; + const float* d; +}; +inline Q81IlParts q8_1_il_parts(const void* x_il, int n_in, int ncols) { + const std::size_t words = std::size_t(n_in / 32) * 8 * std::size_t(native_q8_1_il_cp(ncols)); + const int* bm = static_cast(x_il); + return {bm, bm + words, reinterpret_cast(bm + 2 * words)}; +} + +} // namespace +} // namespace strata::kernels diff --git a/src/kernels/mmvq_il_parity.cu b/src/kernels/mmvq_il_parity.cu new file mode 100644 index 00000000000..a7fc9e04bf3 --- /dev/null +++ b/src/kernels/mmvq_il_parity.cu @@ -0,0 +1,196 @@ +// src/kernels/mmvq_il_parity.cu - native_mmvq_il (2..8 columns from the interleaved q8_1 copy) against native_mmvq, +// bitwise, on synthetic matrices of the model files' dense formats and shapes; the interleaving quantizer's q8_1 blocks +// against native_quantize_q8_1's. --bench times both inside CUDA graphs (weights cycled past the L2). +#include "strata/kernels/native_mmvq.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace strata::kernels; + +#define CK(x) do { const cudaError_t e_ = (x); if (e_ != cudaSuccess) { std::printf("%s: %s\n", #x, cudaGetErrorString(e_)); std::exit(1); } } while (0) + +namespace { + +struct Fmt { int type; const char* name; int block_elems, block_bytes, d_off, dmin_off; }; +// the dense formats of the IQ3_XXS and UD-Q4_K_XL files; d_off: the fp16 block scale (random bytes elsewhere) +const Fmt kFmts[] = { + {23, "IQ4_XS", 256, 136, 0, -1}, {12, "Q4_K", 256, 144, 0, 2}, {13, "Q5_K", 256, 176, 0, 2}, + {14, "Q6_K", 256, 210, 208, -1}, {8, "Q8_0", 32, 34, 0, -1}, {21, "IQ3_S", 256, 110, 0, -1}, +}; + +cudaStream_t g_s; +std::mt19937 g_rng(29); + +void* dev_weights(const Fmt& f, int n_in, int n_out) { + const size_t bytes = native_mmvq_weight_bytes(f.type, n_in, n_out); + std::vector h(bytes); + for (auto& b : h) b = (uint8_t) g_rng(); + std::uniform_int_distribution e(0x1c00, 0x2400); // fp16 scales in ~[0.004, 0.016] + for (size_t o = 0; o < bytes; o += (size_t) f.block_bytes) { + const uint16_t d = (uint16_t) e(g_rng); + std::memcpy(&h[o + f.d_off], &d, 2); + if (f.dmin_off >= 0) { const uint16_t m = (uint16_t) e(g_rng); std::memcpy(&h[o + f.dmin_off], &m, 2); } + } + void* p; + CK(cudaMalloc(&p, bytes)); + CK(cudaMemcpy(p, h.data(), bytes, cudaMemcpyHostToDevice)); + CK(cudaDeviceSynchronize()); // a pageable copy may return before its DMA lands + return p; +} + +struct Act { + float* x = nullptr; + void *q_ref = nullptr, *q = nullptr, *il = nullptr; + int n_in = 0; +}; +Act make_act(int n_in) { + Act a; + a.n_in = n_in; + std::vector h((size_t) 8 * n_in); + std::normal_distribution nd(0.0f, 1.0f); + for (auto& v : h) v = nd(g_rng); + for (size_t i = 0; i < h.size(); i += 97) h[i] *= 20.0f; // outliers set some blocks' scales + for (int j = 0; j < 32; ++j) h[(size_t) 3 * n_in + 64 + j] = 0.0f; // an all-zero block + CK(cudaMalloc(&a.x, h.size() * 4)); + CK(cudaMemcpy(a.x, h.data(), h.size() * 4, cudaMemcpyHostToDevice)); + CK(cudaMalloc(&a.q_ref, native_q8_1_bytes(n_in, 8))); + CK(cudaMalloc(&a.q, native_q8_1_bytes(n_in, 8))); + CK(cudaMalloc(&a.il, native_q8_1_il_bytes(n_in, 8))); + CK(cudaDeviceSynchronize()); + return a; +} + +template +std::vector fetch(const void* d, size_t n) { + std::vector h(n); + CK(cudaMemcpy(h.data(), d, n * sizeof(T), cudaMemcpyDeviceToHost)); + return h; +} + +int check_shape(const Fmt& f, const Act& a, int n_out) { + const int n_in = a.n_in; + void* w = dev_weights(f, n_in, n_out); + float *y0, *y1; + CK(cudaMalloc(&y0, (size_t) 8 * n_out * 4)); + CK(cudaMalloc(&y1, (size_t) 8 * n_out * 4)); + int bad = 0; + for (int nc = 2; nc <= 8; ++nc) { + native_quantize_q8_1(a.x, a.q_ref, n_in, nc, g_s); + native_quantize_q8_1_il(a.x, a.q, a.il, n_in, nc, g_s); + CK(cudaMemsetAsync(y0, 0xff, (size_t) 8 * n_out * 4, g_s)); + CK(cudaMemsetAsync(y1, 0xee, (size_t) 8 * n_out * 4, g_s)); + native_mmvq(f.type, w, a.q_ref, y0, n_in, n_out, nc, g_s); + native_mmvq_il(f.type, w, a.q, a.il, y1, n_in, n_out, nc, g_s); + CK(cudaStreamSynchronize(g_s)); + const size_t qb = native_q8_1_bytes(n_in, nc); + if (fetch(a.q_ref, qb) != fetch(a.q, qb)) { + ++bad; + std::printf(" %s %d->%d, %d columns: the q8_1 blocks differ\n", f.name, n_in, n_out, nc); + } + const auto r0 = fetch(y0, (size_t) nc * n_out), r1 = fetch(y1, (size_t) nc * n_out); + size_t diff = 0; + for (size_t i = 0; i < r0.size(); ++i) diff += r0[i] != r1[i]; + if (diff) { + ++bad; + std::printf(" %s %d->%d, %d columns: %zu of %zu values differ\n", f.name, n_in, n_out, nc, diff, r0.size()); + } + } + cudaFree(w); + cudaFree(y0); + cudaFree(y1); + return bad; +} + +double time_calls(const std::function& fn, int reps = 48) { + cudaGraph_t g; + CK(cudaStreamBeginCapture(g_s, cudaStreamCaptureModeThreadLocal)); + for (int r = 0; r < reps; ++r) fn(r); + CK(cudaStreamEndCapture(g_s, &g)); + cudaGraphExec_t ex; + CK(cudaGraphInstantiate(&ex, g, 0)); + cudaGraphDestroy(g); + const auto t0 = std::chrono::steady_clock::now(); + while (std::chrono::duration(std::chrono::steady_clock::now() - t0).count() < 0.3) { // clocks up + CK(cudaGraphLaunch(ex, g_s)); + CK(cudaStreamSynchronize(g_s)); + } + cudaEvent_t a, b; + CK(cudaEventCreate(&a)); + CK(cudaEventCreate(&b)); + const int iters = 20; + CK(cudaEventRecord(a, g_s)); + for (int i = 0; i < iters; ++i) CK(cudaGraphLaunch(ex, g_s)); + CK(cudaEventRecord(b, g_s)); + CK(cudaEventSynchronize(b)); + float ms = 0; + CK(cudaEventElapsedTime(&ms, a, b)); + cudaGraphExecDestroy(ex); + cudaEventDestroy(a); + cudaEventDestroy(b); + return ms * 1000.0 / iters / reps; +} + +void bench_shape(const Fmt& f, const Act& a, int n_out) { + const int n_in = a.n_in; + const double mb = (double) native_mmvq_weight_bytes(f.type, n_in, n_out) / 1e6; + const int copies = std::max(4, (int) (24.0 / mb) + 1); + std::vector w((size_t) copies); + for (auto& p : w) p = dev_weights(f, n_in, n_out); + float* y; + CK(cudaMalloc(&y, (size_t) 8 * n_out * 4)); + std::printf("%-6s %5d -> %5d (%5.2f MB), us (GB/s):", f.name, n_in, n_out, mb); + for (int nc : {2, 3, 4, 8}) { + native_quantize_q8_1(a.x, a.q_ref, n_in, nc, g_s); + native_quantize_q8_1_il(a.x, a.q, a.il, n_in, nc, g_s); + const double t0 = time_calls([&](int r) { native_mmvq(f.type, w[(size_t) (r % copies)], a.q_ref, y, n_in, n_out, nc, g_s); }); + const double t1 = time_calls([&](int r) { + native_mmvq_il(f.type, w[(size_t) (r % copies)], a.q, a.il, y, n_in, n_out, nc, g_s); + }); + std::printf(" %d: %5.1f -> %5.1f (%3.0f)", nc, t0, t1, mb * 1e3 / t1); + } + std::printf("\n"); + for (auto p : w) cudaFree(p); + cudaFree(y); +} + +} // namespace + +int main(int argc, char** argv) { + const bool bench = argc > 1 && std::strcmp(argv[1], "--bench") == 0; + CK(cudaStreamCreateWithFlags(&g_s, cudaStreamNonBlocking)); + Act a2560 = make_act(2560), a6144 = make_act(6144), a640 = make_act(640); + if (bench) { + for (const Fmt& f : kFmts) { + bench_shape(f, a2560, 10240); + bench_shape(f, a2560, 6144); + bench_shape(f, a6144, 2560); + } + return 0; + } + int bad = 0, cases = 0; + for (const Fmt& f : kFmts) { + const std::pair shapes[] = {{&a2560, 10240}, {&a2560, 6144}, {&a6144, 2560}, {&a2560, 12288}, + {&a2560, 2050}, {&a2560, 4097}, {&a2560, 8192}, {&a2560, 512}}; + for (const auto& sh : shapes) { + bad += check_shape(f, *sh.first, sh.second); + ++cases; + } + if (f.block_elems == 32) { // Q8_0 also at the shared expert's down shape + bad += check_shape(f, a640, 2560); + ++cases; + } + } + std::printf("mmvq_il_parity: %d shape/format cases x 2..8 columns: %s\n", cases, bad ? "FAILED" : "bitwise equal"); + return bad ? 1 : 0; +} From 803e165c69bd16c4d59791d0607c1e6b887c4c45 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 23:51:36 +0200 Subject: [PATCH 46/92] Hyper-connection read: the up projection's per-token sums in one xor tree The up kernel took a warp_sum per (row, token); now a warp takes 2-4 rows at a time and sums all their tokens' dots in one tree that keeps each sum in the lanes that need it (xor_scatter: the butterfly's own additions, so each sum is bitwise warp_sum's), each (row, token) epilogue on the lane holding its sum with its inputs fetched while the dots run. The up kernel at 4 tokens: 18.4 -> 13.4 us; the whole read 44.1 -> 40.2 us (verify_parity: bitwise the single-token read). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- src/kernels/cuda/fused_gr.cu | 89 ++++++++++++++++++-------------- src/kernels/cuda/xor_scatter.cuh | 57 ++++++++++++++++++++ 2 files changed, 106 insertions(+), 40 deletions(-) create mode 100644 src/kernels/cuda/xor_scatter.cuh diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu index 62cb50d370b..9732aacc596 100644 --- a/src/kernels/cuda/fused_gr.cu +++ b/src/kernels/cuda/fused_gr.cu @@ -1,6 +1,7 @@ // src/kernels/cuda/fused_gr.cu - see include/strata/kernels/fused_gr.hpp. #include "strata/kernels/fused_gr.hpp" #include "strata/kernels/bf16_bits.hpp" +#include "xor_scatter.cuh" #include @@ -274,68 +275,72 @@ __global__ void __launch_bounds__(THREADS) gr_down_multi_kernel(GrMulti m) { constexpr int UPM_COLS = 16; // columns per block (x 4 streams = 64 rows, 8 per warp) constexpr int UPM_BLOCKS = N / UPM_COLS; // 160 -constexpr int UPM_RPW = 2; // rows a warp has in flight at once +constexpr int UPM_ROWS = HC * UPM_COLS / WARPS; // rows a warp -// `gr_up_kernel` for T tokens: each row of w_up read once, two rows per warp in flight; the lo vectors staged split -// into chunk halves; the T dots of a row reduced by xor so every lane holds every sum, and lane k runs token k's -// epilogue - the T epilogues in parallel instead of one after another. +// `gr_up_kernel` for TT tokens: each row of w_up read once, RW rows a warp at a time; the TT x RW dots of those rows +// summed by one xor tree (xor_scatter, each sum bitwise warp_sum's) and every (row, token) epilogue on the lane holding +// its sum, its inputs fetched while the dots run. The lo vectors are staged split into chunk halves. +template __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { - __shared__ __align__(16) float4 lo4[kFusedGrMaxT][2][LR / 8]; - __shared__ float g[kFusedGrMaxT][HC][UPM_COLS]; + constexpr int RW = TT <= 4 ? 4 : 2; + constexpr int NV = pow2_at_least(TT * RW); // sums a lane reduces, zero-padded + constexpr int LPV = 32 / NV; // lanes that end up holding each sum + __shared__ __align__(16) float4 lo4[TT][2][LR / 8]; + __shared__ float g[TT][HC][UPM_COLS]; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - const int T = m.T; const int d0 = blockIdx.x * UPM_COLS; - for (int i = t; i < T * LR / 4; i += THREADS) { + for (int i = t; i < TT * LR / 4; i += THREADS) { const int k = i / (LR / 4), f = i - k * (LR / 4); lo4[k][f & 1][f >> 1] = reinterpret_cast(m.a[k].lo)[f]; } __syncthreads(); - for (int r0 = warp * UPM_RPW; r0 < HC * UPM_COLS; r0 += WARPS * UPM_RPW) { - uint4 wa[UPM_RPW], wb[UPM_RPW]; - // the epilogue inputs of this lane's token, fetched while the dots run - float rv[UPM_RPW], wn[UPM_RPW], rsc[UPM_RPW], bo[UPM_RPW], ip[UPM_RPW]; - bool apply = false; + const int vi = xor_scatter_index(lane); // the sum this lane ends with: row vi / TT, token vi % TT + const int vq = vi / TT, vk = vi - vq * TT; + const bool epi = (lane & (LPV - 1)) == 0 && vi < TT * RW; + for (int r0 = warp * UPM_ROWS; r0 < (warp + 1) * UPM_ROWS; r0 += RW) { + uint4 wa[RW], wb[RW]; #pragma unroll - for (int q = 0; q < UPM_RPW; ++q) { + for (int q = 0; q < RW; ++q) { const int r = r0 + q, c = r / UPM_COLS, dd = r - c * UPM_COLS, i = c * N + d0 + dd; const uint4* w4 = reinterpret_cast(m.a[0].w_up + (size_t) i * LR); wa[q] = __ldg(w4 + lane); wb[q] = lane < LR / 8 - 32 ? __ldg(w4 + 32 + lane) : make_uint4(0, 0, 0, 0); - rv[q] = wn[q] = rsc[q] = bo[q] = ip[q] = 0.0f; - if (lane < T) { - const FusedGrArgs& a = m.a[lane]; - rv[q] = a.R[i]; - wn[q] = a.w_norm[i]; - rsc[q] = a.rs[c]; - apply = a.apply; - if (apply) { bo[q] = a.bo_prev[d0 + dd]; ip[q] = a.inj_prev[c]; } - } } + const int er = r0 + vq, ec = er / UPM_COLS, edd = er - ec * UPM_COLS, ei = ec * N + d0 + edd; + float rv = 0.0f, wn = 0.0f, rsc = 0.0f, bo = 0.0f, ip = 0.0f; + bool apply = false; + if (epi) { + const FusedGrArgs& a = m.a[vk]; + rv = a.R[ei]; + wn = a.w_norm[ei]; + rsc = a.rs[ec]; + apply = a.apply; + if (apply) { bo = a.bo_prev[d0 + edd]; ip = a.inj_prev[ec]; } + } + float v[NV]; #pragma unroll - for (int q = 0; q < UPM_RPW; ++q) { - const int r = r0 + q, c = r / UPM_COLS, dd = r - c * UPM_COLS, i = c * N + d0 + dd; - float mine = 0.0f; + for (int q = 0; q < RW; ++q) #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) { - if (k >= T) break; + for (int k = 0; k < TT; ++k) { float acc = dot8h(wa[q], lo4[k][0][lane], lo4[k][1][lane]); if (lane < LR / 8 - 32) acc += dot8h(wb[q], lo4[k][0][32 + lane], lo4[k][1][32 + lane]); - acc = warp_sum(acc); - if (lane == k) mine = acc; + v[q * TT + k] = acc; } - if (lane < T) { - float x = rv[q]; - if (apply) { - x = fmaf(bo[q], 2.0f * sigmoidf_(ip[q] / (float) HC), x); - m.a[lane].R_out[i] = x; - } - const float y = x * wn[q] * rsc[q]; - g[lane][c][dd] = y * sigmoidf_(mine); +#pragma unroll + for (int e = TT * RW; e < NV; ++e) v[e] = 0.0f; + xor_scatter(v, lane); + if (epi) { + float x = rv; + if (apply) { + x = fmaf(bo, 2.0f * sigmoidf_(ip / (float) HC), x); + m.a[vk].R_out[ei] = x; } + const float y = x * wn * rsc; + g[vk][ec][edd] = y * sigmoidf_(v[0]); } } __syncthreads(); - for (int i = t; i < T * UPM_COLS; i += THREADS) { + for (int i = t; i < TT * UPM_COLS; i += THREADS) { const int k = i / UPM_COLS, col = i - k * UPM_COLS; float s = 0.0f; #pragma unroll @@ -377,7 +382,11 @@ void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* xn_scratch, voi gr_down_multi_kernel<320><<>>(m); else gr_down_multi_kernel<160><<>>(m); - gr_up_multi_kernel<<>>(m); + switch (n_tok) { +#define STRATA_UP(T) case T: gr_up_multi_kernel<<>>(m); break; + STRATA_UP(1) STRATA_UP(2) STRATA_UP(3) STRATA_UP(4) STRATA_UP(5) STRATA_UP(6) STRATA_UP(7) STRATA_UP(8) +#undef STRATA_UP + } const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { std::fprintf(stderr, "fused_gr_read_multi: %s\n", cudaGetErrorString(e)); diff --git a/src/kernels/cuda/xor_scatter.cuh b/src/kernels/cuda/xor_scatter.cuh new file mode 100644 index 00000000000..d8fd1a70edb --- /dev/null +++ b/src/kernels/cuda/xor_scatter.cuh @@ -0,0 +1,57 @@ +// src/kernels/cuda/xor_scatter.cuh - several warp sums in one xor tree, each bitwise the butterfly's. +// +// `v += __shfl_xor_sync(v, o)` for o = 16, 8, 4, 2, 1 leaves every lane with the same sum. For NV sums a lane does not +// need all of them in every lane: at distance o, while it holds more than one, it keeps the half its bit o selects and +// adds its partner's copy of them - the butterfly's own additions in the same order - then the last steps as the +// butterfly. NV + NV/2 + ... shuffles instead of 5 NV. +#pragma once + +#include + +namespace strata::kernels { +namespace { + +// NV (a power of two, at most 32) values a lane; afterwards v[0] is the sum of value xor_scatter_index(lane) +template +__device__ __forceinline__ void xor_scatter(float (&v)[NV], int lane) { + static_assert(NV >= 1 && NV <= 32 && (NV & (NV - 1)) == 0, "NV must be a power of two <= 32"); +#pragma unroll + for (int s = 0; s < 5; ++s) { + const int o = 16 >> s, n = NV >> s; // n: the values left before this step + if (n > 1) { + const bool hi = (lane & o) != 0; +#pragma unroll + for (int e = 0; e < n / 2; ++e) { + const float mine = hi ? v[n / 2 + e] : v[e]; + const float other = hi ? v[e] : v[n / 2 + e]; + v[e] = mine + __shfl_xor_sync(0xffffffffu, other, o); + } + } else { + v[0] += __shfl_xor_sync(0xffffffffu, v[0], o); + } + } +} + +// the value a lane's v[0] sums after xor_scatter: the halves its bits 16, 8, ... selected +template +__device__ __forceinline__ int xor_scatter_index(int lane) { + int vi = 0; +#pragma unroll + for (int s = 0; (NV >> s) > 1; ++s) vi += (lane & (16 >> s)) ? NV >> (s + 1) : 0; + return vi; +} + +// the lowest lane holding value vi +template +__device__ __forceinline__ int xor_scatter_lane(int vi) { + int lane = 0; +#pragma unroll + for (int s = 0; (NV >> s) > 1; ++s) lane += (vi & (NV >> (s + 1))) ? 16 >> s : 0; + return lane; +} + +// the smallest power of two >= n +__host__ __device__ constexpr int pow2_at_least(int n) { return n <= 1 ? 1 : 2 * pow2_at_least((n + 1) / 2); } + +} // namespace +} // namespace strata::kernels From 81e276d211104a982de1ffaec1da43c9e86e5681 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sat, 26 Sep 2026 23:51:36 +0200 Subject: [PATCH 47/92] Verify window: the router, the doorbell and the hit plan in one kernel verify_router takes a layer's router logits (bf16_gemv_fp32_mmvf_multi's block, its arithmetic now shared through mmvf_multi.cuh), the first blocks also copying the input rows to mapped memory in 16-byte stores; the last block to finish routes the tokens (native_router_top10's code, route_token), publishes the ids and weights, writes the ring's number (known at capture: no read of the mapped counter) and plans the main GPU's hits (hit_plan_block, shared with verify_hit_plan). It replaces four kernels on the main stream and three on the prediction branch. 21.0 -> 16.3 us at 1 token, 31.5 -> 21.7 at 4; verify_parity checks it bitwise against the four. --window-profile: the router, ring and hit plan are one stage ("router, ring, hits"), followed by the experts' activations ("expert inputs"). With the up-projection change, fixed text against the previous commit, 2 alternating rounds: IQ3_XXS code +1.6%, thinking +2.1%; UD-Q4_K_XL thinking +1.6%, code unchanged. Window hashes identical to the build before the interleaved projections (both models, thinking and code; --spec-split, --pcie-frac 0.3 kernel and direct, --second-gpu -1). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 3 +- include/strata/kernels/native_router.hpp | 24 ++++++ src/core/verify.cpp | 75 ++++++++++------- src/kernels/cuda/hit_plan.cuh | 71 ++++++++++++++++ src/kernels/cuda/mmvf_multi.cuh | 74 ++++++++++++++++ src/kernels/cuda/native_bf16.cu | 55 ++---------- src/kernels/cuda/native_router.cu | 94 ++++++++++++++++++--- src/kernels/cuda/verify_kernels.cu | 52 +----------- src/kernels/verify_parity.cpp | 103 +++++++++++++++++++++++ 9 files changed, 406 insertions(+), 145 deletions(-) create mode 100644 src/kernels/cuda/hit_plan.cuh create mode 100644 src/kernels/cuda/mmvf_multi.cuh diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 6c50afba3db..b790d29ebe2 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -203,6 +203,7 @@ class Verifier { int32_t* sel_ = nullptr; float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *hit_out_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr; + unsigned* ring_count_ = nullptr; // verify_router's block counters: the layer's, the prediction's float *plogits_ = nullptr, *pw_ = nullptr; // the next layer's router (prediction) int32_t* pids_ = nullptr; int32_t* plan_ = nullptr; // device copies of the PCIe share's plan blocks @@ -223,7 +224,7 @@ class Verifier { // --window-profile: stamps [window size][layer, then one row for the window][stage]: a layer's main-stream stages, // then the end of its shared expert's branch - static constexpr int kLayerStamps = 13, kStamps = 14; + static constexpr int kLayerStamps = 12, kStamps = 13; bool profile_ = false; unsigned long long* stamps_ = nullptr; std::vector h_stamps_; diff --git a/include/strata/kernels/native_router.hpp b/include/strata/kernels/native_router.hpp index 0ed95bac79c..4bb9d0e366c 100644 --- a/include/strata/kernels/native_router.hpp +++ b/include/strata/kernels/native_router.hpp @@ -16,4 +16,28 @@ bool native_router_enabled(); void native_router_top10(const float* logits, int32_t* ids, float* weights, void* stream); // `native_router_top10` for n_tok tokens in one launch: logits (n_tok, 512) -> ids, weights (n_tok, 10). void native_router_top10_multi(const float* logits, int32_t* ids, float* weights, int n_tok, void* stream); + +// Plan v0.3 P6: a verify window layer's router in one kernel: bf16_gemv_fp32_mmvf_multi's logits, +// native_router_top10_multi's ids and weights and doorbell_publish's ring, bitwise, and verify_hit_plan's plan. The +// mapped copy of x is written while the logits run; the ids and weights (and their mapped copies) once every token's +// logits are in; then the sequence number; then the plan. +struct VerifyRouterArgs { + const float* x = nullptr; // (n_tok, n_embd) + const uint16_t* w = nullptr; // the router: bf16 (n_expert, n_embd) + float* logits = nullptr; // (n_tok, n_expert) scratch + int32_t* ids = nullptr; // (n_tok, 10) + float* weights = nullptr; // (n_tok, 10) + float* x_out = nullptr; // mapped copy of x, or null + int32_t* ids_out = nullptr; // mapped copies of ids and weights, or null + float* w_out = nullptr; + uint32_t* seq = nullptr; // mapped: set to `ring` once those copies are visible + uint32_t ring = 0; // (doorbell_publish's increment: the rings so far, fixed at capture) + const int32_t* res = nullptr; // verify_hit_plan's arguments; plan null: no plan + const unsigned long long* slot_ptr = nullptr; + int32_t* plan = nullptr; + int cap = 0, ptr_off = 0; + unsigned* counter = nullptr; // device, zero before the first launch (each launch leaves it so) + int n_tok = 0, n_embd = 0, n_expert = 0; // 1..8 tokens, 512 experts, n_embd a multiple of 512 +}; +void verify_router(const VerifyRouterArgs& a, void* stream); } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index a232a052ab0..78eac1eb60f 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -231,6 +231,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& plogits_ = b.take(T * (uint64_t) g.n_expert); pw_ = b.take(T * K); pids_ = b.take(T * K); shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); + ring_count_ = b.take(2); plan_ = b.take(2 * (uint64_t) plan_stride_); dplan_ = b.take(2 * (uint64_t) plan_stride_); res_ = b.take((uint64_t) res_words_); slot_ptr_ = b.take((uint64_t) hits.slots); staging_ = b.take((uint64_t) kStagingBlobs * strata::kernels::cpu::expert_layout().max_blob); @@ -599,12 +600,26 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 2, cs); gr_read_group(1, true, inj_, inj2_); stamp(l, 3, cs); - if (native_router_enabled()) { // one weight read and one launch each for the window's tokens + // the router, the doorbell and the main GPU's routed experts - the ones its VRAM tier holds, decided here from + // the residency snapshot the pool reads too (it leaves their rows to the GPU) + if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { + err = "verify: the residency copy could not join"; + return false; + } + int32_t* hit_plan = dplan_ + (size_t) grp * (size_t) plan_stride_; + if (native_router_enabled()) { // one kernel: logits, top 10, ring, hit plan const WeightRef* wr = need(v, "ffn_gate_inp.weight", err); if (!wr) return false; + VerifyRouterArgs ra; + ra.x = xm; ra.w = (const uint16_t*) wr->data; ra.logits = logits_ + tb * NE; + ra.ids = ids_ + tb * K; ra.weights = w_ + tb * K; + ra.x_out = m_x_ + tb * N; ra.ids_out = m_ids_ + tb * K; ra.w_out = m_w_ + tb * K; ra.seq = m_seq_; + ra.ring = (uint32_t) (l * G + grp + 1); // the rings so far (the host waits for this one) + ra.res = res_ + (size_t) l * NE; ra.slot_ptr = slot_ptr_; ra.plan = hit_plan; + ra.cap = (int) (MT * K); ra.ptr_off = (int) plan_ptr_off_; + ra.counter = ring_count_; ra.n_tok = n; ra.n_embd = (int) N; ra.n_expert = (int) NE; try { - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wr->data, logits_ + tb * NE, N, NE, n, cs); - native_router_top10_multi(logits_ + tb * NE, ids_ + tb * K, w_ + tb * K, n, cs); + verify_router(ra, cs); } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + " router: " + e.what(); return false; @@ -615,26 +630,20 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { mb.logits = logits_ + t * NE; mb.ids = ids_ + t * K; mb.weights = w_ + t * K; if (!moe_route(wt, g, l, K, mb, mixed_ + t * N, cs, err, nullptr)) return false; } + doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, + m_ids_ + tb * K, m_w_ + tb * K, m_seq_, cs); + verify_hit_plan(ids_ + tb * K, (int) (n * K), (int) K, res_ + (size_t) l * NE, (int) NE, slot_ptr_, hit_plan, + MT * K, plan_ptr_off_, cs); } stamp(l, 4, cs); - doorbell_publish(xm, ids_ + tb * K, w_ + tb * K, (int64_t) n * N, (int64_t) n * K, m_x_ + tb * N, - m_ids_ + tb * K, m_w_ + tb * K, m_seq_, cs); - stamp(l, 5, cs); - // after the doorbell: the shared expert's branch first (the longest), the main GPU's routed experts - the ones - // its VRAM tier holds, decided here from the residency snapshot the pool reads too (it leaves their rows to - // the GPU) - and once they are planned, the next layer's prediction + // after the doorbell: the shared expert's branch first (the longest), the routed experts' activations, and the + // next layer's prediction if (beside && !shared(l, grp)) return false; - if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { - err = "verify: the residency copy could not join"; - return false; - } - verify_hit_plan(ids_ + tb * K, (int) (n * K), (int) K, res_ + (size_t) l * NE, (int) NE, slot_ptr_, - dplan_ + (size_t) grp * (size_t) plan_stride_, MT * K, plan_ptr_off_, cs); if (strata::kernels::cpu::expert_layout().native) quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, cs); else quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, cs); - stamp(l, 6, cs); + stamp(l, 5, cs); if (predicts(l)) { const WeightRef* wr1 = need(LayerView(wt, l + 1), "ffn_gate_inp.weight", err); if (!wr1) return false; @@ -643,11 +652,13 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } const size_t half = (size_t) ((l + 1) & 1) * (size_t) (MT * K); + VerifyRouterArgs ra; + ra.x = xm; ra.w = (const uint16_t*) wr1->data; ra.logits = plogits_; ra.ids = pids_; ra.weights = pw_; + ra.ids_out = m_pids_ + half; ra.w_out = m_pw_ + half; ra.seq = m_pseq_; + ra.ring = (uint32_t) (l + 1); // the predictions published so far + ra.counter = ring_count_ + 1; ra.n_tok = n; ra.n_embd = (int) N; ra.n_expert = (int) NE; try { - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wr1->data, plogits_, N, NE, n, side_); - native_router_top10_multi(plogits_, pids_, pw_, n, side_); - doorbell_publish(nullptr, pids_, pw_, 0, (int64_t) n * K, nullptr, m_pids_ + half, m_pw_ + half, m_pseq_, - side_); + verify_router(ra, side_); } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + " prediction: " + e.what(); return false; @@ -698,11 +709,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { // has landed them in staging const Plan vram = plan_at(dplan_ + (size_t) grp * (size_t) plan_stride_); grouped(vram, vram.ptr, vram.start, vram.counts); - stamp(l, 7, cs); + stamp(l, 6, cs); Plan pcie; if (pcie_share_) { wait_flag_ge(m_flagA_, ring, cs); // the pool published this group's PCIe share - stamp(l, 8, cs); + stamp(l, 7, cs); int32_t* pl = plan_ + (size_t) grp * (size_t) plan_stride_; copy_i32_from_mapped(pl, m_plan_ + (size_t) grp * (size_t) plan_i32_, plan_i32_, cs); pcie = plan_at(pl); @@ -715,11 +726,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } grouped(pcie, pcie.ptr2, pcie.start2, pcie.counts + 2); } else { - stamp(l, 8, cs); + stamp(l, 7, cs); } - stamp(l, 9, cs); + stamp(l, 8, cs); wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows - stamp(l, 10, cs); + stamp(l, 9, cs); if (beside && cudaStreamWaitEvent(cs, shjoin_, 0) != cudaSuccess) { err = "verify: the shared expert's branch could not join"; return false; @@ -744,14 +755,14 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; } } - stamp(l, 11, cs); + stamp(l, 10, cs); if (predicts(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { err = "verify: the prediction branch could not join"; return false; } if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); - stamp(l, 12, cs); + stamp(l, 11, cs); return true; }; @@ -1046,9 +1057,9 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, prof_ns_[(size_t) kind * kStamps] += (double) (st[0] - before); for (int i = 1; i < kLayerStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); // the shared expert's branch, from the doorbell (it forks there) to its end - prof_ns_[(size_t) kind * kStamps + kLayerStamps] += (double) (st[kLayerStamps] - st[5]); + prof_ns_[(size_t) kind * kStamps + kLayerStamps] += (double) (st[kLayerStamps] - st[4]); { // the wait for the CPU's rows, by bucket - const double us = (double) (st[10] - st[9]) / 1e3; + const double us = (double) (st[9] - st[8]) / 1e3; int b = 0; while (b < kWaitBuckets - 1 && us >= kWaitEdges[b]) ++b; ++prof_wait_hist_[kind][b]; @@ -1069,13 +1080,13 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, void Verifier::print_profile() const { if (prof_windows_ == 0) return; static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", - "router", "doorbell", "hit plan", "VRAM experts", + "router, ring, hits", "expert inputs", "VRAM experts", "wait for PCIe plan", "PCIe experts", "wait for CPU", "rows + combine", "last HC write", "shared expert, beside"}; const double nw = (double) prof_windows_; std::printf("%-24s GPU time over %lld windows, ms per window: inputs %.3f, head %.3f, whole window %.3f; " - "one stamp %.2f us (each stage below includes one; the last runs from the doorbell beside the " - "four before \"rows + combine\")\n", "window profile", (long long) prof_windows_, + "one stamp %.2f us (each stage below includes one; the last runs from the ring beside the " + "five before \"rows + combine\")\n", "window profile", (long long) prof_windows_, prof_window_ns_[0] / nw / 1e6, prof_window_ns_[1] / nw / 1e6, prof_window_ns_[2] / nw / 1e6, prof_window_ns_[3] / nw / 1e3); std::printf(" %-22s %12s %12s %14s\n", "stage", "GDN us/layer", "QSA us/layer", "ms per window"); diff --git a/src/kernels/cuda/hit_plan.cuh b/src/kernels/cuda/hit_plan.cuh new file mode 100644 index 00000000000..56ef546e315 --- /dev/null +++ b/src/kernels/cuda/hit_plan.cuh @@ -0,0 +1,71 @@ +// src/kernels/cuda/hit_plan.cuh - the verify window's main-GPU hit plan (verify_kernels.hpp, verify_hit_plan) as the +// work of one block, for verify_kernels.cu's kernel and the router kernel that rings the doorbell (native_router.cu). +#pragma once + +#include + +#include + +namespace strata::kernels { +namespace { + +constexpr int kHitPlanMax = 128; + +// A block of at least kHitPlanMax threads, thread i = routed entry i (n <= kHitPlanMax). A group per distinct resident +// expert, in the order of their first entries; a group's entries in routing order. `ids` must be visible to the whole +// block. +__device__ __forceinline__ void hit_plan_block(const int32_t* __restrict__ ids, int n, int k, + const int32_t* __restrict__ res, int n_expert, + const unsigned long long* __restrict__ slot_ptr, + int32_t* __restrict__ plan, int cap, int ptr_off) { + __shared__ int32_t s_id[kHitPlanMax], s_size[kHitPlanMax], s_start[kHitPlanMax]; + const int i = threadIdx.x; + int32_t e = -1, slot = -1; + if (i < n) { + e = ids[i]; + if (e >= 0 && e < n_expert) slot = res[e]; + } + if (i < kHitPlanMax) s_id[i] = e; + __syncthreads(); + const bool hit = slot >= 0; + int first = i; + if (hit) + for (int j = 0; j < i; ++j) + if (s_id[j] == e) { first = j; break; } + const bool lead = hit && first == i; + int size = 0; + if (lead) + for (int j = i; j < n; ++j) size += s_id[j] == e; + if (i < kHitPlanMax) s_size[i] = size; // > 0 at a group's first entry only + const int groups = __syncthreads_count(lead); + const int entries = __syncthreads_count(hit); + int32_t* start = plan + 4; + int32_t* dst = start + cap + 1; + int32_t* tok = dst + cap; + auto* ptr = (unsigned long long*) (plan + ptr_off); + if (lead) { + int grp = 0, at = 0; + for (int j = 0; j < i; ++j) + if (s_size[j] > 0) { ++grp; at += s_size[j]; } + s_start[i] = at; + start[grp] = at; + ptr[grp] = slot_ptr[slot]; + } + __syncthreads(); + if (hit) { + int rank = 0; + for (int j = first; j < i; ++j) rank += s_id[j] == e; + dst[s_start[first] + rank] = i; + tok[s_start[first] + rank] = i / k; + } + if (i == 0) { + plan[0] = groups; + plan[1] = entries; + plan[2] = 0; + plan[3] = 0; + start[groups] = entries; + } +} + +} // namespace +} // namespace strata::kernels diff --git a/src/kernels/cuda/mmvf_multi.cuh b/src/kernels/cuda/mmvf_multi.cuh new file mode 100644 index 00000000000..11ee5981147 --- /dev/null +++ b/src/kernels/cuda/mmvf_multi.cuh @@ -0,0 +1,74 @@ +// src/kernels/cuda/mmvf_multi.cuh - one block's row of the FP32-activation BF16 MMVF for up to 8 activation rows, for +// bf16_gemv_fp32_mmvf_multi (native_bf16.cu) and the verify window's router (native_router.cu); both translation +// units compile with --use_fast_math, the pinned contract's arithmetic. +// +// Adapted from llama.cpp 3cf03257f219afbe7334045ff7c6a06ac68c627d, ggml/src/ggml-cuda/{mmvf.cu,common.cuh}. +// MIT License, Copyright (c) 2023-2026 The ggml authors (see native_bf16.cu for the full notice). +#pragma once + +#include "strata/kernels/bf16_bits.hpp" + +#include + +#include + +namespace strata::kernels { +namespace { + +constexpr int kMmvfMaxRows = 8; + +__device__ __forceinline__ float mmvf_warp_sum(float value) { +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + value += __shfl_xor_sync(0xffffffffu, value, offset, 32); + return value; +} + +// `bf16_f32_mmvf_kernel` for up to kMmvfMaxRows activation rows (x: n_tok rows of n_in): each thread walks the same +// pairs in the same order for every row, and each row is reduced exactly as the single-row kernel reduces it; thread 0 +// ends with the sums in acc[0 .. n_tok). +template +__device__ __forceinline__ void mmvf_multi_row(const float* __restrict__ x, const uint16_t* __restrict__ row, int n_in, + int n_tok, float (&acc)[kMmvfMaxRows]) { + const int t = threadIdx.x; + const uint32_t* weights2 = reinterpret_cast(row); + __shared__ float partials[kMmvfMaxRows][32]; + if constexpr (BLOCK_SIZE > 32) { + if (t < 32) + for (int k = 0; k < kMmvfMaxRows; ++k) partials[k][t] = 0.0f; + __syncthreads(); + } +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) acc[k] = 0.0f; + for (int pair = t; pair < n_in / 2; pair += BLOCK_SIZE) { + const uint32_t weight = weights2[pair]; + const float w0 = f32_from_bf16((uint16_t) weight), w1 = f32_from_bf16((uint16_t) (weight >> 16)); +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + const float2 input = reinterpret_cast(x + (size_t) k * n_in)[pair]; + acc[k] = __fmaf_rn(w0, input.x, acc[k]); + acc[k] = __fmaf_rn(w1, input.y, acc[k]); + } + } +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + acc[k] = mmvf_warp_sum(acc[k]); + } + if constexpr (BLOCK_SIZE > 32) { + if ((t & 31) == 0) + for (int k = 0; k < n_tok; ++k) partials[k][t / 32] = acc[k]; + __syncthreads(); + if (t < 32) { +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + acc[k] = mmvf_warp_sum(partials[k][t]); + } + } + } +} + +} // namespace +} // namespace strata::kernels diff --git a/src/kernels/cuda/native_bf16.cu b/src/kernels/cuda/native_bf16.cu index 7471fedb83d..f9d6b576072 100644 --- a/src/kernels/cuda/native_bf16.cu +++ b/src/kernels/cuda/native_bf16.cu @@ -1,5 +1,6 @@ #include "strata/kernels/bf16_gemv.hpp" #include "strata/kernels/bf16_bits.hpp" +#include "mmvf_multi.cuh" #include #include @@ -33,13 +34,6 @@ namespace { // LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. -__device__ __forceinline__ float mmvf_warp_sum(float value) { -#pragma unroll - for (int offset = 16; offset > 0; offset >>= 1) - value += __shfl_xor_sync(0xffffffffu, value, offset, 32); - return value; -} - template __global__ void bf16_f32_mmvf_kernel(const float* __restrict__ x, const uint16_t* __restrict__ w, float* __restrict__ y, int n_in) { @@ -70,53 +64,14 @@ __global__ void bf16_f32_mmvf_kernel(const float* __restrict__ x, const uint16_t if (t == 0) y[blockIdx.x] = acc; } -// `bf16_f32_mmvf_kernel` for up to kMultiT activation rows: each thread walks the same pairs in the same order for -// every row, and each row is reduced exactly as the single-row kernel reduces it. -constexpr int kMultiT = 8; +// `bf16_f32_mmvf_kernel` for up to kMultiT activation rows (mmvf_multi_row) +constexpr int kMultiT = kMmvfMaxRows; template __global__ void bf16_f32_mmvf_multi_kernel(const float* __restrict__ x, const uint16_t* __restrict__ w, float* __restrict__ y, int n_in, int n_out, int n_tok) { - const int t = threadIdx.x; - const uint16_t* row = w + (size_t) blockIdx.x * n_in; - const uint32_t* weights2 = reinterpret_cast(row); - __shared__ float partials[kMultiT][32]; - if constexpr (BLOCK_SIZE > 32) { - if (t < 32) - for (int k = 0; k < kMultiT; ++k) partials[k][t] = 0.0f; - __syncthreads(); - } float acc[kMultiT]; -#pragma unroll - for (int k = 0; k < kMultiT; ++k) acc[k] = 0.0f; - for (int pair = t; pair < n_in / 2; pair += BLOCK_SIZE) { - const uint32_t weight = weights2[pair]; - const float w0 = f32_from_bf16((uint16_t) weight), w1 = f32_from_bf16((uint16_t) (weight >> 16)); -#pragma unroll - for (int k = 0; k < kMultiT; ++k) { - if (k >= n_tok) break; - const float2 input = reinterpret_cast(x + (size_t) k * n_in)[pair]; - acc[k] = __fmaf_rn(w0, input.x, acc[k]); - acc[k] = __fmaf_rn(w1, input.y, acc[k]); - } - } -#pragma unroll - for (int k = 0; k < kMultiT; ++k) { - if (k >= n_tok) break; - acc[k] = mmvf_warp_sum(acc[k]); - } - if constexpr (BLOCK_SIZE > 32) { - if ((t & 31) == 0) - for (int k = 0; k < n_tok; ++k) partials[k][t / 32] = acc[k]; - __syncthreads(); - if (t < 32) { -#pragma unroll - for (int k = 0; k < kMultiT; ++k) { - if (k >= n_tok) break; - acc[k] = mmvf_warp_sum(partials[k][t]); - } - } - } - if (t == 0) + mmvf_multi_row(x, w + (size_t) blockIdx.x * n_in, n_in, n_tok, acc); + if (threadIdx.x == 0) for (int k = 0; k < n_tok; ++k) y[(size_t) k * n_out + blockIdx.x] = acc[k]; } diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index 00e062996fd..c48928207e6 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -21,6 +21,9 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. #include "strata/kernels/native_router.hpp" +#include "strata/kernels/bf16_bits.hpp" +#include "hit_plan.cuh" +#include "mmvf_multi.cuh" #include #include #include @@ -41,19 +44,13 @@ __device__ __forceinline__ float warp_max(float value) { for (int mask = 16; mask; mask >>= 1) value = fmaxf(value, __shfl_xor_sync(0xffffffffu, value, mask, 32)); return value; } -__launch_bounds__(256, 1) -__global__ void route(const float* __restrict__ logits, int32_t* __restrict__ ids, - float* __restrict__ weights) { - // Preserve the pinned 32x8 block geometry; only row zero is active here. Block b routes token b. - if (threadIdx.y != 0) return; - logits += blockIdx.x * 512; - ids += blockIdx.x * 10; - weights += blockIdx.x * 10; - const int lane = threadIdx.x; +// One token, one warp: 512 logits -> the top 10 ids and weights; ids2 and weights2 take copies when given. The logits +// load through the L2 (another block may have written them). +__device__ __forceinline__ void route_token(const float* __restrict__ logits, int32_t* __restrict__ ids, + float* __restrict__ weights, int32_t* ids2, float* weights2, int lane) { float values[16]; #pragma unroll - for (int i = 0; i < 16; ++i) values[i] = logits[lane + i * 32]; - __syncthreads(); + for (int i = 0; i < 16; ++i) values[i] = __ldcg(logits + lane + i * 32); float maximum = -INFINITY; #pragma unroll for (int i = 0; i < 16; ++i) maximum = max(maximum, values[i]); @@ -87,6 +84,7 @@ __global__ void route(const float* __restrict__ logits, int32_t* __restrict__ id if ((expert & 31) == lane) { values[expert / 32] = -INFINITY; ids[rank] = expert; + if (ids2 != nullptr) ids2[rank] = expert; // Deliberately accumulate by WINNING EXPERT lane, not output rank. // Multiple selected experts in one lane add in selection order. selected_sum += best; @@ -95,7 +93,64 @@ __global__ void route(const float* __restrict__ logits, int32_t* __restrict__ id } selected_sum = max(warp_sum(selected_sum), 6.103515625e-5f); const float inverse_selected_sum = 1.0f / selected_sum; - if (lane < 10) weights[lane] = selected * inverse_selected_sum; + if (lane < 10) { + const float wv = selected * inverse_selected_sum; + weights[lane] = wv; + if (weights2 != nullptr) weights2[lane] = wv; + } +} +__launch_bounds__(256, 1) +__global__ void route(const float* __restrict__ logits, int32_t* __restrict__ ids, + float* __restrict__ weights) { + // Preserve the pinned 32x8 block geometry; only row zero is active here. Block b routes token b. + if (threadIdx.y != 0) return; + route_token(logits + blockIdx.x * 512, ids + blockIdx.x * 10, weights + blockIdx.x * 10, nullptr, nullptr, + threadIdx.x); +} + +// ================================ plan v0.3 P6: the verify window's router in one kernel ================================ +// +// For T tokens: a block a row of the logits (bf16_gemv_fp32_mmvf_multi's 256-thread block, mmvf_multi_row), the first +// blocks also copying the input rows to mapped memory, a 16-byte store (a full PCIe transaction) a thread; the last +// block to finish routes the tokens (`route`, a warp each), publishes their ids and weights, writes the ring's number +// and plans the main GPU's hits (hit_plan_block). In place of the gemv, the top 10, doorbell_publish and +// verify_hit_plan. +constexpr int VR_THREADS = 256; // mmvf_block_size of n_embd a multiple of 512 +template +__global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterArgs a) { + __shared__ bool s_last; + const int t = (int) threadIdx.x, lane = t & 31, warp = t >> 5; + bool mapped = false; + if (a.x_out != nullptr) { + const int i = (int) blockIdx.x * VR_THREADS + t; + if (i < TT * a.n_embd / 4) { + reinterpret_cast(a.x_out)[i] = reinterpret_cast(a.x)[i]; + mapped = true; + } + } + float acc[kMmvfMaxRows]; + mmvf_multi_row(a.x, a.w + (size_t) blockIdx.x * a.n_embd, a.n_embd, TT, acc); + if (t == 0) + for (int k = 0; k < TT; ++k) a.logits[(size_t) k * a.n_expert + blockIdx.x] = acc[k]; + if (mapped) __threadfence_system(); // this thread's writes before the block counts as done + else __threadfence(); + __syncthreads(); + if (t == 0) s_last = atomicAdd(a.counter, 1u) == gridDim.x - 1; + __syncthreads(); + if (!s_last) return; + if (warp < TT) + route_token(a.logits + (size_t) warp * a.n_expert, a.ids + warp * 10, a.weights + warp * 10, + a.ids_out != nullptr ? a.ids_out + warp * 10 : nullptr, + a.w_out != nullptr ? a.w_out + warp * 10 : nullptr, lane); + __threadfence_system(); + __syncthreads(); + if (t == 0) { + __threadfence_system(); + *(volatile uint32_t*) a.seq = a.ring; + *a.counter = 0u; // for the next launch + } + if (a.plan != nullptr) + hit_plan_block(a.ids, TT * 10, 10, a.res, a.n_expert, a.slot_ptr, a.plan, a.cap, a.ptr_off); } bool valid(const void* p, size_t bytes) { const auto address = reinterpret_cast(p); @@ -117,6 +172,21 @@ void native_router_top10(const float* logits, int32_t* ids, float* weights, void const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } +void verify_router(const VerifyRouterArgs& a, void* stream) { + if (!stream || a.n_tok < 1 || a.n_tok > 8 || a.n_expert != 512 || a.n_embd <= 0 || a.n_embd % 512 != 0 || + !a.x || !a.w || !a.logits || !a.ids || !a.weights || !a.seq || !a.counter || (a.plan && (!a.res || !a.slot_ptr))) + throw std::invalid_argument("verify_router: 1..8 tokens, 512 experts, n_embd a multiple of 512, and buffers"); + const auto st = static_cast(stream); + if (a.x_out != nullptr && a.n_tok * a.n_embd / 4 > a.n_expert * VR_THREADS) + throw std::invalid_argument("verify_router: the blocks copy the input rows a float4 a thread"); + switch (a.n_tok) { +#define STRATA_VR(T) case T: verify_router_kernel<<>>(a); break; + STRATA_VR(1) STRATA_VR(2) STRATA_VR(3) STRATA_VR(4) STRATA_VR(5) STRATA_VR(6) STRATA_VR(7) STRATA_VR(8) +#undef STRATA_VR + } + const auto error = cudaGetLastError(); + if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); +} void native_router_top10_multi(const float* logits, int32_t* ids, float* weights, int n_tok, void* stream) { const size_t nl = size_t(n_tok) * 512 * 4, no = size_t(n_tok) * 10 * 4; if (!stream || n_tok < 1 || !valid(logits, nl) || !valid(ids, no) || !valid(weights, no) diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index 2a303ff6e05..41d8a78dc2e 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -3,6 +3,7 @@ // The per-token arithmetic of every kernel here is transcribed from its single-token original (fused_gdn.cu, // elementwise.cu) with the same operation order, so a verify window reproduces plain decode bit for bit. #include "strata/kernels/verify_kernels.hpp" +#include "hit_plan.cuh" #include @@ -229,59 +230,10 @@ __global__ void rebase_ptrs_kernel(unsigned long long* ptr, const int32_t* n, un if (k < *n) ptr[k] = base + (unsigned long long) k * (unsigned long long) bytes; } -// Thread i = routed entry i. A group per distinct resident expert, in the order of their first entries; a group's -// entries in routing order. -constexpr int kHitPlanMax = 128; __global__ void hit_plan_kernel(const int32_t* __restrict__ ids, int n, int k, const int32_t* __restrict__ res, int n_expert, const unsigned long long* __restrict__ slot_ptr, int32_t* __restrict__ plan, int cap, int ptr_off) { - __shared__ int32_t s_id[kHitPlanMax], s_size[kHitPlanMax], s_start[kHitPlanMax]; - const int i = threadIdx.x; - int32_t e = -1, slot = -1; - if (i < n) { - e = ids[i]; - if (e >= 0 && e < n_expert) slot = res[e]; - } - s_id[i] = e; - __syncthreads(); - const bool hit = slot >= 0; - int first = i; - if (hit) - for (int j = 0; j < i; ++j) - if (s_id[j] == e) { first = j; break; } - const bool lead = hit && first == i; - int size = 0; - if (lead) - for (int j = i; j < n; ++j) size += s_id[j] == e; - s_size[i] = size; // > 0 at a group's first entry only - const int groups = __syncthreads_count(lead); - const int entries = __syncthreads_count(hit); - int32_t* start = plan + 4; - int32_t* dst = start + cap + 1; - int32_t* tok = dst + cap; - auto* ptr = (unsigned long long*) (plan + ptr_off); - if (lead) { - int grp = 0, at = 0; - for (int j = 0; j < i; ++j) - if (s_size[j] > 0) { ++grp; at += s_size[j]; } - s_start[i] = at; - start[grp] = at; - ptr[grp] = slot_ptr[slot]; - } - __syncthreads(); - if (hit) { - int rank = 0; - for (int j = first; j < i; ++j) rank += s_id[j] == e; - dst[s_start[first] + rank] = i; - tok[s_start[first] + rank] = i / k; - } - if (i == 0) { - plan[0] = groups; - plan[1] = entries; - plan[2] = 0; - plan[3] = 0; - start[groups] = entries; - } + hit_plan_block(ids, n, k, res, n_expert, slot_ptr, plan, cap, ptr_off); } __global__ void add_streams_broadcast_kernel(const float* __restrict__ h, const float* __restrict__ e, diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 9018d250b21..0e2207a26f4 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -285,6 +285,108 @@ int test_router_multi(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- a layer's router: the gemv, the top 10, doorbell_publish and verify_hit_plan vs verify_router +int test_verify_router(std::mt19937& rng, cudaStream_t s) { + const int N = 2560, NE = 512, K = 10, slots = 300; + const int64_t cap = 8 * K, ptr_off = ((4 + (cap + 1) + 2 * cap) + 1) & ~1ll, words = ptr_off + 4 * cap + 2; + std::vector w((size_t) NE * N); + std::normal_distribution nd(0.0f, 0.05f); + for (auto& v : w) { + const float f = rng() % 64 == 0 ? edgy(rng) : nd(rng); + uint32_t b; + std::memcpy(&b, &f, 4); + v = (uint16_t) (b >> 16); + } + std::vector slot_ptr(slots); + for (int i = 0; i < slots; ++i) slot_ptr[(size_t) i] = 0x700000000ull + (unsigned long long) i * 1382400ull; + uint16_t* d_w = dev(w.size()); + unsigned long long* d_sp = dev(slots); + int32_t* d_res = dev(NE); + unsigned* d_counter = dev(1); + up(d_w, w); + up(d_sp, slot_ptr); + check(cudaMemset(d_counter, 0, 4), "memset"); + // per path: logits, ids, weights, plan (device); x, ids, weights, seq (mapped) + float *d_logits[2], *d_wts[2], *h_x[2], *m_x[2], *h_w[2], *m_w[2]; + int32_t *d_ids[2], *d_plan[2], *h_ids[2], *m_ids[2]; + uint32_t *h_seq[2], *m_seq[2]; + for (int p = 0; p < 2; ++p) { + d_logits[p] = dev((size_t) 8 * NE); + d_wts[p] = dev((size_t) 8 * K); + d_ids[p] = dev((size_t) 8 * K); + d_plan[p] = dev((size_t) words); + check(cudaHostAlloc((void**) &h_x[p], (size_t) 8 * N * 4, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_ids[p], (size_t) 8 * K * 4, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_w[p], (size_t) 8 * K * 4, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_seq[p], 64, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostGetDevicePointer((void**) &m_x[p], h_x[p], 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_ids[p], h_ids[p], 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_w[p], h_w[p], 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_seq[p], h_seq[p], 0), "mapped"); + } + float* d_x = dev((size_t) 8 * N); + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + for (int rep = 0; rep < 4; ++rep) { + std::vector x((size_t) n_tok * N); + for (auto& v : x) v = rng() % 32 == 0 ? edgy(rng) : std::normal_distribution(0.0f, 1.0f)(rng); + std::vector res((size_t) NE, -1); + for (int e = 0; e < NE; ++e) // resident: none, all, or at random + if (rep == 1 || (rep != 0 && rng() % 3 != 0)) res[(size_t) e] = (int32_t) (rng() % slots); + up(d_x, x); + up(d_res, res); + for (int p = 0; p < 2; ++p) { + check(cudaMemset(d_plan[p], 0xff, (size_t) words * 4), "memset"); + std::memset(h_x[p], 0xcd, (size_t) 8 * N * 4); + *h_seq[p] = 5; + } + // the kernels verify_router stands in for + strata::kernels::bf16_gemv_fp32_mmvf_multi(d_x, d_w, d_logits[0], N, NE, n_tok, s); + strata::kernels::native_router_top10_multi(d_logits[0], d_ids[0], d_wts[0], n_tok, s); + strata::kernels::doorbell_publish(d_x, d_ids[0], d_wts[0], (int64_t) n_tok * N, (int64_t) n_tok * K, m_x[0], + m_ids[0], m_w[0], m_seq[0], s); + strata::kernels::verify_hit_plan(d_ids[0], n_tok * K, K, d_res, NE, d_sp, d_plan[0], cap, ptr_off, s); + strata::kernels::VerifyRouterArgs a; + a.x = d_x; a.w = d_w; a.logits = d_logits[1]; a.ids = d_ids[1]; a.weights = d_wts[1]; + a.x_out = m_x[1]; a.ids_out = m_ids[1]; a.w_out = m_w[1]; a.seq = m_seq[1]; a.ring = 6; + a.res = d_res; a.slot_ptr = d_sp; a.plan = d_plan[1]; a.cap = (int) cap; a.ptr_off = (int) ptr_off; + a.counter = d_counter; a.n_tok = n_tok; a.n_embd = N; a.n_expert = NE; + strata::kernels::verify_router(a, s); + check(cudaStreamSynchronize(s), "router"); + const size_t nk = (size_t) n_tok * K; + int b = bitwise_diff(down(d_logits[0], (size_t) n_tok * NE), down(d_logits[1], (size_t) n_tok * NE), "logits"); + b += bitwise_diff(down(d_wts[0], nk), down(d_wts[1], nk), "weights"); + b += bitwise_diff(std::vector(h_w[0], h_w[0] + nk), std::vector(h_w[1], h_w[1] + nk), "mapped weights"); + b += bitwise_diff(std::vector(h_x[0], h_x[0] + (size_t) n_tok * N), + std::vector(h_x[1], h_x[1] + (size_t) n_tok * N), "mapped x"); + if (down(d_ids[0], nk) != down(d_ids[1], nk) || + std::vector(h_ids[0], h_ids[0] + nk) != std::vector(h_ids[1], h_ids[1] + nk)) { + std::fprintf(stderr, "verify_router: n_tok %d rep %d: the ids differ\n", n_tok, rep); + ++b; + } + if (*h_seq[0] != 6 || *h_seq[1] != 6 || down(d_counter, 1)[0] != 0u) { + std::fprintf(stderr, "verify_router: n_tok %d rep %d: seq %u / %u, counter %u\n", n_tok, rep, *h_seq[0], + *h_seq[1], down(d_counter, 1)[0]); + ++b; + } + if (down(d_plan[0], (size_t) words) != down(d_plan[1], (size_t) words)) { + std::fprintf(stderr, "verify_router: n_tok %d rep %d: the hit plans differ\n", n_tok, rep); + ++b; + } + if (b) std::fprintf(stderr, "verify_router: n_tok %d rep %d: %d mismatches\n", n_tok, rep, b); + bad += b; + } + } + for (int p = 0; p < 2; ++p) { + for (void* q : {(void*) d_logits[p], (void*) d_wts[p], (void*) d_ids[p], (void*) d_plan[p]}) cudaFree(q); + for (void* q : {(void*) h_x[p], (void*) h_ids[p], (void*) h_w[p], (void*) h_seq[p]}) cudaFreeHost(q); + } + for (void* q : {(void*) d_w, (void*) d_sp, (void*) d_res, (void*) d_counter, (void*) d_x}) cudaFree(q); + std::printf("verify_router: %s\n", bad ? "MISMATCH" + : "bitwise the gemv, top 10, doorbell and hit plan (1-8 tokens, 4 routings each)"); + return bad; +} + // ---- the hyper-connection read: fused_gr_read per token vs fused_gr_read_multi int test_gr_read(std::mt19937& rng, cudaStream_t s) { const int64_t N = 2560, HC = 4, D = N * HC, LR = 320; @@ -760,6 +862,7 @@ int main(int argc, char** argv) { bad += test_hit_plan(rng, s); bad += test_mmvf_multi(rng, s); bad += test_router_multi(rng, s); + bad += test_verify_router(rng, s); bad += test_gr_read(rng, s); bad += test_rope_tokens(rng, s); bad += test_kv_append(rng, s); From 34a35dbac1e6ea01951841ea866f48710dfbe01d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 00:28:54 +0200 Subject: [PATCH 48/92] Verify window: the GDN gates and the QSA indexer beside the mixer's projections The GDN alpha/beta kernel (a few blocks, latency-bound) and the QSA indexer's chain (its key projection and append, the query projection, norm and rope, the block scores and top-k) only need the mixer's input; they now run on a branch of the graph beside the projections and join before the recurrence and the attention. Per layer, IQ3_XXS (--window-profile): GDN mixer 85.6 -> 76.5 us on thinking text, 100.6 -> 85.9 on code; QSA 151.4 -> 132.9 and 163.3 -> 142.9. Fixed text against the previous commit, 2 alternating rounds: IQ3_XXS code +2.1%, thinking +3.2%; UD-Q4_K_XL code +1.6%, thinking +0.9%. Same kernels on another stream: window hashes identical (both models, thinking and code; --spec-split, --pcie-frac 0.3, --second-gpu -1). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 5 +- src/core/verify.cpp | 89 +++++++++++++++++++++++----------- 2 files changed, 63 insertions(+), 31 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index b790d29ebe2..0a2ca9ed6dc 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -183,8 +183,9 @@ class Verifier { int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k float* h_pw_ = nullptr; float* m_pw_ = nullptr; // 2 x T * k uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; - cudaStream_t side_ = nullptr; // the prediction's branch of the graph - cudaEvent_t fork_ = nullptr, join_ = nullptr; + cudaStream_t side_ = nullptr; // the prediction's branch of the graph, and in the + cudaEvent_t fork_ = nullptr, join_ = nullptr; // mixer the GDN gates' or the QSA indexer's beside + cudaEvent_t abfork_ = nullptr, abjoin_ = nullptr; // the projections cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 78eac1eb60f..3884c3d867a 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -100,7 +100,7 @@ Verifier::~Verifier() { if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (side_) cudaStreamDestroy(side_); if (shs_) cudaStreamDestroy(shs_); - for (cudaEvent_t e : {fork_, join_, shfork_, shjoin_, res_ready_}) + for (cudaEvent_t e : {fork_, join_, abfork_, abjoin_, shfork_, shjoin_, res_ready_}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); @@ -278,6 +278,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&abfork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&abjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shfork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&res_ready_, cudaEventDisableTiming) != cudaSuccess) { @@ -505,13 +507,26 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { float* hb = h_L_ + (size_t) gi * MT * C; float* gate = gate_L_ + (size_t) gi * MT * HV; float* beta = beta_L_ + (size_t) gi * MT * HV; + // alpha/beta beside the projections (a few blocks, latency-bound), joined before the recurrence + if (cudaEventRecord(abfork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, abfork_, 0) != cudaSuccess) { + err = "verify: the GDN gates' branch could not fork"; + return false; + } + gdn_ab_multi(xm, (const uint16_t*) wa->data, (const uint16_t*) wb->data, (const float*) wdt->data, + (const float*) wsa->data, gate + (size_t) tb * HV, beta + (size_t) tb * HV, (int) N, (int) HV, + n, side_); + if (cudaEventRecord(abjoin_, side_) != cudaSuccess) { + err = "verify: the GDN gates' branch could not join"; + return false; + } quant(xm, N); proj(wqkv, qkv + (size_t) tb * C, N, C); gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb); - gdn_ab_multi(xm, (const uint16_t*) wa->data, (const uint16_t*) wb->data, (const float*) wdt->data, - (const float*) wsa->data, gate + (size_t) tb * HV, beta + (size_t) tb * HV, (int) N, (int) HV, - n, cs); proj(wg, z_ + (size_t) tb * ZV, N, ZV); + if (cudaStreamWaitEvent(cs, abjoin_, 0) != cudaSuccess) { + err = "verify: the GDN gates' branch could not join"; + return false; + } // the recurrence from the untouched state over tokens [0, te); outputs only for this group's gdn_step_norm_multi(state, hb, (int) C, gate, beta, z_, (const float*) wnm->data, EPS, y_, (int) HK, (int) HV, te, nullptr, cs, tb); @@ -530,27 +545,50 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!native_of(wq, v.name("attn_q.weight"), err) || !native_of(wk, v.name("attn_k.weight"), err) || !native_of(wv, v.name("attn_v.weight"), err) || !native_of(wo, v.name("attn_output.weight"), err)) return false; - auto norm_rope = [&](float* data, const WeightRef* norm, int rows, int cols, const int32_t* pos) { - if (native_qsa_enabled()) native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, rows, EPS, cs); - else rms_norm_weighted(data, (const float*) norm->data, rows, cols, EPS, cs); - if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, cs); - else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st.cos_tab, st.sin_tab, pos, cs); + auto norm_rope = [&](float* data, const WeightRef* norm, int rows, int cols, const int32_t* pos, + cudaStream_t ns) { + if (native_qsa_enabled()) native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, rows, EPS, ns); + else rms_norm_weighted(data, (const float*) norm->data, rows, cols, EPS, ns); + if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, ns); + else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st.cos_tab, st.sin_tab, pos, ns); }; // the same for the n tokens' heads in one launch each: token t's positions at pos + t * NH const bool batched = native_qsa_enabled() && native_rope_enabled(); - auto norm_rope_tokens = [&](float* data, const WeightRef* norm, int heads, int cols, const int32_t* pos) { - native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, n * heads, EPS, cs); + auto norm_rope_tokens = [&](float* data, const WeightRef* norm, int heads, int cols, const int32_t* pos, + cudaStream_t ns) { + native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, n * heads, EPS, ns); native_rope_apply_tokens(data, data, n * heads, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, heads, - (int) NH, cs); + (int) NH, ns); }; + // the indexer - its keys appended, the queries' block scores and top-k - on a branch beside the + // attention's projections, joined before the attention + if (cudaEventRecord(abfork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, abfork_, 0) != cudaSuccess) { + err = "verify: the indexer's branch could not fork"; + return false; + } float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, side_); + if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, side_); + const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; + native_qsa_indexer_append_multi(idx_raw + tb * ID, step_ + tb * kStepCount + kStepPos, (int) kStepCount, n, + 0, (const float*) wikn->data, EPS, ib, s, st.max_cells, + (float) qsa_freq_base(), side_); + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, N, IQ * ID, n, side_); + if (batched) norm_rope_tokens(qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + tb * NH, side_); + else for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH, side_); + qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, + s, scores_ + (size_t) tb * max_blocks_, side_, qsa_score_grid_blocks); + qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, + sel_ + (size_t) tb * cap_, side_); + if (cudaEventRecord(abjoin_, side_) != cudaSuccess) { + err = "verify: the indexer's branch could not join"; + return false; + } quant(xm, N); - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, cs); proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD); proj(wv, vcur_ + tb * NKV * HD, N, NKV * HD); - if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH); - else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH); - if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, cs); + if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH, cs); + else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH, cs); if (st.kv_int8) kv_append_q8_steps(st.k_q, st.v_q, st.k_scale, st.v_scale, st.page_table, step_ + tb * kStepCount, (int) kStepCount, kcur_ + tb * NKV * HD, vcur_ + tb * NKV * HD, n, s, cs); @@ -558,10 +596,6 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { for (int t = tb; t < te; ++t) kv_append_step(st.k_pool, st.v_pool, st.page_table, step_ + t * kStepCount, kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); - const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; - native_qsa_indexer_append_multi(idx_raw + tb * ID, step_ + tb * kStepCount + kStepPos, (int) kStepCount, n, - 0, (const float*) wikn->data, EPS, ib, s, st.max_cells, - (float) qsa_freq_base(), cs); proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD); // the query halves of every token's heads (rows of 2 * HD: query, then gate) if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, @@ -569,15 +603,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { err = "verify: the q/gate split failed"; return false; } - if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH); - else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH); - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, N, IQ * ID, n, cs); - if (batched) norm_rope_tokens(qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + tb * NH); - else for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH); - qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, - s, scores_ + (size_t) tb * max_blocks_, cs, qsa_score_grid_blocks); - qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, - sel_ + (size_t) tb * cap_, cs); + if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH, cs); + else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH, cs); + if (cudaStreamWaitEvent(cs, abjoin_, 0) != cudaSuccess) { + err = "verify: the indexer's branch could not join"; + return false; + } QsaAttnPools pools; pools.page_table = st.page_table; if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } From b063c13be2f0687a087fd7ea74325e0bf6d04cd6 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 01:01:18 +0200 Subject: [PATCH 49/92] Verify window: more of the layer on short branches The GDN gate projection runs on the shared expert's stream beside qkv and the conv (an i-quant gate matrix is compute-bound where qkv streams), the QSA queries' chain (projection, split, norm and rope) beside the keys and values, and the routed experts' activations beside the router kernel; each joins before its first reader. The branches' fork/mark/wait are three small helpers. Fixed text against the previous commit, 2 alternating rounds: IQ3_XXS code +0.9%, thinking +0.9%; UD-Q4_K_XL thinking +2.4%, code unchanged. Window hashes identical (both models, thinking and code; --spec-split, --pcie-frac 0.3, --second-gpu -1). Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/verify.hpp | 10 ++- src/core/verify.cpp | 118 ++++++++++++++++++--------------- 2 files changed, 71 insertions(+), 57 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 0a2ca9ed6dc..49268a172d7 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -183,9 +183,13 @@ class Verifier { int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k float* h_pw_ = nullptr; float* m_pw_ = nullptr; // 2 x T * k uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; - cudaStream_t side_ = nullptr; // the prediction's branch of the graph, and in the - cudaEvent_t fork_ = nullptr, join_ = nullptr; // mixer the GDN gates' or the QSA indexer's beside - cudaEvent_t abfork_ = nullptr, abjoin_ = nullptr; // the projections + // short branches beside the main stream: on side_ the GDN gates or the QSA indexer (the mixer) and the experts' + // activations (beside the router), then the prediction; on shs_ the GDN gate projection or the QSA queries (the + // mixer), then the shared expert + cudaStream_t side_ = nullptr; + cudaEvent_t fork_ = nullptr, join_ = nullptr; // the prediction's + cudaEvent_t bfork_ = nullptr, bjoin_ = nullptr; // side_'s others + cudaEvent_t pfork_ = nullptr, pjoin_ = nullptr; // shs_'s projections cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 3884c3d867a..27c0da1a8a6 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -100,7 +100,7 @@ Verifier::~Verifier() { if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (side_) cudaStreamDestroy(side_); if (shs_) cudaStreamDestroy(shs_); - for (cudaEvent_t e : {fork_, join_, abfork_, abjoin_, shfork_, shjoin_, res_ready_}) + for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); @@ -278,8 +278,10 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&abfork_, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&abjoin_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&bfork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&bjoin_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&pfork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&pjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shfork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&res_ready_, cudaEventDisableTiming) != cudaSuccess) { @@ -328,6 +330,23 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { gpu_stamp(stamps_ + ((size_t) T * (size_t) (g.n_layers + 1) + (size_t) row) * kStamps + i, st); }; stamp(g.n_layers, 0, cs); + // the layer's short branches: `fork` starts stream `to` from the main stream's work so far, `mark` records what + // `from` has queued, `wait` makes the main stream wait for a mark + auto fork = [&](cudaStream_t to, cudaEvent_t ev) { + if (cudaEventRecord(ev, cs) == cudaSuccess && cudaStreamWaitEvent(to, ev, 0) == cudaSuccess) return true; + err = "verify: a branch could not fork"; + return false; + }; + auto mark = [&](cudaStream_t from, cudaEvent_t ev) { + if (cudaEventRecord(ev, from) == cudaSuccess) return true; + err = "verify: a branch could not join"; + return false; + }; + auto wait = [&](cudaEvent_t ev) { + if (cudaStreamWaitEvent(cs, ev, 0) == cudaSuccess) return true; + err = "verify: a branch could not join"; + return false; + }; // ---- the window's inputs, from mapped staging; the residency snapshot on the shared expert's branch (the first // hit plan waits for it) @@ -484,9 +503,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (n >= 2) native_quantize_q8_1_il(x, xq_, xil_, (int) n_in, n, cs); else native_quantize_q8_1(x, xq_, (int) n_in, n, cs); }; - auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out) { - if (n >= 2) native_mmvq_il(w->native_type, w->native_data, xq_, xil_, y, (int) n_in, (int) n_out, n, cs); - else native_mmvq(w->native_type, w->native_data, xq_, y, (int) n_in, (int) n_out, n, cs); + auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { + if (n >= 2) native_mmvq_il(w->native_type, w->native_data, xq_, xil_, y, (int) n_in, (int) n_out, n, ps); + else native_mmvq(w->native_type, w->native_data, xq_, y, (int) n_in, (int) n_out, n, ps); }; try { if (!is_qsa_layer(g, l)) { @@ -507,31 +526,25 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { float* hb = h_L_ + (size_t) gi * MT * C; float* gate = gate_L_ + (size_t) gi * MT * HV; float* beta = beta_L_ + (size_t) gi * MT * HV; - // alpha/beta beside the projections (a few blocks, latency-bound), joined before the recurrence - if (cudaEventRecord(abfork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, abfork_, 0) != cudaSuccess) { - err = "verify: the GDN gates' branch could not fork"; - return false; - } + // beside qkv and the conv: alpha/beta (a few blocks, latency-bound) and the gate projection, joined + // before the recurrence + if (!fork(side_, bfork_)) return false; gdn_ab_multi(xm, (const uint16_t*) wa->data, (const uint16_t*) wb->data, (const float*) wdt->data, (const float*) wsa->data, gate + (size_t) tb * HV, beta + (size_t) tb * HV, (int) N, (int) HV, n, side_); - if (cudaEventRecord(abjoin_, side_) != cudaSuccess) { - err = "verify: the GDN gates' branch could not join"; - return false; - } + if (!mark(side_, bjoin_)) return false; quant(xm, N); - proj(wqkv, qkv + (size_t) tb * C, N, C); + if (!fork(shs_, pfork_)) return false; + proj(wg, z_ + (size_t) tb * ZV, N, ZV, shs_); + if (!mark(shs_, pjoin_)) return false; + proj(wqkv, qkv + (size_t) tb * C, N, C, cs); gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb); - proj(wg, z_ + (size_t) tb * ZV, N, ZV); - if (cudaStreamWaitEvent(cs, abjoin_, 0) != cudaSuccess) { - err = "verify: the GDN gates' branch could not join"; - return false; - } + if (!wait(bjoin_) || !wait(pjoin_)) return false; // the recurrence from the untouched state over tokens [0, te); outputs only for this group's gdn_step_norm_multi(state, hb, (int) C, gate, beta, z_, (const float*) wnm->data, EPS, y_, (int) HK, (int) HV, te, nullptr, cs, tb); quant(y_ + (size_t) tb * ZV, ZV); - proj(wout, bo_ + tb * N, ZV, N); + proj(wout, bo_ + tb * N, ZV, N, cs); } else { // ======================= QSA ======================= const int64_t qi = qsa_idx[(size_t) l]; @@ -560,12 +573,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { native_rope_apply_tokens(data, data, n * heads, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, heads, (int) NH, ns); }; - // the indexer - its keys appended, the queries' block scores and top-k - on a branch beside the - // attention's projections, joined before the attention - if (cudaEventRecord(abfork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, abfork_, 0) != cudaSuccess) { - err = "verify: the indexer's branch could not fork"; - return false; - } + // two branches beside the keys and values, joined before the attention: the indexer (its keys appended, + // the queries' block scores and top-k), and the queries + if (!fork(side_, bfork_)) return false; float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, side_); if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, side_); @@ -580,13 +590,21 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { s, scores_ + (size_t) tb * max_blocks_, side_, qsa_score_grid_blocks); qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, sel_ + (size_t) tb * cap_, side_); - if (cudaEventRecord(abjoin_, side_) != cudaSuccess) { - err = "verify: the indexer's branch could not join"; + if (!mark(side_, bjoin_)) return false; + quant(xm, N); + if (!fork(shs_, pfork_)) return false; + proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD, shs_); + // the query halves of every token's heads (rows of 2 * HD: query, then gate) + if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, + (size_t) HD * 4, (size_t) (n * NH), cudaMemcpyDeviceToDevice, shs_) != cudaSuccess) { + err = "verify: the q/gate split failed"; return false; } - quant(xm, N); - proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD); - proj(wv, vcur_ + tb * NKV * HD, N, NKV * HD); + if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH, shs_); + else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH, shs_); + if (!mark(shs_, pjoin_)) return false; + proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD, cs); + proj(wv, vcur_ + tb * NKV * HD, N, NKV * HD, cs); if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH, cs); else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH, cs); if (st.kv_int8) @@ -596,19 +614,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { for (int t = tb; t < te; ++t) kv_append_step(st.k_pool, st.v_pool, st.page_table, step_ + t * kStepCount, kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); - proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD); - // the query halves of every token's heads (rows of 2 * HD: query, then gate) - if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, - (size_t) HD * 4, (size_t) (n * NH), cudaMemcpyDeviceToDevice, cs) != cudaSuccess) { - err = "verify: the q/gate split failed"; - return false; - } - if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH, cs); - else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH, cs); - if (cudaStreamWaitEvent(cs, abjoin_, 0) != cudaSuccess) { - err = "verify: the indexer's branch could not join"; - return false; - } + if (!wait(bjoin_) || !wait(pjoin_)) return false; QsaAttnPools pools; pools.page_table = st.page_table; if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } @@ -622,7 +628,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { for (int t = tb; t < te; ++t) qsa_gate_apply_f32(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, s, attn32_ + t * NH * HD, cs); quant(attn32_ + tb * NH * HD, NH * HD); - proj(wo, bo_ + tb * N, NH * HD, N); + proj(wo, bo_ + tb * N, NH * HD, N, cs); } } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + ": " + e.what(); @@ -631,6 +637,14 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 2, cs); gr_read_group(1, true, inj_, inj2_); stamp(l, 3, cs); + // the routed experts' activations beside the router (joined before the experts) + if (!fork(side_, bfork_)) return false; + if (strata::kernels::cpu::expert_layout().native) + quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, side_); + else + quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, + side_); + if (!mark(side_, bjoin_)) return false; // the router, the doorbell and the main GPU's routed experts - the ones its VRAM tier holds, decided here from // the residency snapshot the pool reads too (it leaves their rows to the GPU) if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { @@ -667,14 +681,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { MT * K, plan_ptr_off_, cs); } stamp(l, 4, cs); - // after the doorbell: the shared expert's branch first (the longest), the routed experts' activations, and the - // next layer's prediction + // after the doorbell: the shared expert's branch, and the next layer's prediction if (beside && !shared(l, grp)) return false; - if (strata::kernels::cpu::expert_layout().native) - quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, cs); - else - quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, cs); - stamp(l, 5, cs); if (predicts(l)) { const WeightRef* wr1 = need(LayerView(wt, l + 1), "ffn_gate_inp.weight", err); if (!wr1) return false; @@ -699,6 +707,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } } + if (!wait(bjoin_)) return false; + stamp(l, 5, cs); return beside || shared(l, grp); }; From 96c30c03604ad49a342f5f59b00ad44434bb25ea Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 01:37:35 +0200 Subject: [PATCH 50/92] DETAILS.md: the speeds with fewer serial kernels per layer Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- docs/DETAILS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 58ad8d421ec..62c31941fc9 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -103,7 +103,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 996-1111 tok/s, generation 58-67 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 991-1111 tok/s, generation 57-64 tok/s. ### A second GPU @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1717-1913 tok/s instead of 996-1111, generation 81-96 tok/s instead of 58-67; -IQ3_XXS 1842-2016 instead of 1449-1601, generation 96-117 instead of 92-104. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1725-1924 tok/s instead of 991-1111, generation 82-98 tok/s instead of 57-64; +IQ3_XXS 1828-2015 instead of 1438-1597, generation 103-128 instead of 97-110. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From e1e64f472266eb0e4075c2ebd0da443268750b92 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 03:13:06 +0200 Subject: [PATCH 51/92] Prompt path: the QSA attention of INT8 pools on the tensor cores qsa_prefill_attn runs INT8 pools through int8 mma (m16n8k32): a block per (KV head, query), its 12 query heads the rows of the fragments, 64 cells a tile. q and the probabilities go in as 24-bit fixed point, three 8-bit limbs each (q against its row's power-of-two bound, a probability times its V scale against the tile's), so the integer sums are exact; converting the limbs back takes magic-number adds and FMAs, not I2F. The block is warp-specialized: 4 warps score 16 cells each over all 256 dims (q's limbs in registers) and run the online softmax, 4 multiply a 64-dim group of V each (V through ldmatrix.trans, cells and dims permuted inside the fragments), 4 fetch K, V and their scales by cp.async into rings of 2 and 3 slots, the pool rows two tiles ahead; named barriers between them. FP16 pools keep the FP32 kernel. On a 131K-token prompt's selections (one layer, 2048 queries, cells spread over 131K): 2.5e-6 of a head's largest output from FP64 at worst, 2.6e-7 on average (the FP32 kernel 3.1e-6 and 4.6e-7); 16-bit fixed point for q or the probabilities would be ~5e-4. 10.8 -> 3.5 ms a call (4.5 ms with the clocks settled; gathering the cells alone takes 2.0 ms, the L2's limit). --prefill-profile, IQ3_XXS: QSA attention 16K 0.95 -> 0.39 s, 131K 8.6 -> 3.4 s. Prompt time, both GPUs: 16K 8.96 -> 8.34 s (1832 -> 1970 tok/s), 131K 69.5 -> 64.1 s (1886 -> 2044 tok/s). Not bitwise: 300 tokens of fixed text after the 16K prompt give 4 (IQ3_XXS) and 2 (UD-Q4_K_XL) different argmaxes, all near-ties by llama.cpp's probabilities (0.39/0.36, 0.27/0.20, 0.32/0.43, 0.42/0.41; 0.36/0.52, 0.32/0.57), llama.cpp siding with the old build at 3 and the new at 3. verify_parity: within 2.4e-6 of the decode kernel. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/kernels/qsa_decode_attn.hpp | 6 +- src/kernels/cuda/qsa_decode_attn.cu | 504 ++++++++++++++++++--- 2 files changed, 452 insertions(+), 58 deletions(-) diff --git a/include/strata/kernels/qsa_decode_attn.hpp b/include/strata/kernels/qsa_decode_attn.hpp index 268f7343f20..0705321f53a 100644 --- a/include/strata/kernels/qsa_decode_attn.hpp +++ b/include/strata/kernels/qsa_decode_attn.hpp @@ -42,8 +42,10 @@ void qsa_decode_attn_batch(const float* q, const QsaAttnPools& pools, const int3 int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream); /// The prompt path's form of qsa_decode_attn_batch (no scratch): a block per (KV head, query) walks the query's -/// cells in tiles of 64 with an online softmax, so no partial sums go through memory. FP32 like the decode -/// kernel; the sums are ordered differently. +/// cells in tiles of 64 with an online softmax, so no partial sums go through memory. INT8 pools on the tensor +/// cores: q and the probabilities as 24-bit fixed point (three 8-bit limbs) against the int8 codes, the integer sums +/// exact, ~2.5e-6 of a head's largest output from FP64 on real prompts (FP32 kernels ~3e-6). FP16 pools in FP32, +/// the sums ordered differently from the decode kernel. void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, const QsaShapes& s, float* attn, int64_t n_q, void* stream); diff --git a/src/kernels/cuda/qsa_decode_attn.cu b/src/kernels/cuda/qsa_decode_attn.cu index 577b78018f1..e33fb27e358 100644 --- a/src/kernels/cuda/qsa_decode_attn.cu +++ b/src/kernels/cuda/qsa_decode_attn.cu @@ -166,27 +166,17 @@ __global__ void __launch_bounds__(HD) attn_merge_kernel(const float* __restrict_ attn[(size_t) h * HD + d] = L > 0.0f ? acc / L : 0.0f; } -// ---- the prompt path: a block per (KV head, query) walks all of the query's cells in tiles, online softmax +// ---- the prompt path, FP16 pools: a block per (KV head, query) walks all of the query's cells in tiles, online +// softmax. Scores: thread (cell, quarter) dots its cell's 64 dimensions of that quarter with the 12 query heads, the +// quarters summed through shared memory. Values: thread (dimension pair, cell parity) accumulates its two dimensions +// of the 12 heads over every other cell; the parities are added at the end. constexpr int PT = 64; // cells per tile -static_assert(KV_Q8_GROUP == 64, "a score thread's 64 dimensions are one int8 scale group"); -// Four int8 codes as floats without I2F (a quarter-rate instruction on sm_86): flip the sign bits, put each byte -// in the mantissa of 2^23 and subtract 2^23 + 128. Exact. -__device__ __forceinline__ void i8x4(uint32_t w, float* f) { - const uint32_t u = w ^ 0x80808080u; -#pragma unroll - for (int j = 0; j < 4; ++j) f[j] = __int_as_float((int) __byte_perm(u, 0x4B000000u, 0x7650u | j)) - 8388736.0f; -} - -// Scores: thread (cell, quarter) dots its cell's 64 dimensions of that quarter with the 12 query heads, the quarters -// summed through shared memory. Values: thread (dimension pair, cell parity) accumulates its two dimensions of the -// 12 heads over every other cell; the parities are added at the end. -template -__global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __restrict__ q, QsaAttnPools p, - const int32_t* __restrict__ ids, - const int32_t* __restrict__ steps, int n_kv_heads, - int page_size, float scale, float* __restrict__ attn, - int cap) { +__global__ void __launch_bounds__(THREADS) attn_prefill_f16_kernel(const float* __restrict__ q, QsaAttnPools p, + const int32_t* __restrict__ ids, + const int32_t* __restrict__ steps, int n_kv_heads, + int page_size, float scale, float* __restrict__ attn, + int cap) { const int kvh = blockIdx.x; const size_t qi = blockIdx.y; const size_t head0 = qi * (size_t) (n_kv_heads * G) + (size_t) kvh * G; @@ -227,21 +217,16 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __re for (int j = 0; j < 64; j += 16) { const int d = quarter * 64 + j; float kf[16]; - if constexpr (INT8) { - const uint4 raw = *reinterpret_cast(p.k_q + row * HD + d); - i8x4(raw.x, kf); i8x4(raw.y, kf + 4); i8x4(raw.z, kf + 8); i8x4(raw.w, kf + 12); - } else { - const uint4* src = reinterpret_cast(p.k_pool + row * HD + d); -#pragma unroll - for (int u = 0; u < 2; ++u) { - const uint4 raw = src[u]; - const __half2* h2 = reinterpret_cast(&raw); -#pragma unroll - for (int v = 0; v < 4; ++v) { - const float2 f = __half22float2(h2[v]); - kf[8 * u + 2 * v] = f.x; - kf[8 * u + 2 * v + 1] = f.y; - } + const uint4* src = reinterpret_cast(p.k_pool + row * HD + d); +#pragma unroll + for (int u = 0; u < 2; ++u) { + const uint4 raw = src[u]; + const __half2* h2 = reinterpret_cast(&raw); +#pragma unroll + for (int v = 0; v < 4; ++v) { + const float2 f = __half22float2(h2[v]); + kf[8 * u + 2 * v] = f.x; + kf[8 * u + 2 * v + 1] = f.y; } } #pragma unroll @@ -259,11 +244,6 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __re s[h] = a; } } - if constexpr (INT8) { - const float ks = __half2float(__ushort_as_half(p.k_scale[row * (HD / KV_Q8_GROUP) + quarter])); -#pragma unroll - for (int h = 0; h < G; ++h) s[h] *= ks; - } } #pragma unroll for (int h = 0; h < G; ++h) spart[quarter][h][cell] = s[h]; @@ -296,17 +276,8 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __re } for (int c = parity; c < n_here; c += 2) { const long long r = srow[c]; - float v0, v1; - if constexpr (INT8) { - const uint32_t w = *reinterpret_cast(p.v_q + r * HD + d0) ^ 0x8080u; - const float sc = __half2float(__ushort_as_half(p.v_scale[r * (HD / KV_Q8_GROUP) + d0 / KV_Q8_GROUP])); - v0 = (__int_as_float((int) __byte_perm(w, 0x4B000000u, 0x7650u)) - 8388736.0f) * sc; - v1 = (__int_as_float((int) __byte_perm(w, 0x4B000000u, 0x7651u)) - 8388736.0f) * sc; - } else { - const float2 f = __half22float2(*reinterpret_cast(p.v_pool + r * HD + d0)); - v0 = f.x; - v1 = f.y; - } + const float2 f = __half22float2(*reinterpret_cast(p.v_pool + r * HD + d0)); + const float v0 = f.x, v1 = f.y; const float4* pc = reinterpret_cast(sp[c]); const float4 pa = pc[0], pb = pc[1], pd = pc[2]; const float pr[G] = {pa.x, pa.y, pa.z, pa.w, pb.x, pb.y, pb.z, pb.w, pd.x, pd.y, pd.z, pd.w}; @@ -338,6 +309,421 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_kernel(const float* __re } } +// ---- the prompt path, INT8 pools: the codes on the tensor cores (mma m16n8k32, int8). A block per (KV head, query), +// its 12 query heads the rows of the 16-row fragments, 64 cells a tile. q and the probabilities enter as 24-bit fixed +// point, three 8-bit limbs each; the integer sums are exact, so the error is the fixed point's: q to 2^-23 of its row's +// power-of-two bound, a probability times its V scale to 2^-23 of its tile's. Warp-specialized, one block an SM: +// warps 0-3 score 16 cells of each tile over all 256 dims (q's limbs in registers) and run the online softmax; +// warps 4-7 multiply a 64-dim group of V each: the probabilities times the V scales as limbs against V through +// ldmatrix.trans (cells and dims permuted inside the fragments, the same way on both sides); +// warps 8-11 fetch by cp.async: 8-9 the K codes and scales, 10-11 V's, the pool rows two tiles ahead. +// Rings in shared memory between them (K 2 slots, V 3, probabilities 2), synchronized by named barriers. +namespace pf { +constexpr int NT = 384; +constexpr int KS = 2, VS = 3; // slots of the K and V rings +enum : int { + END = 0, FULL_K = 1, EMPTY_K = FULL_K + KS, FULL_V = EMPTY_K + KS, EMPTY_V = FULL_V + VS, FULL_P = EMPTY_V + VS, + EMPTY_P = FULL_P + 2, SRED = EMPTY_P + 2 +}; +static_assert(SRED < 16, "sixteen named barriers"); +constexpr int NKB = 64 + 128, NVB = 64 + 128, NPB = 128 + 128; // threads at the K, V and P barriers (END as P) +static_assert(KV_Q8_GROUP == 64 && HD == 256, "four 64-dim scale groups a row"); +constexpr int MAGIC = 0x4B400000; // 1.5 * 2^23 as float bits: an int x (|x| < 2^22) added gives 1.5 * 2^23 + x +constexpr float MAGIC_F = 12582912.0f; + +struct __align__(16) Smem { + uint8_t k[KS][PT * HD]; // K codes, 16-byte chunk j of cell c at j ^ ksw(c) + uint8_t v[VS][PT * HD]; // V codes, chunk j of cell c at j ^ (c & 7) + uint2 ks[KS][PT]; // FP16 scales of a cell's four dim groups + uint2 vs[VS][PT]; + float p[2][16][PT]; // probabilities, float4 i of row r at i ^ ((r & 1) << 2) + float alpha[2][16]; // a row's rescale of the running sums, and the tile's largest probability + float pmax[2][16]; + float red[2][4][16]; // the score warps' tile maxima + float lsum[4][16]; + float qmax[16], qsc[16]; +}; + +// Score warp w's B fragments: column j of its n-tile n is cell 16w + 4 (j >> 1) + 2n + (j & 1), so a lane's C +// fragments hold four consecutive cells; this swizzle puts the eight cells an ldmatrix reads on distinct banks. +__device__ __forceinline__ int ksw(int c) { return ((c >> 1) & 6) | (c & 1); } +__device__ __forceinline__ void bar_sync(int id, int n) { + asm volatile("bar.sync %0, %1;\n" ::"r"(id), "r"(n) : "memory"); +} +__device__ __forceinline__ void bar_arrive(int id, int n) { + asm volatile("bar.arrive %0, %1;\n" ::"r"(id), "r"(n) : "memory"); +} +// where cell c of probability row r lives in Smem::p +__device__ __forceinline__ int p_at(int r, int c) { return (((c >> 2) ^ ((r & 1) << 2)) << 2) | (c & 3); } +__device__ __forceinline__ void mma_u8s8(int (&d)[4], const unsigned (&a)[4], unsigned b0, unsigned b1) { + asm volatile("mma.sync.aligned.m16n8k32.row.col.s32.u8.s8.s32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%0,%1,%2,%3};\n" + : "+r"(d[0]), "+r"(d[1]), "+r"(d[2]), "+r"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} +__device__ __forceinline__ void mma_s8s8(int (&d)[4], const unsigned (&a)[4], unsigned b0, unsigned b1) { + asm volatile("mma.sync.aligned.m16n8k32.row.col.s32.s8.s8.s32 {%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%0,%1,%2,%3};\n" + : "+r"(d[0]), "+r"(d[1]), "+r"(d[2]), "+r"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b0), "r"(b1)); +} +__device__ __forceinline__ void ldsm4(unsigned (&r)[4], const void* s) { + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"((unsigned) __cvta_generic_to_shared(s))); +} +__device__ __forceinline__ void ldsm4t(unsigned (&r)[4], const void* s) { + asm volatile("ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) + : "r"((unsigned) __cvta_generic_to_shared(s))); +} +__device__ __forceinline__ void cp_async16(void* s, const void* g) { + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" ::"r"((unsigned) __cvta_generic_to_shared(s)), "l"(g)); +} +__device__ __forceinline__ void cp_async8(void* s, const void* g) { + asm volatile("cp.async.ca.shared.global [%0], [%1], 8;\n" ::"r"((unsigned) __cvta_generic_to_shared(s)), "l"(g)); +} +__device__ __forceinline__ int exp_of(float x) { return (__float_as_int(x) >> 23) & 255; } // x < 2^(E - 126) +__device__ __forceinline__ float pow2f(int e) { return __int_as_float((e + 127) << 23); } // -126 <= e <= 127 +// w (65536 r2 + 256 r1 + r0) from three limbs' sums (|r| < 2^21), w a power of two and wk = -65793 C w: the first +// multiply-add is exact, the other two round +__device__ __forceinline__ float limbs3(int r0, int r1, int r2, float w65536, float w256, float w, float wk) { + const float u0 = __int_as_float(r0 + MAGIC), u1 = __int_as_float(r1 + MAGIC), u2 = __int_as_float(r2 + MAGIC); + return fmaf(u0, w, fmaf(u1, w256, fmaf(u2, w65536, wk))); +} +// bytes 0-2 of four values as three words: byte j of word i is byte i of value j +__device__ __forceinline__ void pack3(unsigned f0, unsigned f1, unsigned f2, unsigned f3, unsigned& w0, unsigned& w1, + unsigned& w2) { + const unsigned t01 = __byte_perm(f0, f1, 0x5140), t23 = __byte_perm(f2, f3, 0x5140); + w0 = __byte_perm(t01, t23, 0x5410); + w1 = __byte_perm(t01, t23, 0x7632); + w2 = __byte_perm(__byte_perm(f0, f1, 0x0062), __byte_perm(f2, f3, 0x0062), 0x5410); +} +// x >= 0 rounded to an integer below 2^23, in the low 23 bits +__device__ __forceinline__ unsigned fx23(float x) { return __float_as_uint(fminf(x, 8388607.0f) + 8388608.0f); } +// dim group g's FP16 scale of a cell's four +__device__ __forceinline__ float scale_of(uint2 s, int g) { + return __half2float(__ushort_as_half((unsigned short) ((g < 2 ? s.x : s.y) >> (16 * (g & 1))))); +} +} // namespace pf + +__global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float* __restrict__ q, QsaAttnPools p, + const int32_t* __restrict__ ids, + const int32_t* __restrict__ steps, int n_kv_heads, + int page_size, float scale, float* __restrict__ attn, + int cap) { + using namespace pf; + extern __shared__ __align__(16) uint8_t smem_raw[]; + Smem& sm = *reinterpret_cast(smem_raw); + const int kvh = blockIdx.x; + const size_t qi = blockIdx.y; + const size_t head0 = qi * (size_t) (n_kv_heads * G) + (size_t) kvh * G; + q += head0 * HD; + attn += head0 * HD; + ids += qi * (size_t) cap; + const int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + const int n_tiles = (n_ids + PT - 1) / PT; + const int t = threadIdx.x, lane = t & 31, warp = t >> 5, gid = lane >> 2, tig = lane & 3; + const float kS = -65793.0f * MAGIC_F; + + // each query row's largest magnitude, and its scale 2^(e - 23) (|q| < 2^e) times the softmax scale + if (t < 256) { + const int r = t >> 4, ch = t & 15; + float mx = 0.0f; + if (r < G) +#pragma unroll + for (int i = 0; i < 16; i += 4) { + const float4 f = *reinterpret_cast(q + (size_t) r * HD + ch * 16 + i); + mx = fmaxf(mx, fmaxf(fmaxf(fabsf(f.x), fabsf(f.y)), fmaxf(fabsf(f.z), fabsf(f.w)))); + } +#pragma unroll + for (int o = 8; o > 0; o >>= 1) mx = fmaxf(mx, __shfl_xor_sync(0xffffffffu, mx, o)); + if (ch == 0) { + sm.qmax[r] = mx; + sm.qsc[r] = mx > 0.0f ? pow2f(exp_of(mx) - 126 - 23) * scale : 0.0f; + } + } + __syncthreads(); + if (n_tiles == 0) { + for (int i = t; i < G * HD; i += NT) attn[i] = 0.0f; + return; + } + + if (warp >= 8) { + // ======== fetch: warps 8-9 K, 10-11 V. Warp 8 + h and 10 + h take cells 32h .. 32h+31 of each tile, a lane a + // cell and its row; tile i is issued, then tile i-1 signalled once its copies have landed + const bool is_k = warp < 10; + const int h = warp & 1, cl = 32 * h + lane, nb = is_k ? NKB : NVB, slots = is_k ? KS : VS; + const int8_t* src = is_k ? p.k_q : p.v_q; + const uint16_t* scl = is_k ? p.k_scale : p.v_scale; + auto row_of = [&](int id, int pg) { return (pg * n_kv_heads + kvh) * page_size + id % page_size; }; + // tile i's row of cell cl; tile i+1's id and page; tile i+2's id + int row = -1, id1 = 0, pg1 = 0, id2 = 0; + if (cl < n_ids) { + const int id = ids[cl]; + row = row_of(id, p.page_table[id / page_size]); + } + if (PT + cl < n_ids) { + id1 = ids[PT + cl]; + pg1 = p.page_table[id1 / page_size]; + } + if (2 * PT + cl < n_ids) id2 = ids[2 * PT + cl]; + for (int i = 0; i < n_tiles; ++i) { + const int s = i % slots, c0 = i * PT, n_here = min(PT, n_ids - c0); + if (i >= slots) bar_sync((is_k ? EMPTY_K : EMPTY_V) + s, nb); + uint8_t* dst = is_k ? sm.k[s] : sm.v[s]; +#pragma unroll + for (int u = 0; u < 16; ++u) { + const int e = lane + 32 * u, cw = e >> 4, j = e & 15, c = 32 * h + cw; + const int r = __shfl_sync(0xffffffffu, row, cw); + if (c < n_here) + cp_async16(dst + c * HD + ((j ^ (is_k ? ksw(c) : c & 7)) << 4), src + (size_t) r * HD + j * 16); + } + uint2* sd = is_k ? &sm.ks[s][cl] : &sm.vs[s][cl]; + if (cl < n_here) cp_async8(sd, scl + (size_t) row * (HD / KV_Q8_GROUP)); + else *sd = make_uint2(0u, 0u); // no scale for the multiply of a missing cell's stale codes + asm volatile("cp.async.commit_group;\n" ::); + if (i >= 1) { + asm volatile("cp.async.wait_group 1;\n" ::); + bar_arrive((is_k ? FULL_K : FULL_V) + (i - 1) % slots, nb); + } + row = cl < n_ids - c0 - PT ? row_of(id1, pg1) : -1; + id1 = id2; + pg1 = c0 + 2 * PT + cl < n_ids ? p.page_table[id2 / page_size] : 0; + id2 = c0 + 3 * PT + cl < n_ids ? ids[c0 + 3 * PT + cl] : 0; + } + asm volatile("cp.async.wait_group 0;\n" ::); + bar_arrive((is_k ? FULL_K : FULL_V) + (n_tiles - 1) % slots, nb); + } else if (warp < 4) { + // ======== scores: cells 16w + 4tig .. +3 of each tile in this lane's C fragments (rows gid, gid + 8) + const int w = warp; + unsigned qa[3][8][4]; // q's limbs as A fragments: [limb][k-step][register] + { + float inv[2]; +#pragma unroll + for (int r = 0; r < 2; ++r) { + const float mx = sm.qmax[gid + 8 * r]; + inv[r] = mx > 0.0f ? pow2f(23 - (exp_of(mx) - 126)) : 0.0f; + } +#pragma unroll + for (int ks = 0; ks < 8; ++ks) +#pragma unroll + for (int r = 0; r < 2; ++r) +#pragma unroll + for (int hf = 0; hf < 2; ++hf) { + const int row = gid + 8 * r; + float4 f = make_float4(0.f, 0.f, 0.f, 0.f); + if (row < G) + f = *reinterpret_cast(q + (size_t) row * HD + 32 * ks + 16 * hf + 4 * tig); + const float fv[4] = {f.x, f.y, f.z, f.w}; + unsigned u[4]; +#pragma unroll + for (int e = 0; e < 4; ++e) + u[e] = (unsigned) max(-8388607, min(8388607, __float2int_rn(fv[e] * inv[r]))); + pack3(u[0], u[1], u[2], u[3], qa[0][ks][r + 2 * hf], qa[1][ks][r + 2 * hf], qa[2][ks][r + 2 * hf]); + } + } + const float qs[2] = {sm.qsc[gid], sm.qsc[gid + 8]}; + float m_run[2] = {-FLT_MAX, -FLT_MAX}, lp[2] = {0.0f, 0.0f}; + for (int i = 0; i < n_tiles; ++i) { + const int s = i % KS, sp = i & 1, n_here = min(PT, n_ids - i * PT); + bar_sync(FULL_K + s, NKB); + uint2 kr[4]; +#pragma unroll + for (int e = 0; e < 4; ++e) kr[e] = sm.ks[s][16 * w + 4 * tig + e]; + float sv[2][4] = {{0.f, 0.f, 0.f, 0.f}, {0.f, 0.f, 0.f, 0.f}}; +#pragma unroll + for (int g = 0; g < 4; ++g) { + unsigned b[2][4]; +#pragma unroll + for (int n = 0; n < 2; ++n) { + const int cb = 16 * w + 4 * ((lane & 7) >> 1) + 2 * n + (lane & 1); + ldsm4(b[n], &sm.k[s][cb * HD + (((4 * g + (lane >> 3)) ^ ksw(cb)) << 4)]); + } + int acc[3][2][4]; +#pragma unroll + for (int j = 0; j < 3; ++j) +#pragma unroll + for (int n = 0; n < 2; ++n) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) acc[j][n][ii] = 0; +#pragma unroll + for (int kk = 0; kk < 2; ++kk) +#pragma unroll + for (int j = 0; j < 3; ++j) +#pragma unroll + for (int n = 0; n < 2; ++n) { + if (j < 2) mma_u8s8(acc[j][n], qa[j][2 * g + kk], b[n][2 * kk], b[n][2 * kk + 1]); + else mma_s8s8(acc[j][n], qa[j][2 * g + kk], b[n][2 * kk], b[n][2 * kk + 1]); + } +#pragma unroll + for (int n = 0; n < 2; ++n) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) + sv[n][ii] = fmaf(limbs3(acc[0][n][ii], acc[1][n][ii], acc[2][n][ii], 65536.0f, 256.0f, 1.0f, kS), + scale_of(kr[2 * n + (ii & 1)], g), sv[n][ii]); + } + if (i + KS < n_tiles) bar_arrive(EMPTY_K + s, NKB); + // online softmax: the tile's maximum over the four warps + float mx[2] = {-FLT_MAX, -FLT_MAX}; +#pragma unroll + for (int n = 0; n < 2; ++n) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) { + const int r = ii >> 1, cell = 16 * w + 4 * tig + 2 * n + (ii & 1); + const float v = cell < n_here ? sv[n][ii] * qs[r] : -FLT_MAX; + sv[n][ii] = v; + mx[r] = fmaxf(mx[r], v); + } +#pragma unroll + for (int r = 0; r < 2; ++r) { + mx[r] = fmaxf(mx[r], __shfl_xor_sync(0xffffffffu, mx[r], 1)); + mx[r] = fmaxf(mx[r], __shfl_xor_sync(0xffffffffu, mx[r], 2)); + } + if (tig == 0) { sm.red[sp][w][gid] = mx[0]; sm.red[sp][w][gid + 8] = mx[1]; } + bar_sync(SRED, 128); + float al[2], pm[2], pv[2][4]; +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + const float tmax = fmaxf(fmaxf(sm.red[sp][0][row], sm.red[sp][1][row]), + fmaxf(sm.red[sp][2][row], sm.red[sp][3][row])); + const float m_new = fmaxf(m_run[r], tmax); + al[r] = __expf(m_run[r] - m_new); + pm[r] = __expf(tmax - m_new); + m_run[r] = m_new; + } +#pragma unroll + for (int n = 0; n < 2; ++n) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) { + const int r = ii >> 1, cell = 16 * w + 4 * tig + 2 * n + (ii & 1); + pv[r][2 * n + (ii & 1)] = cell < n_here ? __expf(sv[n][ii] - m_run[r]) : 0.0f; + } +#pragma unroll + for (int r = 0; r < 2; ++r) lp[r] = fmaf(lp[r], al[r], (pv[r][0] + pv[r][1]) + (pv[r][2] + pv[r][3])); + if (i >= 2) bar_sync(EMPTY_P + sp, NPB); +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + *reinterpret_cast(&sm.p[sp][row][p_at(row, 16 * w + 4 * tig)]) = + make_float4(pv[r][0], pv[r][1], pv[r][2], pv[r][3]); + } + if (w == 0 && tig == 0) { + sm.alpha[sp][gid] = al[0]; sm.alpha[sp][gid + 8] = al[1]; + sm.pmax[sp][gid] = pm[0]; sm.pmax[sp][gid + 8] = pm[1]; + } + bar_arrive(FULL_P + sp, NPB); + } +#pragma unroll + for (int r = 0; r < 2; ++r) { + lp[r] += __shfl_xor_sync(0xffffffffu, lp[r], 1); + lp[r] += __shfl_xor_sync(0xffffffffu, lp[r], 2); + } + if (tig == 0) { sm.lsum[w][gid] = lp[0]; sm.lsum[w][gid + 8] = lp[1]; } + bar_sync(END, NPB); + } else { + // ======== values: dims 64g .. 64g+63 as 16-dim blocks db; n-tile (db, par) column j is dim + // 64g + 16db + 2j + par, and k index 4tig + e of a k-step's half hf is cell 16hf + {2tig, 2tig+1, 2tig+8, + // 2tig+9}[e] (as ldmatrix.trans hands out V and two byte permutes rearrange it) + const int g = warp - 4; + float o[4][2][4]; +#pragma unroll + for (int a = 0; a < 4; ++a) +#pragma unroll + for (int b = 0; b < 2; ++b) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) o[a][b][ii] = 0.0f; + for (int i = 0; i < n_tiles; ++i) { + const int sp = i & 1, s = i % VS; + bar_sync(FULL_P + sp, NPB); + bar_sync(FULL_V + s, NVB); + // the fixed point of p * vs: 2^-23 of the tile's largest p times its largest V scale in this group + float vmax = fmaxf(scale_of(sm.vs[s][lane], g), scale_of(sm.vs[s][lane + 32], g)); +#pragma unroll + for (int o_ = 16; o_ > 0; o_ >>= 1) vmax = fmaxf(vmax, __shfl_xor_sync(0xffffffffu, vmax, o_)); + float inv[2], ps[2], al[2]; +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + const int eb = exp_of(sm.pmax[sp][row] * vmax) - 126; + const bool ok = eb >= -103; + inv[r] = ok ? pow2f(23 - eb) : 0.0f; + ps[r] = ok ? pow2f(eb - 23) : 0.0f; + al[r] = sm.alpha[sp][row]; + } + unsigned pa[3][2][4]; // [limb][k-step][register] +#pragma unroll + for (int kk = 0; kk < 2; ++kk) +#pragma unroll + for (int hf = 0; hf < 2; ++hf) { + const int c = 32 * kk + 16 * hf + 2 * tig; // cells c, c+1, c+8, c+9 + const float v0 = scale_of(sm.vs[s][c], g), v1 = scale_of(sm.vs[s][c + 1], g); + const float v8 = scale_of(sm.vs[s][c + 8], g), v9 = scale_of(sm.vs[s][c + 9], g); +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + const float2 pa2 = *reinterpret_cast(&sm.p[sp][row][p_at(row, c)]); + const float2 pb2 = *reinterpret_cast(&sm.p[sp][row][p_at(row, c + 8)]); + const int x = r + 2 * hf; + pack3(fx23(pa2.x * v0 * inv[r]), fx23(pa2.y * v1 * inv[r]), fx23(pb2.x * v8 * inv[r]), + fx23(pb2.y * v9 * inv[r]), pa[0][kk][x], pa[1][kk][x], pa[2][kk][x]); + } + } + if (i + 2 < n_tiles) bar_arrive(EMPTY_P + sp, NPB); +#pragma unroll + for (int db = 0; db < 4; ++db) { + unsigned bv[2][2][2]; // [k-step][parity][b0, b1] +#pragma unroll + for (int kk = 0; kk < 2; ++kk) { + const int cell = 32 * kk + 8 * (lane >> 3) + (lane & 7); + unsigned r4[4]; + ldsm4t(r4, &sm.v[s][cell * HD + (((4 * g + db) ^ (cell & 7)) << 4)]); + bv[kk][0][0] = __byte_perm(r4[0], r4[1], 0x6420); + bv[kk][1][0] = __byte_perm(r4[0], r4[1], 0x7531); + bv[kk][0][1] = __byte_perm(r4[2], r4[3], 0x6420); + bv[kk][1][1] = __byte_perm(r4[2], r4[3], 0x7531); + } + int acc[3][2][4]; +#pragma unroll + for (int j = 0; j < 3; ++j) +#pragma unroll + for (int par = 0; par < 2; ++par) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) acc[j][par][ii] = 0; +#pragma unroll + for (int kk = 0; kk < 2; ++kk) +#pragma unroll + for (int j = 0; j < 3; ++j) +#pragma unroll + for (int par = 0; par < 2; ++par) mma_u8s8(acc[j][par], pa[j][kk], bv[kk][par][0], bv[kk][par][1]); +#pragma unroll + for (int par = 0; par < 2; ++par) +#pragma unroll + for (int ii = 0; ii < 4; ++ii) { + const int r = ii >> 1; + o[db][par][ii] = fmaf(o[db][par][ii], al[r], + limbs3(acc[0][par][ii], acc[1][par][ii], acc[2][par][ii], 65536.0f * ps[r], + 256.0f * ps[r], ps[r], kS * ps[r])); + } + } + if (i + VS < n_tiles) bar_arrive(EMPTY_V + s, NVB); + } + bar_sync(END, NPB); +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + if (row >= G) continue; + const float l = (sm.lsum[0][row] + sm.lsum[1][row]) + (sm.lsum[2][row] + sm.lsum[3][row]); +#pragma unroll + for (int db = 0; db < 4; ++db) { // dims 64g + 16db + 4tig + {0, 1, 2, 3} + float4 out = make_float4(0.f, 0.f, 0.f, 0.f); + if (l > 0.0f) out = make_float4(o[db][0][2 * r] / l, o[db][1][2 * r] / l, o[db][0][2 * r + 1] / l, + o[db][1][2 * r + 1] / l); + *reinterpret_cast(&attn[(size_t) row * HD + 64 * g + 16 * db + 4 * tig]) = out; + } + } + } +} + } // namespace void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, @@ -351,18 +737,24 @@ void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* } const float scale = 1.0f / sqrtf((float) HD); cudaStream_t st = (cudaStream_t) stream; + static bool attr = false; + if (int8 && !attr) { + cudaFuncSetAttribute(attn_prefill_mma_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, (int) sizeof(pf::Smem)); + attr = true; + } for (int64_t q0 = 0; q0 < n_q; q0 += 65535) { const int64_t nb = n_q - q0 < 65535 ? n_q - q0 : 65535; const dim3 grid((unsigned) s.n_head_kv, (unsigned) nb); const size_t qo = (size_t) q0 * (size_t) s.n_head * HD; + const int32_t* ids_q = ids + (size_t) q0 * (size_t) cap; + const int32_t* steps_q = steps + (size_t) q0 * kStepCount; if (int8) - attn_prefill_kernel<<>>(q + qo, pools, ids + (size_t) q0 * (size_t) cap, - steps + (size_t) q0 * kStepCount, (int) s.n_head_kv, - (int) s.page_size, scale, attn + qo, (int) cap); + attn_prefill_mma_kernel<<>>(q + qo, pools, ids_q, steps_q, + (int) s.n_head_kv, (int) s.page_size, scale, + attn + qo, (int) cap); else - attn_prefill_kernel<<>>(q + qo, pools, ids + (size_t) q0 * (size_t) cap, - steps + (size_t) q0 * kStepCount, (int) s.n_head_kv, - (int) s.page_size, scale, attn + qo, (int) cap); + attn_prefill_f16_kernel<<>>(q + qo, pools, ids_q, steps_q, (int) s.n_head_kv, + (int) s.page_size, scale, attn + qo, (int) cap); } const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { From dcf63ac7617c3d0921bf27e052d1b089874bdb3b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 03:41:30 +0200 Subject: [PATCH 52/92] DETAILS.md: the speeds with the prompt path's attention on the tensor cores Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- docs/DETAILS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 62c31941fc9..5d4adefe1f8 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -103,7 +103,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 991-1111 tok/s, generation 57-64 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 992-1130 tok/s, generation 58-65 tok/s. ### A second GPU @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1725-1924 tok/s instead of 991-1111, generation 82-98 tok/s instead of 57-64; -IQ3_XXS 1828-2015 instead of 1438-1597, generation 103-128 instead of 97-110. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1819-2063 tok/s instead of 992-1130, generation 80-100 tok/s instead of 58-65; +IQ3_XXS 1959-2184 instead of 1516-1699, generation 102-126 instead of 97-111. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From f67f228708537c31a87c75527e3e898a44c628e8 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 05:47:57 +0200 Subject: [PATCH 53/92] MTP draft layer: experts in GGUF formats through the native expert kernels tools/mtp_pack.py --experts iq4_nl (ggml's quantizer, bitwise on the checkpoint's experts) or q8_0, then tools/mtp_rt.py writes the experts as GGUF blocks [gate | up | down] with experts.txt naming the ggml types; the draft layer runs them through native_expert_grouped. Without experts.txt, the Q2_0 blobs as before (window hashes unchanged). Fixed text, both GPUs, IQ4_NL against Q2_0 (+675 MiB of VRAM): IQ3_XXS code +3.5%, thinking +0.6%, after a 16K prompt +4.7% (the 3090's cache keeps its 8000 slots); UD-Q4_K_XL -1.6% / +2.0% / +0.7% (4312 slots instead of 4537). Q8_0 passes the same tokens per window at 2.7x the VRAM. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- include/strata/core/mtp.hpp | 8 +++- src/core/mtp.cpp | 47 ++++++++++++++++------ tools/mtp_pack.py | 77 +++++++++++++++++++++++++++---------- tools/mtp_rt.py | 33 +++++++++++----- 4 files changed, 122 insertions(+), 43 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index 68af2074a0e..ac8cd57c4c1 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -16,13 +16,15 @@ // * the attention is DENSE over every cell the layer has seen - identical to the model's sparse selection // below 2,051 cells - so speculative cells (draft steps, rejected window rows) never touch indexer state and // are simply overwritten when their positions are processed again; -// * all 512 routed experts live in VRAM (708 MB) and run through the grouped hit kernels; +// * all 512 routed experts live in VRAM and run through the grouped kernels: raw GGUF blocks in the formats +// `experts.txt` names, through the main model's native expert kernels, or else the Q2_0 blob layout (708 MB); // * the K/V of a prompt's cells come from the batched prompt path (`prompt_kv`), in its GEMM arithmetic, as the // model's own layers there. #pragma once #include "strata/core/layer.hpp" #include "strata/core/session.hpp" +#include "strata/kernels/iq_kernels.hpp" #include @@ -110,6 +112,8 @@ class MtpDrafter { std::vector tensors_; uint8_t* dense_ = nullptr; uint8_t* experts_ = nullptr; + uint64_t blob_ = 0; // bytes per expert + strata::kernels::NativeExpertLayout nat_; // gu_type >= 0: GGUF blocks (experts.txt); else the Q2_0 blobs void* state_arena_ = nullptr; QsaState st_; void* arena_ = nullptr; @@ -138,7 +142,7 @@ class MtpDrafter { float* attn_scratch_ = nullptr; float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *y_ = nullptr, *sample_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr, *out_ids_ = nullptr; - uint8_t* hit_xq_ = nullptr; + uint8_t *hit_xq_ = nullptr, *nat_xq_ = nullptr; unsigned long long* grp_ptr_ = nullptr; int32_t *grp_start_ = nullptr, *grp_counts_ = nullptr; float* hit_xs_ = nullptr; diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index a9a44ff97c3..159302f92f7 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -133,11 +133,25 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session cudaMemcpy(dense_, blob.data(), blob.size(), cudaMemcpyHostToDevice); vram_ += blob.size(); } - // ---- the 512 routed experts, one blob each + // ---- the 512 routed experts, one blob each: raw GGUF blocks [gate | up | down] in the formats experts.txt names + // (the ggml type ids of gate/up and down), or without it the Q2_0 blob layout of cpu/expert.hpp { - const uint64_t bytes = (uint64_t) g.n_expert * strata::kernels::cpu::BLOB; - std::ifstream f(rt_dir + "/experts.bin", std::ios::binary); + blob_ = strata::kernels::cpu::BLOB; + if (std::ifstream fmt(rt_dir + "/experts.txt"); fmt) { + int gu = -1, dn = -1; + fmt >> gu >> dn; + if (!strata::kernels::native_expert_supported(gu, dn, g.n_embd, g.n_ff)) { + err = "mtp: experts.txt names formats the GPU expert kernels lack"; + return false; + } + nat_ = strata::kernels::native_expert_layout(gu, dn, g.n_embd, g.n_ff); + blob_ = nat_.bytes; + } + const uint64_t bytes = (uint64_t) g.n_expert * blob_; + std::ifstream f(rt_dir + "/experts.bin", std::ios::binary | std::ios::ate); if (!f) { err = "mtp: cannot open experts.bin"; return false; } + if ((uint64_t) f.tellg() != bytes) { err = "mtp: experts.bin does not match its format"; return false; } + f.seekg(0); if (cudaMalloc((void**) &experts_, bytes) != cudaSuccess) { err = "mtp: the 512 experts do not fit in VRAM"; return false; } std::vector chunk(64u << 20); for (uint64_t off = 0; off < bytes;) { @@ -198,7 +212,10 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session grp_ptr_ = b.take(T * K); grp_start_ = b.take(T * K + 1); grp_counts_ = b.take(4); hit_xq_ = b.take(T * (N / 32) * 34); hit_xs_ = b.take(T * (N / 32)); - hit_scratch_ = b.take(strata::kernels::moe_hit_grouped_scratch_bytes((int64_t) (T * K), g.n_embd, g.n_ff)); + nat_xq_ = b.take(T * (N / 32) * 36); + hit_scratch_ = b.take(std::max( + strata::kernels::moe_hit_grouped_scratch_bytes((int64_t) (T * K), g.n_embd, g.n_ff), + strata::kernels::native_expert_scratch_bytes((int64_t) (T * K), g.n_ff))); sh_scratch_ = (float*) b.take(strata::kernels::shared_expert_scratch_bytes(g.n_ff)); x_bf16_ = b.take(N); out_ids_ = b.take(T + 4); @@ -220,8 +237,10 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session cudaMemcpy(ident_, id.data(), id.size() * 4, cudaMemcpyHostToDevice); } if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess) { err = "mtp: stream"; return false; } - std::fprintf(stderr, "strata mtp: draft layer loaded, %.0f MiB of VRAM (experts %.0f, dense %.0f)\n", - (double) vram_ / 1048576.0, (double) g.n_expert * strata::kernels::cpu::BLOB / 1048576.0, + char fmts[48] = "Q2_0 blobs"; + if (nat_.gu_type >= 0) std::snprintf(fmts, sizeof fmts, "ggml types %d/%d", nat_.gu_type, nat_.d_type); + std::fprintf(stderr, "strata mtp: draft layer loaded, %.0f MiB of VRAM (experts %.0f as %s, dense %.0f)\n", + (double) vram_ / 1048576.0, (double) g.n_expert * blob_ / 1048576.0, fmts, (double) tensors_.back().off / 1048576.0); return true; } @@ -362,11 +381,17 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri if (native_router_enabled()) native_router_top10(logits_ + t * g.n_expert, ids_ + t * K, w_ + t * K, cs); else router_top10(logits_ + t * g.n_expert, 1, (int) g.n_expert, (int) K, ids_ + t * K, w_ + t * K, cs); } - moe_group_resident(ids_, (int) (T * K), (int) K, experts_, (int64_t) strata::kernels::cpu::BLOB, grp_ptr_, - grp_start_, grp_counts_, hit_dst_, hit_slot_, cs); - quantize_q8_0_scaled(mixed_, hit_xq_, hit_xs_, (int64_t) T * N, cs); - moe_grouped_s2(grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, (int64_t) T * K, (int64_t) T * K, hit_xq_, - hit_xs_, hit_scratch_, parts_, cs); + moe_group_resident(ids_, (int) (T * K), (int) K, experts_, (int64_t) blob_, grp_ptr_, grp_start_, grp_counts_, + hit_dst_, hit_slot_, cs); + if (nat_.gu_type >= 0) { + quantize_q8_1_rows(mixed_, T, N, nat_xq_, cs); + native_expert_grouped(nat_, grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, (int64_t) T * K, + (int64_t) T * K, nat_xq_, hit_scratch_, parts_, cs); + } else { + quantize_q8_0_scaled(mixed_, hit_xq_, hit_xs_, (int64_t) T * N, cs); + moe_grouped_s2(grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, (int64_t) T * K, (int64_t) T * K, + hit_xq_, hit_xs_, hit_scratch_, parts_, cs); + } NativeSharedWeights nsw; nsw.gate_type = GGML_Q8_0; nsw.gate_data = q8("mlp.shared_expert.gate_proj.weight"); nsw.up_type = GGML_Q8_0; nsw.up_data = q8("mlp.shared_expert.up_proj.weight"); diff --git a/tools/mtp_pack.py b/tools/mtp_pack.py index 022d2ea8ff5..b21c27427d8 100644 --- a/tools/mtp_pack.py +++ b/tools/mtp_pack.py @@ -4,16 +4,18 @@ no longer on this PC, so Strata packs its own from the 31 `mtp.*` tensors fetched by tools/mtp_fetch.py: python tools/mtp_pack.py --src Desktop/Strata/mtp-bf16 --experts q2_0 --out mtp-q2_0.gguf - python tools/mtp_pack.py --src ... --experts q4_0 --out mtp-q4_0.gguf (acceptance comparison arm) + python tools/mtp_pack.py --src ... --experts iq4_nl --out mtp-iq4_nl.gguf Layout: dense tensors (attention, indexer, hyper-connections, shared expert, router, fc/norms) stay BF16 (F32 for 1-D norms), ~0.18 GB. The routed experts keep the checkpoint's fused layout - `gate_up_proj` [512, 1280, 2560] and `down_proj` [512, 2560, 640], quantized along the last (input) axis - in one of: - q2_0 64-element blocks, grid {-1, 0, 1, 2} x d. The scale is chosen per block to MINIMIZE squared error over - that grid (the ggml reference sets d = max|w| and never uses the +2 level). Same format as the main - model's experts, so Strata's CPU VNNI kernel and GPU hit kernel serve it unchanged. ~0.71 GB. - q4_0 ggml reference rounding. ~1.42 GB. q8_0 ggml reference. ~2.67 GB. + q2_0 64-element blocks, grid {-1, 0, 1, 2} x d. The scale is chosen per block to MINIMIZE squared error over + that grid (the ggml reference sets d = max|w| and never uses the +2 level). Same format as the main + model's experts, so Strata's CPU VNNI kernel and GPU hit kernel serve it unchanged. ~0.71 GB. + iq4_nl ggml's quantizer without an importance matrix (relative RMS error ~0.078). ~1.42 GB. + q8_0 ggml reference (~0.006). ~2.67 GB. + iq4_nl and q8_0 run through the main model's native expert kernels (tools/mtp_rt.py writes them as GGUF blocks). This is round-to-nearest, not GSQ: the plan picks the expert format by MEASURED draft acceptance (P0.3/P6), not by this file's reconstruction error, which is reported per tensor only as a sanity check. No model runs here. @@ -23,8 +25,10 @@ import argparse import hashlib import json +import os import sys import time +from concurrent.futures import ThreadPoolExecutor from pathlib import Path import numpy as np @@ -82,26 +86,56 @@ def q2_0(w: np.ndarray) -> np.ndarray: return out.reshape(-1) -def q4_0(w: np.ndarray) -> np.ndarray: - """ggml quantize_row_q4_0_ref: d = max / -8 (signed max), code = clamp(round(x/d + 8.5) truncated, 0, 15).""" +IQ4_NL_GRID = np.array([-127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113], dtype=np.float32) +IQ4_NL_MID = (IQ4_NL_GRID[:-1] + IQ4_NL_GRID[1:]) / 2 + + +def iq4_nl(w: np.ndarray) -> np.ndarray: + """ggml quantize_row_iq4_nl without an importance matrix (quantize_row_iq4_nl_impl, block 32, ntry 7): the scale + that maps the block's largest |w| onto the grid's end, then the best of 15 trial scales by weighted least + squares (weights w^2); codes are the nearest grid values (ties to the upper). fp16 d, then 16 bytes of 4-bit + codes: element j in the low nibble of byte j, element j + 16 in its high nibble.""" x = w.reshape(-1, 32).astype(np.float32) - idx = np.abs(x).argmax(axis=1) - mx = x[np.arange(x.shape[0]), idx][:, None] - d = mx / -8.0 - inv = np.where(d != 0, 1.0 / np.where(d != 0, d, 1.0), 0.0) - q = np.minimum(15, np.trunc(x * inv + 8.5)).astype(np.uint8) - packed = (q[:, :16] | (q[:, 16:] << 4)).astype(np.uint8) + wt = x * x + rows = np.arange(x.shape[0]) + mx = x[rows, np.abs(x).argmax(axis=1)] # the signed value of the largest |w| (first one) + live = np.abs(mx) >= 1e-15 + mx = np.where(live, mx, np.float32(1)) + + def fit(inv: np.ndarray): + q = IQ4_NL_GRID[np.searchsorted(IQ4_NL_MID, inv[:, None] * x, side="right")] + wq = wt * q + a, b = wq * x, wq * q + sqx, sq2 = a[:, 0].copy(), b[:, 0].copy() + for j in range(1, 32): # ggml's order of the sums + sqx += a[:, j] + sq2 += b[:, j] + return sqx, sq2 + + sqx, sq2 = fit(1 / (-mx / IQ4_NL_GRID[0])) + d = sqx / np.where(sq2 > 0, sq2, 1) + best = d * sqx + for t in range(-7, 8): + sqx, sq2 = fit((t + IQ4_NL_GRID[0]) / mx) + better = (sq2 > 0) & (sqx * sqx > best * sq2) + d = np.where(better, sqx / np.where(sq2 > 0, sq2, 1), d) + best = np.where(better, d * sqx, best) + d = np.where(live, d, 0).astype(np.float32) + inv = np.where(d != 0, 1 / np.where(d != 0, d, 1), 0).astype(np.float32) + codes = np.searchsorted(IQ4_NL_MID, inv[:, None] * x, side="right").astype(np.uint8) out = np.empty((x.shape[0], 18), dtype=np.uint8) out[:, :2] = d.astype(np.float16).view(np.uint8).reshape(-1, 2) - out[:, 2:] = packed + out[:, 2:] = codes[:, :16] | (codes[:, 16:] << 4) return out.reshape(-1) def q8_0(w: np.ndarray) -> np.ndarray: + """ggml quantize_row_q8_0_ref: d = max|w| / 127, codes rounded half away from zero.""" x = w.reshape(-1, 32).astype(np.float32) d = np.abs(x).max(axis=1, keepdims=True) / 127.0 inv = np.where(d > 0, 1.0 / np.where(d > 0, d, 1.0), 0.0) - q = np.rint(x * inv).astype(np.int8) + v = x * inv + q = (np.sign(v) * np.floor(np.abs(v) + 0.5)).astype(np.int8) out = np.empty((x.shape[0], 34), dtype=np.uint8) out[:, :2] = d.astype(np.float16).view(np.uint8).reshape(-1, 2) out[:, 2:] = q.view(np.uint8) @@ -115,19 +149,18 @@ def dequant(kind: str, blob: np.ndarray, n: int) -> np.ndarray: qs = b[:, 2:] codes = np.stack([(qs >> s) & 3 for s in (0, 2, 4, 6)], axis=2).reshape(-1, 64).astype(np.float32) return ((codes - 1.0) * d).reshape(-1)[:n] - if kind == "q4_0": + if kind == "iq4_nl": b = blob.reshape(-1, 18) d = b[:, :2].copy().view(np.float16).astype(np.float32) qs = b[:, 2:] - q = np.concatenate([qs & 15, qs >> 4], axis=1).astype(np.float32) - return ((q - 8.0) * d).reshape(-1)[:n] + return (IQ4_NL_GRID[np.concatenate([qs & 15, qs >> 4], axis=1)] * d).reshape(-1)[:n] b = blob.reshape(-1, 34) d = b[:, :2].copy().view(np.float16).astype(np.float32) return (b[:, 2:].view(np.int8).astype(np.float32) * d).reshape(-1)[:n] QUANT = {"q2_0": (q2_0, gguf.GGMLQuantizationType.Q2_0, 64, 18), - "q4_0": (q4_0, gguf.GGMLQuantizationType.Q4_0, 32, 18), + "iq4_nl": (iq4_nl, gguf.GGMLQuantizationType.IQ4_NL, 32, 18), "q8_0": (q8_0, gguf.GGMLQuantizationType.Q8_0, 32, 34)} @@ -146,7 +179,8 @@ def main() -> int: w.add_string("strata.mtp.source_sha256", hashlib.sha256( json.dumps({t["name"]: t["sha256"] for t in manifest}, sort_keys=True).encode()).hexdigest()) w.add_string("strata.mtp.expert_format", a.experts) - w.add_string("strata.mtp.expert_quantizer", "per-block MSE scale search" if a.experts == "q2_0" else "ggml reference") + w.add_string("strata.mtp.expert_quantizer", {"q2_0": "per-block MSE scale search", "iq4_nl": "ggml quantize_iq4_nl", + "q8_0": "ggml reference"}[a.experts]) report = [] for t in sorted(manifest, key=lambda t: t["name"]): name, shape = t["name"], t["shape"] @@ -158,7 +192,8 @@ def main() -> int: x = _Experts(raw) if shape[-1] % block: raise ValueError(f"{name}: inner dim {shape[-1]} not a multiple of {block}") - parts = [fn(x[e]) for e in range(shape[0])] # one expert at a time bounds memory + with ThreadPoolExecutor(os.cpu_count() or 4) as ex: # numpy releases the GIL in the quantizers + parts = list(ex.map(lambda e: fn(x[e]), range(shape[0]))) blob = np.concatenate(parts) errs = [] for e in range(min(a.check_experts, shape[0])): diff --git a/tools/mtp_rt.py b/tools/mtp_rt.py index e3e8f6d78a1..723ff8bb95d 100644 --- a/tools/mtp_rt.py +++ b/tools/mtp_rt.py @@ -3,9 +3,12 @@ python tools/mtp_rt.py --gguf /mtp-bf16/mtp-q2_0.gguf --out /mtp-bf16/rt Writes - experts.bin 512 routed experts in the engine's blob layout (`include/strata/kernels/cpu/expert.hpp`): gate/up - rows interleaved (2r = gate r, 2r+1 = up r), then down rows; the Q2_0 codes in one plane and the fp16 - scales in another. A lossless relayout of the GGUF's Q2_0 blocks (same bytes, `cpu_expert_fixture.py`). + experts.bin 512 routed experts. Q2_0 (tools/mtp_pack.py's default) in the engine's blob layout + (`include/strata/kernels/cpu/expert.hpp`): gate/up rows interleaved (2r = gate r, 2r+1 = up r), then + down rows; the Q2_0 codes in one plane and the fp16 scales in another, a lossless relayout of the + GGUF's blocks (same bytes, `cpu_expert_fixture.py`). Other formats (iq4_nl, q8_0) as the GGUF's + blocks, [gate rows | up rows | down rows] per expert, for the native expert kernels. + experts.txt only with the GGUF-block layout: the ggml type ids of gate/up and down. dense.bin every other tensor: the large projections quantized to Q8_0 (ggml's reference rounding) so the engine's multi-column MMVQ can run them; the hyper-connection and router weights kept BF16; the RMSNorm weights as F32 with the Gemma "+1" applied (vLLM GemmaRMSNorm scales by 1 + w). @@ -68,12 +71,23 @@ def main() -> int: out.mkdir(parents=True, exist_ok=True) r = gguf.GGUFReader(a.gguf) tens = {t.name: t for t in r.tensors} - gu = np.asarray(tens["mtp.layers.0.mlp.experts.gate_up_proj"].data) - dn = np.asarray(tens["mtp.layers.0.mlp.experts.down_proj"].data) - assert gu.shape == (NE, 2 * FF, 720) and dn.shape == (NE, H, 180), (gu.shape, dn.shape) + gut, dnt = tens["mtp.layers.0.mlp.experts.gate_up_proj"], tens["mtp.layers.0.mlp.experts.down_proj"] + gu, dn = np.asarray(gut.data), np.asarray(dnt.data) + q2 = gguf.GGMLQuantizationType.Q2_0 + fmt = out / "experts.txt" with open(out / "experts.bin", "wb") as f: - for e in range(NE): - f.write(blob_of(gu[e], dn[e])) + if gut.tensor_type == q2 and dnt.tensor_type == q2: + assert gu.shape == (NE, 2 * FF, 720) and dn.shape == (NE, H, 180), (gu.shape, dn.shape) + for e in range(NE): + f.write(blob_of(gu[e], dn[e])) + fmt.unlink(missing_ok=True) + else: + assert gu.shape[:2] == (NE, 2 * FF) and dn.shape[:2] == (NE, H), (gu.shape, dn.shape) + for e in range(NE): + f.write(gu[e].tobytes()) + f.write(dn[e].tobytes()) + fmt.write_text(f"{int(gut.tensor_type)} {int(dnt.tensor_type)}\n", encoding="utf-8") + blob = (out / "experts.bin").stat().st_size // NE lines = [] off = 0 with open(out / "dense.bin", "wb") as f: @@ -102,7 +116,8 @@ def main() -> int: lines.append(f"{short} {kind} {rows} {cols} {off} {len(raw)}") off += len(raw) (out / "dense.txt").write_text("\n".join(lines) + "\n", encoding="utf-8") - print(f"experts.bin {NE * BLOB} B, dense.bin {off} B, {len(lines)} tensors -> {out}") + print(f"experts.bin {NE * blob} B ({gut.tensor_type.name}/{dnt.tensor_type.name}), dense.bin {off} B, " + f"{len(lines)} tensors -> {out}") for l in lines: print(" " + l) return 0 From 7a5038f1cc732a4db0571b2666edd7a2cda3316e Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 12:55:44 +0200 Subject: [PATCH 54/92] CPU expert pool: idle workers sleep; the server stops on Ctrl+C The pool's seven workers spun on the phase counter without end: an idle engine, or one reading a prompt on the GPUs, kept 7 of 16 logical CPUs at 100%. A parked worker now spins for 50 ms (kIdleSpin), then sleeps on the pool's state word (std::atomic::wait) until the host opens a phase; the host wakes sleepers only when there are any. Generation does not put them to sleep (--stats: worker sleeps); the first batch after a sleep takes ~0.04 ms longer. The word holds the epoch, a closed bit and the parked workers: a worker joins with a compare-exchange on it, and the host closes each phase once its joined workers have parked again, so a worker that wakes after the host finished a phase cannot join it while the next one is written. pool_test --stress (built by build.cmd pool_test): whole Q2_0 experts and Q4_K/Q5_1, IQ3_XXS/IQ4_NL row splits on synthetic experts, idle gaps that put the workers to sleep, every output bitwise against serial runs. Window hashes unchanged on both models. serve/server.py waited in threading.Event().wait(), which Ctrl+C cannot interrupt on Windows: it now sleeps in a loop, and exits when the engine does. Claude-Session: https://claude.ai/code/session_01YDx8gSYSbts2783qXv2Nxz --- CMakeLists.txt | 7 +- include/strata/kernels/cpu/pool.hpp | 72 ++++----- serve/server.py | 24 ++- src/kernels/cpu/pool.cpp | 98 ++++++++----- src/kernels/cpu/pool_test.cpp | 217 +++++++++++++++++++++++++++- src/program/generate.cpp | 19 ++- 6 files changed, 344 insertions(+), 93 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index afcb9b258a5..450e6bb1ca1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -465,6 +465,10 @@ else() endif() find_package(Threads REQUIRED) target_link_libraries(strata_kernels_cpu PUBLIC Threads::Threads) +# The expert pool against serial runs; `--stress` needs no pack (synthetic experts, idle gaps that put the workers +# to sleep). +add_executable(pool_test src/kernels/cpu/pool_test.cpp) +target_link_libraries(pool_test PRIVATE strata_kernels_cpu) # ---- plan v0.3 P6: the i-quant models' CPU experts (IQ2_XS / IQ3_XXS files) ---- # @@ -551,9 +555,8 @@ if(STRATA_BUILD_TESTS) add_test(NAME expert_parity COMMAND expert_parity --selftest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) - add_executable(pool_test src/kernels/cpu/pool_test.cpp) - target_link_libraries(pool_test PRIVATE strata_kernels_cpu) add_test(NAME pool_test COMMAND pool_test --selftest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) + add_test(NAME pool_stress COMMAND pool_test --stress) endif() if(STRATA_BUILD_TESTS) diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp index d10ebe486a8..4e155009fbf 100644 --- a/include/strata/kernels/cpu/pool.hpp +++ b/include/strata/kernels/cpu/pool.hpp @@ -11,18 +11,26 @@ // one batch. A ring would add a wrap-around to get wrong and buy nothing. What is kept from the phase is // the part that matters: `head`/`done` are single fetch_add counters, one claim per worker, no lock. // -// THE COMPLETION PROTOCOL, because this is where a pool usually goes wrong. `run()` waits for `done == n` -// AND for every worker to PARK. Waiting only for `done` is not enough: a worker can still be inside the -// drain loop after its last `done` increment, and the host resetting `head` underneath it would let that -// worker claim a job from the NEXT batch before the next batch has been published. The -// `done`-then-`parked` pair makes the handover unambiguous, and the second wait costs a few hundred cycles -// against a layer that takes milliseconds. +// THE PHASE PROTOCOL, because this is where a pool usually goes wrong. One atomic word holds the phase's +// epoch, whether it is open, and how many workers are parked. The host writes a phase's jobs while the +// previous phase is closed, then opens the next epoch; a worker joins with a compare-exchange on the whole +// word, so it can only join the phase it saw open. The host closes a phase once `done == n` AND every +// worker that joined has parked again. Waiting only for `done` is not enough: a worker can still be inside +// the drain loop after its last `done` increment, and the host resetting `head` underneath it would let it +// claim a job from the NEXT batch. Closing is what makes a late worker harmless: one that notices a phase +// only after the host finished it (it was asleep, or descheduled) finds it closed and waits for the next. +// +// IDLE WORKERS SLEEP. A parked worker spins for `kIdleSpin`, then blocks on the word (`std::atomic::wait`: +// WaitOnAddress, a futex) until the host opens a phase; the host wakes them only when one sleeps. Generation +// opens phases every few hundred microseconds, and a round's longest gap (head, drafts, the first layer) is +// a few ms, so they sleep only between requests and while a prompt runs on the GPUs. #pragma once #include "strata/kernels/cpu/expert.hpp" #include "strata/kernels/cpu/native_expert.hpp" #include +#include #include #include #include @@ -102,7 +110,7 @@ class ExpertPool { /// is on" is only true if the engine says which adaptation it took. bool host_works() const { return host_works_; } - /// Publish `n` jobs, then block until every one has been claimed AND every worker has parked. + /// Publish `n` jobs, then block until every one is done AND the phase is closed. /// `jobs` must outlive the call (it does, and the workers never touch it afterwards). void run(ExpertJob* jobs, int n); @@ -125,28 +133,27 @@ class ExpertPool { double ms_multi_gu = 0, ms_multi_q = 0, ms_multi_down = 0; int64_t multi_bytes = 0; - /// Total `_mm_pause` iterations spent waiting, over all workers, is no longer counted - see the note on the - /// atomics below. It was a LOCKED read-modify-write in the park loop, so measuring the contention added to - /// it. - long long pauses() const { return 0; } + /// How long a parked worker spins before it sleeps. + static constexpr std::chrono::milliseconds kIdleSpin{50}; + /// How many times a worker has gone to sleep (each worker counts once per idle stretch). + uint32_t sleeps() const { return sleeps_.load(std::memory_order_relaxed); } - /// **WHERE `run()` SPENDS ITS TIME, in milliseconds accumulated over its lifetime.** Three phases per - /// layer - wait for every worker to be parked, wait for the drain, wait for them to re-park - and until now - /// all three were reported as one number. Without the split there is no way to tell a pool that is slow at - /// the WORK from one that is slow at the SYNCHRONISATION, and those need opposite fixes: the first is a - /// kernel problem and the second is a barrier problem. + /// **WHERE `run()` SPENDS ITS TIME, in milliseconds accumulated over its lifetime**: the drain, and closing + /// the phase (the joined workers parking again). Without the split there is no way to tell a pool that is + /// slow at the WORK from one that is slow at the SYNCHRONISATION, and those need opposite fixes. /// /// Only the host thread touches these, in `run()`, so they need no atomics. - void phase_ms(double& wait_park, double& drain, double& repark) const { - wait_park = ms_wait_park_; + void phase_ms(double& drain, double& close) const { drain = ms_drain_; - repark = ms_repark_; + close = ms_close_; } private: void worker(int id); void drain(int id, ExpertScratch& scratch); void run_phase(int mode, int n_tasks, void (*first)(void*) = nullptr, void* ctx = nullptr); + void open_phase(); + void close_phase(); int n_ = 0; bool host_works_ = true; @@ -155,26 +162,21 @@ class ExpertPool { /// The host's own scratch when `host_works_`. A separate object rather than a share of `scratch_[i]`, /// because a worker may own any index and the two must not be able to collide. ExpertScratch host_scratch_; - // `run()`'s three phases, accumulated. Host-thread only; see `phase_ms`. - double ms_wait_park_ = 0.0; + // `run()`'s phases, accumulated. Host-thread only; see `phase_ms`. double ms_drain_ = 0.0; - double ms_repark_ = 0.0; - // ---- EACH ATOMIC GETS ITS OWN CACHE LINE, AND THE SPIN COUNTER IS GONE. (Review finding C3.) - // - // These were six adjacent atomics, which put `head_`, `done_`, `parked_` and `epoch_` on ONE cache line - - // the four that workers and the host actually contend on, invalidating each other on every access. - // - // Worse, the parked spin did `pauses_.fetch_add(1)` on EVERY iteration: a LOCKED read-modify-write, five - // workers against one line, at roughly one iteration per `_mm_pause`. So the line the host must WRITE to - // publish work (`epoch_`) and READ to confirm the workers are parked (`parked_`) was being hammered by the - // very threads waiting for it. That is contention the pool imposes on itself. + double ms_close_ = 0.0; + // ---- EACH ATOMIC GETS ITS OWN CACHE LINE, AND THE PARK LOOP WRITES NOTHING SHARED. (Review finding C3.) // - // The counter was diagnostic only - `pauses()` was read in one place, to print a number nothing branched on - // - so it is deleted rather than amortised. `alignas(64)` then stops the remaining four sharing. + // Adjacent atomics that workers and the host contend on invalidate each other on every access, and a + // diagnostic counter the park loop once incremented on every iteration hammered the very line the host + // writes to publish work. `alignas(64)` keeps them apart; the park loop only reads `state_` (a worker + // writes it once to join a phase and once to park again, and `sleepers_` only when it goes to sleep). alignas(64) std::atomic head_{0}; alignas(64) std::atomic done_{0}; - alignas(64) std::atomic parked_{0}; - alignas(64) std::atomic epoch_{0}; + /// epoch << 32 | kClosed (bit 31) | parked workers + alignas(64) std::atomic state_{0}; + alignas(64) std::atomic sleepers_{0}; + std::atomic sleeps_{0}; alignas(64) std::atomic stop_{false}; std::vector threads_; std::vector scratch_; // one per worker: no allocation, no false sharing of the hot data diff --git a/serve/server.py b/serve/server.py index 4f80379ae9c..1849caf8644 100644 --- a/serve/server.py +++ b/serve/server.py @@ -789,19 +789,27 @@ def main() -> int: print(f"ready: http://{a.host}:{a.port}/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, " f"context {engine.max_context} tokens{', images on' if vision else ''}" f"{', API key required' if svc.api_key else ''})", flush=True) - print(f" open http://{a.host}:{a.port}/ in a browser to chat; close this window to stop the model", flush=True) + print(f" open http://{a.host}:{a.port}/ in a browser to chat; Ctrl+C or close this window to stop the model", + flush=True) if a.open: import webbrowser webbrowser.open(f"http://{'127.0.0.1' if a.host in ('0.0.0.0', '') else a.host}:{a.port}/") + code = 0 try: - threading.Event().wait() + # a sleep, not threading.Event().wait(): on Windows Ctrl+C interrupts only the former + while not hasattr(engine, "proc") or engine.proc.poll() is None: + time.sleep(1) + print(f"the engine exited (code {engine.proc.returncode})" + (f"; see {cfg['log']}" if cfg.get("log") else ""), + flush=True) + code = 1 except KeyboardInterrupt: - httpd.shutdown() - if hasattr(engine, "close"): - engine.close() - if vision: - vision.close() - return 0 + pass + httpd.shutdown() + if hasattr(engine, "close"): + engine.close() + if vision: + vision.close() + return code if __name__ == "__main__": diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cpp index e4537f0aa26..0b1c3e6d586 100644 --- a/src/kernels/cpu/pool.cpp +++ b/src/kernels/cpu/pool.cpp @@ -61,6 +61,11 @@ std::vector physical_cores(bool skip_first) { namespace { +// `state_`: the epoch in the high half, then the closed bit, then the parked workers +constexpr uint64_t kClosed = 1ull << 31; +constexpr uint64_t kParked = kClosed - 1; +constexpr uint64_t kEpoch = 1ull << 32; + void pin_this_thread(int core) { if (core < 0) return; #if defined(_WIN32) @@ -111,6 +116,7 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(h const std::vector cores = physical_cores(true); n_ = n_workers > 0 ? n_workers : (int) cores.size(); if (n_ < 1) n_ = 1; + state_.store(kClosed | (uint64_t) n_, std::memory_order_relaxed); // epoch 0, closed, everyone parked scratch_.resize((size_t) n_); split_.resize((size_t) kMaxSplit); split_multi_.resize((size_t) kMaxSplitMulti); @@ -125,43 +131,66 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(h } ExpertPool::~ExpertPool() { - stop_.store(true, std::memory_order_release); - // Bump the epoch so a PARKED worker notices the stop flag rather than sleeping through it. - epoch_.fetch_add(1, std::memory_order_release); + stop_.store(true, std::memory_order_seq_cst); + // a new value, so a sleeping worker's wait returns and it sees the stop flag + state_.fetch_add(kEpoch, std::memory_order_seq_cst); + state_.notify_all(); for (auto& t : threads_) t.join(); } void ExpertPool::worker(int id) { - uint32_t seen = 0; - // ARRIVE at the park before the first wait, so `parked_ == n_` is true from construction. Counting only - // on the RETURN from a drain leaves `parked_` at 0 until each worker has finished one batch, and the first - // `run()` - which waits for `parked_ == n_` before publishing - then deadlocks. It deadlocks on the very - // first call, which is the good case; a version that deadlocked on the second would be far worse. - parked_.fetch_add(1, std::memory_order_acq_rel); + uint32_t seen = 0; // the last epoch this worker joined for (;;) { - // Park: wait for work. `_mm_pause` rather than a bare spin because it yields the pipeline to the - // sibling hyperthread; `epoch_` is bumped once per LAYER, not once per expert, so most of these - // iterations are spent here with nothing to do. - // - // **AND NOTHING ELSE HAPPENS IN HERE.** This loop used to do `pauses_.fetch_add(1)` on every iteration - // - a locked read-modify-write, five workers against one cache line - so the workers spent their wait - // invalidating each other's caches and the very line the host writes to publish work. The counter was - // diagnostic and nothing branched on it. See the note on the atomics in pool.hpp. - while (epoch_.load(std::memory_order_acquire) == seen) { + // Park: wait for an open phase this worker has not joined. `_mm_pause` rather than a bare spin because + // it yields the pipeline to the sibling hyperthread. The loop writes nothing shared (see the note on + // the atomics in pool.hpp) until it joins, or goes to sleep after `kIdleSpin` without work. + uint64_t s = state_.load(std::memory_order_acquire); + auto idle_from = std::chrono::steady_clock::now(); + for (uint32_t spins = 1;; ++spins) { if (stop_.load(std::memory_order_relaxed)) return; + if (!(s & kClosed) && (uint32_t) (s >> 32) != seen) { + // join; fails when the host has closed this phase or other workers joined since `s` was read + if (state_.compare_exchange_weak(s, s - 1, std::memory_order_acq_rel, std::memory_order_acquire)) break; + continue; + } _mm_pause(); + if (spins % 1024 == 0 && std::chrono::steady_clock::now() - idle_from > kIdleSpin) { + // seq_cst, against open_phase's store then load: the host sees this sleeper or this worker's + // wait sees the new epoch + sleepers_.fetch_add(1, std::memory_order_seq_cst); + sleeps_.fetch_add(1, std::memory_order_relaxed); + state_.wait(s, std::memory_order_seq_cst); + sleepers_.fetch_sub(1, std::memory_order_seq_cst); + idle_from = std::chrono::steady_clock::now(); + } + s = state_.load(std::memory_order_acquire); } - if (stop_.load(std::memory_order_acquire)) return; - seen = epoch_.load(std::memory_order_relaxed); - parked_.fetch_sub(1, std::memory_order_acq_rel); // leaving the park + seen = (uint32_t) (s >> 32); // Drain: one claim per iteration, so a slow worker takes fewer experts and a fast one takes more. // Every job is the same size (all experts are 1,382,400 bytes), so there is nothing to schedule. drain(id, scratch_[(size_t) id]); - parked_.fetch_add(1, std::memory_order_acq_rel); // back at the park + state_.fetch_add(1, std::memory_order_release); // parked again } } +void ExpertPool::open_phase() { + // The previous phase is closed, so no worker is inside a drain and none can join until this store. + const uint64_t s = state_.load(std::memory_order_relaxed); + state_.store(((s >> 32) + 1) << 32 | (uint64_t) n_, std::memory_order_seq_cst); + if (sleepers_.load(std::memory_order_seq_cst) != 0) state_.notify_all(); +} + +void ExpertPool::close_phase() { + // Every job is done; wait for the workers that joined to park again, and shut the phase in the same step, + // so a worker that notices it late cannot join it. + const uint64_t open = (state_.load(std::memory_order_relaxed) & ~kParked) | (uint64_t) n_; + for (uint64_t s = open; + !state_.compare_exchange_weak(s, open | kClosed, std::memory_order_acq_rel, std::memory_order_relaxed); + s = open) + _mm_pause(); +} + void ExpertPool::drain(int id, ExpertScratch& scratch) { (void) id; for (;;) { @@ -240,16 +269,15 @@ void ExpertPool::drain(int id, ExpertScratch& scratch) { } void ExpertPool::run_phase(int mode, int n_tasks, void (*first)(void*), void* ctx) { - while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); mode_ = mode; njobs_ = n_tasks; head_.store(0, std::memory_order_relaxed); done_.store(0, std::memory_order_relaxed); - epoch_.fetch_add(1, std::memory_order_release); + open_phase(); if (first != nullptr) first(ctx); if (host_works_) drain(-1, host_scratch_); while (done_.load(std::memory_order_acquire) != (uint32_t) n_tasks) _mm_pause(); - while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); + close_phase(); } void ExpertPool::run_split(ExpertJob* jobs, int n) { @@ -348,20 +376,17 @@ void ExpertPool::run(ExpertJob* jobs, int n) { for (int i = 0; i < n; ++i) s2_expert_vnni_q(jobs[i].blob, *jobs[i].act, jobs[i].out, scratch_[0]); return; } - // Wait for every worker to be parked BEFORE touching the batch, so the publish below is the only thing - // that can move a worker into the drain loop. + // The previous phase is closed, so no worker can be in the drain loop while the batch is written. // - // THE THREE PHASES ARE TIMED SEPARATELY. They were one number, which cannot distinguish a pool that is - // slow at the WORK from one that is slow at the SYNCHRONISATION - and those need opposite fixes. - const auto t_a = std::chrono::steady_clock::now(); - while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); + // THE PHASES ARE TIMED SEPARATELY. They were one number, which cannot distinguish a pool that is slow at + // the WORK from one that is slow at the SYNCHRONISATION - and those need opposite fixes. const auto t_b = std::chrono::steady_clock::now(); jobs_ = jobs; njobs_ = n; mode_ = 0; head_.store(0, std::memory_order_relaxed); done_.store(0, std::memory_order_relaxed); - epoch_.fetch_add(1, std::memory_order_release); // release: jobs_/njobs_ are visible before the bump + open_phase(); // seq_cst: jobs_/njobs_ are visible to a worker that joins // ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.** // @@ -388,15 +413,14 @@ void ExpertPool::run(ExpertJob* jobs, int n) { } while (done_.load(std::memory_order_acquire) != (uint32_t) n) _mm_pause(); - // And park again, so the next `run` starts from a known state. See the header for why `done` alone is - // not enough. + // And close the phase, so the next `run` starts from a known state. See the header for why `done` alone + // is not enough. const auto t_c = std::chrono::steady_clock::now(); - while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause(); + close_phase(); const auto t_d = std::chrono::steady_clock::now(); - ms_wait_park_ += std::chrono::duration(t_b - t_a).count(); ms_drain_ += std::chrono::duration(t_c - t_b).count(); - ms_repark_ += std::chrono::duration(t_d - t_c).count(); + ms_close_ += std::chrono::duration(t_d - t_c).count(); } } // namespace strata::kernels::cpu diff --git a/src/kernels/cpu/pool_test.cpp b/src/kernels/cpu/pool_test.cpp index dbc7288a1d5..20088ef13e3 100644 --- a/src/kernels/cpu/pool_test.cpp +++ b/src/kernels/cpu/pool_test.cpp @@ -13,8 +13,16 @@ // mode the park protocol exists to prevent, so `run()` is called many times in a row. // 4. A BATCH BIGGER AND SMALLER THAN THE WORKER COUNT, because `n < workers` leaves most workers claiming // nothing and `n > workers` is the real case (10 experts, 5 workers). +// 5. (--stress, synthetic experts, no file) SLEEPING AND LATE WORKERS. Idle workers sleep after `kIdleSpin`; +// a phase opened while they sleep is often finished by the host alone before they wake, and a worker that +// wakes then must not join it: the host may already be writing the next phase. Thousands of batches, with +// idle gaps either side of `kIdleSpin` and one-expert batches after the long ones, every output bitwise +// against a serial run of the same kernels. #include "strata/kernels/cpu/pool.hpp" #include "strata/kernels/cpu/expert.hpp" +#if defined(STRATA_NATIVE_EXPERTS) +#include "ggml.h" +#endif #include #include @@ -22,8 +30,10 @@ #include #include #include +#include #include #include +#include #include namespace cpu = strata::kernels::cpu; @@ -49,19 +59,224 @@ double now_ms() { return duration(steady_clock::now().time_since_epoch()).count(); } +// ---- 5: --stress +const float kSentinel = -1.2345e33f; + +std::vector gauss_vec(std::mt19937& rng, size_t n, float sd) { + std::normal_distribution g(0.0f, sd); + std::vector v(n); + for (auto& x : v) x = g(rng); + return v; +} + +double median(std::vector v) { + if (v.empty()) return 0.0; + std::sort(v.begin(), v.end()); + return v[v.size() / 2]; +} + +/// The pause before a batch: mostly none (generation's layers come back to back), sometimes up to 2 ms, and one +/// in 40 longer than `kIdleSpin`, so every worker is asleep when the batch is published. True for a long one. +bool gap(std::mt19937& rng) { + const unsigned r = rng() % 40; + if (r == 0) { + std::this_thread::sleep_for(cpu::ExpertPool::kIdleSpin + std::chrono::milliseconds(15)); + return true; + } + if (r < 5) std::this_thread::sleep_for(std::chrono::microseconds(100 + rng() % 1900)); + return false; +} + +struct StressStats { + long long batches = 0, outputs = 0, differ = 0; + int long_gaps = 0; + std::vector ms_spinning, ms_after_sleep; // one-expert batches +}; + +void stress_report(const char* what, const StressStats& s, uint32_t sleeps) { + std::printf(" %-34s %s (%lld of %lld outputs differ, %lld batches; %d long gaps, %u worker sleeps)\n", what, + s.differ ? "*** NO ***" : "identical to serial", s.differ, s.outputs, s.batches, s.long_gaps, sleeps); + std::printf(" %-34s %.3f ms while the workers spin, %.3f ms right after they slept (median)\n", + " a one-expert batch", median(s.ms_spinning), median(s.ms_after_sleep)); +} + +/// `run()`: whole Q2_0 experts. Random codes with finite scales: the protocol is under test, not the kernel. +StressStats stress_run(cpu::ExpertPool& pool, std::mt19937& rng, int iters) { + const int NB = 4, NA = 4, MAXJ = 16; + std::vector> blobs((size_t) NB, std::vector(cpu::BLOB)); + for (auto& b : blobs) { + for (auto& v : b) v = (uint8_t) rng(); + for (size_t o = cpu::O_GU_SCALES; o + 2 <= cpu::BLOB; o += 2) { + const uint16_t h = (uint16_t) (0x2000 | (rng() & 0x3FF)); // fp16 in [2^-7, 2^-6) + std::memcpy(b.data() + o, &h, 2); + } + } + std::vector acts((size_t) NA); + for (auto& a : acts) cpu::act_quant_q8_1(gauss_vec(rng, cpu::H, 1.0f).data(), cpu::H, a); + std::vector jobs((size_t) MAXJ); + std::vector out((size_t) MAXJ * cpu::H), ref(cpu::H); + cpu::ExpertScratch ws; + StressStats s; + for (int it = 0; it < iters; ++it) { + const bool slept = gap(rng); + s.long_gaps += slept; + const bool tiny = slept ? rng() % 2 == 0 : rng() % 20 == 0; + const int n = tiny ? 1 : 1 + (int) (rng() % MAXJ); + for (int j = 0; j < n; ++j) { + jobs[(size_t) j].blob = blobs[rng() % NB].data(); + jobs[(size_t) j].act = &acts[rng() % NA]; + jobs[(size_t) j].out = out.data() + (size_t) j * cpu::H; + std::fill(jobs[(size_t) j].out, jobs[(size_t) j].out + cpu::H, kSentinel); + } + const double t0 = now_ms(); + pool.run(jobs.data(), n); + const double ms = now_ms() - t0; + if (tiny) (slept ? s.ms_after_sleep : s.ms_spinning).push_back(ms); + for (int j = 0; j < n; ++j) { + cpu::s2_expert_vnni_q(jobs[(size_t) j].blob, *jobs[(size_t) j].act, ref.data(), ws); + s.differ += std::memcmp(ref.data(), jobs[(size_t) j].out, sizeof(float) * cpu::H) != 0; + } + s.outputs += n; + ++s.batches; + } + return s; +} + +#if defined(STRATA_NATIVE_EXPERTS) +/// `run_split_multi_native`: one layer's experts in GGUF formats, split by rows, several tokens per expert. +StressStats stress_native(cpu::ExpertPool& pool, std::mt19937& rng, int iters, ggml_type gu, ggml_type dn, + std::string& err) { + StressStats s; + cpu::NativeFmt f; + if (!cpu::native_fmt((int) gu, (int) dn, cpu::H, cpu::FF, f, err)) return s; + const int NB = 4, NA = 8, MAXE = 12, MAXNT = 4; + // ggml's quantizers on random weights (valid blocks), one thread per expert: the i-quant ones are slow + std::vector> blobs((size_t) NB, std::vector(f.bytes)); + { + std::vector th; + for (int b = 0; b < NB; ++b) + th.emplace_back([&f, &blobs, gu, dn, b, seed = rng()] { + std::mt19937 r(seed); + uint8_t* p = blobs[(size_t) b].data(); + ggml_quantize_chunk(gu, gauss_vec(r, (size_t) cpu::FF * cpu::H, 0.02f).data(), p, 0, cpu::FF, cpu::H, nullptr); + ggml_quantize_chunk(gu, gauss_vec(r, (size_t) cpu::FF * cpu::H, 0.02f).data(), p + f.up_off, 0, cpu::FF, + cpu::H, nullptr); + ggml_quantize_chunk(dn, gauss_vec(r, (size_t) cpu::H * cpu::FF, 0.02f).data(), p + f.down_off, 0, cpu::H, + cpu::FF, nullptr); + }); + for (auto& t : th) t.join(); + } + std::vector> acts((size_t) NA, std::vector(cpu::kNativeActBytes)); + for (auto& a : acts) cpu::native_quant_act(f, gauss_vec(rng, cpu::H, 1.0f).data(), a.data()); + std::vector jobs((size_t) MAXE); + std::vector out((size_t) MAXE * MAXNT * cpu::H), ref((size_t) MAXNT * cpu::H); + std::vector ff((size_t) MAXNT * cpu::FF); + std::vector hq((size_t) MAXNT * cpu::kNativeHBytes); + for (int it = 0; it < iters; ++it) { + const bool slept = gap(rng); + s.long_gaps += slept; + const bool tiny = slept ? rng() % 2 == 0 : rng() % 20 == 0; + const int n = tiny ? 1 : 1 + (int) (rng() % MAXE); + for (int e = 0; e < n; ++e) { + cpu::ExpertJobMulti& j = jobs[(size_t) e]; + j = cpu::ExpertJobMulti{}; + j.blob = blobs[rng() % NB].data(); + j.nt = tiny ? 1 : 1 + (int) (rng() % MAXNT); + int tok[NA]; + std::iota(tok, tok + NA, 0); + std::shuffle(tok, tok + NA, rng); + for (int t = 0; t < j.nt; ++t) { + j.nact[t] = acts[(size_t) tok[t]].data(); + j.out[t] = out.data() + ((size_t) e * MAXNT + (size_t) t) * cpu::H; + std::fill(j.out[t], j.out[t] + cpu::H, kSentinel); + } + } + const double t0 = now_ms(); + pool.run_split_multi_native(f, jobs.data(), n); + const double ms = now_ms() - t0; + if (tiny) (slept ? s.ms_after_sleep : s.ms_spinning).push_back(ms); + // the same kernels and token groups on this thread, every row at once + for (int e = 0; e < n; ++e) { + const cpu::ExpertJobMulti& j = jobs[(size_t) e]; + float* ffp[cpu::MAXT]; + float* rp[cpu::MAXT]; + const void* hqp[cpu::MAXT]; + for (int t = 0; t < j.nt; ++t) { + ffp[t] = ff.data() + (size_t) t * cpu::FF; + rp[t] = ref.data() + (size_t) t * cpu::H; + hqp[t] = hq.data() + (size_t) t * cpu::kNativeHBytes; + } + cpu::native_gu_rows(f, j.blob, j.nact, j.nt, ffp, 0, cpu::FF); + for (int t = 0; t < j.nt; ++t) cpu::native_quant_h(f, ffp[t], hq.data() + (size_t) t * cpu::kNativeHBytes); + cpu::native_down_rows(f, j.blob, hqp, j.nt, rp, 0, cpu::H); + for (int t = 0; t < j.nt; ++t) s.differ += std::memcmp(rp[t], j.out[t], sizeof(float) * cpu::H) != 0; + s.outputs += j.nt; + } + ++s.batches; + } + return s; +} +#endif + +int stress(int iters) { + std::mt19937 rng(20260927); + cpu::ExpertPool pool; + std::printf(" %-34s %d + the host thread; they sleep after %lld ms without work\n", "workers", pool.workers(), + (long long) cpu::ExpertPool::kIdleSpin.count()); + int bad = 0; + if (cpu::cpu_features().usable()) { + const uint32_t z0 = pool.sleeps(); + const StressStats s = stress_run(pool, rng, iters); + stress_report("run(), whole Q2_0 experts", s, pool.sleeps() - z0); + bad += s.differ != 0 || s.long_gaps == 0 || pool.sleeps() == z0; + } else { + std::printf(" run(): SKIPPED, the CPU lacks %s\n", cpu::cpu_features().reason()); + } +#if defined(STRATA_NATIVE_EXPERTS) + const struct { ggml_type gu, dn; const char* name; } fmts[] = { + {GGML_TYPE_Q4_K, GGML_TYPE_Q5_1, "native rows, Q4_K / Q5_1"}, // ggml-cpu's dot products + {GGML_TYPE_IQ3_XXS, GGML_TYPE_IQ4_NL, "native rows, IQ3_XXS / IQ4_NL"}, // + the AVX-512 i-quant kernel + }; + for (const auto& ft : fmts) { + std::string err; + const uint32_t z0 = pool.sleeps(); + const StressStats s = stress_native(pool, rng, iters, ft.gu, ft.dn, err); + if (!err.empty()) { + std::printf(" %s: %s\n", ft.name, err.c_str()); + ++bad; + continue; + } + stress_report(ft.name, s, pool.sleeps() - z0); + bad += s.differ != 0 || s.long_gaps == 0 || pool.sleeps() == z0; + } +#else + std::printf(" native rows: SKIPPED, built without STRATA_NATIVE_EXPERTS\n"); +#endif + std::printf("\npool stress: %d failures\n", bad); + return bad ? 1 : 0; +} + } // namespace int main(int argc, char** argv) { bool selftest = false; + int stress_iters = 0; const char* path = "pack/full/experts.bin"; long long layer = 0; for (int i = 1; i < argc; ++i) { const std::string a = argv[i]; if (a == "--selftest") selftest = true; + else if (a == "--stress") stress_iters = 1500; + else if (a == "--iters" && i + 1 < argc) stress_iters = std::atoi(argv[++i]); else if (a == "--file" && i + 1 < argc) path = argv[++i]; else if (a == "--layer" && i + 1 < argc) layer = std::atoll(argv[++i]); - else { std::fprintf(stderr, "usage: pool_test [--selftest] [--file P] [--layer N]\n"); return 2; } + else { + std::fprintf(stderr, "usage: pool_test [--selftest] [--file P] [--layer N]\n" + " pool_test --stress [--iters N] (synthetic experts, no file)\n"); + return 2; + } } + if (stress_iters > 0) return stress(stress_iters); const cpu::CpuFeatures feat = cpu::cpu_features(); if (!feat.usable()) { diff --git a/src/program/generate.cpp b/src/program/generate.cpp index d7ab6419eb6..c4c8ebbd682 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -3019,6 +3019,7 @@ int main(int argc, char** argv) { int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; const int64_t misses0 = drive.d.multi_misses, entries0 = drive.d.multi_entries; + const uint32_t sleeps0 = pool.sleeps(); if (o.stats) drive.d.routed.assign((size_t) (g.n_layers * g.n_expert), 0); if (use_mtp) mtp.on_draft = [&](int j, int32_t tok, float prob) { // the drafts the next window will verify @@ -3155,10 +3156,10 @@ int main(int argc, char** argv) { (double) (drive.d.multi_entries - entries0) / (double) (rounds * g.n_layers)); if (rounds > 0) std::printf("%-24s gate/up %.3f quantize %.3f down %.3f ms/round; %.1f GB/s over the rows phases; " - "CPU pool call %.3f ms/round\n", "pool multi", pool.ms_multi_gu / rounds, + "CPU pool call %.3f ms/round; %u worker sleeps\n", "pool multi", pool.ms_multi_gu / rounds, pool.ms_multi_q / rounds, pool.ms_multi_down / rounds, (double) pool.multi_bytes / 1e6 / std::max(1e-9, pool.ms_multi_gu + pool.ms_multi_down), - (drive.cpu_ms - pool_ms0) / rounds); + (drive.cpu_ms - pool_ms0) / rounds, pool.sleeps() - sleeps0); if (rounds > 0) std::printf("%-24s plan %.3f activation quantize %.3f jobs %.3f run %.3f ms/round\n", "dispatch", drive.d.ms_plan / rounds, drive.d.ms_actq / rounds, drive.d.ms_jobs / rounds, @@ -3311,16 +3312,14 @@ int main(int argc, char** argv) { std::printf("%-24s %.3f ms/token over %lld layers (%.0f positions, %lld dispatches)\n", " the CPU expert pool", pool_positions > 0.0 ? drive.cpu_ms / pool_positions : 0.0, (long long) g.n_layers, pool_positions, (long long) drive.calls); - // **AND WHERE INSIDE `run()` IT WENT.** Three phases per layer and they were one number, which cannot - // tell a pool that is slow at the WORK from one that is slow at the SYNCHRONISATION - opposite fixes. - // Wait-for-park is expected to be ~0 (the workers re-parked at the end of the previous layer); the - // question is whether the time is in the drain or in the re-park barrier. + // **AND WHERE INSIDE `run()` IT WENT.** They were one number, which cannot tell a pool that is slow at + // the WORK from one that is slow at the SYNCHRONISATION - opposite fixes: the drain, or the barrier that + // closes the phase. if (pool_positions > 0.0) { - double wp = 0, dr = 0, rp = 0; - pool.phase_ms(wp, dr, rp); + double dr = 0, cl = 0; + pool.phase_ms(dr, cl); const double per = pool_positions; - std::printf("%-24s wait-park %.3f drain %.3f re-park %.3f ms/token\n", - " pool phases", wp / per, dr / per, rp / per); + std::printf("%-24s drain %.3f close %.3f ms/token\n", " pool phases", dr / per, cl / per); } std::printf("%-24s %lld blobs read\n", " expert blobs", (long long) srcp->reads()); // ---- **R4's DISPATCH MEASUREMENT: h, ON THE ENGINE'S OWN ROUTING.** No offline trace, no corpus From 1d492512a5e008cffb91c6665918550e9b2edcf1 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 14:23:38 +0200 Subject: [PATCH 55/92] generate: --window-logits, the verify windows' top 64 tokens per emitted position Each emitted token's line holds its index among the generated tokens, the token, and the 64 likeliest tokens with their log-probabilities from the head's logits: Strata's next-token distributions for a position-by-position comparison with llama.cpp (bench/parity.py in the Strata folder). The logits are copied after the window's graph; the window hashes are unchanged. Claude-Session: https://claude.ai/code/session_01CSrATbDcrJX2zdNDvTYaSr --- include/strata/core/verify.hpp | 2 ++ src/core/verify.cpp | 10 ++++++++++ src/program/generate.cpp | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 49268a172d7..c2e5157d108 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -93,6 +93,8 @@ class Verifier { /// graph writes them there itself, so reading them waits for no copy engine. void set_host_rows(bool on) { host_rows_ = on; } const float* final_R_host() const { return h_rows_; } + /// The head's logits of the last window's first `T` tokens, T * n_vocab floats, copied to host memory `out`. + bool copy_logits(int T, float* out, std::string& err) const; /// Where the pool publishes each layer's PCIe share of the misses; give it to the dispatch /// (`ExpertDispatch::plan`) before the first `run`. diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 27c0da1a8a6..3be2293af87 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -1118,6 +1118,16 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, return true; } +bool Verifier::copy_logits(int T, float* out, std::string& err) const { + if (cudaMemcpyAsync(out, head_logits_, (size_t) T * (size_t) n_vocab_ * sizeof(float), cudaMemcpyDeviceToHost, + cs_) != cudaSuccess || + cudaStreamSynchronize(cs_) != cudaSuccess) { + err = std::string("verify: reading the logits back: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + return true; +} + void Verifier::print_profile() const { if (prof_windows_ == 0) return; static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", diff --git a/src/program/generate.cpp b/src/program/generate.cpp index c4c8ebbd682..b9f43b80dbd 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -254,6 +254,7 @@ struct Options { bool spec_split = false; ///< opt-in split verify window (the overlap study: exact, ~7% slower) bool window_profile = false; ///< GPU timestamps between the verify window's stages, printed by --stats std::string window_hashes; ///< per verify window: a hash of its final residual rows and its argmaxes + std::string window_logits; ///< per emitted position: the 64 likeliest tokens and their log-probabilities bool prefill_profile = false; ///< GPU time per section of the batched prompt path /// Plan v0.3 P8: stay resident and take requests on stdin (see the --serve block in main). bool serve = false; @@ -380,6 +381,9 @@ void usage() { " of its final residual rows and its argmaxes: two builds that do the same\n" " arithmetic write the same file (fixed text through --spec-follow,\n" " --adapt-every 0; not --serve)\n" + " --window-logits PATH per token the verify windows emit, a line with its index among the generated\n" + " tokens, the token and the 64 likeliest as id:log-probability, from the\n" + " head's logits (comparisons with llama.cpp, bench/parity.py; not --serve)\n" " --window-profile with --stats: the GPU time of each stage of the verify window (timestamps\n" " inside the graph; each costs ~2 us)\n" " --prefill-profile the GPU time of each section of the batched prompt path (not --serve)\n" @@ -749,6 +753,7 @@ int main(int argc, char** argv) { else if (a == "--window-profile") o.window_profile = true; else if (a == "--prefill-profile") o.prefill_profile = true; else if (a == "--window-hashes") o.window_hashes = next("--window-hashes"); + else if (a == "--window-logits") o.window_logits = next("--window-logits"); else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode"); else if (a == "--serve") o.serve = true; else if (a == "--vision") o.vision = true; @@ -3015,6 +3020,27 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_hashes.c_str()); return 1; } + std::FILE* wlog = o.window_logits.empty() ? nullptr : std::fopen(o.window_logits.c_str(), "w"); + if (!o.window_logits.empty() && wlog == nullptr) { + std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_logits.c_str()); + return 1; + } + std::vector wl(wlog != nullptr ? (size_t) o.spec * (size_t) n_vocab : 0); + std::vector wl_ids(wlog != nullptr ? (size_t) n_vocab : 0); + auto write_logits = [&](int64_t k, int32_t emitted, const float* l) { // generated token k's line + constexpr int K = 64; + double m = l[0], s = 0; + for (int64_t i = 1; i < n_vocab; ++i) m = std::max(m, (double) l[i]); + for (int64_t i = 0; i < n_vocab; ++i) s += std::exp((double) l[i] - m); + const double lse = m + std::log(s); + for (int64_t i = 0; i < n_vocab; ++i) wl_ids[(size_t) i] = (int32_t) i; + std::partial_sort(wl_ids.begin(), wl_ids.begin() + K, wl_ids.end(), + [&](int32_t a, int32_t b) { return l[a] > l[b] || (l[a] == l[b] && a < b); }); + std::fprintf(wlog, "%lld %d", (long long) k, (int) emitted); + for (int j = 0; j < K; ++j) + std::fprintf(wlog, " %d:%.6f", (int) wl_ids[(size_t) j], (double) l[wl_ids[(size_t) j]] - lse); + std::fprintf(wlog, "\n"); + }; std::vector accepted_hist((size_t) o.spec, 0); int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; const double pool_ms0 = drive.cpu_ms; @@ -3059,6 +3085,10 @@ int main(int argc, char** argv) { drive.d.fail ? drive.d.fail : "(no message)"); return 1; } + if (wlog != nullptr && !ver.copy_logits(T, wl.data(), err)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } if (hashes != nullptr) { // FNV-1a over the window's final residual rows const size_t n_floats = (size_t) T * (size_t) (g.hc * g.n_embd); uint64_t h = 1469598103934665603ull; @@ -3099,6 +3129,8 @@ int main(int argc, char** argv) { ++accepted_hist[(size_t) a]; bool eos = false; for (int i = 0; i <= a && (int64_t) produced.size() < max_new && !eos; ++i) { + if (wlog != nullptr) + write_logits((int64_t) produced.size(), outv[(size_t) i], wl.data() + (size_t) i * (size_t) n_vocab); produced.push_back(outv[(size_t) i]); eos = o.stop_eos && std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) outv[(size_t) i]) != o.eos_ids.end(); if (!follow.empty()) { follow_differ += fdiff[(size_t) i]; ++follow_emitted; } @@ -3132,6 +3164,7 @@ int main(int argc, char** argv) { (long long) produced.size(), (long long) rounds); } if (hashes != nullptr) std::fclose(hashes); + if (wlog != nullptr) std::fclose(wlog); std::printf("%-24s %lld rounds of %d, drafts accepted %lld of %lld (%.3f), %.2f tokens per round\n", "speculation", (long long) rounds, o.spec, (long long) drafts_ok, (long long) drafts_total, drafts_total > 0 ? (double) drafts_ok / (double) drafts_total : 0.0, From 829027926c296c3085cdd8f24b07d935c282a23f Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 17:36:05 +0200 Subject: [PATCH 56/92] xor_scatter: the steps by template recursion, the halves chosen by a value select The inner loop's trip count came from the outer loop's variable, so in larger kernels it stayed rolled and indexed the values at run time, and `hi ? v[i] : v[j]` became a load from a selected address: the values went to local memory (gr_up_multi_kernel<5>, <7>, and a 32-value scatter in a prototype). The steps are now template recursion with constant trip counts, and the select is selp on the values. The same additions in the same order: window hashes identical on both models (thinking and code text), verify_parity passes. The hyper-connection read's up projection of 4 tokens: 16.2 -> 14.1 us (scratch/gr_bench.cu, in graphs). Claude-Session: https://claude.ai/code/session_01CSrATbDcrJX2zdNDvTYaSr --- src/kernels/cuda/xor_scatter.cuh | 35 +++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/src/kernels/cuda/xor_scatter.cuh b/src/kernels/cuda/xor_scatter.cuh index d8fd1a70edb..9a6374878fa 100644 --- a/src/kernels/cuda/xor_scatter.cuh +++ b/src/kernels/cuda/xor_scatter.cuh @@ -11,27 +11,42 @@ namespace strata::kernels { namespace { -// NV (a power of two, at most 32) values a lane; afterwards v[0] is the sum of value xor_scatter_index(lane) -template -__device__ __forceinline__ void xor_scatter(float (&v)[NV], int lane) { - static_assert(NV >= 1 && NV <= 32 && (NV & (NV - 1)) == 0, "NV must be a power of two <= 32"); -#pragma unroll - for (int s = 0; s < 5; ++s) { - const int o = 16 >> s, n = NV >> s; // n: the values left before this step - if (n > 1) { +// c ? a : b as a select of values: the compiler turns `c ? v[i] : v[j]` into a load from a selected address, which +// puts the array in local memory +__device__ __forceinline__ float select_f(bool c, float a, float b) { + float r; + asm("{ .reg .pred p; setp.ne.u32 p, %3, 0; selp.f32 %0, %1, %2, p; }" : "=f"(r) : "f"(a), "f"(b), "r"((unsigned) c)); + return r; +} + +// step S of xor_scatter (distance 16 >> S); the steps are template recursion so that every loop has a constant trip +// count and unrolls (a rolled loop indexes v at run time, in local memory) +template +__device__ __forceinline__ void xor_scatter_step(float (&v)[NV], int lane) { + if constexpr (S < 5) { + constexpr int o = 16 >> S, n = NV >> S; // n: the values left before this step + if constexpr (n > 1) { const bool hi = (lane & o) != 0; #pragma unroll for (int e = 0; e < n / 2; ++e) { - const float mine = hi ? v[n / 2 + e] : v[e]; - const float other = hi ? v[e] : v[n / 2 + e]; + const float mine = select_f(hi, v[n / 2 + e], v[e]); + const float other = select_f(hi, v[e], v[n / 2 + e]); v[e] = mine + __shfl_xor_sync(0xffffffffu, other, o); } } else { v[0] += __shfl_xor_sync(0xffffffffu, v[0], o); } + xor_scatter_step(v, lane); } } +// NV (a power of two, at most 32) values a lane; afterwards v[0] is the sum of value xor_scatter_index(lane) +template +__device__ __forceinline__ void xor_scatter(float (&v)[NV], int lane) { + static_assert(NV >= 1 && NV <= 32 && (NV & (NV - 1)) == 0, "NV must be a power of two <= 32"); + xor_scatter_step(v, lane); +} + // the value a lane's v[0] sums after xor_scatter: the halves its bits 16, 8, ... selected template __device__ __forceinline__ int xor_scatter_index(int lane) { From 3d133bae68c41d77ed868ae4999eafca6397cc5f Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 18:28:37 +0200 Subject: [PATCH 57/92] Hyper-connection read: the down projection split by input slices, the norm folded in The down projection (w_down's 320 rows and w_inject's 4 over the 10240 inputs) ran as 41 blocks that each read every token's whole normalized input through the L2, after a separate norm kernel. Now 80 blocks of 288 threads each take a quarter of the 324 rows over a twentieth of the input (a slice lies in one stream): a block stages its slice of R' * w_norm (R' = R plus the pending write) and the slice's sums of squares, a warp takes 9 rows so each input chunk serves 9 rows, and the last block of each quarter (a counter in the scratch) adds the 20 partials in slice order, per stream times the stream's 1/rms, then silu. The up projection is unchanged. A window above 4 tokens runs the down projection in launches of 4; every token's arithmetic is the same whatever its place in the window and the window's size. The scratch is fused_gr_scratch_bytes(), zeroed once (the arenas, session_zero); the single-token path uses the same kernels. In graphs, 1 / 4 tokens: norm + down 16.8 / 26.2 -> 12.2 / 15.1 us, the whole read 26.9 / 40.1 -> 22.7 / 29.2 us (scratch/hc2_bench.cu); in the window 25-42 us a read, from 34-54. Fixed text, 2 rounds against the previous commit: IQ3_XXS code +1.1%, thinking +4.9%, after 16K +3.5%; UD-Q4_K_XL +2.5%, +0.5%, +1.7% (where the CPU is the later side, part of the gain goes to the 5070 Ti's prefetch copies overlapping the pool). Not bitwise. verify_parity: each token bitwise its 1-token read, and within 1e-4 of the terms' magnitudes of an FP64 reference. Along llama.cpp's greedy texts (bench/parity.py): argmax agreement 99.3 / 97.8 / 98.3% on IQ3_XXS code / thinking / after 16K (before 99.7 / 97.8 / 98.7) and 99.3 / 96.0 / 98.7% on UD-Q4_K_XL (99.6 / 95.3 / 98.3); against a build whose hyper-connection read runs in FP64 the new and old kernels are equally close. Claude-Session: https://claude.ai/code/session_01CSrATbDcrJX2zdNDvTYaSr --- include/strata/core/layer.hpp | 1 + include/strata/core/mtp.hpp | 2 +- include/strata/core/verify.hpp | 3 +- include/strata/kernels/fused_gr.hpp | 43 ++- src/core/layer.cpp | 10 +- src/core/mtp.cpp | 6 +- src/core/session.cpp | 3 + src/core/verify.cpp | 5 +- src/kernels/cuda/fused_gr.cu | 417 +++++++++++----------------- src/kernels/verify_parity.cpp | 108 ++++++- 10 files changed, 294 insertions(+), 304 deletions(-) diff --git a/include/strata/core/layer.hpp b/include/strata/core/layer.hpp index dfb9d9dcbc0..b3f029c98a8 100644 --- a/include/strata/core/layer.hpp +++ b/include/strata/core/layer.hpp @@ -381,6 +381,7 @@ struct BlockBuffers { /// because the fused read of one half reads the other half's injection while writing its own. float* inject2 = nullptr; ///< hc float* gr_rs = nullptr; ///< hc: the fused read's per-stream 1/rms + float* gr_scratch = nullptr; ///< `fused_gr_scratch_bytes()`: the fused read's scratch (session_zero) uint8_t* head_q8k = nullptr; ///< `q8k_bytes(n_embd)`: the head's Q8_K activation image strata::kernels::GrWorkspace gr; ///< `gr_read`'s scratch diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index ac8cd57c4c1..b60716bbb1b 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -136,7 +136,7 @@ class MtpDrafter { int32_t *tok_ = nullptr, *step_ = nullptr, *pos_ = nullptr, *row_ = nullptr, *ident_ = nullptr; float *Rin_ = nullptr, *R_ = nullptr, *emb_ = nullptr, *en_ = nullptr, *e2_ = nullptr, *hn_ = nullptr, *h2_ = nullptr; float *mixed_ = nullptr, *inj_ = nullptr, *inj2_ = nullptr, *lo_ = nullptr, *rs_ = nullptr, *bo_ = nullptr; - float* xn_ = nullptr; + float* grs_ = nullptr; // the hyper-connection reads' scratch (zeroed with the arena) uint8_t* xq_ = nullptr; float *qfull_ = nullptr, *qcur_ = nullptr, *kcur_ = nullptr, *vcur_ = nullptr, *attn_ = nullptr, *attn32_ = nullptr; float* attn_scratch_ = nullptr; diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index c2e5157d108..035f0c3cce3 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -199,7 +199,8 @@ class Verifier { void* arena_ = nullptr; int32_t *tok_ = nullptr, *step_ = nullptr, *pos_ = nullptr, *commit_ = nullptr; float *ple_ = nullptr, *emb_ = nullptr, *R_ = nullptr, *mixed_ = nullptr, *bo_ = nullptr; - float *inj_ = nullptr, *inj2_ = nullptr, *lo_ = nullptr, *rs_ = nullptr, *xn_ = nullptr; + float *inj_ = nullptr, *inj2_ = nullptr, *lo_ = nullptr, *rs_ = nullptr; + float* grs_ = nullptr; // the hyper-connection reads' scratch (zeroed with the arena) uint8_t* xq_ = nullptr; // T columns of q8_1 uint8_t* xil_ = nullptr; // their interleaved copy (2+ tokens) float *qkv_L_ = nullptr, *h_L_ = nullptr, *gate_L_ = nullptr, *beta_L_ = nullptr; // per GDN layer diff --git a/include/strata/kernels/fused_gr.hpp b/include/strata/kernels/fused_gr.hpp index e3136a8fbef..f7ff353a43e 100644 --- a/include/strata/kernels/fused_gr.hpp +++ b/include/strata/kernels/fused_gr.hpp @@ -1,28 +1,28 @@ -// include/strata/kernels/fused_gr.hpp - plan v0.3 P3: the hyper-connection read in TWO kernels, with the -// previous half's write folded in. +// include/strata/kernels/fused_gr.hpp - the hyper-connection read in two kernels, with the previous half's write +// folded in, for up to 8 tokens that share the weights. // -// The native path spends six kernels per `gr_read` (norm, down MMVF, silu, up MMVF, gate+mean, inject MMVF) and -// one per `gr_write`, 96 + 96 times per token, and its up projection (10240 rows of 320) runs one 160-thread -// block per row: 20.6 us for 6.5 MB. Here: +// down : R' = R + bo_prev * 2 sigmoid(inj_prev / hc) (only when `apply`, computed on the fly) +// rs[c] = rsqrt(mean(R'[c]^2) + eps) +// lo[k] = silu((w_down[k] . xn) / hc), xn = R' * w_norm * rs k < hc_lr +// inject[c] = w_inject[c] . xn when w_inject is given +// up : R <- R' in place (when `apply`) +// mixed[d] = mean_c xn[c,d] * sigmoid(w_up[c*n_embd + d] . lo) // -// fused_gr_down : R' = R + bo_prev * 2 sigmoid(inj_prev / hc) (only when `apply`, computed on the fly) -// rs[c] = rsqrt(mean(R'[c]^2) + eps), xn = R' * w_norm * rs -// lo[k] = silu((w_down[k] . xn) / hc) k < hc_lr -// inject[c] = w_inject[c] . xn when w_inject is given -// fused_gr_up : R <- R' in place for this block's columns (when `apply`) -// mixed[d] = mean_c xn[c,d] * sigmoid(w_up[c*n_embd + d] . lo) -// -// FP32 activations and BF16 weights, like the native MMVF contract; the summation order differs from it (G-C -// judges the result). Geometry is the artifact's: n_embd 2560, hc 4, hc_lr 320. `inj_prev` and `inject_out` -// must be different buffers (every block reads the former while one block writes the latter). +// The down projection is split by input slices: 80 blocks each take a quarter of the 324 rows over a twentieth of +// the input, so each block reads 1/20 of every token's R' instead of all of it, and the stream's rs multiplies the +// slice sums (a slice lies in one stream). The last block of each quarter adds the partials in a fixed order. +// FP32 activations and BF16 weights; every token's outputs are the same whatever its place in the window and the +// window's size. Geometry is the artifact's: n_embd 2560, hc 4, hc_lr 320. `inj_prev` and `inject_out` must be +// different buffers (every block reads the former while one block writes the latter). #pragma once +#include #include namespace strata::kernels { struct FusedGrArgs { - const float* R = nullptr; ///< (hc, n_embd), read by `down`; `up` updates it in place when apply + const float* R = nullptr; ///< (hc, n_embd); `up` updates it in place when apply float* R_out = nullptr; ///< == R for the in-place update bool apply = false; ///< fold the previous half's gr_write const float* bo_prev = nullptr; ///< that half's block output, n_embd @@ -39,13 +39,12 @@ struct FusedGrArgs { }; bool fused_gr_supported(int64_t n_embd, int64_t hc, int64_t hc_lr); -void fused_gr_read(const FusedGrArgs& a, void* stream); -/// Plan v0.3 P6: the same read for up to 8 tokens that share the weights (a verify window): the weights are read -/// once for all of them. `a[t]` is token t's arguments (its own R, pending write, lo, rs, inject, mixed; the four -/// weight pointers and eps must be the same for every t); `xn_scratch` is n_tok * hc * n_embd floats. Every -/// token's outputs are bitwise `fused_gr_read(a[t])`. +/// The read of `n_tok` tokens (1-8): `a[t]` is token t's arguments (its own R, pending write, lo, rs, inject, +/// mixed; the four weight pointers and eps must be the same for every t). `scratch` is `fused_gr_scratch_bytes()`, +/// zeroed once when allocated (the kernels leave it zeroed); reads that share it must run in stream order. constexpr int kFusedGrMaxT = 8; -void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* xn_scratch, void* stream); +size_t fused_gr_scratch_bytes(); +void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* scratch, void* stream); } // namespace strata::kernels diff --git a/src/core/layer.cpp b/src/core/layer.cpp index b2a275ff836..1e05f020e58 100644 --- a/src/core/layer.cpp +++ b/src/core/layer.cpp @@ -908,8 +908,10 @@ n += (uint64_t) g.hc * 4 * 2 + 32; // inject2, gr_rs n += q8k_bytes(g.n_embd); // head_q8k -n += strata::kernels::gr_workspace_bytes(s); return align_up16(n) + 256;} -uint64_t block_buffers_init(const ModelGeometry& g, void* base, BlockBuffers& b) { const strata::kernels::GrShapes s{g.n_embd, g.hc, g.hc_lr}; Cursor c{(uint8_t*) base}; b.R = c.take((uint64_t) g.hc * g.n_embd); b.mixed = c.take((uint64_t) g.n_embd); b.block_out = c.take((uint64_t) g.n_embd); b.inject = c.take((uint64_t) g.hc); b.inject2 = c.take((uint64_t) g.hc); b.gr_rs = c.take((uint64_t) g.hc); b.head_q8k = c.take_bytes(q8k_bytes(g.n_embd)); uint8_t* grw = c.take_bytes(strata::kernels::gr_workspace_bytes(s)); strata::kernels::gr_workspace_init(s, grw, b.gr); return c.used;} +n += strata::kernels::gr_workspace_bytes(s); +// gr_scratch +n += align_up16(strata::kernels::fused_gr_scratch_bytes()); return align_up16(n) + 256;} +uint64_t block_buffers_init(const ModelGeometry& g, void* base, BlockBuffers& b) { const strata::kernels::GrShapes s{g.n_embd, g.hc, g.hc_lr}; Cursor c{(uint8_t*) base}; b.R = c.take((uint64_t) g.hc * g.n_embd); b.mixed = c.take((uint64_t) g.n_embd); b.block_out = c.take((uint64_t) g.n_embd); b.inject = c.take((uint64_t) g.hc); b.inject2 = c.take((uint64_t) g.hc); b.gr_rs = c.take((uint64_t) g.hc); b.gr_scratch = (float*) c.take_bytes(strata::kernels::fused_gr_scratch_bytes()); b.head_q8k = c.take_bytes(q8k_bytes(g.n_embd)); uint8_t* grw = c.take_bytes(strata::kernels::gr_workspace_bytes(s)); strata::kernels::gr_workspace_init(s, grw, b.gr); return c.used;} // ---- THE HALF-LEVEL C1 ORACLE. `dump` is a HOST buffer of `2 * n_embd + 2 * hc + n_head * head_dim` floats // per layer; this enqueues one slice of it as a device-to-host copy. **IT RUNS INSIDE THE CAPTURE**, which is // the point: the copy becomes a node of that layer's graph and replays with it, so the layer functions stay one @@ -1020,7 +1022,7 @@ st_begin(layer, 0, stream); fa.w_norm = (const float*) w_norm[0]->data; fa.w_down = (const uint16_t*) w_down[0]->data; fa.w_up = (const uint16_t*) w_up[0]->data; fa.w_inject = (const uint16_t*) w_inject[0]->data; fa.eps = RMS_EPS; fa.lo = bb.gr.lo; fa.rs = bb.gr_rs; fa.inject_out = bb.inject; fa.mixed = bb.mixed; - strata::kernels::fused_gr_read(fa, stream); + strata::kernels::fused_gr_read_multi(&fa, 1, bb.gr_scratch, stream); } else { gr_read(R, (const float*) w_norm[0]->data, (const uint16_t*) w_down[0]->data, (const uint16_t*) w_up[0]->data, (const uint16_t*) w_inject[0]->data, RMS_EPS, gs, bb.gr, bb.mixed, bb.inject, stream); } @@ -1043,7 +1045,7 @@ st_begin(layer, 3, stream); fa.w_norm = (const float*) w_norm[1]->data; fa.w_down = (const uint16_t*) w_down[1]->data; fa.w_up = (const uint16_t*) w_up[1]->data; fa.w_inject = (const uint16_t*) w_inject[1]->data; fa.eps = RMS_EPS; fa.lo = bb.gr.lo; fa.rs = bb.gr_rs; fa.inject_out = bb.inject2; fa.mixed = bb.mixed; - strata::kernels::fused_gr_read(fa, stream); + strata::kernels::fused_gr_read_multi(&fa, 1, bb.gr_scratch, stream); } else { gr_read(R, (const float*) w_norm[1]->data, (const uint16_t*) w_down[1]->data, (const uint16_t*) w_up[1]->data, (const uint16_t*) w_inject[1]->data, RMS_EPS, gs, bb.gr, bb.mixed, bb.inject, stream); } diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 159302f92f7..a0d52a8f998 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -199,7 +199,7 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session hn_ = b.take(T * HC * N); h2_ = b.take(T * HC * N); mixed_ = b.take(T * N); inj_ = b.take(T * HC); inj2_ = b.take(T * HC); lo_ = b.take(T * (uint64_t) g.hc_lr); rs_ = b.take(T * HC); bo_ = b.take(T * N); - xn_ = b.take(T * HC * N); + grs_ = (float*) b.take(strata::kernels::fused_gr_scratch_bytes()); xq_ = b.take(strata::kernels::native_q8_1_bytes((int) (NH * HD), 8)); qfull_ = b.take(T * NH * 2 * HD); qcur_ = b.take(T * NH * HD); kcur_ = b.take(T * NKV * HD); vcur_ = b.take(T * NKV * HD); @@ -319,7 +319,7 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri fa[t].eps = EPS; fa[t].lo = lo_ + t * g.hc_lr; fa[t].rs = rs_ + t * HC; fa[t].inject_out = inj_ + t * HC; fa[t].mixed = mixed_ + t * N; } - fused_gr_read_multi(fa, T, xn_, cs); + fused_gr_read_multi(fa, T, grs_, cs); } // ---- attention: K/V into the layer's own cache, then (full) dense attention over every cell auto norm_rope = [&](float* data, const float* gamma, int rows, int cols, const int32_t* p) { @@ -373,7 +373,7 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri fa[t].eps = EPS; fa[t].lo = lo_ + t * g.hc_lr; fa[t].rs = rs_ + t * HC; fa[t].inject_out = inj2_ + t * HC; fa[t].mixed = mixed_ + t * N; } - fused_gr_read_multi(fa, T, xn_, cs); + fused_gr_read_multi(fa, T, grs_, cs); } // ---- MoE: router, the 512 resident experts, the shared expert, the combine, the write for (int t = 0; t < T; ++t) { diff --git a/src/core/session.cpp b/src/core/session.cpp index 43fc925770a..79761aebc6b 100644 --- a/src/core/session.cpp +++ b/src/core/session.cpp @@ -3,6 +3,7 @@ #include "strata/kernels/qsa.hpp" #include "strata/kernels/elementwise.hpp" +#include "strata/kernels/fused_gr.hpp" #include "strata/kernels/quantize_act.hpp" #include "strata/kernels/s2_expert_grouped.hpp" #include "strata/kernels/cpu/pool.hpp" @@ -114,6 +115,8 @@ void session_zero(SessionState& s, const ModelGeometry& g, const float* R_init, } else { cudaMemsetAsync(s.block.R, 0, (size_t) g.hc * g.n_embd * 4, cs); } + // the fused hyper-connection read's scratch: its counters start at zero + cudaMemsetAsync(s.block.gr_scratch, 0, strata::kernels::fused_gr_scratch_bytes(), cs); // every GDN layer's recurrence and conv history cudaMemsetAsync(s.gdn_state, 0, (size_t) g.n_gdn_layers() * gdn_state_floats(g) * 4, cs); // and every QSA layer's cache and indexer diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 3be2293af87..1247e480191 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -214,7 +214,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& ple_ = b.take(T * N); emb_ = b.take(T * N); R_ = b.take(T * HC * N); mixed_ = b.take(T * N); bo_ = b.take(T * N); inj_ = b.take(T * HC); inj2_ = b.take(T * HC); - lo_ = b.take(T * (uint64_t) g.hc_lr); rs_ = b.take(T * HC); xn_ = b.take(T * HC * N); + lo_ = b.take(T * (uint64_t) g.hc_lr); rs_ = b.take(T * HC); + grs_ = (float*) b.take(strata::kernels::fused_gr_scratch_bytes()); xq_ = b.take(strata::kernels::native_q8_1_bytes(max_in, (int) T)); xil_ = b.take(strata::kernels::native_q8_1_il_bytes(max_in, (int) T)); qkv_L_ = b.take(nG * T * C); h_L_ = b.take(nG * T * C); @@ -492,7 +493,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { a.eps = EPS; a.lo = lo_ + t * g.hc_lr; a.rs = rs_ + t * HC; a.inject_out = inj_out + t * HC; a.mixed = mixed_ + t * N; } - fused_gr_read_multi(fa, n, xn_ + (size_t) tb * HC * N, cs); + fused_gr_read_multi(fa, n, grs_, cs); }; gr_read_group(0, pending, inj2_, inj_); stamp(l, 1, cs); diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu index 9732aacc596..99981798d97 100644 --- a/src/kernels/cuda/fused_gr.cu +++ b/src/kernels/cuda/fused_gr.cu @@ -1,10 +1,10 @@ // src/kernels/cuda/fused_gr.cu - see include/strata/kernels/fused_gr.hpp. #include "strata/kernels/fused_gr.hpp" -#include "strata/kernels/bf16_bits.hpp" #include "xor_scatter.cuh" #include +#include #include #include @@ -15,11 +15,23 @@ constexpr int N = 2560; // n_embd constexpr int HC = 4; // streams constexpr int D = N * HC; // 10240 constexpr int LR = 320; // hc_lr +constexpr int NR = LR + HC; // the down projection's rows: w_down's 320, then w_inject's 4 + +// down: the NR rows in DG groups of DRB = DW warps x DR rows, the input in DS slices of DCS columns (a slice lies in +// one stream); a block per (slice, group), up to DT tokens a launch +constexpr int DG = 4, DS = 20, DW = 9, DR = 9, DT = 4; +constexpr int DTH = DW * 32, DRB = DW * DR, DCS = D / DS, DSPC = DS / HC, DSTEPS = DCS / 256, DNF = DCS / 4; +static_assert(DRB * DG == NR && DCS % 256 == 0 && DS % HC == 0 && DNF <= DTH, "down geometry"); + +// up: UPM_COLS columns of every stream a block, UPM_ROWS rows a warp constexpr int THREADS = 256; constexpr int WARPS = THREADS / 32; -constexpr int DOWN_BLOCKS = LR / WARPS; // 40 blocks of 8 rows; one more for the inject rows -constexpr int UP_COLS = 32; // columns d per `up` block (x 4 streams = 128 rows) -constexpr int UP_BLOCKS = N / UP_COLS; // 80 +constexpr int UPM_COLS = 16; +constexpr int UPM_BLOCKS = N / UPM_COLS; // 160 +constexpr int UPM_ROWS = HC * UPM_COLS / WARPS; // 8 + +// the scratch (floats): DG counters, the slices' sums of squares [DG][DS][DT], the partial dots [DS][DT][NR] +constexpr int SC_CTR = 0, SC_SSP = 64, SC_PART = SC_SSP + DG * DS * DT, SC_FLOATS = SC_PART + DS * DT * NR; __device__ __forceinline__ float warp_sum(float v) { #pragma unroll @@ -28,258 +40,167 @@ __device__ __forceinline__ float warp_sum(float v) { } __device__ __forceinline__ float sigmoidf_(float x) { return 1.0f / (1.0f + __expf(-x)); } -// 8 bf16 packed in a uint4 against 8 floats. -__device__ __forceinline__ float dot8(const uint4 w, const float* x) { +// 8 bf16 packed in a uint4 against 8 floats held as two float4 +__device__ __forceinline__ float dot8h(const uint4 w, const float4 x0, const float4 x1) { float acc = 0.0f; - const uint32_t v[4] = {w.x, w.y, w.z, w.w}; -#pragma unroll - for (int j = 0; j < 4; ++j) { - acc = fmaf(__uint_as_float(v[j] << 16), x[2 * j], acc); - acc = fmaf(__uint_as_float(v[j] & 0xffff0000u), x[2 * j + 1], acc); - } + acc = fmaf(__uint_as_float(w.x << 16), x0.x, acc); + acc = fmaf(__uint_as_float(w.x & 0xffff0000u), x0.y, acc); + acc = fmaf(__uint_as_float(w.y << 16), x0.z, acc); + acc = fmaf(__uint_as_float(w.y & 0xffff0000u), x0.w, acc); + acc = fmaf(__uint_as_float(w.z << 16), x1.x, acc); + acc = fmaf(__uint_as_float(w.z & 0xffff0000u), x1.y, acc); + acc = fmaf(__uint_as_float(w.w << 16), x1.z, acc); + acc = fmaf(__uint_as_float(w.w & 0xffff0000u), x1.w, acc); return acc; } -__global__ void __launch_bounds__(THREADS) gr_down_kernel(FusedGrArgs a) { - __shared__ __align__(16) float xn[D]; - __shared__ float part[WARPS][HC]; - __shared__ float s_rs[HC]; - const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - float gw[HC]; -#pragma unroll - for (int c = 0; c < HC; ++c) gw[c] = a.apply ? 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC) : 0.0f; - // 1. R' * w_norm into shared memory, and the per-stream sums of squares of R'. - float ss[HC] = {0.0f, 0.0f, 0.0f, 0.0f}; - for (int i = t * 4; i < D; i += THREADS * 4) { - const int c = i / N, d = i - c * N; - float4 r = *reinterpret_cast(a.R + i); - if (a.apply) { - const float4 b = *reinterpret_cast(a.bo_prev + d); - r.x = fmaf(b.x, gw[c], r.x); r.y = fmaf(b.y, gw[c], r.y); - r.z = fmaf(b.z, gw[c], r.z); r.w = fmaf(b.w, gw[c], r.w); - } - const float4 g = *reinterpret_cast(a.w_norm + i); - float sq = r.x * r.x + r.y * r.y + r.z * r.z + r.w * r.w; -#pragma unroll - for (int cc = 0; cc < HC; ++cc) if (cc == c) ss[cc] += sq; - *reinterpret_cast(xn + i) = make_float4(r.x * g.x, r.y * g.y, r.z * g.z, r.w * g.w); - } +struct GrMulti { + FusedGrArgs a[kFusedGrMaxT]; + int T; +}; + +__host__ __device__ constexpr int pow2_floor(int n) { return n < 2 ? 1 : 2 * pow2_floor(n / 2); } + +// v[B, V) summed across the warp in power-of-two chunks of xor_scatter; emit(value, sum) on one lane per value +template +__device__ __forceinline__ void reduce_chunks(const float (&v)[VT], int lane, F& emit) { + if constexpr (B < V) { + constexpr int P = pow2_floor(V - B > 32 ? 32 : V - B); + float cv[P]; #pragma unroll - for (int c = 0; c < HC; ++c) { - const float v = warp_sum(ss[c]); - if (lane == 0) part[warp][c] = v; - } - __syncthreads(); - if (t < HC) { - float s = 0.0f; - for (int w = 0; w < WARPS; ++w) s += part[w][t]; - s_rs[t] = rsqrtf(s / (float) N + a.eps); - if (blockIdx.x == 0) a.rs[t] = s_rs[t]; - } - __syncthreads(); - for (int i = t; i < D; i += THREADS) xn[i] *= s_rs[i / N]; - __syncthreads(); - // 2. one warp per output row: 10240 bf16 = 1280 chunks of 8, 40 per lane. - const bool inject_block = blockIdx.x == DOWN_BLOCKS; - const int row = inject_block ? warp : blockIdx.x * WARPS + warp; - if (inject_block && (a.w_inject == nullptr || warp >= HC)) return; - const uint16_t* wrow = (inject_block ? a.w_inject : a.w_down) + (size_t) row * D; - const uint4* w4 = reinterpret_cast(wrow); - float acc = 0.0f; -#pragma unroll 4 - for (int j = lane; j < D / 8; j += 32) acc += dot8(__ldg(w4 + j), xn + j * 8); - acc = warp_sum(acc); - if (lane != 0) return; - if (inject_block) { - a.inject_out[row] = acc; - } else { - const float x = acc / (float) HC; - a.lo[row] = x / (1.0f + __expf(-x)); + for (int e = 0; e < P; ++e) cv[e] = v[B + e]; + xor_scatter

(cv, lane); + if ((lane & (32 / P - 1)) == 0) emit(B + xor_scatter_index

(lane), cv[0]); + reduce_chunks(v, lane, emit); } } -__global__ void __launch_bounds__(THREADS) gr_up_kernel(FusedGrArgs a) { - __shared__ __align__(16) float lo[LR]; - __shared__ float g[HC][UP_COLS]; +// The down projection and the injection of TT tokens. Block (s, g): warp w takes rows g DRB + w DR + [0, DR) of +// [w_down; w_inject] over the slice's DCS columns, a lane 8 columns a step, each activation chunk loaded once for its +// DR rows. The slice's R' = R + bo_prev * 2 sigmoid(inj_prev / hc) times w_norm is staged in shared memory split into +// the halves of the 8-column chunks (conflict-free 16-byte reads), the sums of squares of R' go to the scratch; the stream's 1/rms is applied to the slice sums later, since a slice lies +// in one stream. The group's last block (a counter) adds each row's partials in slice order, per stream times the +// stream's 1/rms, and writes lo = silu(y / hc) or the injection; group 0's also writes rs. Every token's arithmetic +// is the same whatever its place in the window and the window's size. +template +__global__ void __launch_bounds__(DTH) gr_down_split_kernel(GrMulti m, float* __restrict__ scratch) { + static_assert(TT >= 1 && TT <= DT, "tokens a launch"); + __shared__ __align__(16) float4 xs[TT][DNF]; + __shared__ float ssw[TT][DW]; + __shared__ float s_rs[TT][HC]; + __shared__ bool s_last; + unsigned* ctr = reinterpret_cast(scratch + SC_CTR); + float* ssp = scratch + SC_SSP; + float* part = scratch + SC_PART; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - const int d0 = blockIdx.x * UP_COLS; - for (int k = t; k < LR; k += THREADS) lo[k] = a.lo[k]; - __syncthreads(); - // 128 rows (4 streams x 32 columns), 16 per warp: 320 bf16 = 40 chunks of 8. - for (int r = warp; r < HC * UP_COLS; r += WARPS) { - const int c = r / UP_COLS, dd = r - c * UP_COLS, i = c * N + d0 + dd; - const uint4* w4 = reinterpret_cast(a.w_up + (size_t) i * LR); - float acc = dot8(__ldg(w4 + lane), lo + lane * 8); - if (lane < LR / 8 - 32) acc += dot8(__ldg(w4 + 32 + lane), lo + (32 + lane) * 8); - acc = warp_sum(acc); - if (lane == 0) { - float rv = a.R[i]; - if (a.apply) { - rv = fmaf(a.bo_prev[d0 + dd], 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC), rv); - a.R_out[i] = rv; // this block owns column d0+dd of every stream - } - const float x = rv * a.w_norm[i] * a.rs[c]; - g[c][dd] = x * sigmoidf_(acc); + const int s = blockIdx.x, g = blockIdx.y; + const int c = s / DSPC, col0 = s * DCS; + const bool has_inj = m.a[0].w_inject != nullptr; + const int rb0 = g * DRB + warp * DR; + // the staging loads first, then the weights + float4 pr[TT], pb[TT], pw = make_float4(0.0f, 0.0f, 0.0f, 0.0f); + if (t < DNF) { + pw = *reinterpret_cast(m.a[0].w_norm + col0 + 4 * t); +#pragma unroll + for (int k = 0; k < TT; ++k) { + const FusedGrArgs& a = m.a[k]; + pr[k] = *reinterpret_cast(a.R + col0 + 4 * t); + if (a.apply) pb[k] = *reinterpret_cast(a.bo_prev + col0 - c * N + 4 * t); } } - __syncthreads(); - if (t < UP_COLS) { - float s = 0.0f; + uint4 w[DR][DSTEPS]; +#pragma unroll + for (int i = 0; i < DR; ++i) { + const int rb = rb0 + i; + const bool ok = rb < LR || has_inj; + const uint16_t* base = rb < LR ? m.a[0].w_down + (size_t) rb * D : m.a[0].w_inject + (size_t) (ok ? rb - LR : 0) * D; + const uint4* w4 = reinterpret_cast(base + col0) + lane; #pragma unroll - for (int c = 0; c < HC; ++c) s += g[c][t]; - a.mixed[d0 + t] = s / (float) HC; + for (int j = 0; j < DSTEPS; ++j) w[i][j] = ok ? __ldg(w4 + j * 32) : make_uint4(0, 0, 0, 0); } -} - -// ================================ plan v0.3 P6: T tokens, one weight read ================================ -struct GrMulti { - FusedGrArgs a[kFusedGrMaxT]; - float* xn; - int T; -}; - -// Step 1 of `gr_down_kernel`, one block per (token, stream): each thread visits its elements of that stream in the -// order `gr_down_kernel`'s thread does, and the stream's warp partials are summed in the same order. rs and xn to -// global. -__global__ void __launch_bounds__(THREADS) gr_norm_multi_kernel(GrMulti m) { - __shared__ float part[WARPS]; - const int tok = blockIdx.x, c = blockIdx.y; - const FusedGrArgs& a = m.a[tok]; - float* xn = m.xn + (size_t) tok * D; - const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - const float gw = a.apply ? 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC) : 0.0f; - constexpr int STEPS = D / (THREADS * 4); // 10 - float4 p[STEPS]; - float ss = 0.0f; #pragma unroll - for (int j = 0; j < STEPS; ++j) { - const int i = t * 4 + j * THREADS * 4; - if (i / N != c) continue; - float4 r = *reinterpret_cast(a.R + i); - if (a.apply) { - const float4 bo = *reinterpret_cast(a.bo_prev + (i - c * N)); - r.x = fmaf(bo.x, gw, r.x); r.y = fmaf(bo.y, gw, r.y); - r.z = fmaf(bo.z, gw, r.z); r.w = fmaf(bo.w, gw, r.w); + for (int k = 0; k < TT; ++k) { + const FusedGrArgs& a = m.a[k]; + const float gw = a.apply ? 2.0f * sigmoidf_(a.inj_prev[c] / (float) HC) : 0.0f; + float sq = 0.0f; + if (t < DNF) { + float4 r = pr[k]; + if (a.apply) { + r.x = fmaf(pb[k].x, gw, r.x); r.y = fmaf(pb[k].y, gw, r.y); + r.z = fmaf(pb[k].z, gw, r.z); r.w = fmaf(pb[k].w, gw, r.w); + } + sq = fmaf(r.x, r.x, sq); sq = fmaf(r.y, r.y, sq); sq = fmaf(r.z, r.z, sq); sq = fmaf(r.w, r.w, sq); + xs[k][(t & 1) * (DNF / 2) + (t >> 1)] = make_float4(r.x * pw.x, r.y * pw.y, r.z * pw.z, r.w * pw.w); } - const float4 g = *reinterpret_cast(a.w_norm + i); - const float sq = r.x * r.x + r.y * r.y + r.z * r.z + r.w * r.w; - ss += sq; - p[j] = make_float4(r.x * g.x, r.y * g.y, r.z * g.z, r.w * g.w); + sq = warp_sum(sq); + if (lane == 0) ssw[k][warp] = sq; } - ss = warp_sum(ss); - if (lane == 0) part[warp] = ss; __syncthreads(); - float sum = 0.0f; - for (int w = 0; w < WARPS; ++w) sum += part[w]; - const float rs = rsqrtf(sum / (float) N + a.eps); - if (t == 0) a.rs[c] = rs; + if (t < TT) { + float sq = 0.0f; #pragma unroll - for (int j = 0; j < STEPS; ++j) { - const int i = t * 4 + j * THREADS * 4; - if (i / N != c) continue; - *reinterpret_cast(xn + i) = make_float4(p[j].x * rs, p[j].y * rs, p[j].z * rs, p[j].w * rs); + for (int q = 0; q < DW; ++q) sq += ssw[t][q]; + ssp[(g * DS + s) * DT + t] = sq; } -} - -// 8 bf16 packed in a uint4 against 8 floats held as two float4 (the halves of a staged chunk): `dot8`'s order. -__device__ __forceinline__ float dot8h(const uint4 w, const float4 x0, const float4 x1) { - float acc = 0.0f; - acc = fmaf(__uint_as_float(w.x << 16), x0.x, acc); - acc = fmaf(__uint_as_float(w.x & 0xffff0000u), x0.y, acc); - acc = fmaf(__uint_as_float(w.y << 16), x0.z, acc); - acc = fmaf(__uint_as_float(w.y & 0xffff0000u), x0.w, acc); - acc = fmaf(__uint_as_float(w.z << 16), x1.x, acc); - acc = fmaf(__uint_as_float(w.z & 0xffff0000u), x1.y, acc); - acc = fmaf(__uint_as_float(w.w << 16), x1.z, acc); - acc = fmaf(__uint_as_float(w.w & 0xffff0000u), x1.w, acc); - return acc; -} - -// Step 2 of `gr_down_kernel` for T tokens. One warp per row, so each lane accumulates the same chunks in the same -// order as the single-token kernel. The xn tiles of every token are staged split into the first and second halves -// of the 8-float chunks (conflict-free 16-byte reads) and double-buffered; each tile's weights are loaded one tile -// ahead; blocks start their staging at different offsets, which spreads the reads of the same lines over time. -// TC chunks per tile: 320 for up to 4 tokens, 160 above (the two buffers must fit shared memory). -template -__global__ void __launch_bounds__(THREADS) gr_down_multi_kernel(GrMulti m) { - extern __shared__ __align__(16) float4 buf[]; // [2][T][2][TC] - constexpr int TQ = TC / 32, NT = D / 8 / TC; - const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - const int T = m.T; - const bool inject_block = blockIdx.x == DOWN_BLOCKS; - const int row = inject_block ? warp : blockIdx.x * WARPS + warp; - const bool active = !(inject_block && (m.a[0].w_inject == nullptr || warp >= HC)); - const uint4* w4 = reinterpret_cast((inject_block ? m.a[0].w_inject : m.a[0].w_down) + - (size_t) (active ? row : 0) * D); - float acc[kFusedGrMaxT]; + float acc[DR * TT]; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) acc[k] = 0.0f; - const int per_buf = T * 2 * TC, cnt = T * 2 * TC; - const int rot = (int) ((blockIdx.x * 331u) % (unsigned) cnt); - const float4* src4 = reinterpret_cast(m.xn); - auto stage = [&](int tile, float4* dst) { - for (int ii = t; ii < cnt; ii += THREADS) { - int i = ii + rot; - if (i >= cnt) i -= cnt; - const int k = i / (2 * TC), rem = i - k * 2 * TC, j = rem >> 1, h = rem & 1; - dst[(k * 2 + h) * TC + j] = src4[((size_t) k * D + (size_t) tile * TC * 8) / 4 + j * 2 + h]; - } - }; - uint4 wv[TQ], wn[TQ]; - if (active) { + for (int e = 0; e < DR * TT; ++e) acc[e] = 0.0f; #pragma unroll - for (int q = 0; q < TQ; ++q) wn[q] = __ldg(w4 + lane + 32 * q); - } - stage(0, buf); - __syncthreads(); - for (int tile = 0; tile < NT; ++tile) { - const float4* xb = buf + (size_t) (tile & 1) * per_buf; + for (int j = 0; j < DSTEPS; ++j) { + float4 x0[TT], x1[TT]; #pragma unroll - for (int q = 0; q < TQ; ++q) wv[q] = wn[q]; - if (tile + 1 < NT) { - if (active) { + for (int k = 0; k < TT; ++k) { x0[k] = xs[k][j * 32 + lane]; x1[k] = xs[k][DNF / 2 + j * 32 + lane]; } #pragma unroll - for (int q = 0; q < TQ; ++q) wn[q] = __ldg(w4 + (tile + 1) * TC + lane + 32 * q); - } - stage(tile + 1, buf + (size_t) ((tile + 1) & 1) * per_buf); - } - if (active) { + for (int i = 0; i < DR; ++i) #pragma unroll - for (int q = 0; q < TQ; ++q) { - const int j = lane + 32 * q; + for (int k = 0; k < TT; ++k) acc[i * TT + k] += dot8h(w[i][j], x0[k], x1[k]); + } + auto emit = [=](int vi, float sum) { + const int i = vi / TT, k = vi - i * TT, rb = rb0 + i; + if (rb < LR || has_inj) part[(s * DT + k) * NR + rb] = sum; + }; + reduce_chunks(acc, lane, emit); + // the group's last block + __threadfence(); + __syncthreads(); + if (t == 0) s_last = atomicAdd(ctr + g, 1u) == (unsigned) (DS - 1); + __syncthreads(); + if (!s_last) return; + __threadfence(); + if (t < TT * HC) { + const int k = t / HC, cc = t - k * HC; + float sq = 0.0f; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) { - if (k >= T) break; - acc[k] += dot8h(wv[q], xb[(k * 2) * TC + j], xb[(k * 2 + 1) * TC + j]); - } - } - } - __syncthreads(); + for (int q = 0; q < DSPC; ++q) sq += __ldcg(ssp + (g * DS + cc * DSPC + q) * DT + k); + s_rs[k][cc] = rsqrtf(sq / (float) N + m.a[0].eps); } - if (!active) return; - float s[kFusedGrMaxT]; + __syncthreads(); + for (int e = t; e < TT * DRB; e += DTH) { + const int k = e / DRB, r = e - k * DRB, rb = g * DRB + r; + if (rb >= LR && !has_inj) continue; + float y = 0.0f; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) s[k] = k < T ? warp_sum(acc[k]) : 0.0f; - // lane k writes token k (every lane holds every sum after the xor reduction) + for (int cc = 0; cc < HC; ++cc) { + float p = 0.0f; #pragma unroll - for (int k = 0; k < kFusedGrMaxT; ++k) { - if (k >= T || lane != k) continue; - if (inject_block) { - m.a[k].inject_out[row] = s[k]; + for (int q = 0; q < DSPC; ++q) p += __ldcg(part + ((cc * DSPC + q) * DT + k) * NR + rb); + y = fmaf(s_rs[k][cc], p, y); + } + if (rb >= LR) { + m.a[k].inject_out[rb - LR] = y; } else { - const float x = s[k] / (float) HC; - m.a[k].lo[row] = x / (1.0f + __expf(-x)); + const float x = y / (float) HC; + m.a[k].lo[rb] = x / (1.0f + __expf(-x)); } } + if (g == 0 && t < TT * HC) m.a[t / HC].rs[t % HC] = s_rs[t / HC][t % HC]; + if (t == 0) ctr[g] = 0u; } -constexpr int UPM_COLS = 16; // columns per block (x 4 streams = 64 rows, 8 per warp) -constexpr int UPM_BLOCKS = N / UPM_COLS; // 160 -constexpr int UPM_ROWS = HC * UPM_COLS / WARPS; // rows a warp - -// `gr_up_kernel` for TT tokens: each row of w_up read once, RW rows a warp at a time; the TT x RW dots of those rows -// summed by one xor tree (xor_scatter, each sum bitwise warp_sum's) and every (row, token) epilogue on the lane holding -// its sum, its inputs fetched while the dots run. The lo vectors are staged split into chunk halves. +// The up projection of TT tokens: each row of w_up read once, RW rows a warp at a time; the TT x RW dots of those rows +// summed by one xor tree (xor_scatter) and every (row, token) epilogue on the lane holding its sum, its inputs fetched +// while the dots run: R <- R' for this block's columns (when apply), mixed[d] = mean_c xn[c,d] * sigmoid(u[c,d]). +// The lo vectors are staged split into chunk halves. template __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { constexpr int RW = TT <= 4 ? 4 : 2; @@ -351,8 +272,10 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { } // namespace -void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* xn_scratch, void* stream) { - if (n_tok < 1 || n_tok > kFusedGrMaxT || xn_scratch == nullptr) { +size_t fused_gr_scratch_bytes() { return (size_t) SC_FLOATS * sizeof(float); } + +void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* scratch, void* stream) { + if (n_tok < 1 || n_tok > kFusedGrMaxT || scratch == nullptr) { std::fprintf(stderr, "fused_gr_read_multi: invalid arguments\n"); std::exit(1); } @@ -361,27 +284,26 @@ void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* xn_scratch, voi m.a[t] = a[t]; const FusedGrArgs& x = a[t]; if (!x.R || !x.w_norm || !x.w_down || !x.w_up || !x.lo || !x.rs || !x.mixed || (x.w_inject && !x.inject_out) || - (x.apply && (!x.bo_prev || !x.inj_prev || !x.R_out)) || x.w_down != a[0].w_down || x.w_up != a[0].w_up || - x.w_inject != a[0].w_inject || x.w_norm != a[0].w_norm) { + (x.apply && (!x.bo_prev || !x.inj_prev || !x.R_out || x.inj_prev == x.inject_out)) || + x.w_down != a[0].w_down || x.w_up != a[0].w_up || x.w_inject != a[0].w_inject || + x.w_norm != a[0].w_norm || x.eps != a[0].eps) { std::fprintf(stderr, "fused_gr_read_multi: invalid arguments for token %d\n", t); std::exit(1); } } - m.xn = xn_scratch; m.T = n_tok; cudaStream_t st = (cudaStream_t) stream; - gr_norm_multi_kernel<<>>(m); - static bool attr = false; - if (!attr) { // two buffers of the largest tile: 4 tokens x 320 chunks, or 8 x 160 - const int bytes = 2 * 4 * 2 * 320 * (int) sizeof(float4); - cudaFuncSetAttribute(gr_down_multi_kernel<320>, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes); - cudaFuncSetAttribute(gr_down_multi_kernel<160>, cudaFuncAttributeMaxDynamicSharedMemorySize, bytes); - attr = true; + const dim3 down_grid(DS, DG); + for (int t0 = 0; t0 < n_tok; t0 += DT) { // DT tokens a down launch; each token's arithmetic is the same + GrMulti md; + md.T = std::min(DT, n_tok - t0); + for (int k = 0; k < md.T; ++k) md.a[k] = a[t0 + k]; + switch (md.T) { +#define STRATA_DOWN(T) case T: gr_down_split_kernel<<>>(md, scratch); break; + STRATA_DOWN(1) STRATA_DOWN(2) STRATA_DOWN(3) STRATA_DOWN(4) +#undef STRATA_DOWN + } } - if (n_tok <= 4) - gr_down_multi_kernel<320><<>>(m); - else - gr_down_multi_kernel<160><<>>(m); switch (n_tok) { #define STRATA_UP(T) case T: gr_up_multi_kernel<<>>(m); break; STRATA_UP(1) STRATA_UP(2) STRATA_UP(3) STRATA_UP(4) STRATA_UP(5) STRATA_UP(6) STRATA_UP(7) STRATA_UP(8) @@ -398,21 +320,4 @@ bool fused_gr_supported(int64_t n_embd, int64_t hc, int64_t hc_lr) { return n_embd == N && hc == HC && hc_lr == LR; } -void fused_gr_read(const FusedGrArgs& a, void* stream) { - if (!a.R || !a.w_norm || !a.w_down || !a.w_up || !a.lo || !a.rs || !a.mixed || - (a.w_inject && !a.inject_out) || (a.apply && (!a.bo_prev || !a.inj_prev || !a.R_out)) || - (a.apply && a.inj_prev == a.inject_out)) { - std::fprintf(stderr, "fused_gr_read: invalid arguments\n"); - std::exit(1); - } - cudaStream_t st = (cudaStream_t) stream; - gr_down_kernel<<>>(a); - gr_up_kernel<<>>(a); - const cudaError_t e = cudaGetLastError(); - if (e != cudaSuccess) { - std::fprintf(stderr, "fused_gr_read: %s\n", cudaGetErrorString(e)); - std::exit(1); - } -} - } // namespace strata::kernels diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 0e2207a26f4..45ed9b846ed 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -387,7 +387,8 @@ int test_verify_router(std::mt19937& rng, cudaStream_t s) { return bad; } -// ---- the hyper-connection read: fused_gr_read per token vs fused_gr_read_multi +// ---- the hyper-connection read: every token of an n-token read bitwise its 1-token read, and within a rounding +// bound of an FP64 reference (|error| <= 1e-4 x the sum of the terms' magnitudes, carried through silu and sigmoid) int test_gr_read(std::mt19937& rng, cudaStream_t s) { const int64_t N = 2560, HC = 4, D = N * HC, LR = 320; auto bf16 = [&](size_t n) { @@ -400,16 +401,21 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { } return v; }; + auto bf = [](uint16_t v) { const uint32_t b = (uint32_t) v << 16; float f; std::memcpy(&f, &b, 4); return (double) f; }; std::vector wn((size_t) D); for (auto& v : wn) v = edgy(rng); - uint16_t *d_wd = dev((size_t) (LR * D)), *d_wu = dev((size_t) (D * LR)), - *d_wi = dev((size_t) (HC * D)); + const std::vector wd = bf16((size_t) (LR * D)), wu = bf16((size_t) (D * LR)), wi = bf16((size_t) (HC * D)); + uint16_t *d_wd = dev(wd.size()), *d_wu = dev(wu.size()), *d_wi = dev(wi.size()); float* d_wn = dev(wn.size()); - up(d_wd, bf16((size_t) (LR * D))); - up(d_wu, bf16((size_t) (D * LR))); - up(d_wi, bf16((size_t) (HC * D))); + up(d_wd, wd); + up(d_wu, wu); + up(d_wi, wi); up(d_wn, wn); - int bad = 0; + const size_t sb = strata::kernels::fused_gr_scratch_bytes(); + float* d_scratch = (float*) dev(sb); + check(cudaMemset(d_scratch, 0, sb), "memset"); + int bad = 0, off = 0; + double worst = 0.0; // the largest error as a fraction of its bound for (int n_tok = 1; n_tok <= 8; ++n_tok) { for (int variant = 0; variant < 3; ++variant) { // apply + inject, no apply, no inject const bool apply = variant != 1, inject = variant != 2; @@ -417,8 +423,8 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { for (auto& v : R) v = edgy(rng); for (auto& v : bo) v = edgy(rng); for (auto& v : inj) v = edgy(rng); - // one set of outputs per path; R is updated in place when apply, so each path has its own copy - float* d[2][6]; // R, lo, rs, inject, mixed, xn + // path 0: one read of n_tok tokens; path 1: n_tok reads of one token. R is updated in place when apply. + float* d[2][5]; // R, lo, rs, inject, mixed for (int p = 0; p < 2; ++p) { d[p][0] = dev(R.size()); up(d[p][0], R); @@ -426,7 +432,6 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { d[p][2] = dev((size_t) (n_tok * HC)); d[p][3] = dev((size_t) (n_tok * HC)); d[p][4] = dev((size_t) (n_tok * N)); - d[p][5] = dev((size_t) (n_tok * D)); check(cudaMemset(d[p][3], 0, (size_t) (n_tok * HC) * 4), "memset"); } float *d_bo = dev(bo.size()), *d_inj = dev(inj.size()); @@ -444,25 +449,98 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { a.mixed = d[p][4] + t * N; args[p].push_back(a); } - for (int t = 0; t < n_tok; ++t) strata::kernels::fused_gr_read(args[0][(size_t) t], s); - strata::kernels::fused_gr_read_multi(args[1].data(), n_tok, d[1][5], s); + strata::kernels::fused_gr_read_multi(args[0].data(), n_tok, d_scratch, s); + for (int t = 0; t < n_tok; ++t) strata::kernels::fused_gr_read_multi(&args[1][(size_t) t], 1, d_scratch, s); check(cudaStreamSynchronize(s), "gr read"); const char* names[5] = {"R", "lo", "rs", "inject", "mixed"}; const size_t sizes[5] = {R.size(), (size_t) (n_tok * LR), (size_t) (n_tok * HC), (size_t) (n_tok * HC), (size_t) (n_tok * N)}; + std::vector got[5]; for (int o = 0; o < 5; ++o) { - const int b = bitwise_diff(down(d[0][o], sizes[o]), down(d[1][o], sizes[o]), names[o]); + got[o] = down(d[0][o], sizes[o]); + const int b = bitwise_diff(got[o], down(d[1][o], sizes[o]), names[o]); if (b) std::fprintf(stderr, "gr_read: n_tok %d variant %d: %s: %d differ\n", n_tok, variant, names[o], b); bad += b; } + // FP64 for the first token of the 1- and 4-token reads + if (n_tok == 1 || n_tok == 4) { + const float* Rt = R.data(); + // xa: the magnitude xn's rounding scales with (R' = R + gw bo may cancel) + std::vector Rp((size_t) D), Ra((size_t) D), xn((size_t) D), xa((size_t) D), rs(HC); + for (int c = 0; c < HC; ++c) { + const double gw = apply ? 2.0 / (1.0 + std::exp(-(double) inj[(size_t) c] / HC)) : 0.0; + double ss = 0.0; + for (int64_t dd = 0; dd < N; ++dd) { + const size_t i = (size_t) (c * N + dd); + Rp[i] = (double) Rt[i] + gw * bo[(size_t) dd]; + Ra[i] = std::fabs((double) Rt[i]) + std::fabs(gw * bo[(size_t) dd]); + ss += Rp[i] * Rp[i]; + } + rs[(size_t) c] = 1.0 / std::sqrt(ss / N + 1e-6); + for (int64_t dd = 0; dd < N; ++dd) { + const size_t i = (size_t) (c * N + dd); + xn[i] = Rp[i] * wn[i] * rs[(size_t) c]; + xa[i] = Ra[i] * std::fabs((double) wn[i]) * rs[(size_t) c]; + } + } + auto near = [&](double gpu, double ref, double bound, const char* what, int64_t i) { + if (!std::isfinite(ref) && std::isinf(gpu) && (gpu > 0) == (ref > 0)) return; + const double e = std::fabs(gpu - ref); + worst = std::max(worst, e / (bound + 1e-300)); + if (!(e <= bound) && off++ < 5) + std::fprintf(stderr, " gr_read FP64: n_tok %d variant %d: %s[%lld] %.9g vs %.9g (bound %.3g)\n", n_tok, + variant, what, (long long) i, gpu, ref, bound); + }; + for (int c = 0; c < HC; ++c) near(got[2][(size_t) c], rs[(size_t) c], 1e-5 * rs[(size_t) c], "rs", c); + if (apply) + for (int64_t i = 0; i < D; ++i) near(got[0][(size_t) i], Rp[(size_t) i], 1e-6 * Ra[(size_t) i] + 1e-30, "R", i); + std::vector lo((size_t) LR), lo_b((size_t) LR); + for (int64_t r = 0; r < LR + (inject ? HC : 0); ++r) { + const uint16_t* w = r < LR ? wd.data() + (size_t) (r * D) : wi.data() + (size_t) ((r - LR) * D); + double y = 0.0, mag = 0.0; + for (int64_t i = 0; i < D; ++i) { + y += bf(w[i]) * xn[(size_t) i]; + mag += std::fabs(bf(w[i])) * xa[(size_t) i]; + } + if (r < LR) { + const double x = y / HC; + lo[(size_t) r] = x / (1.0 + std::exp(-x)); + lo_b[(size_t) r] = 1.1 * 1e-4 * mag / HC + 1e-30; + near(got[1][(size_t) r], lo[(size_t) r], lo_b[(size_t) r], "lo", r); + } else { + near(got[3][(size_t) (r - LR)], y, 1e-4 * mag + 1e-30, "inject", r - LR); + } + } + for (int64_t dd = 0; dd < N; ++dd) { + double mixed = 0.0, bound = 0.0; + for (int c = 0; c < HC; ++c) { + const size_t i = (size_t) (c * N + dd); + double u = 0.0, du = 0.0; + for (int64_t k = 0; k < LR; ++k) { + const double w = bf(wu[i * LR + (size_t) k]); + u += w * lo[(size_t) k]; + du += std::fabs(w) * (1e-4 * std::fabs(lo[(size_t) k]) + lo_b[(size_t) k]); + } + const double sg = 1.0 / (1.0 + std::exp(-u)); + mixed += xn[i] * sg / HC; + bound += (1e-4 * xa[i] + 0.25 * std::fabs(xn[i]) * du) / HC; + } + near(got[4][(size_t) dd], mixed, bound + 1e-30, "mixed", dd); + } + } for (int p = 0; p < 2; ++p) for (float* q : d[p]) cudaFree(q); cudaFree(d_bo); cudaFree(d_inj); } } - for (void* p : {(void*) d_wd, (void*) d_wu, (void*) d_wi, (void*) d_wn}) cudaFree(p); - std::printf("gr_read: %s\n", bad ? "MISMATCH" : "bitwise equal to the single-token read (1-8 tokens, 3 variants)"); + for (void* p : {(void*) d_wd, (void*) d_wu, (void*) d_wi, (void*) d_wn, (void*) d_scratch}) cudaFree(p); + bad += off; + char msg[160]; + std::snprintf(msg, sizeof msg, + "each token bitwise its 1-token read (1-8 tokens, 3 variants); FP64 within bounds (largest error %.2g of its bound)", + worst); + std::printf("gr_read: %s\n", bad ? "MISMATCH" : msg); return bad; } From f14591215f01afb8c0cf188f5800aba82d5fa9fc Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 19:12:10 +0200 Subject: [PATCH 58/92] verify_router: the input's mapped copy in blocks of its own, one round of fences, the top 10 on warp reductions The first gemv blocks also copied the window's input rows to mapped memory and waited on a system fence before counting themselves done, so the copy (~6 GB/s over the 3090's x4 link) stretched the kernel. The copy now runs in blocks of its own, scheduled first, beside the gemv blocks. The last block fenced twice: every thread, then thread 0 again (~2 us each, even from one thread); now only the routing warps, which wrote the mapped ids and weights, fence before the ring's number. route_token's rounds take the largest probability and the lowest expert among equal ones with __reduce_max_sync / __reduce_min_sync on an order-preserving key, the winning lane alone rescanning its values, in place of a 16-value scan and a 5-level shuffle tree every round (the draft layer's router shares it). Bitwise: the logits, ids, weights, mapped copies and plan are verify_router's (random and tied logits); window hashes identical on both models. scratch/router2_bench.cu: 17.6 -> 15.6 us at 1 token, 24.1 -> 21.6 at 4, 35.0 -> 32.4 at 8; in the window 22-30 us a layer, from 26-42. Fixed text, 2 rounds: IQ3_XXS code +2.5%, thinking +0.9%, after 16K +2.7%; UD-Q4_K_XL within noise (the CPU is the later side there). --- src/kernels/cuda/native_router.cu | 89 +++++++++++++++++-------------- 1 file changed, 50 insertions(+), 39 deletions(-) diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index c48928207e6..ac94e97880f 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -44,8 +44,17 @@ __device__ __forceinline__ float warp_max(float value) { for (int mask = 16; mask; mask >>= 1) value = fmaxf(value, __shfl_xor_sync(0xffffffffu, value, mask, 32)); return value; } +// A float as a uint32 in the same order (sign flipped for positives, all bits for negatives), and back. +__device__ __forceinline__ unsigned order_key(float f) { + const unsigned b = __float_as_uint(f); + return (b & 0x80000000u) ? ~b : (b | 0x80000000u); +} +__device__ __forceinline__ float order_key_float(unsigned k) { + return __uint_as_float((k & 0x80000000u) ? (k & 0x7fffffffu) : ~k); +} // One token, one warp: 512 logits -> the top 10 ids and weights; ids2 and weights2 take copies when given. The logits -// load through the L2 (another block may have written them). +// load through the L2 (another block may have written them). Each round takes the largest probability and, among +// equal ones, the lowest expert (a lane's own best, then two warp reductions); only the winning lane rescans. __device__ __forceinline__ void route_token(const float* __restrict__ logits, int32_t* __restrict__ ids, float* __restrict__ weights, int32_t* ids2, float* weights2, int lane) { float values[16]; @@ -67,22 +76,27 @@ __device__ __forceinline__ void route_token(const float* __restrict__ logits, in values[i] *= reciprocal; if (__isnanf(values[i])) values[i] = -FLT_MAX; } + float lbest = values[0]; // the lane's largest value, the lowest index among equal ones + int li = 0; +#pragma unroll + for (int i = 1; i < 16; ++i) + if (values[i] > lbest) { lbest = values[i]; li = i; } float selected = 0.0f, selected_sum = 0.0f; +#pragma unroll 1 for (int rank = 0; rank < 10; ++rank) { - float best = values[0]; - int expert = lane; + const unsigned mine = order_key(lbest); + const unsigned top = __reduce_max_sync(0xffffffffu, mine); + const int expert = __reduce_min_sync(0xffffffffu, mine == top ? lane + li * 32 : 0x7fffffff); + const float best = order_key_float(top); + if ((expert & 31) == lane) { #pragma unroll - for (int i = 1; i < 16; ++i) { - if (values[i] > best) { best = values[i]; expert = lane + i * 32; } - } + for (int i = 0; i < 16; ++i) + if (i == li) values[i] = -INFINITY; + lbest = values[0]; + li = 0; #pragma unroll - for (int mask = 16; mask; mask >>= 1) { - const float other = __shfl_xor_sync(0xffffffffu, best, mask, 32); - const int other_id = __shfl_xor_sync(0xffffffffu, expert, mask, 32); - if (other > best || (other == best && other_id < expert)) { best = other; expert = other_id; } - } - if ((expert & 31) == lane) { - values[expert / 32] = -INFINITY; + for (int i = 1; i < 16; ++i) + if (values[i] > lbest) { lbest = values[i]; li = i; } ids[rank] = expert; if (ids2 != nullptr) ids2[rank] = expert; // Deliberately accumulate by WINNING EXPERT lane, not output rank. @@ -110,42 +124,40 @@ __global__ void route(const float* __restrict__ logits, int32_t* __restrict__ id // ================================ plan v0.3 P6: the verify window's router in one kernel ================================ // -// For T tokens: a block a row of the logits (bf16_gemv_fp32_mmvf_multi's 256-thread block, mmvf_multi_row), the first -// blocks also copying the input rows to mapped memory, a 16-byte store (a full PCIe transaction) a thread; the last -// block to finish routes the tokens (`route`, a warp each), publishes their ids and weights, writes the ring's number -// and plans the main GPU's hits (hit_plan_block). In place of the gemv, the top 10, doorbell_publish and -// verify_hit_plan. +// For T tokens: when the input rows have a mapped copy, the first blocks (scheduled first) only copy them, a 16-byte +// store (a full PCIe transaction) a thread, and fence; then a block a row of the logits (bf16_gemv_fp32_mmvf_multi's +// 256-thread block, mmvf_multi_row). The last block to finish routes the tokens (`route_token`, a warp each); the +// routing warps, which wrote the mapped ids and weights, fence; then the ring's number, and the plan of the main +// GPU's hits (hit_plan_block). In place of the gemv, the top 10, doorbell_publish and verify_hit_plan. constexpr int VR_THREADS = 256; // mmvf_block_size of n_embd a multiple of 512 template -__global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterArgs a) { +__global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterArgs a, int n_copy) { __shared__ bool s_last; - const int t = (int) threadIdx.x, lane = t & 31, warp = t >> 5; - bool mapped = false; - if (a.x_out != nullptr) { - const int i = (int) blockIdx.x * VR_THREADS + t; - if (i < TT * a.n_embd / 4) { + const int t = (int) threadIdx.x, lane = t & 31, warp = t >> 5, b = (int) blockIdx.x; + if (b < n_copy) { + for (int i = b * VR_THREADS + t; i < TT * a.n_embd / 4; i += n_copy * VR_THREADS) reinterpret_cast(a.x_out)[i] = reinterpret_cast(a.x)[i]; - mapped = true; - } + __threadfence_system(); // this thread's writes before the block counts as done + } else { + const int row = b - n_copy; + float acc[kMmvfMaxRows]; + mmvf_multi_row(a.x, a.w + (size_t) row * a.n_embd, a.n_embd, TT, acc); + if (t == 0) + for (int k = 0; k < TT; ++k) a.logits[(size_t) k * a.n_expert + row] = acc[k]; + __threadfence(); } - float acc[kMmvfMaxRows]; - mmvf_multi_row(a.x, a.w + (size_t) blockIdx.x * a.n_embd, a.n_embd, TT, acc); - if (t == 0) - for (int k = 0; k < TT; ++k) a.logits[(size_t) k * a.n_expert + blockIdx.x] = acc[k]; - if (mapped) __threadfence_system(); // this thread's writes before the block counts as done - else __threadfence(); __syncthreads(); if (t == 0) s_last = atomicAdd(a.counter, 1u) == gridDim.x - 1; __syncthreads(); if (!s_last) return; - if (warp < TT) + if (warp < TT) { route_token(a.logits + (size_t) warp * a.n_expert, a.ids + warp * 10, a.weights + warp * 10, a.ids_out != nullptr ? a.ids_out + warp * 10 : nullptr, a.w_out != nullptr ? a.w_out + warp * 10 : nullptr, lane); - __threadfence_system(); - __syncthreads(); - if (t == 0) { __threadfence_system(); + } + __syncthreads(); + if (t == 0) { // warp 0 routed and fenced *(volatile uint32_t*) a.seq = a.ring; *a.counter = 0u; // for the next launch } @@ -177,10 +189,9 @@ void verify_router(const VerifyRouterArgs& a, void* stream) { !a.x || !a.w || !a.logits || !a.ids || !a.weights || !a.seq || !a.counter || (a.plan && (!a.res || !a.slot_ptr))) throw std::invalid_argument("verify_router: 1..8 tokens, 512 experts, n_embd a multiple of 512, and buffers"); const auto st = static_cast(stream); - if (a.x_out != nullptr && a.n_tok * a.n_embd / 4 > a.n_expert * VR_THREADS) - throw std::invalid_argument("verify_router: the blocks copy the input rows a float4 a thread"); + const int n_copy = a.x_out != nullptr ? (a.n_tok * a.n_embd / 4 + VR_THREADS - 1) / VR_THREADS : 0; switch (a.n_tok) { -#define STRATA_VR(T) case T: verify_router_kernel<<>>(a); break; +#define STRATA_VR(T) case T: verify_router_kernel<<>>(a, n_copy); break; STRATA_VR(1) STRATA_VR(2) STRATA_VR(3) STRATA_VR(4) STRATA_VR(5) STRATA_VR(6) STRATA_VR(7) STRATA_VR(8) #undef STRATA_VR } From df5166e095101908a3a954c2082f690abfe72f15 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Sun, 27 Sep 2026 19:55:18 +0200 Subject: [PATCH 59/92] fused_gr, native_router, xor_scatter: lines within 120 columns --- src/kernels/cuda/fused_gr.cu | 12 +++++++----- src/kernels/cuda/native_router.cu | 3 ++- src/kernels/cuda/xor_scatter.cuh | 3 ++- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu index 99981798d97..70f323ea56b 100644 --- a/src/kernels/cuda/fused_gr.cu +++ b/src/kernels/cuda/fused_gr.cu @@ -78,10 +78,11 @@ __device__ __forceinline__ void reduce_chunks(const float (&v)[VT], int lane, F& // The down projection and the injection of TT tokens. Block (s, g): warp w takes rows g DRB + w DR + [0, DR) of // [w_down; w_inject] over the slice's DCS columns, a lane 8 columns a step, each activation chunk loaded once for its // DR rows. The slice's R' = R + bo_prev * 2 sigmoid(inj_prev / hc) times w_norm is staged in shared memory split into -// the halves of the 8-column chunks (conflict-free 16-byte reads), the sums of squares of R' go to the scratch; the stream's 1/rms is applied to the slice sums later, since a slice lies -// in one stream. The group's last block (a counter) adds each row's partials in slice order, per stream times the -// stream's 1/rms, and writes lo = silu(y / hc) or the injection; group 0's also writes rs. Every token's arithmetic -// is the same whatever its place in the window and the window's size. +// the halves of the 8-column chunks (conflict-free 16-byte reads), and the sums of squares of R' go to the scratch: +// a slice lies in one stream, so the stream's 1/rms multiplies the slice sums later. The group's last block (a +// counter) adds each row's partials in slice order, per stream times the stream's 1/rms, and writes lo = silu(y / hc) +// or the injection; group 0's also writes rs. Every token's arithmetic is the same whatever its place in the window +// and the window's size. template __global__ void __launch_bounds__(DTH) gr_down_split_kernel(GrMulti m, float* __restrict__ scratch) { static_assert(TT >= 1 && TT <= DT, "tokens a launch"); @@ -113,7 +114,8 @@ __global__ void __launch_bounds__(DTH) gr_down_split_kernel(GrMulti m, float* __ for (int i = 0; i < DR; ++i) { const int rb = rb0 + i; const bool ok = rb < LR || has_inj; - const uint16_t* base = rb < LR ? m.a[0].w_down + (size_t) rb * D : m.a[0].w_inject + (size_t) (ok ? rb - LR : 0) * D; + const uint16_t* base = + rb < LR ? m.a[0].w_down + (size_t) rb * D : m.a[0].w_inject + (size_t) (ok ? rb - LR : 0) * D; const uint4* w4 = reinterpret_cast(base + col0) + lane; #pragma unroll for (int j = 0; j < DSTEPS; ++j) w[i][j] = ok ? __ldg(w4 + j * 32) : make_uint4(0, 0, 0, 0); diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index ac94e97880f..9e104d038dd 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -191,7 +191,8 @@ void verify_router(const VerifyRouterArgs& a, void* stream) { const auto st = static_cast(stream); const int n_copy = a.x_out != nullptr ? (a.n_tok * a.n_embd / 4 + VR_THREADS - 1) / VR_THREADS : 0; switch (a.n_tok) { -#define STRATA_VR(T) case T: verify_router_kernel<<>>(a, n_copy); break; +#define STRATA_VR(T) \ + case T: verify_router_kernel<<>>(a, n_copy); break; STRATA_VR(1) STRATA_VR(2) STRATA_VR(3) STRATA_VR(4) STRATA_VR(5) STRATA_VR(6) STRATA_VR(7) STRATA_VR(8) #undef STRATA_VR } diff --git a/src/kernels/cuda/xor_scatter.cuh b/src/kernels/cuda/xor_scatter.cuh index 9a6374878fa..9db9496b80c 100644 --- a/src/kernels/cuda/xor_scatter.cuh +++ b/src/kernels/cuda/xor_scatter.cuh @@ -15,7 +15,8 @@ namespace { // puts the array in local memory __device__ __forceinline__ float select_f(bool c, float a, float b) { float r; - asm("{ .reg .pred p; setp.ne.u32 p, %3, 0; selp.f32 %0, %1, %2, p; }" : "=f"(r) : "f"(a), "f"(b), "r"((unsigned) c)); + asm("{ .reg .pred p; setp.ne.u32 p, %3, 0; selp.f32 %0, %1, %2, p; }" + : "=f"(r) : "f"(a), "f"(b), "r"((unsigned) c)); return r; } From 2130de245ba95240db1daf09f239073f4975d39f Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 00:05:35 +0200 Subject: [PATCH 60/92] Second GPU: the adaptive tier's copies between windows The second GPU's adaptive tier moved up to four experts per layer, submitted after each layer's prefetch copies. On that GPU's one copy engine they started behind the prefetch and ran inside the next layer's CPU pool (UD-Q4_K_XL code: four took ~0.4 ms and ended ~0.58 ms after the next ring), where they took the pool's RAM. Between windows (the last layer's CPU rows to the next window's first pool: the head, the commit, the drafts) the RAM stayed idle for ~3.5 ms. Now its copies go there. The verify window calls a tail hook once its last CPU rows are in; the adaptive update starts on its thread from there, and the second tier's copies go out as far as they end before the next window's first pool: the gap's running average at the rate of the tier's own copies (timed with events, ~47 GB/s beside the first tier's x4 copies). The last tail before the next update sends the rest; an update drops what is still queued (nothing in these runs). The first tier's copies wait for the window's graph (Verifier::window_done), which may still read its cache when the update starts. The tier is paced whenever the second GPU's is adaptive. Window hashes identical with --adapt-every 0 (no tiers). Fixed text, both GPUs, 2 rounds against the previous commit: UD-Q4_K_XL code +5.5% (the pool 19.5 -> 17.7 ms a round, 43 -> 46 GB/s; the tier's moves unchanged at 42 a round), thinking +1.3%, after 16K +3.0%; IQ3_XXS code +1.3%, thinking 0.0%, after 16K +0.3%. Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- include/strata/core/adaptive_tier.hpp | 20 +++-- include/strata/core/verify.hpp | 10 +++ src/core/adaptive_tier.cpp | 57 +++++++++++- src/core/verify.cpp | 11 ++- src/program/generate.cpp | 119 ++++++++++++++++++++------ 5 files changed, 178 insertions(+), 39 deletions(-) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp index aa731985035..c69b28b91c2 100644 --- a/include/strata/core/adaptive_tier.hpp +++ b/include/strata/core/adaptive_tier.hpp @@ -38,20 +38,26 @@ class AdaptiveTier { /// A tier in front of this one: the experts it holds or is loading are not candidates here, and this tier's /// copies of them are evicted first. void set_upper(const AdaptiveTier* upper) { upper_ = upper; } + /// The copies wait for `ev` (on the cache's GPU): the last work that may read the slots they overwrite. + void set_after(cudaEvent_t ev) { after_ = ev; } /// Ranks and submits this call's moves, then decays `usage` (n_layers x n_expert) unless `decay` is false (a /// tier ranked before another on the same counts). Nothing to do while the previous moves are in flight. /// False when a copy could not be submitted. bool adapt(std::vector& usage, std::string& err, bool decay = true); - /// Paced: adapt() only queues its moves, and each pump() submits up to `per_pump` of them (a copy queue that - /// never runs far ahead, so copies submitted after a pump do not wait behind a whole update). - void set_pace(int per_pump) { pace_ = per_pump; } - bool pump(std::string& err); + /// Paced: adapt() only queues its moves, and `pump` submits them in order while they fit `budget` bytes, `sent` + /// the bytes submitted: the copies go where the RAM and the copy engine are idle. The next adapt() drops the + /// moves not submitted by then and admits the landed ones before it ranks. `copy_rate()`: a running average of + /// the batches' measured rates (bytes per ms), 0 until the first has landed. + bool set_paced(); + bool pump(uint64_t budget, uint64_t& sent, std::string& err); + double copy_rate() const { return rate_; } /// Admits the landed moves into `host_res` (and `d_res`) once all are submitted; `wait` submits the queued ones /// and blocks until they have landed. void apply_pending(bool wait); int64_t swaps = 0, fills = 0; ///< experts moved into an occupied / an empty slot + int64_t dropped = 0; ///< paced: moves the next update dropped before their copies went out double ms = 0; ///< host time in `adapt` private: @@ -69,7 +75,11 @@ class AdaptiveTier { struct Move { int32_t layer, in, out, slot; }; // out: the evicted expert, < 0 for an empty slot std::vector queued_; // this update's moves; [next_, end) not submitted yet size_t next_ = 0; - int pace_ = 0; + bool paced_ = false; + cudaEvent_t after_ = nullptr; + cudaEvent_t t0_ = nullptr, t1_ = nullptr; // paced: around the last batch, whose `batch_` bytes are not timed yet + uint64_t batch_ = 0; + double rate_ = 0; std::string failed_; // a copy apply_pending could not submit bool submit(const Move& m, std::string& err); bool pump_n(size_t n, std::string& err); diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 035f0c3cce3..96dc4d2f437 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -47,6 +47,8 @@ using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, i /// Layer `layer`'s likely experts, its router applied to the previous layer's FFN input: ids and weights (n_tok, k), /// host memory. using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); +/// The window's last CPU rows are in. +using TailFn = void (*)(void* user); /// The main GPU's VRAM expert tier. struct VerifyHits { @@ -118,6 +120,11 @@ class Verifier { /// rows of a layer are in, `fn` gets the next layer's prediction (the RAM is idle until the next ring), waited /// for when it comes later (`predict_late`). Windows of one token group. Set before the first `run`. void set_predict(PredictFn fn, void* user) { predict_ = fn; predict_user_ = user; } + /// `fn` runs once the window's last CPU rows are in: the RAM stays idle until the next window's first pool (the + /// head, the commit, the drafts). The main GPU may still run the window then: `window_done` follows its graph. + /// Set before the first `run`. + void set_tail(TailFn fn, void* user) { tail_ = fn; tail_user_ = user; } + cudaEvent_t window_done() const { return done_; } double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0, ms_predict = 0; int64_t windows = 0, predict_late = 0; @@ -182,6 +189,8 @@ class Verifier { // taken it; the counter holds the predictions published in this window PredictFn predict_ = nullptr; void* predict_user_ = nullptr; + TailFn tail_ = nullptr; + void* tail_user_ = nullptr; int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k float* h_pw_ = nullptr; float* m_pw_ = nullptr; // 2 x T * k uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; @@ -194,6 +203,7 @@ class Verifier { cudaEvent_t pfork_ = nullptr, pjoin_ = nullptr; // shs_'s projections cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; + cudaEvent_t done_ = nullptr; // after the last window's graph // device void* arena_ = nullptr; diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 01c55f3e87d..26ec747e94f 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -19,10 +19,17 @@ struct OnDevice { // the tier's GPU current for one call (when it has one), th AdaptiveTier::~AdaptiveTier() { OnDevice on(dev_, main_); if (stream_) cudaStreamSynchronize(stream_); - if (ev_) cudaEventDestroy(ev_); + cudaEvent_t evs[] = {ev_, t0_, t1_}; + for (cudaEvent_t e : evs) if (e) cudaEventDestroy(e); if (stream_) cudaStreamDestroy(stream_); } +bool AdaptiveTier::set_paced() { + OnDevice on(dev_, main_); + paced_ = cudaEventCreate(&t0_) == cudaSuccess && cudaEventCreate(&t1_) == cudaSuccess; + return paced_; +} + bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, int64_t n_layers, int64_t n_expert, int max_moves, std::string& err, int device, int main_device) { cache_ = &cache; @@ -66,17 +73,61 @@ bool AdaptiveTier::submit(const Move& m, std::string& err) { bool AdaptiveTier::pump_n(size_t n, std::string& err) { if (next_ >= queued_.size()) return true; OnDevice on(dev_, main_); + if (after_ != nullptr && cudaStreamWaitEvent(stream_, after_, 0) != cudaSuccess) { + err = "adaptive tier: a refill copy failed"; + return false; + } for (size_t i = 0; i < n && next_ < queued_.size(); ++i) if (!submit(queued_[next_++], err)) return false; if (cudaEventRecord(ev_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } return true; } -bool AdaptiveTier::pump(std::string& err) { return pump_n((size_t) pace_, err); } +bool AdaptiveTier::pump(uint64_t budget, uint64_t& sent, std::string& err) { + sent = 0; + size_t n = 0; + for (; next_ + n < queued_.size(); ++n) { + const uint64_t b = strata::kernels::cpu::expert_layout().blob_bytes(queued_[next_ + n].layer); + if (sent + b > budget) break; + sent += b; + } + if (n == 0) return true; + OnDevice on(dev_, main_); + float took = 0; + if (batch_ > 0 && cudaEventQuery(t1_) == cudaSuccess && cudaEventElapsedTime(&took, t0_, t1_) == cudaSuccess && + took > 0) { + const double r = (double) batch_ / (double) took; // the last batch has landed + rate_ = rate_ > 0 ? 0.8 * rate_ + 0.2 * r : r; + batch_ = 0; + } + const bool timed = batch_ == 0 && n >= 4; // a batch of a few copies: its start and end dominate + if (timed && cudaEventRecord(t0_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + if (!pump_n(n, err)) return false; + if (timed) { + if (cudaEventRecord(t1_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + batch_ = sent; + } + return true; +} bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay) { const auto t0 = std::chrono::steady_clock::now(); if (!failed_.empty()) { err = failed_; return false; } + if (paced_) { // the moves not submitted by now give way to this call's (ranked again if still worth it) + for (size_t i = next_; i < queued_.size(); ++i) { + const Move& m = queued_[i]; + if (m.out >= 0) { + --swaps; + } else { + free_[(size_t) m.layer].push_back(m.slot); + --fills; + } + } + dropped += (int64_t) (queued_.size() - next_); + queued_.resize(next_); + pending_.resize(next_); + apply_pending(false); + } if (!pending_.empty()) return true; // the previous moves are still in flight if (upper_ != nullptr) { upper_has_.assign((size_t) (n_layers_ * n_expert_), 0); @@ -128,7 +179,7 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay queued_.push_back({m.layer, m.in, m.out, m.slot}); pending_.emplace_back((int32_t) (m.layer * n_expert_ + m.in), m.slot); // resident once the copy has landed } - if (pace_ <= 0 && !pump_n(queued_.size(), err)) return false; + if (!paced_ && !pump_n(queued_.size(), err)) return false; if (decay) for (float& v : usage) v *= 0.7f; ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 1247e480191..def7350724c 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -100,7 +100,7 @@ Verifier::~Verifier() { if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (side_) cudaStreamDestroy(side_); if (shs_) cudaStreamDestroy(shs_); - for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_}) + for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); @@ -285,7 +285,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& cudaEventCreateWithFlags(&pjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shfork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&shjoin_, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&res_ready_, cudaEventDisableTiming) != cudaSuccess) { + cudaEventCreateWithFlags(&res_ready_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&done_, cudaEventDisableTiming) != cudaSuccess) { err = "verify: stream create failed"; return false; } @@ -1002,7 +1003,10 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, ms_host += ms_since(t0); VDBG("staged; launching\n"); const cudaError_t le = cudaGraphLaunch(exec_[T], cs_); - if (le != cudaSuccess) { err = std::string("verify: launch: ") + cudaGetErrorString(le); return false; } + if (le != cudaSuccess || cudaEventRecord(done_, cs_) != cudaSuccess) { + err = std::string("verify: launch: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } (void) cudaStreamQuery(cs_); VDBG("launched\n"); if (ple_on) { // before layer 1: the graph waits for the flag, raised even when a read failed @@ -1064,6 +1068,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *flag = want; ms_wait += std::chrono::duration(b - a).count(); ms_pool += ms_since(b); + if (tail_ != nullptr && k + 1 == g.n_layers * G) tail_(tail_user_); // the next layer's prediction, published during this layer's pool, else microseconds later (its branch is // joined before the next layer): waited for, so the experts the second GPU takes never depend on timing if (predict_ != nullptr && G == 1 && l + 1 < g.n_layers && native_router_enabled()) { diff --git a/src/program/generate.cpp b/src/program/generate.cpp index b9f43b80dbd..00a3ceecb00 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -447,9 +447,32 @@ struct Drive { /// needs no locking. `d.layers` is the CURRENT layer on entry (the adapter increments it as it walks the /// blob), which is why the layer index comes from there rather than from a counter of our own. std::FILE* routing = nullptr; - /// The second GPU's adaptive tier when its moves are paced behind the prefetch copies. + /// The second GPU's adaptive tier, paced: its copies go between windows, where the RAM is idle (the head, the + /// commit, the drafts); through the layers they slowed the CPU pool's RAM reads, and on that GPU's one copy engine + /// they held up the prefetch copies. An update is ranked at a window's tail, and its copies go out as far as they + /// end before the next window's first pool (`gap_ms`, a running average of that gap, at the tier's measured copy + /// rate), the rest at the last tail before the next update. strata::core::AdaptiveTier* tier2 = nullptr; + double gap_ms = 0; + Clock::time_point tail_at{}, copies_end{}; // the last window's CPU rows done; the gap's queued copies end + bool gap_open = false; // no pool since the tail + bool adapt_next = false; // an update is ranked at this window's tail + bool flush_next = false; // the next window's tail ranks one: this gap takes the rest std::string tier_err; + /// The adaptive tiers' update (ranking, copy submission) on a thread of its own, joined before the next window: + /// started at the window's tail when the second tier is paced, else after the window. + std::function adapt; + std::thread adapt_thr; + bool adapt_ok = true; + void start_adapt() { + adapt_ok = true; + adapt_thr = std::thread([this] { adapt_ok = adapt(); }); + } + bool join_adapt() { + if (adapt_thr.joinable()) adapt_thr.join(); + return adapt_ok; + } + ~Drive() { join_adapt(); } }; void drive_pool(void* user, const float* x_f, const int32_t* ids, const float* weights, int64_t n_embd, int64_t k, @@ -485,8 +508,34 @@ void drive_pool(void* user, const float* x_f, const int32_t* ids, const float* w void drive_predict(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k) { Drive* t = (Drive*) user; strata::core::expert_prefetch_multi(t->d, layer, ids, w, n_tok, k); - // a few of the second tier's moves, queued behind the prefetch rather than in front of the next ones - if (t->tier2 != nullptr && !t->d.failed && !t->tier2->pump(t->tier_err)) { +} + +/// The second tier's queued moves that end before the next window's first pool, or all of them. +bool pump_gap(Drive* t, std::string& err, bool all = false) { + if (t->tier2 == nullptr) return true; + const Clock::time_point start = std::max(Clock::now(), t->copies_end); + const double left = t->gap_ms - std::chrono::duration(start - t->tail_at).count(); + const double rate = t->tier2->copy_rate(); + const bool paced = !all && rate > 0 && t->gap_ms > 0; + if (paced && left <= 0) return true; + uint64_t sent = 0; + if (!t->tier2->pump(paced ? (uint64_t) (left * rate) : UINT64_MAX, sent, err)) return false; + if (rate > 0) + t->copies_end = start + std::chrono::duration_cast( + std::chrono::duration((double) sent / rate)); + return true; +} + +/// A window's last CPU rows are in (Verifier::set_tail). +void drive_tail(void* user) { + Drive* t = (Drive*) user; + t->tail_at = Clock::now(); + t->copies_end = t->tail_at; + t->gap_open = true; + if (t->d.failed) return; + if (t->adapt_next) { + t->start_adapt(); + } else if (!pump_gap(t, t->tier_err, t->flush_next)) { t->d.failed = true; t->d.fail = t->tier_err.c_str(); } @@ -498,6 +547,11 @@ void drive_pool_multi(void* user, const float* x_f, const int32_t* ids, int64_t Drive* t = (Drive*) user; t->d.layers = layer; const Clock::time_point a = Clock::now(); + if (t->gap_open) { // the first pool after a window: the RAM's idle time between them + t->gap_open = false; + const double gap = std::chrono::duration(a - t->tail_at).count(); + if (gap < 20.0) t->gap_ms = t->gap_ms > 0 ? 0.8 * t->gap_ms + 0.2 * gap : gap; + } strata::core::expert_pool_dispatch_multi(t->d, x_f, ids, n_tok, k, out); t->cpu_ms += std::chrono::duration(Clock::now() - a).count(); ++t->calls; @@ -2136,12 +2190,7 @@ int main(int argc, char** argv) { // lowest slots first: the last ones, which the prompt path borrows, stay empty longest for (size_t i = empty2.size(); i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); tier2.set_upper(&tier); - // with the prefetch, the tier's copies go four per layer behind it, an update within ~24 layers: - // queued in front of the prefetch copies, an update's ~300 MB would hold them up for ~7 ms - if (gpu2.prefetch_slots() > 0) { - tier2.set_pace(4); - drive.tier2 = &tier2; - } + if (tier2.set_paced()) drive.tier2 = &tier2; // its copies between windows (Drive::tier2) } } if (ok) { @@ -2161,8 +2210,10 @@ int main(int argc, char** argv) { std::string adapt_err; auto adapt = [&]() -> bool { const bool two = tier2.on(); - return tier.adapt(drive.d.usage, adapt_err, !two) && (!two || tier2.adapt(drive.d.usage, adapt_err)); + return tier.adapt(drive.d.usage, adapt_err, !two) && + (!two || (tier2.adapt(drive.d.usage, adapt_err) && pump_gap(&drive, adapt_err))); }; + drive.adapt = adapt; auto apply_pending = [&](bool wait) { tier.apply_pending(wait); tier2.apply_pending(wait); @@ -2286,6 +2337,10 @@ int main(int argc, char** argv) { ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache + ver.set_tail(&drive_tail, &drive); + tier.set_after(ver.window_done()); + } prefill.draft = &mtp; drive.d.plan = ver.plan_sink(); drive.d.host_res = ver.residency(); // the snapshot the windows decide their GPU hits by @@ -2537,6 +2592,7 @@ int main(int argc, char** argv) { drive.d.layers = 0; drive.d.experts = 0; drive.d.failed = false; + drive.adapt_next = drive.flush_next = false; apply_pending(false); if (!ver.run(T, window.data(), j, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) { if (drive.d.failed && drive.d.fail) err = drive.d.fail; @@ -2595,18 +2651,19 @@ int main(int argc, char** argv) { drive.d.failed = false; apply_pending(false); tr("window", p, T); + drive.adapt_next = !drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0; + drive.flush_next = !drive.d.usage.empty() && ((rounds + 2) % o.adapt_every) == 0; if (!ver.run(T, window.data(), p, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) { + drive.join_adapt(); std::printf("ERR %s\n", drive.d.failed && drive.d.fail ? drive.d.fail : err.c_str()); return 1; } int a = 0; while (a < T - 1 && window[(size_t) a + 1] == outv[(size_t) a]) ++a; - std::thread adapt_thr; // the adaptive tier beside the commit and the draft (as in generate) - bool adapt_ok = true; - if (!drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0) - adapt_thr = std::thread([&] { adapt_ok = adapt(); }); + // the adaptive tier beside the commit and the draft (as in generate) + if (drive.adapt_next && !drive.adapt_thr.joinable()) drive.start_adapt(); if (!ver.commit(a + 1, err, false)) { // beside the draft - if (adapt_thr.joinable()) adapt_thr.join(); + drive.join_adapt(); std::printf("ERR %s\n", err.c_str()); return 1; } @@ -2624,8 +2681,7 @@ int main(int argc, char** argv) { const bool drafted = (eos || produced_n >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && ver.wait_commit(err); - if (adapt_thr.joinable()) adapt_thr.join(); - if (!adapt_ok) { + if (!drive.join_adapt()) { std::printf("ERR an adaptive refill failed\n"); return 1; } @@ -2992,6 +3048,10 @@ int main(int argc, char** argv) { ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache + ver.set_tail(&drive_tail, &drive); + tier.set_after(ver.window_done()); + } drive.d.plan = ver.plan_sink(); drive.d.host_res = ver.residency(); // the snapshot the windows decide their GPU hits by drive.d.pcie_num = (int) (o.pcie_frac * 256.0 + 0.5); @@ -3075,17 +3135,22 @@ int main(int argc, char** argv) { drive.d.experts = 0; drive.d.failed = false; apply_pending(false); + drive.adapt_next = !drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0; + drive.flush_next = !drive.d.usage.empty() && ((rounds + 2) % o.adapt_every) == 0; if (!ver.run(T, window.data(), p, &drive_pool_multi, &drive, outv.data(), err)) { + drive.join_adapt(); std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } if (drive.d.failed) { + drive.join_adapt(); std::fprintf(stderr, "strata generate: the expert pool failed at layer %lld expert %lld: %s\n", (long long) drive.d.fail_layer, (long long) drive.d.fail_expert, drive.d.fail ? drive.d.fail : "(no message)"); return 1; } if (wlog != nullptr && !ver.copy_logits(T, wl.data(), err)) { + drive.join_adapt(); std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -3113,12 +3178,9 @@ int main(int argc, char** argv) { } // plan v0.3 P6: the adaptive tier's host work (ranking, copy submission) runs on its own thread while the // GPU commits and drafts; it touches only the residency tables, which nothing reads until the next window - std::thread adapt_thr; - bool adapt_ok = true; - if (!drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0) - adapt_thr = std::thread([&] { adapt_ok = adapt(); }); + if (drive.adapt_next && !drive.adapt_thr.joinable()) drive.start_adapt(); if (!ver.commit(a + 1, err, false)) { // beside the draft - if (adapt_thr.joinable()) adapt_thr.join(); + drive.join_adapt(); std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -3136,7 +3198,7 @@ int main(int argc, char** argv) { if (!follow.empty()) { follow_differ += fdiff[(size_t) i]; ++follow_emitted; } } if (eos) { - if (adapt_thr.joinable()) adapt_thr.join(); + drive.join_adapt(); if (!ver.wait_commit(err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; @@ -3147,8 +3209,7 @@ int main(int argc, char** argv) { const bool drafted = (!use_mtp || (int64_t) produced.size() >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && ver.wait_commit(err); - if (adapt_thr.joinable()) adapt_thr.join(); - if (!adapt_ok) { + if (!drive.join_adapt()) { std::fprintf(stderr, "strata generate: %s\n", adapt_err.c_str()); return 1; } @@ -3204,12 +3265,14 @@ int main(int argc, char** argv) { if (rounds > 0 && drive.d.gpu2 != nullptr) std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " "%.3f ms/round waiting for it after the CPU pool; %lld experts swapped in and %lld into empty " - "slots (%lld still empty); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round " - "of host time, %lld predictions waited for)\n", "second GPU", + "slots (%lld still empty, %lld moves dropped; its copies between windows, %.2f ms, at %.1f " + "GB/s); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round of host time, %lld " + "predictions waited for)\n", "second GPU", (double) gpu2.entries_done / (double) rounds, (double) gpu2.experts / (double) rounds, (double) gpu2.layers / (double) rounds, (double) drive.d.gpu2_skipped / (double) rounds, gpu2.ms_wait / (double) rounds, (long long) tier2.swaps, (long long) tier2.fills, - (long long) tier2.free_slots(), (double) gpu2.prefetch_copied / (double) rounds, + (long long) tier2.free_slots(), (long long) tier2.dropped, drive.gap_ms, + tier2.copy_rate() / 1e6, (double) gpu2.prefetch_copied / (double) rounds, (double) gpu2.prefetch_used / (double) rounds, ver.ms_predict / (double) rounds, (long long) ver.predict_late); if (rounds > 0 && !drive.d.routed.empty()) { From 45678753534914a4e0f6d03ab54d7a25d2e73acc Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 00:11:06 +0200 Subject: [PATCH 61/92] DETAILS.md: the speeds with the second GPU's adaptive tier copying between windows Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- docs/DETAILS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 5d4adefe1f8..147436fd555 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1819-2063 tok/s instead of 992-1130, generation 80-100 tok/s instead of 58-65; -IQ3_XXS 1959-2184 instead of 1516-1699, generation 102-126 instead of 97-111. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1852-2075 tok/s instead of 992-1130, generation 89-104 tok/s instead of 58-65; +IQ3_XXS 1986-2190 instead of 1516-1699, generation 112-139 instead of 97-111. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From 3ab7c61a7e4135e9e2f56416ac0ece66e09137c3 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 03:53:28 +0200 Subject: [PATCH 62/92] Second GPU: the prefetch predicted from an estimate of the next layer's FFN input Each layer's router also ran on the previous layer's FFN input to pick the experts copied ahead to the second GPU: 52% of them were used on UD-Q4_K_XL code, 34% on thinking; IQ3_XXS 44% / 18%. With hyper-connections each layer's FFN input is its own gated mix of the four residual streams (its own norm weights and gates), which that misses. Now each layer's FFN read also writes the next layer's estimated FFN input: that layer's read of this state, with the read's gates (sigmoids of two projections) replaced by their running average over the windows, computed in the up kernel's epilogue, which holds the updated state and its streams' norms (FusedGrArgs::est). The prediction branch runs the next layer's router on it, each expert's logit corrected by a running average of its error (VerifyRouterArgs::bias, verify_router_bias_update, after the prediction; the predicted logits in two buffers by layer parity). The FFN reads keep the gates' average (FusedGrArgs::gate_ema). Both learn only while the tiers adapt: with --adapt-every 0 the gates stay at one half and the bias at zero, so the prediction depends on the window alone and a regenerated turn reproduces its output. Share of the prefetched experts used, fixed text: UD-Q4_K_XL code 71%, thinking 57%, after 16K 69%; IQ3_XXS 61%, 34%, 53%. Both GPUs, 3 rounds against the previous commit, ms per round (the drafts accepted differ between builds even on fixed text; t/s in brackets): UD-Q4_K_XL code -4.0% (+5.0%; the CPU's experts 4.81 -> 4.25 a layer), thinking -2.9% (+2.8%), after 16K -3.4% (+2.3%); IQ3_XXS code -4.7% (+5.6%), thinking -0.3% (+0.3%), after 16K -1.5% (+1.3%). The next layer's full read on the prediction branch predicted about as well (offline 67% against the estimate's 70% with the bias, UD-Q4_K_XL code), but its 3090 time cost more than it gained where the 3090 is the later side. 12 MiB stays: 16 was 0.2-4% slower per round. Window hashes: identical between runs with --adapt-every 0, and to the previous commit with the prefetch off (--second-gpu-prefetch-mb 0); with it, 29 / 39 of 1200 argmaxes differ on thinking text, all near-ties (the experts the 5070 Ti computes change). verify_parity: the reads keeping the average and the estimate bitwise the plain ones, both within FP64 bounds, the biased logits and the update bitwise their host emulation. Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- include/strata/core/verify.hpp | 22 ++++++--- include/strata/kernels/fused_gr.hpp | 13 ++++- include/strata/kernels/native_router.hpp | 4 ++ src/core/verify.cpp | 63 ++++++++++++++++++------ src/kernels/cuda/fused_gr.cu | 35 +++++++++++-- src/kernels/cuda/native_router.cu | 19 ++++++- src/kernels/verify_parity.cpp | 63 +++++++++++++++++++++--- src/program/generate.cpp | 6 ++- 8 files changed, 187 insertions(+), 38 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 96dc4d2f437..e37bcadc7f9 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -44,8 +44,8 @@ class NativeHead; /// second GPU's rows, the main GPU's zeroed unless `GpuPlanSink::host_rows_only`. using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out, int64_t layer); -/// Layer `layer`'s likely experts, its router applied to the previous layer's FFN input: ids and weights (n_tok, k), -/// host memory. +/// Layer `layer`'s likely experts, its router applied to an estimate of its FFN input (`Verifier::set_predict`): ids +/// and weights (n_tok, k), host memory. using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); /// The window's last CPU rows are in. using TailFn = void (*)(void* user); @@ -116,10 +116,13 @@ class Verifier { /// launch, ~2 us). Set before the first `run`; `print_profile` writes the per-stage table to stdout. void set_profile(bool on) { profile_ = on; } void print_profile() const; - /// Each layer's router also runs on the previous layer's FFN input, on a side branch of the graph; once the CPU's - /// rows of a layer are in, `fn` gets the next layer's prediction (the RAM is idle until the next ring), waited - /// for when it comes later (`predict_late`). Windows of one token group. Set before the first `run`. - void set_predict(PredictFn fn, void* user) { predict_ = fn; predict_user_ = user; } + /// Each layer's router also runs on an estimate of its FFN input from the previous layer's state, on a side branch + /// of the graph: the layer's FFN read of that state, its gates their running average over the windows, and the + /// router's logits corrected by a running average of their error. Once the CPU's rows of a layer are in, `fn` + /// gets the next layer's prediction (the RAM is idle until the next ring), waited for when it comes later + /// (`predict_late`). Without `learn` the averages stay at their start (gates of one half, no correction), so the + /// prediction depends on the window alone. Windows of one token group. Set before the first `run`. + void set_predict(PredictFn fn, void* user, bool learn) { predict_ = fn; predict_user_ = user; learn_ = learn; } /// `fn` runs once the window's last CPU rows are in: the RAM stays idle until the next window's first pool (the /// head, the commit, the drafts). The main GPU may still run the window then: `window_done` follows its graph. /// Set before the first `run`. @@ -189,6 +192,7 @@ class Verifier { // taken it; the counter holds the predictions published in this window PredictFn predict_ = nullptr; void* predict_user_ = nullptr; + bool learn_ = false; TailFn tail_ = nullptr; void* tail_user_ = nullptr; int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k @@ -222,8 +226,12 @@ class Verifier { float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *hit_out_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr; unsigned* ring_count_ = nullptr; // verify_router's block counters: the layer's, the prediction's - float *plogits_ = nullptr, *pw_ = nullptr; // the next layer's router (prediction) + float* plogits_ = nullptr; // the predictions' logits, by layer parity + float* pw_ = nullptr; // the next layer's router (prediction) int32_t* pids_ = nullptr; + float* pmixed_ = nullptr; // its input: the next layer's FFN read, estimated + float* gate_ema_ = nullptr; // [layer][hc][n_embd]: the FFN reads' gates, averaged + float* pbias_ = nullptr; // [layer][n_expert]: the predictions' logit bias int32_t* plan_ = nullptr; // device copies of the PCIe share's plan blocks int32_t* dplan_ = nullptr; // the main GPU's hit plans, decided on the device int32_t* res_ = nullptr; // the residency snapshot, copied as a window starts diff --git a/include/strata/kernels/fused_gr.hpp b/include/strata/kernels/fused_gr.hpp index f7ff353a43e..abe486a616a 100644 --- a/include/strata/kernels/fused_gr.hpp +++ b/include/strata/kernels/fused_gr.hpp @@ -7,6 +7,10 @@ // inject[c] = w_inject[c] . xn when w_inject is given // up : R <- R' in place (when `apply`) // mixed[d] = mean_c xn[c,d] * sigmoid(w_up[c*n_embd + d] . lo) +// gate_ema[c,d] += (mean over the tokens of sigmoid(w_up[c*n_embd + d] . lo) - gate_ema[c,d]) / 4 +// when `gate_ema` is given +// est[d] = mean_c R'[c,d] * rs[c] * est_norm[c,d] * est_gates[c,d] when `est_norm` is given: the mixed of +// another read of R' (its norm weights) with the gates `est_gates` // // The down projection is split by input slices: 80 blocks each take a quarter of the 324 rows over a twentieth of // the input, so each block reads 1/20 of every token's R' instead of all of it, and the stream's rs multiplies the @@ -36,13 +40,18 @@ struct FusedGrArgs { float* rs = nullptr; ///< workspace, hc floats float* inject_out = nullptr; ///< hc floats (when w_inject) float* mixed = nullptr; ///< n_embd + float* gate_ema = nullptr; ///< (hc, n_embd): the gates' running average over the reads, or null + const float* est_norm = nullptr; ///< another read's w_norm, or null + const float* est_gates = nullptr; ///< (hc, n_embd): the gates for its estimate + float* est = nullptr; ///< n_embd: the estimate }; bool fused_gr_supported(int64_t n_embd, int64_t hc, int64_t hc_lr); /// The read of `n_tok` tokens (1-8): `a[t]` is token t's arguments (its own R, pending write, lo, rs, inject, -/// mixed; the four weight pointers and eps must be the same for every t). `scratch` is `fused_gr_scratch_bytes()`, -/// zeroed once when allocated (the kernels leave it zeroed); reads that share it must run in stream order. +/// mixed, est; the four weight pointers, gate_ema, est_norm, est_gates and eps must be the same for every t). +/// `scratch` is `fused_gr_scratch_bytes()`, zeroed once when allocated (the kernels leave it zeroed); reads that share +/// it must run in stream order. constexpr int kFusedGrMaxT = 8; size_t fused_gr_scratch_bytes(); void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* scratch, void* stream); diff --git a/include/strata/kernels/native_router.hpp b/include/strata/kernels/native_router.hpp index 4bb9d0e366c..d180860ffac 100644 --- a/include/strata/kernels/native_router.hpp +++ b/include/strata/kernels/native_router.hpp @@ -38,6 +38,10 @@ struct VerifyRouterArgs { int cap = 0, ptr_off = 0; unsigned* counter = nullptr; // device, zero before the first launch (each launch leaves it so) int n_tok = 0, n_embd = 0, n_expert = 0; // 1..8 tokens, 512 experts, n_embd a multiple of 512 + const float* bias = nullptr; // (n_expert) added to the logits, or null }; void verify_router(const VerifyRouterArgs& a, void* stream); +// A prediction's bias: bias[e] = fmaf(1/8 / n_tok, sum over the n_tok tokens of (logits[t][e] - predicted[t][e]), +// bias[e]), where predicted holds the logits a verify_router with this bias computed for the same tokens (512 experts). +void verify_router_bias_update(const float* logits, const float* predicted, int n_tok, float* bias, void* stream); } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index def7350724c..05a35752794 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -229,7 +229,10 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& attn_scratch_ = b.take(T * (uint64_t) attn_scratch_floats_); tail_snap_ = b.take(nQ * TS); logits_ = b.take(T * (uint64_t) g.n_expert); w_ = b.take(T * K); ids_ = b.take(T * K); - plogits_ = b.take(T * (uint64_t) g.n_expert); pw_ = b.take(T * K); pids_ = b.take(T * K); + plogits_ = b.take(2 * T * (uint64_t) g.n_expert); pw_ = b.take(T * K); + pids_ = b.take(T * K); + pmixed_ = b.take(T * N); gate_ema_ = b.take((uint64_t) g.n_layers * HC * N); + pbias_ = b.take((uint64_t) (g.n_layers * g.n_expert)); shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); ring_count_ = b.take(2); @@ -260,6 +263,13 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sink_.staging = (unsigned long long) staging_; sink_.staging_cap = kStagingBlobs; (void) TS; + { // the gates' running averages start at a gate's middle + const std::vector half((size_t) (g.n_layers * g.hc * g.n_embd), 0.5f); + if (cudaMemcpy(gate_ema_, half.data(), half.size() * sizeof(float), cudaMemcpyHostToDevice) != cudaSuccess) { + err = "verify: the gates' averages could not be set"; + return false; + } + } { std::vector sp((size_t) hits.slots); for (int64_t i = 0; i < hits.slots; ++i) @@ -385,11 +395,15 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { broadcast_streams(emb_, R_, N, (int) HC, T, cs); } - // set_predict: the next layer's router on this layer's FFN input, on a side branch forked after the doorbell and - // joined before the next layer rewrites that input + // set_predict: this layer's FFN read also estimates the next layer's FFN input (that layer's read of this state, + // its gates their running average over the windows), and on a side branch forked after the doorbell the next + // layer's router runs on it with that layer's logit bias. When learning, the FFN read of a layer the previous one + // predicts keeps its gates' average, and the layer corrects its bias by its own logits on that branch. auto predicts = [&](int64_t l) { return predict_ != nullptr && G == 1 && l + 1 < g.n_layers && native_router_enabled(); }; + auto corrects = [&](int64_t l) { return learn_ && l > 0 && predicts(l - 1); }; + auto branches = [&](int64_t l) { return predicts(l) || corrects(l); }; // per-layer state indices (GDN and QSA layers are numbered separately) std::vector gdn_idx((size_t) g.n_layers, -1), qsa_idx((size_t) g.n_layers, -1); @@ -483,7 +497,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } pending = false; } - auto gr_read_group = [&](int half, bool apply, float* inj_prev, float* inj_out) { + auto gr_read_group = [&](int half, bool apply, float* inj_prev, float* inj_out, float* gate_ema, + const float* est_norm, const float* est_gates) { FusedGrArgs fa[kFusedGrMaxT]; for (int t = tb; t < te; ++t) { FusedGrArgs& a = fa[t - tb]; @@ -492,11 +507,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { a.w_norm = (const float*) wn[half]->data; a.w_down = (const uint16_t*) wd[half]->data; a.w_up = (const uint16_t*) wu[half]->data; a.w_inject = (const uint16_t*) wi[half]->data; a.eps = EPS; a.lo = lo_ + t * g.hc_lr; a.rs = rs_ + t * HC; - a.inject_out = inj_out + t * HC; a.mixed = mixed_ + t * N; + a.inject_out = inj_out + t * HC; a.mixed = mixed_ + t * N; a.gate_ema = gate_ema; + a.est_norm = est_norm; a.est_gates = est_gates; a.est = pmixed_ + t * N; } fused_gr_read_multi(fa, n, grs_, cs); }; - gr_read_group(0, pending, inj2_, inj_); + gr_read_group(0, pending, inj2_, inj_, nullptr, nullptr, nullptr); stamp(l, 1, cs); float* xm = mixed_ + tb * N; // the mixer's projections: a group of 2+ tokens is quantized with the interleaved copy the multi-token kernels @@ -637,7 +653,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } stamp(l, 2, cs); - gr_read_group(1, true, inj_, inj2_); + // the FFN read, and the next layer's estimated FFN input for the prediction (set_predict) + const WeightRef* wn1 = predicts(l) ? need(LayerView(wt, l + 1), "hc_ffn_norm.weight", err) : nullptr; + if (predicts(l) && !wn1) return false; + gr_read_group(1, true, inj_, inj2_, corrects(l) ? gate_ema_ + (size_t) l * HC * N : nullptr, + wn1 != nullptr ? (const float*) wn1->data : nullptr, + wn1 != nullptr ? gate_ema_ + (size_t) (l + 1) * HC * N : nullptr); stamp(l, 3, cs); // the routed experts' activations beside the router (joined before the experts) if (!fork(side_, bfork_)) return false; @@ -685,30 +706,42 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 4, cs); // after the doorbell: the shared expert's branch, and the next layer's prediction if (beside && !shared(l, grp)) return false; + if (branches(l) && + (cudaEventRecord(fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, fork_, 0) != cudaSuccess)) { + err = "verify: the prediction branch could not fork"; + return false; + } if (predicts(l)) { const WeightRef* wr1 = need(LayerView(wt, l + 1), "ffn_gate_inp.weight", err); if (!wr1) return false; - if (cudaEventRecord(fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, fork_, 0) != cudaSuccess) { - err = "verify: the prediction branch could not fork"; - return false; - } const size_t half = (size_t) ((l + 1) & 1) * (size_t) (MT * K); VerifyRouterArgs ra; - ra.x = xm; ra.w = (const uint16_t*) wr1->data; ra.logits = plogits_; ra.ids = pids_; ra.weights = pw_; + ra.x = pmixed_; ra.w = (const uint16_t*) wr1->data; + ra.logits = plogits_ + (size_t) ((l + 1) & 1) * MT * NE; ra.ids = pids_; ra.weights = pw_; ra.ids_out = m_pids_ + half; ra.w_out = m_pw_ + half; ra.seq = m_pseq_; ra.ring = (uint32_t) (l + 1); // the predictions published so far ra.counter = ring_count_ + 1; ra.n_tok = n; ra.n_embd = (int) N; ra.n_expert = (int) NE; + if (learn_) ra.bias = pbias_ + (size_t) (l + 1) * NE; try { verify_router(ra, side_); } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + " prediction: " + e.what(); return false; } - if (cudaEventRecord(join_, side_) != cudaSuccess) { - err = "verify: the prediction branch could not join"; + } + if (corrects(l)) { // after the prediction, from the one the previous layer made + try { + verify_router_bias_update(logits_ + tb * NE, plogits_ + (size_t) (l & 1) * MT * NE, n, + pbias_ + (size_t) l * NE, side_); + } catch (const std::exception& e) { + err = "verify layer " + std::to_string(l) + " prediction bias: " + e.what(); return false; } } + if (branches(l) && cudaEventRecord(join_, side_) != cudaSuccess) { + err = "verify: the prediction branch could not join"; + return false; + } if (!wait(bjoin_)) return false; stamp(l, 5, cs); return beside || shared(l, grp); @@ -799,7 +832,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } } stamp(l, 10, cs); - if (predicts(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { + if (branches(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { err = "verify: the prediction branch could not join"; return false; } diff --git a/src/kernels/cuda/fused_gr.cu b/src/kernels/cuda/fused_gr.cu index 70f323ea56b..9c2fc805118 100644 --- a/src/kernels/cuda/fused_gr.cu +++ b/src/kernels/cuda/fused_gr.cu @@ -201,8 +201,9 @@ __global__ void __launch_bounds__(DTH) gr_down_split_kernel(GrMulti m, float* __ // The up projection of TT tokens: each row of w_up read once, RW rows a warp at a time; the TT x RW dots of those rows // summed by one xor tree (xor_scatter) and every (row, token) epilogue on the lane holding its sum, its inputs fetched -// while the dots run: R <- R' for this block's columns (when apply), mixed[d] = mean_c xn[c,d] * sigmoid(u[c,d]). -// The lo vectors are staged split into chunk halves. +// while the dots run: R <- R' for this block's columns (when apply), mixed[d] = mean_c xn[c,d] * sigmoid(u[c,d]), the +// gates' running average (when gate_ema) and the other read's estimate (when est_norm). The lo vectors are staged +// split into chunk halves. template __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { constexpr int RW = TT <= 4 ? 4 : 2; @@ -210,6 +211,9 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { constexpr int LPV = 32 / NV; // lanes that end up holding each sum __shared__ __align__(16) float4 lo4[TT][2][LR / 8]; __shared__ float g[TT][HC][UPM_COLS]; + __shared__ float gs[TT][HC][UPM_COLS]; // the gates + __shared__ float ge[TT][HC][UPM_COLS]; // the estimate's terms + const bool est = m.a[0].est_norm != nullptr; const int t = threadIdx.x, lane = t & 31, warp = t >> 5; const int d0 = blockIdx.x * UPM_COLS; for (int i = t; i < TT * LR / 4; i += THREADS) { @@ -230,7 +234,7 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { wb[q] = lane < LR / 8 - 32 ? __ldg(w4 + 32 + lane) : make_uint4(0, 0, 0, 0); } const int er = r0 + vq, ec = er / UPM_COLS, edd = er - ec * UPM_COLS, ei = ec * N + d0 + edd; - float rv = 0.0f, wn = 0.0f, rsc = 0.0f, bo = 0.0f, ip = 0.0f; + float rv = 0.0f, wn = 0.0f, rsc = 0.0f, bo = 0.0f, ip = 0.0f, en = 0.0f, eg = 0.0f; bool apply = false; if (epi) { const FusedGrArgs& a = m.a[vk]; @@ -239,6 +243,7 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { rsc = a.rs[ec]; apply = a.apply; if (apply) { bo = a.bo_prev[d0 + edd]; ip = a.inj_prev[ec]; } + if (est) { en = a.est_norm[ei]; eg = a.est_gates[ei]; } } float v[NV]; #pragma unroll @@ -259,7 +264,10 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { m.a[vk].R_out[ei] = x; } const float y = x * wn * rsc; - g[vk][ec][edd] = y * sigmoidf_(v[0]); + const float gate = sigmoidf_(v[0]); + g[vk][ec][edd] = y * gate; + gs[vk][ec][edd] = gate; + ge[vk][ec][edd] = x * rsc * en * eg; } } __syncthreads(); @@ -270,6 +278,22 @@ __global__ void __launch_bounds__(THREADS) gr_up_multi_kernel(GrMulti m) { for (int c = 0; c < HC; ++c) s += g[k][c][col]; m.a[k].mixed[d0 + col] = s / (float) HC; } + if (est) + for (int i = t; i < TT * UPM_COLS; i += THREADS) { + const int k = i / UPM_COLS, col = i - k * UPM_COLS; + float s = 0.0f; +#pragma unroll + for (int c = 0; c < HC; ++c) s += ge[k][c][col]; + m.a[k].est[d0 + col] = s / (float) HC; + } + if (m.a[0].gate_ema != nullptr && t < HC * UPM_COLS) { + const int c = t / UPM_COLS, col = t - c * UPM_COLS; + float s = 0.0f; +#pragma unroll + for (int k = 0; k < TT; ++k) s += gs[k][c][col]; + float* e = m.a[0].gate_ema + (size_t) c * N + d0 + col; + *e += 0.25f * (s / (float) TT - *e); + } } } // namespace @@ -288,7 +312,8 @@ void fused_gr_read_multi(const FusedGrArgs* a, int n_tok, float* scratch, void* if (!x.R || !x.w_norm || !x.w_down || !x.w_up || !x.lo || !x.rs || !x.mixed || (x.w_inject && !x.inject_out) || (x.apply && (!x.bo_prev || !x.inj_prev || !x.R_out || x.inj_prev == x.inject_out)) || x.w_down != a[0].w_down || x.w_up != a[0].w_up || x.w_inject != a[0].w_inject || - x.w_norm != a[0].w_norm || x.eps != a[0].eps) { + x.w_norm != a[0].w_norm || x.gate_ema != a[0].gate_ema || x.eps != a[0].eps || + x.est_norm != a[0].est_norm || x.est_gates != a[0].est_gates || (x.est_norm && (!x.est_gates || !x.est))) { std::fprintf(stderr, "fused_gr_read_multi: invalid arguments for token %d\n", t); std::exit(1); } diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index 9e104d038dd..6897f74f9bd 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -143,7 +143,8 @@ __global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterA float acc[kMmvfMaxRows]; mmvf_multi_row(a.x, a.w + (size_t) row * a.n_embd, a.n_embd, TT, acc); if (t == 0) - for (int k = 0; k < TT; ++k) a.logits[(size_t) k * a.n_expert + row] = acc[k]; + for (int k = 0; k < TT; ++k) + a.logits[(size_t) k * a.n_expert + row] = a.bias ? acc[k] + a.bias[row] : acc[k]; __threadfence(); } __syncthreads(); @@ -164,6 +165,14 @@ __global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterA if (a.plan != nullptr) hit_plan_block(a.ids, TT * 10, 10, a.res, a.n_expert, a.slot_ptr, a.plan, a.cap, a.ptr_off); } +__global__ void __launch_bounds__(512) bias_update(const float* __restrict__ logits, + const float* __restrict__ predicted, int n_tok, float scale, + float* __restrict__ bias) { + const int e = (int) threadIdx.x; + float d = 0.0f; + for (int t = 0; t < n_tok; ++t) d += logits[t * 512 + e] - predicted[t * 512 + e]; + bias[e] = fmaf(scale, d, bias[e]); +} bool valid(const void* p, size_t bytes) { const auto address = reinterpret_cast(p); return p && address % 4 == 0 && bytes <= UINTPTR_MAX - address; @@ -199,6 +208,14 @@ void verify_router(const VerifyRouterArgs& a, void* stream) { const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } +void verify_router_bias_update(const float* logits, const float* predicted, int n_tok, float* bias, void* stream) { + if (!stream || n_tok < 1 || n_tok > 8 || !logits || !predicted || !bias) + throw std::invalid_argument("verify_router_bias_update: 1..8 tokens and buffers"); + bias_update<<<1, 512, 0, static_cast(stream)>>>(logits, predicted, n_tok, 0.125f / (float) n_tok, + bias); + const auto error = cudaGetLastError(); + if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); +} void native_router_top10_multi(const float* logits, int32_t* ids, float* weights, int n_tok, void* stream) { const size_t nl = size_t(n_tok) * 512 * 4, no = size_t(n_tok) * 10 * 4; if (!stream || n_tok < 1 || !valid(logits, nl) || !valid(ids, no) || !valid(weights, no) diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 45ed9b846ed..e7898b2e4ee 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -324,7 +324,7 @@ int test_verify_router(std::mt19937& rng, cudaStream_t s) { check(cudaHostGetDevicePointer((void**) &m_w[p], h_w[p], 0), "mapped"); check(cudaHostGetDevicePointer((void**) &m_seq[p], h_seq[p], 0), "mapped"); } - float* d_x = dev((size_t) 8 * N); + float *d_x = dev((size_t) 8 * N), *d_bias = dev((size_t) NE); int bad = 0; for (int n_tok = 1; n_tok <= 8; ++n_tok) { for (int rep = 0; rep < 4; ++rep) { @@ -373,6 +373,26 @@ int test_verify_router(std::mt19937& rng, cudaStream_t s) { std::fprintf(stderr, "verify_router: n_tok %d rep %d: the hit plans differ\n", n_tok, rep); ++b; } + // with a bias the logits are the plain ones plus it; the update then corrects it by the plain ones + std::vector bias((size_t) NE); + for (auto& v : bias) v = std::normal_distribution(0.0f, 0.5f)(rng); + up(d_bias, bias); + a.bias = d_bias; + strata::kernels::verify_router(a, s); + strata::kernels::verify_router_bias_update(d_logits[0], d_logits[1], n_tok, d_bias, s); + check(cudaStreamSynchronize(s), "router bias"); + const std::vector plain = down(d_logits[0], (size_t) n_tok * NE); + std::vector want((size_t) n_tok * NE), fixed((size_t) NE); + for (int t = 0; t < n_tok; ++t) + for (int e = 0; e < NE; ++e) + want[(size_t) (t * NE + e)] = plain[(size_t) (t * NE + e)] + bias[(size_t) e]; + for (int e = 0; e < NE; ++e) { + float dsum = 0.0f; + for (int t = 0; t < n_tok; ++t) dsum += plain[(size_t) (t * NE + e)] - want[(size_t) (t * NE + e)]; + fixed[(size_t) e] = std::fmaf(0.125f / (float) n_tok, dsum, bias[(size_t) e]); + } + b += bitwise_diff(want, down(d_logits[1], (size_t) n_tok * NE), "biased logits"); + b += bitwise_diff(fixed, down(d_bias, (size_t) NE), "corrected bias"); if (b) std::fprintf(stderr, "verify_router: n_tok %d rep %d: %d mismatches\n", n_tok, rep, b); bad += b; } @@ -381,9 +401,12 @@ int test_verify_router(std::mt19937& rng, cudaStream_t s) { for (void* q : {(void*) d_logits[p], (void*) d_wts[p], (void*) d_ids[p], (void*) d_plan[p]}) cudaFree(q); for (void* q : {(void*) h_x[p], (void*) h_ids[p], (void*) h_w[p], (void*) h_seq[p]}) cudaFreeHost(q); } - for (void* q : {(void*) d_w, (void*) d_sp, (void*) d_res, (void*) d_counter, (void*) d_x}) cudaFree(q); - std::printf("verify_router: %s\n", bad ? "MISMATCH" - : "bitwise the gemv, top 10, doorbell and hit plan (1-8 tokens, 4 routings each)"); + for (void* q : {(void*) d_w, (void*) d_sp, (void*) d_res, (void*) d_counter, (void*) d_x, (void*) d_bias}) + cudaFree(q); + std::printf("verify_router: %s\n", + bad ? "MISMATCH" + : "bitwise the gemv, top 10, doorbell and hit plan (1-8 tokens, 4 routings each), and with a bias " + "the logits plus it and its update"); return bad; } @@ -437,6 +460,17 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { float *d_bo = dev(bo.size()), *d_inj = dev(inj.size()); up(d_bo, bo); up(d_inj, inj); + // path 0 also keeps the gates' running average and estimates another read (its norm weights wn2, the + // gates eg) + std::vector ema0((size_t) D), wn2((size_t) D), eg((size_t) D); + for (auto& v : ema0) v = std::uniform_real_distribution(0.0f, 1.0f)(rng); + for (auto& v : eg) v = std::uniform_real_distribution(0.0f, 1.0f)(rng); + for (auto& v : wn2) v = edgy(rng); + float *d_ema = dev((size_t) D), *d_est = dev((size_t) (n_tok * N)); + float *d_wn2 = dev((size_t) D), *d_eg = dev((size_t) D); + up(d_ema, ema0); + up(d_wn2, wn2); + up(d_eg, eg); std::vector args[2]; for (int p = 0; p < 2; ++p) for (int t = 0; t < n_tok; ++t) { @@ -447,11 +481,13 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { a.eps = 1e-6f; a.lo = d[p][1] + t * LR; a.rs = d[p][2] + t * HC; a.inject_out = d[p][3] + t * HC; a.mixed = d[p][4] + t * N; + if (p == 0) { a.gate_ema = d_ema; a.est_norm = d_wn2; a.est_gates = d_eg; a.est = d_est + t * N; } args[p].push_back(a); } strata::kernels::fused_gr_read_multi(args[0].data(), n_tok, d_scratch, s); for (int t = 0; t < n_tok; ++t) strata::kernels::fused_gr_read_multi(&args[1][(size_t) t], 1, d_scratch, s); check(cudaStreamSynchronize(s), "gr read"); + const std::vector ema1 = down(d_ema, (size_t) D), est = down(d_est, (size_t) (n_tok * N)); const char* names[5] = {"R", "lo", "rs", "inject", "mixed"}; const size_t sizes[5] = {R.size(), (size_t) (n_tok * LR), (size_t) (n_tok * HC), (size_t) (n_tok * HC), (size_t) (n_tok * N)}; @@ -524,10 +560,23 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { const double sg = 1.0 / (1.0 + std::exp(-u)); mixed += xn[i] * sg / HC; bound += (1e-4 * xa[i] + 0.25 * std::fabs(xn[i]) * du) / HC; + if (n_tok == 1) + near(ema1[i], ema0[i] + 0.25 * (sg - ema0[i]), 0.25 * (0.25 * du + 1e-6) + 1e-6, + "gate average", (int64_t) i); } near(got[4][(size_t) dd], mixed, bound + 1e-30, "mixed", dd); } + for (int64_t dd = 0; dd < N; ++dd) { // the other read's estimate + double ref = 0.0, mag = 0.0; + for (int c = 0; c < HC; ++c) { + const size_t i = (size_t) (c * N + dd); + ref += Rp[i] * rs[(size_t) c] * wn2[i] * eg[i] / HC; + mag += Ra[i] * rs[(size_t) c] * std::fabs((double) wn2[i]) * eg[i] / HC; + } + near(est[(size_t) dd], ref, 1e-4 * mag + 1e-30, "estimate", dd); + } } + for (float* q : {d_ema, d_est, d_wn2, d_eg}) cudaFree(q); for (int p = 0; p < 2; ++p) for (float* q : d[p]) cudaFree(q); cudaFree(d_bo); @@ -536,9 +585,11 @@ int test_gr_read(std::mt19937& rng, cudaStream_t s) { } for (void* p : {(void*) d_wd, (void*) d_wu, (void*) d_wi, (void*) d_wn, (void*) d_scratch}) cudaFree(p); bad += off; - char msg[160]; + char msg[256]; std::snprintf(msg, sizeof msg, - "each token bitwise its 1-token read (1-8 tokens, 3 variants); FP64 within bounds (largest error %.2g of its bound)", + "each token bitwise its 1-token read (1-8 tokens, 3 variants, keeping the gates' average and " + "estimating another read or not); FP64, the gates' average and the estimate within bounds (largest " + "error %.2g of its bound)", worst); std::printf("gr_read: %s\n", bad ? "MISMATCH" : msg); return bad; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 00a3ceecb00..1b0a34ac7d3 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -2336,7 +2336,8 @@ int main(int argc, char** argv) { ver.set_profile(o.window_profile); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 - if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) + ver.set_predict(&drive_predict, &drive, o.adapt_every > 0 && o.adapt_swaps > 0); if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache ver.set_tail(&drive_tail, &drive); tier.set_after(ver.window_done()); @@ -3047,7 +3048,8 @@ int main(int argc, char** argv) { ver.set_host_rows(!o.window_hashes.empty()); ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2 : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 - if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive); + if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) + ver.set_predict(&drive_predict, &drive, o.adapt_every > 0 && o.adapt_swaps > 0); if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache ver.set_tail(&drive_tail, &drive); tier.set_after(ver.window_done()); From 84c39840f71b8573257b20d4e3f9f3dc78d4a56c Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 06:09:07 +0200 Subject: [PATCH 63/92] Second GPU: its rows reach the 3090 on a branch while the CPU works; its own experts before the prefetched ones After the CPU pool the host waited for the second GPU's graph (SecondGpu::finish), copied its rows into the pool's mapped rows and raised the flag; the 3090's combine then read them over the x4 link with the CPU's. A layer that took a prefetched expert launched the whole share behind the prefetch copies. Now the second GPU's graph writes its rows into the window's mapped rows itself, at their entries, and its last block raises a flag of the token group's once they are visible. A branch of the 3090's graph, forked at the doorbell, waits for that flag and copies the rows into VRAM (fetch_listed_rows, from the entries the pool publishes), and the combine takes them from there as they are (native_moe_gather_combine's third list). The host raises the CPU's flag as soon as the CPU's rows are in, and the flag of a layer without a share at once. The share goes out in two launches: the experts in its cache at once, the prefetched ones once their copies have landed. The second tier's copies between windows wait for the last share (AdaptiveTier::set_after), which may still read its slots; a failure there is seen every ~2 ms while the host waits for a ring (Verifier::set_watch), and the window's waits are then released. --stats: the layers whose second-GPU rows came after the CPU's, and the host time from the CPU's to theirs; --window-profile: the combine's wait for them and their branch. Window hashes identical to the previous commit (--adapt-every 0): thinking text on both models, code text with and without --spec-split, and without the prefetch. verify_parity: the gather-combine with the second GPU's rows fetched. Fixed text, both GPUs, 2 rounds against the previous commit, ms per round (t/s): UD-Q4_K_XL code -8.6% (+10.3%; the host's pool call 17.0 -> 14.5 ms a round, 43 -> 48 GB/s over the rows), thinking -1.8% (+1.1%), after 16K -1.0% (+1.1%); IQ3_XXS code -0.9% (+0.9%), thinking -0.2% (+0.1%), after 16K -1.0% (-0.7%). The second GPU's rows now come after the CPU's in 10-20 layers a round on UD-Q4_K_XL (0.56-1.08 ms a round), 3-11 on IQ3_XXS (0.11-0.50 ms). Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- include/strata/core/expert_source.hpp | 10 +- include/strata/core/second_gpu.hpp | 44 ++++--- include/strata/core/verify.hpp | 31 ++++- include/strata/kernels/iq_kernels.hpp | 13 +- include/strata/kernels/native_moe.hpp | 13 +- include/strata/kernels/verify_kernels.hpp | 5 + src/core/adaptive_tier.cpp | 5 + src/core/expert_source.cpp | 35 +++-- src/core/second_gpu.cpp | 149 +++++++++++++--------- src/core/verify.cpp | 118 ++++++++++++++--- src/kernels/cuda/iq_kernels.cu | 33 ++++- src/kernels/cuda/native_moe.cu | 30 +++-- src/kernels/cuda/verify_kernels.cu | 29 +++++ src/kernels/verify_parity.cpp | 66 +++++++--- src/program/generate.cpp | 26 +++- 15 files changed, 439 insertions(+), 168 deletions(-) diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index c4539081aac..bbc1a61d14a 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -89,6 +89,13 @@ struct GpuPlanSink { int pcie_mode = 0; /// The GPU reads only the rows it did not compute, so the pool leaves the GPU's rows of `out` unwritten. bool host_rows_only = false; + /// The second GPU's share of this token group (mapped; null without one): its entries, [n, pad x3, entries], and + /// the flag its graph raises to `ring` once their rows are in `out`, which the main GPU waits for. The pool + /// raises the flag itself when the second GPU takes no share, and sets `gpu2_ring` to `ring` once the flag is + /// raised or a share is submitted. + int32_t* gpu2_list = nullptr; + uint32_t* gpu2_flag = nullptr; + uint32_t ring = 0, gpu2_ring = 0; }; /// The adapter's own state. One per session, reused every layer so the token path allocates nothing (P2.T10). @@ -248,7 +255,8 @@ void expert_pool_dispatch(void* user, const float* x_f, const int32_t* ids, cons /// Plan v0.3 P6: the pool for a verify window of `n_tok` tokens. `x_f` is (n_tok, n_embd), `ids` (n_tok, k) and /// `out` (n_tok * k, n_embd). Each distinct missed expert is computed once for all the tokens routed to it; -/// resident experts' rows are the main GPU's (zeroed, or left unwritten with `GpuPlanSink::host_rows_only`). +/// resident experts' rows are the main GPU's (zeroed, or left unwritten with `GpuPlanSink::host_rows_only`), and the +/// second GPU writes its share's rows itself, after this returns (`GpuPlanSink::gpu2_flag`). void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out); /// Copies `layer`'s likeliest experts that neither GPU holds to the second GPU's prefetch slots, where that layer's diff --git a/include/strata/core/second_gpu.hpp b/include/strata/core/second_gpu.hpp index e12a5b96cd3..796a39e9407 100644 --- a/include/strata/core/second_gpu.hpp +++ b/include/strata/core/second_gpu.hpp @@ -2,17 +2,18 @@ // // It holds experts the first GPU's cache does not (the profile's next ranks, then the conversation's, through its // own AdaptiveTier) and computes them for each layer of a verify window while the CPU pool computes the misses. -// It never spins on host memory: per layer the host launches one captured graph (copy the plan in, quantize the -// activations, the grouped experts, their rows out; activations, plan and rows in mapped host memory) before the CPU -// pool and collects the rows after it, so it keeps its latency on a GPU that also drives a display (a graph spinning -// there stalls whenever the desktop draws). The rows cross PCIe in 16-byte stores: written one float at a time they -// took ~0.2 ms a layer and slowed the CPU pool's RAM reads. One graph per format pair and group-count bucket: the -// grouped kernels' grids are sized for the bucket. Native packs only: the kernels are the first GPU's hit path. +// It never spins on host memory: per layer the host launches captured graphs once the CPU pool's workers have +// started (copy the plan in, quantize the activations, the grouped experts, their rows out), and the last one writes +// the rows into the window's own rows in mapped host memory and raises a flag there, which the first GPU waits for; +// so it keeps its latency on a GPU that also drives a display (a graph spinning there stalls whenever the desktop +// draws). The rows cross PCIe in 16-byte stores: written one float at a time they took ~0.2 ms a layer and slowed the +// CPU pool's RAM reads. One graph per format pair, group-count bucket and part: the grouped kernels' grids are sized +// for the bucket. Native packs only: the kernels are the first GPU's hit path. // // Prefetch: between a layer's CPU pool and the next layer's ring the RAM is idle (the first GPU's serial part, ~0.28 ms // a layer), and this GPU's x16 link copies ~12 MiB in that time: 4 UD-Q4_K_XL or ~7 IQ3_XXS experts. The next layer's // likeliest experts that neither GPU holds go into a few slots of their own, and that layer's share takes them from -// there. +// there: its own experts start at once, the prefetched ones once their copies have landed. #pragma once #include "strata/core/expert_cache.hpp" @@ -52,32 +53,37 @@ class SecondGpu { } /// One layer's share: group g is the expert in slot `slots[g]`, serving the routed entries - /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations. + /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations. Entry + /// i's row goes to row i of `out` (n_embd floats, mapped host memory), and then `*flag` (mapped) is set to `ring`. + /// Waits for the last share's graphs first (the plan and staging are theirs until then). bool submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, - const int32_t* entries, int n_groups, std::string& err); - /// Waits for the submitted layer and writes its rows into `out` (row i = entry i, n_embd floats). - bool finish(float* out, std::string& err); + const int32_t* entries, int n_groups, float* out, uint32_t* flag, uint32_t ring, std::string& err); + /// False, with the reason, when its work so far failed; does not wait. + bool healthy(std::string& err) const; + /// Recorded after the last submitted share: the last work that may read its cache and prefetch slots. + cudaEvent_t done_event() const { return ev_; } int64_t layers = 0, experts = 0, entries_done = 0; int64_t prefetch_copied = 0, prefetch_used = 0; ///< experts copied ahead / of those, computed here - double ms_wait = 0; ///< host time waiting for it after the CPU pool private: - bool graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& exec, std::string& err); + bool graph_for(int gu_type, int d_type, int groups, int part, bool prep, cudaGraphExec_t& exec, std::string& err); int dev_ = -1, main_ = 0, max_t_ = 0; int64_t n_embd_ = 0, n_ff_ = 0, cap_ = 0; ExpertCache cache_; cudaStream_t s_ = nullptr; cudaEvent_t ev_ = nullptr; - // pinned staging (portable, mapped): the activations, the plan, and the rows the down kernel writes - float *h_x_ = nullptr, *m_x_ = nullptr, *h_out_ = nullptr, *m_out_ = nullptr; + // pinned staging (portable, mapped): the activations and the plan + float *h_x_ = nullptr, *m_x_ = nullptr; uint8_t *h_plan_ = nullptr, *m_plan_ = nullptr; uint8_t *d_xq_ = nullptr, *d_plan_ = nullptr, *d_scratch_ = nullptr; - float* d_rows_ = nullptr; // the rows, then copied out to the mapped ones in full transactions - struct Graph { int gu, d, groups; cudaGraphExec_t exec; }; - std::vector graphs_; // per (gate/up, down) format pair and group-count bucket - std::vector pending_; // the submitted layer's entries, in row order + float* d_rows_ = nullptr; // the rows, then copied out in full transactions + unsigned* d_count_ = nullptr; // the rows-out kernel's block counter + // part 0: the experts in its cache, launched at once; part 1: the prefetched ones, after their copies. `prep`: the + // graph copies the plan and quantizes the activations (the first launch of a layer) + struct Graph { int gu, d, groups, part; bool prep; cudaGraphExec_t exec; }; + std::vector graphs_; // prefetch slots, their copy stream, and the event of the last copies static constexpr int kPrefetchMax = 16; uint8_t* d_pre_ = nullptr; diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index e37bcadc7f9..7dcbff4a53a 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -40,10 +40,12 @@ namespace strata::core { class NativeHead; -/// The CPU pool for a window: x_f (n_tok, n_embd), ids (n_tok, k) -> out (n_tok * k, n_embd): the CPU's and the -/// second GPU's rows, the main GPU's zeroed unless `GpuPlanSink::host_rows_only`. +/// The CPU pool for a window: x_f (n_tok, n_embd), ids (n_tok, k) -> out (n_tok * k, n_embd): the CPU's rows, the +/// main GPU's zeroed unless `GpuPlanSink::host_rows_only`; a second GPU writes its own (`set_gpu2`). using PoolMultiFn = void (*)(void* user, const float* x_f, const int32_t* ids, int64_t n_tok, int64_t k, float* out, int64_t layer); +/// False, with the reason, when work the window waits for failed (the second GPU's). +using WatchFn = bool (*)(void* user, std::string& err); /// Layer `layer`'s likely experts, its router applied to an estimate of its FFN input (`Verifier::set_predict`): ids /// and weights (n_tok, k), host memory. using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); @@ -128,9 +130,19 @@ class Verifier { /// Set before the first `run`. void set_tail(TailFn fn, void* user) { tail_ = fn; tail_user_ = user; } cudaEvent_t window_done() const { return done_; } + /// The pool gives a second GPU a share of each layer (`GpuPlanSink::gpu2_flag`): it writes its rows into the + /// pool's rows and raises its token group's flag, and a branch of the window, forked at the ring, takes them + /// into VRAM while the CPU works. Set before the first `run`. + void set_gpu2(bool on) { gpu2_ = on; } + /// `fn` is asked every ~2 ms while the host waits for a ring: on failure the window's waits are released and `run` + /// returns the reason. Set before the first `run`. + void set_watch(WatchFn fn, void* user) { watch_ = fn; watch_user_ = user; } double ms_wait = 0, ms_pool = 0, ms_host = 0, ms_commit = 0, ms_predict = 0; int64_t windows = 0, predict_late = 0; + /// Layers whose second-GPU rows came after the CPU's, and the host time from the CPU's rows to theirs. + int64_t late2 = 0; + double ms_late2 = 0; private: bool capture(int T, std::string& err); @@ -188,6 +200,15 @@ class Verifier { bool pcie_share_ = true; int groups_[9] = {}; float* h_ymiss_ = nullptr; float* m_ymiss_ = nullptr; // T * k * n_embd + // the second GPU's share (set_gpu2), per token group: the flag it raises (64 bytes apart) and its entries, + // [n, pad x3, entries] (mapped; `list2_` their device copies, which the combine reads) + bool gpu2_ = false; + uint32_t* h_flag2_ = nullptr; uint32_t* m_flag2_ = nullptr; + int32_t* h_list2_ = nullptr; int32_t* m_list2_ = nullptr; + int32_t* list2_ = nullptr; + int64_t list2_stride_ = 0; + WatchFn watch_ = nullptr; + void* watch_user_ = nullptr; // the next layer's predicted experts: two halves by layer parity, so a prediction stays put until the host has // taken it; the counter holds the predictions published in this window PredictFn predict_ = nullptr; @@ -207,6 +228,8 @@ class Verifier { cudaEvent_t pfork_ = nullptr, pjoin_ = nullptr; // shs_'s projections cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; + cudaStream_t g2s_ = nullptr; // the second GPU's rows, per token group + cudaEvent_t g2fork_ = nullptr, g2join_[2] = {}; cudaEvent_t done_ = nullptr; // after the last window's graph // device @@ -249,8 +272,8 @@ class Verifier { int64_t cap_ = 0, max_blocks_ = 0, attn_scratch_floats_ = 0; // --window-profile: stamps [window size][layer, then one row for the window][stage]: a layer's main-stream stages, - // then the end of its shared expert's branch - static constexpr int kLayerStamps = 12, kStamps = 13; + // then the ends of its shared expert's branch and of the second GPU's rows' branch + static constexpr int kLayerStamps = 13, kStamps = 15; bool profile_ = false; unsigned long long* stamps_ = nullptr; std::vector h_stamps_; diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index c98c7454717..593cc0f3a11 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -64,10 +64,13 @@ size_t native_expert_scratch_bytes(int64_t cap_entries, int64_t n_ff); void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long* grp_ptr, const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_dst, const int32_t* ent_tok, int64_t cap_groups, int64_t cap_entries, const void* x_q8_1, void* scratch, float* out, void* stream); -/// Copies the first grp_start[*n_groups] rows of `rows` (n_embd floats each, n_embd % 4 == 0) to `out`, at most -/// `max_rows`, in 16-byte stores: rows bound for mapped host memory cross PCIe in full transactions (the grouped -/// kernels' one-float stores there cost ~0.2 ms a layer). Counts are read on the device. -void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, int64_t n_embd, - int64_t max_rows, float* out, void* stream); +/// Entries [grp_start[0], grp_start[*n_groups]): entry e's row ent_dst[e] of `rows` (n_embd floats each, +/// n_embd % 4 == 0) to the same row of `*out`, at most `max_rows` rows, in 16-byte stores: rows bound for mapped host +/// memory cross PCIe in full transactions (the grouped kernels' one-float stores there cost ~0.2 ms a layer). Then, +/// unless `*flag` is null, `**flag = *ring` once every row is visible to the host. Counts and pointers are read on +/// the device; `counter` (device, zero) is left zero. +void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, + const int32_t* ent_dst, int64_t n_embd, int64_t max_rows, float* const* out, + uint32_t* const* flag, const int32_t* ring, unsigned* counter, void* stream); } // namespace strata::kernels diff --git a/include/strata/kernels/native_moe.hpp b/include/strata/kernels/native_moe.hpp index 42735f9e131..69cd2a2e3cb 100644 --- a/include/strata/kernels/native_moe.hpp +++ b/include/strata/kernels/native_moe.hpp @@ -17,11 +17,12 @@ void native_moe_combine(const float* parts, const float* weights, const float* s float* output, int64_t n_embd, int64_t k, void* stream); // The verify window's combine for n_tok tokens, bitwise `native_moe_combine` per token over the rows the window -// used to assemble: row t*k + j is the GPU's (`gpu_rows`, row-indexed, as the host's zeroed row plus the hit) when -// an entry of dst[0..*count) or dst2[0..*count2) names it (`dst2` may be null), and otherwise the host's -// (`host_rows`, mapped memory: only these rows cross PCIe). weights (n_tok, k), shared and output (n_tok, n_embd). -// Graph-capturable. +// used to assemble: row t*k + j comes from `gpu_rows` (row-indexed) as the host's zeroed row plus the hit when an +// entry of dst[0..*count) or dst2[0..*count2) names it, as it is when an entry of dst3[0..*count3) does, and +// otherwise from `host_rows` (mapped memory: only these rows cross PCIe). `dst2` and `dst3` may be null. +// weights (n_tok, k), shared and output (n_tok, n_embd). Graph-capturable. void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, - const int32_t* count, const int32_t* dst2, const int32_t* count2, const float* weights, - const float* shared, float* output, int64_t n_embd, int64_t k, int n_tok, void* stream); + const int32_t* count, const int32_t* dst2, const int32_t* count2, const int32_t* dst3, + const int32_t* count3, const float* weights, const float* shared, float* output, + int64_t n_embd, int64_t k, int n_tok, void* stream); } diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index 8902ccf51d5..bafc85afead 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -55,6 +55,11 @@ void copy_indexed(float* dst, const float* src, int64_t stride, const int32_t* i /// into dst + k * blob_bytes with coalesced 16-byte loads - the PCIe share of a layer's missed experts, staged /// into VRAM before the grouped expert kernel reads them. Launched for a capacity of `cap` blobs. void fetch_blobs(const unsigned long long* src, const int32_t* n, uint8_t* dst, int64_t blob_bytes, int cap, void* stream); +/// A list in mapped host memory, [n, pad x3, rows...]: each listed row of `src` (mapped, n_embd floats, a multiple of +/// 4) to the same row of `dst`, and the list to `list_dev` (device, the same layout). At most `cap` rows; rows +/// outside [0, cap) are not copied. The second GPU's rows of a verify window's layer, taken while the CPU works. +void fetch_listed_rows(const int32_t* list, const float* src, float* dst, int32_t* list_dev, int cap, int64_t n_embd, + void* stream); /// ptr[k] = base + k * blob_bytes for k < *n (the staged copies `fetch_blobs` made). void rebase_ptrs(unsigned long long* ptr, const int32_t* n, uint8_t* base, int64_t blob_bytes, void* stream); /// The main GPU's share of a verify window's layer, decided on the device: the entries of `ids` (n = tokens * k, diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 26ec747e94f..735a4995278 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -101,6 +101,11 @@ bool AdaptiveTier::pump(uint64_t budget, uint64_t& sent, std::string& err) { batch_ = 0; } const bool timed = batch_ == 0 && n >= 4; // a batch of a few copies: its start and end dominate + // timed from when the work that may read its slots is done (pump_n waits for it too) + if (after_ != nullptr && cudaStreamWaitEvent(stream_, after_, 0) != cudaSuccess) { + err = "adaptive tier: a refill copy failed"; + return false; + } if (timed && cudaEventRecord(t0_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } if (!pump_n(n, err)) return false; if (timed) { diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index 4dfc00bafbb..7faa943ef6b 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -316,8 +316,10 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 } // The second GPU takes this layer's share only when the layer's misses are big enough that the CPU would need // longer for all of them than its round trip (~100 us: ~4 MB of experts at the pool's ~45 GB/s); otherwise it - // would only add its latency. - if (d.gpu2 != nullptr) { + // would only add its latency. Its rows reach the main GPU through the sink. + GpuPlanSink* const S2 = d.gpu2 != nullptr && d.plan != nullptr && d.plan->gpu2_flag != nullptr ? d.plan : nullptr; + d.gpu2_used = false; + if (S2 != nullptr) { uint64_t miss_bytes = 0, gpu2_bytes = 0; for (int q = 0; q < nd; ++q) { const int32_t e = ids[distinct[q]]; @@ -370,6 +372,15 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 if (first_of[i] == i0) kind[i] = kd; } g2_start[n2g] = n2e; + if (S2 != nullptr) { // the main GPU takes the second GPU's rows once their flag rises: at once without a share + S2->gpu2_list[0] = n2e; + for (int i = 0; i < n2e; ++i) S2->gpu2_list[4 + i] = g2_ent[i]; + if (n2g == 0) { + std::atomic_thread_fence(std::memory_order_seq_cst); + *(volatile uint32_t*) S2->gpu2_flag = S2->ring; + S2->gpu2_ring = S2->ring; + } + } if (P != nullptr) { // the PCIe groups: staging slot q, their entries in routing order const uint64_t bb = lay.blob_bytes(d.layers); int entries = 0; @@ -421,8 +432,8 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 } if (kind[i] >= 0) { // a GPU computes this entry (a VRAM hit, a PCIe read, the second GPU) if (kind[i] == 0) ++d.cache_hits; - if (kind[i] == 2) ++d.gpu2_entries; - if (!skip_gpu_rows) std::memset(row, 0, (size_t) H * sizeof(float)); + if (kind[i] == 2) ++d.gpu2_entries; // its row is the second GPU's to write + else if (!skip_gpu_rows) std::memset(row, 0, (size_t) H * sizeof(float)); continue; } ++d.cache_refused; @@ -451,18 +462,22 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 } const auto c3 = std::chrono::steady_clock::now(); // the second GPU's share goes out once the CPU's workers have started on theirs: its launch takes the host - // tens of microseconds + // tens of microseconds. It writes its rows into `out` itself. struct Submit2 { ExpertDispatch* d; const float* x; int n_tok, n_groups; int64_t k; const int32_t *slots, *starts, *entries; + float* out; bool ok; - } s2{&d, x_f, (int) n_tok, n2g, k, g2_slot, g2_start, g2_ent, true}; + } s2{&d, x_f, (int) n_tok, n2g, k, g2_slot, g2_start, g2_ent, out, true}; void (*submit2)(void*) = [](void* p) { Submit2& s = *(Submit2*) p; - s.ok = s.d->gpu2->submit(s.d->layers, s.x, s.n_tok, s.k, s.slots, s.starts, s.entries, s.n_groups, s.d->gpu2_err); + GpuPlanSink& S = *s.d->plan; + s.ok = s.d->gpu2->submit(s.d->layers, s.x, s.n_tok, s.k, s.slots, s.starts, s.entries, s.n_groups, s.out, + S.gpu2_flag, S.ring, s.d->gpu2_err); + if (s.ok) S.gpu2_ring = S.ring; }; if (n2g <= 0) submit2 = nullptr; pt("run", njobs); @@ -478,12 +493,6 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 d.fail_layer = d.layers; return; } - if (n2g > 0 && !d.gpu2->finish(out, d.gpu2_err)) { // the second GPU's rows, over the zeroed ones - d.failed = true; - d.fail = d.gpu2_err.c_str(); - d.fail_layer = d.layers; - return; - } const auto c4 = std::chrono::steady_clock::now(); pt("ran"); auto ms = [](auto a, auto b) { return std::chrono::duration(b - a).count(); }; diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp index 68adfae3a9a..6b673f97e1c 100644 --- a/src/core/second_gpu.cpp +++ b/src/core/second_gpu.cpp @@ -7,16 +7,21 @@ #include -#include #include #include namespace strata::core { namespace { -// the plan block: [n_groups, pad x3] [starts: cap+1] [dst: cap] [tok: cap] (8-byte aligned) [slot addresses: cap] -int64_t ptr_off(int64_t cap) { return (4 + (cap + 1) + 2 * cap + 1) / 2 * 2; } // in int32 units -size_t plan_bytes(int64_t cap) { return (size_t) ptr_off(cap) * 4 + (size_t) cap * 8; } +// the plan block: [groups of part 0, of part 1, ring, pad] [starts, part 0: cap+1] [part 1: cap+1] [dst: cap] +// [tok: cap] (8-byte aligned) [slot addresses, part 0: cap] [part 1: cap] [out] [flag after part 0] [after part 1] +struct PlanLayout { + int64_t starts[2], dst, tok, ptr; // int32 offsets; `ptr` even + size_t bytes; + explicit PlanLayout(int64_t cap) + : starts{4, 4 + cap + 1}, dst(4 + 2 * (cap + 1)), tok(dst + cap), ptr((tok + cap + 1) & ~1ll), + bytes((size_t) ptr * 4 + (size_t) (2 * cap + 3) * 8) {} +}; // the grouped kernels' grids are sized for the bucket a layer's group count falls in int bucket(int groups) { @@ -39,9 +44,9 @@ SecondGpu::~SecondGpu() { if (pre_s_) cudaStreamSynchronize(pre_s_); for (auto& gr : graphs_) cudaGraphExecDestroy(gr.exec); cache_.close(); - void* dev[] = {d_xq_, d_plan_, d_scratch_, d_rows_, d_pre_}; + void* dev[] = {d_xq_, d_plan_, d_scratch_, d_rows_, d_count_, d_pre_}; for (void* p : dev) if (p) cudaFree(p); - void* host[] = {h_x_, h_out_, h_plan_}; + void* host[] = {h_x_, h_plan_}; for (void* p : host) if (p) cudaFreeHost(p); if (ev_) cudaEventDestroy(ev_); if (pre_ev_) cudaEventDestroy(pre_ev_); @@ -58,30 +63,36 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, cap_ = (int64_t) max_t * k; DeviceScope scope(dev_, main_); const size_t xb = (size_t) max_t * (size_t) n_embd * sizeof(float), ob = (size_t) cap_ * (size_t) n_embd * sizeof(float); + const size_t pb = PlanLayout(cap_).bytes; const unsigned pm = cudaHostAllocPortable | cudaHostAllocMapped; const char* step = nullptr; cudaError_t e = cudaSuccess; + int unified = 0; auto run = [&](const char* what, cudaError_t r) { if (e == cudaSuccess && r != cudaSuccess) { e = r; step = what; } }; run("context", cudaFree(nullptr)); + run("unified addressing", cudaDeviceGetAttribute(&unified, cudaDevAttrUnifiedAddressing, dev_)); run("stream", cudaStreamCreateWithFlags(&s_, cudaStreamNonBlocking)); run("event", cudaEventCreateWithFlags(&ev_, cudaEventDisableTiming)); run("pinned staging", cudaHostAlloc((void**) &h_x_, xb, pm)); - run("pinned staging", cudaHostAlloc((void**) &h_out_, ob, pm)); - run("pinned staging", cudaHostAlloc((void**) &h_plan_, plan_bytes(cap_), pm)); + run("pinned staging", cudaHostAlloc((void**) &h_plan_, pb, pm)); run("mapped staging", cudaHostGetDevicePointer((void**) &m_x_, h_x_, 0)); - run("mapped staging", cudaHostGetDevicePointer((void**) &m_out_, h_out_, 0)); run("mapped staging", cudaHostGetDevicePointer((void**) &m_plan_, h_plan_, 0)); run("device buffers", cudaMalloc((void**) &d_xq_, (size_t) max_t * (size_t) (n_embd / 32) * 36)); - run("device buffers", cudaMalloc((void**) &d_plan_, plan_bytes(cap_))); + run("device buffers", cudaMalloc((void**) &d_plan_, pb)); run("device buffers", cudaMalloc((void**) &d_scratch_, strata::kernels::native_expert_scratch_bytes(cap_, n_ff))); run("device buffers", cudaMalloc((void**) &d_rows_, ob)); + run("device buffers", cudaMalloc((void**) &d_count_, sizeof(unsigned))); + run("device buffers", cudaMemset(d_count_, 0, sizeof(unsigned))); if (e != cudaSuccess) { err = std::string("second GPU: ") + step + ": " + cudaGetErrorString(e); return false; } + if (!unified) { // it writes its rows through the host pointers of the first GPU's mapped memory + err = "second GPU: no unified addressing"; + return false; + } std::memset(h_x_, 0, xb); - std::memset(h_plan_, 0, plan_bytes(cap_)); - pending_.reserve((size_t) cap_); + std::memset(h_plan_, 0, pb); return true; } @@ -121,25 +132,35 @@ bool SecondGpu::prefetch(int64_t layer, const int32_t* ids, const uint8_t* const return true; } -bool SecondGpu::graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& exec, std::string& err) { +bool SecondGpu::graph_for(int gu_type, int d_type, int groups, int part, bool prep, cudaGraphExec_t& exec, + std::string& err) { for (const auto& gr : graphs_) - if (gr.gu == gu_type && gr.d == d_type && gr.groups == groups) { exec = gr.exec; return true; } + if (gr.gu == gu_type && gr.d == d_type && gr.groups == groups && gr.part == part && gr.prep == prep) { + exec = gr.exec; + return true; + } const strata::kernels::NativeExpertLayout L = strata::kernels::native_expert_layout(gu_type, d_type, n_embd_, n_ff_); + const PlanLayout P(cap_); const auto* pi = (const int32_t*) d_plan_; + const auto* p64 = (const unsigned long long*) (pi + P.ptr); cudaGraph_t graph = nullptr; if (cudaStreamBeginCapture(s_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { err = "second GPU: cannot capture"; return false; } try { - // a kernel copies the plan (a copy node costs ~15 us on this link); the grouped kernels read the group - // count from it and skip the groups past it - strata::kernels::copy_i32_from_mapped((int32_t*) d_plan_, (const int32_t*) m_plan_, (int64_t) plan_bytes(cap_) / 4, s_); - strata::kernels::quantize_q8_1_rows(m_x_, max_t_, n_embd_, d_xq_, s_); - strata::kernels::native_expert_grouped(L, (const unsigned long long*) (pi + ptr_off(cap_)), pi + 4, pi, - pi + 4 + cap_ + 1, pi + 4 + 2 * cap_ + 1, groups, cap_, d_xq_, d_scratch_, - d_rows_, s_); - strata::kernels::native_expert_rows_out(d_rows_, pi, pi + 4, n_embd_, cap_, m_out_, s_); + if (prep) { + // a kernel copies the plan (a copy node costs ~15 us on this link); the grouped kernels read the group + // count from it and skip the groups past it + strata::kernels::copy_i32_from_mapped((int32_t*) d_plan_, (const int32_t*) m_plan_, (int64_t) P.bytes / 4, + s_); + strata::kernels::quantize_q8_1_rows(m_x_, max_t_, n_embd_, d_xq_, s_); + } + strata::kernels::native_expert_grouped(L, p64 + part * cap_, pi + P.starts[part], pi + part, pi + P.dst, + pi + P.tok, groups, cap_, d_xq_, d_scratch_, d_rows_, s_); + strata::kernels::native_expert_rows_out(d_rows_, pi + part, pi + P.starts[part], pi + P.dst, n_embd_, cap_, + (float* const*) (p64 + 2 * cap_), + (uint32_t* const*) (p64 + 2 * cap_ + 1 + part), pi + 2, d_count_, s_); } catch (const std::exception& ex) { cudaStreamEndCapture(s_, &graph); if (graph) cudaGraphDestroy(graph); @@ -155,43 +176,57 @@ bool SecondGpu::graph_for(int gu_type, int d_type, int groups, cudaGraphExec_t& cudaGraphDestroy(graph); cudaGraphUpload(exec, s_); cudaStreamSynchronize(s_); - graphs_.push_back({gu_type, d_type, groups, exec}); + graphs_.push_back({gu_type, d_type, groups, part, prep, exec}); return true; } bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, - const int32_t* entries, int n_groups, std::string& err) { + const int32_t* entries, int n_groups, float* out, uint32_t* flag, uint32_t ring, + std::string& err) { const int n = starts[n_groups]; if (n_groups <= 0 || n > cap_ || n_tok > max_t_) { err = "second GPU: a layer's share is out of range"; return false; } - // the plan: rows 0..n-1 in entry order - auto* pi = (int32_t*) h_plan_; - int32_t* start = pi + 4; - int32_t* dst = start + cap_ + 1; - int32_t* tok = dst + cap_; - auto* ptr = (unsigned long long*) (pi + ptr_off(cap_)); - pi[0] = n_groups; - pending_.assign(entries, entries + n); - for (int g = 0; g <= n_groups; ++g) start[g] = starts[g]; - for (int j = 0; j < n; ++j) { - dst[j] = j; - tok[j] = (int32_t) (entries[j] / k); + cudaError_t q; + while ((q = cudaEventQuery(ev_)) == cudaErrorNotReady) _mm_pause(); + if (q != cudaSuccess) { + err = std::string("second GPU: ") + cudaGetErrorString(q); + return false; } - bool pre = false; - for (int g = 0; g < n_groups; ++g) { - if (slots[g] <= -2) { // a prefetch slot - ptr[g] = (unsigned long long) (d_pre_ + (size_t) (-2 - slots[g]) * pre_cap_); - pre = true; - ++prefetch_used; - } else { - ptr[g] = (unsigned long long) cache_.device_slot(slots[g]); + // the plan: the experts in its cache first (part 0), then the prefetched ones (part 1), their entries in that order + const PlanLayout P(cap_); + auto* pi = (int32_t*) h_plan_; + auto* p64 = (unsigned long long*) (pi + P.ptr); + int ng[2] = {0, 0}, e = 0; + for (int part = 0; part < 2; ++part) { + int32_t* st = pi + P.starts[part]; + for (int g = 0; g < n_groups; ++g) { + const bool pre = slots[g] <= -2; + if (pre != (part == 1)) continue; + p64[(size_t) part * (size_t) cap_ + (size_t) ng[part]] = + pre ? (unsigned long long) (d_pre_ + (size_t) (-2 - slots[g]) * pre_cap_) + : (unsigned long long) cache_.device_slot(slots[g]); + st[ng[part]++] = e; + for (int j = starts[g]; j < starts[g + 1]; ++j, ++e) { + pi[P.dst + e] = entries[j]; + pi[P.tok + e] = (int32_t) (entries[j] / k); + } } + st[ng[part]] = e; + pi[part] = ng[part]; } + pi[2] = (int32_t) ring; + p64[2 * cap_] = (unsigned long long) out; + p64[2 * cap_ + 1] = ng[1] == 0 ? (unsigned long long) flag : 0; // the last launch raises it + p64[2 * cap_ + 2] = ng[1] > 0 ? (unsigned long long) flag : 0; std::memcpy(h_x_, x, (size_t) n_tok * (size_t) n_embd_ * sizeof(float)); const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; DeviceScope scope(dev_, main_); - cudaGraphExec_t exec = nullptr; - if (!graph_for(f.gu_type, f.d_type, bucket(n_groups), exec, err)) return false; - if ((pre && cudaStreamWaitEvent(s_, pre_ev_, 0) != cudaSuccess) || cudaGraphLaunch(exec, s_) != cudaSuccess || + cudaGraphExec_t own = nullptr, pre = nullptr; + if ((ng[0] > 0 && !graph_for(f.gu_type, f.d_type, bucket(ng[0]), 0, true, own, err)) || + (ng[1] > 0 && !graph_for(f.gu_type, f.d_type, bucket(ng[1]), 1, ng[0] == 0, pre, err))) + return false; + if ((own != nullptr && cudaGraphLaunch(own, s_) != cudaSuccess) || + (pre != nullptr && + (cudaStreamWaitEvent(s_, pre_ev_, 0) != cudaSuccess || cudaGraphLaunch(pre, s_) != cudaSuccess)) || cudaEventRecord(ev_, s_) != cudaSuccess) { err = std::string("second GPU: ") + cudaGetErrorString(cudaGetLastError()); return false; @@ -199,23 +234,15 @@ bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, cons ++layers; experts += n_groups; entries_done += n; + prefetch_used += ng[1]; return true; } -bool SecondGpu::finish(float* out, std::string& err) { - const auto t0 = std::chrono::steady_clock::now(); - cudaError_t q; - while ((q = cudaEventQuery(ev_)) == cudaErrorNotReady) _mm_pause(); - ms_wait += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); - if (q != cudaSuccess) { - err = std::string("second GPU: ") + cudaGetErrorString(q); - return false; - } - for (size_t j = 0; j < pending_.size(); ++j) - std::memcpy(out + (size_t) pending_[j] * (size_t) n_embd_, h_out_ + j * (size_t) n_embd_, - (size_t) n_embd_ * sizeof(float)); - pending_.clear(); - return true; +bool SecondGpu::healthy(std::string& err) const { + const cudaError_t q = cudaEventQuery(ev_); + if (q == cudaSuccess || q == cudaErrorNotReady) return true; + err = std::string("second GPU: ") + cudaGetErrorString(q); + return false; } } // namespace strata::core diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 05a35752794..9c452bb4fdd 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -58,8 +58,8 @@ struct Bump { } }; -bool mapped(size_t bytes, void** h, void** d) { - if (cudaHostAlloc(h, bytes, cudaHostAllocMapped) != cudaSuccess) return false; +bool mapped(size_t bytes, void** h, void** d, unsigned flags = cudaHostAllocMapped) { + if (cudaHostAlloc(h, bytes, flags) != cudaSuccess) return false; std::memset(*h, 0, bytes); return cudaHostGetDevicePointer(d, *h, 0) == cudaSuccess; } @@ -100,12 +100,15 @@ Verifier::~Verifier() { if (copy_) { cudaStreamSynchronize(copy_); cudaStreamDestroy(copy_); } if (side_) cudaStreamDestroy(side_); if (shs_) cudaStreamDestroy(shs_); - for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_}) + if (g2s_) cudaStreamDestroy(g2s_); + for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_, g2fork_, + g2join_[0], g2join_[1]}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, - h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_, h_res_}; + h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_, h_res_, h_flag2_, + h_list2_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -154,7 +157,9 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& const uint64_t TS = (uint64_t) (s.idx_block - 1) * ID; const int max_in = (int) std::max(std::max(N, ZV), NH * HD); - // ---- mapped staging + // ---- mapped staging (portable where the second GPU writes: its rows and flags) + const unsigned portable = cudaHostAllocMapped | cudaHostAllocPortable; + list2_stride_ = (4 + (int64_t) (T * K) + 3) & ~3ll; bool ok = mapped(T * 4, (void**) &h_tok_, (void**) &m_tok_) && mapped(T * strata::kernels::kStepCount * 4, (void**) &h_step_, (void**) &m_step_) && mapped(T * NH * 4, (void**) &h_pos_, (void**) &m_pos_) && @@ -168,7 +173,9 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(64, (void**) &h_flag_, (void**) &m_flag_) && mapped(64, (void**) &h_flagA_, (void**) &m_flagA_) && mapped(64, (void**) &h_flagB_, (void**) &m_flagB_) && - mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_) && + mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_, portable) && + mapped(128, (void**) &h_flag2_, (void**) &m_flag2_, portable) && + mapped((size_t) (2 * list2_stride_) * 4, (void**) &h_list2_, (void**) &m_list2_) && mapped(2 * T * K * 4, (void**) &h_pids_, (void**) &m_pids_) && mapped(2 * T * K * 4, (void**) &h_pw_, (void**) &m_pw_) && mapped(64, (void**) &h_pseq_, (void**) &m_pseq_) && @@ -235,6 +242,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& pbias_ = b.take((uint64_t) (g.n_layers * g.n_expert)); shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); + list2_ = b.take(2 * (uint64_t) list2_stride_); ring_count_ = b.take(2); plan_ = b.take(2 * (uint64_t) plan_stride_); dplan_ = b.take(2 * (uint64_t) plan_stride_); res_ = b.take((uint64_t) res_words_); slot_ptr_ = b.take((uint64_t) hits.slots); @@ -287,6 +295,10 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&side_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&g2s_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&g2fork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&g2join_[0], cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&g2join_[1], cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&bfork_, cudaEventDisableTiming) != cudaSuccess || @@ -704,7 +716,16 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { MT * K, plan_ptr_off_, cs); } stamp(l, 4, cs); - // after the doorbell: the shared expert's branch, and the next layer's prediction + // after the doorbell: the second GPU's rows of this token group, taken into VRAM once their flag rises; the + // shared expert's branch; and the next layer's prediction + if (gpu2_) { + if (!fork(g2s_, g2fork_)) return false; + wait_flag_ge(m_flag2_ + (size_t) grp * 16, (uint32_t) (l * G + grp + 1), g2s_); + fetch_listed_rows(m_list2_ + grp * list2_stride_, m_ymiss_ + (size_t) tb * K * N, + hit_out_ + (size_t) tb * K * N, list2_ + grp * list2_stride_, (int) (n * K), N, g2s_); + stamp(l, kLayerStamps + 1, g2s_); + if (!mark(g2s_, g2join_[grp])) return false; + } if (beside && !shared(l, grp)) return false; if (branches(l) && (cudaEventRecord(fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, fork_, 0) != cudaSuccess)) { @@ -807,15 +828,19 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 8, cs); wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows stamp(l, 9, cs); + if (gpu2_ && !wait(g2join_[grp])) return false; // the second GPU's, in `hit_out` + stamp(l, 10, cs); if (beside && cudaStreamWaitEvent(cs, shjoin_, 0) != cudaSuccess) { err = "verify: the shared expert's branch could not join"; return false; } if (native_moe_combine_enabled()) { - // only the host's rows cross PCIe; the GPU's come from `hit_out` (the pool leaves their rows unwritten) + // only the CPU's rows cross PCIe here; the GPUs' come from `hit_out` (the pool leaves their rows unwritten) + const int32_t* list2 = gpu2_ ? list2_ + grp * list2_stride_ : nullptr; try { native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, vram.dst, vram.counts + 1, pcie.dst, - pcie.counts != nullptr ? pcie.counts + 1 : nullptr, w_ + tb * K, shared_ + tb * N, + pcie.counts != nullptr ? pcie.counts + 1 : nullptr, + list2 != nullptr ? list2 + 4 : nullptr, list2, w_ + tb * K, shared_ + tb * N, bo_ + tb * N, N, K, n, cs); } catch (const std::exception& e) { err = "verify layer " + std::to_string(l) + " combine: " + e.what(); @@ -831,14 +856,14 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; } } - stamp(l, 10, cs); + stamp(l, 11, cs); if (branches(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { err = "verify: the prediction branch could not join"; return false; } if (l == g.n_layers - 1) for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); - stamp(l, 11, cs); + stamp(l, 12, cs); return true; }; @@ -1029,6 +1054,8 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *(volatile uint32_t*) h_flagB_ = 0; *(volatile uint32_t*) h_pseq_ = 0; *(volatile uint32_t*) h_pleflag_ = 0; + h_flag2_[0] = h_flag2_[16] = 0; + sink_.gpu2_ring = 0; std::atomic_thread_fence(std::memory_order_seq_cst); last_t_ = T; last_pos0_ = pos0; @@ -1042,6 +1069,15 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, } (void) cudaStreamQuery(cs_); VDBG("launched\n"); + // on a failure every wait of the graph passes, so it runs to its end before the next window resets the flags + auto release = [&] { + h_list2_[0] = h_list2_[list2_stride_] = 0; + std::atomic_thread_fence(std::memory_order_seq_cst); + for (uint32_t* f : {h_flag_, h_flagA_, h_flagB_, h_pleflag_, h_flag2_, h_flag2_ + 16}) + *(volatile uint32_t*) f = 0xffffffffu; + cudaStreamSynchronize(cs_); + return false; + }; if (ple_on) { // before layer 1: the graph waits for the flag, raised even when a read failed const Clock::time_point tp = Clock::now(); bool ok = true; @@ -1050,12 +1086,22 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, _mm_sfence(); *(volatile uint32_t*) h_pleflag_ = 1; ms_host += ms_since(tp); - if (!ok) return false; + if (!ok) return release(); } volatile uint32_t* const seq = h_seq_; volatile uint32_t* const flag = h_flag_; const int G = groups_[T] > 0 ? groups_[T] : 1; const int gtb[2] = {0, (T + 1) / 2}, gte[2] = {G == 2 ? (T + 1) / 2 : T, T}; + // a pool whose second-GPU rows came after the CPU's: the host time until they were in + const volatile uint32_t* late_flag = nullptr; + uint32_t late_want = 0; + Clock::time_point late_at{}; + auto settle = [&](bool now) { + if (late_flag != nullptr && (now || *late_flag >= late_want)) { + ms_late2 += ms_since(late_at); + late_flag = nullptr; + } + }; for (int64_t k = 0; k < g.n_layers * G; ++k) { const int64_t l = k / G; const int grp = (int) (k % G); @@ -1064,6 +1110,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, auto last_flush = a; uint32_t spins = 0; while (*seq < want) { + settle(false); _mm_pause(); if ((++spins & 1023u) != 0) continue; const auto now = Clock::now(); @@ -1073,10 +1120,14 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, if (q != cudaErrorNotReady && *seq < want) { err = "verify: layer " + std::to_string(l) + " never rang (" + (q == cudaSuccess ? std::string("graph finished") : std::string(cudaGetErrorString(q))) + ")"; - return false; + return release(); } + if (watch_ != nullptr && !watch_(watch_user_, err)) return release(); + } + if (now - a > std::chrono::seconds(20)) { + err = "verify: timed out at layer " + std::to_string(l); + return release(); } - if (now - a > std::chrono::seconds(20)) { err = "verify: timed out at layer " + std::to_string(l); return false; } } const Clock::time_point b = Clock::now(); VDBG("layer %lld rang\n", (long long) l); @@ -1098,9 +1149,21 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, *(volatile uint32_t*) h_flagA_ = want; raise_flag(h_flagB_, want); } + if (gpu2_ && sink_.gpu2_ring != want) { // no second-GPU share went out (the pool failed): an empty one + sink_.gpu2_list[0] = 0; + std::atomic_thread_fence(std::memory_order_seq_cst); + *(volatile uint32_t*) sink_.gpu2_flag = want; + } *flag = want; ms_wait += std::chrono::duration(b - a).count(); ms_pool += ms_since(b); + if (gpu2_ && *(volatile uint32_t*) sink_.gpu2_flag < want) { + settle(true); // a split window's other token group + late_flag = sink_.gpu2_flag; + late_want = want; + late_at = Clock::now(); + ++late2; + } if (tail_ != nullptr && k + 1 == g.n_layers * G) tail_(tail_user_); // the next layer's prediction, published during this layer's pool, else microseconds later (its branch is // joined before the next layer): waited for, so the experts the second GPU takes never depend on timing @@ -1108,8 +1171,11 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, const Clock::time_point c = Clock::now(); if (*(volatile uint32_t*) h_pseq_ < (uint32_t) (l + 1)) { ++predict_late; - while (*(volatile uint32_t*) h_pseq_ < (uint32_t) (l + 1) && Clock::now() - c < std::chrono::milliseconds(50)) + while (*(volatile uint32_t*) h_pseq_ < (uint32_t) (l + 1) && + Clock::now() - c < std::chrono::milliseconds(50)) { + settle(false); _mm_pause(); + } } if (*(volatile uint32_t*) h_pseq_ >= (uint32_t) (l + 1)) { std::atomic_thread_fence(std::memory_order_acquire); @@ -1119,6 +1185,11 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, ms_predict += ms_since(c); } } + while (late_flag != nullptr && ms_since(late_at) < 50.0) { // the last layer's (the graph ends after them) + settle(false); + _mm_pause(); + } + settle(true); const cudaError_t se = cudaStreamSynchronize(cs_); if (se != cudaSuccess) { err = std::string("verify: ") + cudaGetErrorString(se); return false; } cudaStreamSynchronize(copy_); // no host function of this window may raise flag B in the next one @@ -1136,8 +1207,9 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, const unsigned long long before = l == 0 ? w[1] : (st - kStamps)[kLayerStamps - 1]; prof_ns_[(size_t) kind * kStamps] += (double) (st[0] - before); for (int i = 1; i < kLayerStamps; ++i) prof_ns_[(size_t) kind * kStamps + i] += (double) (st[i] - st[i - 1]); - // the shared expert's branch, from the doorbell (it forks there) to its end + // the shared expert's branch and the second GPU's rows', from the doorbell (they fork there) to their ends prof_ns_[(size_t) kind * kStamps + kLayerStamps] += (double) (st[kLayerStamps] - st[4]); + if (gpu2_) prof_ns_[(size_t) kind * kStamps + kLayerStamps + 1] += (double) (st[kLayerStamps + 1] - st[4]); { // the wait for the CPU's rows, by bucket const double us = (double) (st[9] - st[8]) / 1e3; int b = 0; @@ -1171,12 +1243,13 @@ void Verifier::print_profile() const { if (prof_windows_ == 0) return; static const char* const names[kStamps] = {"stamp gap", "HC read, mixer", "mixer", "HC read, FFN", "router, ring, hits", "expert inputs", "VRAM experts", - "wait for PCIe plan", "PCIe experts", "wait for CPU", "rows + combine", - "last HC write", "shared expert, beside"}; + "wait for PCIe plan", "PCIe experts", "wait for CPU", + "wait for 2nd GPU", "rows + combine", "last HC write", + "shared expert, beside", "2nd GPU rows, beside"}; const double nw = (double) prof_windows_; std::printf("%-24s GPU time over %lld windows, ms per window: inputs %.3f, head %.3f, whole window %.3f; " - "one stamp %.2f us (each stage below includes one; the last runs from the ring beside the " - "five before \"rows + combine\")\n", "window profile", (long long) prof_windows_, + "one stamp %.2f us (each stage below includes one; the last two run from the ring beside the " + "six before \"rows + combine\")\n", "window profile", (long long) prof_windows_, prof_window_ns_[0] / nw / 1e6, prof_window_ns_[1] / nw / 1e6, prof_window_ns_[2] / nw / 1e6, prof_window_ns_[3] / nw / 1e3); std::printf(" %-22s %12s %12s %14s\n", "stage", "GDN us/layer", "QSA us/layer", "ms per window"); @@ -1218,6 +1291,11 @@ void Verifier::set_plan_slot(int grp) { const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs; sink_.staging = (unsigned long long) (staging_ + (size_t) (grp * per) * strata::kernels::cpu::expert_layout().max_blob); sink_.staging_cap = per; + if (gpu2_) { + sink_.gpu2_list = h_list2_ + (size_t) grp * (size_t) list2_stride_; + sink_.gpu2_flag = h_flag2_ + (size_t) grp * 16; + sink_.ring = cur_layer_ + 1; + } } // Flag B only rises: a host function of an earlier layer may run after a later layer already raised it directly. diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index 0013b1c132c..b551926bcdb 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -182,10 +182,27 @@ __global__ void __launch_bounds__(DOWN_WARPS * 32) native_down_kernel(const unsi } __global__ void rows_out_kernel(const float4* __restrict__ src, const int32_t* __restrict__ n_groups, - const int32_t* __restrict__ grp_start, long long n4, float4* __restrict__ dst) { - const long long total = (long long) grp_start[*n_groups] * n4; - for (long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; i < total; i += (long long) gridDim.x * blockDim.x) - dst[i] = src[i]; + const int32_t* __restrict__ grp_start, const int32_t* __restrict__ ent_dst, + long long n4, float4* const* out, uint32_t* const* flag, const int32_t* ring, + unsigned* counter) { + const int e0 = grp_start[0]; + const long long total = (long long) (grp_start[*n_groups] - e0) * n4, step = (long long) gridDim.x * blockDim.x; + float4* o = *out; + for (long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; i < total; i += step) { + const long long at = (long long) ent_dst[e0 + i / n4] * n4 + i % n4; + o[at] = src[at]; + } + uint32_t* f = *flag; + if (f == nullptr) return; + __shared__ bool last; + __threadfence_system(); // this thread's rows before the block counts as done + __syncthreads(); + if (threadIdx.x == 0) last = atomicAdd(counter, 1u) == gridDim.x - 1; + __syncthreads(); + if (last && threadIdx.x == 0) { + *counter = 0u; // for the next launch + *(volatile uint32_t*) f = (uint32_t) *ring; + } } // ---------------------------------------------------------------- q8_1 (quantize.cu) @@ -697,11 +714,13 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long check("native_expert_grouped/down"); } -void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, int64_t n_embd, - int64_t max_rows, float* out, void* stream) { +void native_expert_rows_out(const float* rows, const int32_t* n_groups, const int32_t* grp_start, + const int32_t* ent_dst, int64_t n_embd, int64_t max_rows, float* const* out, + uint32_t* const* flag, const int32_t* ring, unsigned* counter, void* stream) { const long long n4 = n_embd / 4, most = max_rows * n4; const unsigned blocks = (unsigned) std::min((most + 255) / 256, 256); - rows_out_kernel<<>>((const float4*) rows, n_groups, grp_start, n4, (float4*) out); + rows_out_kernel<<>>((const float4*) rows, n_groups, grp_start, ent_dst, n4, + (float4* const*) out, flag, ring, counter); check("native_expert_rows_out"); } diff --git a/src/kernels/cuda/native_moe.cu b/src/kernels/cuda/native_moe.cu index f9144ba9a29..d84356b5d5d 100644 --- a/src/kernels/cuda/native_moe.cu +++ b/src/kernels/cuda/native_moe.cu @@ -49,28 +49,31 @@ constexpr int kMaxK = 15; __global__ void gather_combine(const float* __restrict__ gpu_rows, const float* host_rows, const int32_t* __restrict__ dst, const int32_t* __restrict__ count, const int32_t* __restrict__ dst2, const int32_t* __restrict__ count2, + const int32_t* __restrict__ dst3, const int32_t* __restrict__ count3, const float* __restrict__ weights, const float* __restrict__ shared, float* __restrict__ output, int64_t n_embd, int k) { - __shared__ unsigned on_gpu; // bit j: row t*k + j is the GPU's + __shared__ unsigned on_gpu, as_is; // bit j: row t*k + j is in gpu_rows, a hit / a row taken as it is const int t = blockIdx.y; - if (threadIdx.x == 0) on_gpu = 0u; + if (threadIdx.x == 0) on_gpu = as_is = 0u; __syncthreads(); - const int c = *count, c2 = dst2 != nullptr ? *count2 : 0; - for (int i = threadIdx.x; i < c + c2; i += blockDim.x) { - const int r = (i < c ? dst[i] : dst2[i - c]) - t * k; - if (r >= 0 && r < k) atomicOr(&on_gpu, 1u << r); + const int c = *count, c2 = dst2 != nullptr ? *count2 : 0, c3 = dst3 != nullptr ? *count3 : 0; + for (int i = threadIdx.x; i < c + c2 + c3; i += blockDim.x) { + const int r = (i < c ? dst[i] : i < c + c2 ? dst2[i - c] : dst3[i - c - c2]) - t * k; + if (r >= 0 && r < k) atomicOr(i < c + c2 ? &on_gpu : &as_is, 1u << r); } __syncthreads(); const int64_t col = int64_t(blockIdx.x) * blockDim.x + threadIdx.x; if (col >= n_embd) return; - const unsigned mask = on_gpu; + const unsigned mask = on_gpu, same = as_is; const float* w = weights + int64_t(t) * k; float v[kMaxK]; #pragma unroll for (int j = 0; j < kMaxK; ++j) { if (j >= k) break; const int64_t at = (int64_t(t) * k + j) * n_embd + col; - if ((mask >> j) & 1u) { + if ((same >> j) & 1u) { + v[j] = gpu_rows[at]; + } else if ((mask >> j) & 1u) { const float h = gpu_rows[at]; v[j] = __float_as_uint(h) == 0x80000000u ? 0.0f : h; // 0 + h: the host's zeroed row plus the hit } else { @@ -115,15 +118,18 @@ void native_moe_combine(const float* parts, const float* weights, const float* s if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, - const int32_t* count, const int32_t* dst2, const int32_t* count2, const float* weights, - const float* shared, float* output, int64_t n_embd, int64_t k, int n_tok, void* stream) { + const int32_t* count, const int32_t* dst2, const int32_t* count2, const int32_t* dst3, + const int32_t* count3, const float* weights, const float* shared, float* output, + int64_t n_embd, int64_t k, int n_tok, void* stream) { if (!stream || n_embd <= 0 || n_embd > std::numeric_limits::max() || k < 1 || k > kMaxK || n_tok < 1 || - !gpu_rows || !host_rows || !dst || !count || (dst2 != nullptr && count2 == nullptr) || !weights || !output) + !gpu_rows || !host_rows || !dst || !count || (dst2 != nullptr && count2 == nullptr) || + (dst3 != nullptr && count3 == nullptr) || !weights || !output) throw std::invalid_argument("native MoE gather-combine requires a stream, positive width, 1..15 experts " "and its buffers"); const dim3 grid(unsigned((n_embd + 255) / 256), unsigned(n_tok)); gather_combine<<(stream)>>>(gpu_rows, host_rows, dst, count, dst2, count2, - weights, shared, output, n_embd, int(k)); + dst3, count3, weights, shared, output, n_embd, + int(k)); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index 41d8a78dc2e..03fe04fb929 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -225,6 +225,26 @@ __global__ void fetch_blobs_kernel(const unsigned long long* __restrict__ src, c } } +// block (x, i): part x of listed row i, block (0, i) also its entry of the device list +__global__ void fetch_listed_rows_kernel(const int32_t* list, const float4* src, float4* __restrict__ dst, + int32_t* __restrict__ list_dev, int cap, long long n4) { + const int n = min(list[0], cap), i = (int) blockIdx.y; + const bool first = blockIdx.x == 0 && threadIdx.x == 0; + if (i >= n) { + if (i == 0 && first) list_dev[0] = 0; + return; + } + const int r = list[4 + i]; + if (first) { + list_dev[4 + i] = r; + if (i == 0) list_dev[0] = n; + } + if (r < 0 || r >= cap) return; + const long long step = (long long) gridDim.x * blockDim.x; + for (long long c = (long long) blockIdx.x * blockDim.x + threadIdx.x; c < n4; c += step) + dst[r * n4 + c] = src[r * n4 + c]; +} + __global__ void rebase_ptrs_kernel(unsigned long long* ptr, const int32_t* n, unsigned long long base, long long bytes) { const int k = threadIdx.x; if (k < *n) ptr[k] = base + (unsigned long long) k * (unsigned long long) bytes; @@ -317,6 +337,15 @@ void fetch_blobs(const unsigned long long* src, const int32_t* n, uint8_t* dst, check("fetch_blobs"); } +void fetch_listed_rows(const int32_t* list, const float* src, float* dst, int32_t* list_dev, int cap, int64_t n_embd, + void* stream) { + if (cap <= 0) return; + const long long n4 = n_embd / 4; + fetch_listed_rows_kernel<<>>( + list, (const float4*) src, (float4*) dst, list_dev, cap, n4); + check("fetch_listed_rows"); +} + void rebase_ptrs(unsigned long long* ptr, const int32_t* n, uint8_t* base, int64_t blob_bytes, void* stream) { rebase_ptrs_kernel<<<1, 128, 0, (cudaStream_t) stream>>>(ptr, n, (unsigned long long) base, (long long) blob_bytes); check("rebase_ptrs"); diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index e7898b2e4ee..6a8a5ca57e4 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -78,7 +78,8 @@ int bitwise_diff(const std::vector& a, const std::vector& b, const return bad; } -// ---- the MoE combine: copy all rows + add the hits + one combine per token vs native_moe_gather_combine +// ---- the MoE combine: copy all rows + add the hits + one combine per token vs the second GPU's rows taken into VRAM +// (fetch_listed_rows) and native_moe_gather_combine int test_gather_combine(std::mt19937& rng, cudaStream_t s) { const int64_t N = 2560, K = 10; int bad = 0; @@ -91,23 +92,35 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { for (auto& v : host) v = edgy(rng); for (auto& v : w) v = std::uniform_real_distribution(0.0f, 1.0f)(rng); for (auto& v : shared) v = edgy(rng); - // the GPU's entries, in a random order and a random share (none, some, all) - std::vector dst; - for (int32_t r = 0; r < rows; ++r) + // the GPU's entries, in a random order and a random share (none, some, all); from rep 4 on the second GPU + // takes some of the rest + std::vector dst, raw; + for (int32_t r = 0; r < rows; ++r) { if (rep == 1 || (rep != 0 && rng() % 3 != 0)) dst.push_back(r); + else if (rep >= 4 && rng() % 2 == 0) raw.push_back(r); + } std::shuffle(dst.begin(), dst.end(), rng); + std::shuffle(raw.begin(), raw.end(), rng); const int32_t count = (int32_t) dst.size(); - // the old path's host rows: the pool zeroed the GPU's + // the old path's host rows: the pool zeroed the GPU's, the host copied the second GPU's in std::vector host_zeroed = host; for (int32_t r : dst) std::fill(host_zeroed.begin() + r * N, host_zeroed.begin() + (r + 1) * N, 0.0f); - // the new path's mapped rows: the GPU's rows hold garbage it must not read - std::vector host_poison = host; + for (int32_t r : raw) + std::copy(hit.begin() + r * N, hit.begin() + (r + 1) * N, host_zeroed.begin() + r * N); + // the new path's mapped rows: the GPU's rows hold garbage it must not read, the second GPU wrote its own; + // its device rows hold garbage until they are fetched + std::vector host_poison = host, hit_poison = hit; for (int32_t r : dst) std::fill(host_poison.begin() + r * N, host_poison.begin() + (r + 1) * N, NAN); + for (int32_t r : raw) { + std::copy(hit.begin() + r * N, hit.begin() + (r + 1) * N, host_poison.begin() + r * N); + std::fill(hit_poison.begin() + r * N, hit_poison.begin() + (r + 1) * N, NAN); + } float *d_hit = dev(hit.size()), *d_parts = dev(hit.size()), *d_w = dev(w.size()), *d_sh = dev(shared.size()), *d_out_old = dev(shared.size()), *d_out_new = dev(shared.size()); - int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(3); + int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(3), + *d_list = dev((size_t) (4 + rows)); up(d_hit, hit); up(d_w, w); up(d_sh, shared); @@ -115,10 +128,15 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { // the new path takes them as two lists (the window's VRAM share and the pool's PCIe share), split at random const int32_t split = rep % 2 == 0 ? count : (int32_t) (rng() % (dst.size() + 1)); up(d_count, std::vector{count, split, count - split}); - float* h_map = nullptr; - float* m_map = nullptr; + check(cudaMemset(d_list, 0xff, (size_t) (4 + rows) * 4), "memset"); + float *h_map = nullptr, *m_map = nullptr; + int32_t *h_list = nullptr, *m_list = nullptr; check(cudaHostAlloc((void**) &h_map, host.size() * sizeof(float), cudaHostAllocMapped), "cudaHostAlloc"); check(cudaHostGetDevicePointer((void**) &m_map, h_map, 0), "cudaHostGetDevicePointer"); + check(cudaHostAlloc((void**) &h_list, (size_t) (4 + rows) * 4, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostGetDevicePointer((void**) &m_list, h_list, 0), "cudaHostGetDevicePointer"); + h_list[0] = (int32_t) raw.size(); + std::copy(raw.begin(), raw.end(), h_list + 4); // old: copy all rows, add the hits, combine per token std::memcpy(h_map, host_zeroed.data(), host.size() * sizeof(float)); @@ -130,19 +148,37 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { check(cudaStreamSynchronize(s), "old path"); // new std::memcpy(h_map, host_poison.data(), host.size() * sizeof(float)); - strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count + 1, rep % 2 == 0 ? nullptr : d_dst + split, - d_count + 2, d_w, d_sh, d_out_new, N, K, n_tok, s); + up(d_hit, hit_poison); + const bool second = rep >= 4; + if (second) strata::kernels::fetch_listed_rows(m_list, m_map, d_hit, d_list, (int) rows, N, s); + strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count + 1, + rep % 2 == 0 ? nullptr : d_dst + split, d_count + 2, + second ? d_list + 4 : nullptr, second ? d_list : nullptr, d_w, + d_sh, d_out_new, N, K, n_tok, s); check(cudaStreamSynchronize(s), "new path"); - const int b = bitwise_diff(down(d_out_old, shared.size()), down(d_out_new, shared.size()), "gather_combine"); + int b = bitwise_diff(down(d_out_old, shared.size()), down(d_out_new, shared.size()), "gather_combine"); + if (second) { + std::vector want((size_t) (4 + rows), -1), got = down(d_list, (size_t) (4 + rows)); + want[0] = (int32_t) raw.size(); + std::copy(raw.begin(), raw.end(), want.begin() + 4); + for (int i = 1; i < 4; ++i) got[(size_t) i] = -1; + if (got != want) { + std::fprintf(stderr, "fetch_listed_rows: n_tok %d rep %d: the device list differs\n", n_tok, rep); + ++b; + } + } if (b) std::fprintf(stderr, "gather_combine: n_tok %d rep %d: %d of %zu differ\n", n_tok, rep, b, shared.size()); bad += b; cudaFreeHost(h_map); + cudaFreeHost(h_list); for (void* p : {(void*) d_hit, (void*) d_parts, (void*) d_w, (void*) d_sh, (void*) d_out_old, - (void*) d_out_new, (void*) d_dst, (void*) d_count}) + (void*) d_out_new, (void*) d_dst, (void*) d_count, (void*) d_list}) cudaFree(p); } } - std::printf("gather_combine: %s\n", bad ? "MISMATCH" : "bitwise equal (n_tok 1-4, 8; 8 plans each, one or two lists)"); + std::printf("gather_combine: %s\n", + bad ? "MISMATCH" + : "bitwise equal (n_tok 1-4, 8; 8 plans each, one or two lists, the second GPU's rows fetched)"); return bad; } diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 1b0a34ac7d3..e0db948ef0a 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -526,6 +526,12 @@ bool pump_gap(Drive* t, std::string& err, bool all = false) { return true; } +/// Whether the second GPU's work so far succeeded (Verifier::set_watch). +bool drive_watch(void* user, std::string& err) { + Drive* t = (Drive*) user; + return t->d.gpu2 == nullptr || t->d.gpu2->healthy(err); +} + /// A window's last CPU rows are in (Verifier::set_tail). void drive_tail(void* user) { Drive* t = (Drive*) user; @@ -2190,6 +2196,7 @@ int main(int argc, char** argv) { // lowest slots first: the last ones, which the prompt path borrows, stay empty longest for (size_t i = empty2.size(); i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); tier2.set_upper(&tier); + tier2.set_after(gpu2.done_event()); // a window's last share may still read its slots if (tier2.set_paced()) drive.tier2 = &tier2; // its copies between windows (Drive::tier2) } } @@ -2338,6 +2345,10 @@ int main(int argc, char** argv) { : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive, o.adapt_every > 0 && o.adapt_swaps > 0); + if (drive.d.gpu2 != nullptr) { + ver.set_gpu2(true); + ver.set_watch(&drive_watch, &drive); + } if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache ver.set_tail(&drive_tail, &drive); tier.set_after(ver.window_done()); @@ -3050,6 +3061,10 @@ int main(int argc, char** argv) { : native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0 if (drive.d.gpu2 != nullptr && gpu2.prefetch_slots() > 0) ver.set_predict(&drive_predict, &drive, o.adapt_every > 0 && o.adapt_swaps > 0); + if (drive.d.gpu2 != nullptr) { + ver.set_gpu2(true); + ver.set_watch(&drive_watch, &drive); + } if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache ver.set_tail(&drive_tail, &drive); tier.set_after(ver.window_done()); @@ -3266,13 +3281,14 @@ int main(int argc, char** argv) { (long long) tier.swaps, o.adapt_every, tier.ms / rounds); if (rounds > 0 && drive.d.gpu2 != nullptr) std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " - "%.3f ms/round waiting for it after the CPU pool; %lld experts swapped in and %lld into empty " - "slots (%lld still empty, %lld moves dropped; its copies between windows, %.2f ms, at %.1f " - "GB/s); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round of host time, %lld " - "predictions waited for)\n", "second GPU", + "its rows after the CPU's in %.2f layers, %.3f ms/round; %lld experts swapped in and %lld into " + "empty slots (%lld still empty, %lld moves dropped; its copies between windows, %.2f ms, at " + "%.1f GB/s); %.2f experts per round prefetched, %.2f of them used (%.3f ms/round of host " + "time, %lld predictions waited for)\n", "second GPU", (double) gpu2.entries_done / (double) rounds, (double) gpu2.experts / (double) rounds, (double) gpu2.layers / (double) rounds, (double) drive.d.gpu2_skipped / (double) rounds, - gpu2.ms_wait / (double) rounds, (long long) tier2.swaps, (long long) tier2.fills, + (double) ver.late2 / (double) rounds, ver.ms_late2 / (double) rounds, + (long long) tier2.swaps, (long long) tier2.fills, (long long) tier2.free_slots(), (long long) tier2.dropped, drive.gap_ms, tier2.copy_rate() / 1e6, (double) gpu2.prefetch_copied / (double) rounds, (double) gpu2.prefetch_used / (double) rounds, ver.ms_predict / (double) rounds, From d9ce223164a735b00f5a1c9d9dba348a41140194 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 08:15:21 +0200 Subject: [PATCH 64/92] Verify window: the router kernel writes the FFN input as the pool's Q8_K rows and the GPUs' q8_1 rows The router kernel copied each layer's FFN input to mapped memory as floats (the router stage's long pole), and the CPU pool quantized it to Q8_K (ggml's quantize_row_q8_K, before its workers start: 0.27-0.65 ms a round); the host copied the floats again for the second GPU, whose graph quantized them to q8_1, and a branch beside the router quantized them to q8_1 for the 3090's own experts. Now the router kernel's first blocks, four superblocks of 256 values each, write the q8_1 rows (quantize_q8_1_rows' arithmetic) to the device for the 3090's experts and to mapped memory for the second GPU, and ggml's Q8_K rows (quantize_row_q8_K_ref) to mapped memory for the pool, bitwise: their float operations are PTX add/mul/div.rn, which the file's fast-math flags do not change. Everything leaves in 16-byte stores: with a block per superblock, its 292-byte Q8_K block written 4 bytes at a time, the kernel was 1.6 us slower than the float copy at 4 tokens; four superblocks are whole 16-byte chunks. The pool takes the Q8_K rows of the layers whose gate/up activation is Q8_K (pool_takes_q8k: every layer of both model files), the floats otherwise; the second GPU copies the q8_1 rows in. The kernel alone, in graphs (scratch/router_rows_bench.cu), us at 1 / 2 / 3 / 4 / 8 tokens: the float copy 13.7 / 15.2 / 17.3 / 18.3 / 31.8, the quantized rows 14.2 / 15.4 / 16.6 / 17.5 / 24.7. Window hashes identical to the previous commit (--adapt-every 0): thinking text on both models, code text with and without --spec-split. verify_parity: the q8_1 rows bitwise quantize_q8_1_rows', the Q8_K rows ggml's own (native_quant_act), with denormals, zero blocks and tied maxima. Fixed text, both GPUs, against the previous commit, ms per round (t/s): code 4 rounds, UD-Q4_K_XL -1.9% (+1.7%; +0.6% without the base's slow first run), IQ3_XXS -0.2% (+0.4%); thinking and after 16K 4 rounds, UD-Q4_K_XL +0.3% (-0.1%) and -0.2% (+0.4%), IQ3_XXS +0.4% (-0.3%) and +1.2% (-0.3%, more tokens a pass). The pool's call is shorter by the quantization (0.27-0.65 ms a round), which reaches the round only where the CPU is the later side; where the 3090 is (IQ3_XXS thinking) the host waits that much longer for the next ring. Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- CMakeLists.txt | 2 +- include/strata/core/expert_source.hpp | 10 ++ include/strata/core/second_gpu.hpp | 17 +-- include/strata/core/verify.hpp | 11 +- include/strata/kernels/native_router.hpp | 9 +- src/core/expert_source.cpp | 19 ++-- src/core/second_gpu.cpp | 19 ++-- src/core/verify.cpp | 46 ++++++-- src/kernels/cuda/native_router.cu | 135 +++++++++++++++++++++-- src/kernels/verify_parity.cpp | 100 +++++++++++++++++ 10 files changed, 321 insertions(+), 47 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 450e6bb1ca1..932e93a09ea 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -365,7 +365,7 @@ if(STRATA_ENABLE_CUDA) # The verify window's batched kernels vs the per-token kernel sequences they replace, bitwise (synthetic data). add_executable(verify_parity src/kernels/verify_parity.cpp) target_include_directories(verify_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) - target_link_libraries(verify_parity PRIVATE strata_kernels strata_prefill CUDA::cudart) + target_link_libraries(verify_parity PRIVATE strata_kernels strata_prefill strata_kernels_cpu CUDA::cudart) add_test(NAME verify_parity COMMAND verify_parity --selftest) # The QSA block scores and top-k against the kernels they replaced, kept in the test: bitwise (synthetic data). diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index bbc1a61d14a..63354985455 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -96,8 +96,18 @@ struct GpuPlanSink { int32_t* gpu2_list = nullptr; uint32_t* gpu2_flag = nullptr; uint32_t ring = 0, gpu2_ring = 0; + /// This token group's activations as the window's router wrote them (mapped): ggml's Q8_K rows, `xk_stride` bytes + /// apart, when the layer takes them (`pool_takes_q8k`), else null (the pool quantizes the floats); and the q8_1 + /// rows (n_embd / 32 blocks each) the second GPU takes. + const uint8_t* xk = nullptr; + int64_t xk_stride = 0; + const uint8_t* x1 = nullptr; }; +/// Whether the pool takes a native layer's gate/up activation as ggml's Q8_K rows (`GpuPlanSink::xk`): its format's +/// vec_dot_type (the i-quants', the K-quants'), and not the Q2_0 kernels' own. +inline bool pool_takes_q8k(const strata::kernels::cpu::NativeFmt& f) { return f.gu_type != 42 && f.gu_act == 15; } + /// The adapter's own state. One per session, reused every layer so the token path allocates nothing (P2.T10). struct ExpertDispatch { strata::kernels::cpu::ExpertPool* pool = nullptr; diff --git a/include/strata/core/second_gpu.hpp b/include/strata/core/second_gpu.hpp index 796a39e9407..7348bc2494d 100644 --- a/include/strata/core/second_gpu.hpp +++ b/include/strata/core/second_gpu.hpp @@ -3,7 +3,7 @@ // It holds experts the first GPU's cache does not (the profile's next ranks, then the conversation's, through its // own AdaptiveTier) and computes them for each layer of a verify window while the CPU pool computes the misses. // It never spins on host memory: per layer the host launches captured graphs once the CPU pool's workers have -// started (copy the plan in, quantize the activations, the grouped experts, their rows out), and the last one writes +// started (copy the plan and the activations in, the grouped experts, their rows out), and the last one writes // the rows into the window's own rows in mapped host memory and raises a flag there, which the first GPU waits for; // so it keeps its latency on a GPU that also drives a display (a graph spinning there stalls whenever the desktop // draws). The rows cross PCIe in 16-byte stores: written one float at a time they took ~0.2 ms a layer and slowed the @@ -53,10 +53,11 @@ class SecondGpu { } /// One layer's share: group g is the expert in slot `slots[g]`, serving the routed entries - /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations. Entry - /// i's row goes to row i of `out` (n_embd floats, mapped host memory), and then `*flag` (mapped) is set to `ring`. - /// Waits for the last share's graphs first (the plan and staging are theirs until then). - bool submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, + /// `entries[starts[g] .. starts[g+1])` (entry i = token i/k). `x` holds the window's n_tok activations as q8_1 + /// rows (n_embd / 32 blocks each). Entry i's row goes to row i of `out` (n_embd floats, mapped host memory), and + /// then `*flag` (mapped) is set to `ring`. Waits for the last share's graphs first (the plan and staging are + /// theirs until then). + bool submit(int64_t layer, const uint8_t* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, const int32_t* entries, int n_groups, float* out, uint32_t* flag, uint32_t ring, std::string& err); /// False, with the reason, when its work so far failed; does not wait. bool healthy(std::string& err) const; @@ -74,14 +75,14 @@ class SecondGpu { ExpertCache cache_; cudaStream_t s_ = nullptr; cudaEvent_t ev_ = nullptr; - // pinned staging (portable, mapped): the activations and the plan - float *h_x_ = nullptr, *m_x_ = nullptr; + // pinned staging (portable, mapped): the activations (q8_1 rows) and the plan + uint8_t *h_x_ = nullptr, *m_x_ = nullptr; uint8_t *h_plan_ = nullptr, *m_plan_ = nullptr; uint8_t *d_xq_ = nullptr, *d_plan_ = nullptr, *d_scratch_ = nullptr; float* d_rows_ = nullptr; // the rows, then copied out in full transactions unsigned* d_count_ = nullptr; // the rows-out kernel's block counter // part 0: the experts in its cache, launched at once; part 1: the prefetched ones, after their copies. `prep`: the - // graph copies the plan and quantizes the activations (the first launch of a layer) + // graph copies the plan and the activations in (the first launch of a layer) struct Graph { int gu, d, groups, part; bool prep; cudaGraphExec_t exec; }; std::vector graphs_; // prefetch slots, their copy stream, and the event of the last copies diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 7dcbff4a53a..d61408e9f79 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -20,7 +20,9 @@ // EXPERTS. The main GPU computes the routed experts its VRAM tier holds, decided in the graph from a snapshot of the // residency table that the pool reads too (`residency`), so neither waits for the other; its shared expert runs on a // branch of its own beside them. The pool computes the rest on the CPU and the second GPU, and with a PCIe share -// publishes the missed experts the GPU reads over PCIe. +// publishes the missed experts the GPU reads over PCIe. With the native router and a native pack the router kernel +// writes their input in the forms they take: ggml's Q8_K rows for the CPU (`pool_takes_q8k` layers) and q8_1 rows for +// the GPUs, so neither quantizes it. // // Requires the default native decode configuration (native projections, fused GR, fused GDN, fast attention and // selection, native indexer) and a profile-filled VRAM expert tier. @@ -200,6 +202,13 @@ class Verifier { bool pcie_share_ = true; int groups_[9] = {}; float* h_ymiss_ = nullptr; float* m_ymiss_ = nullptr; // T * k * n_embd + // the FFN input as the router kernel writes it (quant_router_): Q8_K rows `xk_stride_` bytes apart for the layers + // the pool takes them (xk_layer_), q8_1 rows for the second GPU + bool quant_router_ = false; + std::vector xk_layer_; + int64_t xk_stride_ = 0; + uint8_t* h_xk_ = nullptr; uint8_t* m_xk_ = nullptr; // T rows + uint8_t* h_xq1_ = nullptr; uint8_t* m_xq1_ = nullptr; // T rows of n_embd / 32 blocks // the second GPU's share (set_gpu2), per token group: the flag it raises (64 bytes apart) and its entries, // [n, pad x3, entries] (mapped; `list2_` their device copies, which the combine reads) bool gpu2_ = false; diff --git a/include/strata/kernels/native_router.hpp b/include/strata/kernels/native_router.hpp index d180860ffac..c7d02db91b3 100644 --- a/include/strata/kernels/native_router.hpp +++ b/include/strata/kernels/native_router.hpp @@ -19,8 +19,9 @@ void native_router_top10_multi(const float* logits, int32_t* ids, float* weights // Plan v0.3 P6: a verify window layer's router in one kernel: bf16_gemv_fp32_mmvf_multi's logits, // native_router_top10_multi's ids and weights and doorbell_publish's ring, bitwise, and verify_hit_plan's plan. The -// mapped copy of x is written while the logits run; the ids and weights (and their mapped copies) once every token's -// logits are in; then the sequence number; then the plan. +// forms of x are written while the logits run (with `xq1`, its q8_1 rows bitwise quantize_q8_1_rows' and its Q8_K +// rows ggml's quantize_row_q8_K's); the ids and weights (and their mapped copies) once every token's logits are in; +// then the sequence number; then the plan. struct VerifyRouterArgs { const float* x = nullptr; // (n_tok, n_embd) const uint16_t* w = nullptr; // the router: bf16 (n_expert, n_embd) @@ -28,6 +29,10 @@ struct VerifyRouterArgs { int32_t* ids = nullptr; // (n_tok, 10) float* weights = nullptr; // (n_tok, 10) float* x_out = nullptr; // mapped copy of x, or null + uint8_t* xq1 = nullptr; // x as q8_1 rows (n_embd / 32 blocks each), or null + uint8_t* xq1_out = nullptr; // mapped copy of them, or null (needs xq1) + uint8_t* xk_out = nullptr; // mapped: x as Q8_K rows, `xk_stride` bytes apart (a multiple + int xk_stride = 0; // of 16), or null (needs xq1) int32_t* ids_out = nullptr; // mapped copies of ids and weights, or null float* w_out = nullptr; uint32_t* seq = nullptr; // mapped: set to `ring` once those copies are visible diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index 7faa943ef6b..2a3ca69aa92 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -409,12 +409,18 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 } const bool skip_gpu_rows = d.plan != nullptr && d.plan->host_rows_only; const auto c1 = std::chrono::steady_clock::now(); + // the gate/up activations: the window's router wrote them as Q8_K rows when the layer takes those + const uint8_t* xk = native && d.plan != nullptr ? d.plan->xk : nullptr; + auto nact = [&](int64_t t) -> const void* { + return xk != nullptr ? xk + (size_t) t * (size_t) d.plan->xk_stride + : d.nact_multi.data() + (size_t) t * kNativeActBytes; + }; if (native && lay.fmt[(size_t) d.layers].gu_type == 42) // a native Q2_0 pack: the Q2_0 kernels' activations for (int64_t t = 0; t < n_tok; ++t) act_quant_any(x_f + (size_t) t * H, H, d.act_multi[(size_t) t]); - else if (native) + else if (native && xk == nullptr) for (int64_t t = 0; t < n_tok; ++t) native_quant_act(lay.fmt[(size_t) d.layers], x_f + (size_t) t * H, d.nact_multi.data() + (size_t) t * kNativeActBytes); - else + else if (!native) for (int64_t t = 0; t < n_tok; ++t) act_quant_q8_1(x_f + (size_t) t * H, H, d.act_multi[(size_t) t]); const auto c2 = std::chrono::steady_clock::now(); int njobs = 0; @@ -455,27 +461,26 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 } ExpertJobMulti& jb = d.jobs_multi[(size_t) jo]; jb.act[jb.nt] = &d.act_multi[(size_t) t]; - jb.nact[jb.nt] = native ? d.nact_multi.data() + (size_t) t * kNativeActBytes : nullptr; + jb.nact[jb.nt] = native ? nact(t) : nullptr; jb.out[jb.nt] = row; ++jb.nt; ++d.multi_entries; } const auto c3 = std::chrono::steady_clock::now(); // the second GPU's share goes out once the CPU's workers have started on theirs: its launch takes the host - // tens of microseconds. It writes its rows into `out` itself. + // tens of microseconds. It takes the q8_1 rows the window's router wrote, and writes its rows into `out` itself. struct Submit2 { ExpertDispatch* d; - const float* x; int n_tok, n_groups; int64_t k; const int32_t *slots, *starts, *entries; float* out; bool ok; - } s2{&d, x_f, (int) n_tok, n2g, k, g2_slot, g2_start, g2_ent, out, true}; + } s2{&d, (int) n_tok, n2g, k, g2_slot, g2_start, g2_ent, out, true}; void (*submit2)(void*) = [](void* p) { Submit2& s = *(Submit2*) p; GpuPlanSink& S = *s.d->plan; - s.ok = s.d->gpu2->submit(s.d->layers, s.x, s.n_tok, s.k, s.slots, s.starts, s.entries, s.n_groups, s.out, + s.ok = s.d->gpu2->submit(s.d->layers, S.x1, s.n_tok, s.k, s.slots, s.starts, s.entries, s.n_groups, s.out, S.gpu2_flag, S.ring, s.d->gpu2_err); if (s.ok) S.gpu2_ring = S.ring; }; diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp index 6b673f97e1c..701e57ab8cd 100644 --- a/src/core/second_gpu.cpp +++ b/src/core/second_gpu.cpp @@ -62,7 +62,8 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, n_ff_ = n_ff; cap_ = (int64_t) max_t * k; DeviceScope scope(dev_, main_); - const size_t xb = (size_t) max_t * (size_t) n_embd * sizeof(float), ob = (size_t) cap_ * (size_t) n_embd * sizeof(float); + const size_t xb = (size_t) max_t * (size_t) (n_embd / 32) * 36; // q8_1 rows + const size_t ob = (size_t) cap_ * (size_t) n_embd * sizeof(float); const size_t pb = PlanLayout(cap_).bytes; const unsigned pm = cudaHostAllocPortable | cudaHostAllocMapped; const char* step = nullptr; @@ -77,7 +78,7 @@ bool SecondGpu::init(int device, int main_device, int64_t n_embd, int64_t n_ff, run("pinned staging", cudaHostAlloc((void**) &h_plan_, pb, pm)); run("mapped staging", cudaHostGetDevicePointer((void**) &m_x_, h_x_, 0)); run("mapped staging", cudaHostGetDevicePointer((void**) &m_plan_, h_plan_, 0)); - run("device buffers", cudaMalloc((void**) &d_xq_, (size_t) max_t * (size_t) (n_embd / 32) * 36)); + run("device buffers", cudaMalloc((void**) &d_xq_, xb)); run("device buffers", cudaMalloc((void**) &d_plan_, pb)); run("device buffers", cudaMalloc((void**) &d_scratch_, strata::kernels::native_expert_scratch_bytes(cap_, n_ff))); run("device buffers", cudaMalloc((void**) &d_rows_, ob)); @@ -150,11 +151,11 @@ bool SecondGpu::graph_for(int gu_type, int d_type, int groups, int part, bool pr } try { if (prep) { - // a kernel copies the plan (a copy node costs ~15 us on this link); the grouped kernels read the group - // count from it and skip the groups past it + // kernels copy the plan and the activations (a copy node costs ~15 us on this link); the grouped kernels + // read the group count from the plan and skip the groups past it strata::kernels::copy_i32_from_mapped((int32_t*) d_plan_, (const int32_t*) m_plan_, (int64_t) P.bytes / 4, s_); - strata::kernels::quantize_q8_1_rows(m_x_, max_t_, n_embd_, d_xq_, s_); + strata::kernels::copy_from_mapped((float*) d_xq_, (const float*) m_x_, max_t_ * (n_embd_ / 32) * 9, s_); } strata::kernels::native_expert_grouped(L, p64 + part * cap_, pi + P.starts[part], pi + part, pi + P.dst, pi + P.tok, groups, cap_, d_xq_, d_scratch_, d_rows_, s_); @@ -180,9 +181,9 @@ bool SecondGpu::graph_for(int gu_type, int d_type, int groups, int part, bool pr return true; } -bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, const int32_t* slots, const int32_t* starts, - const int32_t* entries, int n_groups, float* out, uint32_t* flag, uint32_t ring, - std::string& err) { +bool SecondGpu::submit(int64_t layer, const uint8_t* x, int n_tok, int64_t k, const int32_t* slots, + const int32_t* starts, const int32_t* entries, int n_groups, float* out, uint32_t* flag, + uint32_t ring, std::string& err) { const int n = starts[n_groups]; if (n_groups <= 0 || n > cap_ || n_tok > max_t_) { err = "second GPU: a layer's share is out of range"; return false; } cudaError_t q; @@ -217,7 +218,7 @@ bool SecondGpu::submit(int64_t layer, const float* x, int n_tok, int64_t k, cons p64[2 * cap_] = (unsigned long long) out; p64[2 * cap_ + 1] = ng[1] == 0 ? (unsigned long long) flag : 0; // the last launch raises it p64[2 * cap_ + 2] = ng[1] > 0 ? (unsigned long long) flag : 0; - std::memcpy(h_x_, x, (size_t) n_tok * (size_t) n_embd_ * sizeof(float)); + std::memcpy(h_x_, x, (size_t) n_tok * (size_t) (n_embd_ / 32) * 36); const auto& f = strata::kernels::cpu::expert_layout().fmt[(size_t) layer]; DeviceScope scope(dev_, main_); cudaGraphExec_t own = nullptr, pre = nullptr; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 9c452bb4fdd..934b41c0830 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -108,7 +108,7 @@ Verifier::~Verifier() { if (stamps_) cudaFree(stamps_); void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_, h_res_, h_flag2_, - h_list2_}; + h_list2_, h_xk_, h_xq1_}; for (void* h : hosts) if (h) cudaFreeHost(h); } @@ -160,6 +160,12 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& // ---- mapped staging (portable where the second GPU writes: its rows and flags) const unsigned portable = cudaHostAllocMapped | cudaHostAllocPortable; list2_stride_ = (4 + (int64_t) (T * K) + 3) & ~3ll; + const auto& lay = strata::kernels::cpu::expert_layout(); + quant_router_ = strata::kernels::native_router_enabled() && lay.native; + xk_layer_.assign((size_t) g.n_layers, 0); + for (int64_t l = 0; quant_router_ && l < g.n_layers; ++l) + xk_layer_[(size_t) l] = pool_takes_q8k(lay.fmt[(size_t) l]); + xk_stride_ = ((int64_t) (N / 256) * 292 + 15) & ~15ll; bool ok = mapped(T * 4, (void**) &h_tok_, (void**) &m_tok_) && mapped(T * strata::kernels::kStepCount * 4, (void**) &h_step_, (void**) &m_step_) && mapped(T * NH * 4, (void**) &h_pos_, (void**) &m_pos_) && @@ -176,6 +182,8 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped(T * K * N * 4, (void**) &h_ymiss_, (void**) &m_ymiss_, portable) && mapped(128, (void**) &h_flag2_, (void**) &m_flag2_, portable) && mapped((size_t) (2 * list2_stride_) * 4, (void**) &h_list2_, (void**) &m_list2_) && + mapped(T * (uint64_t) xk_stride_, (void**) &h_xk_, (void**) &m_xk_) && + mapped(T * (N / 32) * 36, (void**) &h_xq1_, (void**) &m_xq1_) && mapped(2 * T * K * 4, (void**) &h_pids_, (void**) &m_pids_) && mapped(2 * T * K * 4, (void**) &h_pw_, (void**) &m_pw_) && mapped(64, (void**) &h_pseq_, (void**) &m_pseq_) && @@ -354,6 +362,10 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { gpu_stamp(stamps_ + ((size_t) T * (size_t) (g.n_layers + 1) + (size_t) row) * kStamps + i, st); }; stamp(g.n_layers, 0, cs); + if (gpu2_ && !quant_router_) { + err = "verify: the second GPU takes the native router's q8_1 rows (a native pack)"; + return false; + } // the layer's short branches: `fork` starts stream `to` from the main stream's work so far, `mark` records what // `from` has queued, `wait` makes the main stream wait for a mark auto fork = [&](cudaStream_t to, cudaEvent_t ev) { @@ -672,14 +684,16 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { wn1 != nullptr ? (const float*) wn1->data : nullptr, wn1 != nullptr ? gate_ema_ + (size_t) (l + 1) * HC * N : nullptr); stamp(l, 3, cs); - // the routed experts' activations beside the router (joined before the experts) - if (!fork(side_, bfork_)) return false; - if (strata::kernels::cpu::expert_layout().native) - quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, side_); - else - quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), (int64_t) n * N, - side_); - if (!mark(side_, bjoin_)) return false; + // the routed experts' activations: from the router kernel, else beside it (joined before the experts) + if (!quant_router_) { + if (!fork(side_, bfork_)) return false; + if (strata::kernels::cpu::expert_layout().native) + quantize_q8_1_rows(xm, n, N, nat_xq_ + (size_t) tb * (N / 32) * 36, side_); + else + quantize_q8_0_scaled(xm, hit_xq_ + (size_t) tb * (N / 32) * 34, hit_xs_ + (size_t) tb * (N / 32), + (int64_t) n * N, side_); + if (!mark(side_, bjoin_)) return false; + } // the router, the doorbell and the main GPU's routed experts - the ones its VRAM tier holds, decided here from // the residency snapshot the pool reads too (it leaves their rows to the GPU) if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { @@ -693,7 +707,14 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { VerifyRouterArgs ra; ra.x = xm; ra.w = (const uint16_t*) wr->data; ra.logits = logits_ + tb * NE; ra.ids = ids_ + tb * K; ra.weights = w_ + tb * K; - ra.x_out = m_x_ + tb * N; ra.ids_out = m_ids_ + tb * K; ra.w_out = m_w_ + tb * K; ra.seq = m_seq_; + ra.x_out = xk_layer_[(size_t) l] ? nullptr : m_x_ + tb * N; + ra.ids_out = m_ids_ + tb * K; ra.w_out = m_w_ + tb * K; ra.seq = m_seq_; + if (quant_router_) { // the q8_1 rows for both GPUs' experts, the Q8_K rows for the CPU's + ra.xq1 = nat_xq_ + (size_t) tb * (N / 32) * 36; + ra.xq1_out = gpu2_ ? m_xq1_ + (size_t) tb * (N / 32) * 36 : nullptr; + ra.xk_out = xk_layer_[(size_t) l] ? m_xk_ + (size_t) tb * (size_t) xk_stride_ : nullptr; + ra.xk_stride = (int) xk_stride_; + } ra.ring = (uint32_t) (l * G + grp + 1); // the rings so far (the host waits for this one) ra.res = res_ + (size_t) l * NE; ra.slot_ptr = slot_ptr_; ra.plan = hit_plan; ra.cap = (int) (MT * K); ra.ptr_off = (int) plan_ptr_off_; @@ -763,7 +784,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { err = "verify: the prediction branch could not join"; return false; } - if (!wait(bjoin_)) return false; + if (!quant_router_ && !wait(bjoin_)) return false; stamp(l, 5, cs); return beside || shared(l, grp); }; @@ -1134,6 +1155,9 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, cur_layer_ = want - 1; set_plan_slot(grp); const int tb = gtb[grp], n = gte[grp] - gtb[grp]; + sink_.xk = xk_layer_[(size_t) l] ? h_xk_ + (size_t) tb * (size_t) xk_stride_ : nullptr; + sink_.xk_stride = xk_stride_; + sink_.x1 = h_xq1_ + (size_t) tb * (size_t) (g.n_embd / 32) * 36; if (pool != nullptr) pool(user, h_x_ + (size_t) tb * g.n_embd, h_ids_ + (size_t) tb * ss.k, n, ss.k, h_ymiss_ + (size_t) tb * ss.k * g.n_embd, l); diff --git a/src/kernels/cuda/native_router.cu b/src/kernels/cuda/native_router.cu index 6897f74f9bd..64784dc6dec 100644 --- a/src/kernels/cuda/native_router.cu +++ b/src/kernels/cuda/native_router.cu @@ -24,6 +24,7 @@ #include "strata/kernels/bf16_bits.hpp" #include "hit_plan.cuh" #include "mmvf_multi.cuh" +#include #include #include #include @@ -124,19 +125,130 @@ __global__ void route(const float* __restrict__ logits, int32_t* __restrict__ id // ================================ plan v0.3 P6: the verify window's router in one kernel ================================ // -// For T tokens: when the input rows have a mapped copy, the first blocks (scheduled first) only copy them, a 16-byte -// store (a full PCIe transaction) a thread, and fence; then a block a row of the logits (bf16_gemv_fp32_mmvf_multi's -// 256-thread block, mmvf_multi_row). The last block to finish routes the tokens (`route_token`, a warp each); the -// routing warps, which wrote the mapped ids and weights, fence; then the ring's number, and the plan of the main -// GPU's hits (hit_plan_block). In place of the gemv, the top 10, doorbell_publish and verify_hit_plan. +// For T tokens: the first blocks (scheduled first) write the input rows' other forms, and fence; then a block a row +// of the logits (bf16_gemv_fp32_mmvf_multi's 256-thread block, mmvf_multi_row). The last block to finish routes the +// tokens (`route_token`, a warp each); the routing warps, which wrote the mapped ids and weights, fence; then the +// ring's number, and the plan of the main GPU's hits (hit_plan_block). In place of the gemv, the top 10, +// doorbell_publish and verify_hit_plan. +// +// The rows' forms: a mapped copy; or their q8_1 blocks for the GPUs' experts (quantize_q8_1_rows) and ggml's Q8_K +// blocks for the CPU pool (quantize_row_q8_K_ref), bitwise: the quantizers' float operations are written in PTX, which +// this file's fast-math flags (no denormals, approximate division) do not change. A block takes four Q8_K blocks (292 +// bytes each), so its rows go out in 16-byte stores, full PCIe transactions: a block each, written 4 bytes at a time, +// cost twice as much a byte. constexpr int VR_THREADS = 256; // mmvf_block_size of n_embd a multiple of 512 +constexpr int kQ81Bytes = 36, kQ8KBytes = 292; +__device__ __forceinline__ float add_rn(float x, float y) { + float r; + asm("add.rn.f32 %0, %1, %2;" : "=f"(r) : "f"(x), "f"(y)); + return r; +} +__device__ __forceinline__ float mul_rn(float x, float y) { + float r; + asm("mul.rn.f32 %0, %1, %2;" : "=f"(r) : "f"(x), "f"(y)); + return r; +} +__device__ __forceinline__ float div_rn(float x, float y) { + float r; + asm("div.rn.f32 %0, %1, %2;" : "=f"(r) : "f"(x), "f"(y)); + return r; +} +// four superblocks of 256 values a block (their 4 x 292 bytes of Q8_K and 4 x 288 of q8_1 are whole 16-byte stores; +// a warp a q8_1 block of each): block b's q8_1 blocks to xq1 (and xq1_out), its Q8_K blocks to xk_out, its floats to +// x_out, each when given +constexpr int kGroup = 4; +__device__ __forceinline__ void quantize_rows(const VerifyRouterArgs& a, int b) { + __shared__ uint4 s_q1[kGroup * 8 * kQ81Bytes / 16]; + __shared__ uint4 s_qk[(kGroup * kQ8KBytes + 15) / 16]; + __shared__ unsigned s_amax[kGroup][8]; + __shared__ float s_max[kGroup][8]; + const int t = (int) threadIdx.x, lane = t & 31, warp = t >> 5, per = a.n_embd / 256; + const int groups = (per + kGroup - 1) / kGroup, tok = b / groups, sb0 = (b % groups) * kGroup; + const int ns = min(kGroup, per - sb0); + const float* xr = a.x + (size_t) tok * a.n_embd + (size_t) sb0 * 256; + float x[kGroup]; +#pragma unroll + for (int j = 0; j < kGroup; ++j) x[j] = j < ns ? xr[j * 256 + t] : 0.0f; +#pragma unroll + for (int j = 0; j < kGroup; ++j) { + if (j >= ns) break; + const unsigned ax = __float_as_uint(x[j]) & 0x7fffffffu; // |x|, ordered as an integer + // q8_1 (quantize_q8_1_kernel): d = amax / 127, q = round(x / d), the warp's sum in xor-shuffle order + const unsigned amax = __reduce_max_sync(0xffffffffu, ax); + float sum = x[j]; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) sum = add_rn(sum, __shfl_xor_sync(0xffffffffu, sum, o)); + const float d = div_rn(__uint_as_float(amax), 127.0f); + float q = 0.0f; + if (amax != 0u) { // roundf: half away from zero + const float v = div_rn(x[j], d), r = truncf(v); + q = fabsf(v - r) >= 0.5f ? r + copysignf(1.0f, v) : r; + } + uint8_t* q1 = reinterpret_cast(s_q1) + (j * 8 + warp) * kQ81Bytes; + q1[4 + lane] = (uint8_t) (int8_t) q; + if (lane == 0) *reinterpret_cast(q1) = __halves2half2(__float2half(d), __float2half(sum)); + // Q8_K: the superblock's first value of the largest magnitude, from each warp's + const int first = __reduce_min_sync(0xffffffffu, ax == amax ? lane : 32); + const float wmax = __shfl_sync(0xffffffffu, x[j], first & 31); + if (lane == 0) { + s_amax[j][warp] = amax; + s_max[j][warp] = wmax; + } + } + __syncthreads(); + // Q8_K: iscale = -127 / that value, q = min(127, nearest_int(iscale * x)), d = 1 / iscale, the sums of 16 quants + uint8_t* qk = reinterpret_cast(s_qk); +#pragma unroll + for (int j = 0; j < kGroup; ++j) { + if (j >= ns) break; + unsigned best = 0u; + float mx = 0.0f; + for (int w = 0; w < 8; ++w) + if (s_amax[j][w] > best) { + best = s_amax[j][w]; + mx = s_max[j][w]; + } + uint8_t* blk = qk + j * kQ8KBytes; + int qi = 0; + if (best != 0u) { + const float iscale = div_rn(-127.0f, mx); + qi = min(127, (int) (__float_as_uint(add_rn(mul_rn(iscale, x[j]), 12582912.0f)) & 0x007fffffu) - + 0x00400000); + if (t == 0) *reinterpret_cast(blk) = div_rn(1.0f, iscale); + } else if (t == 0) { + *reinterpret_cast(blk) = 0.0f; + } + blk[4 + t] = (uint8_t) (int8_t) qi; + int bsum = qi; +#pragma unroll + for (int o = 8; o > 0; o >>= 1) bsum += __shfl_xor_sync(0xffffffffu, bsum, o); // within the 16-lane halves + if ((lane & 15) == 0) reinterpret_cast(blk + 4 + 256)[t >> 4] = (int16_t) bsum; + } + __syncthreads(); + const size_t row1 = (size_t) per * 8 * kQ81Bytes, at1 = (size_t) sb0 * 8 * kQ81Bytes; + const int n1 = ns * 8 * kQ81Bytes / 16, nk = (ns * kQ8KBytes + 15) / 16; + if (t < n1) { + const uint4 v = s_q1[t]; + reinterpret_cast(a.xq1 + tok * row1 + at1)[t] = v; + if (a.xq1_out != nullptr) reinterpret_cast(a.xq1_out + tok * row1 + at1)[t] = v; + } + if (a.xk_out != nullptr && t < nk) + reinterpret_cast(a.xk_out + (size_t) tok * a.xk_stride + (size_t) sb0 * kQ8KBytes)[t] = s_qk[t]; + if (a.x_out != nullptr && t < ns * 64) + reinterpret_cast(a.x_out + (size_t) tok * a.n_embd + (size_t) sb0 * 256)[t] = + reinterpret_cast(xr)[t]; +} template __global__ void __launch_bounds__(VR_THREADS) verify_router_kernel(VerifyRouterArgs a, int n_copy) { __shared__ bool s_last; const int t = (int) threadIdx.x, lane = t & 31, warp = t >> 5, b = (int) blockIdx.x; if (b < n_copy) { - for (int i = b * VR_THREADS + t; i < TT * a.n_embd / 4; i += n_copy * VR_THREADS) - reinterpret_cast(a.x_out)[i] = reinterpret_cast(a.x)[i]; + if (a.xq1 != nullptr) { + quantize_rows(a, b); + } else { + for (int i = b * VR_THREADS + t; i < TT * a.n_embd / 4; i += n_copy * VR_THREADS) + reinterpret_cast(a.x_out)[i] = reinterpret_cast(a.x)[i]; + } __threadfence_system(); // this thread's writes before the block counts as done } else { const int row = b - n_copy; @@ -197,8 +309,15 @@ void verify_router(const VerifyRouterArgs& a, void* stream) { if (!stream || a.n_tok < 1 || a.n_tok > 8 || a.n_expert != 512 || a.n_embd <= 0 || a.n_embd % 512 != 0 || !a.x || !a.w || !a.logits || !a.ids || !a.weights || !a.seq || !a.counter || (a.plan && (!a.res || !a.slot_ptr))) throw std::invalid_argument("verify_router: 1..8 tokens, 512 experts, n_embd a multiple of 512, and buffers"); + if ((a.xk_out != nullptr && + (a.xq1 == nullptr || a.xk_stride < (a.n_embd / 256 * kQ8KBytes + 15) / 16 * 16 || a.xk_stride % 16)) || + (a.xq1_out != nullptr && a.xq1 == nullptr)) + throw std::invalid_argument("verify_router: the Q8_K and mapped q8_1 rows come with the q8_1 rows, the Q8_K " + "rows a multiple of 16 bytes apart and at least a row's"); const auto st = static_cast(stream); - const int n_copy = a.x_out != nullptr ? (a.n_tok * a.n_embd / 4 + VR_THREADS - 1) / VR_THREADS : 0; + const int n_copy = a.xq1 != nullptr ? a.n_tok * ((a.n_embd / 256 + kGroup - 1) / kGroup) + : a.x_out != nullptr ? (a.n_tok * a.n_embd / 4 + VR_THREADS - 1) / VR_THREADS + : 0; switch (a.n_tok) { #define STRATA_VR(T) \ case T: verify_router_kernel<<>>(a, n_copy); break; diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 6a8a5ca57e4..7948248fa09 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -6,8 +6,10 @@ // prompt path's batched PLE and GDN arithmetic is held to the same standard; its attention kernel, which sums in // another order, is checked against the decode kernel within a tolerance. #include "strata/kernels/bf16_gemv.hpp" +#include "strata/kernels/cpu/native_expert.hpp" #include "strata/kernels/elementwise.hpp" #include "strata/kernels/fused_gr.hpp" +#include "strata/kernels/iq_kernels.hpp" #include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_moe.hpp" #include "strata/kernels/native_ple_postops.hpp" @@ -446,6 +448,103 @@ int test_verify_router(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the router kernel's forms of its input: the q8_1 rows vs quantize_q8_1_rows, the Q8_K rows vs ggml's +// quantize_row_q8_K (the CPU pool's native_quant_act), the floats vs the input +int test_router_rows(std::mt19937& rng, cudaStream_t s) { + const int N = 2560, NE = 512, FF = 640; + const size_t row1 = N / 32 * 36, stride = 2928; + strata::kernels::cpu::NativeFmt f; + std::string ferr; + if (!strata::kernels::cpu::native_fmt(18, 20, N, FF, f, ferr) || f.act_bytes != (size_t) (N / 256 * 292)) { + std::fprintf(stderr, "router_rows: %s\n", ferr.empty() ? "IQ3_XXS's activation is not Q8_K" : ferr.c_str()); + return 1; + } + std::vector w((size_t) NE * N); + std::normal_distribution nd(0.0f, 0.05f); + for (auto& v : w) { + const float x = nd(rng); + uint32_t b; + std::memcpy(&b, &x, 4); + v = (uint16_t) (b >> 16); + } + uint16_t* d_w = dev(w.size()); + up(d_w, w); + float *d_x = dev((size_t) 8 * N), *d_logits = dev((size_t) 8 * NE), *d_wts = dev(80); + int32_t* d_ids = dev(80); + unsigned* d_counter = dev(1); + check(cudaMemset(d_counter, 0, 4), "memset"); + uint8_t *d_q1 = dev(8 * row1), *d_ref = dev(8 * row1); + uint8_t *h_q1 = nullptr, *m_q1 = nullptr, *h_xk = nullptr, *m_xk = nullptr; + float *h_x = nullptr, *m_x = nullptr; + uint32_t *h_seq = nullptr, *m_seq = nullptr; + check(cudaHostAlloc((void**) &h_q1, 8 * row1, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_xk, 8 * stride, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_x, (size_t) 8 * N * 4, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostAlloc((void**) &h_seq, 64, cudaHostAllocMapped), "cudaHostAlloc"); + check(cudaHostGetDevicePointer((void**) &m_q1, h_q1, 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_xk, h_xk, 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_x, h_x, 0), "mapped"); + check(cudaHostGetDevicePointer((void**) &m_seq, h_seq, 0), "mapped"); + int bad = 0; + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + for (int rep = 0; rep < 4; ++rep) { + std::vector x((size_t) n_tok * N); + for (auto& v : x) v = rng() % 32 == 0 ? edgy(rng) : std::normal_distribution(0.0f, 1.0f)(rng); + if (rep == 1) std::fill(x.begin(), x.begin() + 256, 0.0f); // a zero block (and eight q8_1 ones) + if (rep == 2) { // the largest magnitude twice, in two warps and in one: the first one counts + x[7] = 5e4f; + x[40] = -5e4f; + x[256 + 3] = -6e4f; + x[256 + 20] = 6e4f; + } + up(d_x, x); + std::memset(h_q1, 0xcd, 8 * row1); + std::memset(h_xk, 0xcd, 8 * stride); + std::memset(h_x, 0xcd, (size_t) 8 * N * 4); + strata::kernels::VerifyRouterArgs a; + a.x = d_x; a.w = d_w; a.logits = d_logits; a.ids = d_ids; a.weights = d_wts; + a.xq1 = d_q1; a.xq1_out = m_q1; + a.xk_out = rep == 3 ? nullptr : m_xk; a.xk_stride = (int) stride; + a.x_out = rep == 3 ? m_x : nullptr; // a layer the pool takes as floats + a.seq = m_seq; a.ring = 1; a.counter = d_counter; a.n_tok = n_tok; a.n_embd = N; a.n_expert = NE; + strata::kernels::verify_router(a, s); + strata::kernels::quantize_q8_1_rows(d_x, n_tok, N, d_ref, s); + check(cudaStreamSynchronize(s), "router rows"); + int b = 0; + const std::vector ref = down(d_ref, (size_t) n_tok * row1); + if (down(d_q1, ref.size()) != ref || std::memcmp(h_q1, ref.data(), ref.size()) != 0) { + std::fprintf(stderr, "router_rows: n_tok %d rep %d: the q8_1 rows differ\n", n_tok, rep); + ++b; + } + for (int t = 0; t < n_tok && rep != 3; ++t) { + std::vector want(strata::kernels::cpu::kNativeActBytes, 0); + strata::kernels::cpu::native_quant_act(f, x.data() + (size_t) t * N, want.data()); + const uint8_t* got = h_xk + (size_t) t * stride; + for (size_t i = 0; i < f.act_bytes; ++i) + if (got[i] != want[i]) { + std::fprintf(stderr, "router_rows: n_tok %d rep %d token %d: Q8_K byte %zu (block %zu) %02x " + "vs ggml's %02x\n", n_tok, rep, t, i, i / 292, got[i], want[i]); + ++b; + break; + } + } + if (rep == 3 && std::memcmp(h_x, x.data(), x.size() * 4) != 0) { + std::fprintf(stderr, "router_rows: n_tok %d: the float copy differs\n", n_tok); + ++b; + } + bad += b; + } + } + for (void* q : {(void*) h_q1, (void*) h_xk, (void*) h_x, (void*) h_seq}) cudaFreeHost(q); + for (void* q : {(void*) d_w, (void*) d_x, (void*) d_logits, (void*) d_wts, (void*) d_ids, (void*) d_counter, + (void*) d_q1, (void*) d_ref}) + cudaFree(q); + std::printf("router_rows: %s\n", bad ? "MISMATCH" + : "the q8_1 rows bitwise quantize_q8_1_rows', the Q8_K rows ggml's, the float " + "copy the input (1-8 tokens, denormals, zero blocks, tied maxima)"); + return bad; +} + // ---- the hyper-connection read: every token of an n-token read bitwise its 1-token read, and within a rounding // bound of an FP64 reference (|error| <= 1e-4 x the sum of the terms' magnitudes, carried through silu and sigmoid) int test_gr_read(std::mt19937& rng, cudaStream_t s) { @@ -1028,6 +1127,7 @@ int main(int argc, char** argv) { bad += test_mmvf_multi(rng, s); bad += test_router_multi(rng, s); bad += test_verify_router(rng, s); + bad += test_router_rows(rng, s); bad += test_gr_read(rng, s); bad += test_rope_tokens(rng, s); bad += test_kv_append(rng, s); From f06e9c2460d882cac056d3acb22b95a23e5ee3b5 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 08:23:00 +0200 Subject: [PATCH 65/92] DETAILS.md: generation with the second GPU's rows on a branch and the router's quantized rows API runs of both configs (bench/api_bench.sh, free runs): UD-Q4_K_XL generation 94-112 tok/s (28 tokens to the 131K prompt and thinking), IQ3_XXS 115-147. Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- docs/DETAILS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 147436fd555..c0db430d0d5 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1852-2075 tok/s instead of 992-1130, generation 89-104 tok/s instead of 58-65; -IQ3_XXS 1986-2190 instead of 1516-1699, generation 112-139 instead of 97-111. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1852-2075 tok/s instead of 992-1130, generation 94-112 tok/s instead of 58-65; +IQ3_XXS 1986-2190 instead of 1516-1699, generation 115-147 instead of 97-111. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From bedc0c6c8a853b77a7a0309974283e2777eda01e Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 10:43:34 +0200 Subject: [PATCH 66/92] QSA block scores: query batches in a per-thread GEMM form; bitwise The prompt path's batches (256 queries at a time) went through the decode windows' kernel: an 8-lane group per key block, each score's tree finished across the group by shuffles, ~28 clocks a (query, block) pair. batch_scores_kernel takes batches of 12 queries or more. Lane l of a warp scores key blocks l + 32 j (j < 4) of a 128-block tile against the warp's 2 queries, all four heads: a dot4 per leaf in the original contraction, the 32 leaves summed on a stack in the butterfly's tree (bit-reversed order), so every score is bitwise the warp form's. A warp's lanes read the same query values, a shared-memory broadcast (~2.4 clocks a 16-byte load against ~4.3 for distinct addresses). Persistent: one block an SM with an equal share of the (tile, chunk of 16 queries) units; the keys reload between barriers when the tile changes, and each warp stages its queries into a slot of its own, the next unit's even leaves once this unit is done with them, the odd ones after its last leaf. The decode windows keep block_scores_kernel, which now takes the same dot4. qsa_select_parity: scores bitwise and selections equal over decode windows and prompt batches of 11-256 queries; --bench, a 256-query batch: 16K 254 -> 77 us, 64K 893 -> 493, 131K 1734 -> 892. Window hashes after the 16K prompt identical on both models. Prompt time (IQ3_XXS, both GPUs, 2 rounds): 16K 8203 -> 8024 ms (-2.2%), 65K 30970 -> 30125 (-2.7%), 131K 63680 -> 60764 (-4.6%). Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- include/strata/kernels/qsa_select.hpp | 8 +- src/kernels/cuda/qsa_select.cu | 208 +++++++++++++++++++++++++- src/kernels/qsa_select_parity.cu | 24 ++- 3 files changed, 225 insertions(+), 15 deletions(-) diff --git a/include/strata/kernels/qsa_select.hpp b/include/strata/kernels/qsa_select.hpp index 231324febc9..4f37d0a4729 100644 --- a/include/strata/kernels/qsa_select.hpp +++ b/include/strata/kernels/qsa_select.hpp @@ -8,8 +8,12 @@ // qsa_block_scores : FP32; relu per indexer head, summed; block n_bid (the incomplete tail) scores the `dead` key // and gets +1e9 when it has cells - which is exactly what the per-token path reads there, // because pooled[n_bid] holds `dead` at that time (in a chunk it may already hold a block -// completed later, so it is not read). A 4-lane group per key block reads the key once for -// all queries and sums each score in the order one warp per (query, block) did. +// completed later, so it is not read). Each score sums in the order one warp per +// (query, block) did. Decode windows (< 12 queries): an 8-lane group per key block reads the +// key once for all queries. Batches: a per-thread GEMM form, a lane's 2 queries x 4 key +// blocks over all heads, the tree on a stack of partial sums, the queries read by the whole +// warp at once; one persistent block an SM with an equal share of the (key tile, query chunk) +// units. // qsa_block_topk : one 1024-thread block per query; a 4-pass radix select over the query's n_bid + 1 blocks, // each weighted by its cell count, then the cells emitted in ascending order with ties to the // lowest index - the same selection as topk_kernel, over a quarter of the elements. diff --git a/src/kernels/cuda/qsa_select.cu b/src/kernels/cuda/qsa_select.cu index 1a4864f3b1c..39ececf2929 100644 --- a/src/kernels/cuda/qsa_select.cu +++ b/src/kernels/cuda/qsa_select.cu @@ -6,6 +6,7 @@ #include #include #include +#include namespace strata::kernels { namespace { @@ -16,6 +17,12 @@ constexpr int SCORE_WARPS = 8; constexpr int SCORE_KB = 2; // key blocks per 8-lane group constexpr int SCORE_TILE = SCORE_WARPS * 4 * SCORE_KB; // key blocks per thread block and round constexpr int SCORE_QC = 16; // queries staged in shared memory at a time (32 KB) +constexpr int BATCH_T = 256, BATCH_W = BATCH_T / 32; // batch_scores_kernel: one block an SM +constexpr int BATCH_QT = 2, BATCH_BT = 4; // a lane: 2 queries x 4 key blocks, all heads +constexpr int BATCH_TILE = 32 * BATCH_BT; // key blocks in shared memory (64 KB) +constexpr int BATCH_QC = BATCH_W * BATCH_QT; // queries a unit (a warp's slots: 32 KB) +constexpr int QD = IDX_HEADS * IDX_DIM; // a query's floats +constexpr int kBatchMinQueries = 12; // fewer (decode windows): block_scores_kernel constexpr int TOPK_T = 1024, TOPK_W = TOPK_T / 32; constexpr int TOPK_ROWS = 32; // rows of TOPK_T blocks held in registers constexpr int TOPK_MAX_ROWS = 64; // 65536 blocks: 262144 cells @@ -27,6 +34,12 @@ __device__ __forceinline__ uint32_t order_key(float s) { return (b & 0x80000000u) ? ~b : (b | 0x80000000u); } +// One leaf of a score (dims 4l..4l+3 of a head): k.x*q.x + k.y*q.y + k.z*q.z + k.w*q.w as the original warp-per-pair +// kernel compiled it (qsa_select_parity keeps it) +__device__ __forceinline__ float dot4(float4 k, float4 q) { + return __fmaf_rn(k.w, q.w, __fmaf_rn(k.z, q.z, __fmaf_rn(k.x, q.x, __fmul_rn(k.y, q.y)))); +} + // One query's scores for the group's SCORE_KB key blocks. Lane j of the 8-lane group holds each key's dims 4l..4l+3 // for l = j, j+8, j+16, j+24; per head its four 4-term dot products d_l are summed in the lane as // (d_j + d_j+16) + (d_j+8 + d_j+24), then across the group by xor 4, 2 and 1: the tree in which a warp-wide xor @@ -49,7 +62,7 @@ __device__ __forceinline__ void group_scores(const float4 (&k)[SCORE_KB][4], con for (int i = 0; i < 4; ++i) { float4 k4 = k[kb][i]; if (TAIL && kb == tail_kb) k4 = *reinterpret_cast(dead + 4 * (j + 8 * i)); - d[i] = k4.x * q4[i].x + k4.y * q4[i].y + k4.z * q4[i].z + k4.w * q4[i].w; + d[i] = dot4(k4, q4[i]); } float c = (d[0] + d[2]) + (d[1] + d[3]); c += __shfl_xor_sync(FULL, c, 4); @@ -119,6 +132,174 @@ __global__ void __launch_bounds__(SCORE_WARPS * 32) block_scores_kernel(const fl } } +__host__ __device__ constexpr int rev5(int i) { + return ((i & 1) << 4) | ((i & 2) << 2) | (i & 4) | ((i & 8) >> 2) | ((i & 16) >> 4); +} +__host__ __device__ constexpr int popc5(int i) { + return (i & 1) + ((i >> 1) & 1) + ((i >> 2) & 1) + ((i >> 3) & 1) + ((i >> 4) & 1); +} +// key rows in shared memory, 16-byte chunk c of row r at c ^ (r & 7): a quarter warp's rows on distinct bank quads +__device__ __forceinline__ int key_at(int r, int c) { return r * IDX_DIM + 4 * (c ^ (r & 7)); } +__device__ __forceinline__ void cp_async16(void* s, const void* g) { + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" ::"r"((unsigned) __cvta_generic_to_shared(s)), "l"(g)); +} + +// The same scores for batches of queries, in a per-thread GEMM form. Lane l of a warp scores key blocks l + 32 j +// (j < BATCH_BT) of the tile against the warp's BATCH_QT queries, all heads: a dot4 per leaf (dims 4l..4l+3), the 32 +// leaves summed on a stack in the tree of the warp form's butterfly (bit-reversed order: leaves l and l + 16 first), +// so each score is bit for bit the warp form's. The warp reads its queries' values all at once, a shared-memory +// broadcast at about half the cost of distinct reads. Persistent and balanced: block g takes units +// [g U / G, (g + 1) U / G) of the U = tiles x chunks of BATCH_QC queries, tile-major. The tile's keys are loaded +// between barriers when the tile changes; each warp stages its queries into a slot of its own, the next unit's even +// leaves once this unit is done with them, then the odd ones, so the copies run while it computes. A query's tail +// block scores the `dead` key in the warp form. +__global__ void __launch_bounds__(BATCH_T, 1) batch_scores_kernel(const float* __restrict__ pooled, + const float* __restrict__ dead, + const float* __restrict__ q_idx, + const int32_t* __restrict__ steps, int nq, + int64_t max_blocks, float* __restrict__ out) { + extern __shared__ __align__(16) float smem[]; + __shared__ int s_top; + const int t = threadIdx.x, lane = t & 31, warp = t >> 5; + float* ks = smem; // BATCH_TILE x IDX_DIM + float* qs = smem + BATCH_TILE * IDX_DIM + warp * BATCH_QT * QD; + { + int top = -1; // the batch's last block + for (int i = t; i < nq; i += BATCH_T) { + const int n_bid = steps[(int64_t) i * kStepCount + kStepNBid]; + top = max(top, (int64_t) n_bid < max_blocks - 1 ? n_bid : (int) (max_blocks - 1)); + } + top = __reduce_max_sync(FULL, top); + if (t == 0) s_top = -1; + __syncthreads(); + if (lane == 0) atomicMax(&s_top, top); + __syncthreads(); + } + const int top = s_top; + if (top < 0) return; + const int n_chunks = (nq + BATCH_QC - 1) / BATCH_QC, units = (top / BATCH_TILE + 1) * n_chunks; + const int u1 = (int) ((int64_t) (blockIdx.x + 1) * units / gridDim.x); + // leaves of parity `half` of queries q0, q0 + 1 into the warp's slot, one cp.async group + auto stage = [&](int q0, int half) { +#pragma unroll + for (int e = lane; e < BATCH_QT * IDX_HEADS * 16; e += 32) { + const int a = e / (IDX_HEADS * 16), off = (e / 16) % IDX_HEADS * IDX_DIM + 4 * (2 * (e % 16) + half); + if (q0 + a < nq) cp_async16(qs + a * QD + off, q_idx + (int64_t) (q0 + a) * QD + off); + } + asm volatile("cp.async.commit_group;\n" ::); + }; + int have = -1; // the first query of the slot's rows + for (int u = (int) ((int64_t) blockIdx.x * units / gridDim.x); u < u1;) { + const int b0 = u / n_chunks * BATCH_TILE, ue = min(u1, (u / n_chunks + 1) * n_chunks); + __syncthreads(); // every warp is done with the previous tile + for (int e = t; e < BATCH_TILE * 32; e += BATCH_T) { + const int r = e >> 5, c = e & 31; + cp_async16(ks + key_at(r, c), pooled + (int64_t) min(b0 + r, top) * IDX_DIM + 4 * c); + } + asm volatile("cp.async.commit_group;\n" ::); + asm volatile("cp.async.wait_group 0;\n" ::); + __syncthreads(); + for (; u < ue; ++u) { + const int q0 = u % n_chunks * BATCH_QC + warp * BATCH_QT; + int n_bid[BATCH_QT], n_kv[BATCH_QT], wlast = -1; +#pragma unroll + for (int a = 0; a < BATCH_QT; ++a) { + const bool in = q0 + a < nq; + n_bid[a] = in ? steps[(int64_t) (q0 + a) * kStepCount + kStepNBid] : -1; + n_kv[a] = in ? steps[(int64_t) (q0 + a) * kStepCount + kStepNKv] : 0; + wlast = max(wlast, n_bid[a]); + } + if (wlast < b0) continue; // the warp's queries end before the tile + if (have != q0) { + // a skipped unit's prefetch may still be landing in the slot: copies to one address are unordered + asm volatile("cp.async.wait_group 0;\n" ::); + __syncwarp(); + stage(q0, 0); + stage(q0, 1); + have = q0; + } + const int q0n = u + 1 < u1 ? (u + 1) % n_chunks * BATCH_QC + warp * BATCH_QT : -1; + const bool ahead = q0n >= 0 && q0n != q0; + asm volatile("cp.async.wait_group 1;\n" ::); // the even leaves + __syncwarp(); + float stk[BATCH_QT][IDX_HEADS][BATCH_BT][6]; + auto leaves = [&](auto first) { +#pragma unroll + for (int ii = 0; ii < 16; ++ii) { + const int i = decltype(first)::value + ii, l = rev5(i), depth = popc5(i); + float4 kf[BATCH_BT], qf[BATCH_QT][IDX_HEADS]; +#pragma unroll + for (int j = 0; j < BATCH_BT; ++j) + kf[j] = *reinterpret_cast(ks + key_at(lane + 32 * j, l)); +#pragma unroll + for (int a = 0; a < BATCH_QT; ++a) +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) + qf[a][h] = *reinterpret_cast(qs + a * QD + h * IDX_DIM + 4 * l); +#pragma unroll + for (int a = 0; a < BATCH_QT; ++a) +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) +#pragma unroll + for (int j = 0; j < BATCH_BT; ++j) { + float* s = stk[a][h][j]; + s[depth] = dot4(kf[j], qf[a][h]); +#pragma unroll + for (int lv = 0; lv < 5; ++lv) { // complete subtrees combine + if (((i + 1) >> lv) & 1) break; + s[depth - lv - 1] = s[depth - lv - 1] + s[depth - lv]; + } + } + } + }; + leaves(std::integral_constant()); // rev5(0..15): the even leaves + __syncwarp(); + if (ahead) { + stage(q0n, 0); + asm volatile("cp.async.wait_group 1;\n" ::); // this unit's odd leaves + } else { + asm volatile("cp.async.wait_group 0;\n" ::); + } + __syncwarp(); + leaves(std::integral_constant()); + __syncwarp(); + if (ahead) { + stage(q0n, 1); + have = q0n; + } +#pragma unroll + for (int a = 0; a < BATCH_QT; ++a) { + const int qi = q0 + a; + const int64_t last = (int64_t) n_bid[a] < max_blocks - 1 ? n_bid[a] : max_blocks - 1; +#pragma unroll + for (int j = 0; j < BATCH_BT; ++j) { + float score = 0.0f; +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) { + const float d = stk[a][h][j][0]; + score += d > 0.0f ? d : 0.0f; + } + const int b = b0 + lane + 32 * j; + if (qi < nq && b <= last && b != n_bid[a]) out[(int64_t) qi * max_blocks + b] = score; + } + if (qi < nq && n_bid[a] >= b0 && n_bid[a] < b0 + BATCH_TILE && (int64_t) n_bid[a] <= max_blocks - 1) { + const float4 k4 = *reinterpret_cast(dead + 4 * lane); + float score = 0.0f; +#pragma unroll + for (int h = 0; h < IDX_HEADS; ++h) { + float d = dot4(k4, *reinterpret_cast(q_idx + (int64_t) qi * QD + h * IDX_DIM + + 4 * lane)); +#pragma unroll + for (int o = 16; o > 0; o >>= 1) d += __shfl_xor_sync(FULL, d, o); + score += d > 0.0f ? d : 0.0f; + } + if (lane == 0) out[(int64_t) qi * max_blocks + n_bid[a]] = n_kv[a] % R != 0 ? score + 1e9f : score; + } + } + } + } +} + __device__ __forceinline__ int warp_sum(int v) { #pragma unroll for (int o = 16; o > 0; o >>= 1) v += __shfl_xor_sync(FULL, v, o); @@ -335,9 +516,25 @@ void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx std::exit(1); } const int64_t covered = grid_blocks < max_blocks ? grid_blocks : max_blocks; - const unsigned grid = (unsigned) ((covered + SCORE_TILE - 1) / SCORE_TILE); - block_scores_kernel<<>>(pooled, dead, q_idx, steps, (int) nq, - max_blocks, scores); + if (nq >= kBatchMinQueries) { + // one block an SM (the prompt path's GPU), each with the same share of the units + constexpr int smem = (BATCH_TILE * IDX_DIM + BATCH_QC * QD) * (int) sizeof(float); + static int sms = 0; + if (sms == 0) { + int dev = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev); + cudaFuncSetAttribute(batch_scores_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem); + } + const int64_t units = ((covered + BATCH_TILE - 1) / BATCH_TILE) * ((nq + BATCH_QC - 1) / BATCH_QC); + const unsigned grid = (unsigned) (units < sms ? units : sms); + batch_scores_kernel<<>>(pooled, dead, q_idx, steps, (int) nq, + max_blocks, scores); + } else { + const unsigned grid = (unsigned) ((covered + SCORE_TILE - 1) / SCORE_TILE); + block_scores_kernel<<>>(pooled, dead, q_idx, steps, (int) nq, + max_blocks, scores); + } const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { std::fprintf(stderr, "qsa_block_scores: %s\n", cudaGetErrorString(e)); std::exit(1); } } @@ -345,7 +542,8 @@ void qsa_block_scores(const float* pooled, const float* dead, const float* q_idx void qsa_block_topk(const float* scores, const int32_t* steps, int64_t nq, int64_t max_blocks, int64_t cap, const QsaShapes& s, int32_t* ids, void* stream) { if (nq <= 0) return; - if (s.idx_block != R || cap < qsa_selection_width(kTopkMaxCells, s) || max_blocks > (int64_t) TOPK_MAX_ROWS * TOPK_T) { + if (s.idx_block != R || cap < qsa_selection_width(kTopkMaxCells, s) || + max_blocks > (int64_t) TOPK_MAX_ROWS * TOPK_T) { std::fprintf(stderr, "qsa_block_topk: unsupported geometry or cap\n"); std::exit(1); } diff --git a/src/kernels/qsa_select_parity.cu b/src/kernels/qsa_select_parity.cu index 358ef1ea898..933006af576 100644 --- a/src/kernels/qsa_select_parity.cu +++ b/src/kernels/qsa_select_parity.cu @@ -1,6 +1,6 @@ // src/kernels/qsa_select_parity.cu - the QSA block scores and top-k (qsa_select.cu) against the kernels they // replaced, kept here as the reference: scores bitwise, selections identical, for decode windows (1-8 queries, -// the captured graph's fixed grid) and the prompt path's batches (256 queries), from the identity case to 200K +// the captured graph's fixed grid) and the prompt path's batches (up to 256 queries), from the identity case to 200K // cells, with exact score ties, zero scores and every tail length. --bench times both. #include "strata/kernels/qsa.hpp" #include "strata/kernels/qsa_select.hpp" @@ -259,7 +259,8 @@ int compare(Data& d, int64_t pos0, int64_t nq, int64_t grid_blocks, cudaStream_t check(cudaMemcpy(ib.data(), d.ids_new, ib.size() * 4, cudaMemcpyDeviceToHost), "down"); int bad = 0; for (int64_t i = 0; i < nq; ++i) { - const int64_t n_bid = st[(size_t) (i * kStepCount + kStepNBid)], width = st[(size_t) (i * kStepCount + kStepWidth)]; + const int64_t n_bid = st[(size_t) (i * kStepCount + kStepNBid)]; + const int64_t width = st[(size_t) (i * kStepCount + kStepWidth)]; const int64_t last = std::min(n_bid, d.max_blocks - 1); for (int64_t k = 0; k <= last; ++k) { const size_t o = (size_t) (i * d.max_blocks + k); @@ -297,10 +298,13 @@ int selftest(Data& d, cudaStream_t s) { bad += compare(d, p, nq, qsa_score_grid_blocks, s, "decode"); ++cases; } - // prompt batches: 256 queries, the grid covering the batch's last block - const int64_t pos_pre[] = {0, 1900, 16384, 65536 - 100, 131072 - 256, 199936 - 1}; - for (int64_t p : pos_pre) { - const int64_t nq = 256, last = p + nq - 1; + // prompt batches: 256 queries and a sub-chunk's last, shorter batch (11 go to the decode windows' kernel), the grid + // covering the batch's last block + const int64_t pos_pre[][2] = {{0, 256}, {1900, 256}, {16384, 256}, {65536 - 100, 256}, + {131072 - 256, 256}, {199936 - 1, 256}, {5000, 12}, {70001, 37}, + {131072 - 100, 100}, {199936 - 255, 255}, {9000, 11}}; + for (const auto& pq : pos_pre) { + const int64_t p = pq[0], nq = pq[1], last = p + nq - 1; bad += compare(d, p, nq, (last + 1) / 4 + 1, s, "prompt"); ++cases; } @@ -341,7 +345,9 @@ void bench(Data& d, cudaStream_t s) { {"decode 4 q, 16K cells", 16384, 4, false}, {"decode 4 q, 64K cells", 65536, 4, false}, {"decode 4 q, 131K cells", 131072, 4, false}, {"decode 1 q, 131K cells", 131072, 1, false}, {"prompt 256 q at 16K", 16384 - 256, 256, true}, - {"prompt 256 q at 64K", 65536 - 256, 256, true}, {"prompt 256 q at 131K", 131072 - 256, 256, true}}; + {"prompt 256 q at 64K", 65536 - 256, 256, true}, + {"prompt 256 q at 131K", 131072 - 256, 256, true}, + {"prompt 16 q at 131K", 131072 - 16, 16, true}, {"prompt 64 q at 131K", 131072 - 64, 64, true}}; for (const Case& c : cs) { steps_from(d, c.pos, c.nq); const int64_t grid = c.prompt ? (c.pos + c.nq) / 4 + 1 : qsa_score_grid_blocks; @@ -352,7 +358,9 @@ void bench(Data& d, cudaStream_t s) { const float ns = time_ms(s, reps, [&] { qsa_block_scores(d.pooled, d.dead, d.q, d.steps, c.nq, d.max_blocks, d.s, d.sc_new, s, grid); }); - const float rt = time_ms(s, reps, [&] { ref::topk(d.sc_ref, d.steps, c.nq, d.max_blocks, d.cap, d.ids_ref, s); }); + const float rt = time_ms(s, reps, [&] { + ref::topk(d.sc_ref, d.steps, c.nq, d.max_blocks, d.cap, d.ids_ref, s); + }); const float nt = time_ms(s, reps, [&] { qsa_block_topk(d.sc_new, d.steps, c.nq, d.max_blocks, d.cap, d.s, d.ids_new, s); }); From 0046ba64437adedcdc66a1d61144a7c7dc4a754d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 11:09:41 +0200 Subject: [PATCH 67/92] Prompt-path QSA attention: each fetch warp issues its cells' K and V together; bitwise Warps 8-9 fetched the K codes and scales of a tile and 10-11 the V ones. With clocks around the waits (scratch/attn2_bench.cu), the value warps waited ~2K clocks a tile for V, while the fetch warps hardly waited for a free slot (~40) or for their copies (~10): they spent 1.5-1.7K issuing (the memory pipe pushing back) and ~0.5K on the row lookups, and K ran ahead of V, so V's copies queued behind K's. Now each of the four fetch warps takes 16 cells of a tile and issues their K and V copies and scales together; the K and V barriers count all four. The arithmetic is unchanged: the 131K dump's attention call is bitwise the old kernel's, and window hashes after the 16K prompt are identical on both models; verify_parity passes. The value warps' wait for V falls to ~1K clocks a tile. The kernel on the 131K dump (layer 5, 2048 queries, alternating, 3 rounds): 3.71-3.91 ms against 4.30-4.55. In the engine (IQ3_XXS, --prefill-profile) the QSA attention takes 373 ms of a 16K prompt instead of 395 (-5.6%) and 3346 of a 131K one instead of 3459 (-3.3%); prompt time (both GPUs, 2 rounds) 16K 8095 -> 8058 ms, 65K 30276 -> 30152, 131K 60812 -> 60693: within noise. Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- src/kernels/cuda/qsa_decode_attn.cu | 44 ++++++++++++++++------------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/src/kernels/cuda/qsa_decode_attn.cu b/src/kernels/cuda/qsa_decode_attn.cu index e33fb27e358..fc69e8b9256 100644 --- a/src/kernels/cuda/qsa_decode_attn.cu +++ b/src/kernels/cuda/qsa_decode_attn.cu @@ -316,7 +316,9 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_f16_kernel(const float* // warps 0-3 score 16 cells of each tile over all 256 dims (q's limbs in registers) and run the online softmax; // warps 4-7 multiply a 64-dim group of V each: the probabilities times the V scales as limbs against V through // ldmatrix.trans (cells and dims permuted inside the fragments, the same way on both sides); -// warps 8-11 fetch by cp.async: 8-9 the K codes and scales, 10-11 V's, the pool rows two tiles ahead. +// warps 8-11 fetch by cp.async, 16 cells of each tile a warp: their K and V codes and scales together (with +// K and V in warps of their own, V's copies queued behind K's, which ran ahead), the pool rows two +// tiles ahead. // Rings in shared memory between them (K 2 slots, V 3, probabilities 2), synchronized by named barriers. namespace pf { constexpr int NT = 384; @@ -326,7 +328,7 @@ enum : int { EMPTY_P = FULL_P + 2, SRED = EMPTY_P + 2 }; static_assert(SRED < 16, "sixteen named barriers"); -constexpr int NKB = 64 + 128, NVB = 64 + 128, NPB = 128 + 128; // threads at the K, V and P barriers (END as P) +constexpr int NKB = 128 + 128, NVB = 128 + 128, NPB = 128 + 128; // threads at the K, V, P barriers (END as P) static_assert(KV_Q8_GROUP == 64 && HD == 256, "four 64-dim scale groups a row"); constexpr int MAGIC = 0x4B400000; // 1.5 * 2^23 as float bits: an int x (|x| < 2^22) added gives 1.5 * 2^23 + x constexpr float MAGIC_F = 12582912.0f; @@ -448,14 +450,10 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float } if (warp >= 8) { - // ======== fetch: warps 8-9 K, 10-11 V. Warp 8 + h and 10 + h take cells 32h .. 32h+31 of each tile, a lane a - // cell and its row; tile i is issued, then tile i-1 signalled once its copies have landed - const bool is_k = warp < 10; - const int h = warp & 1, cl = 32 * h + lane, nb = is_k ? NKB : NVB, slots = is_k ? KS : VS; - const int8_t* src = is_k ? p.k_q : p.v_q; - const uint16_t* scl = is_k ? p.k_scale : p.v_scale; + // ======== fetch, K and V together: warp 8 + f takes cells 16f .. 16f+15 of each tile (lane l & 15 looks up + // a cell's row); tile i is issued, then tile i-1 signalled once its copies have landed + const int f = warp - 8, cl = 16 * f + (lane & 15); auto row_of = [&](int id, int pg) { return (pg * n_kv_heads + kvh) * page_size + id % page_size; }; - // tile i's row of cell cl; tile i+1's id and page; tile i+2's id int row = -1, id1 = 0, pg1 = 0, id2 = 0; if (cl < n_ids) { const int id = ids[cl]; @@ -467,23 +465,28 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float } if (2 * PT + cl < n_ids) id2 = ids[2 * PT + cl]; for (int i = 0; i < n_tiles; ++i) { - const int s = i % slots, c0 = i * PT, n_here = min(PT, n_ids - c0); - if (i >= slots) bar_sync((is_k ? EMPTY_K : EMPTY_V) + s, nb); - uint8_t* dst = is_k ? sm.k[s] : sm.v[s]; + const int sk = i % KS, sv = i % VS, c0 = i * PT, n_here = min(PT, n_ids - c0); + if (i >= KS) bar_sync(EMPTY_K + sk, NKB); + if (i >= VS) bar_sync(EMPTY_V + sv, NVB); #pragma unroll - for (int u = 0; u < 16; ++u) { - const int e = lane + 32 * u, cw = e >> 4, j = e & 15, c = 32 * h + cw; + for (int u = 0; u < 8; ++u) { + const int e = lane + 32 * u, cw = e >> 4, j = e & 15, c = 16 * f + cw; const int r = __shfl_sync(0xffffffffu, row, cw); - if (c < n_here) - cp_async16(dst + c * HD + ((j ^ (is_k ? ksw(c) : c & 7)) << 4), src + (size_t) r * HD + j * 16); + if (c < n_here) { + cp_async16(&sm.k[sk][c * HD + ((j ^ ksw(c)) << 4)], p.k_q + (size_t) r * HD + j * 16); + cp_async16(&sm.v[sv][c * HD + ((j ^ (c & 7)) << 4)], p.v_q + (size_t) r * HD + j * 16); + } } - uint2* sd = is_k ? &sm.ks[s][cl] : &sm.vs[s][cl]; - if (cl < n_here) cp_async8(sd, scl + (size_t) row * (HD / KV_Q8_GROUP)); + const bool vl = lane >= 16; // lanes 0-15 a cell's K scales, 16-31 its V scales + uint2* sd = vl ? &sm.vs[sv][cl] : &sm.ks[sk][cl]; + if (cl < n_here) + cp_async8(sd, (vl ? p.v_scale : p.k_scale) + (size_t) row * (HD / KV_Q8_GROUP)); else *sd = make_uint2(0u, 0u); // no scale for the multiply of a missing cell's stale codes asm volatile("cp.async.commit_group;\n" ::); if (i >= 1) { asm volatile("cp.async.wait_group 1;\n" ::); - bar_arrive((is_k ? FULL_K : FULL_V) + (i - 1) % slots, nb); + bar_arrive(FULL_K + (i - 1) % KS, NKB); + bar_arrive(FULL_V + (i - 1) % VS, NVB); } row = cl < n_ids - c0 - PT ? row_of(id1, pg1) : -1; id1 = id2; @@ -491,7 +494,8 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float id2 = c0 + 3 * PT + cl < n_ids ? ids[c0 + 3 * PT + cl] : 0; } asm volatile("cp.async.wait_group 0;\n" ::); - bar_arrive((is_k ? FULL_K : FULL_V) + (n_tiles - 1) % slots, nb); + bar_arrive(FULL_K + (n_tiles - 1) % KS, NKB); + bar_arrive(FULL_V + (n_tiles - 1) % VS, NVB); } else if (warp < 4) { // ======== scores: cells 16w + 4tig .. +3 of each tile in this lane's C fragments (rows gid, gid + 8) const int w = warp; From fdba43b208796fd8c2a414f05830589cd46e9034 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 11:28:37 +0200 Subject: [PATCH 68/92] DETAILS.md: speeds after the batched QSA block scores and the attention's combined fetch The second-GPU and 3090-alone ranges from the API runs of 0046ba6 (2026-09-28). Claude-Session: https://claude.ai/code/session_01Ljcr81CGEb4dQcdiAZcXAs --- docs/DETAILS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index c0db430d0d5..92b5b60050c 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -103,7 +103,7 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native

/-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 992-1130 tok/s, generation 58-65 tok/s. +UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 1004-1134 tok/s, generation 63-73 tok/s. ### A second GPU @@ -111,8 +111,8 @@ With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0 each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1852-2075 tok/s instead of 992-1130, generation 94-112 tok/s instead of 58-65; -IQ3_XXS 1986-2190 instead of 1516-1699, generation 115-147 instead of 97-111. When the card has no free VRAM, the +VRAM), 16K-131K prompts: UD-Q4_K_XL 1856-2121 tok/s instead of 1004-1134, generation 94-112 tok/s instead of 63-73; +IQ3_XXS 2028-2256 instead of 1548-1744, generation 115-147 instead of 107-121. When the card has no free VRAM, the engine runs on the main GPU alone. ## Before you start From 962b99c70338dbb9028d4eecabe0ee65793fa9cc Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 13:21:11 +0200 Subject: [PATCH 69/92] Verify window: the shared expert and the prediction fork before the router, the CPU's rows awaited on a branch The shared expert's branch and the next layer's prediction need only the FFN read. They now fork there, recorded after the router so its blocks are dispatched first, and run beside its routing, ring and hit plan; the prediction's bias update waits for the router. They forked at the doorbell before: the prediction router's 512 blocks were dispatched ahead of the routed experts, which started 5-7 us after the router, and the shared expert's branch, whose kernels wait while IQ3_XXS's expert kernel holds every SM's registers, ended last before the combine in 70-82% of the layers on IQ3_XXS thinking text (a CUPTI trace of the verify graph). The wait for the CPU's rows runs on a branch of its own from the doorbell, joined before the combine, instead of after the routed experts. Bitwise: window hashes identical (thinking text, both models; --spec-split, --second-gpu -1, prefetch off). Fixed text, both GPUs, 2 rounds against 0046ba6, ms per round: IQ3_XXS code -2.8%, thinking -3.1%, after 16K -2.2%; UD-Q4_K_XL code -0.9%, thinking -2.5%, after 16K -1.9%. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/verify.hpp | 13 +++++---- src/core/verify.cpp | 51 ++++++++++++++++++++++++---------- 2 files changed, 45 insertions(+), 19 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index d61408e9f79..b4400384244 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -19,10 +19,10 @@ // // EXPERTS. The main GPU computes the routed experts its VRAM tier holds, decided in the graph from a snapshot of the // residency table that the pool reads too (`residency`), so neither waits for the other; its shared expert runs on a -// branch of its own beside them. The pool computes the rest on the CPU and the second GPU, and with a PCIe share -// publishes the missed experts the GPU reads over PCIe. With the native router and a native pack the router kernel -// writes their input in the forms they take: ggml's Q8_K rows for the CPU (`pool_takes_q8k` layers) and q8_1 rows for -// the GPUs, so neither quantizes it. +// branch of its own, forked before the router, and the wait for the CPU's rows on another. The pool computes the rest +// on the CPU and the second GPU, and with a PCIe share publishes the missed experts the GPU reads over PCIe. With the +// native router and a native pack the router kernel writes their input in the forms they take: ggml's Q8_K rows for +// the CPU (`pool_takes_q8k` layers) and q8_1 rows for the GPUs, so neither quantizes it. // // Requires the default native decode configuration (native projections, fused GR, fused GDN, fast attention and // selection, native indexer) and a profile-filled VRAM expert tier. @@ -232,13 +232,16 @@ class Verifier { // activations (beside the router), then the prediction; on shs_ the GDN gate projection or the QSA queries (the // mixer), then the shared expert cudaStream_t side_ = nullptr; - cudaEvent_t fork_ = nullptr, join_ = nullptr; // the prediction's + cudaEvent_t join_ = nullptr; // the prediction's cudaEvent_t bfork_ = nullptr, bjoin_ = nullptr; // side_'s others cudaEvent_t pfork_ = nullptr, pjoin_ = nullptr; // shs_'s projections cudaStream_t shs_ = nullptr; // the shared expert's branch (and the snapshot's copy) cudaEvent_t shfork_ = nullptr, shjoin_ = nullptr, res_ready_ = nullptr; cudaStream_t g2s_ = nullptr; // the second GPU's rows, per token group cudaEvent_t g2fork_ = nullptr, g2join_[2] = {}; + cudaStream_t cps_ = nullptr; // the wait for the CPU's rows, per token group + cudaEvent_t cpfork_ = nullptr, cpjoin_[2] = {}; + cudaEvent_t efork_ = nullptr, rdone_ = nullptr; // the FFN read's end, the router's cudaEvent_t done_ = nullptr; // after the last window's graph // device diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 934b41c0830..095425091b2 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -101,8 +101,9 @@ Verifier::~Verifier() { if (side_) cudaStreamDestroy(side_); if (shs_) cudaStreamDestroy(shs_); if (g2s_) cudaStreamDestroy(g2s_); - for (cudaEvent_t e : {fork_, join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_, g2fork_, - g2join_[0], g2join_[1]}) + if (cps_) cudaStreamDestroy(cps_); + for (cudaEvent_t e : {join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_, g2fork_, + g2join_[0], g2join_[1], cpfork_, cpjoin_[0], cpjoin_[1], efork_, rdone_}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); @@ -304,10 +305,15 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& cudaStreamCreateWithFlags(&side_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&g2s_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&cps_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&cpfork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&cpjoin_[0], cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&cpjoin_[1], cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&efork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&rdone_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&g2fork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&g2join_[0], cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&g2join_[1], cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&fork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&join_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&bfork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&bjoin_, cudaEventDisableTiming) != cudaSuccess || @@ -420,8 +426,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } // set_predict: this layer's FFN read also estimates the next layer's FFN input (that layer's read of this state, - // its gates their running average over the windows), and on a side branch forked after the doorbell the next - // layer's router runs on it with that layer's logit bias. When learning, the FFN read of a layer the previous one + // its gates their running average over the windows), and on a side branch forked after the read the next layer's + // router runs on it with that layer's logit bias. When learning, the FFN read of a layer the previous one // predicts keeps its gates' average, and the layer corrects its bias by its own logits on that branch. auto predicts = [&](int64_t l) { return predict_ != nullptr && G == 1 && l + 1 < g.n_layers && native_router_enabled(); @@ -439,8 +445,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } } - // the shared expert: in a window of one token group on a branch of its own that forks at the doorbell, beside the - // routed experts (joined before the combine); in a split window inline + // the shared expert: in a window of one token group on a branch of its own forked after the FFN read (`efork_`), + // beside the router and the routed experts (joined before the combine); in a split window inline const bool beside = G == 1; auto shared = [&](int64_t l, int grp) -> bool { const int tb = tb_[grp], te = te_[grp], n = te - tb; @@ -458,7 +464,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { nsw.q8_1 = xq_; // the main stream's next use is the next layer's mixer cudaStream_t ws = cs; if (beside) { - if (cudaEventRecord(shfork_, cs) != cudaSuccess || cudaStreamWaitEvent(shs_, shfork_, 0) != cudaSuccess) { + if (cudaStreamWaitEvent(shs_, efork_, 0) != cudaSuccess) { err = "verify: the shared expert's branch could not fork"; return false; } @@ -694,6 +700,12 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { (int64_t) n * N, side_); if (!mark(side_, bjoin_)) return false; } + // the branches that need only the FFN read fork here; they are recorded after the router, whose blocks are + // then dispatched first, and run beside its routing, the ring and the hit plan + if (beside && cudaEventRecord(efork_, cs) != cudaSuccess) { + err = "verify: the FFN read's branches could not fork"; + return false; + } // the router, the doorbell and the main GPU's routed experts - the ones its VRAM tier holds, decided here from // the residency snapshot the pool reads too (it leaves their rows to the GPU) if (l == 0 && grp == 0 && cudaStreamWaitEvent(cs, res_ready_, 0) != cudaSuccess) { @@ -737,8 +749,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { MT * K, plan_ptr_off_, cs); } stamp(l, 4, cs); - // after the doorbell: the second GPU's rows of this token group, taken into VRAM once their flag rises; the - // shared expert's branch; and the next layer's prediction + // after the doorbell: the second GPU's rows of this token group, taken into VRAM once their flag rises, and + // the wait for the CPU's rows, each on a branch beside the routed experts; then the branches forked after the + // FFN read: the shared expert, and the next layer's prediction if (gpu2_) { if (!fork(g2s_, g2fork_)) return false; wait_flag_ge(m_flag2_ + (size_t) grp * 16, (uint32_t) (l * G + grp + 1), g2s_); @@ -747,9 +760,15 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, kLayerStamps + 1, g2s_); if (!mark(g2s_, g2join_[grp])) return false; } + if (!fork(cps_, cpfork_)) return false; + wait_flag_ge(m_flag_, (uint32_t) (l * G + grp + 1), cps_); // the CPU's share is in the mapped rows + if (!mark(cps_, cpjoin_[grp])) return false; + if (beside && cudaEventRecord(rdone_, cs) != cudaSuccess) { + err = "verify: the router's end could not be marked"; + return false; + } if (beside && !shared(l, grp)) return false; - if (branches(l) && - (cudaEventRecord(fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, fork_, 0) != cudaSuccess)) { + if (branches(l) && cudaStreamWaitEvent(side_, efork_, 0) != cudaSuccess) { err = "verify: the prediction branch could not fork"; return false; } @@ -771,7 +790,11 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } } - if (corrects(l)) { // after the prediction, from the one the previous layer made + if (corrects(l)) { // after the prediction and the router, from the prediction the previous layer made + if (cudaStreamWaitEvent(side_, rdone_, 0) != cudaSuccess) { + err = "verify: the prediction's bias could not wait for the router"; + return false; + } try { verify_router_bias_update(logits_ + tb * NE, plogits_ + (size_t) (l & 1) * MT * NE, n, pbias_ + (size_t) l * NE, side_); @@ -847,7 +870,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 7, cs); } stamp(l, 8, cs); - wait_flag_ge(m_flag_, ring, cs); // the CPU's share is in the mapped rows + if (!wait(cpjoin_[grp])) return false; // the CPU's share is in the mapped rows stamp(l, 9, cs); if (gpu2_ && !wait(g2join_[grp])) return false; // the second GPU's, in `hit_out` stamp(l, 10, cs); From aa209f7124a1a8eb6e2c092ea05bd3ab76f2814d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 16:12:09 +0200 Subject: [PATCH 70/92] Verify window: the shared expert's gate in the combine, fewer QSA mixer launches, one PLE pass a window; bitwise - The shared expert's branch ends with its down projection: its scalar gate's logits come from a branch of their own, and the gather-combine applies sigmoid and product in shared_expert.cu's instructions (inline PTX without flush to zero; shared_expert_gate_rows, which the one-token path now uses, is the reference). Its last kernels no longer wait behind the routed experts' grids: on IQ3_XXS thinking text it ends last before the combine in 0.4-16% of the layers instead of 58-66%, and at 2+ tokens the prediction's bias update, dispatched behind them, no longer delays the next layer. - The QSA mixer: the indexer's key and query projections in one launch (bf16_gemv_fp32_mmvf_multi takes a second matrix), the key tail's snapshot in the append, each group of heads' RMS norm and RoPE in one kernel (native_qsa_norm_rope_tokens; the queries read from their projection's rows, the split copy gone), the keys and values dispatched ahead of the queries' grid, and that grid after the indexer's projections, so the append goes ahead of it: 8-12 us less a QSA layer. - The PLE block: both projections of a window's tokens in one pass each, then native_ple_postops_tokens, which now writes the history after each token for the commit. bf16_gemv_fp32_mmvf_multi takes 4 rows a block from 2 tokens (each pair's inputs loaded once for them) and the one-row kernel at 1 token: the PLE section 110-174 us a window of 1-4 tokens instead of 112-428 (IQ3_XXS). - The verify window requires the native combine, shared expert gate, QSA norms and RoPE and PLE block (the --native preset's); their fallbacks are gone from it. - verify_parity: the gated combine, the fused norm and RoPE, the two-matrix and 4-row projections, the append's tail snapshot, the history after each token. Window hashes identical (thinking, code, after 16K; both models; --spec-split and --second-gpu -1). Fixed text, both GPUs, against 962b99c, ms per round (2 rounds; UD-Q4_K_XL code 5): IQ3_XXS code -6.1%, thinking -2.4%, after 16K -2.5%; UD-Q4_K_XL code +1.6% (its CPU pool's drifting RAM rate: the serial part -0.3 ms, the pool +0.9), thinking -2.0%, after 16K -1.4%. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/verify.hpp | 13 +- include/strata/kernels/bf16_gemv.hpp | 5 +- include/strata/kernels/native_moe.hpp | 9 +- include/strata/kernels/native_ple_postops.hpp | 11 +- include/strata/kernels/native_qsa.hpp | 7 + include/strata/kernels/native_qsa_indexer.hpp | 5 +- include/strata/kernels/ple.hpp | 1 + include/strata/kernels/shared_expert.hpp | 20 +- src/core/layer.cpp | 4 + src/core/verify.cpp | 207 +++++++++--------- src/kernels/cuda/mmvf_multi.cuh | 73 +++++- src/kernels/cuda/native_bf16.cu | 40 +++- src/kernels/cuda/native_moe.cu | 35 ++- src/kernels/cuda/native_ple_postops.cu | 26 ++- src/kernels/cuda/native_qsa.cu | 63 ++++++ src/kernels/cuda/native_qsa_indexer.cu | 18 +- src/kernels/cuda/ple.cu | 1 + src/kernels/cuda/shared_expert.cu | 33 +-- src/kernels/verify_parity.cpp | 162 +++++++++++--- 19 files changed, 534 insertions(+), 199 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index b4400384244..e2b67c5803e 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -25,7 +25,8 @@ // the CPU (`pool_takes_q8k` layers) and q8_1 rows for the GPUs, so neither quantizes it. // // Requires the default native decode configuration (native projections, fused GR, fused GDN, fast attention and -// selection, native indexer) and a profile-filled VRAM expert tier. +// selection, native indexer, QSA norms, RoPE, combine, shared expert gate and PLE block) and a profile-filled VRAM +// expert tier. #pragma once #include "strata/core/expert_source.hpp" @@ -241,6 +242,10 @@ class Verifier { cudaEvent_t g2fork_ = nullptr, g2join_[2] = {}; cudaStream_t cps_ = nullptr; // the wait for the CPU's rows, per token group cudaEvent_t cpfork_ = nullptr, cpjoin_[2] = {}; + cudaStream_t kst_ = nullptr; // the QSA mixer's keys + cudaEvent_t kjoin_ = nullptr, iproj_ = nullptr; // and the end of its indexer's projections + cudaStream_t sgs_ = nullptr; // the shared expert's scalar gate + cudaEvent_t sgjoin_ = nullptr; cudaEvent_t efork_ = nullptr, rdone_ = nullptr; // the FFN read's end, the router's cudaEvent_t done_ = nullptr; // after the last window's graph @@ -258,7 +263,7 @@ class Verifier { float *qidx_ = nullptr, *scores_ = nullptr, *attn_ = nullptr, *attn32_ = nullptr, *attn_scratch_ = nullptr; float* tail_snap_ = nullptr; // per QSA layer int32_t* sel_ = nullptr; - float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *hit_out_ = nullptr; + float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *hit_out_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr; unsigned* ring_count_ = nullptr; // verify_router's block counters: the layer's, the prediction's float* plogits_ = nullptr; // the predictions' logits, by layer parity @@ -278,9 +283,11 @@ class Verifier { float* hit_xs_ = nullptr; void* hit_scratch_ = nullptr; float *head_mixed_ = nullptr, *head_inj_ = nullptr, *head_logits_ = nullptr; - uint16_t* sh_bf16_ = nullptr; float *sh_gate_ = nullptr, *sh_up_ = nullptr, *sh_g_ = nullptr; float* hist_snap_ = nullptr; // T * NG_HIST * NG_HC_DIM + // the PLE block's projections (keys T * NG_HC_DIM, values T * n_embd) and the rest of the block's rows + float *ple_key_ = nullptr, *ple_val_ = nullptr, *ple_nkey_ = nullptr, *ple_norm_ = nullptr, *ple_gated_ = nullptr; + float* ple_gate_ = nullptr; int64_t cap_ = 0, max_blocks_ = 0, attn_scratch_floats_ = 0; // --window-profile: stamps [window size][layer, then one row for the window][stage]: a layer's main-stream stages, diff --git a/include/strata/kernels/bf16_gemv.hpp b/include/strata/kernels/bf16_gemv.hpp index 020bf30b7f0..822202ea955 100644 --- a/include/strata/kernels/bf16_gemv.hpp +++ b/include/strata/kernels/bf16_gemv.hpp @@ -45,8 +45,9 @@ void bf16_gemv_split(const uint16_t* x, const uint16_t* w, float* y, int64_t n_i void bf16_gemv_fp32_mmvf(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, void* stream); /// `bf16_gemv_fp32_mmvf` for n_tok (1..8) activation rows x (n_tok, n_in) -> y (n_tok, n_out), each weight read once; -/// every row bitwise what the single-row call gives. +/// every row bitwise what the single-row call gives. With `w2` the same launch also gives y2 (n_tok, n_out2) from a +/// second matrix of the same input. void bf16_gemv_fp32_mmvf_multi(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, int n_tok, - void* stream); + void* stream, const uint16_t* w2 = nullptr, float* y2 = nullptr, int64_t n_out2 = 0); } // namespace strata::kernels diff --git a/include/strata/kernels/native_moe.hpp b/include/strata/kernels/native_moe.hpp index 69cd2a2e3cb..cb2dc21895c 100644 --- a/include/strata/kernels/native_moe.hpp +++ b/include/strata/kernels/native_moe.hpp @@ -20,9 +20,12 @@ void native_moe_combine(const float* parts, const float* weights, const float* s // used to assemble: row t*k + j comes from `gpu_rows` (row-indexed) as the host's zeroed row plus the hit when an // entry of dst[0..*count) or dst2[0..*count2) names it, as it is when an entry of dst3[0..*count3) does, and // otherwise from `host_rows` (mapped memory: only these rows cross PCIe). `dst2` and `dst3` may be null. -// weights (n_tok, k), shared and output (n_tok, n_embd). Graph-capturable. +// weights (n_tok, k), shared and output (n_tok, n_embd). With `shared_gate` (n_tok logits of the shared expert's +// scalar gate, shared_expert_multi) the shared rows are its down projection's, and each is gated first, bitwise +// shared_expert_gate_rows. Graph-capturable. void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, const int32_t* count, const int32_t* dst2, const int32_t* count2, const int32_t* dst3, - const int32_t* count3, const float* weights, const float* shared, float* output, - int64_t n_embd, int64_t k, int n_tok, void* stream); + const int32_t* count3, const float* weights, const float* shared, + const float* shared_gate, float* output, int64_t n_embd, int64_t k, int n_tok, + void* stream); } diff --git a/include/strata/kernels/native_ple_postops.hpp b/include/strata/kernels/native_ple_postops.hpp index ad6d0c4f881..2fe69748206 100644 --- a/include/strata/kernels/native_ple_postops.hpp +++ b/include/strata/kernels/native_ple_postops.hpp @@ -27,10 +27,11 @@ void native_ple_postops(const float* projected_key, const float* hidden, const PleWeights& weights, const NativePlePostopsBuffers& buffers, void* stream); -// n_tok consecutive tokens (the prompt path), row-major with the token slow: projected_key, hidden, key, query, -// gated, normalized and result [n_tok, 10240], value [n_tok, 2560], gate [n_tok, 4]. Bitwise n_tok calls of -// native_ple_postops, each followed by ple_history_advance: history is read, then left holding the last nine -// normalized rows. result may equal hidden and query may equal normalized; nothing else may overlap. +// n_tok consecutive tokens (the prompt path, a verify window's group), row-major with the token slow: projected_key, +// hidden, key, query, gated, normalized and result [n_tok, 10240], value [n_tok, 2560], gate [n_tok, 4]. Bitwise +// n_tok calls of native_ple_postops, each followed by ple_history_advance: history is read, then left holding the +// last nine normalized rows; `snaps` (optional, [n_tok, 9 * 10240]) receives the history after each token. result +// may equal hidden and query may equal normalized; nothing else may overlap. struct NativePleTokensBuffers { float* key; float* query; @@ -41,6 +42,6 @@ struct NativePleTokensBuffers { }; void native_ple_postops_tokens(const float* projected_key, const float* hidden, const float* value, float* history, const PleWeights& weights, const NativePleTokensBuffers& buffers, - int n_tok, void* stream); + int n_tok, void* stream, float* snaps = nullptr); } // namespace strata::kernels diff --git a/include/strata/kernels/native_qsa.hpp b/include/strata/kernels/native_qsa.hpp index c56eb4d29c3..961eed7eeee 100644 --- a/include/strata/kernels/native_qsa.hpp +++ b/include/strata/kernels/native_qsa.hpp @@ -21,6 +21,13 @@ bool native_qsa_enabled(); void native_qsa_rms_norm_weighted(const float* input, const float* gamma, float* output, int n_cols, int n_rows, float epsilon, void* stream); +// native_qsa_rms_norm_weighted, then native_rope_apply_tokens (64 rotated columns) on its output, in one launch and +// bitwise both: n_rows rows of n_cols (128 or 256) read at input + r * in_stride, written contiguously to output; +// row r's position is positions[(r / heads) * pos_stride + r % heads]. output == input needs in_stride == n_cols. +void native_qsa_norm_rope_tokens(const float* input, int in_stride, const float* gamma, float* output, int n_cols, + int n_rows, float epsilon, int n_rot, float freq_base, const int* positions, int heads, + int pos_stride, void* stream); + // attn/output [head_dim,n_head]; q_full [2*head_dim,n_head], each row is // [query channels, gate channels]. Output is attn*sigmoid(second-half gate). // Exact output==attn is supported; all other spans must be disjoint. diff --git a/include/strata/kernels/native_qsa_indexer.hpp b/include/strata/kernels/native_qsa_indexer.hpp index d51f57a560d..ce53fef8274 100644 --- a/include/strata/kernels/native_qsa_indexer.hpp +++ b/include/strata/kernels/native_qsa_indexer.hpp @@ -32,10 +32,11 @@ void native_qsa_indexer_append(const float* raw, const int32_t* relative_pos_dev const QsaIndexerBuffers& buffers, const QsaShapes& shapes, int64_t max_cells, float freq_base, void* stream); // n_tok appends in one launch, in order: token t's raw key at raw + t * 128, its position at -// relative_pos_device[t * pos_stride]. Bitwise n_tok calls of native_qsa_indexer_append. +// relative_pos_device[t * pos_stride]. Bitwise n_tok calls of native_qsa_indexer_append. With `tail_snap` +// (3 * 128 floats) the key tail as it was before them is copied there first. void native_qsa_indexer_append_multi(const float* raw, const int32_t* relative_pos_device, int pos_stride, int n_tok, int32_t pos_base, const float* gamma, float epsilon, const QsaIndexerBuffers& buffers, const QsaShapes& shapes, - int64_t max_cells, float freq_base, void* stream); + int64_t max_cells, float freq_base, void* stream, float* tail_snap = nullptr); } // namespace strata::kernels diff --git a/include/strata/kernels/ple.hpp b/include/strata/kernels/ple.hpp index fabad838a3e..09c2da99e1c 100644 --- a/include/strata/kernels/ple.hpp +++ b/include/strata/kernels/ple.hpp @@ -33,6 +33,7 @@ namespace strata::kernels { /// Opt in to pinned CUDA BF16/F32 MMVF for the PLE value projection only (default false). /// Configure before session capture; captured graphs retain their selected projection kernels. void ple_set_native_bf16(bool enabled); +bool ple_native_bf16(); /// Opt in to pinned CUDA postprojection norms, gate, convolution and residual /// arithmetic (default false). Set before capture; existing graphs keep their diff --git a/include/strata/kernels/shared_expert.hpp b/include/strata/kernels/shared_expert.hpp index 948fd896e7a..b69f219c62c 100644 --- a/include/strata/kernels/shared_expert.hpp +++ b/include/strata/kernels/shared_expert.hpp @@ -69,13 +69,19 @@ void shared_expert(const uint8_t* x_q8_0, const uint8_t* x_q8k, const uint16_t* int64_t n_embd, int64_t n_ff, int tpr, void* stream, const float* x_f32 = nullptr, const NativeSharedWeights* native = nullptr); -/// Plan v0.3 P6: the shared expert for `n_tok` <= 8 tokens (a verify window) with all three projections native: -/// multi-column MMVQ, so the weights are read once; every token is bitwise `shared_expert` on that token. -/// `x` (n_tok, n_embd) f32, `x_bf16` the same rounded (only read when the native BF16 gate is off), `gate`/`up` -/// (n_tok, n_ff) scratch, `g` n_tok floats, `out` (n_tok, n_embd). `nw.q8_1` must hold n_tok columns of n_embd. -void shared_expert_multi(int n_tok, const float* x, const uint16_t* x_bf16, const NativeSharedWeights& nw, - const uint16_t* gate_inp_bf16, float* gate, float* up, float* g, float* out, int64_t n_embd, - int64_t n_ff, void* stream); +/// The native scalar gate applied: `g` (n_tok logits) becomes sigmoid(g), and each token's row of `out` (n_tok, +/// n_embd) is multiplied by it. +void shared_expert_gate_rows(float* out, float* g, int64_t n_embd, int n_tok, void* stream); + +/// Plan v0.3 P6: the shared expert for `n_tok` <= 8 tokens (a verify window) with all three projections and the gate +/// native: multi-column MMVQ, so the weights are read once. `x` (n_tok, n_embd) f32, `gate`/`up` (n_tok, n_ff) +/// scratch; `out` (n_tok, n_embd) receives the down projection's rows and `g` the scalar gate's n_tok logits, +/// computed on `gate_stream` (may be `stream`), so that the chain ends with the down projection: +/// `shared_expert_gate_rows` on them, or `native_moe_gather_combine` applying them, gives every token bitwise +/// `shared_expert` on that token. `nw.q8_1` must hold n_tok columns of n_embd. +void shared_expert_multi(int n_tok, const float* x, const NativeSharedWeights& nw, const uint16_t* gate_inp_bf16, + float* gate, float* up, float* g, float* out, int64_t n_embd, int64_t n_ff, void* stream, + void* gate_stream); /// The MoE block's final combination, `ref/moe.py::moe` L156: /// diff --git a/src/core/layer.cpp b/src/core/layer.cpp index 1e05f020e58..eee96a65997 100644 --- a/src/core/layer.cpp +++ b/src/core/layer.cpp @@ -447,6 +447,10 @@ bool layer_verify_compatible(std::string& why) { else if (!g_fast_attn || native_flash_attn_short) why = "the split-K decode attention is off"; else if (!g_fast_select) why = "the block top-k selection is off"; else if (!strata::kernels::native_qsa_indexer_enabled()) why = "the native QSA indexer is off"; + else if (!strata::kernels::native_qsa_enabled() || !strata::kernels::native_rope_enabled()) + why = "the native QSA norms or RoPE are off"; + else if (!strata::kernels::native_moe_combine_enabled()) why = "the native MoE combine is off"; + else if (!strata::kernels::shared_expert_native_bf16()) why = "the shared expert's native gate is off"; else return true; return false; } diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 095425091b2..75cc4924f38 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -15,6 +15,7 @@ #include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_mmvq.hpp" #include "strata/kernels/native_moe.hpp" +#include "strata/kernels/native_ple_postops.hpp" #include "strata/kernels/native_qsa.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_router.hpp" @@ -25,7 +26,6 @@ #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/qsa_select.hpp" #include "strata/kernels/quantize_act.hpp" -#include "strata/kernels/rope.hpp" #include "strata/kernels/s2_expert_grouped.hpp" #include "strata/kernels/sampler.hpp" #include "strata/kernels/shared_expert.hpp" @@ -102,8 +102,11 @@ Verifier::~Verifier() { if (shs_) cudaStreamDestroy(shs_); if (g2s_) cudaStreamDestroy(g2s_); if (cps_) cudaStreamDestroy(cps_); + if (kst_) cudaStreamDestroy(kst_); + if (sgs_) cudaStreamDestroy(sgs_); for (cudaEvent_t e : {join_, bfork_, bjoin_, pfork_, pjoin_, shfork_, shjoin_, res_ready_, done_, g2fork_, - g2join_[0], g2join_[1], cpfork_, cpjoin_[0], cpjoin_[1], efork_, rdone_}) + g2join_[0], g2join_[1], cpfork_, cpjoin_[0], cpjoin_[1], efork_, rdone_, kjoin_, iproj_, + sgjoin_}) if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); @@ -140,6 +143,11 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& err = "verify: geometry differs from the artifact's"; return false; } + if (ss.ple.ready() && (!strata::kernels::ple_native_postops_enabled() || !strata::kernels::ple_native_bf16() || + (ss.ple.w.key_bf16 == nullptr && ss.ple.w.key_native_data == nullptr))) { + err = "verify: the PLE block needs the native postops and value and a BF16 or native key"; + return false; + } const WeightRef* wo = wt.find("output.weight"); if (wo == nullptr) { err = "verify: output.weight is missing"; return false; } n_vocab_ = wo->ne1; @@ -211,7 +219,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sink_.start2 = h_plan_ + plan_ptr_off_ + 4 * cap; sink_.cap = cap; sink_.publish = &Verifier::publish_plan; - sink_.host_rows_only = strata::kernels::native_moe_combine_enabled(); // the gather-combine (post) + sink_.host_rows_only = true; // the gather-combine (post) reads the GPUs' from VRAM sink_.fetch = &Verifier::fetch_dma; sink_.ctx = this; } @@ -249,7 +257,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& pids_ = b.take(T * K); pmixed_ = b.take(T * N); gate_ema_ = b.take((uint64_t) g.n_layers * HC * N); pbias_ = b.take((uint64_t) (g.n_layers * g.n_expert)); - shared_ = b.take(T * N); parts_ = b.take(T * K * N); hit_out_ = b.take(T * K * N); + shared_ = b.take(T * N); hit_out_ = b.take(T * K * N); hit_slot_ = b.take(T * K); hit_dst_ = b.take(T * K); hit_count_ = b.take(4); list2_ = b.take(2 * (uint64_t) list2_stride_); ring_count_ = b.take(2); @@ -262,10 +270,13 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& strata::kernels::moe_hit_grouped_scratch_bytes((int64_t) (T * K), g.n_embd, g.n_ff), strata::kernels::native_expert_scratch_bytes((int64_t) (T * K), g.n_ff))); head_mixed_ = b.take(T * N); head_inj_ = b.take(HC); - sh_bf16_ = b.take(T * N); sh_gate_ = b.take(T * (uint64_t) g.n_ff); + sh_gate_ = b.take(T * (uint64_t) g.n_ff); sh_up_ = b.take(T * (uint64_t) g.n_ff); sh_g_ = b.take(T + 4); head_logits_ = b.take(T * (uint64_t) n_vocab_); hist_snap_ = b.take(T * HS); + const uint64_t PD = (uint64_t) strata::kernels::NG_HC_DIM; + ple_key_ = b.take(T * PD); ple_val_ = b.take(T * N); ple_nkey_ = b.take(T * PD); + ple_norm_ = b.take(T * PD); ple_gated_ = b.take(T * PD); ple_gate_ = b.take(T * HC); }; Bump count; carve(count); @@ -306,6 +317,11 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& cudaStreamCreateWithFlags(&shs_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&g2s_, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&cps_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&kst_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&kjoin_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&iproj_, cudaEventDisableTiming) != cudaSuccess || + cudaStreamCreateWithFlags(&sgs_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&sgjoin_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&cpfork_, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&cpjoin_[0], cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&cpjoin_[1], cudaEventDisableTiming) != cudaSuccess || @@ -446,7 +462,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } // the shared expert: in a window of one token group on a branch of its own forked after the FFN read (`efork_`), - // beside the router and the routed experts (joined before the combine); in a split window inline + // beside the router and the routed experts (joined before the combine); in a split window inline. It ends with + // its down projection: the combine applies its scalar gate, whose logits come from a branch of their own. const bool beside = G == 1; auto shared = [&](int64_t l, int grp) -> bool { const int tb = tb_[grp], te = te_[grp], n = te - tb; @@ -462,27 +479,26 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { nsw.up_type = wsu->native_type; nsw.up_data = wsu->native_data; nsw.down_type = wsd->native_type; nsw.down_data = wsd->native_data; nsw.q8_1 = xq_; // the main stream's next use is the next layer's mixer - cudaStream_t ws = cs; + cudaStream_t ws = cs, gs = cs; if (beside) { - if (cudaStreamWaitEvent(shs_, efork_, 0) != cudaSuccess) { + if (cudaStreamWaitEvent(shs_, efork_, 0) != cudaSuccess || + cudaStreamWaitEvent(sgs_, efork_, 0) != cudaSuccess) { err = "verify: the shared expert's branch could not fork"; return false; } ws = shs_; + gs = sgs_; } - if (!shared_expert_native_bf16()) // the scalar gate's BF16 input (the native gate reads x) - for (int t = tb; t < te; ++t) f32_to_bf16_bulk(mixed_ + t * N, sh_bf16_ + t * N, N, ws); try { - shared_expert_multi(n, mixed_ + tb * N, sh_bf16_ + tb * N, nsw, (const uint16_t*) wgi->data, - sh_gate_ + (size_t) tb * g.n_ff, sh_up_ + (size_t) tb * g.n_ff, sh_g_ + tb, shared_ + tb * N, - N, g.n_ff, ws); + shared_expert_multi(n, mixed_ + tb * N, nsw, (const uint16_t*) wgi->data, sh_gate_ + (size_t) tb * g.n_ff, + sh_up_ + (size_t) tb * g.n_ff, sh_g_ + tb, shared_ + tb * N, N, g.n_ff, ws, gs); } catch (const std::exception& e) { err = std::string("verify shared expert: ") + e.what(); return false; } if (beside) { stamp(l, kLayerStamps, shs_); - if (cudaEventRecord(shjoin_, shs_) != cudaSuccess) { + if (cudaEventRecord(shjoin_, shs_) != cudaSuccess || cudaEventRecord(sgjoin_, sgs_) != cudaSuccess) { err = "verify: the shared expert's branch could not join"; return false; } @@ -505,25 +521,46 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!wn[h] || !wd[h] || !wu[h] || !wi[h]) return false; } bool pending = l > 0; // the previous layer's FFN write, folded into this layer's first read + // the projections of the group's tokens: 2+ tokens are quantized with the interleaved copy the multi-token + // kernels read + auto quant = [&](const float* x, int64_t n_in) { + if (n >= 2) native_quantize_q8_1_il(x, xq_, xil_, (int) n_in, n, cs); + else native_quantize_q8_1(x, xq_, (int) n_in, n, cs); + }; + auto mm = [&](int type, const void* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { + if (n >= 2) native_mmvq_il(type, w, xq_, xil_, y, (int) n_in, (int) n_out, n, ps); + else native_mmvq(type, w, xq_, y, (int) n_in, (int) n_out, n, ps); + }; + auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { + mm(w->native_type, w->native_data, y, n_in, n_out, ps); + }; if (l == 1 && ple_on) { if (grp == 0) { // the host reads the rows while layer 0 runs wait_flag_ge(m_pleflag_, 1, cs); copy_from_mapped(ple_, m_ple_, (int64_t) T * N, cs); } - float* normalized = (float*) ((uint8_t*) ss.ple.scratch + ple_block_scratch_bytes()); - for (int t = tb; t < te; ++t) { - gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); - PleOut po; - po.normalized = normalized; - po.result = Rt(t); - try { - ple_block(ple_ + t * N, Rt(t), ss.ple.hist, ss.ple.w, po, ss.ple.scratch, cs); - ple_history_advance(ss.ple.hist, normalized, cs); - } catch (const std::exception& e) { - err = std::string("verify PLE: ") + e.what(); - return false; + for (int t = tb; t < te; ++t) gr_write(Rt(t), bo_ + t * N, inj2_ + t * HC, gs, Rt(t), cs); + // the PLE block: the group's projections with each weight read once, then the rest of the block over its + // tokens in order, which leaves the history after each of them for the commit + const PleWeights& pw = ss.ple.w; + const float* emb = ple_ + tb * N; + float* key = ple_key_ + (size_t) tb * NG_HC_DIM; + float* norm = ple_norm_ + (size_t) tb * NG_HC_DIM; + try { + if (pw.key_bf16 != nullptr) { + bf16_gemv_fp32_mmvf_multi(emb, pw.key_bf16, key, N, NG_HC_DIM, n, cs); + } else { + quant(emb, N); + mm(pw.key_native_type, pw.key_native_data, key, N, NG_HC_DIM, cs); } - copy_from_mapped(hist_snap_ + (size_t) t * HS, ss.ple.hist, HS, cs); + bf16_gemv_fp32_mmvf_multi(emb, pw.value_bf16, ple_val_ + tb * N, N, N, n, cs); + const NativePleTokensBuffers pb{ple_nkey_ + (size_t) tb * NG_HC_DIM, norm, ple_gate_ + tb * HC, + ple_gated_ + (size_t) tb * NG_HC_DIM, norm, Rt(tb)}; + native_ple_postops_tokens(key, Rt(tb), ple_val_ + tb * N, ss.ple.hist, pw, pb, n, cs, + hist_snap_ + (size_t) tb * HS); + } catch (const std::exception& e) { + err = std::string("verify PLE: ") + e.what(); + return false; } pending = false; } @@ -545,16 +582,6 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { gr_read_group(0, pending, inj2_, inj_, nullptr, nullptr, nullptr); stamp(l, 1, cs); float* xm = mixed_ + tb * N; - // the mixer's projections: a group of 2+ tokens is quantized with the interleaved copy the multi-token kernels - // read - auto quant = [&](const float* x, int64_t n_in) { - if (n >= 2) native_quantize_q8_1_il(x, xq_, xil_, (int) n_in, n, cs); - else native_quantize_q8_1(x, xq_, (int) n_in, n, cs); - }; - auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { - if (n >= 2) native_mmvq_il(w->native_type, w->native_data, xq_, xil_, y, (int) n_in, (int) n_out, n, ps); - else native_mmvq(w->native_type, w->native_data, xq_, y, (int) n_in, (int) n_out, n, ps); - }; try { if (!is_qsa_layer(g, l)) { // ======================= GDN ======================= @@ -606,55 +633,51 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!native_of(wq, v.name("attn_q.weight"), err) || !native_of(wk, v.name("attn_k.weight"), err) || !native_of(wv, v.name("attn_v.weight"), err) || !native_of(wo, v.name("attn_output.weight"), err)) return false; - auto norm_rope = [&](float* data, const WeightRef* norm, int rows, int cols, const int32_t* pos, - cudaStream_t ns) { - if (native_qsa_enabled()) native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, rows, EPS, ns); - else rms_norm_weighted(data, (const float*) norm->data, rows, cols, EPS, ns); - if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, ns); - else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st.cos_tab, st.sin_tab, pos, ns); - }; - // the same for the n tokens' heads in one launch each: token t's positions at pos + t * NH - const bool batched = native_qsa_enabled() && native_rope_enabled(); - auto norm_rope_tokens = [&](float* data, const WeightRef* norm, int heads, int cols, const int32_t* pos, - cudaStream_t ns) { - native_qsa_rms_norm_weighted(data, (const float*) norm->data, data, cols, n * heads, EPS, ns); - native_rope_apply_tokens(data, data, n * heads, cols, (int) s.n_rot, (float) qsa_freq_base(), pos, heads, - (int) NH, ns); + // the rms norm and rope of the group's heads in one launch: rows of `cols` read at in + r * stride, + // token t's positions at pos_ + t * NH + auto norm_rope = [&](const float* in, int stride, float* out, const WeightRef* norm, int heads, + int cols, cudaStream_t ns) { + native_qsa_norm_rope_tokens(in, stride, (const float*) norm->data, out, cols, n * heads, EPS, + (int) s.n_rot, (float) qsa_freq_base(), pos_ + tb * NH, heads, (int) NH, + ns); }; - // two branches beside the keys and values, joined before the attention: the indexer (its keys appended, - // the queries' block scores and top-k), and the queries + // three branches beside the values, joined before the attention: the indexer (its key and query + // projections in one launch, the keys appended, the queries' block scores and top-k), the keys and the + // queries. A grid is dispatched whole before a later kernel's blocks, so the keys' and values' + // projections go ahead of the queries' grid, and that grid waits for the indexer's projections: the + // append is dispatched ahead of it, and the indexer's last kernels end about when the queries' do. if (!fork(side_, bfork_)) return false; float* idx_raw = idx_raw_L_ + (size_t) qi * MT * ID; - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, side_); - if (grp == 0) copy_from_mapped(tail_snap_ + (size_t) qi * TS, st.idx_tail, TS, side_); + bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wik->data, idx_raw + tb * ID, N, ID, n, side_, + (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, IQ * ID); + if (!mark(side_, iproj_)) return false; const QsaIndexerBuffers ib{st.idx_tail, st.idx_dead, st.idx_pooled, st.idx_block_pos}; native_qsa_indexer_append_multi(idx_raw + tb * ID, step_ + tb * kStepCount + kStepPos, (int) kStepCount, n, 0, (const float*) wikn->data, EPS, ib, s, st.max_cells, - (float) qsa_freq_base(), side_); - bf16_gemv_fp32_mmvf_multi(xm, (const uint16_t*) wiq->data, qidx_ + tb * IQ * ID, N, IQ * ID, n, side_); - if (batched) norm_rope_tokens(qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + tb * NH, side_); - else for (int t = tb; t < te; ++t) norm_rope(qidx_ + t * IQ * ID, wiqn, (int) IQ, (int) ID, pos_ + t * NH, side_); + (float) qsa_freq_base(), side_, + grp == 0 ? tail_snap_ + (size_t) qi * TS : nullptr); + norm_rope(qidx_ + tb * IQ * ID, (int) ID, qidx_ + tb * IQ * ID, wiqn, (int) IQ, (int) ID, side_); qsa_block_scores(st.idx_pooled, st.idx_dead, qidx_ + tb * IQ * ID, step_ + tb * kStepCount, n, max_blocks_, s, scores_ + (size_t) tb * max_blocks_, side_, qsa_score_grid_blocks); qsa_block_topk(scores_ + (size_t) tb * max_blocks_, step_ + tb * kStepCount, n, max_blocks_, cap_, s, sel_ + (size_t) tb * cap_, side_); if (!mark(side_, bjoin_)) return false; quant(xm, N); - if (!fork(shs_, pfork_)) return false; - proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD, shs_); - // the query halves of every token's heads (rows of 2 * HD: query, then gate) - if (cudaMemcpy2DAsync(qcur_ + tb * NH * HD, (size_t) HD * 4, qfull_ + tb * NH * 2 * HD, (size_t) HD * 2 * 4, - (size_t) HD * 4, (size_t) (n * NH), cudaMemcpyDeviceToDevice, shs_) != cudaSuccess) { - err = "verify: the q/gate split failed"; + if (cudaEventRecord(pfork_, cs) != cudaSuccess || cudaStreamWaitEvent(kst_, pfork_, 0) != cudaSuccess || + cudaStreamWaitEvent(shs_, pfork_, 0) != cudaSuccess || + cudaStreamWaitEvent(shs_, iproj_, 0) != cudaSuccess) { + err = "verify: a branch could not fork"; return false; } - if (batched) norm_rope_tokens(qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, pos_ + tb * NH, shs_); - else for (int t = tb; t < te; ++t) norm_rope(qcur_ + t * NH * HD, wqn, (int) NH, (int) HD, pos_ + t * NH, shs_); - if (!mark(shs_, pjoin_)) return false; - proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD, cs); + proj(wk, kcur_ + tb * NKV * HD, N, NKV * HD, kst_); + norm_rope(kcur_ + tb * NKV * HD, (int) HD, kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, kst_); + if (!mark(kst_, kjoin_)) return false; proj(wv, vcur_ + tb * NKV * HD, N, NKV * HD, cs); - if (batched) norm_rope_tokens(kcur_ + tb * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + tb * NH, cs); - else for (int t = tb; t < te; ++t) norm_rope(kcur_ + t * NKV * HD, wkn, (int) NKV, (int) HD, pos_ + t * NH, cs); + proj(wq, qfull_ + tb * NH * 2 * HD, N, NH * 2 * HD, shs_); + // the query halves of every token's heads (rows of 2 * HD: query, then gate) + norm_rope(qfull_ + tb * NH * 2 * HD, (int) (2 * HD), qcur_ + tb * NH * HD, wqn, (int) NH, (int) HD, + shs_); + if (!mark(shs_, pjoin_) || !wait(kjoin_)) return false; if (st.kv_int8) kv_append_q8_steps(st.k_q, st.v_q, st.k_scale, st.v_scale, st.page_table, step_ + tb * kStepCount, (int) kStepCount, kcur_ + tb * NKV * HD, vcur_ + tb * NKV * HD, n, s, cs); @@ -669,12 +692,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } qsa_decode_attn_batch(qcur_ + tb * NH * HD, pools, sel_ + (size_t) tb * cap_, step_ + tb * kStepCount, cap_, s, attn_scratch_ + (size_t) tb * attn_scratch_floats_, attn_ + tb * NH * HD, n, cs); - if (native_qsa_enabled()) - native_qsa_gate_apply(attn_ + tb * NH * HD, qfull_ + tb * NH * 2 * HD, attn32_ + tb * NH * HD, - (int) (n * NH), (int) HD, cs); - else - for (int t = tb; t < te; ++t) - qsa_gate_apply_f32(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, s, attn32_ + t * NH * HD, cs); + native_qsa_gate_apply(attn_ + tb * NH * HD, qfull_ + tb * NH * 2 * HD, attn32_ + tb * NH * HD, + (int) (n * NH), (int) HD, cs); quant(attn32_ + tb * NH * HD, NH * HD); proj(wo, bo_ + tb * N, NH * HD, N, cs); } @@ -874,31 +893,21 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { stamp(l, 9, cs); if (gpu2_ && !wait(g2join_[grp])) return false; // the second GPU's, in `hit_out` stamp(l, 10, cs); - if (beside && cudaStreamWaitEvent(cs, shjoin_, 0) != cudaSuccess) { + if (beside && (cudaStreamWaitEvent(cs, shjoin_, 0) != cudaSuccess || + cudaStreamWaitEvent(cs, sgjoin_, 0) != cudaSuccess)) { err = "verify: the shared expert's branch could not join"; return false; } - if (native_moe_combine_enabled()) { - // only the CPU's rows cross PCIe here; the GPUs' come from `hit_out` (the pool leaves their rows unwritten) - const int32_t* list2 = gpu2_ ? list2_ + grp * list2_stride_ : nullptr; - try { - native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, vram.dst, vram.counts + 1, pcie.dst, - pcie.counts != nullptr ? pcie.counts + 1 : nullptr, - list2 != nullptr ? list2 + 4 : nullptr, list2, w_ + tb * K, shared_ + tb * N, - bo_ + tb * N, N, K, n, cs); - } catch (const std::exception& e) { - err = "verify layer " + std::to_string(l) + " combine: " + e.what(); - return false; - } - } else { - copy_from_mapped(parts_ + (size_t) tb * K * N, m_ymiss_ + (size_t) tb * K * N, (int64_t) n * K * N, cs); - moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, vram.dst, vram.counts + 1, cap, N, cs); - if (pcie_share_) moe_hit_add(parts_ + (size_t) tb * K * N, hit_out, pcie.dst, pcie.counts + 1, cap, N, cs); - for (int t = tb; t < te; ++t) { - MoEBuffers mb = ss.moe; - mb.weights = w_ + t * K; mb.shared = shared_ + t * N; - if (!moe_combine_parts(g, l, K, mb, parts_ + (size_t) t * K * N, bo_ + t * N, cs, err)) return false; - } + // only the CPU's rows cross PCIe here; the GPUs' come from `hit_out` (the pool leaves their rows unwritten) + const int32_t* list2 = gpu2_ ? list2_ + grp * list2_stride_ : nullptr; + try { + native_moe_gather_combine(hit_out, m_ymiss_ + (size_t) tb * K * N, vram.dst, vram.counts + 1, pcie.dst, + pcie.counts != nullptr ? pcie.counts + 1 : nullptr, + list2 != nullptr ? list2 + 4 : nullptr, list2, w_ + tb * K, shared_ + tb * N, + sh_g_ + tb, bo_ + tb * N, N, K, n, cs); + } catch (const std::exception& e) { + err = "verify layer " + std::to_string(l) + " combine: " + e.what(); + return false; } stamp(l, 11, cs); if (branches(l) && cudaStreamWaitEvent(cs, join_, 0) != cudaSuccess) { diff --git a/src/kernels/cuda/mmvf_multi.cuh b/src/kernels/cuda/mmvf_multi.cuh index 11ee5981147..b2ddc663d66 100644 --- a/src/kernels/cuda/mmvf_multi.cuh +++ b/src/kernels/cuda/mmvf_multi.cuh @@ -1,6 +1,6 @@ -// src/kernels/cuda/mmvf_multi.cuh - one block's row of the FP32-activation BF16 MMVF for up to 8 activation rows, for -// bf16_gemv_fp32_mmvf_multi (native_bf16.cu) and the verify window's router (native_router.cu); both translation -// units compile with --use_fast_math, the pinned contract's arithmetic. +// src/kernels/cuda/mmvf_multi.cuh - one block's row, or rows, of the FP32-activation BF16 MMVF for up to 8 activation +// rows, for bf16_gemv_fp32_mmvf_multi (native_bf16.cu) and the verify window's router (native_router.cu); both +// translation units compile with --use_fast_math, the pinned contract's arithmetic. // // Adapted from llama.cpp 3cf03257f219afbe7334045ff7c6a06ac68c627d, ggml/src/ggml-cuda/{mmvf.cu,common.cuh}. // MIT License, Copyright (c) 2023-2026 The ggml authors (see native_bf16.cu for the full notice). @@ -70,5 +70,72 @@ __device__ __forceinline__ void mmvf_multi_row(const float* __restrict__ x, cons } } +// mmvf_multi_row for R consecutive rows of one matrix (the first `rows` of them real): each pair's inputs are loaded +// once for the R rows, and every row's sums and their order are mmvf_multi_row's; thread 0 ends with acc[r][0 .. n_tok). +template +__device__ __forceinline__ void mmvf_multi_rows(const float* __restrict__ x, const uint16_t* __restrict__ row0, int n_in, + int n_tok, int rows, float (&acc)[R][kMmvfMaxRows]) { + const int t = threadIdx.x; + __shared__ float partials[R][kMmvfMaxRows][32]; + if constexpr (BLOCK_SIZE > 32) { + if (t < 32) +#pragma unroll + for (int r = 0; r < R; ++r) +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) partials[r][k][t] = 0.0f; + __syncthreads(); + } +#pragma unroll + for (int r = 0; r < R; ++r) +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) acc[r][k] = 0.0f; + for (int pair = t; pair < n_in / 2; pair += BLOCK_SIZE) { + float w0[R], w1[R]; +#pragma unroll + for (int r = 0; r < R; ++r) { + const uint32_t weight = r < rows ? reinterpret_cast(row0 + (size_t) r * n_in)[pair] : 0u; + w0[r] = f32_from_bf16((uint16_t) weight); + w1[r] = f32_from_bf16((uint16_t) (weight >> 16)); + } +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + const float2 input = reinterpret_cast(x + (size_t) k * n_in)[pair]; +#pragma unroll + for (int r = 0; r < R; ++r) { + acc[r][k] = __fmaf_rn(w0[r], input.x, acc[r][k]); + acc[r][k] = __fmaf_rn(w1[r], input.y, acc[r][k]); + } + } + } +#pragma unroll + for (int r = 0; r < R; ++r) +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + acc[r][k] = mmvf_warp_sum(acc[r][k]); + } + if constexpr (BLOCK_SIZE > 32) { + if ((t & 31) == 0) +#pragma unroll + for (int r = 0; r < R; ++r) +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + partials[r][k][t / 32] = acc[r][k]; + } + __syncthreads(); + if (t < 32) { +#pragma unroll + for (int r = 0; r < R; ++r) +#pragma unroll + for (int k = 0; k < kMmvfMaxRows; ++k) { + if (k >= n_tok) break; + acc[r][k] = mmvf_warp_sum(partials[r][k][t]); + } + } + } +} + } // namespace } // namespace strata::kernels diff --git a/src/kernels/cuda/native_bf16.cu b/src/kernels/cuda/native_bf16.cu index f9d6b576072..069cc212099 100644 --- a/src/kernels/cuda/native_bf16.cu +++ b/src/kernels/cuda/native_bf16.cu @@ -64,15 +64,25 @@ __global__ void bf16_f32_mmvf_kernel(const float* __restrict__ x, const uint16_t if (t == 0) y[blockIdx.x] = acc; } -// `bf16_f32_mmvf_kernel` for up to kMultiT activation rows (mmvf_multi_row) -constexpr int kMultiT = kMmvfMaxRows; +// `bf16_f32_mmvf_kernel` for up to kMultiT activation rows, kRows weight rows a block (mmvf_multi_rows); the blocks +// past n_out's take the rows of a second matrix of the same input +constexpr int kMultiT = kMmvfMaxRows, kRows = 4; template __global__ void bf16_f32_mmvf_multi_kernel(const float* __restrict__ x, const uint16_t* __restrict__ w, - float* __restrict__ y, int n_in, int n_out, int n_tok) { - float acc[kMultiT]; - mmvf_multi_row(x, w + (size_t) blockIdx.x * n_in, n_in, n_tok, acc); + float* __restrict__ y, int n_in, int n_out, int n_tok, + const uint16_t* __restrict__ w2, float* __restrict__ y2, int n_out2) { + const int blocks1 = (n_out + kRows - 1) / kRows; + const bool first = (int) blockIdx.x < blocks1; + const int row0 = (first ? (int) blockIdx.x : (int) blockIdx.x - blocks1) * kRows, n = first ? n_out : n_out2; + const int rows = min(kRows, n - row0); + float acc[kRows][kMultiT]; + mmvf_multi_rows(x, (first ? w : w2) + (size_t) row0 * n_in, n_in, n_tok, rows, acc); if (threadIdx.x == 0) - for (int k = 0; k < n_tok; ++k) y[(size_t) k * n_out + blockIdx.x] = acc[k]; +#pragma unroll + for (int r = 0; r < kRows; ++r) +#pragma unroll + for (int k = 0; k < kMultiT; ++k) + if (r < rows && k < n_tok) (first ? y : y2)[(size_t) k * n + row0 + r] = acc[r][k]; } int mmvf_block_size(int64_t n_in) { @@ -121,20 +131,26 @@ void bf16_gemv_fp32_mmvf(const float* x, const uint16_t* w, float* y, void bf16_gemv_fp32_mmvf_multi(const float* x, const uint16_t* w, float* y, int64_t n_in, int64_t n_out, int n_tok, - void* stream) { + void* stream, const uint16_t* w2, float* y2, int64_t n_out2) { if (n_tok < 1 || n_tok > kMultiT) throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: 1..8 activation rows"); if (n_in <= 0 || (n_in & 1) != 0 || n_in > std::numeric_limits::max() || - n_out <= 0 || n_out > std::numeric_limits::max()) + n_out <= 0 || n_out2 < 0 || n_out + n_out2 > std::numeric_limits::max()) throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: require positive even n_in and positive n_out <= INT_MAX"); - if (x == nullptr || w == nullptr || y == nullptr || + if (x == nullptr || w == nullptr || y == nullptr || (n_out2 > 0 && (w2 == nullptr || y2 == nullptr)) || (reinterpret_cast(x) & 7u) != 0 || - (reinterpret_cast(w) & 3u) != 0 || - (reinterpret_cast(y) & 3u) != 0) + (reinterpret_cast(w) & 3u) != 0 || (reinterpret_cast(w2) & 3u) != 0 || + (reinterpret_cast(y) & 3u) != 0 || (reinterpret_cast(y2) & 3u) != 0) throw std::invalid_argument("bf16_gemv_fp32_mmvf_multi: null or misaligned pointer"); + if (n_tok == 1 && n_out2 == 0) { // the same sums: a row a block streams the weights faster + bf16_gemv_fp32_mmvf(x, w, y, n_in, n_out, stream); + return; + } const cudaStream_t st = (cudaStream_t) stream; + const unsigned blocks = (unsigned) ((n_out + kRows - 1) / kRows + (n_out2 + kRows - 1) / kRows); #define STRATA_MMVF_CASE(N) case N: \ - bf16_f32_mmvf_multi_kernel<<<(unsigned) n_out, N, 0, st>>>(x, w, y, (int) n_in, (int) n_out, n_tok); break + bf16_f32_mmvf_multi_kernel<<>>(x, w, y, (int) n_in, (int) n_out, n_tok, w2, y2, (int) n_out2); \ + break switch (mmvf_block_size(n_in)) { STRATA_MMVF_CASE(32); STRATA_MMVF_CASE(64); diff --git a/src/kernels/cuda/native_moe.cu b/src/kernels/cuda/native_moe.cu index d84356b5d5d..fec14c0fdf8 100644 --- a/src/kernels/cuda/native_moe.cu +++ b/src/kernels/cuda/native_moe.cu @@ -44,14 +44,29 @@ __global__ void combine(const float* __restrict__ parts, const float* __restrict output[col] = sum; } constexpr int kMaxK = 15; +// A shared expert's row times its scalar gate, in shared_expert_gate_rows' instructions (shared_expert.cu builds +// without fast math: nothing flushes to zero): sigmoid(logit) as __fdividef(1, 1 + __expf(-logit)), then the product. +// The explicit rounding keeps ptxas from fusing the product into the sum after it. +__device__ __forceinline__ float gated_shared(float row, float logit) { + float e, d, g, y; + const float one = 1.0f; + asm("mul.rn.f32 %0, %1, 0fBFB8AA3B;" : "=f"(e) : "f"(logit)); + asm("ex2.approx.f32 %0, %1;" : "=f"(d) : "f"(e)); + asm("add.rn.f32 %0, %1, %2;" : "=f"(e) : "f"(d), "f"(one)); + asm("div.approx.f32 %0, %1, %2;" : "=f"(g) : "f"(one), "f"(e)); + asm("mul.rn.f32 %0, %1, %2;" : "=f"(y) : "f"(row), "f"(g)); + return y; +} // One block row per token. The rows a GPU computed are marked from the plan's entries first; each column then -// loads its k values (the host's over PCIe, all in flight at once) and sums them as `combine` does. +// loads its k values (the host's over PCIe, all in flight at once) and sums them as `combine` does, then adds the +// shared expert's row, times its gate when `shared_gate` holds the gates' logits. __global__ void gather_combine(const float* __restrict__ gpu_rows, const float* host_rows, const int32_t* __restrict__ dst, const int32_t* __restrict__ count, const int32_t* __restrict__ dst2, const int32_t* __restrict__ count2, const int32_t* __restrict__ dst3, const int32_t* __restrict__ count3, const float* __restrict__ weights, const float* __restrict__ shared, - float* __restrict__ output, int64_t n_embd, int k) { + const float* __restrict__ shared_gate, float* __restrict__ output, int64_t n_embd, + int k) { __shared__ unsigned on_gpu, as_is; // bit j: row t*k + j is in gpu_rows, a hit / a row taken as it is const int t = blockIdx.y; if (threadIdx.x == 0) on_gpu = as_is = 0u; @@ -86,7 +101,10 @@ __global__ void gather_combine(const float* __restrict__ gpu_rows, const float* if (j >= k) break; sum += v[j] * w[j]; } - if (shared) sum += shared[int64_t(t) * n_embd + col]; + if (shared) { + const float s = shared[int64_t(t) * n_embd + col]; + sum += shared_gate ? gated_shared(s, shared_gate[t]) : s; + } output[int64_t(t) * n_embd + col] = sum; } bool valid_span(const void* p, size_t bytes) { @@ -119,17 +137,18 @@ void native_moe_combine(const float* parts, const float* weights, const float* s } void native_moe_gather_combine(const float* gpu_rows, const float* host_rows, const int32_t* dst, const int32_t* count, const int32_t* dst2, const int32_t* count2, const int32_t* dst3, - const int32_t* count3, const float* weights, const float* shared, float* output, - int64_t n_embd, int64_t k, int n_tok, void* stream) { + const int32_t* count3, const float* weights, const float* shared, + const float* shared_gate, float* output, int64_t n_embd, int64_t k, int n_tok, + void* stream) { if (!stream || n_embd <= 0 || n_embd > std::numeric_limits::max() || k < 1 || k > kMaxK || n_tok < 1 || !gpu_rows || !host_rows || !dst || !count || (dst2 != nullptr && count2 == nullptr) || - (dst3 != nullptr && count3 == nullptr) || !weights || !output) + (dst3 != nullptr && count3 == nullptr) || !weights || !output || (shared_gate != nullptr && !shared)) throw std::invalid_argument("native MoE gather-combine requires a stream, positive width, 1..15 experts " "and its buffers"); const dim3 grid(unsigned((n_embd + 255) / 256), unsigned(n_tok)); gather_combine<<(stream)>>>(gpu_rows, host_rows, dst, count, dst2, count2, - dst3, count3, weights, shared, output, n_embd, - int(k)); + dst3, count3, weights, shared, shared_gate, + output, n_embd, int(k)); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/cuda/native_ple_postops.cu b/src/kernels/cuda/native_ple_postops.cu index 7a2cf51d7ee..c80b371749c 100644 --- a/src/kernels/cuda/native_ple_postops.cu +++ b/src/kernels/cuda/native_ple_postops.cu @@ -110,13 +110,22 @@ __global__ void conv_residual_tokens_kernel(const float* history, const float* n const float activation = sum / (1.0f + expf(-sum)); result[i] = __fadd_rn(hidden[i], __fadd_rn(gated[i], activation)); } -// The history after the chunk: its last nine normalized rows, older rows moved up when the chunk is shorter. -__global__ void history_tokens_kernel(float* history, const float* normalized, int n_tok) { +// The history after the chunk: its last nine normalized rows, older rows moved up when the chunk is shorter; with +// `snaps`, the history after each of its tokens too. +__global__ void history_tokens_kernel(float* history, const float* normalized, int n_tok, float* snaps) { const int c = blockIdx.x * blockDim.x + threadIdx.x; if (c >= D) return; float old[HISTORY]; #pragma unroll for (int r = 0; r < HISTORY; ++r) old[r] = history[c * HISTORY + r]; + if (snaps != nullptr) + for (int t = 1; t <= n_tok; ++t) +#pragma unroll + for (int r = 0; r < HISTORY; ++r) { + const int j = t - HISTORY + r; + snaps[size_t(t - 1) * HISTORY * D + c * HISTORY + r] = + j >= 0 ? normalized[size_t(j) * D + c] : old[r + t]; + } #pragma unroll for (int r = 0; r < HISTORY; ++r) { const int j = n_tok - HISTORY + r; @@ -170,7 +179,8 @@ void native_ple_postops(const float* projected_key, const float* hidden, } void native_ple_postops_tokens(const float* projected_key, const float* hidden, const float* value, float* history, - const PleWeights& w, const NativePleTokensBuffers& b, int n_tok, void* stream) { + const PleWeights& w, const NativePleTokensBuffers& b, int n_tok, void* stream, + float* snaps) { if (!stream || n_tok < 1) throw std::invalid_argument("native PLE postops: tokens need a stream and n_tok >= 1"); const size_t rows = size_t(n_tok) * D * 4; const Span inputs[] = {{projected_key,rows,4}, {hidden,rows,4}, {value,size_t(n_tok)*N*4,4}, @@ -188,6 +198,14 @@ void native_ple_postops_tokens(const float* projected_key, const float* hidden, if (!(i == 1 && j == 4 && b.query == b.normalized) && overlaps(outputs[i], outputs[j])) throw std::invalid_argument("native PLE postops writable spans overlap"); } + if (snaps != nullptr) { + const Span s{snaps, size_t(n_tok) * HISTORY * D * 4, 4}; + validate(s); + for (const auto& span : inputs) + if (overlaps(s, span)) throw std::invalid_argument("native PLE postops: the snapshots overlap an input"); + for (const auto& span : outputs) + if (overlaps(s, span)) throw std::invalid_argument("native PLE postops: the snapshots overlap an output"); + } const int R = H * n_tok; const unsigned blocks = unsigned((size_t(n_tok) * D + 255) / 256); auto st = static_cast(stream); @@ -198,7 +216,7 @@ void native_ple_postops_tokens(const float* projected_key, const float* hidden, launch_check(); native_gr_rms_norm_weighted_repeat(b.gated,w.norm_conv,b.normalized,N,R,H,NG_RMS_EPS,stream); conv_residual_tokens_kernel<<>>(history,b.normalized,w.conv1d_f16,hidden,b.gated,b.result,n_tok); - history_tokens_kernel<<>>(history,b.normalized,n_tok); + history_tokens_kernel<<>>(history,b.normalized,n_tok,snaps); launch_check(); } } // namespace strata::kernels diff --git a/src/kernels/cuda/native_qsa.cu b/src/kernels/cuda/native_qsa.cu index 32e2511c213..14a29f08641 100644 --- a/src/kernels/cuda/native_qsa.cu +++ b/src/kernels/cuda/native_qsa.cu @@ -23,6 +23,7 @@ // SOFTWARE. #include "strata/kernels/native_qsa.hpp" +#include "strata/kernels/mrope.hpp" #include #include #include @@ -66,6 +67,47 @@ __global__ void norm(const float* input, const float* __restrict__ gamma, float* for (std::size_t col = tid; col < std::size_t(n_cols); col += BlockSize) output[col] = scale * input[col] * gamma[col]; } +// `norm<256>`, then native_rope.cu's `apply` on its output, one block a row of at most 256 columns: the normalized +// row waits in shared memory. Row r is read at input + r * in_stride. +__global__ void norm_rope(const float* input, int in_stride, const float* __restrict__ gamma, float* output, + int n_cols, float epsilon, int n_rot, float theta_scale, const int* __restrict__ positions, + const int32_t* __restrict__ mtab, int heads, int pos_stride) { + constexpr int BlockSize = 256; + const int tid = threadIdx.x, row = blockIdx.x; + input += std::size_t(row) * in_stride; + output += std::size_t(row) * n_cols; + float partial = 0.0f; + for (std::size_t col = tid; col < std::size_t(n_cols); col += BlockSize) { + const float x = input[col]; + partial += x * x; + } + __shared__ float sums[32]; + __shared__ float values[BlockSize]; + partial = warp_sum(partial); + const int lane = tid % 32; + if (lane == 0) sums[tid / 32] = partial; + __syncthreads(); + partial = lane < BlockSize / 32 ? sums[lane] : 0.0f; + partial = warp_sum(partial); + const float mean = partial / n_cols; + const float scale = rsqrtf(mean + epsilon); + for (std::size_t col = tid; col < std::size_t(n_cols); col += BlockSize) + values[col] = scale * input[col] * gamma[col]; + __syncthreads(); // every read of input precedes this: output may be input + const int pair = tid; + if (pair >= n_cols / 2) return; + if (pair >= n_rot / 2) { + output[2 * pair] = values[2 * pair]; + output[2 * pair + 1] = values[2 * pair + 1]; + return; + } + const int position = positions[(row / heads) * pos_stride + row % heads]; + const float theta = mrope_pos(mtab, position, pair) * powf(theta_scale, float(pair)); + const float c = cosf(theta), s = sinf(theta); + const float a = values[pair], b = values[pair + n_rot / 2]; + output[pair] = a * c - b * s; + output[pair + n_rot / 2] = a * s + b * c; +} __global__ void gate(const float* attn, const float* __restrict__ q_full, float* output, int n_head, int head_dim) { const std::size_t i = std::size_t(blockIdx.x) * blockDim.x + threadIdx.x; @@ -117,6 +159,27 @@ void native_qsa_rms_norm_weighted(const float* input, const float* gamma, float* norm<1024><<(stream)>>>(input, gamma, output, n_cols, epsilon); check_launch(); } +void native_qsa_norm_rope_tokens(const float* input, int in_stride, const float* gamma, float* output, int n_cols, + int n_rows, float epsilon, int n_rot, float freq_base, const int* positions, int heads, + int pos_stride, void* stream) { + const auto count = elements(n_cols, n_rows); + if ((n_cols != 128 && n_cols != 256) || n_rot != 64 || in_stride < n_cols || heads < 1 || pos_stride < heads || + !std::isfinite(epsilon) || epsilon < 0.0f || !std::isfinite(freq_base) || freq_base <= 1.0f || !positions) + throw std::invalid_argument("native QSA norm-rope requires 128 or 256 columns, 64 rotated, valid strides, " + "epsilon and base"); + const std::size_t in_bytes = (std::size_t(n_rows - 1) * in_stride + n_cols) * 4; + if (!stream || !valid(input, in_bytes) || !valid(gamma, std::size_t(n_cols) * 4) || !valid(output, count * 4) || + (input != output && overlap(input, in_bytes, output, count * 4)) || + (input == output && in_stride != n_cols) || overlap(output, count * 4, gamma, std::size_t(n_cols) * 4)) + throw std::invalid_argument("native QSA norm-rope requires a stream, aligned spans, and disjoint buffers or an " + "exact input/output alias"); + // pinned host-side float powf, as native_rope_apply_tokens + const float theta_scale = powf(freq_base, -2.0f / n_rot); + norm_rope<<(stream)>>>( + input, in_stride, gamma, output, n_cols, epsilon, n_rot, theta_scale, positions, mrope_table(), heads, + pos_stride); + check_launch(); +} void native_qsa_gate_apply(const float* attn, const float* q_full, float* output, int n_head, int head_dim, void* stream) { const auto count = elements(head_dim, n_head); diff --git a/src/kernels/cuda/native_qsa_indexer.cu b/src/kernels/cuda/native_qsa_indexer.cu index 3a2d51cd0c7..4203a949801 100644 --- a/src/kernels/cuda/native_qsa_indexer.cu +++ b/src/kernels/cuda/native_qsa_indexer.cu @@ -97,11 +97,17 @@ __device__ void append_one(const float* __restrict__ raw, const int pos, } __syncthreads(); // values and partials are reused by the next call } +// With `tail_snap`, the tail as it was is copied there first. __global__ void append(const float* __restrict__ raw, const int32_t* __restrict__ pos_dev, int pos_stride, int n_tok, int pos_base, const float* __restrict__ gamma, float epsilon, float* __restrict__ tail, float* __restrict__ dead, float* __restrict__ pooled, int32_t* __restrict__ block_pos, - int max_cells, float theta_scale, const int32_t* __restrict__ mtab) { + int max_cells, float theta_scale, const int32_t* __restrict__ mtab, + float* __restrict__ tail_snap) { + if (tail_snap != nullptr) { + for (int i = threadIdx.x; i < (R - 1) * D; i += blockDim.x) tail_snap[i] = tail[i]; + __syncthreads(); + } for (int t = 0; t < n_tok; ++t) { append_one(raw + std::size_t(t) * D, pos_dev[std::size_t(t) * pos_stride], pos_base, gamma, epsilon, tail, dead, pooled, block_pos, max_cells, theta_scale, mtab); @@ -131,7 +137,7 @@ void native_qsa_indexer_append(const float* raw, const int32_t* relative_pos_dev void native_qsa_indexer_append_multi(const float* raw, const int32_t* relative_pos_device, int pos_stride, int n_tok, int32_t pos_base, const float* gamma, float epsilon, const QsaIndexerBuffers& b, const QsaShapes& s, int64_t max_cells, - float freq_base, void* stream) { + float freq_base, void* stream, float* tail_snap) { if (n_tok < 1 || pos_stride < 0 || (n_tok > 1 && pos_stride < 1)) throw std::invalid_argument("native QSA indexer: n_tok >= 1 appends with a positive position stride"); if (!stream || s.idx_dim != D || s.idx_block != R || s.n_rot != ROT || @@ -145,9 +151,15 @@ void native_qsa_indexer_append_multi(const float* raw, const int32_t* relative_p for (const auto& span : spans) validate(span); for (int i = 0; i < 7; ++i) for (int j = i + 1; j < 7; ++j) if (overlaps(spans[i], spans[j])) throw std::invalid_argument("native QSA indexer buffers overlap"); + if (tail_snap != nullptr) { + const Span snap{tail_snap, (R - 1) * D * 4}; + validate(snap); + for (const auto& span : spans) + if (overlaps(snap, span)) throw std::invalid_argument("native QSA indexer: the tail snapshot overlaps"); + } const float theta_scale = powf(freq_base, -2.0f / ROT); append<<<1,THREADS,0,static_cast(stream)>>>(raw,relative_pos_device,pos_stride,n_tok,pos_base,gamma, - epsilon,b.tail,b.dead,b.pooled,b.block_pos,int(max_cells),theta_scale,mrope_table()); + epsilon,b.tail,b.dead,b.pooled,b.block_pos,int(max_cells),theta_scale,mrope_table(),tail_snap); const auto error = cudaGetLastError(); if (error != cudaSuccess) throw std::runtime_error(cudaGetErrorString(error)); } diff --git a/src/kernels/cuda/ple.cu b/src/kernels/cuda/ple.cu index 0e0e914f815..092ca68b4c0 100644 --- a/src/kernels/cuda/ple.cu +++ b/src/kernels/cuda/ple.cu @@ -197,6 +197,7 @@ void ck(cudaError_t e, const char* what) { } // namespace void ple_set_native_bf16(bool enabled) { native_bf16 = enabled; } +bool ple_native_bf16() { return native_bf16; } void ple_set_native_postops(bool enabled) { native_postops = enabled; } bool ple_native_postops_enabled() { return native_postops; } diff --git a/src/kernels/cuda/shared_expert.cu b/src/kernels/cuda/shared_expert.cu index d911a45fb15..7e7b874f33c 100644 --- a/src/kernels/cuda/shared_expert.cu +++ b/src/kernels/cuda/shared_expert.cu @@ -160,12 +160,22 @@ __global__ void scale_rows_kernel(float* __restrict__ out, const float* __restri } } // namespace -void shared_expert_multi(int n_tok, const float* x, const uint16_t* x_bf16, const NativeSharedWeights& nw, - const uint16_t* gate_inp_bf16, float* gate, float* up, float* g, float* out, int64_t n_embd, - int64_t n_ff, void* stream) { - if (n_tok < 1 || n_tok > 8 || !nw.q8_1 || !nw.gate_data || !nw.up_data || !nw.down_data || !stream) - throw std::invalid_argument("shared_expert_multi: needs 1..8 tokens, native weights, scratch and a stream"); +void shared_expert_gate_rows(float* out, float* g, int64_t n_embd, int n_tok, void* stream) { cudaStream_t cs = (cudaStream_t) stream; + native_scalar_sigmoid_kernel<<<1, (unsigned) n_tok, 0, cs>>>(g); + scale_rows_kernel<<>>( + out, g, (int) n_embd); +} + +void shared_expert_multi(int n_tok, const float* x, const NativeSharedWeights& nw, const uint16_t* gate_inp_bf16, + float* gate, float* up, float* g, float* out, int64_t n_embd, int64_t n_ff, void* stream, + void* gate_stream) { + if (n_tok < 1 || n_tok > 8 || !nw.q8_1 || !nw.gate_data || !nw.up_data || !nw.down_data || !stream || + !gate_stream || !native_bf16) + throw std::invalid_argument("shared_expert_multi: needs 1..8 tokens, native weights and gate, scratch and " + "streams"); + cudaStream_t cs = (cudaStream_t) stream; + bf16_gemv_fp32_mmvf_multi(x, gate_inp_bf16, g, n_embd, 1, n_tok, gate_stream); native_quantize_q8_1(x, nw.q8_1, (int) n_embd, n_tok, stream); native_mmvq(nw.gate_type, nw.gate_data, nw.q8_1, gate, (int) n_embd, (int) n_ff, n_tok, stream); native_mmvq(nw.up_type, nw.up_data, nw.q8_1, up, (int) n_embd, (int) n_ff, n_tok, stream); @@ -173,15 +183,6 @@ void shared_expert_multi(int n_tok, const float* x, const uint16_t* x_bf16, cons native_swiglu_kernel<<<(unsigned) ((n + THREADS - 1) / THREADS), THREADS, 0, cs>>>(gate, up, gate, n); native_quantize_q8_1(gate, nw.q8_1, (int) n_ff, n_tok, stream); native_mmvq(nw.down_type, nw.down_data, nw.q8_1, out, (int) n_ff, (int) n_embd, n_tok, stream); - if (native_bf16) { - bf16_gemv_fp32_mmvf_multi(x, gate_inp_bf16, g, n_embd, 1, n_tok, stream); - native_scalar_sigmoid_kernel<<<1, (unsigned) n_tok, 0, cs>>>(g); - } else { - for (int t = 0; t < n_tok; ++t) - scalar_gate_kernel<<<1, 256, 0, cs>>>(x_bf16 + (size_t) t * n_embd, gate_inp_bf16, g + t, (int) n_embd); - } - scale_rows_kernel<<>>( - out, g, (int) n_embd); const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) throw std::runtime_error(std::string("shared_expert_multi: ") + cudaGetErrorString(e)); } @@ -295,11 +296,11 @@ void shared_expert(const uint8_t* x_q8_0, const uint8_t* x_q8k, const uint16_t* // round trip. 256 threads is the reduction's width, not the problem's size. if (use_native) { bf16_gemv_fp32_mmvf(x_f32, gate_inp_bf16, g, n_embd, 1, stream); - native_scalar_sigmoid_kernel<<<1, 1, 0, (cudaStream_t) stream>>>(g); + shared_expert_gate_rows(out, g, n_embd, 1, stream); } else { scalar_gate_kernel<<<1, 256, 0, (cudaStream_t) stream>>>(x_bf16, gate_inp_bf16, g, (int) n_embd); + scale_kernel<<>>(out, g, (int) n_embd); } - scale_kernel<<>>(out, g, (int) n_embd); if (stream == nullptr) { const cudaError_t e = cudaDeviceSynchronize(); diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 7948248fa09..4638031bdb9 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -13,11 +13,13 @@ #include "strata/kernels/kv_q8.hpp" #include "strata/kernels/native_moe.hpp" #include "strata/kernels/native_ple_postops.hpp" +#include "strata/kernels/native_qsa.hpp" #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_rope.hpp" #include "strata/kernels/qsa_decode_attn.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" +#include "strata/kernels/shared_expert.hpp" #include "strata/kernels/verify_kernels.hpp" #include "strata/prefill/kernels.hpp" @@ -81,7 +83,8 @@ int bitwise_diff(const std::vector& a, const std::vector& b, const } // ---- the MoE combine: copy all rows + add the hits + one combine per token vs the second GPU's rows taken into VRAM -// (fetch_listed_rows) and native_moe_gather_combine +// (fetch_listed_rows) and native_moe_gather_combine; from rep 2 on the shared rows gated by shared_expert_gate_rows vs +// the gather-combine gating them int test_gather_combine(std::mt19937& rng, cudaStream_t s) { const int64_t N = 2560, K = 10; int bad = 0; @@ -94,6 +97,11 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { for (auto& v : host) v = edgy(rng); for (auto& v : w) v = std::uniform_real_distribution(0.0f, 1.0f)(rng); for (auto& v : shared) v = edgy(rng); + std::vector glog((size_t) n_tok); // the gates' logits: sigmoid near 1, near 0 (denormal products) + for (auto& v : glog) + v = rng() % 4 == 0 ? std::uniform_real_distribution(-100.0f, -80.0f)(rng) + : std::uniform_real_distribution(-8.0f, 8.0f)(rng); + const bool gated = rep >= 2; // the GPU's entries, in a random order and a random share (none, some, all); from rep 4 on the second GPU // takes some of the rest std::vector dst, raw; @@ -120,12 +128,16 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { float *d_hit = dev(hit.size()), *d_parts = dev(hit.size()), *d_w = dev(w.size()), *d_sh = dev(shared.size()), *d_out_old = dev(shared.size()), - *d_out_new = dev(shared.size()); + *d_out_new = dev(shared.size()), *d_shg = dev(shared.size()), + *d_g = dev(glog.size()), *d_glog = dev(glog.size()); int32_t *d_dst = dev(std::max(1, dst.size()) + 64), *d_count = dev(3), *d_list = dev((size_t) (4 + rows)); up(d_hit, hit); up(d_w, w); up(d_sh, shared); + up(d_shg, shared); + up(d_g, glog); + up(d_glog, glog); if (!dst.empty()) up(d_dst, dst); // the new path takes them as two lists (the window's VRAM share and the pool's PCIe share), split at random const int32_t split = rep % 2 == 0 ? count : (int32_t) (rng() % (dst.size() + 1)); @@ -144,9 +156,10 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { std::memcpy(h_map, host_zeroed.data(), host.size() * sizeof(float)); strata::kernels::copy_from_mapped(d_parts, m_map, rows * N, s); strata::kernels::moe_hit_add(d_parts, d_hit, d_dst, d_count, rows, N, s); + if (gated) strata::kernels::shared_expert_gate_rows(d_shg, d_g, N, n_tok, s); for (int t = 0; t < n_tok; ++t) - strata::kernels::native_moe_combine(d_parts + t * K * N, d_w + t * K, d_sh + t * N, d_out_old + t * N, - N, K, s); + strata::kernels::native_moe_combine(d_parts + t * K * N, d_w + t * K, (gated ? d_shg : d_sh) + t * N, + d_out_old + t * N, N, K, s); check(cudaStreamSynchronize(s), "old path"); // new std::memcpy(h_map, host_poison.data(), host.size() * sizeof(float)); @@ -156,7 +169,7 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { strata::kernels::native_moe_gather_combine(d_hit, m_map, d_dst, d_count + 1, rep % 2 == 0 ? nullptr : d_dst + split, d_count + 2, second ? d_list + 4 : nullptr, second ? d_list : nullptr, d_w, - d_sh, d_out_new, N, K, n_tok, s); + d_sh, gated ? d_glog : nullptr, d_out_new, N, K, n_tok, s); check(cudaStreamSynchronize(s), "new path"); int b = bitwise_diff(down(d_out_old, shared.size()), down(d_out_new, shared.size()), "gather_combine"); if (second) { @@ -174,13 +187,15 @@ int test_gather_combine(std::mt19937& rng, cudaStream_t s) { cudaFreeHost(h_map); cudaFreeHost(h_list); for (void* p : {(void*) d_hit, (void*) d_parts, (void*) d_w, (void*) d_sh, (void*) d_out_old, - (void*) d_out_new, (void*) d_dst, (void*) d_count, (void*) d_list}) + (void*) d_out_new, (void*) d_shg, (void*) d_g, (void*) d_glog, (void*) d_dst, + (void*) d_count, (void*) d_list}) cudaFree(p); } } std::printf("gather_combine: %s\n", bad ? "MISMATCH" - : "bitwise equal (n_tok 1-4, 8; 8 plans each, one or two lists, the second GPU's rows fetched)"); + : "bitwise equal (n_tok 1-4, 8; 8 plans each, one or two lists, the second GPU's rows fetched, " + "the shared rows gated)"); return bad; } @@ -257,40 +272,52 @@ int test_hit_plan(std::mt19937& rng, cudaStream_t s) { return bad; } -// ---- BF16 x FP32 MMVF: one call per row vs bf16_gemv_fp32_mmvf_multi (the router, indexer and gate shapes) +// ---- BF16 x FP32 MMVF: one call per row vs bf16_gemv_fp32_mmvf_multi (the router, indexer and gate shapes), and +// with a second matrix of 125 rows in the same launch (the indexer's keys and queries; partial groups of 4 rows) int test_mmvf_multi(std::mt19937& rng, cudaStream_t s) { - int bad = 0; - for (int64_t n_out : {512, 128, 1}) { - const int64_t n_in = 2560; - std::vector w((size_t) (n_in * n_out)); + auto bf16_matrix = [&](int64_t n) { + std::vector w((size_t) n); for (auto& v : w) { const float f = edgy(rng); uint32_t b; std::memcpy(&b, &f, 4); v = (uint16_t) (b >> 16); } - uint16_t* d_w = dev(w.size()); - up(d_w, w); + uint16_t* d = dev(w.size()); + up(d, w); + return d; + }; + int bad = 0; + const int64_t n_in = 2560, n_out2 = 125; + uint16_t* d_w2 = bf16_matrix(n_in * n_out2); + for (int64_t n_out : {512, 130, 1}) { + uint16_t* d_w = bf16_matrix(n_in * n_out); for (int n_tok = 1; n_tok <= 8; ++n_tok) { std::vector x((size_t) (n_tok * n_in)); for (auto& v : x) v = edgy(rng); - float *d_x = dev(x.size()), *d_a = dev((size_t) (n_tok * n_out)), - *d_b = dev((size_t) (n_tok * n_out)); + const size_t ny = (size_t) (n_tok * n_out), ny2 = (size_t) (n_tok * n_out2); + float *d_x = dev(x.size()), *d_a = dev(ny), *d_b = dev(ny), *d_c = dev(ny), + *d_a2 = dev(ny2), *d_c2 = dev(ny2); up(d_x, x); - for (int t = 0; t < n_tok; ++t) + for (int t = 0; t < n_tok; ++t) { strata::kernels::bf16_gemv_fp32_mmvf(d_x + t * n_in, d_w, d_a + t * n_out, n_in, n_out, s); + strata::kernels::bf16_gemv_fp32_mmvf(d_x + t * n_in, d_w2, d_a2 + t * n_out2, n_in, n_out2, s); + } strata::kernels::bf16_gemv_fp32_mmvf_multi(d_x, d_w, d_b, n_in, n_out, n_tok, s); + strata::kernels::bf16_gemv_fp32_mmvf_multi(d_x, d_w, d_c, n_in, n_out, n_tok, s, d_w2, d_c2, n_out2); check(cudaStreamSynchronize(s), "mmvf"); - const int b = bitwise_diff(down(d_a, (size_t) (n_tok * n_out)), down(d_b, (size_t) (n_tok * n_out)), "mmvf_multi"); + const int b = bitwise_diff(down(d_a, ny), down(d_b, ny), "mmvf_multi") + + bitwise_diff(down(d_a, ny), down(d_c, ny), "mmvf_multi, two matrices") + + bitwise_diff(down(d_a2, ny2), down(d_c2, ny2), "mmvf_multi, the second matrix"); if (b) std::fprintf(stderr, "mmvf_multi: n_out %lld n_tok %d: %d differ\n", (long long) n_out, n_tok, b); bad += b; - cudaFree(d_x); - cudaFree(d_a); - cudaFree(d_b); + for (float* p : {d_x, d_a, d_b, d_c, d_a2, d_c2}) cudaFree(p); } cudaFree(d_w); } - std::printf("mmvf_multi: %s\n", bad ? "MISMATCH" : "bitwise equal (n_out 512, 128, 1; 1-8 rows)"); + cudaFree(d_w2); + std::printf("mmvf_multi: %s\n", + bad ? "MISMATCH" : "bitwise equal (n_out 512, 130, 1; 1-8 rows; with a second matrix)"); return bad; } @@ -765,6 +792,60 @@ int test_rope_tokens(std::mt19937& rng, cudaStream_t s) { return bad; } +// ---- the QSA heads' norm and RoPE: native_qsa_rms_norm_weighted + native_rope_apply_tokens vs +// native_qsa_norm_rope_tokens, in place and from rows of twice the width (the queries' halves) +int test_norm_rope(std::mt19937& rng, cudaStream_t s) { + const int NH = 24; + int bad = 0; + for (int heads : {2, 4, 24}) { + for (int head_dim : {128, 256}) { + std::vector gamma((size_t) head_dim); + for (auto& g : gamma) g = 0.5f + std::uniform_real_distribution(0.0f, 1.0f)(rng); + float* d_gamma = dev(gamma.size()); + up(d_gamma, gamma); + for (int n_tok = 1; n_tok <= 8; ++n_tok) { + const int rows = n_tok * heads; + std::vector x((size_t) rows * 2 * head_dim); // rows of 2 * head_dim: the first half is read + for (auto& v : x) v = std::normal_distribution(0.0f, 3.0f)(rng); + std::vector half((size_t) rows * head_dim); + for (int r = 0; r < rows; ++r) + std::copy(x.begin() + (size_t) r * 2 * head_dim, x.begin() + (size_t) r * 2 * head_dim + head_dim, + half.begin() + (size_t) r * head_dim); + std::vector pos((size_t) n_tok * NH); + const int p0 = (int) (rng() % 200000); + for (int t = 0; t < n_tok; ++t) + for (int h = 0; h < NH; ++h) pos[(size_t) t * NH + h] = p0 + t; + float *d_a = dev(half.size()), *d_b = dev(half.size()), *d_x = dev(x.size()), + *d_c = dev(half.size()); + int32_t* d_pos = dev(pos.size()); + up(d_a, half); + up(d_b, half); + up(d_x, x); + up(d_pos, pos); + strata::kernels::native_qsa_rms_norm_weighted(d_a, d_gamma, d_a, head_dim, rows, 1e-6f, s); + strata::kernels::native_rope_apply_tokens(d_a, d_a, rows, head_dim, 64, 5000000.0f, d_pos, heads, NH, + s); + strata::kernels::native_qsa_norm_rope_tokens(d_b, head_dim, d_gamma, d_b, head_dim, rows, 1e-6f, 64, + 5000000.0f, d_pos, heads, NH, s); + strata::kernels::native_qsa_norm_rope_tokens(d_x, 2 * head_dim, d_gamma, d_c, head_dim, rows, 1e-6f, 64, + 5000000.0f, d_pos, heads, NH, s); + check(cudaStreamSynchronize(s), "norm_rope"); + const int b = bitwise_diff(down(d_a, half.size()), down(d_b, half.size()), "norm_rope") + + bitwise_diff(down(d_a, half.size()), down(d_c, half.size()), "norm_rope, strided"); + if (b) + std::fprintf(stderr, "norm_rope: %d heads of %d, n_tok %d: %d differ\n", heads, head_dim, n_tok, b); + bad += b; + for (float* p : {d_a, d_b, d_x, d_c}) cudaFree(p); + cudaFree(d_pos); + } + cudaFree(d_gamma); + } + } + std::printf("norm_rope_tokens: %s\n", bad ? "MISMATCH" : "bitwise equal (2, 4, 24 heads of 128, 256; 1-8 tokens; " + "in place and strided)"); + return bad; +} + // ---- the int8 KV append: kv_append_q8_step per token vs kv_append_q8_steps int test_kv_append(std::mt19937& rng, cudaStream_t s) { const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); @@ -867,9 +948,15 @@ int test_indexer_append(std::mt19937& rng, cudaStream_t s) { for (int t = 0; t < n_tok; ++t) strata::kernels::native_qsa_indexer_append(d_raw + (size_t) t * D, d_pos + t, 0, d_gamma, 1e-6f, ib0, sh, max_cells, 5000000.0f, s); + float* d_snap = dev(tail.size()); strata::kernels::native_qsa_indexer_append_multi(d_raw, d_pos, 1, n_tok, 0, d_gamma, 1e-6f, ib1, sh, - max_cells, 5000000.0f, s); + max_cells, 5000000.0f, s, d_snap); check(cudaStreamSynchronize(s), "indexer"); + if (down(d_snap, tail.size()) != tail) { + std::fprintf(stderr, "indexer: n_tok %d start %d: the tail snapshot differs\n", n_tok, start); + ++bad; + } + cudaFree(d_snap); const size_t sizes[3] = {tail.size(), dead.size(), pooled.size()}; const char* names[3] = {"tail", "dead", "pooled"}; for (int o = 0; o < 3; ++o) { @@ -891,7 +978,8 @@ int test_indexer_append(std::mt19937& rng, cudaStream_t s) { } } cudaFree(d_gamma); - std::printf("indexer_append_multi: %s\n", bad ? "MISMATCH" : "bitwise equal (1-8 tokens, 8 starts, rejected cells)"); + std::printf("indexer_append_multi: %s\n", + bad ? "MISMATCH" : "bitwise equal (1-8 tokens, 8 starts, rejected cells), the tail snapshot the tail"); return bad; } @@ -923,8 +1011,9 @@ int test_ple_tokens(std::mt19937& rng, cudaStream_t s) { for (auto& x : hist) x = nd(rng); float *d_key = dev(key.size()), *d_hidden = dev(hidden.size()), *d_value = dev(value.size()); up(d_key, key); up(d_hidden, hidden); up(d_value, value); - // per token: the decode path's calls - float *h0 = dev(hist.size()), *r0 = dev(hidden.size()); + // per token: the decode path's calls, the history after each token kept + float *h0 = dev(hist.size()), *r0 = dev(hidden.size()), + *s0 = dev((size_t) n_tok * hist.size()); float *k1 = dev(D), *q1 = dev(D), *g1 = dev(H), *gd1 = dev(D), *c1 = dev(D); up(h0, hist); for (int t = 0; t < n_tok; ++t) { @@ -932,22 +1021,30 @@ int test_ple_tokens(std::mt19937& rng, cudaStream_t s) { strata::kernels::native_ple_postops(d_key + (size_t) t * D, d_hidden + (size_t) t * D, d_value + (size_t) t * N, h0, w, b, s); strata::kernels::ple_history_advance(h0, q1, s); + check(cudaMemcpyAsync(s0 + (size_t) t * hist.size(), h0, hist.size() * 4, cudaMemcpyDeviceToDevice, s), + "ple snapshot"); } // the chunk at once float *h2 = dev(hist.size()), *r2 = dev(hidden.size()), *k2 = dev(key.size()), - *q2 = dev(key.size()), *g2 = dev((size_t) n_tok * H), *gd2 = dev(key.size()); + *q2 = dev(key.size()), *g2 = dev((size_t) n_tok * H), *gd2 = dev(key.size()), + *s2 = dev((size_t) n_tok * hist.size()); up(h2, hist); - strata::kernels::native_ple_postops_tokens(d_key, d_hidden, d_value, h2, w, {k2, q2, g2, gd2, q2, r2}, n_tok, s); + strata::kernels::native_ple_postops_tokens(d_key, d_hidden, d_value, h2, w, {k2, q2, g2, gd2, q2, r2}, n_tok, s, + s2); check(cudaStreamSynchronize(s), "ple tokens"); const int br = bitwise_diff(down(r0, hidden.size()), down(r2, hidden.size()), "result"); const int bh = bitwise_diff(down(h0, hist.size()), down(h2, hist.size()), "history"); - if (br || bh) std::fprintf(stderr, "ple tokens: n_tok %d: result or history differ\n", n_tok); - bad += br + bh; - for (float* p : {d_key, d_hidden, d_value, h0, r0, k1, q1, g1, gd1, c1, h2, r2, k2, q2, g2, gd2}) cudaFree(p); + const int bs = bitwise_diff(down(s0, (size_t) n_tok * hist.size()), down(s2, (size_t) n_tok * hist.size()), + "histories"); + if (br || bh || bs) std::fprintf(stderr, "ple tokens: n_tok %d: results or histories differ\n", n_tok); + bad += br + bh + bs; + for (float* p : {d_key, d_hidden, d_value, h0, r0, s0, k1, q1, g1, gd1, c1, h2, r2, k2, q2, g2, gd2, s2}) + cudaFree(p); } for (float* p : {d_nk, d_nq, d_nc}) cudaFree(p); cudaFree(d_taps); - std::printf("ple_postops_tokens: %s\n", bad ? "MISMATCH" : "bitwise equal (1-23 tokens, results and history)"); + std::printf("ple_postops_tokens: %s\n", + bad ? "MISMATCH" : "bitwise equal (1-23 tokens, results, history, the history after each token)"); return bad; } @@ -1130,6 +1227,7 @@ int main(int argc, char** argv) { bad += test_router_rows(rng, s); bad += test_gr_read(rng, s); bad += test_rope_tokens(rng, s); + bad += test_norm_rope(rng, s); bad += test_kv_append(rng, s); bad += test_indexer_append(rng, s); bad += test_ple_tokens(rng, s); From b67a2913f3d41f0c8ec2a647295e8f9ea35d0d72 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 20:09:07 +0200 Subject: [PATCH 71/92] Verify window: the projections' rows a warp by format and tokens, the GDN qkv grid first, IQ3_S decoded once; bitwise - native_mmvq_il picks the rows a warp (1, 2 or 4) from a table by format, token count (1-4) and row class, measured alone at the engine's clocks (scratch/mm1_bench.cu) and in the engine, where the kernels run beside other branches. It took 2 rows, 4 for Q4_K and Q6_K from 8192 rows: at 3-4 tokens those 4-row instances need 115-193 registers, and Q4_K's qkv took 34-39 us where 2 rows take 25-30; Q6_K runs fastest at 2-3 tokens with 1 row (its out projection at 3 tokens 34.6 -> 20.8 us); UD's Q8_0 head at 2 tokens with 1 row (837 -> 773 us). - One column also goes through native_mmvq_il (its kernels read the q8_1 blocks, Q81One): IQ3_S takes them at every token count, and the shared expert's IQ3_S gate and up projections at 2+ tokens decode their weights once for all tokens (shared_expert_multi quantizes with the interleaved copy, the verify window's xil_): 11.1 / 15.1 / 17.9 -> 9.0 / 10.5 / 8.2 us at 2 / 3 / 4 tokens. IQ3SCols loads a block's 8 activation positions before its sums: its 3-column instance, allotted 40 registers, loaded each after the last one's use. - The GDN mixer records qkv's grid before the gate projection's: the conv runs beside the gate's last blocks. - The interleaved kernels keep a grid-stride loop though the grid gives each warp one group: without it ptxas gives several instances fewer registers, and Q6_K's at 3 tokens run 10-50% slower. - mmvq_il_parity: 1-8 columns with each rows a warp; --bench at the engine's clocks (a one-thread spin before each launch: a continuous load reaches the 3090's 260 W cap). The mixers (IQ3_XXS thinking, us at 1 / 2 / 3 / 4 tokens): GDN 64.0 / 78.0 / 90.2 / 87.3 -> 62.7 / 71.9 / 76.9 / 84.1, QSA 105.4 / 118.9 / 144.3 / 141.3 -> 106.0 / 116.2 / 133.4 / 143.1; the shared expert ends last before the combine in 1.3 / 0.1 / 0.8 / 1.1% of the layers (1.3 / 0.4 / 16.5 / 14.6). Window hashes identical (thinking, code and after 16K; both models; --spec-split and --second-gpu -1). Fixed text, both GPUs, against aa209f7, ms per round (6 runs of each build, half of them first in their pair: a run's place moved code-text runs by 3-6%): IQ3_XXS code -0.9%, thinking -1.0%, after 16K +0.3%; UD-Q4_K_XL code -0.9%, thinking -1.0%, after 16K -1.3%. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/kernels/iq_kernels.hpp | 6 +- include/strata/kernels/native_mmvq.hpp | 9 +- include/strata/kernels/shared_expert.hpp | 4 +- src/core/verify.cpp | 9 +- src/kernels/cuda/iq_dot.cuh | 9 +- src/kernels/cuda/iq_kernels.cu | 87 ++++++++----- src/kernels/cuda/native_mmvq.cu | 158 ++++++++++++++--------- src/kernels/cuda/q8_1_il.cuh | 21 +++ src/kernels/cuda/shared_expert.cu | 13 +- src/kernels/mmvq_il_parity.cu | 79 +++++++----- 10 files changed, 258 insertions(+), 137 deletions(-) diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp index 593cc0f3a11..721790f834f 100644 --- a/include/strata/kernels/iq_kernels.hpp +++ b/include/strata/kernels/iq_kernels.hpp @@ -22,9 +22,11 @@ void quantize_q8_1_rows(const float* x, int64_t n_rows, int64_t n_cols, void* y, /// y[c][r] = W[r] . x[c] for `ncols` columns of q8_1 activations (x stride n_in/32 blocks per column). void iq_mmvq(int ggml_type, const void* w, const void* x_q8_1, float* y, int n_in, int n_out, int ncols, void* stream); -/// iq_mmvq for 2..8 columns from their interleaved copy (native_quantize_q8_1_il), bitwise: IQ3_S. +/// iq_mmvq for 2..8 columns from their interleaved copy (native_quantize_q8_1_il), or one from its q8_1 blocks, +/// bitwise, a warp taking `rows` rows (1, 2 or 4): IQ3_S. bool iq_mmvq_il_supported(int ggml_type) noexcept; -void iq_mmvq_il(int ggml_type, const void* w, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream); +void iq_mmvq_il(int ggml_type, const void* w, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, + int ncols, void* stream, int rows); /// Dequantize `n` contiguous values (n a multiple of 256) to fp16 / fp32. void iq_dequant_f16(int ggml_type, const void* src, int64_t n, uint16_t* dst, void* stream); diff --git a/include/strata/kernels/native_mmvq.hpp b/include/strata/kernels/native_mmvq.hpp index 2b8c14e8e8f..959e1d05672 100644 --- a/include/strata/kernels/native_mmvq.hpp +++ b/include/strata/kernels/native_mmvq.hpp @@ -116,10 +116,13 @@ constexpr int native_q8_1_il_cp(int ncols) { return ncols <= 2 ? 2 : ncols <= 4 std::size_t native_q8_1_il_bytes(int n_in, int ncols); // native_quantize_q8_1 that also writes the interleaved copy to x_il (native_q8_1_il_bytes); 2 <= ncols <= 8. void native_quantize_q8_1_il(const float* x, void* x_q8_1, void* x_il, int n_in, int ncols, void* stream); -// native_mmvq for 2..8 columns quantized by native_quantize_q8_1_il, bitwise native_mmvq's output: kernels that read -// the interleaved copy for IQ4_XS, Q4_K, Q5_K, Q6_K, Q8_0 and IQ3_S matrices of at least 2048 rows, native_mmvq -// otherwise. +// native_mmvq for 1 column (x_il unused) or 2..8 quantized by native_quantize_q8_1_il, bitwise native_mmvq's output: +// kernels in which a warp takes 1, 2 or 4 rows, reading 2+ columns from the interleaved copy, where they are the faster +// on the RTX 3090 (a table by format, columns and rows: IQ4_XS, Q4_K, Q5_K, Q6_K and Q8_0 matrices of 2+ columns and +// 2048+ rows, IQ3_S ones); native_mmvq otherwise. void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream); +// Tests and benchmarks: every native_mmvq_il call takes `rows` a warp (0: the table), and with `one` 1 column too. +void native_mmvq_il_tune(int rows, bool one); } // namespace strata::kernels diff --git a/include/strata/kernels/shared_expert.hpp b/include/strata/kernels/shared_expert.hpp index b69f219c62c..e0c990b8d51 100644 --- a/include/strata/kernels/shared_expert.hpp +++ b/include/strata/kernels/shared_expert.hpp @@ -38,6 +38,7 @@ struct NativeSharedWeights { const void* up_data = nullptr; const void* down_data = nullptr; void* q8_1 = nullptr; + void* q8_1_il = nullptr; // shared_expert_multi: the interleaved copy of 2+ tokens' inputs (native_q8_1_il_bytes) }; /// Bytes of caller-owned scratch `shared_expert` needs. **THE KERNEL USED TO `cudaMalloc` FOUR BUFFERS ON @@ -78,7 +79,8 @@ void shared_expert_gate_rows(float* out, float* g, int64_t n_embd, int n_tok, vo /// scratch; `out` (n_tok, n_embd) receives the down projection's rows and `g` the scalar gate's n_tok logits, /// computed on `gate_stream` (may be `stream`), so that the chain ends with the down projection: /// `shared_expert_gate_rows` on them, or `native_moe_gather_combine` applying them, gives every token bitwise -/// `shared_expert` on that token. `nw.q8_1` must hold n_tok columns of n_embd. +/// `shared_expert` on that token. `nw.q8_1` must hold n_tok columns of n_embd; with `nw.q8_1_il` 2+ tokens' gate and +/// up projections run through `native_mmvq_il` (IQ3_S's weights decoded once for all tokens). void shared_expert_multi(int n_tok, const float* x, const NativeSharedWeights& nw, const uint16_t* gate_inp_bf16, float* gate, float* up, float* g, float* out, int64_t n_embd, int64_t n_ff, void* stream, void* gate_stream); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 75cc4924f38..98899da3b86 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -479,6 +479,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { nsw.up_type = wsu->native_type; nsw.up_data = wsu->native_data; nsw.down_type = wsd->native_type; nsw.down_data = wsd->native_data; nsw.q8_1 = xq_; // the main stream's next use is the next layer's mixer + nsw.q8_1_il = xil_; cudaStream_t ws = cs, gs = cs; if (beside) { if (cudaStreamWaitEvent(shs_, efork_, 0) != cudaSuccess || @@ -528,8 +529,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { else native_quantize_q8_1(x, xq_, (int) n_in, n, cs); }; auto mm = [&](int type, const void* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { - if (n >= 2) native_mmvq_il(type, w, xq_, xil_, y, (int) n_in, (int) n_out, n, ps); - else native_mmvq(type, w, xq_, y, (int) n_in, (int) n_out, n, ps); + native_mmvq_il(type, w, xq_, xil_, y, (int) n_in, (int) n_out, n, ps); }; auto proj = [&](const WeightRef* w, float* y, int64_t n_in, int64_t n_out, cudaStream_t ps) { mm(w->native_type, w->native_data, y, n_in, n_out, ps); @@ -602,7 +602,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { float* gate = gate_L_ + (size_t) gi * MT * HV; float* beta = beta_L_ + (size_t) gi * MT * HV; // beside qkv and the conv: alpha/beta (a few blocks, latency-bound) and the gate projection, joined - // before the recurrence + // before the recurrence. qkv's grid is recorded first, so it is dispatched first and the conv runs + // beside the gate projection's last blocks. if (!fork(side_, bfork_)) return false; gdn_ab_multi(xm, (const uint16_t*) wa->data, (const uint16_t*) wb->data, (const float*) wdt->data, (const float*) wsa->data, gate + (size_t) tb * HV, beta + (size_t) tb * HV, (int) N, (int) HV, @@ -610,9 +611,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (!mark(side_, bjoin_)) return false; quant(xm, N); if (!fork(shs_, pfork_)) return false; + proj(wqkv, qkv + (size_t) tb * C, N, C, cs); proj(wg, z_ + (size_t) tb * ZV, N, ZV, shs_); if (!mark(shs_, pjoin_)) return false; - proj(wqkv, qkv + (size_t) tb * C, N, C, cs); gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb); if (!wait(bjoin_) || !wait(pjoin_)) return false; // the recurrence from the untouched state over tokens [0, te); outputs only for this group's diff --git a/src/kernels/cuda/iq_dot.cuh b/src/kernels/cuda/iq_dot.cuh index e53da235423..c8fdb6879c4 100644 --- a/src/kernels/cuda/iq_dot.cuh +++ b/src/kernels/cuda/iq_dot.cuh @@ -245,17 +245,18 @@ struct IQ3SCols { } template __device__ __forceinline__ void apply(const X& x, const int& b, float (&out)[NC]) const { + int u[8][NC]; // every position's activations first, so that their loads overlap +#pragma unroll + for (int l = 0; l < 8; ++l) x.u(b, l, u[l]); int sp[NC], sn[NC]; #pragma unroll for (int c = 0; c < NC; ++c) sp[c] = sn[c] = 0; #pragma unroll for (int l = 0; l < 8; ++l) { - int u[NC]; - x.u(b, l, u); #pragma unroll for (int c = 0; c < NC; ++c) { - sp[c] = ggml_cuda_dp4a((int) g[l], u[c], sp[c]); - sn[c] = ggml_cuda_dp4a((int) gm[l], u[c], sn[c]); + sp[c] = ggml_cuda_dp4a((int) g[l], u[l][c], sp[c]); + sn[c] = ggml_cuda_dp4a((int) gm[l], u[l][c], sn[c]); } } float d8[NC]; diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu index b551926bcdb..8b4621580c8 100644 --- a/src/kernels/cuda/iq_kernels.cu +++ b/src/kernels/cuda/iq_kernels.cu @@ -33,43 +33,47 @@ __global__ void __launch_bounds__(128) mmvq_kernel(const uint8_t* __restrict__ w } } -// mmvq_kernel for NC columns from their interleaved (position-major) copy: a warp takes R rows, decodes each call's -// weights once and keeps row_dot's lane-strided sum per (row, column), so every value is bitwise mmvq_kernel's. +// mmvq_kernel for NC columns from their interleaved (position-major) copy, or one from its q8_1 blocks: a warp takes R +// rows, decodes each call's weights once and keeps row_dot's lane-strided sum per (row, column), so every value is +// bitwise mmvq_kernel's. template __global__ void __launch_bounds__(128) mmvq_il_iq3_s_kernel(const uint8_t* __restrict__ w, size_t row_bytes, - const int* __restrict__ xq, const float* __restrict__ xd, + const void* __restrict__ xq, const float* __restrict__ xd, float* __restrict__ y, int n_in, int n_out) { using F = Fmt<21>; const int lane = threadIdx.x & 31; - const int row0 = (blockIdx.x * 4 + (threadIdx.x >> 5)) * R; - if (row0 >= n_out) return; - const Q81Il x{xq, xd, n_in / 32}; + const auto x = q8_1_cols(xq, xd, n_in / 32); const int nb = n_in / F::qk; - float acc[R][NC]; + // a grid-stride loop, though the grid gives each warp one group: native_mmvq_il_kernel's form, which its + // rows-a-warp table is measured on + for (int grp = blockIdx.x * 4 + (threadIdx.x >> 5); grp * R < n_out; grp += gridDim.x * 4) { + const int row0 = grp * R; + float acc[R][NC]; #pragma unroll - for (int i = 0; i < R; ++i) + for (int i = 0; i < R; ++i) #pragma unroll - for (int c = 0; c < NC; ++c) acc[i][c] = 0.0f; - for (int k = lane; k < nb * F::ipb; k += 32) { - const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); + for (int c = 0; c < NC; ++c) acc[i][c] = 0.0f; + for (int k = lane; k < nb * F::ipb; k += 32) { + const int kbx = k / F::ipb, iqs = F::step * (k % F::ipb); #pragma unroll - for (int i = 0; i < R; ++i) { - const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row - IQ3SCols wv; - wv.load(w + (size_t) row * row_bytes, kbx, iqs); - float o[NC]; - wv.apply(x, kbx * (F::qk / 32) + iqs / 2, o); + for (int i = 0; i < R; ++i) { + const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row + IQ3SCols wv; + wv.load(w + (size_t) row * row_bytes, kbx, iqs); + float o[NC]; + wv.apply(x, kbx * (F::qk / 32) + iqs / 2, o); #pragma unroll - for (int c = 0; c < NC; ++c) acc[i][c] += o[c]; + for (int c = 0; c < NC; ++c) acc[i][c] += o[c]; + } } - } #pragma unroll - for (int i = 0; i < R; ++i) + for (int i = 0; i < R; ++i) #pragma unroll - for (int c = 0; c < NC; ++c) { - const float s = warp_sum(acc[i][c]); - if (lane == 0 && row0 + i < n_out) y[(size_t) c * n_out + row0 + i] = s; - } + for (int c = 0; c < NC; ++c) { + const float s = warp_sum(acc[i][c]); + if (lane == 0 && row0 + i < n_out) y[(size_t) c * n_out + row0 + i] = s; + } + } } // ---------------------------------------------------------------- grouped native experts @@ -590,21 +594,38 @@ void iq_mmvq(int t, const void* w, const void* x_q8_1, float* y, int n_in, int n bool iq_mmvq_il_supported(int t) noexcept { return t == 21; } -void iq_mmvq_il(int t, const void* w, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream) { - if (t != 21 || ncols < 2 || ncols > 8 || n_in % 256 != 0) { +namespace { +template +void launch_iq3_s_il(const uint8_t* w, size_t rb, const void* x_q8_1, const void* x_il, float* y, int n_in, + int n_out, int ncols, cudaStream_t s) { + const unsigned blocks = (unsigned) (((n_out + R - 1) / R + 3) / 4); + if (ncols == 1) { + mmvq_il_iq3_s_kernel<1, R><<>>(w, rb, x_q8_1, nullptr, y, n_in, n_out); + return; + } + const Q81IlParts parts = q8_1_il_parts(x_il, n_in, ncols); + switch (ncols) { +#define STRATA_IL(N) \ + case N: mmvq_il_iq3_s_kernel<<>>(w, rb, parts.pm, parts.d, y, n_in, n_out); break; + STRATA_IL(2) STRATA_IL(3) STRATA_IL(4) STRATA_IL(5) STRATA_IL(6) STRATA_IL(7) STRATA_IL(8) +#undef STRATA_IL + } +} +} // namespace + +void iq_mmvq_il(int t, const void* w, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, int ncols, + void* stream, int rows) { + if (t != 21 || ncols < 1 || ncols > 8 || n_in % 256 != 0) { std::fprintf(stderr, "iq_mmvq_il: type %d, %d columns are not supported\n", t, ncols); std::exit(1); } - constexpr int R = 2; - const Q81IlParts parts = q8_1_il_parts(x_il, n_in, ncols); const size_t rb = iq_row_bytes(t, n_in); const auto* W = (const uint8_t*) w; - const unsigned blocks = (unsigned) (((n_out + R - 1) / R + 3) / 4); cudaStream_t s = (cudaStream_t) stream; - switch (ncols) { -#define STRATA_IL(N) case N: mmvq_il_iq3_s_kernel<<>>(W, rb, parts.pm, parts.d, y, n_in, n_out); break; - STRATA_IL(2) STRATA_IL(3) STRATA_IL(4) STRATA_IL(5) STRATA_IL(6) STRATA_IL(7) STRATA_IL(8) -#undef STRATA_IL + switch (rows) { + case 1: launch_iq3_s_il<1>(W, rb, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 4: launch_iq3_s_il<4>(W, rb, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + default: launch_iq3_s_il<2>(W, rb, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; } check("iq_mmvq_il"); } diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu index 7495662564d..ae37c545ceb 100644 --- a/src/kernels/cuda/native_mmvq.cu +++ b/src/kernels/cuda/native_mmvq.cu @@ -30,6 +30,7 @@ #include #include +#include #include #include #include @@ -1091,8 +1092,8 @@ void launch_multi(const void* weights, const void* x_q8_1, float* y, int n_in, i struct IlIQ4XS { using F = IQ4XSTraits; static constexpr bool PM = true; - template - __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + template + __device__ static void apply(const F::W& r, const X& x, int kby, int iqs, float (&out)[NC]) { const int b = kby + iqs / 4; int sumi[NC]; #pragma unroll @@ -1121,8 +1122,8 @@ struct IlIQ4XS { struct IlQ4K { using F = Q4KTraits; static constexpr bool PM = false; - template - __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + template + __device__ static void apply(const F::W& r, const X& x, int kby, int iqs, float (&out)[NC]) { int u[4][NC]; float d8[2][NC]; #pragma unroll @@ -1155,8 +1156,8 @@ struct IlQ4K { struct IlQ5K { using F = Q5KTraits; static constexpr bool PM = false; - template - __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + template + __device__ static void apply(const F::W& r, const X& x, int kby, int iqs, float (&out)[NC]) { int u[4][NC]; float d8[2][NC]; #pragma unroll @@ -1193,8 +1194,8 @@ struct IlQ5K { struct IlQ6K { using F = Q6KTraits; static constexpr bool PM = false; - template - __device__ static void apply(const F::W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + template + __device__ static void apply(const F::W& r, const X& x, int kby, int iqs, float (&out)[NC]) { int u[2][NC]; float d8[2][NC]; #pragma unroll @@ -1231,8 +1232,8 @@ struct IlQ80 { r.d0 = w->d; return r; } - template - __device__ static void apply(const W& r, const Q81Il& x, int kby, int iqs, float (&out)[NC]) { + template + __device__ static void apply(const W& r, const X& x, int kby, int iqs, float (&out)[NC]) { int u0[NC], u1[NC]; x.u(kby, iqs, u0); x.u(kby, iqs + 1, u1); @@ -1260,67 +1261,85 @@ template <> struct IlLoad { template __launch_bounds__(4 * WARP) -__global__ void native_mmvq_il_kernel(const typename I::F::Block* __restrict__ w, const int* __restrict__ xq, +__global__ void native_mmvq_il_kernel(const typename I::F::Block* __restrict__ w, const void* __restrict__ xq, const float* __restrict__ xd, float* __restrict__ y, int n_in, int n_out) { using F = typename I::F; constexpr int SUB = WARP / F::T; // weight blocks a chunk covers const int lane = int(threadIdx.x) & (WARP - 1); - const int row0 = (int(blockIdx.x) * 4 + (int(threadIdx.x) >> 5)) * R; - if (row0 >= n_out) return; - const Q81Il x{xq, xd, n_in / Q8K}; + const auto x = q8_1_cols(xq, xd, n_in / Q8K); const int bpr = n_in / F::DIV; const int kqs = F::kqs(lane); const int nm = (bpr + SUB - 1) / SUB; - float acc[4][R][NC]; + // A grid-stride loop over the row groups, though the grid gives each warp one: without it ptxas gives several + // instances fewer registers, and Q6_K's at 3 columns run 10-50% slower (the rows-a-warp table is measured on it). + for (int grp = int(blockIdx.x) * 4 + (int(threadIdx.x) >> 5); grp * R < n_out; grp += int(gridDim.x) * 4) { + const int row0 = grp * R; + float acc[4][R][NC]; #pragma unroll - for (int v = 0; v < 4; ++v) + for (int v = 0; v < 4; ++v) #pragma unroll - for (int i = 0; i < R; ++i) + for (int i = 0; i < R; ++i) #pragma unroll - for (int c = 0; c < NC; ++c) acc[v][i][c] = 0.0f; - for (int m0 = 0; m0 < nm; m0 += 4) { + for (int c = 0; c < NC; ++c) acc[v][i][c] = 0.0f; + for (int m0 = 0; m0 < nm; m0 += 4) { #pragma unroll - for (int v = 0; v < 4; ++v) { - const int kbx = SUB * (m0 + v) + lane / F::T; - if (kbx < bpr) { + for (int v = 0; v < 4; ++v) { + const int kbx = SUB * (m0 + v) + lane / F::T; + if (kbx < bpr) { #pragma unroll - for (int i = 0; i < R; ++i) { - const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row - const typename IlLoad::W wv = IlLoad::load(w + std::size_t(row) * bpr + kbx, kqs); - float o[NC]; - I::template apply(wv, x, kbx * F::KBY, kqs, o); + for (int i = 0; i < R; ++i) { + const int row = min(row0 + i, n_out - 1); // a partial last group recomputes its last row + const typename IlLoad::W wv = IlLoad::load(w + std::size_t(row) * bpr + kbx, kqs); + float o[NC]; + I::template apply(wv, x, kbx * F::KBY, kqs, o); #pragma unroll - for (int c = 0; c < NC; ++c) acc[v][i][c] += o[c]; + for (int c = 0; c < NC; ++c) acc[v][i][c] += o[c]; + } } } } - } #pragma unroll - for (int i = 0; i < R; ++i) + for (int i = 0; i < R; ++i) #pragma unroll - for (int c = 0; c < NC; ++c) { - float s = acc[0][i][c]; - s += acc[1][i][c]; - s += acc[2][i][c]; - s += acc[3][i][c]; - s = warp_sum(s); - if (lane == 0 && row0 + i < n_out) y[std::size_t(c) * n_out + row0 + i] = s; - } + for (int c = 0; c < NC; ++c) { + float s = acc[0][i][c]; + s += acc[1][i][c]; + s += acc[2][i][c]; + s += acc[3][i][c]; + s = warp_sum(s); + if (lane == 0 && row0 + i < n_out) y[std::size_t(c) * n_out + row0 + i] = s; + } + } } template -void launch_il(const void* weights, const void* x_il, float* y, int n_in, int n_out, int ncols, cudaStream_t s) { +void launch_il(const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, int ncols, + cudaStream_t s) { + const auto* w = static_cast(weights); + const unsigned blocks = unsigned(((n_out + R - 1) / R + 3) / 4); + if (ncols == 1) { + native_mmvq_il_kernel<<>>(w, x_q8_1, nullptr, y, n_in, n_out); + return; + } const Q81IlParts parts = q8_1_il_parts(x_il, n_in, ncols); const int* xq = I::PM ? parts.pm : parts.bm; const float* xd = parts.d; - const auto* w = static_cast(weights); - const unsigned blocks = unsigned(((n_out + R - 1) / R + 3) / 4); switch (ncols) { #define STRATA_IL_CASE(N) case N: native_mmvq_il_kernel<<>>(w, xq, xd, y, n_in, n_out); break; STRATA_IL_CASE(2) STRATA_IL_CASE(3) STRATA_IL_CASE(4) STRATA_IL_CASE(5) STRATA_IL_CASE(6) STRATA_IL_CASE(7) STRATA_IL_CASE(8) #undef STRATA_IL_CASE - default: throw std::invalid_argument("native_mmvq_il requires 2 <= ncols <= 8"); + default: throw std::invalid_argument("native_mmvq_il requires 1 <= ncols <= 8"); + } +} + +template +void launch_il_rows(int r, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, + int ncols, cudaStream_t s) { + switch (r) { + case 1: launch_il(weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 4: launch_il(weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + default: launch_il(weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; } } @@ -1781,37 +1800,60 @@ void native_quantize_q8_1_il(const float* x, void* x_q8_1, void* x_il, int n_in, launch_check(); } +namespace { +// Rows a warp at 1-4 columns ([ncols - 1]) by the row count, in classes around the model's shapes: below 2048 (the +// shared expert's 640, the keys' 512), 4096 (the output projections' 2560), 8192 (the gate projection's 6144), 12288 +// (qkv's 10240), and more (the queries' 12288, the head's); 0: native_mmvq's kernels, which spread a row over four +// warps (IQ3_S's decode the weights once a column). The fastest in the verify window on the RTX 3090 +// (scratch/mm1_bench.cu alone, CUPTI traces beside the other branches); 5-8 columns take 2 rows from 2048. +struct IlRows { int type; uint8_t r[4][5]; }; +constexpr IlRows kIlRows[] = { + {23, {{0, 0, 0, 0, 0}, {0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}, {0, 2, 2, 1, 2}}}, // IQ4_XS + {12, {{0, 0, 0, 0, 0}, {0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}}}, // Q4_K + {13, {{0, 0, 0, 0, 0}, {0, 2, 1, 1, 2}, {0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}}}, // Q5_K + {14, {{0, 0, 0, 0, 0}, {0, 1, 1, 1, 1}, {0, 1, 1, 1, 1}, {0, 2, 2, 1, 2}}}, // Q6_K + {8, {{0, 0, 0, 0, 0}, {0, 0, 4, 2, 1}, {0, 2, 2, 2, 2}, {0, 2, 2, 2, 2}}}, // Q8_0 + {21, {{1, 1, 1, 2, 2}, {1, 2, 4, 2, 2}, {1, 1, 1, 1, 1}, {1, 2, 2, 1, 1}}}, // IQ3_S +}; +int il_rows(int type, int ncols, int n_out) { + const int cls = n_out < 2048 ? 0 : n_out < 4096 ? 1 : n_out < 8192 ? 2 : n_out < 12288 ? 3 : 4; + for (const IlRows& e : kIlRows) + if (e.type == type) return ncols <= 4 ? e.r[ncols - 1][cls] : cls ? 2 : 0; + return 0; +} + +int g_tune_rows = 0; // native_mmvq_il_tune (tests, benchmarks): rows a warp for every shape (0: the table), +bool g_tune_one = false; // and 1 column through these kernels too +} // namespace +void native_mmvq_il_tune(int rows, bool one) { + g_tune_rows = rows; + g_tune_one = one; +} + void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in, int n_out, int ncols, void* stream) { - // below ~2048 rows the warps are too few for one row each: the multi-column kernel spreads a row over four (and - // for Q8_0 2 columns of fewer than 4096 rows) const bool il = ggml_type == 23 || ggml_type == 12 || ggml_type == 13 || ggml_type == 14 || ggml_type == 8; - if (ncols < 2 || n_out < 2048 || (ggml_type == 8 && ncols == 2 && n_out < 4096) || - (!il && !iq_mmvq_il_supported(ggml_type))) { + const int r = g_tune_rows && (ncols > 1 || g_tune_one) ? g_tune_rows : il_rows(ggml_type, ncols, n_out); + if (r == 0 || (!il && !iq_mmvq_il_supported(ggml_type))) { native_mmvq(ggml_type, weights, x_q8_1, y, n_in, n_out, ncols, stream); return; } if (!il) { - iq_mmvq_il(ggml_type, weights, x_il, y, n_in, n_out, ncols, stream); + iq_mmvq_il(ggml_type, weights, x_q8_1, x_il, y, n_in, n_out, ncols, stream, r); return; } validate_shape(n_in, ncols, ggml_type == 8 ? Q8K : QK); validate_pointer(weights); - validate_pointer(x_il); + validate_pointer(ncols == 1 ? x_q8_1 : x_il); validate_pointer(y); validate_stream(stream); const auto s = static_cast(stream); - // rows a warp: 2, and 4 for Q4_K and Q6_K from 8192 rows (the faster on the model files' shapes, mmvq_il_parity - // --bench) - const bool wide = n_out >= 8192; switch (ggml_type) { - case 23: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; - case 12: wide ? launch_il(weights, x_il, y, n_in, n_out, ncols, s) - : launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; - case 13: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; - case 14: wide ? launch_il(weights, x_il, y, n_in, n_out, ncols, s) - : launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; - case 8: launch_il(weights, x_il, y, n_in, n_out, ncols, s); break; + case 23: launch_il_rows(r, weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 12: launch_il_rows(r, weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 13: launch_il_rows(r, weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 14: launch_il_rows(r, weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; + case 8: launch_il_rows(r, weights, x_q8_1, x_il, y, n_in, n_out, ncols, s); break; } launch_check(); } diff --git a/src/kernels/cuda/q8_1_il.cuh b/src/kernels/cuda/q8_1_il.cuh index 0034893a6b5..49c7a5a1fb3 100644 --- a/src/kernels/cuda/q8_1_il.cuh +++ b/src/kernels/cuda/q8_1_il.cuh @@ -4,9 +4,11 @@ #include "strata/kernels/native_mmvq.hpp" +#include #include #include +#include namespace strata::kernels { namespace { @@ -62,6 +64,25 @@ template struct Q81Il { } }; +// One column's plain q8_1 blocks (36 bytes: the half2 scale and sum, then 32 codes) behind Q81Il's interface +struct Q81One { + const uint8_t* __restrict__ x; + __device__ __forceinline__ void u(int b, int p, int (&o)[1]) const { + o[0] = reinterpret_cast(x + std::size_t(b) * 36 + 4)[p]; + } + __device__ __forceinline__ void scales(int b, float (&o)[1]) const { + o[0] = __low2float(*reinterpret_cast(x + std::size_t(b) * 36)); + } +}; + +// The activations a kernel for NC columns reads: one column's q8_1 blocks, 2+ columns' interleaved copy (nb blocks a +// column) +template +__device__ __forceinline__ auto q8_1_cols(const void* xq, const float* xd, int nb) { + if constexpr (NC == 1) return Q81One{static_cast(xq)}; + else return Q81Il{static_cast(xq), xd, nb}; +} + // The copy's parts inside the buffer native_q8_1_il_bytes sizes: bm, pm (int32) and d (float). struct Q81IlParts { const int* bm; diff --git a/src/kernels/cuda/shared_expert.cu b/src/kernels/cuda/shared_expert.cu index 7e7b874f33c..16e025e0ca3 100644 --- a/src/kernels/cuda/shared_expert.cu +++ b/src/kernels/cuda/shared_expert.cu @@ -176,9 +176,16 @@ void shared_expert_multi(int n_tok, const float* x, const NativeSharedWeights& n "streams"); cudaStream_t cs = (cudaStream_t) stream; bf16_gemv_fp32_mmvf_multi(x, gate_inp_bf16, g, n_embd, 1, n_tok, gate_stream); - native_quantize_q8_1(x, nw.q8_1, (int) n_embd, n_tok, stream); - native_mmvq(nw.gate_type, nw.gate_data, nw.q8_1, gate, (int) n_embd, (int) n_ff, n_tok, stream); - native_mmvq(nw.up_type, nw.up_data, nw.q8_1, up, (int) n_embd, (int) n_ff, n_tok, stream); + // native_mmvq_il's kernels where they are the faster (1 token, or 2+ with the interleaved copy) + const bool il = n_tok == 1 || nw.q8_1_il; + if (n_tok >= 2 && nw.q8_1_il) native_quantize_q8_1_il(x, nw.q8_1, nw.q8_1_il, (int) n_embd, n_tok, stream); + else native_quantize_q8_1(x, nw.q8_1, (int) n_embd, n_tok, stream); + auto proj = [&](int type, const void* w, float* y) { + if (il) native_mmvq_il(type, w, nw.q8_1, nw.q8_1_il, y, (int) n_embd, (int) n_ff, n_tok, stream); + else native_mmvq(type, w, nw.q8_1, y, (int) n_embd, (int) n_ff, n_tok, stream); + }; + proj(nw.gate_type, nw.gate_data, gate); + proj(nw.up_type, nw.up_data, up); const int n = (int) (n_ff * n_tok); native_swiglu_kernel<<<(unsigned) ((n + THREADS - 1) / THREADS), THREADS, 0, cs>>>(gate, up, gate, n); native_quantize_q8_1(gate, nw.q8_1, (int) n_ff, n_tok, stream); diff --git a/src/kernels/mmvq_il_parity.cu b/src/kernels/mmvq_il_parity.cu index a7fc9e04bf3..61d3a531d8f 100644 --- a/src/kernels/mmvq_il_parity.cu +++ b/src/kernels/mmvq_il_parity.cu @@ -1,6 +1,7 @@ -// src/kernels/mmvq_il_parity.cu - native_mmvq_il (2..8 columns from the interleaved q8_1 copy) against native_mmvq, -// bitwise, on synthetic matrices of the model files' dense formats and shapes; the interleaving quantizer's q8_1 blocks -// against native_quantize_q8_1's. --bench times both inside CUDA graphs (weights cycled past the L2). +// src/kernels/mmvq_il_parity.cu - native_mmvq_il (1 column from its q8_1 blocks, 2..8 from the interleaved copy) +// against native_mmvq, bitwise, on synthetic matrices of the model files' dense formats and shapes, with every rows a +// warp; the interleaving quantizer's q8_1 blocks against native_quantize_q8_1's. --bench times both inside CUDA graphs +// (weights cycled past the L2). #include "strata/kernels/native_mmvq.hpp" #include @@ -78,6 +79,7 @@ std::vector fetch(const void* d, size_t n) { return h; } +// every column count 1..8 (1: the one-column kernels), with the table's rows a warp and each of 1, 2, 4 int check_shape(const Fmt& f, const Act& a, int n_out) { const int n_in = a.n_in; void* w = dev_weights(f, n_in, n_out); @@ -85,25 +87,33 @@ int check_shape(const Fmt& f, const Act& a, int n_out) { CK(cudaMalloc(&y0, (size_t) 8 * n_out * 4)); CK(cudaMalloc(&y1, (size_t) 8 * n_out * 4)); int bad = 0; - for (int nc = 2; nc <= 8; ++nc) { + for (int nc = 1; nc <= 8; ++nc) { native_quantize_q8_1(a.x, a.q_ref, n_in, nc, g_s); - native_quantize_q8_1_il(a.x, a.q, a.il, n_in, nc, g_s); + if (nc >= 2) native_quantize_q8_1_il(a.x, a.q, a.il, n_in, nc, g_s); + else native_quantize_q8_1(a.x, a.q, n_in, nc, g_s); CK(cudaMemsetAsync(y0, 0xff, (size_t) 8 * n_out * 4, g_s)); - CK(cudaMemsetAsync(y1, 0xee, (size_t) 8 * n_out * 4, g_s)); native_mmvq(f.type, w, a.q_ref, y0, n_in, n_out, nc, g_s); - native_mmvq_il(f.type, w, a.q, a.il, y1, n_in, n_out, nc, g_s); CK(cudaStreamSynchronize(g_s)); const size_t qb = native_q8_1_bytes(n_in, nc); if (fetch(a.q_ref, qb) != fetch(a.q, qb)) { ++bad; std::printf(" %s %d->%d, %d columns: the q8_1 blocks differ\n", f.name, n_in, n_out, nc); } - const auto r0 = fetch(y0, (size_t) nc * n_out), r1 = fetch(y1, (size_t) nc * n_out); - size_t diff = 0; - for (size_t i = 0; i < r0.size(); ++i) diff += r0[i] != r1[i]; - if (diff) { - ++bad; - std::printf(" %s %d->%d, %d columns: %zu of %zu values differ\n", f.name, n_in, n_out, nc, diff, r0.size()); + const auto r0 = fetch(y0, (size_t) nc * n_out); + for (int rows : {0, 1, 2, 4}) { + CK(cudaMemsetAsync(y1, 0xee, (size_t) 8 * n_out * 4, g_s)); + native_mmvq_il_tune(rows, nc == 1); + native_mmvq_il(f.type, w, a.q, a.il, y1, n_in, n_out, nc, g_s); + native_mmvq_il_tune(0, false); + CK(cudaStreamSynchronize(g_s)); + const auto r1 = fetch(y1, (size_t) nc * n_out); + size_t diff = 0; + for (size_t i = 0; i < r0.size(); ++i) diff += r0[i] != r1[i]; + if (diff) { + ++bad; + std::printf(" %s %d->%d, %d columns, rows %d: %zu of %zu values differ\n", f.name, n_in, n_out, nc, + rows, diff, r0.size()); + } } } cudaFree(w); @@ -112,6 +122,13 @@ int check_shape(const Fmt& f, const Act& a, int n_out) { return bad; } +__global__ void spin_kernel(long long cycles) { + const long long t0 = clock64(); + while (clock64() - t0 < cycles) {} +} + +// The median of 20 launches, each after ~3 ms of a one-thread spin, as the engine's load: the GPU active at little +// power (a continuous load reaches the 3090's 260 W cap and lowers the clocks, an idle GPU lowers them too). double time_calls(const std::function& fn, int reps = 48) { cudaGraph_t g; CK(cudaStreamBeginCapture(g_s, cudaStreamCaptureModeThreadLocal)); @@ -120,25 +137,25 @@ double time_calls(const std::function& fn, int reps = 48) { cudaGraphExec_t ex; CK(cudaGraphInstantiate(&ex, g, 0)); cudaGraphDestroy(g); - const auto t0 = std::chrono::steady_clock::now(); - while (std::chrono::duration(std::chrono::steady_clock::now() - t0).count() < 0.3) { // clocks up - CK(cudaGraphLaunch(ex, g_s)); - CK(cudaStreamSynchronize(g_s)); - } cudaEvent_t a, b; CK(cudaEventCreate(&a)); CK(cudaEventCreate(&b)); - const int iters = 20; - CK(cudaEventRecord(a, g_s)); - for (int i = 0; i < iters; ++i) CK(cudaGraphLaunch(ex, g_s)); - CK(cudaEventRecord(b, g_s)); - CK(cudaEventSynchronize(b)); - float ms = 0; - CK(cudaEventElapsedTime(&ms, a, b)); + std::vector ts; + for (int k = 0; k < 24; ++k) { + spin_kernel<<<1, 1, 0, g_s>>>(5'500'000); + CK(cudaEventRecord(a, g_s)); + CK(cudaGraphLaunch(ex, g_s)); + CK(cudaEventRecord(b, g_s)); + CK(cudaEventSynchronize(b)); + float ms = 0; + CK(cudaEventElapsedTime(&ms, a, b)); + if (k >= 4) ts.push_back(ms * 1000.0 / reps); + } + std::sort(ts.begin(), ts.end()); cudaGraphExecDestroy(ex); cudaEventDestroy(a); cudaEventDestroy(b); - return ms * 1000.0 / iters / reps; + return ts[ts.size() / 2]; } void bench_shape(const Fmt& f, const Act& a, int n_out) { @@ -150,7 +167,7 @@ void bench_shape(const Fmt& f, const Act& a, int n_out) { float* y; CK(cudaMalloc(&y, (size_t) 8 * n_out * 4)); std::printf("%-6s %5d -> %5d (%5.2f MB), us (GB/s):", f.name, n_in, n_out, mb); - for (int nc : {2, 3, 4, 8}) { + for (int nc : {2, 3, 4}) { native_quantize_q8_1(a.x, a.q_ref, n_in, nc, g_s); native_quantize_q8_1_il(a.x, a.q, a.il, n_in, nc, g_s); const double t0 = time_calls([&](int r) { native_mmvq(f.type, w[(size_t) (r % copies)], a.q_ref, y, n_in, n_out, nc, g_s); }); @@ -172,16 +189,19 @@ int main(int argc, char** argv) { Act a2560 = make_act(2560), a6144 = make_act(6144), a640 = make_act(640); if (bench) { for (const Fmt& f : kFmts) { + bench_shape(f, a2560, 12288); bench_shape(f, a2560, 10240); bench_shape(f, a2560, 6144); bench_shape(f, a6144, 2560); + bench_shape(f, a2560, 640); } return 0; } int bad = 0, cases = 0; for (const Fmt& f : kFmts) { const std::pair shapes[] = {{&a2560, 10240}, {&a2560, 6144}, {&a6144, 2560}, {&a2560, 12288}, - {&a2560, 2050}, {&a2560, 4097}, {&a2560, 8192}, {&a2560, 512}}; + {&a2560, 2050}, {&a2560, 4097}, {&a2560, 8192}, {&a2560, 512}, + {&a2560, 640}}; for (const auto& sh : shapes) { bad += check_shape(f, *sh.first, sh.second); ++cases; @@ -191,6 +211,7 @@ int main(int argc, char** argv) { ++cases; } } - std::printf("mmvq_il_parity: %d shape/format cases x 2..8 columns: %s\n", cases, bad ? "FAILED" : "bitwise equal"); + std::printf("mmvq_il_parity: %d shape/format cases x 1..8 columns x rows a warp: %s\n", cases, + bad ? "FAILED" : "bitwise equal"); return bad ? 1 : 0; } From fd23d911b9e175dc1b2ea92ecb6b23a8e773c8a2 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Mon, 28 Sep 2026 23:47:32 +0200 Subject: [PATCH 72/92] QSA decode attention: the prompt path's kernels split among blocks, their partials merged with the output gate The decode attention ran a block per 64 selected cells and KV head, a warp a cell with each head's score a shuffle reduction, then a merge and the output gate as kernels of their own: ~43 us a QSA layer at 1 token whatever the cell count, and the draft layer's attention over its 32K window ~88 us (a 512-chunk merge). Now: - The prompt path's kernels take a split (a SPLIT template flag): a block per (KV head, query, split) walks its share of the query's cells and writes its unnormalized sums with each row's maximum and sum. INT8 pools on the tensor cores as in the prompt path, FP16 pools in FP32. Up to 20 splits a query, all queries' blocks in one wave, whole 64-cell tiles. The prompt path's outputs are bitwise unchanged. - attn_merge_kernel adds the splits with the log-sum-exp rescale, issuing all its loads in one round, and applies the output gate (native_qsa_gate_apply's instructions in PTX, bitwise), so the verify window and the draft layer drop their gate launches and the ungated buffer. - A query's scratch no longer grows with the capacity (the draft layer's 32K window: 12 MiB less VRAM). - verify_parity: the decode attention (6 queries at once and each alone) within 2e-5 of the prompt path's kernels, the gate bitwise native_qsa_gate_apply's. The QSA mixer 106 / 116 / 133 / 143 -> 73 / 83 / 89 / 99 us at 1-4 tokens (IQ3_XXS thinking, CUPTI): the attention 8-9 us, the merge 3-4. Not bitwise: along llama.cpp's texts the KL to llama.cpp is within noise of b67a291 on all six texts (UD after 16K closer), and the two builds are as far apart as one- and four-token windows. Fixed text, both GPUs, 6 runs of each build in both orders, ms per round: IQ3_XXS code -3.4%, thinking -3.8%, after 16K -5.5%; UD-Q4_K_XL -0.9%, -2.7%, -3.8%. After the 131K prompt (IQ3_XXS) 113.9 -> 125.1 t/s. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/mtp.hpp | 2 +- include/strata/core/verify.hpp | 2 +- include/strata/kernels/qsa_decode_attn.hpp | 38 +- src/core/mtp.cpp | 6 +- src/core/verify.cpp | 7 +- src/kernels/cuda/qsa_decode_attn.cu | 452 ++++++++++----------- src/kernels/verify_parity.cpp | 79 ++-- 7 files changed, 297 insertions(+), 289 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index b60716bbb1b..2c45a27d654 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -138,7 +138,7 @@ class MtpDrafter { float *mixed_ = nullptr, *inj_ = nullptr, *inj2_ = nullptr, *lo_ = nullptr, *rs_ = nullptr, *bo_ = nullptr; float* grs_ = nullptr; // the hyper-connection reads' scratch (zeroed with the arena) uint8_t* xq_ = nullptr; - float *qfull_ = nullptr, *qcur_ = nullptr, *kcur_ = nullptr, *vcur_ = nullptr, *attn_ = nullptr, *attn32_ = nullptr; + float *qfull_ = nullptr, *qcur_ = nullptr, *kcur_ = nullptr, *vcur_ = nullptr, *attn32_ = nullptr; float* attn_scratch_ = nullptr; float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *parts_ = nullptr, *y_ = nullptr, *sample_ = nullptr; int32_t *ids_ = nullptr, *hit_slot_ = nullptr, *hit_dst_ = nullptr, *hit_count_ = nullptr, *out_ids_ = nullptr; diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index e2b67c5803e..0cee1782011 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -260,7 +260,7 @@ class Verifier { float *qkv_L_ = nullptr, *h_L_ = nullptr, *gate_L_ = nullptr, *beta_L_ = nullptr; // per GDN layer float *z_ = nullptr, *y_ = nullptr, *y_dummy_ = nullptr; float *qfull_ = nullptr, *qcur_ = nullptr, *kcur_ = nullptr, *vcur_ = nullptr, *idx_raw_L_ = nullptr; - float *qidx_ = nullptr, *scores_ = nullptr, *attn_ = nullptr, *attn32_ = nullptr, *attn_scratch_ = nullptr; + float *qidx_ = nullptr, *scores_ = nullptr, *attn32_ = nullptr, *attn_scratch_ = nullptr; float* tail_snap_ = nullptr; // per QSA layer int32_t* sel_ = nullptr; float *logits_ = nullptr, *w_ = nullptr, *shared_ = nullptr, *hit_out_ = nullptr; diff --git a/include/strata/kernels/qsa_decode_attn.hpp b/include/strata/kernels/qsa_decode_attn.hpp index 0705321f53a..3df34c21811 100644 --- a/include/strata/kernels/qsa_decode_attn.hpp +++ b/include/strata/kernels/qsa_decode_attn.hpp @@ -1,17 +1,15 @@ -// include/strata/kernels/qsa_decode_attn.hpp - plan v0.3 P3/P7: split-K decode attention over the selected cells. +// include/strata/kernels/qsa_decode_attn.hpp - the QSA attention over each query's selected cells. // -// The first attention kernel runs one block per query head (24 blocks on a 48-SM part), reads each key row with one -// thread (uncoalesced) and walks every selected cell serially for the values; at a 4K context (2,051 selected -// cells) that costs ~0.3-0.5 ms per QSA layer. This one reads the KV POOLS directly through the page table and the -// selection ids (no gather copy), in chunks of CHUNK cells per block, and serves all `n_head / n_head_kv` query -// heads that share a KV head from one read of the chunk: +// The kernels read the KV POOLS directly through the page table and the selection ids (no gather copy) and serve +// all `n_head / n_head_kv` query heads that share a KV head from one read of each cell: a block per (KV head, query) +// walks the query's cells in tiles of 64 with an online softmax. INT8 pools on the tensor cores: q and the +// probabilities as 24-bit fixed point (three 8-bit limbs) against the int8 codes, the integer sums exact, ~2.5e-6 of +// a head's largest output from FP64 on real prompts (FP32 kernels ~3e-6). FP16 pools in FP32. // -// grid (ceil(cap / CHUNK), n_head_kv): s = q.k * scale for 12 heads x CHUNK cells, chunk max m and sum l, -// acc = p . V -> partials -// grid (n_head): merge the chunks with the usual log-sum-exp rescale -// -// FP16 pools or INT8 pools (codes + fp16 scale per 64 values, `kv_q8.hpp`). The grid is sized by the capacity; -// the real count comes from `step[kStepWidth]` as for every other capturable QSA kernel. +// The prompt path's queries fill the GPU by their number. A verify window's 1-4 queries would leave most of it idle, +// so the decode splits each query's cells among up to 20 blocks (whole tiles, all queries' blocks in one wave) whose +// partial sums a merge adds with the usual log-sum-exp rescale. Grids are sized by the capacity; the real count comes +// from `step[kStepWidth]` as for every other capturable QSA kernel. #pragma once #include "strata/kernels/qsa.hpp" @@ -30,22 +28,22 @@ struct QsaAttnPools { const int32_t* page_table = nullptr; }; -/// Scratch floats for `cap` selected cells: partial accumulators, maxima and sums. +/// Scratch floats a query with `cap` selected cells needs: its splits' partial sums, maxima and sums. uint64_t qsa_decode_attn_scratch_floats(int64_t cap, const QsaShapes& s); void qsa_decode_attn_step(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* step, int64_t cap, const QsaShapes& s, float* scratch, float* attn, void* stream); /// Plan v0.3 P5: `n_q` queries at once, each with its own selection: q [n_q, n_head, 256], ids [n_q, cap], steps -/// [n_q, kStepCount], attn [n_q, n_head, 256]; scratch is `n_q` times the single-query size. +/// [n_q, kStepCount], attn [n_q, n_head, 256]; scratch is `n_q` times the single-query size. With `gate` (the +/// query projection's rows [n_q, n_head, 2 * 256]: a head's queries, then its gate logits), attn is the gated output, +/// bitwise native_qsa_gate_apply's. void qsa_decode_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, - int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream); + int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream, + const float* gate = nullptr); -/// The prompt path's form of qsa_decode_attn_batch (no scratch): a block per (KV head, query) walks the query's -/// cells in tiles of 64 with an online softmax, so no partial sums go through memory. INT8 pools on the tensor -/// cores: q and the probabilities as 24-bit fixed point (three 8-bit limbs) against the int8 codes, the integer sums -/// exact, ~2.5e-6 of a head's largest output from FP64 on real prompts (FP32 kernels ~3e-6). FP16 pools in FP32, -/// the sums ordered differently from the decode kernel. +/// The prompt path's form of qsa_decode_attn_batch (no scratch): a block per (KV head, query) walks all of the +/// query's cells, so no partial sums go through memory. void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap, const QsaShapes& s, float* attn, int64_t n_q, void* stream); diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index a0d52a8f998..460cd8b9b2e 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -203,7 +203,7 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session xq_ = b.take(strata::kernels::native_q8_1_bytes((int) (NH * HD), 8)); qfull_ = b.take(T * NH * 2 * HD); qcur_ = b.take(T * NH * HD); kcur_ = b.take(T * NKV * HD); vcur_ = b.take(T * NKV * HD); - attn_ = b.take(T * NH * HD); attn32_ = b.take(T * NH * HD); + attn32_ = b.take(T * NH * HD); attn_scratch_ = b.take((uint64_t) attn_scratch_floats_); // the full layer runs one row at a time logits_ = b.take(T * (uint64_t) g.n_expert); w_ = b.take(T * K); ids_ = b.take(T * K); shared_ = b.take(T * N); parts_ = b.take(T * K * N); y_ = b.take(T * N); @@ -355,9 +355,7 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri if (st_.kv_int8) { pools.k_q = st_.k_q; pools.v_q = st_.v_q; pools.k_scale = st_.k_scale; pools.v_scale = st_.v_scale; } else { pools.k_pool = st_.k_pool; pools.v_pool = st_.v_pool; } if (window_ > 0) window_ids(const_cast(step), T, (int) window_, ident_, cap_, cs); - qsa_decode_attn_batch(qcur_, pools, ident_, step, cap_, s, attn_scratch_, attn_, T, cs); - for (int t = 0; t < T; ++t) - native_qsa_gate_apply(attn_ + t * NH * HD, qfull_ + t * NH * 2 * HD, attn32_ + t * NH * HD, (int) NH, (int) HD, cs); + qsa_decode_attn_batch(qcur_, pools, ident_, step, cap_, s, attn_scratch_, attn32_, T, cs, qfull_); native_quantize_q8_1(attn32_, xq_, (int) (NH * HD), T, cs); native_mmvq(GGML_Q8_0, q8("self_attn.o_proj.weight"), xq_, bo_, (int) (NH * HD), (int) N, T, cs); // ---- the MLP hyper-connection (the attention write folded in) diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 98899da3b86..12cb420397e 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -249,7 +249,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& kcur_ = b.take(T * NKV * HD); vcur_ = b.take(T * NKV * HD); idx_raw_L_ = b.take(nQ * T * ID); qidx_ = b.take(T * IQ * ID); scores_ = b.take(T * (uint64_t) max_blocks_); sel_ = b.take(T * (uint64_t) cap_); - attn_ = b.take(T * NH * HD); attn32_ = b.take(T * NH * HD); + attn32_ = b.take(T * NH * HD); attn_scratch_ = b.take(T * (uint64_t) attn_scratch_floats_); tail_snap_ = b.take(nQ * TS); logits_ = b.take(T * (uint64_t) g.n_expert); w_ = b.take(T * K); ids_ = b.take(T * K); @@ -692,9 +692,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { if (st.kv_int8) { pools.k_q = st.k_q; pools.v_q = st.v_q; pools.k_scale = st.k_scale; pools.v_scale = st.v_scale; } else { pools.k_pool = st.k_pool; pools.v_pool = st.v_pool; } qsa_decode_attn_batch(qcur_ + tb * NH * HD, pools, sel_ + (size_t) tb * cap_, step_ + tb * kStepCount, cap_, - s, attn_scratch_ + (size_t) tb * attn_scratch_floats_, attn_ + tb * NH * HD, n, cs); - native_qsa_gate_apply(attn_ + tb * NH * HD, qfull_ + tb * NH * 2 * HD, attn32_ + tb * NH * HD, - (int) (n * NH), (int) HD, cs); + s, attn_scratch_ + (size_t) tb * attn_scratch_floats_, attn32_ + tb * NH * HD, + n, cs, qfull_ + tb * NH * 2 * HD); quant(attn32_ + tb * NH * HD, NH * HD); proj(wo, bo_ + tb * N, NH * HD, N, cs); } diff --git a/src/kernels/cuda/qsa_decode_attn.cu b/src/kernels/cuda/qsa_decode_attn.cu index fc69e8b9256..3ef91c3c372 100644 --- a/src/kernels/cuda/qsa_decode_attn.cu +++ b/src/kernels/cuda/qsa_decode_attn.cu @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -14,9 +15,10 @@ namespace { constexpr int HD = 256; // head_dim constexpr int G = 12; // query heads per KV head (24 / 2) -constexpr int CHUNK = 64; // cells per block constexpr int THREADS = 256; constexpr int WARPS = THREADS / 32; +constexpr int PT = 64; // cells per tile +constexpr int SPLIT_MAX = 20; // a decode query's splits at most __device__ __forceinline__ float warp_sum(float v) { #pragma unroll @@ -29,161 +31,43 @@ __device__ __forceinline__ float warp_max(float v) { return v; } -// 8 consecutive values of one cell's key or value row for KV head `kvh`, dimensions [d0, d0+8). -template -__device__ __forceinline__ void load8(const QsaAttnPools& p, bool value, long long row, int d0, float* out) { - if constexpr (!INT8) { - const uint16_t* base = (value ? p.v_pool : p.k_pool) + row * HD + d0; - const uint4 raw = *reinterpret_cast(base); - const __half2* h2 = reinterpret_cast(&raw); -#pragma unroll - for (int j = 0; j < 4; ++j) { - const float2 f = __half22float2(h2[j]); - out[2 * j] = f.x; - out[2 * j + 1] = f.y; - } - } else { - const int8_t* codes = (value ? p.v_q : p.k_q) + row * HD + d0; - const uint16_t sbits = (value ? p.v_scale : p.k_scale)[row * (HD / KV_Q8_GROUP) + d0 / KV_Q8_GROUP]; - const float sc = __half2float(__ushort_as_half(sbits)); - const uint2 raw = *reinterpret_cast(codes); - const int8_t* c = reinterpret_cast(&raw); -#pragma unroll - for (int j = 0; j < 8; ++j) out[j] = (float) c[j] * sc; - } -} - -template -__global__ void __launch_bounds__(THREADS) attn_chunk_kernel(const float* __restrict__ q, QsaAttnPools p, - const int32_t* __restrict__ ids, - const int32_t* __restrict__ step, int n_kv_heads, - int page_size, float scale, float* __restrict__ part_acc, - float* __restrict__ part_m, float* __restrict__ part_l, - int n_chunks, int cap = 0, long long scratch_stride = 0) { - // batched form: query blockIdx.z, with its own q row, selection, step and scratch - q += (size_t) blockIdx.z * (size_t) (n_kv_heads * G) * HD; - ids += (size_t) blockIdx.z * (size_t) cap; - step += (size_t) blockIdx.z * kStepCount; - part_acc += (size_t) blockIdx.z * (size_t) scratch_stride; - part_m += (size_t) blockIdx.z * (size_t) scratch_stride; - part_l += (size_t) blockIdx.z * (size_t) scratch_stride; - __shared__ __align__(16) float sq[G][HD]; // 12 KB: this KV head's query heads - __shared__ float sp[G][CHUNK]; // scores, then probabilities - __shared__ long long srow[CHUNK]; // pool row of each cell (page, kv head, slot) - const int n_ids = __ldg(step + kStepWidth); - const int chunk = blockIdx.x, kvh = blockIdx.y; - const int t = threadIdx.x, lane = t & 31, warp = t >> 5; - const int c0 = chunk * CHUNK; - const int n_here = min(CHUNK, n_ids - c0); - const int slot = kvh * n_chunks + chunk; - if (n_here <= 0) { - if (t < G) { part_m[slot * G + t] = -FLT_MAX; part_l[slot * G + t] = 0.0f; } - return; - } - for (int i = t; i < G * HD; i += THREADS) sq[i / HD][i % HD] = q[(size_t) (kvh * G) * HD + i]; - if (t < CHUNK) { - long long r = -1; - if (t < n_here) { - const int cell = ids[c0 + t]; - const long long page = (long long) p.page_table[cell / page_size]; - r = (page * n_kv_heads + kvh) * page_size + (cell % page_size); - } - srow[t] = r; - } - __syncthreads(); - // scores: each warp takes cells warp, warp+8, ...; each lane holds 8 of the 256 dimensions. - for (int c = warp; c < CHUNK; c += WARPS) { - if (c >= n_here) { - if (lane < G) sp[lane][c] = -FLT_MAX; - continue; - } - float k8[8]; - load8(p, false, srow[c], lane * 8, k8); -#pragma unroll - for (int h = 0; h < G; ++h) { - const float4 qa = *reinterpret_cast(&sq[h][lane * 8]); - const float4 qb = *reinterpret_cast(&sq[h][lane * 8 + 4]); - float s = k8[0] * qa.x + k8[1] * qa.y + k8[2] * qa.z + k8[3] * qa.w + - k8[4] * qb.x + k8[5] * qb.y + k8[6] * qb.z + k8[7] * qb.w; - s = warp_sum(s); - if (lane == 0) sp[h][c] = s * scale; - } - } - __syncthreads(); - // per-head chunk max and exp-sum: warp w handles heads w and w+8. - for (int h = warp; h < G; h += WARPS) { - const float a = sp[h][lane], b = sp[h][lane + 32]; - const float m = warp_max(fmaxf(a, b)); - const float ea = (lane < n_here) ? __expf(a - m) : 0.0f; - const float eb = (lane + 32 < n_here) ? __expf(b - m) : 0.0f; - sp[h][lane] = ea; - sp[h][lane + 32] = eb; - const float l = warp_sum(ea + eb); - if (lane == 0) { part_m[slot * G + h] = m; part_l[slot * G + h] = l; } - } - __syncthreads(); - // values: thread t owns dimension t for all 12 heads. - float acc[G]; -#pragma unroll - for (int h = 0; h < G; ++h) acc[h] = 0.0f; - for (int c = 0; c < n_here; ++c) { - float v; - if constexpr (!INT8) { - v = __half2float(__ushort_as_half(p.v_pool[srow[c] * HD + t])); - } else { - const float sc = __half2float(__ushort_as_half(p.v_scale[srow[c] * (HD / KV_Q8_GROUP) + t / KV_Q8_GROUP])); - v = (float) p.v_q[srow[c] * HD + t] * sc; - } -#pragma unroll - for (int h = 0; h < G; ++h) acc[h] = fmaf(sp[h][c], v, acc[h]); - } -#pragma unroll - for (int h = 0; h < G; ++h) part_acc[((size_t) slot * G + h) * HD + t] = acc[h]; -} - -__global__ void __launch_bounds__(HD) attn_merge_kernel(const float* __restrict__ part_acc, - const float* __restrict__ part_m, - const float* __restrict__ part_l, int n_chunks, - float* __restrict__ attn, long long scratch_stride = 0) { - part_acc += (size_t) blockIdx.y * (size_t) scratch_stride; - part_m += (size_t) blockIdx.y * (size_t) scratch_stride; - part_l += (size_t) blockIdx.y * (size_t) scratch_stride; - attn += (size_t) blockIdx.y * (size_t) gridDim.x * HD; - const int h = blockIdx.x; // global query head - const int kvh = h / G, hl = h % G; - const int d = threadIdx.x; - float M = -FLT_MAX; - for (int c = 0; c < n_chunks; ++c) M = fmaxf(M, part_m[(kvh * n_chunks + c) * G + hl]); - float L = 0.0f, acc = 0.0f; - for (int c = 0; c < n_chunks; ++c) { - const int slot = kvh * n_chunks + c; - const float m = part_m[slot * G + hl]; - if (m == -FLT_MAX) continue; - const float w = __expf(m - M); - L = fmaf(part_l[slot * G + hl], w, L); - acc = fmaf(part_acc[((size_t) slot * G + hl) * HD + d], w, acc); - } - attn[(size_t) h * HD + d] = L > 0.0f ? acc / L : 0.0f; +// The decode's partial results, per query `stride` floats: [split][head][HD] sums of p * v (p relative to the split's +// maximum), then at `m` and `l` (offsets from `acc`) [split][head] maxima and sums of p. +struct Partials { + float* acc = nullptr; + long long m = 0, l = 0, stride = 0; + int splits = 0; +}; +// the cells of each of a query's splits: split k takes [k c, min((k + 1) c, n)), whole tiles while the cells last +__host__ __device__ __forceinline__ int split_cells(int n, int splits) { + const int c = (n + splits - 1) / splits; + return c > PT ? c : PT; } -// ---- the prompt path, FP16 pools: a block per (KV head, query) walks all of the query's cells in tiles, online -// softmax. Scores: thread (cell, quarter) dots its cell's 64 dimensions of that quarter with the 12 query heads, the -// quarters summed through shared memory. Values: thread (dimension pair, cell parity) accumulates its two dimensions -// of the 12 heads over every other cell; the parities are added at the end. -constexpr int PT = 64; // cells per tile - -__global__ void __launch_bounds__(THREADS) attn_prefill_f16_kernel(const float* __restrict__ q, QsaAttnPools p, - const int32_t* __restrict__ ids, - const int32_t* __restrict__ steps, int n_kv_heads, - int page_size, float scale, float* __restrict__ attn, - int cap) { +// ---- FP16 pools: a block per (KV head, query) walks all of the query's cells in tiles, online softmax; SPLIT: a +// block per (KV head, query, split) its split's cells, writing partials. Scores: thread (cell, quarter) dots its +// cell's 64 dimensions of that quarter with the 12 query heads, the quarters summed through shared memory. Values: +// thread (dimension pair, cell parity) accumulates its two dimensions of the 12 heads over every other cell; the +// parities are added at the end. +template +__global__ void __launch_bounds__(THREADS) attn_f16_kernel(const float* __restrict__ q, QsaAttnPools p, + const int32_t* __restrict__ ids, + const int32_t* __restrict__ steps, int n_kv_heads, + int page_size, float scale, float* __restrict__ attn, + int cap, Partials part) { const int kvh = blockIdx.x; const size_t qi = blockIdx.y; const size_t head0 = qi * (size_t) (n_kv_heads * G) + (size_t) kvh * G; q += head0 * HD; - attn += head0 * HD; + if constexpr (!SPLIT) attn += head0 * HD; ids += qi * (size_t) cap; - const int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + if constexpr (SPLIT) { + const int c = split_cells(n_ids, part.splits), c0 = (int) blockIdx.z * c; + if (c0 >= n_ids) return; + ids += c0; + n_ids = min(c, n_ids - c0); + } __shared__ __align__(16) float sq[G][HD]; // 12 KB: this KV head's query heads __shared__ __align__(16) float spart[4][G][PT]; // 12 KB: the quarters' partial scores; the parity sums at the end __shared__ __align__(16) float sp[PT][G]; // probabilities, cell-major @@ -299,7 +183,21 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_f16_kernel(const float* } } __syncthreads(); - if (parity == 0) { + if constexpr (SPLIT) { + float* base = part.acc + qi * part.stride; + const size_t hq = (size_t) blockIdx.z * n_kv_heads * G + (size_t) kvh * G; // the block's first head + if (parity == 0) { +#pragma unroll + for (int h = 0; h < G; ++h) { + base[(hq + h) * HD + d0] = acc0[h] + odd[h * HD + d0]; + base[(hq + h) * HD + d0 + 1] = acc1[h] + odd[h * HD + d0 + 1]; + } + } + if (t < G) { + base[part.m + hq + t] = s_m[t]; + base[part.l + hq + t] = s_l[t]; + } + } else if (parity == 0) { #pragma unroll for (int h = 0; h < G; ++h) { const float l = s_l[h]; @@ -309,10 +207,11 @@ __global__ void __launch_bounds__(THREADS) attn_prefill_f16_kernel(const float* } } -// ---- the prompt path, INT8 pools: the codes on the tensor cores (mma m16n8k32, int8). A block per (KV head, query), -// its 12 query heads the rows of the 16-row fragments, 64 cells a tile. q and the probabilities enter as 24-bit fixed -// point, three 8-bit limbs each; the integer sums are exact, so the error is the fixed point's: q to 2^-23 of its row's -// power-of-two bound, a probability times its V scale to 2^-23 of its tile's. Warp-specialized, one block an SM: +// ---- INT8 pools: the codes on the tensor cores (mma m16n8k32, int8). A block per (KV head, query), or per (KV head, +// query, split) with SPLIT, its 12 query heads the rows of the 16-row fragments, 64 cells a tile. q and the +// probabilities enter as 24-bit fixed point, three 8-bit limbs each; the integer sums are exact, so the error is the +// fixed point's: q to 2^-23 of its row's power-of-two bound, a probability times its V scale to 2^-23 of its tile's. +// Warp-specialized, one block an SM: // warps 0-3 score 16 cells of each tile over all 256 dims (q's limbs in registers) and run the online softmax; // warps 4-7 multiply a 64-dim group of V each: the probabilities times the V scales as limbs against V through // ldmatrix.trans (cells and dims permuted inside the fragments, the same way on both sides); @@ -344,6 +243,7 @@ struct __align__(16) Smem { float red[2][4][16]; // the score warps' tile maxima float lsum[4][16]; float qmax[16], qsc[16]; + float mfin[16]; // SPLIT: the rows' final maxima }; // Score warp w's B fragments: column j of its n-tile n is cell 16w + 4 (j >> 1) + 2n + (j & 1), so a lane's C @@ -407,11 +307,12 @@ __device__ __forceinline__ float scale_of(uint2 s, int g) { } } // namespace pf -__global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float* __restrict__ q, QsaAttnPools p, - const int32_t* __restrict__ ids, - const int32_t* __restrict__ steps, int n_kv_heads, - int page_size, float scale, float* __restrict__ attn, - int cap) { +template +__global__ void __launch_bounds__(pf::NT, 1) attn_mma_kernel(const float* __restrict__ q, QsaAttnPools p, + const int32_t* __restrict__ ids, + const int32_t* __restrict__ steps, int n_kv_heads, + int page_size, float scale, float* __restrict__ attn, + int cap, Partials part) { using namespace pf; extern __shared__ __align__(16) uint8_t smem_raw[]; Smem& sm = *reinterpret_cast(smem_raw); @@ -419,9 +320,15 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float const size_t qi = blockIdx.y; const size_t head0 = qi * (size_t) (n_kv_heads * G) + (size_t) kvh * G; q += head0 * HD; - attn += head0 * HD; + if constexpr (!SPLIT) attn += head0 * HD; ids += qi * (size_t) cap; - const int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + int n_ids = __ldg(steps + qi * kStepCount + kStepWidth); + if constexpr (SPLIT) { + const int c = split_cells(n_ids, part.splits), c0 = (int) blockIdx.z * c; + if (c0 >= n_ids) return; + ids += c0; + n_ids = min(c, n_ids - c0); + } const int n_tiles = (n_ids + PT - 1) / PT; const int t = threadIdx.x, lane = t & 31, warp = t >> 5, gid = lane >> 2, tig = lane & 3; const float kS = -65793.0f * MAGIC_F; @@ -444,7 +351,7 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float } } __syncthreads(); - if (n_tiles == 0) { + if (!SPLIT && n_tiles == 0) { for (int i = t; i < G * HD; i += NT) attn[i] = 0.0f; return; } @@ -623,6 +530,7 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float lp[r] += __shfl_xor_sync(0xffffffffu, lp[r], 2); } if (tig == 0) { sm.lsum[w][gid] = lp[0]; sm.lsum[w][gid + 8] = lp[1]; } + if (SPLIT && w == 0 && tig == 0) { sm.mfin[gid] = m_run[0]; sm.mfin[gid + 8] = m_run[1]; } bar_sync(END, NPB); } else { // ======== values: dims 64g .. 64g+63 as 16-dim blocks db; n-tile (db, par) column j is dim @@ -712,20 +620,133 @@ __global__ void __launch_bounds__(pf::NT, 1) attn_prefill_mma_kernel(const float if (i + VS < n_tiles) bar_arrive(EMPTY_V + s, NVB); } bar_sync(END, NPB); + if constexpr (SPLIT) { // o as it is, with each row's maximum and sum + float* base = part.acc + qi * part.stride; + const size_t hq = (size_t) blockIdx.z * n_kv_heads * G + (size_t) kvh * G; // the block's first head #pragma unroll - for (int r = 0; r < 2; ++r) { - const int row = gid + 8 * r; - if (row >= G) continue; - const float l = (sm.lsum[0][row] + sm.lsum[1][row]) + (sm.lsum[2][row] + sm.lsum[3][row]); -#pragma unroll - for (int db = 0; db < 4; ++db) { // dims 64g + 16db + 4tig + {0, 1, 2, 3} - float4 out = make_float4(0.f, 0.f, 0.f, 0.f); - if (l > 0.0f) out = make_float4(o[db][0][2 * r] / l, o[db][1][2 * r] / l, o[db][0][2 * r + 1] / l, - o[db][1][2 * r + 1] / l); - *reinterpret_cast(&attn[(size_t) row * HD + 64 * g + 16 * db + 4 * tig]) = out; + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + if (row >= G) continue; +#pragma unroll + for (int db = 0; db < 4; ++db) + *reinterpret_cast(&base[(hq + row) * HD + 64 * g + 16 * db + 4 * tig]) = + make_float4(o[db][0][2 * r], o[db][1][2 * r], o[db][0][2 * r + 1], o[db][1][2 * r + 1]); + } + if (g == 0 && lane < G) { + base[part.m + hq + lane] = sm.mfin[lane]; + base[part.l + hq + lane] = + (sm.lsum[0][lane] + sm.lsum[1][lane]) + (sm.lsum[2][lane] + sm.lsum[3][lane]); + } + } else { +#pragma unroll + for (int r = 0; r < 2; ++r) { + const int row = gid + 8 * r; + if (row >= G) continue; + const float l = (sm.lsum[0][row] + sm.lsum[1][row]) + (sm.lsum[2][row] + sm.lsum[3][row]); +#pragma unroll + for (int db = 0; db < 4; ++db) { // dims 64g + 16db + 4tig + {0, 1, 2, 3} + float4 out = make_float4(0.f, 0.f, 0.f, 0.f); + if (l > 0.0f) out = make_float4(o[db][0][2 * r] / l, o[db][1][2 * r] / l, o[db][0][2 * r + 1] / l, + o[db][1][2 * r + 1] / l); + *reinterpret_cast(&attn[(size_t) row * HD + 64 * g + 16 * db + 4 * tig]) = out; + } + } + } + } +} + +// x * sigmoid(raw) as native_qsa_gate_apply computes it: its TU's fast-math instructions +__device__ __forceinline__ float gated(float x, float raw) { + float r; + asm("{\n\t.reg .f32 e;\n\tmul.ftz.f32 e, %2, 0fBFB8AA3B;\n\tex2.approx.ftz.f32 e, e;\n\t" + "add.ftz.f32 e, e, 0f3F800000;\n\trcp.approx.ftz.f32 e, e;\n\tmul.ftz.f32 %0, %1, e;\n\t}" + : "=f"(r) + : "f"(x), "f"(raw)); + return r; +} + +// The decode's merge: a block per (head, query), 4 groups of 64 threads adding every 4th split's sums, a thread 4 +// dims. Every thread reads all the splits' maxima and sums, so all of its loads go out at once. +__global__ void __launch_bounds__(THREADS) attn_merge_kernel(Partials part, const int32_t* __restrict__ steps, + const float* __restrict__ gate, float* __restrict__ attn) { + const int h = blockIdx.x, n_head = gridDim.x; + const size_t qi = blockIdx.y; + const int n_ids = __ldg(steps + qi * kStepCount + kStepWidth), c = split_cells(n_ids, part.splits); + const int n_sp = (n_ids + c - 1) / c; + const float* base = part.acc + qi * part.stride; + const int t = threadIdx.x, grp = t >> 6, d4 = 4 * (t & 63); + float4 o[SPLIT_MAX / 4]; + float m[SPLIT_MAX], l[SPLIT_MAX]; +#pragma unroll + for (int k = 0; k < SPLIT_MAX / 4; ++k) + if (grp + 4 * k < n_sp) + o[k] = *reinterpret_cast(base + ((size_t) (grp + 4 * k) * n_head + h) * HD + d4); +#pragma unroll + for (int k = 0; k < SPLIT_MAX; ++k) + if (k < n_sp) { + m[k] = base[part.m + k * n_head + h]; + l[k] = base[part.l + k * n_head + h]; + } + float4 raw = make_float4(0.f, 0.f, 0.f, 0.f); + if (gate && grp == 0) raw = *reinterpret_cast(gate + (qi * n_head + h) * 2 * HD + HD + d4); + float M = -FLT_MAX, L = 0.0f; +#pragma unroll + for (int k = 0; k < SPLIT_MAX; ++k) + if (k < n_sp) M = fmaxf(M, m[k]); + float4 a = make_float4(0.f, 0.f, 0.f, 0.f); +#pragma unroll + for (int k = 0; k < SPLIT_MAX; ++k) + if (k < n_sp) { + const float w = __expf(m[k] - M); + L = fmaf(l[k], w, L); + if ((k & 3) == grp) { + const float4 x = o[k >> 2]; + a = make_float4(fmaf(x.x, w, a.x), fmaf(x.y, w, a.y), fmaf(x.z, w, a.z), fmaf(x.w, w, a.w)); } } + __shared__ float4 red[3][64]; + if (grp > 0) red[grp - 1][t & 63] = a; + __syncthreads(); + if (grp > 0) return; +#pragma unroll + for (int k = 0; k < 3; ++k) { + const float4 b = red[k][t]; + a = make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); } + float4 r = make_float4(0.f, 0.f, 0.f, 0.f); + if (L > 0.0f) r = make_float4(a.x / L, a.y / L, a.z / L, a.w / L); + if (gate) r = make_float4(gated(r.x, raw.x), gated(r.y, raw.y), gated(r.z, raw.z), gated(r.w, raw.w)); + *reinterpret_cast(attn + (qi * n_head + h) * HD + d4) = r; +} + +// A decode query's splits at most: SPLIT_MAX (more cost the merge and each block's setup more than they gain), whole +// tiles. +int64_t max_splits(int64_t cap) { return std::max(1, std::min(SPLIT_MAX, (cap + PT - 1) / PT)); } +// n_q queries' splits: a block of the INT8 kernel takes an SM, so all queries' blocks in one wave +int decode_splits(int64_t cap, int64_t n_q, int64_t n_kv) { + int dev = 0, sms = 0; + cudaGetDevice(&dev); + cudaDeviceGetAttribute(&sms, cudaDevAttrMultiProcessorCount, dev); + return (int) std::max(1, std::min(max_splits(cap), sms / (n_kv * n_q))); +} + +// the per-query scratch's layout for `splits` +Partials partials(float* scratch, int splits, int64_t n_head, int64_t stride) { + Partials part; + part.acc = scratch; + part.m = (long long) splits * n_head * HD; + part.l = part.m + (long long) splits * n_head; + part.stride = stride; + part.splits = splits; + return part; +} + +// the INT8 kernel's shared memory, above the default limit (once for each form) +template +void allow_mma_smem() { + static const cudaError_t e = cudaFuncSetAttribute( + attn_mma_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, (int) sizeof(pf::Smem)); + (void) e; } } // namespace @@ -741,11 +762,7 @@ void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* } const float scale = 1.0f / sqrtf((float) HD); cudaStream_t st = (cudaStream_t) stream; - static bool attr = false; - if (int8 && !attr) { - cudaFuncSetAttribute(attn_prefill_mma_kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, (int) sizeof(pf::Smem)); - attr = true; - } + if (int8) allow_mma_smem(); for (int64_t q0 = 0; q0 < n_q; q0 += 65535) { const int64_t nb = n_q - q0 < 65535 ? n_q - q0 : 65535; const dim3 grid((unsigned) s.n_head_kv, (unsigned) nb); @@ -753,12 +770,11 @@ void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* const int32_t* ids_q = ids + (size_t) q0 * (size_t) cap; const int32_t* steps_q = steps + (size_t) q0 * kStepCount; if (int8) - attn_prefill_mma_kernel<<>>(q + qo, pools, ids_q, steps_q, - (int) s.n_head_kv, (int) s.page_size, scale, - attn + qo, (int) cap); + attn_mma_kernel<<>>( + q + qo, pools, ids_q, steps_q, (int) s.n_head_kv, (int) s.page_size, scale, attn + qo, (int) cap, {}); else - attn_prefill_f16_kernel<<>>(q + qo, pools, ids_q, steps_q, (int) s.n_head_kv, - (int) s.page_size, scale, attn + qo, (int) cap); + attn_f16_kernel<<>>( + q + qo, pools, ids_q, steps_q, (int) s.n_head_kv, (int) s.page_size, scale, attn + qo, (int) cap, {}); } const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { @@ -768,74 +784,44 @@ void qsa_prefill_attn(const float* q, const QsaAttnPools& pools, const int32_t* } void qsa_decode_attn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, - int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream) { + int64_t cap, const QsaShapes& s, float* scratch, float* attn, int64_t n_q, void* stream, + const float* gate) { if (n_q <= 0) return; + const bool int8 = pools.k_q != nullptr; if (s.head_dim != HD || s.n_head != (int64_t) G * s.n_head_kv || cap <= 0 || !scratch || !ids || !steps || - !pools.page_table || n_q > 65535) { - std::fprintf(stderr, "qsa_decode_attn_batch: unsupported geometry or missing buffers\n"); + !pools.page_table || n_q > 65535 || + (int8 ? (!pools.v_q || !pools.k_scale || !pools.v_scale) : (!pools.k_pool || !pools.v_pool))) { + std::fprintf(stderr, "qsa_decode_attn: unsupported geometry or missing buffers\n"); std::exit(1); } - const bool int8 = pools.k_q != nullptr; - const int n_chunks = (int) ((cap + CHUNK - 1) / CHUNK); - // per query: [acc: n_chunks*n_head*HD][m: n_chunks*n_head][l: n_chunks*n_head], all offsets from one stride - const long long stride = (long long) qsa_decode_attn_scratch_floats(cap, s); - float* part_acc = scratch; - float* part_m = scratch + (size_t) n_chunks * s.n_head * HD; - float* part_l = part_m + (size_t) n_chunks * s.n_head; + const int splits = decode_splits(cap, n_q, s.n_head_kv); + const Partials part = partials(scratch, splits, s.n_head, (long long) qsa_decode_attn_scratch_floats(cap, s)); const float scale = 1.0f / sqrtf((float) HD); - const dim3 grid((unsigned) n_chunks, (unsigned) s.n_head_kv, (unsigned) n_q); + const dim3 grid((unsigned) s.n_head_kv, (unsigned) n_q, (unsigned) splits); cudaStream_t st = (cudaStream_t) stream; - if (int8) - attn_chunk_kernel<<>>(q, pools, ids, steps, (int) s.n_head_kv, (int) s.page_size, - scale, part_acc, part_m, part_l, n_chunks, (int) cap, stride); - else - attn_chunk_kernel<<>>(q, pools, ids, steps, (int) s.n_head_kv, (int) s.page_size, - scale, part_acc, part_m, part_l, n_chunks, (int) cap, stride); - attn_merge_kernel<<>>(part_acc, part_m, part_l, n_chunks, - attn, stride); + if (int8) { + allow_mma_smem(); + attn_mma_kernel<<>>( + q, pools, ids, steps, (int) s.n_head_kv, (int) s.page_size, scale, nullptr, (int) cap, part); + } else { + attn_f16_kernel<<>>( + q, pools, ids, steps, (int) s.n_head_kv, (int) s.page_size, scale, nullptr, (int) cap, part); + } + attn_merge_kernel<<>>(part, steps, gate, attn); const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { - std::fprintf(stderr, "qsa_decode_attn_batch: %s\n", cudaGetErrorString(e)); + std::fprintf(stderr, "qsa_decode_attn: %s\n", cudaGetErrorString(e)); std::exit(1); } } uint64_t qsa_decode_attn_scratch_floats(int64_t cap, const QsaShapes& s) { - const int64_t chunks = (cap + CHUNK - 1) / CHUNK; - return (uint64_t) chunks * (uint64_t) s.n_head * (HD + 2) + 64; + return (uint64_t) max_splits(cap) * (uint64_t) s.n_head * (HD + 2) + 64; } void qsa_decode_attn_step(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* step, int64_t cap, const QsaShapes& s, float* scratch, float* attn, void* stream) { - if (s.head_dim != HD || s.n_head != (int64_t) G * s.n_head_kv || cap <= 0 || !scratch || !ids || !step || - !pools.page_table) { - std::fprintf(stderr, "qsa_decode_attn: unsupported geometry or missing buffers\n"); - std::exit(1); - } - const bool int8 = pools.k_q != nullptr; - if (int8 ? (!pools.v_q || !pools.k_scale || !pools.v_scale) : (!pools.k_pool || !pools.v_pool)) { - std::fprintf(stderr, "qsa_decode_attn: incomplete KV pools\n"); - std::exit(1); - } - const int n_chunks = (int) ((cap + CHUNK - 1) / CHUNK); - float* part_acc = scratch; - float* part_m = scratch + (size_t) n_chunks * s.n_head * HD; - float* part_l = part_m + (size_t) n_chunks * s.n_head; - const float scale = 1.0f / sqrtf((float) HD); - const dim3 grid((unsigned) n_chunks, (unsigned) s.n_head_kv); - cudaStream_t st = (cudaStream_t) stream; - if (int8) - attn_chunk_kernel<<>>(q, pools, ids, step, (int) s.n_head_kv, (int) s.page_size, - scale, part_acc, part_m, part_l, n_chunks); - else - attn_chunk_kernel<<>>(q, pools, ids, step, (int) s.n_head_kv, (int) s.page_size, - scale, part_acc, part_m, part_l, n_chunks); - attn_merge_kernel<<<(unsigned) s.n_head, HD, 0, st>>>(part_acc, part_m, part_l, n_chunks, attn); - const cudaError_t e = cudaGetLastError(); - if (e != cudaSuccess) { - std::fprintf(stderr, "qsa_decode_attn: %s\n", cudaGetErrorString(e)); - std::exit(1); - } + qsa_decode_attn_batch(q, pools, ids, step, cap, s, scratch, attn, 1, stream); } } // namespace strata::kernels diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 4638031bdb9..045e2f819ac 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -1048,8 +1048,8 @@ int test_ple_tokens(std::mt19937& rng, cudaStream_t s) { return bad; } -// ---- the prompt path's attention vs the decode kernel (INT8 and FP16 pools; 1 to 2051 selected cells) -int test_prefill_attn(std::mt19937& rng, cudaStream_t s) { +// ---- the decode attention (split, merged, gated) vs the prompt path's kernels (INT8 and FP16 pools; 1-2051 cells) +int test_decode_attn(std::mt19937& rng, cudaStream_t s) { const strata::kernels::QsaShapes sh = strata::kernels::qsa_real_shapes(); const int HD = (int) sh.head_dim, NH = (int) sh.n_head, NKV = (int) sh.n_head_kv; const int cells = 4096, pages = cells / (int) sh.page_size; @@ -1089,44 +1089,71 @@ int test_prefill_attn(std::mt19937& rng, cudaStream_t s) { std::copy(all.begin(), all.begin() + widths[(size_t) i], ids.begin() + (size_t) i * cap); steps[(size_t) i * strata::kernels::kStepCount + strata::kernels::kStepWidth] = widths[(size_t) i]; } - std::vector q((size_t) n_q * NH * HD); + std::vector q((size_t) n_q * NH * HD), qf((size_t) n_q * NH * 2 * HD); // qf: queries, then gate logits for (auto& x : q) x = 3.0f * std::normal_distribution(0.0f, 1.0f)(rng); + for (auto& x : qf) x = 2.0f * std::normal_distribution(0.0f, 1.0f)(rng); int32_t *d_ids = dev(ids.size()), *d_steps = dev(steps.size()); float *d_q = dev(q.size()), *d_a = dev(q.size()), *d_b = dev(q.size()); - float* d_scratch = dev((size_t) n_q * strata::kernels::qsa_decode_attn_scratch_floats(cap, sh)); - up(d_ids, ids); up(d_steps, steps); up(d_q, q); - int bad = 0; + float *d_qf = dev(qf.size()), *d_g = dev(q.size()), *d_c = dev(q.size()); + const uint64_t scr = strata::kernels::qsa_decode_attn_scratch_floats(cap, sh); + float* d_scratch = dev((size_t) n_q * scr); + up(d_ids, ids); up(d_steps, steps); up(d_q, q); up(d_qf, qf); + check(cudaDeviceSynchronize(), "attn inputs"); + const int64_t SC = strata::kernels::kStepCount; + int bad = 0, gate_bad = 0; double worst = 0.0; for (int int8 = 0; int8 < 2; ++int8) { strata::kernels::QsaAttnPools pools; pools.page_table = d_table; if (int8) { pools.k_q = d_kq; pools.v_q = d_vq; pools.k_scale = d_ks; pools.v_scale = d_vs; } else { pools.k_pool = d_kh; pools.v_pool = d_vh; } - strata::kernels::qsa_decode_attn_batch(d_q, pools, d_ids, d_steps, cap, sh, d_scratch, d_a, n_q, s); strata::kernels::qsa_prefill_attn(d_q, pools, d_ids, d_steps, cap, sh, d_b, n_q, s); - check(cudaStreamSynchronize(s), "prefill attn"); - const std::vector a = down(d_a, q.size()), b = down(d_b, q.size()); - for (int r = 0; r < n_q * NH; ++r) { // relative to the largest value of the head's output row - double big = 0.0, diff = 0.0; - for (int d = 0; d < HD; ++d) { - big = std::max(big, (double) std::fabs(a[(size_t) r * HD + d])); - diff = std::max(diff, (double) std::fabs(a[(size_t) r * HD + d] - b[(size_t) r * HD + d])); - } - const double rel = diff / std::max(big, 1e-30); - worst = std::max(worst, rel); - if (!(rel <= 2e-5)) { - if (bad < 5) std::fprintf(stderr, "prefill attn: %s pools, query %d head %d: relative difference %.3g\n", - int8 ? "INT8" : "FP16", r / NH, r % NH, rel); - ++bad; + for (int alone = 0; alone < 2; ++alone) { // the queries in one call, and each alone (the most splits) + if (alone) + for (int i = 0; i < n_q; ++i) + strata::kernels::qsa_decode_attn_batch(d_q + (size_t) i * NH * HD, pools, d_ids + (size_t) i * cap, + d_steps + i * SC, cap, sh, d_scratch, + d_a + (size_t) i * NH * HD, 1, s); + else strata::kernels::qsa_decode_attn_batch(d_q, pools, d_ids, d_steps, cap, sh, d_scratch, d_a, n_q, s); + check(cudaStreamSynchronize(s), "decode attn"); + const std::vector a = down(d_a, q.size()), b = down(d_b, q.size()); + for (int r = 0; r < n_q * NH; ++r) { // relative to the largest value of the head's output row + double big = 0.0, diff = 0.0; + for (int d = 0; d < HD; ++d) { + big = std::max(big, (double) std::fabs(a[(size_t) r * HD + d])); + diff = std::max(diff, (double) std::fabs(a[(size_t) r * HD + d] - b[(size_t) r * HD + d])); + } + const double rel = diff / std::max(big, 1e-30); + worst = std::max(worst, rel); + if (!(rel <= 2e-5)) { + if (bad < 5) + std::fprintf(stderr, "decode attn: %s pools, %s, query %d head %d: relative difference %.3g\n", + int8 ? "INT8" : "FP16", alone ? "alone" : "batched", r / NH, r % NH, rel); + ++bad; + } } } + // the gate in the merge against native_qsa_gate_apply on the ungated output + strata::kernels::qsa_decode_attn_batch(d_q, pools, d_ids, d_steps, cap, sh, d_scratch, d_a, n_q, s); + strata::kernels::native_qsa_gate_apply(d_a, d_qf, d_c, n_q * NH, HD, s); + strata::kernels::qsa_decode_attn_batch(d_q, pools, d_ids, d_steps, cap, sh, d_scratch, d_g, n_q, s, d_qf); + check(cudaStreamSynchronize(s), "gated attn"); + const std::vector g = down(d_g, q.size()), c = down(d_c, q.size()); + if (std::memcmp(g.data(), c.data(), g.size() * 4) != 0) { + std::fprintf(stderr, "decode attn: %s pools, the fused gate differs from native_qsa_gate_apply\n", + int8 ? "INT8" : "FP16"); + ++gate_bad; + } } for (void* p : {(void*) d_table, (void*) d_kq, (void*) d_vq, (void*) d_ks, (void*) d_vs, (void*) d_kh, (void*) d_vh, - (void*) d_ids, (void*) d_steps, (void*) d_q, (void*) d_a, (void*) d_b, (void*) d_scratch}) + (void*) d_ids, (void*) d_steps, (void*) d_q, (void*) d_a, (void*) d_b, (void*) d_qf, (void*) d_g, + (void*) d_c, (void*) d_scratch}) cudaFree(p); - std::printf("prefill_attn: %s (INT8 and FP16 pools, 1-2051 cells; largest relative difference %.2g)\n", - bad ? "OUTSIDE TOLERANCE" : "within 2e-5 of the decode kernel", worst); - return bad; + std::printf("decode_attn: %s, the gate %s (INT8 and FP16 pools, 1-2051 cells, 6 queries at once and each alone; " + "largest relative difference %.2g)\n", + bad ? "OUTSIDE TOLERANCE" : "within 2e-5 of the prompt path's kernels", + gate_bad ? "DIFFERS" : "bitwise native_qsa_gate_apply's", worst); + return bad + gate_bad; } // ---- the prompt path's GDN conv and recurrence over a chunk vs the verify window's kernels, 8 tokens at a time @@ -1231,7 +1258,7 @@ int main(int argc, char** argv) { bad += test_kv_append(rng, s); bad += test_indexer_append(rng, s); bad += test_ple_tokens(rng, s); - bad += test_prefill_attn(rng, s); + bad += test_decode_attn(rng, s); bad += test_prefill_gdn(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); From 37c1d81e7102cdd3b29d6c76f23f04387209a280 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Tue, 29 Sep 2026 03:51:41 +0200 Subject: [PATCH 73/92] Prompt lookup: windows of up to 8 tokens from the text's earlier occurrence of its last 16+ tokens --spec-lookup N: when the text's last N+ tokens occurred before (the prompt or the answer) and the tokens that followed there are more drafts than --spec gives the draft layer, the next verify window takes up to 7 of them and the draft layer's chain stops at its first draft; the verifier, the draft layer and the second GPU are built for 8-token windows then. SuffixDrafter keeps each trigram's latest position with a link from every position to its trigram's previous one, checks 16 earlier occurrences, and keeps its index across --serve requests: `assign` undoes the positions past the next request's common prefix, so proposals depend on the history alone. Its test builds with build.cmd. setup.py's configs take --spec-lookup 16. Fixed text, both GPUs, 6 runs each in both orders: a file returned with a rename 178.9 -> 225.0 t/s (IQ3_XXS) and 134.5 -> 166.0 (UD-Q4_K_XL), 7.5 tokens per pass instead of 3.8; thinking and after a 16K prompt unchanged. The output is the greedy output. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- CMakeLists.txt | 8 +- docs/DETAILS.md | 3 +- include/strata/core/mtp.hpp | 4 +- include/strata/spec/suffix_drafter.hpp | 26 ++++-- setup.py | 2 +- src/core/mtp.cpp | 3 +- src/program/generate.cpp | 108 ++++++++++++++++++++----- src/spec/suffix_drafter.cpp | 64 +++++++++++---- src/spec/suffix_drafter_test.cpp | 51 +++++++++++- 9 files changed, 216 insertions(+), 53 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 932e93a09ea..ebb4a6b847f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -256,7 +256,7 @@ if(STRATA_ENABLE_CUDA) add_executable(strata src/program/generate.cpp) target_include_directories(strata PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) - target_link_libraries(strata PRIVATE strata_engine strata_prefill CUDA::cudart) + target_link_libraries(strata PRIVATE strata_engine strata_prefill strata_spec CUDA::cudart) add_executable(dequant_s2_parity src/kernels/dequant_s2_parity.cpp) # The tests are HOST C++ files that call cudaMalloc/cudaMemcpy themselves, so they need the toolkit headers @@ -440,10 +440,10 @@ endif() # Plan v0.3 P6: speculation components that need no GPU. add_library(strata_spec STATIC src/spec/suffix_drafter.cpp src/spec/controller.cpp) target_include_directories(strata_spec PUBLIC include) +add_executable(suffix_drafter_test src/spec/suffix_drafter_test.cpp) +target_link_libraries(suffix_drafter_test PRIVATE strata_spec) +add_test(NAME suffix_drafter_test COMMAND suffix_drafter_test) if(STRATA_BUILD_TESTS) - add_executable(suffix_drafter_test src/spec/suffix_drafter_test.cpp) - target_link_libraries(suffix_drafter_test PRIVATE strata_spec) - add_test(NAME suffix_drafter_test COMMAND suffix_drafter_test) add_executable(controller_test src/spec/controller_test.cpp) target_link_libraries(controller_test PRIVATE strata_spec) add_test(NAME controller_test COMMAND controller_test) diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 92b5b60050c..488c4bfd0c2 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -351,7 +351,8 @@ test images. as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants). - **SSD:** the 28.8 GB n-gram table, read a few rows per token through the OS cache. - **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2 - tokens per pass on average. + tokens per pass on average. Where the answer repeats earlier text for 16+ tokens (a file returned with an edit), the + pass checks up to 7 tokens that followed it there instead. - **Prompts** are processed in 2,048-token chunks with the experts streamed to the GPU over PCIe. The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](paper/Strata-Paper.pdf)**. diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index 2c45a27d654..a44cad8a8c4 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -63,6 +63,8 @@ class MtpDrafter { int64_t window = 32768); /// The prompt's length: the prompt's K/V skip the cells the attention window can never reach again. void set_prompt_len(int64_t n) { prompt_len_ = n; } + /// The draft chain stops after `n` drafts (0: max_t - 1). + void set_max_drafts(int n) { max_drafts_ = n; } uint64_t vram_bytes() const { return vram_; } /// The layer's own K/V cells (the prompt cache stashes them with the model's). QsaState& kv_state() { return st_; } @@ -102,7 +104,7 @@ class MtpDrafter { const WeightTable* wt_ = nullptr; const NativeHead* head_ = nullptr; const float* window_R_ = nullptr; - int max_t_ = 0; + int max_t_ = 0, max_drafts_ = 0; int64_t n_vocab_ = 0; uint64_t vram_ = 0; cudaStream_t cs_ = nullptr; diff --git a/include/strata/spec/suffix_drafter.hpp b/include/strata/spec/suffix_drafter.hpp index 731bbef1b3b..c033832c2f9 100644 --- a/include/strata/spec/suffix_drafter.hpp +++ b/include/strata/spec/suffix_drafter.hpp @@ -4,10 +4,15 @@ // output quotes its input (edits, refactors, repeated code) the continuation is usually exact, and a verify pass // accepts many tokens at once. It needs no model and costs microseconds. // -// Index: every trigram of the history (prompt + accepted output) maps to its WAYS most recent end positions in a -// fixed-size open-addressing table, so memory is bounded (~20 bytes per history token) and appends are O(1). -// A proposal checks those candidates, extends each match backwards up to `max_match`, and takes the longest -// (most recent on ties). Matches shorter than `min_match` propose nothing. +// Index: an open-addressing table maps every trigram of the history (prompt + accepted output) to its most recent +// end position, and each position links to its trigram's previous one, so appends are O(1) (the table is rebuilt, +// larger if the history needs it, when half full). A proposal checks the trigram's WAYS most recent earlier +// occurrences, extends each match backwards up to `max_match`, and takes the longest (most recent on ties). +// Matches shorter than `min_match` propose nothing. +// +// `assign` replaces the history by another text that may share a prefix with it (a conversation's next request): +// the positions past the common prefix are undone newest first, which leaves the index as appending the prefix +// alone would, and the rest is appended. Proposals therefore depend on the history alone. #pragma once #include @@ -18,7 +23,7 @@ namespace strata::spec { class SuffixDrafter { public: - static constexpr int WAYS = 4; + static constexpr int WAYS = 16; // earlier occurrences a proposal checks (in code a trigram recurs often) explicit SuffixDrafter(int min_match = 3, int max_match = 32, size_t capacity_tokens = 1u << 19); @@ -26,6 +31,8 @@ class SuffixDrafter { /// Add tokens to the history (the prompt, then every accepted token). void append(const int32_t* tokens, size_t n); void append(int32_t token) { append(&token, 1); } + /// Make the history `tokens` (see above). + void assign(const int32_t* tokens, size_t n); /// Write up to `max_k` proposed next tokens to `out`; returns how many (0 = no match of at least min_match). int propose(int max_k, int32_t* out); @@ -36,17 +43,20 @@ class SuffixDrafter { private: struct Slot { - uint64_t key = 0; // trigram hash + 1 (0 = empty) - uint32_t pos[WAYS] = {}; // end positions, most recent first - uint8_t n = 0; + uint64_t key = 0; // trigram hash (0 = empty) + int32_t last = -1; // the trigram's most recent end position (-1: none left after `assign`) }; Slot* find_slot(uint64_t key, bool insert); uint64_t key_at(size_t end) const; + void link(size_t end); // index hist_[end], the newest token + void rebuild(); int min_match_, max_match_; std::vector hist_; + std::vector prev_; // per end position: its trigram's previous end position (-1: none) std::vector table_; size_t mask_ = 0; + size_t used_ = 0; // occupied slots, those of undone positions included int last_match_ = 0; }; diff --git a/setup.py b/setup.py index a17eb68c7a6..7bad77e7da8 100644 --- a/setup.py +++ b/setup.py @@ -814,7 +814,7 @@ def main() -> int: fail("the model has no per_layer_token_embd tensor (is this a Qwen3.8-Flash-Next GGUF?)") args = ["--pack", str(pack), "--native", str(shards[0]), "--ple-gguf", str(ple), "--expert-profile", str(ROOT / "data" / "expert-profile.bin"), "--expert-cache", "auto", - "--prefill", "2048", "--spec", "4", "--spec-min-p", "0.5", "--mtp", str(rt), + "--prefill", "2048", "--spec", "4", "--spec-min-p", "0.5", "--spec-lookup", "16", "--mtp", str(rt), "--max-context", str(ctx)] if ctx > 8192: args += ["--kv", "int8"] diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 460cd8b9b2e..13885772a8c 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -540,7 +540,8 @@ bool MtpDrafter::draft(int T, const int32_t* tokens, int64_t p, int a, int32_t* if (on_draft) on_draft(0, drafts[0], pj); int n = 1; // the chain continues while the last draft is likely enough to be verified - for (int j = 1; j < max_t_ - 1 && pj >= min_p; ++j) { + const int chain = max_drafts_ > 0 ? std::min(max_drafts_, max_t_ - 1) : max_t_ - 1; + for (int j = 1; j < chain && pj >= min_p; ++j) { if (!capture_step(j, err)) return false; put(max_t_ + j - 1, p + a + j); std::atomic_thread_fence(std::memory_order_seq_cst); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index e0db948ef0a..756614d35ce 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -40,12 +40,14 @@ #include "strata/kernels/native_rope.hpp" #include "strata/kernels/mrope.hpp" #include "strata/kernels/qsa.hpp" +#include "strata/kernels/verify_kernels.hpp" #include "strata/core/native_head.hpp" #include "strata/core/verify.hpp" #include "strata/core/mtp.hpp" #include "strata/core/prompt_cache.hpp" #include "strata/prefill/experts.hpp" #include "strata/prefill/prefill.hpp" +#include "strata/spec/suffix_drafter.hpp" #include "strata/core/native_dense.hpp" #include "strata/program/logits_selection.hpp" @@ -248,6 +250,12 @@ struct Options { /// Plan v0.3 P6: a draft enters the verify window only while every draft before it (and itself) has at least /// this probability under the draft layer; 0 = always --spec-1 drafts. double spec_min_p = 0.0; + /// Prompt lookup: when the text's last N+ tokens occurred before and the tokens after that occurrence are more + /// drafts than --spec gives the draft layer, the next window verifies up to kVerifyMaxT - 1 of them instead; + /// 0 = off. + int spec_lookup = 0; + /// The verify windows' capacity: --spec, or the kernels' maximum with prompt lookup. + int max_window() const { return spec_lookup > 0 ? std::max(spec, strata::kernels::kVerifyMaxT) : spec; } /// Stop when the model emits an end-of-turn token (<|endoftext|> 248044, <|im_end|> 248046, or --eos-ids). bool stop_eos = false; std::vector eos_ids = {248044, 248046}; @@ -346,6 +354,9 @@ void usage() { " --spec T verify windows of up to T tokens: the last one and T-1 MTP drafts (2..8)\n" " --spec-min-p P a draft enters the window only while it and the drafts before it have at\n" " least probability P under the draft layer (default 0: always T-1 drafts)\n" + " --spec-lookup N when the text's last N+ tokens occurred before, the next window verifies\n" + " up to 7 of the tokens that followed there instead, if they are more than\n" + " --spec allows the draft layer (windows of up to 8 tokens; 0 = off)\n" " --mtp DIR the MTP draft layer's runtime files (tools/mtp_rt.py)\n" " --mtp-window N the draft layer attends to the last N cells (default 32768; 0 = every cell)\n" " --main-gpu N run on CUDA device N of the visible ones, in nvidia-smi's order unless\n" @@ -808,6 +819,7 @@ int main(int argc, char** argv) { else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac")); else if (a == "--adapt-every") o.adapt_every = std::atoi(next("--adapt-every")); else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); + else if (a == "--spec-lookup") o.spec_lookup = std::max(0, std::atoi(next("--spec-lookup"))); else if (a == "--stop-eos") o.stop_eos = true; else if (a == "--spec-split") o.spec_split = true; else if (a == "--window-profile") o.window_profile = true; @@ -1308,7 +1320,11 @@ int main(int argc, char** argv) { o.mtp.clear(); } if (!o.mtp.empty()) mtp.set_prompt_len((int64_t) o.tokens.size()); - if (!o.mtp.empty() && !mtp.load(o.mtp, g, ss, o.spec, err, o.mtp_window)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } + if (!o.mtp.empty() && !mtp.load(o.mtp, g, ss, o.max_window(), err, o.mtp_window)) { + std::fprintf(stderr, "strata generate: %s\n", err.c_str()); + return 1; + } + mtp.set_max_drafts(o.spec - 1); } // --ple-io ram: the table goes into RAM once the SSD has delivered the experts and the draft layer if (ple_table.is_open() && !ple_table.start_ram_load(err)) @@ -2119,7 +2135,7 @@ int main(int argc, char** argv) { for (int64_t l = 0; l < g.n_layers; ++l) min_blob = std::min(min_blob, lay0.blob_bytes(l)); const uint64_t pf_bytes = (uint64_t) (std::max(0.0, o.second_gpu_prefetch_mb) * 1048576.0); drive.d.gpu2_prefetch_bytes = pf_bytes; - bool ok = gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.spec, ss.k, err) && + bool ok = gpu2.init(o.second_gpu, main_dev, g.n_embd, g.n_ff, o.max_window(), ss.k, err) && gpu2.init_prefetch((int) std::min(pf_bytes / min_blob, 64), lay0.max_blob, err); const auto& lay = strata::kernels::cpu::expert_layout(); std::vector sizes; @@ -2333,7 +2349,7 @@ int main(int argc, char** argv) { vh.slot_off = xcache.slot_offsets(); vh.slots = xcache.slots(); vh.blob = thits.blob; - if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.spec, err) || + if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.max_window(), err) || !mtp.bind(wt, &native_head, ver.final_R_all(), err)) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); return 1; @@ -2424,7 +2440,10 @@ int main(int argc, char** argv) { std::fflush(stdout); std::string line; int64_t rounds = 0; - const int S = o.spec; + const int S = o.spec, W = o.max_window(); // the draft layer's windows; with prompt lookup's + // prompt lookup: the text so far, kept across requests (a conversation's next request shares its prefix) + strata::spec::SuffixDrafter lookup(o.spec_lookup, 32, o.spec_lookup > 0 ? (size_t) o.max_context + 16 : 16); + std::vector ids32, ldrafts((size_t) W, 0); constexpr int64_t kImStart = 248045; // <|im_start|>: a new turn begins here // The vision path (--vision): GENI carries images. The file is one // or more strata-vision records (int32 'SVE1', n, nx, ny, n_embd, then n x n_embd floats) in prompt order; @@ -2551,8 +2570,8 @@ int main(int argc, char** argv) { const int64_t reuse = pcache.begin(ids, !geni); tr("request", n, reuse); mtp.set_prompt_len(n); - std::vector drafts((size_t) S, 0), window((size_t) S), outv((size_t) S); - std::vector dprob((size_t) S, 0.0f); + std::vector drafts((size_t) W, 0), window((size_t) W), outv((size_t) W); + std::vector dprob((size_t) W, 0.0f); // ---- the rest of the prompt, [reuse, n - 1). A turn usually shares everything before its prompt's // last <|im_start|> with the next one (chat templates re-render the answer, e.g. without its // reasoning), so that position gets a checkpoint, and so do the prompt's end (a regenerated answer) @@ -2635,8 +2654,14 @@ int main(int argc, char** argv) { int64_t produced_n = 0; const char* finish = "length"; const Clock::time_point d0 = Clock::now(); + int lk = 0; // the next window's drafts from the lookup (0: the draft layer's) + if (o.spec_lookup > 0) { + ids32.resize((size_t) n); + for (int64_t i = 0; i < n; ++i) ids32[(size_t) i] = (int32_t) ids[(size_t) i]; + lookup.assign(ids32.data(), ids32.size()); + } mtp.on_draft = [&](int j, int32_t tok, float prob) { // the drafts the next window will verify - if (prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); + if (lk == 0 && prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); }; while (produced_n < max_new) { if (stop_req.load()) { finish = "cancel"; break; } // the client went away @@ -2645,19 +2670,22 @@ int main(int argc, char** argv) { T = 1; while (T < S && dprob[(size_t) T - 1] >= (float) o.spec_min_p) ++T; } + if (lk > 0) T = 1 + lk; if (first_window) T = 1; + const std::vector& dr = lk > 0 ? ldrafts : drafts; // Never verify past what this request emits: the committed cells then hold exactly the tokens the // client saw, so its next turn continues from them. So the window stops at the output limit and // at a drafted end-of-turn token. T = (int) std::min(T, max_new - produced_n); for (int i = 1; i < T; ++i) - if (std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) drafts[(size_t) i - 1]) != o.eos_ids.end()) { + if (std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) dr[(size_t) i - 1]) != + o.eos_ids.end()) { T = i + 1; break; } if (p + T > o.max_context) break; window[0] = x; - for (int i = 1; i < T; ++i) window[(size_t) i] = drafts[(size_t) i - 1]; + for (int i = 1; i < T; ++i) window[(size_t) i] = dr[(size_t) i - 1]; drive.d.layers = 0; drive.d.experts = 0; drive.d.failed = false; @@ -2685,11 +2713,21 @@ int main(int argc, char** argv) { bool eos = false; for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) { std::printf("T %d\n", (int) outv[(size_t) i]); + if (o.spec_lookup > 0) lookup.append(outv[(size_t) i]); ++produced_n; eos = std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) outv[(size_t) i]) != o.eos_ids.end(); } std::fflush(stdout); ++rounds; + // the next window's drafts: the lookup's when they are more than the draft layer's chain allows (the + // chain stops at its first draft then) + lk = 0; + if (o.spec_lookup > 0 && !eos && produced_n < max_new) { + lk = lookup.propose(W - 1, ldrafts.data()); + if (lk < S) lk = 0; + for (int j = 0; j < lk; ++j) ver.ple_ahead(j + 1, ldrafts[(size_t) j]); + } + mtp.set_max_drafts(lk > 0 ? 1 : S - 1); const bool drafted = (eos || produced_n >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && ver.wait_commit(err); @@ -2706,6 +2744,7 @@ int main(int argc, char** argv) { p += a + 1; } mtp.on_draft = nullptr; + mtp.set_max_drafts(S - 1); const double decode_ms = std::chrono::duration(Clock::now() - d0).count(); std::printf("DONE %lld %lld %.1f %.1f %s %lld\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms, finish, (long long) reuse); @@ -3044,7 +3083,7 @@ int main(int argc, char** argv) { vh.slot_off = xcache.slot_offsets(); vh.slots = xcache.slots(); vh.blob = thits.blob; - if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.spec, err)) { + if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.max_window(), err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -3079,9 +3118,10 @@ int main(int argc, char** argv) { if (o.adapt_every > 0 && o.adapt_swaps > 0) drive.d.usage.assign((size_t) (g.n_layers * g.n_expert), 0.0f); int64_t p = spec_pos; int32_t x = (int32_t) tok; - std::vector drafts((size_t) o.spec, 0); - std::vector dprob((size_t) o.spec, 1.0f); - std::vector window_hist((size_t) o.spec + 1, 0); + const int W = o.max_window(); + std::vector drafts((size_t) W, 0); + std::vector dprob((size_t) W, 1.0f); + std::vector window_hist((size_t) W + 1, 0); // plan v0.3 P6: with a native pack the first window is the last prompt token alone (it produces the first // generated token and the MTP's first cell); otherwise the token loop already did that. bool first_window = native_pack; @@ -3090,8 +3130,8 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } - std::vector window((size_t) o.spec), outv((size_t) o.spec); - std::vector fdiff((size_t) o.spec, 0); + std::vector window((size_t) W), outv((size_t) W); + std::vector fdiff((size_t) W, 0); std::FILE* hashes = o.window_hashes.empty() ? nullptr : std::fopen(o.window_hashes.c_str(), "w"); if (!o.window_hashes.empty() && hashes == nullptr) { std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_hashes.c_str()); @@ -3102,7 +3142,7 @@ int main(int argc, char** argv) { std::fprintf(stderr, "strata generate: cannot write %s\n", o.window_logits.c_str()); return 1; } - std::vector wl(wlog != nullptr ? (size_t) o.spec * (size_t) n_vocab : 0); + std::vector wl(wlog != nullptr ? (size_t) W * (size_t) n_vocab : 0); std::vector wl_ids(wlog != nullptr ? (size_t) n_vocab : 0); auto write_logits = [&](int64_t k, int32_t emitted, const float* l) { // generated token k's line constexpr int K = 64; @@ -3118,15 +3158,25 @@ int main(int argc, char** argv) { std::fprintf(wlog, " %d:%.6f", (int) wl_ids[(size_t) j], (double) l[wl_ids[(size_t) j]] - lse); std::fprintf(wlog, "\n"); }; - std::vector accepted_hist((size_t) o.spec, 0); + std::vector accepted_hist((size_t) W, 0); int64_t rounds = 0, drafts_total = 0, drafts_ok = 0, corrupt_counter = 0; + // prompt lookup: the text so far, the prompt and then each emitted token + strata::spec::SuffixDrafter lookup(o.spec_lookup, 32, + o.spec_lookup > 0 ? (size_t) (n_prompt + max_new) + 16 : 16); + if (o.spec_lookup > 0) { + for (int64_t t : o.tokens) lookup.append((int32_t) t); + for (int64_t t : produced) lookup.append((int32_t) t); + } + std::vector ldrafts((size_t) W, 0); + int lk = 0; // the next window's drafts from the lookup (0: the draft layer's) + int64_t lookup_rounds = 0, lookup_drafts = 0, lookup_ok = 0; const double pool_ms0 = drive.cpu_ms; const int64_t misses0 = drive.d.multi_misses, entries0 = drive.d.multi_entries; const uint32_t sleeps0 = pool.sleeps(); if (o.stats) drive.d.routed.assign((size_t) (g.n_layers * g.n_expert), 0); if (use_mtp) mtp.on_draft = [&](int j, int32_t tok, float prob) { // the drafts the next window will verify - if (prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); + if (lk == 0 && prob >= (float) o.spec_min_p) ver.ple_ahead(j + 1, tok); }; while ((int64_t) produced.size() < max_new) { const Clock::time_point t0 = Clock::now(); @@ -3135,6 +3185,7 @@ int main(int argc, char** argv) { T = 1; while (T < o.spec && dprob[(size_t) T - 1] >= (float) o.spec_min_p) ++T; } + if (lk > 0) T = 1 + lk; if (first_window) T = 1; ++window_hist[(size_t) T]; if (p + T > o.max_context) { @@ -3144,7 +3195,8 @@ int main(int argc, char** argv) { window[0] = x; for (int i = 1; i < T; ++i) { const size_t at = produced.size() - 1 + (size_t) i; - int32_t d = use_mtp ? drafts[(size_t) i - 1] : at < oracle.size() ? (int32_t) oracle[at] : 0; + int32_t d = lk > 0 ? ldrafts[(size_t) i - 1] : use_mtp ? drafts[(size_t) i - 1] + : at < oracle.size() ? (int32_t) oracle[at] : 0; if (o.spec_corrupt > 0 && (++corrupt_counter % o.spec_corrupt) == 0) d = (d + 1) % (int32_t) n_vocab; window[(size_t) i] = d; } @@ -3189,6 +3241,11 @@ int main(int argc, char** argv) { } int a = 0; while (a < T - 1 && window[(size_t) a + 1] == outv[(size_t) a]) ++a; + if (lk > 0) { + ++lookup_rounds; + lookup_drafts += lk; + lookup_ok += a; + } if (first_window) { first_window = false; ttft_ms = std::chrono::duration(Clock::now() - t_start).count(); @@ -3211,6 +3268,7 @@ int main(int argc, char** argv) { if (wlog != nullptr) write_logits((int64_t) produced.size(), outv[(size_t) i], wl.data() + (size_t) i * (size_t) n_vocab); produced.push_back(outv[(size_t) i]); + if (o.spec_lookup > 0) lookup.append(outv[(size_t) i]); eos = o.stop_eos && std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) outv[(size_t) i]) != o.eos_ids.end(); if (!follow.empty()) { follow_differ += fdiff[(size_t) i]; ++follow_emitted; } } @@ -3223,6 +3281,15 @@ int main(int argc, char** argv) { total_ms += std::chrono::duration(Clock::now() - t0).count(); break; } + // the next window's drafts: the lookup's when they are more than the draft layer's chain allows (the + // chain stops at its first draft then) + lk = 0; + if (o.spec_lookup > 0 && (int64_t) produced.size() < max_new) { + lk = lookup.propose(W - 1, ldrafts.data()); + if (lk < o.spec) lk = 0; + for (int j = 0; j < lk; ++j) ver.ple_ahead(j + 1, ldrafts[(size_t) j]); + } + mtp.set_max_drafts(lk > 0 ? 1 : o.spec - 1); const bool drafted = (!use_mtp || (int64_t) produced.size() >= max_new || mtp.draft(T, outv.data(), p, a, drafts.data(), err, dprob.data(), (float) o.spec_min_p)) && ver.wait_commit(err); @@ -3258,6 +3325,9 @@ int main(int argc, char** argv) { std::printf("%-24s", "accepted per round"); for (size_t i = 0; i < accepted_hist.size(); ++i) std::printf(" %zu:%lld", i, (long long) accepted_hist[i]); std::printf("\n"); + if (o.spec_lookup > 0) + std::printf("%-24s %lld windows of its drafts, %lld of %lld accepted\n", "prompt lookup", + (long long) lookup_rounds, (long long) lookup_ok, (long long) lookup_drafts); if (rounds > 0) std::printf("%-24s wait for rings %.3f pool %.3f host %.3f commit %.3f ms/round; CPU experts %.2f " "distinct / %.2f routed per layer\n", diff --git a/src/spec/suffix_drafter.cpp b/src/spec/suffix_drafter.cpp index 6f04f76d595..12611797ff1 100644 --- a/src/spec/suffix_drafter.cpp +++ b/src/spec/suffix_drafter.cpp @@ -17,16 +17,19 @@ uint64_t mix(uint64_t x) { SuffixDrafter::SuffixDrafter(int min_match, int max_match, size_t capacity_tokens) : min_match_(std::max(3, min_match)), max_match_(std::max(min_match, max_match)) { - size_t cap = 1; + size_t cap = 16; while (cap < capacity_tokens * 2) cap <<= 1; // load factor <= 0.5 at the nominal capacity table_.assign(cap, Slot{}); mask_ = cap - 1; hist_.reserve(capacity_tokens); + prev_.reserve(capacity_tokens); } void SuffixDrafter::reset() { hist_.clear(); + prev_.clear(); std::fill(table_.begin(), table_.end(), Slot{}); + used_ = 0; last_match_ = 0; } @@ -36,46 +39,73 @@ uint64_t SuffixDrafter::key_at(size_t end) const { } SuffixDrafter::Slot* SuffixDrafter::find_slot(uint64_t key, bool insert) { - for (size_t i = key & mask_, probes = 0; probes <= mask_; i = (i + 1) & mask_, ++probes) { + for (size_t i = key & mask_;; i = (i + 1) & mask_) { // never full: rebuilt at half load Slot& s = table_[i]; if (s.key == key) return &s; if (s.key == 0) { if (!insert) return nullptr; s.key = key; + ++used_; return &s; } } - return nullptr; // table full: the history outgrew its capacity +} + +void SuffixDrafter::link(size_t end) { + int32_t before = -1; + if (end >= 2) { + Slot* s = find_slot(key_at(end), true); + before = s->last; + s->last = (int32_t) end; + } + prev_.push_back(before); +} + +// The indexed positions again, without the slots of undone ones, in a table that holds them at a quarter load. +void SuffixDrafter::rebuild() { + const size_t n = prev_.size(); + size_t cap = table_.size(); + while (cap < 4 * n) cap <<= 1; + table_.assign(cap, Slot{}); + mask_ = cap - 1; + used_ = 0; + prev_.clear(); + for (size_t e = 0; e < n; ++e) link(e); } void SuffixDrafter::append(const int32_t* tokens, size_t n) { for (size_t i = 0; i < n; ++i) { + if (2 * (used_ + 1) > table_.size()) rebuild(); hist_.push_back(tokens[i]); - const size_t end = hist_.size() - 1; - if (end < 2) continue; - Slot* s = find_slot(key_at(end), true); - if (s == nullptr) continue; - for (int w = WAYS - 1; w > 0; --w) s->pos[w] = s->pos[w - 1]; - s->pos[0] = (uint32_t) end; - if (s->n < WAYS) ++s->n; + link(hist_.size() - 1); } } +void SuffixDrafter::assign(const int32_t* tokens, size_t n) { + size_t keep = 0; + const size_t common = std::min(n, hist_.size()); + while (keep < common && hist_[keep] == tokens[keep]) ++keep; + for (size_t e = hist_.size(); e-- > std::max(keep, 2);) // newest first: each trigram's last goes back + if (Slot* s = find_slot(key_at(e), false)) s->last = prev_[e]; + hist_.resize(keep); + prev_.resize(keep); + last_match_ = 0; + append(tokens + keep, n - keep); +} + int SuffixDrafter::propose(int max_k, int32_t* out) { last_match_ = 0; const size_t n = hist_.size(); if (n < 4 || max_k <= 0) return 0; const size_t cur = n - 1; - const Slot* s = find_slot(key_at(cur), false); - if (s == nullptr) return 0; size_t best_end = 0; int best_len = 0; - for (int w = 0; w < s->n; ++w) { - const size_t p = s->pos[w]; - if (p >= cur) continue; // the current suffix itself + int32_t p = prev_[cur]; // the current trigram's earlier occurrences, newest first + for (int w = 0; w < WAYS && p >= 0; ++w, p = prev_[(size_t) p]) { int len = 0; - while (len < max_match_ && len <= (int) p && hist_[p - len] == hist_[cur - len]) ++len; - if (len > best_len) { best_len = len; best_end = p; } // most recent first, so ties keep the newer + while (len < max_match_ && len <= p && hist_[(size_t) (p - len)] == hist_[cur - len]) ++len; + if (len > best_len) { best_len = len; best_end = (size_t) p; } // most recent first, so ties keep the newer + if (best_len == max_match_) break; } if (best_len < min_match_) return 0; last_match_ = best_len; diff --git a/src/spec/suffix_drafter_test.cpp b/src/spec/suffix_drafter_test.cpp index 9fa8611e7c8..c67723a9642 100644 --- a/src/spec/suffix_drafter_test.cpp +++ b/src/spec/suffix_drafter_test.cpp @@ -12,6 +12,7 @@ // No model runs; this measures the drafter on text, not on the model's own outputs. #include "strata/spec/suffix_drafter.hpp" +#include #include #include #include @@ -103,7 +104,7 @@ void unit_tests() { const int n = d.propose(6, out); check(n >= 3 && out[0] == 1 && out[1] == 2 && out[2] == 3, "periodic continuation"); } - { // bounded memory: appending beyond the nominal capacity does not crash and still works + { // beyond the nominal capacity the table grows SuffixDrafter d(3, 32, 1024); std::vector doc; for (int i = 0; i < 5000; ++i) doc.push_back(i % 997); @@ -111,6 +112,54 @@ void unit_tests() { int32_t out[4]; check(d.propose(4, out) > 0, "propose after overflow of nominal capacity"); } + { // assign: the shared prefix stays, the rest of the old text proposes nothing + SuffixDrafter d; + const std::vector a = {1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3}; + d.append(a.data(), a.size()); + int32_t out[4]; + check(d.propose(4, out) > 0 && out[0] == 4, "before assign"); + const std::vector b = {1, 2, 3, 40, 50, 60, 70, 1, 2, 3}; + d.assign(b.data(), b.size()); + check(d.size() == b.size(), "assign replaces the history"); + const int n = d.propose(4, out); + check(n == 4 && out[0] == 40 && out[3] == 70, "the new text's continuation, not the old one's"); + d.assign(a.data(), a.size()); + check(d.propose(4, out) == 4 && out[0] == 4 && out[3] == 7, "the old text again"); + } + { // many texts: the slots of undone positions are dropped when the table is rebuilt + SuffixDrafter d(3, 32, 256); + std::vector doc(200); + for (int r = 0; r < 50; ++r) { + for (int i = 0; i < 200; ++i) doc[(size_t) i] = r * 1000 + i % 150; + d.assign(doc.data(), doc.size()); + } + int32_t out[4]; + check(d.propose(4, out) == 4 && out[0] == 49000 + 50, "propose after many texts"); + } + { // proposals depend on the history alone: after assign undid a longer text and the table grew, as a fresh + // drafter's (a 6-token alphabet: every trigram recurs, more often than WAYS) + uint32_t rng = 12345; + auto next = [&] { rng = rng * 1664525u + 1013904223u; return (int32_t) ((rng >> 16) % 6); }; + std::vector a(3000), b; + for (int32_t& t : a) t = next(); + b.assign(a.begin(), a.begin() + 1700); + for (int i = 0; i < 1300; ++i) b.push_back(next()); + SuffixDrafter x(3, 32, 16), y(3, 32, 1u << 14); + x.append(a.data(), a.size()); + x.assign(b.data(), 1000); + x.append(b.data() + 1000, b.size() - 1000); + y.append(b.data(), b.size()); + bool same = x.size() == y.size(); + int32_t ox[8], oy[8]; + for (int i = 0; i < 300 && same; ++i) { + const int nx = x.propose(8, ox), ny = y.propose(8, oy); + same = nx == ny && x.last_match() == y.last_match() && std::equal(ox, ox + nx, oy); + const int32_t t = next(); + x.append(t); + y.append(t); + } + check(same, "proposals after assign and growth are a fresh drafter's"); + } std::printf("suffix_drafter unit tests: %s\n", g_fail ? "FAILED" : "OK"); } } // namespace From 4404541c485395fa26eddd29a2a8ef8ae18428a4 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Tue, 29 Sep 2026 06:20:24 +0200 Subject: [PATCH 74/92] Prompt parts through verify windows as wide as the verifier's (8 tokens with --spec-lookup) --serve's feed took the draft layer's 4-token windows for a turn's new tokens; with --spec-lookup the verifier holds 8, and an 8-token window costs 1.6-1.8x a 4-token one. The feed's drafts are never used, so the draft layer's chain stops at its first draft there. Follow-up turns of 34-272 new tokens (bench/feed_test.py, both GPUs, 4 runs each): 13-16% less prompt time from 50 tokens up on both models (IQ3_XXS 64 tokens 479 -> 408 ms, UD-Q4_K_XL 586 from 679). Against the batched prompt path the break-even moves from ~84 to ~113 tokens (IQ3_XXS) and from ~60 to ~73 (UD-Q4_K_XL); on the 3090 alone ~400. cache_test.py (--adapt-every 0): the regenerated turn and the stash round trip reproduce their output, both models, and a second process too. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- src/program/generate.cpp | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 756614d35ce..333930b29d6 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -277,9 +277,9 @@ struct Options { int64_t cache_every = 16384; /// Prompt parts of up to this many tokens go through verify windows instead of the batched prompt path: a new /// part that short, and a last prompt chunk that short. A batched call has a fixed cost (it lends and refills - /// expert-cache slots and reads every expert its tokens use): ~0.6 s with a second GPU, ~1.5 s with IQ3_XXS on - /// the 3090 alone; verify windows cost 10-15 ms per token. They break even at ~50-65 tokens with a second GPU, - /// ~185 without (bench/feed_test.py). + /// expert-cache slots and reads every expert its tokens use): ~0.55 s with a second GPU, ~1.6 s with IQ3_XXS on + /// the 3090 alone; verify windows of 8 tokens (--spec-lookup) cost 5-8 ms per token. They break even at ~73 + /// (UD-Q4_K_XL) and ~113 (IQ3_XXS) tokens with a second GPU, ~400 without (bench/feed_test.py). int64_t feed_max = 128; int adapt_swaps = 96; }; @@ -2614,11 +2614,12 @@ int main(int argc, char** argv) { } tr("prompt done (slots refilled)"); } - // prompt tokens through verify windows: each window is committed whole, and the draft layer catches up - // on the same cells (its drafts are not used) + // prompt tokens through verify windows as wide as the verifier's: each window is committed whole, and the + // draft layer catches up on the same cells (its drafts are not used, so its chain stops at the first) auto feed = [&](int64_t from, int64_t to) -> bool { + mtp.set_max_drafts(1); for (int64_t j = from; j < to;) { - const int T = (int) std::min(S, to - j); + const int T = (int) std::min(W, to - j); for (int i = 0; i < T; ++i) window[(size_t) i] = (int32_t) ids[(size_t) (j + i)]; drive.d.layers = 0; drive.d.experts = 0; @@ -2744,7 +2745,6 @@ int main(int argc, char** argv) { p += a + 1; } mtp.on_draft = nullptr; - mtp.set_max_drafts(S - 1); const double decode_ms = std::chrono::duration(Clock::now() - d0).count(); std::printf("DONE %lld %lld %.1f %.1f %s %lld\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms, finish, (long long) reuse); From 7105fccbe3ec8c0bbb85f93973c91624505074be Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Tue, 29 Sep 2026 10:54:14 +0200 Subject: [PATCH 75/92] The 3090 tier's copies in pieces while its link carries nothing else An update of the 3090's adaptive tier copied its moves at once (UD-Q4_K_XL 142-301 MB, 5-50 ms at 6 GB/s over the x4 link), and a copy in flight there slowed every transfer of the verify window over that link: the router kernel's rows and doorbell to the host 19 -> 40 us a layer at 1 token, the combine's read of the CPU's rows 6 -> 9 us, the wait for the CPU's rows +20-170 us (CUPTI, UD thinking); 2-5% of a fixed-text run. Now the moves go out in pieces: once a layer's CPU rows are in (Verifier::set_rows_in, after the flag and the prediction), as much as lands before the next layer's router writes to the host (that layer's running average of the time from the previous layer's CPU rows to its ring, by window size, less 80 us). AdaptiveTier stages a window's moves before it (their residents evicted before the window's residency snapshot), sends a blob in as many pieces as it takes, and admits a move once its batch has landed (a ring of events); an update drops the queued moves not staged yet and skips the experts still under way. The 3090 tier counts an expert the 5070 Ti holds at half its use (moving it only shifts work between the GPUs), and the 5070 Ti tier yields only the experts the 3090 tier has staged. The tier keeps the host residency table only: a synchronous copy of its device table at every admission waited behind the pieces. Without a second GPU the pieces cannot carry what the tier needs, so a window's start sends all the queued moves there and an update waits for the previous one's, as before. Fixed text, both GPUs, 6 runs of each build in both orders, t/s: IQ3_XXS code / thinking / after 16K / edit +0.2 / +1.6 / +4.5 / -2.2%, UD-Q4_K_XL +2.3 / +7.7 / +6.7 / -0.1%. The pieces carry less than the updates asked for (UD code 29 MB a round against 86): the 3090's hits rose on thinking text and fell on code and edits. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/adaptive_tier.hpp | 94 +++++++----- include/strata/core/verify.hpp | 8 + src/core/adaptive_tier.cpp | 203 ++++++++++++++++---------- src/core/verify.cpp | 1 + src/program/generate.cpp | 116 +++++++++++---- 5 files changed, 287 insertions(+), 135 deletions(-) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp index c69b28b91c2..77ae932c96e 100644 --- a/include/strata/core/adaptive_tier.hpp +++ b/include/strata/core/adaptive_tier.hpp @@ -2,9 +2,9 @@ // // Every few rounds the missing experts routed most often (decayed counts, at least twice) move into the tier: into // their layer's empty slots first, then into the slots of the layer's least-routed residents when they were routed -// clearly more often. The copies run on their own stream between rounds; an evicted expert is a miss at once, a new -// one resident once its copy has landed (`apply_pending`). Both decode loops (--serve and the speculative loop) use -// it, for the first GPU's cache and for a second GPU's. +// clearly more often. The copies run on their own stream while the rounds go on; an evicted expert is a miss at once, +// a new one resident once its copy has landed (`apply_pending`). Both decode loops (--serve and the speculative loop) +// use it, for the first GPU's cache and for a second GPU's. #pragma once #include "strata/core/expert_cache.hpp" @@ -26,11 +26,11 @@ class AdaptiveTier { AdaptiveTier(const AdaptiveTier&) = delete; AdaptiveTier& operator=(const AdaptiveTier&) = delete; - /// `host_res` is the residency table (n_layers x n_expert, slot or kNotResident) the dispatch reads, `d_res` - /// its device copy (may be null); up to `max_moves` experts move per call. `device`: the cache's GPU (its - /// stream and copies), `main_device` current again afterwards. - bool init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, int64_t n_layers, - int64_t n_expert, int max_moves, std::string& err, int device = -1, int main_device = 0); + /// `host_res` is the residency table (n_layers x n_expert, slot or kNotResident) the dispatch reads (the verify + /// windows take a snapshot of it; a device copy of it is its owner's to refresh); up to `max_moves` experts move + /// per call. `device`: the cache's GPU (its stream and copies), `main_device` current again afterwards. + bool init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int64_t n_layers, int64_t n_expert, + int max_moves, std::string& err, int device = -1, int main_device = 0); bool on() const { return res_ != nullptr; } /// An empty slot sized for `layer`'s experts. void add_free(int64_t layer, int32_t slot) { free_[(size_t) layer].push_back(slot); } @@ -38,53 +38,79 @@ class AdaptiveTier { /// A tier in front of this one: the experts it holds or is loading are not candidates here, and this tier's /// copies of them are evicted first. void set_upper(const AdaptiveTier* upper) { upper_ = upper; } - /// The copies wait for `ev` (on the cache's GPU): the last work that may read the slots they overwrite. - void set_after(cudaEvent_t ev) { after_ = ev; } + /// A tier behind this one: a candidate it holds counts half its use here (moving it only shifts work from that + /// tier's GPU to this one's). + void set_lower(const AdaptiveTier* lower) { lower_ = lower; } + /// An update waits while the previous one's moves are under way, instead of dropping the ones not staged yet (a + /// tier whose moves all go out before the next window). + void set_wait(bool on) { wait_ = on; } - /// Ranks and submits this call's moves, then decays `usage` (n_layers x n_expert) unless `decay` is false (a - /// tier ranked before another on the same counts). Nothing to do while the previous moves are in flight. - /// False when a copy could not be submitted. + /// Ranks and queues this call's moves, then decays `usage` (n_layers x n_expert) unless `decay` is false (a tier + /// ranked before another on the same counts). The queued moves whose residents are still in place give way to + /// this call's (ranked again if still worth it); the experts still under way are not candidates. bool adapt(std::vector& usage, std::string& err, bool decay = true); - /// Paced: adapt() only queues its moves, and `pump` submits them in order while they fit `budget` bytes, `sent` - /// the bytes submitted: the copies go where the RAM and the copy engine are idle. The next adapt() drops the - /// moves not submitted by then and admits the landed ones before it ranks. `copy_rate()`: a running average of - /// the batches' measured rates (bytes per ms), 0 until the first has landed. - bool set_paced(); + + /// By default the copies go out in pieces (the first GPU's tier: its copies cross the link that the verify + /// window's own transfers need, and slowed them while they ran). `stage` evicts the next queued moves' residents + /// until `bytes` are staged and not sent: called before a window, whose residency snapshot then leaves their slots + /// alone. `pump_bytes` sends the staged moves' next `bytes`, a blob in as many calls as it takes. + void stage(uint64_t bytes); + bool pump_bytes(uint64_t bytes, std::string& err); + + /// Paced (the second GPU's tier, its copies between windows): whole moves, each resident evicted as its copy goes + /// out. `pump` submits them in order while they fit `budget` bytes, `sent` the bytes submitted, after `ev` (on + /// the cache's GPU: the last work that may read the slots they overwrite). + void set_paced(cudaEvent_t ev) { paced_ = true; after_ = ev; } bool pump(uint64_t budget, uint64_t& sent, std::string& err); + + /// A running average of the copies' measured rate (bytes per ms), 0 until the first timed ones have landed. double copy_rate() const { return rate_; } - /// Admits the landed moves into `host_res` (and `d_res`) once all are submitted; `wait` submits the queued ones - /// and blocks until they have landed. + /// Admits the moves whose copies have landed into `host_res`; `wait` sends the queued ones and + /// blocks until they have landed. void apply_pending(bool wait); int64_t swaps = 0, fills = 0; ///< experts moved into an occupied / an empty slot - int64_t dropped = 0; ///< paced: moves the next update dropped before their copies went out + int64_t dropped = 0; ///< moves the next update dropped before their residents left + uint64_t sent_bytes = 0; ///< bytes copied double ms = 0; ///< host time in `adapt` private: + struct Move { int32_t layer, in, out, slot; }; // out: the evicted expert, < 0 for an empty slot ExpertCache* cache_ = nullptr; ExpertSource* src_ = nullptr; std::vector* res_ = nullptr; - int32_t* d_res_ = nullptr; int64_t n_layers_ = 0, n_expert_ = 0; int max_moves_ = 0; int dev_ = -1, main_ = 0; + bool paced_ = false; + cudaEvent_t after_ = nullptr; const AdaptiveTier* upper_ = nullptr; + const AdaptiveTier* lower_ = nullptr; + bool wait_ = false; + std::vector blocked_; // per (layer, expert): not a candidate, rebuilt each call std::vector upper_has_; // per (layer, expert), rebuilt each call std::vector> free_; // per layer - std::vector> pending_; // (residency index, slot) once the copies have landed - struct Move { int32_t layer, in, out, slot; }; // out: the evicted expert, < 0 for an empty slot - std::vector queued_; // this update's moves; [next_, end) not submitted yet - size_t next_ = 0; - bool paced_ = false; - cudaEvent_t after_ = nullptr; - cudaEvent_t t0_ = nullptr, t1_ = nullptr; // paced: around the last batch, whose `batch_` bytes are not timed yet - uint64_t batch_ = 0; + // The queued moves in order and their (residency index, slot) once landed: [0, admitted_) admitted, up to sent_ + // sent (and off_ bytes of the next), up to staged_ with their residents evicted. + std::vector queued_; + std::vector> pending_; + size_t admitted_ = 0, sent_ = 0, staged_ = 0; + uint64_t off_ = 0; + cudaEvent_t t0_ = nullptr, t1_ = nullptr; // around the last timed copies, whose `timed_` bytes have not landed + uint64_t timed_ = 0; double rate_ = 0; - std::string failed_; // a copy apply_pending could not submit - bool submit(const Move& m, std::string& err); - bool pump_n(size_t n, std::string& err); + std::string failed_; // a copy apply_pending could not submit cudaStream_t stream_ = nullptr; - cudaEvent_t ev_ = nullptr; + // the batches in flight, oldest first: a ring of events, each with the moves sent when it was recorded + static constexpr int kBatches = 64; + cudaEvent_t evs_[kBatches] = {}; + size_t batch_sent_[kBatches] = {}; + int first_ = 0, flying_ = 0; + uint64_t bytes_of(const Move& m) const; + void evict(const Move& m); + bool copy(const Move& m, uint64_t off, uint64_t n, std::string& err); + bool time_begin(bool worth); + bool end_batch(bool timed, uint64_t bytes, std::string& err); }; } // namespace strata::core diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 0cee1782011..01b5e433c82 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -54,6 +54,8 @@ using WatchFn = bool (*)(void* user, std::string& err); using PredictFn = void (*)(void* user, int64_t layer, const int32_t* ids, const float* w, int64_t n_tok, int64_t k); /// The window's last CPU rows are in. using TailFn = void (*)(void* user); +/// Layer `layer`'s CPU rows are in (`Verifier::set_rows_in`). +using RowsFn = void (*)(void* user, int64_t layer); /// The main GPU's VRAM expert tier. struct VerifyHits { @@ -132,6 +134,10 @@ class Verifier { /// head, the commit, the drafts). The main GPU may still run the window then: `window_done` follows its graph. /// Set before the first `run`. void set_tail(TailFn fn, void* user) { tail_ = fn; tail_user_ = user; } + /// `fn` runs once each layer's CPU rows are in and the next layer's prediction has gone out, but the last's (the + /// tail's): the main GPU's link then has nothing to carry until the next layer's router writes its rows to the + /// host. Set before the first `run`. + void set_rows_in(RowsFn fn, void* user) { rows_in_ = fn; rows_user_ = user; } cudaEvent_t window_done() const { return done_; } /// The pool gives a second GPU a share of each layer (`GpuPlanSink::gpu2_flag`): it writes its rows into the /// pool's rows and raises its token group's flag, and a branch of the window, forked at the ring, takes them @@ -226,6 +232,8 @@ class Verifier { bool learn_ = false; TailFn tail_ = nullptr; void* tail_user_ = nullptr; + RowsFn rows_in_ = nullptr; + void* rows_user_ = nullptr; int32_t* h_pids_ = nullptr; int32_t* m_pids_ = nullptr; // 2 x T * k float* h_pw_ = nullptr; float* m_pw_ = nullptr; // 2 x T * k uint32_t* h_pseq_ = nullptr; uint32_t* m_pseq_ = nullptr; diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index 735a4995278..f4c4bb4f94f 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -19,23 +19,17 @@ struct OnDevice { // the tier's GPU current for one call (when it has one), th AdaptiveTier::~AdaptiveTier() { OnDevice on(dev_, main_); if (stream_) cudaStreamSynchronize(stream_); - cudaEvent_t evs[] = {ev_, t0_, t1_}; + for (cudaEvent_t e : evs_) if (e) cudaEventDestroy(e); + cudaEvent_t evs[] = {t0_, t1_}; for (cudaEvent_t e : evs) if (e) cudaEventDestroy(e); if (stream_) cudaStreamDestroy(stream_); } -bool AdaptiveTier::set_paced() { - OnDevice on(dev_, main_); - paced_ = cudaEventCreate(&t0_) == cudaSuccess && cudaEventCreate(&t1_) == cudaSuccess; - return paced_; -} - -bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int32_t* d_res, - int64_t n_layers, int64_t n_expert, int max_moves, std::string& err, int device, int main_device) { +bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector& host_res, int64_t n_layers, + int64_t n_expert, int max_moves, std::string& err, int device, int main_device) { cache_ = &cache; src_ = &src; res_ = &host_res; - d_res_ = d_res; n_layers_ = n_layers; n_expert_ = n_expert; max_moves_ = max_moves; @@ -43,13 +37,14 @@ bool AdaptiveTier::init(ExpertCache& cache, ExpertSource& src, std::vector= 0) (*res_)[(size_t) (m.layer * n_expert_ + m.out)] = kNotResident; // a miss from now on +} + +bool AdaptiveTier::copy(const Move& m, uint64_t off, uint64_t n, std::string& err) { const uint8_t* b = src_->blob(m.layer, m.in); - if (b == nullptr) { err = "adaptive tier: a refill copy failed"; return false; } - if (m.out >= 0) (*res_)[(size_t) (m.layer * n_expert_ + m.out)] = kNotResident; // evicted now: a miss meanwhile - if (cudaMemcpyAsync(cache_->device_slot(m.slot), b, (size_t) strata::kernels::cpu::expert_layout().blob_bytes(m.layer), - cudaMemcpyHostToDevice, stream_) != cudaSuccess) { + if (b == nullptr || + cudaMemcpyAsync(cache_->device_slot(m.slot) + off, b + off, (size_t) n, cudaMemcpyHostToDevice, stream_) != + cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + sent_bytes += n; return true; } -bool AdaptiveTier::pump_n(size_t n, std::string& err) { - if (next_ >= queued_.size()) return true; - OnDevice on(dev_, main_); - if (after_ != nullptr && cudaStreamWaitEvent(stream_, after_, 0) != cudaSuccess) { - err = "adaptive tier: a refill copy failed"; - return false; +// Takes the last timed copies' rate once they have landed; then starts timing these when `worth` (long enough for +// their start and end not to dominate) and none are being timed. +bool AdaptiveTier::time_begin(bool worth) { + float took = 0; + if (timed_ > 0 && cudaEventQuery(t1_) == cudaSuccess && cudaEventElapsedTime(&took, t0_, t1_) == cudaSuccess && + took > 0) { + const double r = (double) timed_ / (double) took; + rate_ = rate_ > 0 ? 0.8 * rate_ + 0.2 * r : r; + timed_ = 0; } - for (size_t i = 0; i < n && next_ < queued_.size(); ++i) - if (!submit(queued_[next_++], err)) return false; - if (cudaEventRecord(ev_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + return worth && timed_ == 0 && cudaEventRecord(t0_, stream_) == cudaSuccess; +} + +// After a batch of copies: its timing, and its event in the ring, with the moves sent so far (a full ring's newest +// event takes this batch too). +bool AdaptiveTier::end_batch(bool timed, uint64_t bytes, std::string& err) { + if (timed) { + if (cudaEventRecord(t1_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } + timed_ = bytes; + } + if (flying_ < kBatches) ++flying_; + const int i = (first_ + flying_ - 1) % kBatches; + batch_sent_[i] = sent_; + if (cudaEventRecord(evs_[i], stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } return true; } +void AdaptiveTier::stage(uint64_t bytes) { + uint64_t have = 0; + for (size_t i = sent_; i < staged_; ++i) have += bytes_of(queued_[i]); + have -= off_; + for (; staged_ < queued_.size() && have < bytes; ++staged_) { + evict(queued_[staged_]); + have += bytes_of(queued_[staged_]); + } +} + +bool AdaptiveTier::pump_bytes(uint64_t bytes, std::string& err) { + if (sent_ >= staged_ || bytes < (64u << 10)) return true; + OnDevice on(dev_, main_); + const bool timed = time_begin(bytes >= (256u << 10)); + uint64_t done = 0; + while (sent_ < staged_ && done < bytes) { + const Move& m = queued_[sent_]; + const uint64_t blob = bytes_of(m); + uint64_t n = std::min(bytes - done, blob - off_); + if (off_ + n < blob) n &= ~(uint64_t) 4095; // whole pages, but the blob's end + if (n == 0) break; + if (!copy(m, off_, n, err)) return false; + done += n; + off_ += n; + if (off_ == blob) { + off_ = 0; + ++sent_; + } + } + return done == 0 || end_batch(timed, done, err); +} + bool AdaptiveTier::pump(uint64_t budget, uint64_t& sent, std::string& err) { sent = 0; size_t n = 0; - for (; next_ + n < queued_.size(); ++n) { - const uint64_t b = strata::kernels::cpu::expert_layout().blob_bytes(queued_[next_ + n].layer); + for (; sent_ + n < queued_.size(); ++n) { + const uint64_t b = bytes_of(queued_[sent_ + n]); if (sent + b > budget) break; sent += b; } if (n == 0) return true; OnDevice on(dev_, main_); - float took = 0; - if (batch_ > 0 && cudaEventQuery(t1_) == cudaSuccess && cudaEventElapsedTime(&took, t0_, t1_) == cudaSuccess && - took > 0) { - const double r = (double) batch_ / (double) took; // the last batch has landed - rate_ = rate_ > 0 ? 0.8 * rate_ + 0.2 * r : r; - batch_ = 0; - } - const bool timed = batch_ == 0 && n >= 4; // a batch of a few copies: its start and end dominate - // timed from when the work that may read its slots is done (pump_n waits for it too) + // timed from when the work that may read its slots is done if (after_ != nullptr && cudaStreamWaitEvent(stream_, after_, 0) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } - if (timed && cudaEventRecord(t0_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } - if (!pump_n(n, err)) return false; - if (timed) { - if (cudaEventRecord(t1_, stream_) != cudaSuccess) { err = "adaptive tier: a refill copy failed"; return false; } - batch_ = sent; + const bool timed = time_begin(n >= 4); // a batch of a few copies: its start and end dominate + for (size_t i = 0; i < n; ++i, ++sent_) { + evict(queued_[sent_]); + if (!copy(queued_[sent_], 0, bytes_of(queued_[sent_]), err)) return false; } - return true; + staged_ = sent_; + return end_batch(timed, sent, err); } bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay) { const auto t0 = std::chrono::steady_clock::now(); if (!failed_.empty()) { err = failed_; return false; } - if (paced_) { // the moves not submitted by now give way to this call's (ranked again if still worth it) - for (size_t i = next_; i < queued_.size(); ++i) { - const Move& m = queued_[i]; - if (m.out >= 0) { - --swaps; - } else { - free_[(size_t) m.layer].push_back(m.slot); - --fills; - } + apply_pending(false); + if (wait_ && admitted_ < queued_.size()) return true; + // the moves whose residents are still in place give way to this call's (ranked again if still worth it) + for (size_t i = staged_; i < queued_.size(); ++i) { + const Move& m = queued_[i]; + if (m.out >= 0) { + --swaps; + } else { + free_[(size_t) m.layer].push_back(m.slot); + --fills; } - dropped += (int64_t) (queued_.size() - next_); - queued_.resize(next_); - pending_.resize(next_); - apply_pending(false); } - if (!pending_.empty()) return true; // the previous moves are still in flight + dropped += (int64_t) (queued_.size() - staged_); + queued_.resize(staged_); + pending_.resize(staged_); + queued_.erase(queued_.begin(), queued_.begin() + (ptrdiff_t) admitted_); + pending_.erase(pending_.begin(), pending_.begin() + (ptrdiff_t) admitted_); + for (int b = 0; b < flying_; ++b) batch_sent_[(first_ + b) % kBatches] -= admitted_; + sent_ -= admitted_; + staged_ -= admitted_; + admitted_ = 0; + blocked_.assign((size_t) (n_layers_ * n_expert_), 0); // the experts still under way + for (const auto& pr : pending_) blocked_[(size_t) pr.first] = 1; if (upper_ != nullptr) { upper_has_.assign((size_t) (n_layers_ * n_expert_), 0); for (size_t i = 0; i < upper_has_.size(); ++i) upper_has_[i] = (*upper_->res_)[i] >= 0; - for (const auto& pr : upper_->pending_) upper_has_[(size_t) pr.first] = 1; + for (size_t i = 0; i < upper_->staged_; ++i) upper_has_[(size_t) upper_->pending_[i].first] = 1; } struct Ranked { float gain; int32_t layer, in, out, slot; }; // out < 0: an empty slot std::vector moves; @@ -148,9 +196,12 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay vict.clear(); const float* u = usage.data() + l * n_expert_; const int32_t* r = res.data() + l * n_expert_; + const uint8_t* bl = blocked_.data() + l * n_expert_; const uint8_t* up = upper_ != nullptr ? upper_has_.data() + l * n_expert_ : nullptr; + const int32_t* lo = lower_ != nullptr ? lower_->res_->data() + l * n_expert_ : nullptr; for (int32_t e = 0; e < (int32_t) n_expert_; ++e) { - if (r[e] < 0) { if (u[e] >= 2.0f && (up == nullptr || !up[e])) cand.emplace_back(u[e], e); } + const float ue = lo != nullptr && lo[e] >= 0 ? 0.5f * u[e] : u[e]; + if (r[e] < 0) { if (ue >= 2.0f && !bl[e] && (up == nullptr || !up[e])) cand.emplace_back(ue, e); } else vict.emplace_back(up != nullptr && up[e] ? -1.0f : u[e], e); // the upper tier's copy goes first } if (cand.empty()) continue; @@ -170,9 +221,7 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay } std::sort(moves.begin(), moves.end(), [](const Ranked& a, const Ranked& b) { return a.gain > b.gain; }); if ((int) moves.size() > max_moves_) moves.resize((size_t) max_moves_); - // the copies run beside the next windows; a move counts once admitted (apply_pending) - queued_.clear(); - next_ = 0; + // a move counts once queued; its resident leaves when it is staged (or sent, paced), its expert comes once landed for (const Ranked& m : moves) { if (m.out >= 0) { ++swaps; @@ -182,9 +231,8 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay ++fills; } queued_.push_back({m.layer, m.in, m.out, m.slot}); - pending_.emplace_back((int32_t) (m.layer * n_expert_ + m.in), m.slot); // resident once the copy has landed + pending_.emplace_back((int32_t) (m.layer * n_expert_ + m.in), m.slot); } - if (!paced_ && !pump_n(queued_.size(), err)) return false; if (decay) for (float& v : usage) v *= 0.7f; ms += std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); @@ -192,18 +240,23 @@ bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay } void AdaptiveTier::apply_pending(bool wait) { - if (pending_.empty()) return; - if (next_ < queued_.size()) { // paced moves not submitted yet - if (!wait) return; + if (wait && admitted_ < queued_.size()) { // send the rest and wait for it std::string err; - if (!pump_n(queued_.size(), err)) { failed_ = err; return; } + uint64_t sent = 0; + if (!paced_) stage(UINT64_MAX); + if (!(paced_ ? pump(UINT64_MAX, sent, err) : pump_bytes(UINT64_MAX, err))) { + failed_ = err; + return; + } + OnDevice on(dev_, main_); + cudaStreamSynchronize(stream_); } - if (wait) cudaEventSynchronize(ev_); - else if (cudaEventQuery(ev_) != cudaSuccess) return; + size_t landed = admitted_; // the batches landed, oldest first + for (; flying_ > 0 && cudaEventQuery(evs_[first_]) == cudaSuccess; first_ = (first_ + 1) % kBatches, --flying_) + landed = std::max(landed, batch_sent_[first_]); + if (landed == admitted_) return; std::vector& res = *res_; - for (const auto& [i, slot] : pending_) res[(size_t) i] = slot; - pending_.clear(); - if (d_res_ != nullptr) cudaMemcpy(d_res_, res.data(), res.size() * sizeof(int32_t), cudaMemcpyHostToDevice); + for (; admitted_ < landed; ++admitted_) res[(size_t) pending_[admitted_].first] = pending_[admitted_].second; } } // namespace strata::core diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 12cb420397e..e1abd399117 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -1240,6 +1240,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, } ms_predict += ms_since(c); } + if (rows_in_ != nullptr && k + 1 < g.n_layers * G) rows_in_(rows_user_, l); } while (late_flag != nullptr && ms_since(late_at) < 50.0) { // the last layer's (the graph ends after them) settle(false); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 333930b29d6..d62219f2e8f 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -458,6 +458,20 @@ struct Drive { /// needs no locking. `d.layers` is the CURRENT layer on entry (the adapter increments it as it walks the /// blob), which is why the layer index comes from there rather than from a counter of our own. std::FILE* routing = nullptr; + /// The first GPU's adaptive tier: its copies cross that GPU's link, and while they ran they slowed the window's + /// own transfers there (the router's rows and doorbell to the host, the CPU's rows to the combine) by 20-60 us a + /// layer. So they go out in pieces while the link has nothing else to carry: once a layer's CPU rows are in, as + /// much as lands before the next layer's router writes to the host (that layer's running average of the time from + /// the previous layer's CPU rows to its ring, by window size, less the time since and `kLinkReserveUs`). A + /// window's moves are staged before it (`drive_window`). Without a second GPU the pieces cannot carry what the + /// tier moves (the CPU takes every miss; pieces during the pool too slowed its RAM reads): `burst`, a window's + /// start sends all the queued moves, and an update waits for the previous one's. + strata::core::AdaptiveTier* tier1 = nullptr; + bool burst = false; + std::vector ring_us; // by window size and layer + Clock::time_point rows_at{}; // this window's last CPU rows (none yet: the epoch) + int win_t = 0; // this window's tokens + int64_t n_layers = 0; /// The second GPU's adaptive tier, paced: its copies go between windows, where the RAM is idle (the head, the /// commit, the drafts); through the layers they slowed the CPU pool's RAM reads, and on that GPU's one copy engine /// they held up the prefetch copies. An update is ranked at a window's tail, and its copies go out as far as they @@ -470,8 +484,8 @@ struct Drive { bool adapt_next = false; // an update is ranked at this window's tail bool flush_next = false; // the next window's tail ranks one: this gap takes the rest std::string tier_err; - /// The adaptive tiers' update (ranking, copy submission) on a thread of its own, joined before the next window: - /// started at the window's tail when the second tier is paced, else after the window. + /// The adaptive tiers' update (ranking, the second tier's first copies) on a thread of its own, started at the + /// window's tail and joined before the next window. std::function adapt; std::thread adapt_thr; bool adapt_ok = true; @@ -521,6 +535,50 @@ void drive_predict(void* user, int64_t layer, const int32_t* ids, const float* w strata::core::expert_prefetch_multi(t->d, layer, ids, w, n_tok, k); } +// The first tier's piece ends this long before the next ring: the router (20-30 us), the doorbell, and the copy's +// start ~20 us after the call under WDDM (50 and 30 us were slower; pieces between windows too, slower again). +constexpr double kLinkReserveUs = 80; + +/// The first tier's copy rate in bytes per ms (a first guess until measured). +double tier1_rate(const Drive* t) { + const double r = t->tier1->copy_rate(); + return r > 0 ? r : 5e6; +} + +/// Before a window of `T` tokens: the tiers' landed moves admitted, and the first tier's next moves staged for what +/// this window's layers take (their residents leave before the window's residency snapshot). +void drive_window(Drive* t, int T) { + t->win_t = T; + t->rows_at = Clock::time_point{}; + if (t->tier2 != nullptr) t->tier2->apply_pending(false); + if (t->tier1 == nullptr) return; + t->tier1->apply_pending(false); + if (t->burst) { // all of them, at once + t->tier1->stage(UINT64_MAX); + if (!t->tier1->pump_bytes(UINT64_MAX, t->tier_err)) { + t->d.failed = true; + t->d.fail = t->tier_err.c_str(); + } + return; + } + double us = 0; + for (int64_t l = 1; l < t->n_layers; ++l) + us += std::max(0.0, t->ring_us[(size_t) (T * t->n_layers + l)] - kLinkReserveUs); + t->tier1->stage((uint64_t) (1.25 * tier1_rate(t) * us / 1e3)); +} + +/// A layer's CPU rows are in (Verifier::set_rows_in): the first tier's piece that lands before the next router. +void drive_rows_in(void* user, int64_t layer) { + Drive* t = (Drive*) user; + if (t->tier1 == nullptr || t->burst || t->d.failed || layer + 1 >= t->n_layers) return; + const double us = t->ring_us[(size_t) (t->win_t * t->n_layers + layer + 1)] - kLinkReserveUs - + std::chrono::duration(Clock::now() - t->rows_at).count(); + if (us > 0 && !t->tier1->pump_bytes((uint64_t) (tier1_rate(t) * us / 1e3), t->tier_err)) { + t->d.failed = true; + t->d.fail = t->tier_err.c_str(); + } +} + /// The second tier's queued moves that end before the next window's first pool, or all of them. bool pump_gap(Drive* t, std::string& err, bool all = false) { if (t->tier2 == nullptr) return true; @@ -564,13 +622,19 @@ void drive_pool_multi(void* user, const float* x_f, const int32_t* ids, int64_t Drive* t = (Drive*) user; t->d.layers = layer; const Clock::time_point a = Clock::now(); + const size_t at = (size_t) (t->win_t * t->n_layers + std::min(layer, t->n_layers - 1)); + if (t->rows_at != Clock::time_point{}) { // the last CPU rows to this ring + const double us = std::chrono::duration(a - t->rows_at).count(); + t->ring_us[at] = t->ring_us[at] > 0 ? 0.8 * t->ring_us[at] + 0.2 * us : us; + } if (t->gap_open) { // the first pool after a window: the RAM's idle time between them t->gap_open = false; const double gap = std::chrono::duration(a - t->tail_at).count(); if (gap < 20.0) t->gap_ms = t->gap_ms > 0 ? 0.8 * t->gap_ms + 0.2 * gap : gap; } strata::core::expert_pool_dispatch_multi(t->d, x_f, ids, n_tok, k, out); - t->cpu_ms += std::chrono::duration(Clock::now() - a).count(); + t->rows_at = Clock::now(); + t->cpu_ms += std::chrono::duration(t->rows_at - a).count(); ++t->calls; } @@ -2110,7 +2174,7 @@ int main(int argc, char** argv) { // moves every --adapt-every rounds on the adapt thread, admitted once their copies have landed strata::core::AdaptiveTier tier; const bool adaptive = !host_res.empty() && o.adapt_every > 0 && o.adapt_swaps > 0; - if (adaptive && !tier.init(xcache, *srcp, host_res, d_res, g.n_layers, g.n_expert, o.adapt_swaps, err, + if (adaptive && !tier.init(xcache, *srcp, host_res, g.n_layers, g.n_expert, o.adapt_swaps, err, o.main_gpu, o.main_gpu)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; @@ -2206,14 +2270,15 @@ int main(int argc, char** argv) { for (int64_t e = 0; e < g.n_expert; ++e) host_res2[(size_t) (l * g.n_expert + e)] = gpu2.cache().slot_of(l, e); } if (ok && adaptive) { - ok = tier2.init(gpu2.cache(), *srcp, host_res2, nullptr, g.n_layers, g.n_expert, o.adapt_swaps, err, + ok = tier2.init(gpu2.cache(), *srcp, host_res2, g.n_layers, g.n_expert, o.adapt_swaps, err, o.second_gpu, main_dev); if (ok) { // lowest slots first: the last ones, which the prompt path borrows, stay empty longest for (size_t i = empty2.size(); i-- > 0;) tier2.add_free(empty2[i], (int32_t) (pre.size() + i)); tier2.set_upper(&tier); - tier2.set_after(gpu2.done_event()); // a window's last share may still read its slots - if (tier2.set_paced()) drive.tier2 = &tier2; // its copies between windows (Drive::tier2) + tier.set_lower(&tier2); + tier2.set_paced(gpu2.done_event()); // between windows; a window's last share may still read its slots + drive.tier2 = &tier2; } } if (ok) { @@ -2237,10 +2302,11 @@ int main(int argc, char** argv) { (!two || (tier2.adapt(drive.d.usage, adapt_err) && pump_gap(&drive, adapt_err))); }; drive.adapt = adapt; - auto apply_pending = [&](bool wait) { - tier.apply_pending(wait); - tier2.apply_pending(wait); - }; + if (tier.on()) drive.tier1 = &tier; // its copies in pieces, or at once without a second GPU (Drive::tier1) + drive.burst = !tier2.on(); + tier.set_wait(drive.burst); + drive.n_layers = g.n_layers; + drive.ring_us.assign((size_t) ((strata::kernels::kVerifyMaxT + 1) * g.n_layers), 0.0); if (!o.no_capture && !o.no_token_graph && layer_dump == nullptr && half_dump == nullptr && (hit_fn == nullptr || thits.on()) && !native_pack) { if (!strata::core::session_capture_token(wt, g, ss, d_parts, loop_scratch.y_miss, loop_scratch.parts_bytes, @@ -2365,9 +2431,9 @@ int main(int argc, char** argv) { ver.set_gpu2(true); ver.set_watch(&drive_watch, &drive); } - if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache + if (drive.tier1 != nullptr || drive.tier2 != nullptr) { // the tiers' updates start at the tail ver.set_tail(&drive_tail, &drive); - tier.set_after(ver.window_done()); + ver.set_rows_in(&drive_rows_in, &drive); } prefill.draft = &mtp; drive.d.plan = ver.plan_sink(); @@ -2591,7 +2657,8 @@ int main(int argc, char** argv) { if (last > reuse && batched_end - last <= o.feed_max) batched_end = last; } if (batched_end > reuse) { - apply_pending(true); // no copy may still land in a lent slot + tier.apply_pending(true); // no copy may still land in a lent slot + tier2.apply_pending(true); if (!pbuf.take(std::min(o.prefill_chunk, batched_end - reuse), err)) { report(err); return 1; } tr("prompt start", reuse, batched_end); for (int64_t a0 = reuse; a0 < batched_end;) { @@ -2625,7 +2692,7 @@ int main(int argc, char** argv) { drive.d.experts = 0; drive.d.failed = false; drive.adapt_next = drive.flush_next = false; - apply_pending(false); + drive_window(&drive, T); if (!ver.run(T, window.data(), j, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) { if (drive.d.failed && drive.d.fail) err = drive.d.fail; return false; @@ -2690,7 +2757,7 @@ int main(int argc, char** argv) { drive.d.layers = 0; drive.d.experts = 0; drive.d.failed = false; - apply_pending(false); + drive_window(&drive, T); tr("window", p, T); drive.adapt_next = !drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0; drive.flush_next = !drive.d.usage.empty() && ((rounds + 2) % o.adapt_every) == 0; @@ -2701,8 +2768,6 @@ int main(int argc, char** argv) { } int a = 0; while (a < T - 1 && window[(size_t) a + 1] == outv[(size_t) a]) ++a; - // the adaptive tier beside the commit and the draft (as in generate) - if (drive.adapt_next && !drive.adapt_thr.joinable()) drive.start_adapt(); if (!ver.commit(a + 1, err, false)) { // beside the draft drive.join_adapt(); std::printf("ERR %s\n", err.c_str()); @@ -3104,9 +3169,9 @@ int main(int argc, char** argv) { ver.set_gpu2(true); ver.set_watch(&drive_watch, &drive); } - if (drive.tier2 != nullptr) { // the update then starts at the tail, while the 3090 may still read its cache + if (drive.tier1 != nullptr || drive.tier2 != nullptr) { // the tiers' updates start at the tail ver.set_tail(&drive_tail, &drive); - tier.set_after(ver.window_done()); + ver.set_rows_in(&drive_rows_in, &drive); } drive.d.plan = ver.plan_sink(); drive.d.host_res = ver.residency(); // the snapshot the windows decide their GPU hits by @@ -3203,7 +3268,7 @@ int main(int argc, char** argv) { drive.d.layers = 0; drive.d.experts = 0; drive.d.failed = false; - apply_pending(false); + drive_window(&drive, T); drive.adapt_next = !drive.d.usage.empty() && ((rounds + 1) % o.adapt_every) == 0; drive.flush_next = !drive.d.usage.empty() && ((rounds + 2) % o.adapt_every) == 0; if (!ver.run(T, window.data(), p, &drive_pool_multi, &drive, outv.data(), err)) { @@ -3250,9 +3315,6 @@ int main(int argc, char** argv) { first_window = false; ttft_ms = std::chrono::duration(Clock::now() - t_start).count(); } - // plan v0.3 P6: the adaptive tier's host work (ranking, copy submission) runs on its own thread while the - // GPU commits and drafts; it touches only the residency tables, which nothing reads until the next window - if (drive.adapt_next && !drive.adapt_thr.joinable()) drive.start_adapt(); if (!ver.commit(a + 1, err, false)) { // beside the draft drive.join_adapt(); std::fprintf(stderr, "strata generate: %s\n", err.c_str()); @@ -3347,8 +3409,10 @@ int main(int argc, char** argv) { drive.d.ms_run / rounds); ver.print_profile(); if (rounds > 0 && !drive.d.usage.empty()) - std::printf("%-24s %lld experts swapped into the VRAM tier (every %d rounds, %.3f ms/round)\n", "adaptive tier", - (long long) tier.swaps, o.adapt_every, tier.ms / rounds); + std::printf("%-24s %lld experts swapped in and %lld into empty slots (every %d rounds, %.3f ms/round), " + "%lld moves dropped; %.1f MB/round copied\n", "adaptive tier", (long long) tier.swaps, + (long long) tier.fills, o.adapt_every, tier.ms / rounds, (long long) tier.dropped, + (double) tier.sent_bytes / 1e6 / (double) rounds); if (rounds > 0 && drive.d.gpu2 != nullptr) std::printf("%-24s %.2f routed entries and %.2f experts per round in %.2f layers (%.2f left to the CPU), " "its rows after the CPU's in %.2f layers, %.3f ms/round; %lld experts swapped in and %lld into " From 5137b33e907b58acba81a7f7f75992d8b9d6e48b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Tue, 29 Sep 2026 23:27:06 +0200 Subject: [PATCH 76/92] The host thread on the last core, the pool off the second GPU's interrupt core, copies in batches Windows sends each GPU's interrupts to one logical processor (here the 5070 Ti's to the first, the 3090's to the second, one MSI each), and the host thread ran on the first: generating IQ3_XXS thinking text the 5070 Ti raised ~12,000 interrupts a second, each ~13 us in dxgkrnl's ISR and ~3 us in DPCs, a fifth of that processor. A copy raises one when it lands, so the prefetch's copies, one per expert, raised most of them. The host saw a layer's ring up to 30-110 us late (p99), its call submitting the 5070 Ti's share took 18-24 us (14-17 on another core), and in the pool's gate/up phase it finished last in 42% of the layers with a share. - The host thread on the last physical core, the pool's workers on the others; with a second GPU the first core stays with the OS and its interrupts (6 workers here, from 7). core_plan() decides; the startup log says where the threads run. - The 5070 Ti's prefetch copies and each adaptive-tier call's copies in one cudaMemcpyBatchAsync (copy_blobs): the first processor's interrupt time 10% -> 4% on IQ3_XXS thinking text. Window hashes identical to 7105fcc's (--adapt-every 0, both models); pool_test --stress passes. Fixed text, 4 runs each in both orders, t/s: both GPUs IQ3_XXS code +1.2%, thinking +0.5%, after 16K +1.3%, edit -0.7%; UD-Q4_K_XL +5.9 / +0.7 / +1.7 / -0.1%; the 3090 alone IQ3_XXS code +4.9%, thinking +0.9%, UD-Q4_K_XL +3.2 / +2.1%. 16K prompt times unchanged (IQ3_XXS 8.11 -> 7.93 s, UD-Q4_K_XL 8.53 -> 8.48 s). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/adaptive_tier.hpp | 5 +++++ include/strata/core/expert_cache.hpp | 5 +++++ include/strata/kernels/cpu/pool.hpp | 31 +++++++++++++++++---------- src/core/adaptive_tier.cpp | 20 ++++++++++++----- src/core/expert_cache.cpp | 9 ++++++++ src/core/second_gpu.cpp | 14 +++++++----- src/core/session.cpp | 10 ++++----- src/kernels/cpu/pool.cpp | 20 +++++++++++------ src/kernels/cpu/pool_test.cpp | 4 ++-- src/program/generate.cpp | 12 ++++++++--- 10 files changed, 93 insertions(+), 37 deletions(-) diff --git a/include/strata/core/adaptive_tier.hpp b/include/strata/core/adaptive_tier.hpp index 77ae932c96e..f5c9c1c4ba5 100644 --- a/include/strata/core/adaptive_tier.hpp +++ b/include/strata/core/adaptive_tier.hpp @@ -106,9 +106,14 @@ class AdaptiveTier { cudaEvent_t evs_[kBatches] = {}; size_t batch_sent_[kBatches] = {}; int first_ = 0, flying_ = 0; + // a call's copies, sent as one batch (copy_blobs) + std::vector cp_dst_; + std::vector cp_src_; + std::vector cp_bytes_; uint64_t bytes_of(const Move& m) const; void evict(const Move& m); bool copy(const Move& m, uint64_t off, uint64_t n, std::string& err); + bool send(std::string& err); bool time_begin(bool worth); bool end_batch(bool timed, uint64_t bytes, std::string& err); }; diff --git a/include/strata/core/expert_cache.hpp b/include/strata/core/expert_cache.hpp index da856d75d79..a0dfd1b0a29 100644 --- a/include/strata/core/expert_cache.hpp +++ b/include/strata/core/expert_cache.hpp @@ -152,4 +152,9 @@ class ExpertCache { int64_t admitted_ = 0; }; +/// Copies `n` pinned host blobs into VRAM on `stream` as one submission (cudaMemcpyBatchAsync: the copies in any +/// order, the batch in the stream's). Under WDDM a copy submitted alone raises an interrupt when it completes, on the +/// CPU its GPU's interrupts go to (~10 us there, in the ISR and a DPC); a batch raises one. +bool copy_blobs(void* const* dst, const void* const* src, const size_t* bytes, size_t n, void* stream); + } // namespace strata::core diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp index 4e155009fbf..a9a20a8aac9 100644 --- a/include/strata/kernels/cpu/pool.hpp +++ b/include/strata/kernels/cpu/pool.hpp @@ -62,15 +62,24 @@ struct ExpertJobMulti { /// worker. On the 6-core/12-thread machine this project measures on, `hardware_concurrency()/2` workers on /// logical processors 0..5 would put every worker on a sibling pair and halve the useful bandwidth - which is /// exactly the kind of error that shows up as "the CPU path is slower than the model says" with no clue why. -/// -/// `skip_first` drops the first core, which P2.S3 reserves for the host loop. -std::vector physical_cores(bool skip_first); +std::vector physical_cores(); + +/// Where the host loop and the pool's workers run, one logical processor each: the host on the last physical core, +/// the workers on the others. With `spare_first` (a second GPU) the first core stays with the OS: Windows sends the +/// GPUs' interrupts there (here the 5070 Ti's to its first logical processor, the 3090's to its second), and the +/// second GPU's copies and launches raise thousands a second during generation, each ~16 us there in the ISR and a +/// DPC. With four physical cores or fewer the first core takes a worker anyway. +struct CorePlan { + int host = -1; + std::vector workers; +}; +CorePlan core_plan(bool spare_first); /// **THE RESERVATION IS A FICTION UNLESS THE HOST IS ACTUALLY PUT THERE.** /// -/// `physical_cores(true)` keeps the workers off the first physical core so that the host loop can spin on -/// `cudaEventQuery` without stealing a worker's cycles. Nothing in the pool can enforce the other half of -/// that, so this is it: the host loop calls this on entry and restores on exit. +/// `core_plan()` keeps the workers off the host's core so that the host loop can spin on `cudaEventQuery` +/// without stealing a worker's cycles. Nothing in the pool can enforce the other half of that, so this is it: +/// the host loop calls this on entry and restores on exit. /// /// MEASURED, and this is why it exists: the pool runs at **36.32 GB/s on 5 workers with nothing else running** /// - exactly 5/6 of L9's 44.14 on 6 - and at **26.9 GB/s inside the host loop**, where the unpinned spinning @@ -86,13 +95,13 @@ void restore_thread_affinity(long long previous); #endif class ExpertPool { public: - /// `n_workers <= 0` means "every physical core except the first". Workers are pinned to physical cores - /// (minus core 0 by default) and each owns one `ExpertScratch`, so nothing in the token path allocates. + /// `n_workers <= 0` means a worker on each of `core_plan(spare_first).workers`. Workers are pinned to those + /// cores and each owns one `ExpertScratch`, so nothing in the token path allocates. /// /// **`host_works` PUTS THE HOST THREAD INTO THE DRAIN (R2.2's FIRST HALF).** /// - /// The pool reserves core 0 for the host loop so the doorbell spin cannot steal a worker's cycles - but - /// during `run()` the host does not spin, it waits, so core 0 is idle for the whole drain. Measured on the + /// The pool keeps a core for the host loop so the doorbell spin cannot steal a worker's cycles - but + /// during `run()` the host does not spin, it waits, so that core is idle for the whole drain. Measured on the /// 6-core machine this project targets: the engine's pool drains at **33.7 GB/s** (663.6 MB of expert /// blobs in 19.71 ms/token) where the same kernel on 5 workers should reach 5/6 x 44.14 = 36.8 and the /// machine measures 44.14 GB/s on all six. So the sixth core is being paid for and not used. @@ -100,7 +109,7 @@ class ExpertPool { /// With `host_works`, `run()` claims jobs itself instead of spinning on `done_`, and the pool is six /// threads on six cores. `false` is the A/B arm and exists so the change is measurable rather than /// asserted - the counter it moves is `pool phases ... drain`, which is host-side and needs no profiler. - explicit ExpertPool(int n_workers = 0, bool pin = true, bool host_works = true); + explicit ExpertPool(int n_workers = 0, bool pin = true, bool host_works = true, bool spare_first = false); ~ExpertPool(); ExpertPool(const ExpertPool&) = delete; ExpertPool& operator=(const ExpertPool&) = delete; diff --git a/src/core/adaptive_tier.cpp b/src/core/adaptive_tier.cpp index f4c4bb4f94f..a298afecf3f 100644 --- a/src/core/adaptive_tier.cpp +++ b/src/core/adaptive_tier.cpp @@ -63,16 +63,26 @@ void AdaptiveTier::evict(const Move& m) { bool AdaptiveTier::copy(const Move& m, uint64_t off, uint64_t n, std::string& err) { const uint8_t* b = src_->blob(m.layer, m.in); - if (b == nullptr || - cudaMemcpyAsync(cache_->device_slot(m.slot) + off, b + off, (size_t) n, cudaMemcpyHostToDevice, stream_) != - cudaSuccess) { + if (b == nullptr) { err = "adaptive tier: a refill copy failed"; return false; } + cp_dst_.push_back(cache_->device_slot(m.slot) + off); + cp_src_.push_back(b + off); + cp_bytes_.push_back((size_t) n); sent_bytes += n; return true; } +bool AdaptiveTier::send(std::string& err) { + const bool ok = copy_blobs(cp_dst_.data(), cp_src_.data(), cp_bytes_.data(), cp_dst_.size(), stream_); + cp_dst_.clear(); + cp_src_.clear(); + cp_bytes_.clear(); + if (!ok) err = "adaptive tier: a refill copy failed"; + return ok; +} + // Takes the last timed copies' rate once they have landed; then starts timing these when `worth` (long enough for // their start and end not to dominate) and none are being timed. bool AdaptiveTier::time_begin(bool worth) { @@ -129,7 +139,7 @@ bool AdaptiveTier::pump_bytes(uint64_t bytes, std::string& err) { ++sent_; } } - return done == 0 || end_batch(timed, done, err); + return done == 0 || (send(err) && end_batch(timed, done, err)); } bool AdaptiveTier::pump(uint64_t budget, uint64_t& sent, std::string& err) { @@ -153,7 +163,7 @@ bool AdaptiveTier::pump(uint64_t budget, uint64_t& sent, std::string& err) { if (!copy(queued_[sent_], 0, bytes_of(queued_[sent_]), err)) return false; } staged_ = sent_; - return end_batch(timed, sent, err); + return send(err) && end_batch(timed, sent, err); } bool AdaptiveTier::adapt(std::vector& usage, std::string& err, bool decay) { diff --git a/src/core/expert_cache.cpp b/src/core/expert_cache.cpp index 9f5fc981fb3..46dfadc2ac4 100644 --- a/src/core/expert_cache.cpp +++ b/src/core/expert_cache.cpp @@ -245,6 +245,15 @@ bool ExpertCache::fill_slot(int32_t slot, const uint8_t* host_blob, void* stream return true; } +bool copy_blobs(void* const* dst, const void* const* src, const size_t* bytes, size_t n, void* stream) { + if (n == 0) return true; + cudaMemcpyAttributes attr{}; + attr.srcAccessOrder = cudaMemcpySrcAccessOrderStream; + attr.flags = cudaMemcpyFlagPreferOverlapWithCompute; // the copy engines, beside the kernels + size_t first = 0; + return cudaMemcpyBatchAsync(dst, src, bytes, n, &attr, &first, 1, (cudaStream_t) stream) == cudaSuccess; +} + bool ExpertCache::fill_slot_blocking(int32_t slot, const uint8_t* host_blob, std::string& err, int64_t bytes) { const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_); uint8_t* dst = device_slot(slot); diff --git a/src/core/second_gpu.cpp b/src/core/second_gpu.cpp index 701e57ab8cd..4cfb793aa4a 100644 --- a/src/core/second_gpu.cpp +++ b/src/core/second_gpu.cpp @@ -117,11 +117,15 @@ bool SecondGpu::prefetch(int64_t layer, const int32_t* ids, const uint8_t* const if (n <= 0) return true; if (n > pre_max_ || bytes > pre_cap_) { err = "second GPU: a prefetch does not fit its slots"; return false; } DeviceScope scope(dev_, main_); - bool ok = cudaStreamWaitEvent(pre_s_, ev_, 0) == cudaSuccess; // the last submitted layer may read the slots - for (int i = 0; ok && i < n; ++i) - ok = cudaMemcpyAsync(d_pre_ + (size_t) i * pre_cap_, src[i], (size_t) bytes, cudaMemcpyHostToDevice, pre_s_) == - cudaSuccess; - if (!ok || cudaEventRecord(pre_ev_, pre_s_) != cudaSuccess) { + void* dst[kPrefetchMax]; + size_t size[kPrefetchMax]; + for (int i = 0; i < n; ++i) { + dst[i] = d_pre_ + (size_t) i * pre_cap_; + size[i] = (size_t) bytes; + } + if (cudaStreamWaitEvent(pre_s_, ev_, 0) != cudaSuccess || // the last submitted layer may read the slots + !copy_blobs(dst, (const void* const*) src, size, (size_t) n, pre_s_) || + cudaEventRecord(pre_ev_, pre_s_) != cudaSuccess) { err = std::string("second GPU: prefetch: ") + cudaGetErrorString(cudaGetLastError()); return false; } diff --git a/src/core/session.cpp b/src/core/session.cpp index 79761aebc6b..e3d5e1e818d 100644 --- a/src/core/session.cpp +++ b/src/core/session.cpp @@ -569,15 +569,15 @@ bool SessionLoopScratch::init(size_t parts_bytes_in, std::string& err) { free(); return false; } - // **PIN THE HOST ONCE, NOT ONCE PER TOKEN.** `ExpertPool` builds its workers from `physical_cores(true)`, - // which drops the first physical core so the host loop can spin without taking a worker's cycles - and + // **PIN THE HOST ONCE, NOT ONCE PER TOKEN.** `ExpertPool` builds its workers from `core_plan()`, which + // keeps them off the host's core so the host loop can spin without taking a worker's cycles - and // nothing in the pool can pin the host, so if this does not happen the spin is free to land on a worker's // core or its SMT sibling. The symptom is not an error: it is a CPU path at 26.9 GB/s where the same pool // runs at 36.32. It was being done and undone on EVERY token, which is a syscall pair on the critical path // for a property that wants to hold for the whole session. - const std::vector cores = strata::kernels::cpu::physical_cores(false); - if (!cores.empty()) { - pinned_core = strata::kernels::cpu::pin_current_thread(cores[0]); + const int host = strata::kernels::cpu::core_plan(false).host; // the last core, a second GPU or not + if (host >= 0) { + pinned_core = strata::kernels::cpu::pin_current_thread(host); pinned = true; } return true; diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cpp index 0b1c3e6d586..b3280faf6f2 100644 --- a/src/kernels/cpu/pool.cpp +++ b/src/kernels/cpu/pool.cpp @@ -19,7 +19,7 @@ namespace strata::kernels::cpu { -std::vector physical_cores(bool skip_first) { +std::vector physical_cores() { std::vector cores; #if defined(_WIN32) // Ask the OS rather than assuming a layout. `hardware_concurrency()` returns LOGICAL processors, and on @@ -55,10 +55,18 @@ std::vector physical_cores(bool skip_first) { else for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back((int) i); #endif - if (skip_first && !cores.empty()) cores.erase(cores.begin()); return cores; } +CorePlan core_plan(bool spare_first) { + const std::vector cores = physical_cores(); + CorePlan plan; + if (cores.empty()) return plan; + plan.host = cores.back(); + plan.workers.assign(cores.begin() + (spare_first && cores.size() > 4 ? 1 : 0), cores.end() - 1); + return plan; +} + namespace { // `state_`: the epoch in the high half, then the closed bit, then the parked workers @@ -112,8 +120,8 @@ void restore_thread_affinity(long long previous) { #endif } -ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(host_works) { - const std::vector cores = physical_cores(true); +ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, bool spare_first) : host_works_(host_works) { + const std::vector cores = core_plan(spare_first).workers; n_ = n_workers > 0 ? n_workers : (int) cores.size(); if (n_ < 1) n_ = 1; state_.store(kClosed | (uint64_t) n_, std::memory_order_relaxed); // epoch 0, closed, everyone parked @@ -390,8 +398,8 @@ void ExpertPool::run(ExpertJob* jobs, int n) { // ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.** // - // The loop below used to be `while (done_ != n) _mm_pause();`. The host is pinned to core 0 - the core - // `physical_cores(true)` deliberately keeps the five workers off - so for the whole drain that core was + // The loop below used to be `while (done_ != n) _mm_pause();`. The host is pinned to its own core - the + // one `core_plan()` deliberately keeps the workers off - so for the whole drain that core was // idle while five cores did six cores' worth of work. Measured before the change: 33.7 GB/s against // 5/6 x 44.14 = 36.8 for five workers and 44.14 for six. // diff --git a/src/kernels/cpu/pool_test.cpp b/src/kernels/cpu/pool_test.cpp index 20088ef13e3..5b68290d3a0 100644 --- a/src/kernels/cpu/pool_test.cpp +++ b/src/kernels/cpu/pool_test.cpp @@ -314,9 +314,9 @@ int main(int argc, char** argv) { } // ---- the pool - const std::vector cores = cpu::physical_cores(true); + const std::vector cores = cpu::core_plan(false).workers; const int hw = (int) std::thread::hardware_concurrency(); - std::printf(" %-44s %d logical, %d physical (skipping the first)\n", "cores the pool will use", + std::printf(" %-44s %d logical, %d physical (not the last, the host's)\n", "cores the pool will use", hw, (int) cores.size()); cpu::ExpertPool pool; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index d62219f2e8f..f44c72c1bdd 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -1393,7 +1393,9 @@ int main(int argc, char** argv) { // --ple-io ram: the table goes into RAM once the SSD has delivered the experts and the draft layer if (ple_table.is_open() && !ple_table.start_ram_load(err)) std::fprintf(stderr, "strata generate: %s; its rows stay on the SSD\n", err.c_str()); - strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker); + // with a second GPU the first core stays with the OS: that GPU's interrupts go there (core_plan) + const bool spare_first = o.second_gpu >= 0; + strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker, spare_first); if (o.no_ple_prefetch) strata::kernels::ple_prefetch_enable(false); std::fprintf(stderr, "strata generate: session is up; locating the head\n"); const strata::core::WeightRef* wo = wt.find("output.weight"); @@ -1688,8 +1690,12 @@ int main(int argc, char** argv) { strata::core::HitFn hit_fn = (o.no_pool || o.expert_cache <= 0) ? nullptr : &strata::core::expert_hit_run; void* pool_user = o.no_pool ? nullptr : (void*) &drive; - std::fprintf(stderr, "strata generate: %d expert-pool workers%s%s\n", pool.workers(), - pool.host_works() ? " + the host thread" : "", + const strata::kernels::cpu::CorePlan cores = strata::kernels::cpu::core_plan(spare_first); + std::string on; + for (int i = 0; i < pool.workers() && i < (int) cores.workers.size(); ++i) + on += (i > 0 ? "," : "") + std::to_string(cores.workers[(size_t) i]); + std::fprintf(stderr, "strata generate: %d pool workers on logical processors %s, the host thread on %d%s%s\n", + pool.workers(), on.c_str(), cores.host, pool.host_works() ? " (draining too)" : "", o.no_pool ? " (UNUSED: --no-pool)" : ""); // **THE MISALIGNMENT WARNING THAT STOOD HERE IS GONE, BECAUSE THE MISALIGNMENT IS FIXED.** From 6fe6825ce4e0a82e9e605c2ea55ffea6b801e300 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 00:52:59 +0200 Subject: [PATCH 77/92] The pool's threads move off the processors that take interrupts Windows moves device interrupts between processors as the load shifts (the power plan's interrupt steering, "Default" here): the GPUs' went to logical processors 0 and 1 one morning and to 1 and 2 that night, beside a network card's ~26,000 a second on 0, so the 3090's landed on a pool worker (up to ~45% of its processor in ISRs and DPCs). Moving that worker off its processor mid-run left the interrupts where they were (two traced runs: to its core's other processor, and to another core). So CorePlacement starts the threads where 5137b33 put them and, once a second between windows, reads each processor's interrupt and DPC time: a thread whose processor spent over 5% there two seconds in a row moves to a quiet processor, a free core's, else its own core's other one. Each move is logged; --stats counts them. Window hashes identical to 5137b33's (--adapt-every 0, both models); pool_test --stress passes. Fixed text, 2 runs each in both orders against 5137b33: no thread had to move (the interrupts stayed on the first core), speeds within the runs' spread (IQ3_XXS +0.2 / +0.2 / +0.2% on code / thinking / edit, UD-Q4_K_XL +7.8 / -0.4 / -0.4%). With the threshold lowered for a test, a worker on the first core moved to its other processor on the 3090 alone. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/session.hpp | 5 +- include/strata/kernels/cpu/pool.hpp | 65 ++++++++--- src/core/session.cpp | 9 +- src/kernels/cpu/pool.cpp | 172 ++++++++++++++++++++++------ src/kernels/cpu/pool_test.cpp | 2 +- src/program/generate.cpp | 30 +++-- 6 files changed, 214 insertions(+), 69 deletions(-) diff --git a/include/strata/core/session.hpp b/include/strata/core/session.hpp index 3b32942fbd8..26c1a4b72ec 100644 --- a/include/strata/core/session.hpp +++ b/include/strata/core/session.hpp @@ -364,8 +364,9 @@ struct SessionLoopScratch { long long pinned_core = -1; ///< the affinity to restore, or -1 if the host was never pinned bool pinned = false; - /// Allocates the buffers and pins the host thread. Call ONCE, at session setup. - bool init(size_t parts_bytes_in, std::string& err); + /// Allocates the buffers and pins the host thread to logical processor `host` (CorePlacement::host(); -1: the + /// last physical core's first). Call ONCE, at session setup. + bool init(size_t parts_bytes_in, std::string& err, int host = -1); /// Frees everything and restores the affinity. Safe to call twice, or on a default-constructed object. void free(); }; diff --git a/include/strata/kernels/cpu/pool.hpp b/include/strata/kernels/cpu/pool.hpp index a9a20a8aac9..7fbf49c8cbe 100644 --- a/include/strata/kernels/cpu/pool.hpp +++ b/include/strata/kernels/cpu/pool.hpp @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -58,26 +59,50 @@ struct ExpertJobMulti { const void* nact[MAXT] = {}; }; -/// One logical processor per PHYSICAL core, so a worker is never scheduled onto an SMT sibling of another -/// worker. On the 6-core/12-thread machine this project measures on, `hardware_concurrency()/2` workers on -/// logical processors 0..5 would put every worker on a sibling pair and halve the useful bandwidth - which is -/// exactly the kind of error that shows up as "the CPU path is slower than the model says" with no clue why. -std::vector physical_cores(); - -/// Where the host loop and the pool's workers run, one logical processor each: the host on the last physical core, -/// the workers on the others. With `spare_first` (a second GPU) the first core stays with the OS: Windows sends the -/// GPUs' interrupts there (here the 5070 Ti's to its first logical processor, the 3090's to its second), and the -/// second GPU's copies and launches raise thousands a second during generation, each ~16 us there in the ISR and a -/// DPC. With four physical cores or fewer the first core takes a worker anyway. -struct CorePlan { - int host = -1; - std::vector workers; +/// Each PHYSICAL core's logical processors (SMT siblings together), in the OS's order. The pool runs one thread +/// per physical core, so a worker is never scheduled onto an SMT sibling of another worker. On the 6-core/12-thread +/// machine this project measures on, `hardware_concurrency()/2` workers on logical processors 0..5 would put every +/// worker on a sibling pair and halve the useful bandwidth - which is exactly the kind of error that shows up as +/// "the CPU path is slower than the model says" with no clue why. +std::vector> physical_cores(); + +class ExpertPool; + +/// Where the host loop and the pool's workers run - one logical processor each, one thread per physical core - and +/// how they move off the processors that take interrupts. A GPU's copies and launches interrupt the CPU when they +/// complete (~10-40 us each in the ISR and a DPC, thousands a second during generation), on processors Windows picks +/// and moves as the load shifts (the power plan's interrupt steering; here the first few). So `tick`, called on the +/// host thread between windows, reads each processor's interrupt and DPC time once a second, and a thread whose +/// processor spent over 5% there two seconds in a row moves to a quiet processor: a free core's, else its own core's +/// other one. Without the counters (not Windows) the threads stay where they started. +class CorePlacement { +public: + /// The start: the host on the last physical core, the workers on the others; with `spare_first` (a second GPU, + /// whose work raises most of the interrupts) and more than four cores, the first core stays free: where Windows + /// sends interrupts first, and a place to move to. + explicit CorePlacement(bool spare_first); + int host() const { return host_; } + /// The workers' logical processors, by worker (the pool pins worker i to `workers()[i]`). + const std::vector& workers() const { return workers_; } + /// At most once a second: samples each processor's interrupt and DPC time and moves the threads whose processors + /// took them (`pool`'s workers; the calling thread for the host's). Returns what moved, or an empty string. + std::string tick(ExpertPool& pool); + int64_t moves = 0; + +private: + int quiet_processor(int from, const std::vector& load) const; + std::vector> cores_; + std::vector core_of_; // logical processor -> physical core, -1 when unknown + int host_ = -1; + std::vector workers_; + std::vector busy_; // each processor's interrupt and DPC time at the last sample (100 ns units) + std::vector hot_; // the samples in a row a processor spent over 5% there + std::chrono::steady_clock::time_point at_{}; }; -CorePlan core_plan(bool spare_first); /// **THE RESERVATION IS A FICTION UNLESS THE HOST IS ACTUALLY PUT THERE.** /// -/// `core_plan()` keeps the workers off the host's core so that the host loop can spin on `cudaEventQuery` +/// `CorePlacement` keeps the workers off the host's core so that the host loop can spin on `cudaEventQuery` /// without stealing a worker's cycles. Nothing in the pool can enforce the other half of that, so this is it: /// the host loop calls this on entry and restores on exit. /// @@ -95,8 +120,8 @@ void restore_thread_affinity(long long previous); #endif class ExpertPool { public: - /// `n_workers <= 0` means a worker on each of `core_plan(spare_first).workers`. Workers are pinned to those - /// cores and each owns one `ExpertScratch`, so nothing in the token path allocates. + /// Worker i is pinned to `cores[i]` (none of `cores`: `CorePlacement(false).workers()`); `n_workers <= 0` means + /// one on each. Each owns one `ExpertScratch`, so nothing in the token path allocates. /// /// **`host_works` PUTS THE HOST THREAD INTO THE DRAIN (R2.2's FIRST HALF).** /// @@ -109,12 +134,14 @@ class ExpertPool { /// With `host_works`, `run()` claims jobs itself instead of spinning on `done_`, and the pool is six /// threads on six cores. `false` is the A/B arm and exists so the change is measurable rather than /// asserted - the counter it moves is `pool phases ... drain`, which is host-side and needs no profiler. - explicit ExpertPool(int n_workers = 0, bool pin = true, bool host_works = true, bool spare_first = false); + explicit ExpertPool(int n_workers = 0, bool pin = true, bool host_works = true, std::vector cores = {}); ~ExpertPool(); ExpertPool(const ExpertPool&) = delete; ExpertPool& operator=(const ExpertPool&) = delete; int workers() const { return n_; } + /// Moves worker `i` to logical processor `core` (CorePlacement::tick). + void pin_worker(int i, int core); /// Whether the host thread also drains. Reported at startup, because "the engine adapts to the machine it /// is on" is only true if the engine says which adaptation it took. bool host_works() const { return host_works_; } diff --git a/src/core/session.cpp b/src/core/session.cpp index e3d5e1e818d..b5427d0a343 100644 --- a/src/core/session.cpp +++ b/src/core/session.cpp @@ -552,7 +552,7 @@ void session_graphs_free(SessionGraphs& gr) { gr.captured = false; } -bool SessionLoopScratch::init(size_t parts_bytes_in, std::string& err) { +bool SessionLoopScratch::init(size_t parts_bytes_in, std::string& err, int host) { if (y_miss != nullptr || probe != nullptr) { err = "SessionLoopScratch::init: already initialised"; return false; @@ -569,13 +569,16 @@ bool SessionLoopScratch::init(size_t parts_bytes_in, std::string& err) { free(); return false; } - // **PIN THE HOST ONCE, NOT ONCE PER TOKEN.** `ExpertPool` builds its workers from `core_plan()`, which + // **PIN THE HOST ONCE, NOT ONCE PER TOKEN.** `ExpertPool` builds its workers from `CorePlacement`, which // keeps them off the host's core so the host loop can spin without taking a worker's cycles - and // nothing in the pool can pin the host, so if this does not happen the spin is free to land on a worker's // core or its SMT sibling. The symptom is not an error: it is a CPU path at 26.9 GB/s where the same pool // runs at 36.32. It was being done and undone on EVERY token, which is a syscall pair on the critical path // for a property that wants to hold for the whole session. - const int host = strata::kernels::cpu::core_plan(false).host; // the last core, a second GPU or not + if (host < 0) { // the last physical core, CorePlacement's start + const std::vector> cores = strata::kernels::cpu::physical_cores(); + host = cores.back().front(); + } if (host >= 0) { pinned_core = strata::kernels::cpu::pin_current_thread(host); pinned = true; diff --git a/src/kernels/cpu/pool.cpp b/src/kernels/cpu/pool.cpp index b3280faf6f2..85810a2203d 100644 --- a/src/kernels/cpu/pool.cpp +++ b/src/kernels/cpu/pool.cpp @@ -19,52 +19,146 @@ namespace strata::kernels::cpu { -std::vector physical_cores() { - std::vector cores; +std::vector> physical_cores() { + std::vector> cores; #if defined(_WIN32) // Ask the OS rather than assuming a layout. `hardware_concurrency()` returns LOGICAL processors, and on // every SMT machine half of them are siblings - pinning one worker to each of the first N would put two // workers on each physical core and halve the bandwidth the expert kernel is bound by. DWORD len = 0; GetLogicalProcessorInformationEx(RelationProcessorCore, nullptr, &len); - if (len == 0) { - for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back((int) i); - } else { - std::vector buf(len); - if (GetLogicalProcessorInformationEx(RelationProcessorCore, - (PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX) buf.data(), &len)) { - const char* p = buf.data(); - const char* end = p + len; - while (p < end) { - const auto* e = (const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) p; - if (e->Relationship == RelationProcessorCore) { - const GROUP_AFFINITY& g = e->Processor.GroupMask[0]; - for (int bit = 0; bit < 64; ++bit) - if (g.Mask & (1ull << bit)) { cores.push_back((int) (g.Group * 64 + bit)); break; } - } - p += e->Size; + std::vector buf(len); + if (len > 0 && GetLogicalProcessorInformationEx(RelationProcessorCore, + (PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX) buf.data(), &len)) { + for (const char* p = buf.data(); p < buf.data() + len;) { + const auto* e = (const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) p; + if (e->Relationship == RelationProcessorCore) { + const GROUP_AFFINITY& g = e->Processor.GroupMask[0]; + std::vector core; + for (int bit = 0; bit < 64; ++bit) + if (g.Mask & (1ull << bit)) core.push_back((int) (g.Group * 64 + bit)); + if (!core.empty()) cores.push_back(core); } + p += e->Size; } } #else - cpu_set_t set; + cpu_set_t set; // each logical processor as a core of its own CPU_ZERO(&set); - if (sched_getaffinity(0, sizeof set, &set) == 0) + if (sched_getaffinity(0, sizeof set, &set) == 0) { for (int i = 0; i < CPU_SETSIZE; ++i) - if (CPU_ISSET(i, &set)) cores.push_back(i); - else - for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back((int) i); + if (CPU_ISSET(i, &set)) cores.push_back({i}); + } #endif + if (cores.empty()) + for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back({(int) i}); return cores; } -CorePlan core_plan(bool spare_first) { - const std::vector cores = physical_cores(); - CorePlan plan; - if (cores.empty()) return plan; - plan.host = cores.back(); - plan.workers.assign(cores.begin() + (spare_first && cores.size() > 4 ? 1 : 0), cores.end() - 1); - return plan; +namespace { + +#if defined(_WIN32) +// Each logical processor's time in interrupts and DPCs so far (100 ns units), from NtQuerySystemInformation's +// processor performance information (winternl.h names its DpcTime and InterruptTime Reserved1); the kernel adds to +// them at each clock tick, so over a second they resolve ~1.6%. +std::vector interrupt_times() { + struct Info { LARGE_INTEGER idle, kernel, user, dpc, interrupt; ULONG interrupts; }; + using Query = LONG(WINAPI*)(int, void*, ULONG, ULONG*); + static const Query query = reinterpret_cast( + reinterpret_cast(GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtQuerySystemInformation"))); + SYSTEM_INFO si; + GetSystemInfo(&si); + std::vector v(si.dwNumberOfProcessors); + ULONG len = 0; + std::vector t; + if (query == nullptr || query(8, v.data(), (ULONG) (v.size() * sizeof(Info)), &len) != 0) return t; + for (const Info& i : v) t.push_back((uint64_t) (i.dpc.QuadPart + i.interrupt.QuadPart)); + return t; +} +#endif + +constexpr double kBusy = 0.05; // a processor over this share of a second in interrupts and DPCs takes them +constexpr double kQuiet = 0.02; // one below it may take a thread + +} // namespace + +CorePlacement::CorePlacement(bool spare_first) : cores_(physical_cores()) { + for (size_t c = 0; c < cores_.size(); ++c) + for (int p : cores_[c]) { + if (p >= (int) core_of_.size()) core_of_.resize((size_t) p + 1, -1); + core_of_[(size_t) p] = (int) c; + } + host_ = cores_.back().front(); + for (size_t c = spare_first && cores_.size() > 4 ? 1 : 0; c + 1 < cores_.size(); ++c) + workers_.push_back(cores_[c].front()); +} + +// A quiet processor for a thread on `from`: the first of the quietest free core whose processors are all quiet, else +// the other processor of `from`'s own core, or -1. +int CorePlacement::quiet_processor(int from, const std::vector& load) const { + auto quiet = [&](int p) { return p < (int) load.size() && load[(size_t) p] < kQuiet && hot_[(size_t) p] == 0; }; + std::vector taken(cores_.size(), 0); + for (int p : workers_) taken[(size_t) core_of_[(size_t) p]] = 1; + taken[(size_t) core_of_[(size_t) host_]] = 1; + int best = -1; + double best_load = 0; + for (size_t c = 0; c < cores_.size(); ++c) { + if (taken[c]) continue; + bool ok = true; + double sum = 0; + for (int p : cores_[c]) { + ok = ok && quiet(p); + sum += p < (int) load.size() ? load[(size_t) p] : 0.0; + } + if (ok && (best < 0 || sum < best_load)) { + best = cores_[c].front(); + best_load = sum; + } + } + if (best >= 0) return best; + for (int p : cores_[(size_t) core_of_[(size_t) from]]) + if (p != from && quiet(p)) return p; + return -1; +} + +std::string CorePlacement::tick(ExpertPool& pool) { +#if defined(_WIN32) + const auto now = std::chrono::steady_clock::now(); + const double s = std::chrono::duration(now - at_).count(); + if (!busy_.empty() && s < 1.0) return {}; + std::vector cur = interrupt_times(); + // a first sample, or the engine idle since the last one (between requests): a new start, nothing to judge + const bool fresh = busy_.empty() || s > 3.0 || cur.size() != busy_.size(); + std::vector load(cur.size(), 0.0); + if (!fresh) + for (size_t p = 0; p < cur.size(); ++p) load[p] = (double) (cur[p] - busy_[p]) / 1e7 / s; + busy_ = std::move(cur); + at_ = now; + hot_.resize(busy_.size(), 0); + for (size_t p = 0; p < hot_.size(); ++p) hot_[p] = !fresh && load[p] > kBusy ? hot_[p] + 1 : 0; + std::string moved; + for (int t = -1; t < (int) workers_.size(); ++t) { // the host, then the workers + int& at = t < 0 ? host_ : workers_[(size_t) t]; + if (at >= (int) hot_.size() || hot_[(size_t) at] < 2) continue; + const int to = quiet_processor(at, load); + if (to < 0) continue; + if (t < 0) pin_current_thread(to); + else pool.pin_worker(t, to); + char line[160]; + std::snprintf(line, sizeof line, + "%slogical processor %d spent %.0f%% of the last second in interrupts and DPCs: " + "the %s moved to %d", moved.empty() ? "" : "; ", at, 100.0 * load[(size_t) at], + t < 0 ? "host thread" : "pool worker", to); + moved += line; + hot_[(size_t) at] = 0; + at = to; + ++moves; + } + return moved; +#else + (void) pool; + return {}; +#endif } namespace { @@ -120,8 +214,8 @@ void restore_thread_affinity(long long previous) { #endif } -ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, bool spare_first) : host_works_(host_works) { - const std::vector cores = core_plan(spare_first).workers; +ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, std::vector cores) : host_works_(host_works) { + if (cores.empty()) cores = CorePlacement(false).workers(); n_ = n_workers > 0 ? n_workers : (int) cores.size(); if (n_ < 1) n_ = 1; state_.store(kClosed | (uint64_t) n_, std::memory_order_relaxed); // epoch 0, closed, everyone parked @@ -138,6 +232,18 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works, bool spare_firs } } +void ExpertPool::pin_worker(int i, int core) { + if (i < 0 || i >= (int) threads_.size() || core < 0) return; +#if defined(_WIN32) + SetThreadAffinityMask((HANDLE) threads_[(size_t) i].native_handle(), (DWORD_PTR) 1 << (core & 63)); +#else + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(core, &set); + pthread_setaffinity_np(threads_[(size_t) i].native_handle(), sizeof set, &set); +#endif +} + ExpertPool::~ExpertPool() { stop_.store(true, std::memory_order_seq_cst); // a new value, so a sleeping worker's wait returns and it sees the stop flag @@ -399,7 +505,7 @@ void ExpertPool::run(ExpertJob* jobs, int n) { // ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.** // // The loop below used to be `while (done_ != n) _mm_pause();`. The host is pinned to its own core - the - // one `core_plan()` deliberately keeps the workers off - so for the whole drain that core was + // one `CorePlacement` deliberately keeps the workers off - so for the whole drain that core was // idle while five cores did six cores' worth of work. Measured before the change: 33.7 GB/s against // 5/6 x 44.14 = 36.8 for five workers and 44.14 for six. // diff --git a/src/kernels/cpu/pool_test.cpp b/src/kernels/cpu/pool_test.cpp index 5b68290d3a0..50d59246ff2 100644 --- a/src/kernels/cpu/pool_test.cpp +++ b/src/kernels/cpu/pool_test.cpp @@ -314,7 +314,7 @@ int main(int argc, char** argv) { } // ---- the pool - const std::vector cores = cpu::core_plan(false).workers; + const std::vector cores = cpu::CorePlacement(false).workers(); const int hw = (int) std::thread::hardware_concurrency(); std::printf(" %-44s %d logical, %d physical (not the last, the host's)\n", "cores the pool will use", hw, (int) cores.size()); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index f44c72c1bdd..25188fe66eb 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -467,6 +467,8 @@ struct Drive { /// tier moves (the CPU takes every miss; pieces during the pool too slowed its RAM reads): `burst`, a window's /// start sends all the queued moves, and an update waits for the previous one's. strata::core::AdaptiveTier* tier1 = nullptr; + /// Where the host and the pool's workers run: checked before each window, once a second (CorePlacement::tick). + strata::kernels::cpu::CorePlacement* placement = nullptr; bool burst = false; std::vector ring_us; // by window size and layer Clock::time_point rows_at{}; // this window's last CPU rows (none yet: the epoch) @@ -550,6 +552,10 @@ double tier1_rate(const Drive* t) { void drive_window(Drive* t, int T) { t->win_t = T; t->rows_at = Clock::time_point{}; + if (t->placement != nullptr && t->d.pool != nullptr) { + const std::string moved = t->placement->tick(*t->d.pool); + if (!moved.empty()) std::fprintf(stderr, "strata generate: %s\n", moved.c_str()); + } if (t->tier2 != nullptr) t->tier2->apply_pending(false); if (t->tier1 == nullptr) return; t->tier1->apply_pending(false); @@ -1393,9 +1399,11 @@ int main(int argc, char** argv) { // --ple-io ram: the table goes into RAM once the SSD has delivered the experts and the draft layer if (ple_table.is_open() && !ple_table.start_ram_load(err)) std::fprintf(stderr, "strata generate: %s; its rows stay on the SSD\n", err.c_str()); - // with a second GPU the first core stays with the OS: that GPU's interrupts go there (core_plan) - const bool spare_first = o.second_gpu >= 0; - strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker, spare_first); + // where the host loop and the pool's workers run, moved off the processors that take interrupts as generation + // goes (drive_window); with a second GPU, whose work raises most of them, the first core starts free + strata::kernels::cpu::CorePlacement placement(/*spare_first=*/o.second_gpu >= 0); + strata::kernels::cpu::ExpertPool pool(o.pool_workers, /*pin=*/true, /*host_works=*/!o.no_host_worker, + placement.workers()); if (o.no_ple_prefetch) strata::kernels::ple_prefetch_enable(false); std::fprintf(stderr, "strata generate: session is up; locating the head\n"); const strata::core::WeightRef* wo = wt.find("output.weight"); @@ -1582,6 +1590,7 @@ int main(int argc, char** argv) { drive.d.hit_cpu_order = o.expert_cache_cpu_order; drive.d.split_rows = !o.no_split_rows; drive.d.pool = &pool; + drive.placement = &placement; drive.d.src = srcp; drive.d.n_expert = g.n_expert; drive.d.jobs.resize((size_t) K); @@ -1690,12 +1699,11 @@ int main(int argc, char** argv) { strata::core::HitFn hit_fn = (o.no_pool || o.expert_cache <= 0) ? nullptr : &strata::core::expert_hit_run; void* pool_user = o.no_pool ? nullptr : (void*) &drive; - const strata::kernels::cpu::CorePlan cores = strata::kernels::cpu::core_plan(spare_first); std::string on; - for (int i = 0; i < pool.workers() && i < (int) cores.workers.size(); ++i) - on += (i > 0 ? "," : "") + std::to_string(cores.workers[(size_t) i]); + for (int i = 0; i < pool.workers() && i < (int) placement.workers().size(); ++i) + on += (i > 0 ? "," : "") + std::to_string(placement.workers()[(size_t) i]); std::fprintf(stderr, "strata generate: %d pool workers on logical processors %s, the host thread on %d%s%s\n", - pool.workers(), on.c_str(), cores.host, pool.host_works() ? " (draining too)" : "", + pool.workers(), on.c_str(), placement.host(), pool.host_works() ? " (draining too)" : "", o.no_pool ? " (UNUSED: --no-pool)" : ""); // **THE MISALIGNMENT WARNING THAT STOOD HERE IS GONE, BECAUSE THE MISALIGNMENT IS FIXED.** @@ -2128,7 +2136,7 @@ int main(int argc, char** argv) { // Initialised unconditionally, including under --no-pool: the loop validates the scratch it is handed, so // passing a default-constructed one is an error rather than a fallback. (It was, and the guard caught it - // which is the point of the guard.) One allocation at setup either way. - if (!loop_scratch.init((size_t) K * g.n_embd * 4, err)) { + if (!loop_scratch.init((size_t) K * g.n_embd * 4, err, placement.host())) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; } @@ -3405,10 +3413,10 @@ int main(int argc, char** argv) { (double) (drive.d.multi_entries - entries0) / (double) (rounds * g.n_layers)); if (rounds > 0) std::printf("%-24s gate/up %.3f quantize %.3f down %.3f ms/round; %.1f GB/s over the rows phases; " - "CPU pool call %.3f ms/round; %u worker sleeps\n", "pool multi", pool.ms_multi_gu / rounds, - pool.ms_multi_q / rounds, pool.ms_multi_down / rounds, + "CPU pool call %.3f ms/round; %u worker sleeps; %lld thread moves\n", "pool multi", + pool.ms_multi_gu / rounds, pool.ms_multi_q / rounds, pool.ms_multi_down / rounds, (double) pool.multi_bytes / 1e6 / std::max(1e-9, pool.ms_multi_gu + pool.ms_multi_down), - (drive.cpu_ms - pool_ms0) / rounds, pool.sleeps() - sleeps0); + (drive.cpu_ms - pool_ms0) / rounds, pool.sleeps() - sleeps0, (long long) placement.moves); if (rounds > 0) std::printf("%-24s plan %.3f activation quantize %.3f jobs %.3f run %.3f ms/round\n", "dispatch", drive.d.ms_plan / rounds, drive.d.ms_actq / rounds, drive.d.ms_jobs / rounds, From 45dae9d688ba0bc464c2936416481924126891e5 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 05:50:42 +0200 Subject: [PATCH 78/92] The full expert profile: both GPUs' tiers start filled data/expert-profile.bin is upstream's ranking of all 24,576 (layer, expert) pairs (it ranked 8,000). The 3090's cache takes the pairs it fits, the 5070 Ti's tier the next ones: with 8,000 pairs that tier started with 523 of IQ3_S's 6,619 slots filled (UD-Q4_K_XL: 3,463 of ~4,300) and its adaptive tier filled the rest over thousands of tokens. Fixed text, both GPUs, 4 runs each in both orders, t/s against the 8,000-pair profile: IQ3_S code 173.5 (+15.8%), thinking 130.2 (+4.6%), after 16K 138.6 (+6.3%), edit 236.4 (+9.6%) UD-Q4_K_XL code 115.4 (+4.8%), thinking 102.3 (-1.1%), after 16K 119.8 (-0.2%), edit 161.8 (-1.1%) Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- data/expert-profile.bin | Bin 130328 -> 196632 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/data/expert-profile.bin b/data/expert-profile.bin index bf4aeed408d518e22d8e9dfd2fa821dcd05f232f..a4ed7e5feb68e68fa76df66b9109e99c1ef143b8 100644 GIT binary patch literal 196632 zcmX8c34l*k`v>rQ$CxpTW$as+F*BGkGq&s^OLn2fQlbrQN{bd*qa;g~5^YLJvPX%O z7SbY>koM@UC@qvC^8cLqzus^6{^ob@InVR`KF>Mlo_p^7U2tLlzHt;qjUY;ZVgHYs zQJuIjdL=4~jz*c$ZBfIhS5!OtGOiVU7?+6dh-yUR;u_J7QAY81Bcrm>;J9k^dQ?4X z?fcK2`%_fP=YnWiR44i*sux`o)sJ3_(xY4B{OBTO9f@j1r@L|m5<=_NBL3XsAP0yoEx38zsvDou(=68XT>$6KI*$Kt`O~s>#M1R zW6MV8^1TO}jw&ySEoI_z(fy7a9A`zBMTyZ*aj9sF?RxfgjLWD$F)HW#AD#cVvL-qA zR~*c6{@2JI5@oZauD{od+VXo$Tt4c{rvvJoYr87S^0|&W>PN}^`&K#g;(E$R!+D<4 z3h`GI-K3tc@R-ELrR+N4*gtVFQrVm1>|&eSI`5XK0Xt*7)r-2*q;Z@>(Y$Cb8!u2! zZ`+Q__)E$6&~S_6`b3rSQ-uu;SX(ptUKxe#ujkWtHZD`{Ufkx#)uR4*yw=~deE+dy zZ>7sJ9CgNLUvbjjF+q#raLRr`To=E!q9eF^QGH)Iew8>G&zIoGpZtAQ`S<(&hqy{~ zC%u*`Z-)Ius^sCOSDYD*^|_CFKjCvVEEKY1U6jJlD*UYPyu#w`K;?eytf9Vt#s0Lo zDC(sC6Lh^y9CdJP1)pDxD@LbjTy3&m2FW5?ojIQ^bh_KvuEbgB5fj4ov`|0v#0`|ehpAHiimgcIH#Hz zD1qx8Y%R&}j_SHvL>$G@OUj+Yo<+WEqpaWff3CmHWNRC~Ud@L>|1PlYgrgqHtH+vr z$N!6yjJQlR9(LlUta3YR7mlj`xOm^-sGH+dd6p3^l5?%FSl&5h`1}Q*uV(4~I5FzN z*P3!XW5-P2#=}e8uh`)lgsTe9M=g_?sW6%lotNnH?q2y0Ln<;3DS3 zdF+1)_op1285c!^v?E#UJmuK+%BrKRU)fbtUC*#+g3qn-wa_^~^X)?ZKEbxH`TUde zZc)zjVy%*xsjZAr&V5uqq^ff~Uo+`lTDgmy_q}7s;r4+2b-pVZmxvzb^L`w>h}*09 zyVmy^tjNdL)5;i4yA?3a-w)#L31xgIF8BK!+K{%6Eum%I?A(@oPoYB{?adPQ590f5 zd}fHfQ~q|1__&&;kNf-yE2LX zSE^-%BL?97MRAzw*jJUij9Qnm>oZ*artY`-7vk|7HZ4+j7F#B>W{5J5D(hPHzw*ASU-s;?(#MYxu>ypX;enM zlyy!uoE1gwlrvqt-X?<=`^Nca*}s8}Hz>P0Uca^<;qV0hzGKf{>RRO3%JOSFWZ~~pzCDiPMznkY zUv^bU~!&o@Vr<3gI=(`5W`H=5p#P~1DYaridI^!@d zC*!ZkU()2*Jrt-Y7PG{`_v+aZmy{=p}O!!W#j&fg>c+AS&_-NvUCVz@cm53u$8I8SV)tGA*ywF-SQ)m@j(Iok3Zenq>y-Z_KKskr&}Xg_yB8|2 zq+`ZKCG6K>d1i4t`;)&9VgLQ?XvD)Ev>U~TF3!8kdFgmrhLe6c_)~lo(Q+|A`muS7 z<4V(Rk@CtZuO=;?rukHLgqXcZSvT{mwfwpmUn}S^m>+X-T8l5coz)J$Ro*V;eV|P~ zC0^eYE4R_$Y4)_i*IoX$K^t_JG6T=!9G|D&3~||4y-w^hc$KBJutFPVqFfJw6Q?xM^+1h~KdC_KdpI}>cNB$u<|Kn?- zx>oRGqxyQ%?{56Q>-f3K&!hDY?5yVd&&qsMo4gQTt(35lowI0r7M=F+e~x1l9Xrdh z+r&k_W9QTJX=M$k(R*wugy!<8H-ARKay+)Bdm}ddPyHR(-i|JFq7-?OYTS~p-zi<( z-cDurOg2yFPvpFq&wt3hJNUd`{R7mKfZsfkchq^8sV8#GRdk%`@BPKmM{;m*l&pr* z(WSm`i_b7do{pOr*}uy-x%m8=CrjB^P3(_X*E;n?cxj3C4smRZQafr9*Np~>i)S6* zT-kNl{D5=X!EWe_r+?(uqxAn!J^A=LECw#&=RWM@#6^xtEbgB=YER0E#c|I0U7e%g zS-wtE_h#p1IB$~tnaIy@9de!aAdFFGiG^7{C-LD$o|hF5|FUOOoEEhdQ$yv|DfJJ- z@fUcQuFgB-gy;>i+JY8)akUOt*Ncm{9eZJ1Sy@#aU)30|sy?W;T&}Bc$kC3~i~hsG zC2Y^d)iv6j%`{3COOJ@%r`bDJT!it+e*NL~G+hO!*!LFuUS`)tY|a%&(;UB$Z?m*H zE9Jo$^^6ls>2ht2`kqzKFupv9pWf>EfSoVXtcJRls3-jU*ZHA6dS3b8u<3T%J&nh3 zt++5w7l~!mUs)N|*jP;-)QT2Ib>&L~b>zFg&~`ay51X%H$8~sq3C~q&y^5N-Z24V& zPKFkI@6V1!j$aIowE^wLW5}(C*f9WC*~&Pnj6LFU6dxw5>!jSh&M`gtAF(J$Ty+N<9B;D zwS;M`4P(1L%DD!&cfd;J^-@+ZetaC4bbXO#JeGlx3UZ;M(ksdH%G$~*+5^`PG@Z(~ z6d23D6MQ;9F2&~*ZE-pOs>%Jl;9#W@8ux?R1A>YR_0^VGXj+&&(Bpj%%x zz2oe8&_&+-!tSB`{y=p7pq^uV9?1F)aam&tV@R4j&!;_n+K%s`%KgnT1+@5ATm=sj z)w>9P(|q>?^!EKBiVTa)dBlmdPA$H!ohLrVh=-Fn{uuJuRnvD@;B~0>rXPO}i-Q5Q z=|Y#8@V5Fti*xlgMY63(yPg;|!(W=18i12Y?Cj2$D&lo%h->+CgEG>c_XnH4Q_m@! ztj1#rD8;^e;Wh2r9Ocbc&K1i0QJn1KO@Vgg8h*bmSDvHEUG!}Ve~FnY%5BVtS)$=F ze3jzM*UEY(inYVZ#v18SSDf6ftR%dA5tS8Z<&1%hn{gOs0=~lWU2zHRYFTY_S+SVG z+X}9=D&VD}F<~`1maSdPp;2!10{h?8W@O{)2G*Za-fYLE(m(W%ryU<+xE`Atin%7V zKk4`_bnlL{rF`5JC%fJ&MfX&3m@ZaI+be_Pict$b_QUO)i_VzUSwOO4~g|tCFD!8 zHlVC=ekJW;HSt_iuAhac+G@+hUZycsU446g^c1Ag=TAI6FMhJsy(~^+udyVwpi%e0 z7mb?B!|l|+mk<9cw~Tyfqs&p_dM{sYW7FAi35K@erYdf(f$7Sx&c4&^3T^8KoLukR z4f5z_adQ(}-WGS=_Iu%2fQV*0Ofd_7$pgF(2}1@C@vW%~`Sk zKK8f7S6zNr5JL+c7v>UvrSS=8g`R&Dyv2q?{As|SOl3ZXmG6{x4jg0m7WUo{7xF(L zdP=+w7PldOve_~ZAIsVD9el2glggNflf&BX7v$6=C3a#{=zqezz*Wd9RDZ|t`|AeWZ!AA-d7z*~5Ua_v?;SEc1i^)*)CVD*Le>-MOA^emr_spEk1 z%P4yRn=j_i!>~vUjL?qkasI0`=!=i%*j-yZ%#aI*op%w8gr8uaGCI&?8hgSxyS(!+ z5hLkr3bUtIu;n^&{w4nB^LL2)cdGvlvAmS+As4^IRjOS2AKk*(_C|GY6D>_~lIZyN z;d$I#jQ`2%x(N@j&^~B)BbzgnQ_}epXkMA#OT<<_{rakJ8JkDQ$Lndi1(#v|<5e+t zO#S(6t?P{I=yMNWA7@u1u^Q&brtqb)a>ujpY1qu4Tg3Y-;w+3+KEg|wpH7N1jXmlb z`{n5Oa--omnW~Nibn$vf;_EKZ_rvhP#0$7PD=y>G?)kFxMe{%@3Ldz3d>d0&d#$Ke=0O2R2M#_Z_e6IR7`|9*y(g@LW6H<5Ze0T%Bv=>I??RexJbX4V2)0oNQ1B77qL(_ zPSs|XF^((CPh(b?F4yMZt|N>SLt&m}GmQtT?<09PMSZ`Cjb6&HhtuBv`;L6Ml-A$I zrOeN#C^=nUR>AnKig>Q(I=qIzoJI3m(dY1Kw&5CT0Bn-sjo8-(C)>ox1M2uqoO}eo!+kg%ua2AHZhS0(c2l4-DdANbCFF1cK z-&@5o{Y&bn%&pNpAx_3>in`PF{iU^&W$0L@_?os*fshZ&7a-b+v(WaZ@@jjQ&wyCH1XS zc4$AA@uw3n7d!J5pMIyu4}5r2&iqP)u`~|%Kd$7@Kgz5f_+t0*xIp|A(kzbt7nRVT zlyJS2sDDT_=TcH%nIv}EO!HmpnV|e*+J>QYSZ_QqK)W^q=8CH~Xz~K=Qcs#X2EsD9 zkRPwp_z5{bL!A$)v$Jxa*Ux+*{sTGy0gmZe=_(}UfWW^wxYhfqV~6{KBKz+ zs)l&0rGKhr43_D;EaTQ}mS)T0TzQnQFO9@;BpwpAM@e!q#nCC+fi&asGR~-APM}iM zT$#(*y-d5;!VweXb#vvEWcN`hqr5D52)3!lox zxnit6ZQAoQj7#p4cfTm}bKIrj@?n1V6~|3Q%LYDO&$kK6e+rkQU_CDHgd%?SRNq~E z{KJ|3#PuBQVGn+OuHNnZJ1Dj<;a?j6KH}e>>hGbAp2DI6d_Rw+$!xe3$M>px7@iww zU%tWMYFy4zPZf2%fzNdQ%vDDZTpom1akxU{w&Yv5C(x3XjnvapJug6G{)YZ+svPHm`SbEWhd z>B=eX_zdIS^7N^wtV*s0E6b6}+PW&m_u|LWp`D!iUES-|-4{3C(&DK4Izi~m4~elV zyh+2$XZYC9j<#?`T)_4sZBr3DV(pAMr?@2U+}CDTGw~kg($@3;ZanCl~U~6*Y9- zS?IbaVpCi^pB?Vge5TzB_nE@H>`3;7>kp$~*Gr{vnnKGo*C%O?NjE-8mz(KwsIm>4$rWrLnj=8xDxwPuWv}E&K3Qfi3;b283!$KFp)h6;v8*huCZJLeRH1sH4Wu& zq1Y-^PLX-3*tL5JWBNpMBe)R5!`ZY67ek;mK0aa3ICi`Rd*pI5)PwG9*+8SK;A^&o zIjOK7Sr5&n++f06Bx(3OjdA2#M9N*P1p6f3VCz-N!Hl)z*V_Xan z8{yu`U2OPW-3fALw7PGUFZp6PS-nlfNIMF>rQUzl-&;QC;Ula|c!eept0T<)J_FB2 zxy4+ZjgMAhrLDSW;$x9~E=8Zm#m%|4530WkZffCWF|Ce>ms9u%s|-%yB9A?P;G+_( zVbg`Qx{_U!?@0t5Xb9icD6YF3omE0^DW#Yis{Q})sMZS zqPp6Ey6&yjl}FjyUHA6+a7wM6cpb*zKeF>FHWt7w>`S0o5zRgkEw{6=Gdu#{uH>a0x zoYK(slgU4}R72b!3Y4Dtwycqwr#mC?J zq&Z^aa=Dv^_x<7{OZ|hPmNxCQe2Mjg4@8-G%VcYo@n)9$q;*|0W$Qz7j1O|PUj_Ir zGFMY%ZcqOxhr*24TX-3QiL+oFErzr6W*l8dj|H#?mdf$B@zoAbe?miCErVTR=}mrw zzTmhRN`il&68rVy`o2oWPv%qL8#s-_1=_()+MYBub`v}E*nW-Nn#A^%?7o4$tzcTL z9l~p(INuaE&^P9>D_@+tpRYZ-iv2k}|Bl^nvw1R`?_+1UuW~herm>?1^cC0l!ZA7b z8LQrA+i%*&yYTxTZuhZsCOa=+->p<=rWdFuw#(4@DRyoa+vR9|kbRwOzZYkTFc1F0 z@qV%X4NJoOcqZM$TFnMHu8-S$+4#R$TR{Hw$6q# z;(85^&Z74va_vK!hjm(`*|dO7Yv}(X-5cSo3S5e_gX|b1&%%uTFY@ebsLP%(50(WT z@bxx4#h$kCCC%Gu8Ee9kY952D z5^xjLX6Lth@q5K$nD3ab)^LyIKX!f1uAtOjI{rq(8{-1&C#?iwM_J$0fx+-(6c^7~ zjbu}5T+-Ms$+=0zb6C&Gn|b(}&Zd{d+Z9k9cMsFGmwb69E-SAp;H;u~luG)xN^+x8 zalQ^`?`+r^yuxiW_Rhy=SVuRMt*LR2>lxQVunOkH^~6_wYxm40kBQ{ebW2m&2wokG%}u zVb=o?+WK4Alf|w}v@K0xFm%G%9QkkwyDGD35}W$qZzKM$ln?P41Dz||6XK`j2w z)@8%}kS|^Cbt*}=-R+r0H+Wx?YW@2E3NE}V~_wgC76|RyI3;41P8jH&|^k0eB z9RB9 zUq*?G6ZBcbkE;9_1bgTc^6CnuhE-G-@Zmpk(vICPvN2iioK1sFT5J*I9_JOs)OI#UP=(#?#nbbUW&eNTXtWp~&yQ4G zhdHIU_|hIui}3~GsijPwAfA55|2n>eaZM|}w5GvGcotgHq8A-jiK{!s)8%vsE4$q> z(_hrk#?)|(HDr;sefXcqpS?J*#iu*i-N=zIz&5sbWP7+)_>=8x*!-{@2=`le;CrGt z>4f7g>{%eT58-t+e89`0;`({r6v^ z9C4p(p3|B+oNkTtU2ilL=S5KqxS1_oap*2n)Cw+VSD5pDSS;MFJ=@5(^YM9)cpd@o zLVI@pELytYcPV~r+x{V@%i#Gqp2M7Jd0a2SbLdmE`LkNA%!ZY4kUv}avX>9Hh?$DG zJt=N-U^;vSjrdjw=kx6m{DyTaP2e#(dIg?G!6)LU93R&>vpy8S5x!1`P4FE&D*pGy z^|bj7=$yyTJncokIBRH3Zr%_F^C2L}*`<_O%6wi5 zThjQHX3jaySh=(|v^3tznBOVK)^g^btfAIshH+>fTnj7DVxQjD zS6ktTo7%4T>Nqx2zGsCf#%-3k%yxgLp6mHM?Oq<<^VuC4H$-AJGM5{h??}*IC+J^G z;2}|&iN>@g%?Tvw2TS3rlrcgn?RbiDSgLEsbYq}&W69Fi-jv2uY1b>}b*XqE`;*{W zF*t!`tKd`n`^Dfi`A`CGhjq}6hMVL?xWC$fR$IhjKD`b=$c>ru{|kPNplD& z&ChlGY#%ic*A27*d9DHSwWImwE!-#27JsCT$P|nH>Do{}hTL34(_2MD1@V}}-_v3- ztZ57LC9x*qXZyWhjo?rdN+I4&gYm(NpzY`o+`Vh9=2jSX%HqJY<_1WrvAK&l6?>O(F{)%uC-(l>q8Q*8vQlBj^v7;*9>%a`CC7xc6 z%Zh<=$}5Mb3}dD8VzWYY4*S;Qyp{I!VrQ+!^Ejw3W}Xx?SFrUV`CcD>#C27Q42JLJ z|BY&Riyxiw{S!Y9<2kJPdm7JMG4vR}mgDv%yxz~Bci~&u!JmXUHkOKA<0WV(6P%YM z?vjiRle8Vl;w)J|TuOVMqMb{T3n}(@GmP@ z!A_iahwu2)Q>@(2hsWhWdv+Q*8MD+az6Vl*tzl04Y`6)=!U5PB*VB(Q&}QYC8!hyG zq5ds$eG_ZXW7h=>PwTg#mP7=#nzPKXAHvpX>GRO&E_ZQaFLd4 z4F0c&DeMpH>bt@n^12m0KBC8y{8$2g>C%=j1#;mzeoTX1&{!;OqR~Pch4rOZ^X+`w zlQjB=HZy6mmo{(Hr3pWG(4{XwD}j}x+Q1ywRXJiOSIp&Vk32)g-_c?)k6uQp_$kzf z7qK@ozZ`3$O1SPWsf{UVp2K}QT74$>pQBUA^{^J^Nq$!5Z@Aa_pj;2nf>aZ4zw&h* zl#=&X@cC>0e#+m0a^xOXO>ek|WgK`LC&zYM;880W~VTzOt! z``N%;UISzGV7WCuY%K>H#n&4&`b(Qt9$tYz>GVTf62D1~NMcJV_l5K+@+aI=J_@;f z*b7JL^%$JwLs&byTkeEs8aC7KR@*QRU(biB?C(sg2Oz91eTwaS#obZ4vx?34uy+hC z=Fy=fe9U&MMa|>pMoq=pem-nt|E267t510>ZWy(Q3iw{Yjw1bJk+>;xogJAckBi6e zhgcuxteVrIDSx`f?&a{O248-|)^+?DK!@-w(P)^*k0vzfMH91^=32^X^U9gq%@F71 zeP6-#NJaOyD$DN5uKOxm!&Kcd)#+Sa+gHPww5E)&sZBXc-%*S0o*^mzOC8q$ndUjN z*~quJzP_P>xM@&4C;BE$*7NIqUf*rIm49IlXTP@2T?y?&k+wc^-4U5jO`vNDWBFuu znJ?f+Q<@AGTUKAWJ}kqFGR6NI(&K&hmt+4k6z;)?NBIzD#@o}R9v{N`yYC=FPBoHC z3)C2{JG%1YLVkpM?IZZ{mUbcZ_oe9bT2wdsn=gORXaS8DC@HK5I!dP%eEOJg$7nU3 zUz2UWg`fFzo=l#=pC#Ifu%2ZK%{K7qdcNI3w-;y@o@v;}$FRQQSAISqChO3yjW(kV ze|JVH)*Pf7zooIqdJu6q5{}dFcAno4MfNAc3%D3c#~FC&Ny{(AyOH+}09aDvWZ zmSTZ+rBjrpO~`hwhby}Ojjt!PA*1m0HJ%=!buCtOz)2YA{w()iHg2e4dxp<5`1}`y z^;wt4Df-3~a}_D{N?~iNu|*oL()9_z-^chK*6qAU>(+b>Ylg$P;dj2B;n!Zs<<}C)iLU#g7hlJV$i;HAKHLuTV5j(gli&C7^%&nS z;oCrZPKe9uSIf~cL!F+FqUR<2y%A=>e)y5^uW3K8r)!v3Ot(FWgD2^F2mdqadLQ2} zaUP=C-{N}$&90zXXg^nLf6l-`x?KTV`TAaL?YWr#oqmn@eMD@BYn5to z0X{r82V=!_4u3;m_>@>~%;$^alIBs8%t0l~mgM4gsI#_c2)stmSX_rSyz}W9)+-%` zu(s`PzP||F?T^F3ul)X&f0xST$$YCRe_wr+;aW z!upeEXc^|$!|ZHL)?5b{@$V}Bt>D{KK8@njTbP+17xJ$VPlbF9cgu_9aU_T0;`J%J zX&Tmcq(N=|O{d?teB8muD`_}EEsbo)L0GFYou>2nngHS6{UUMvFTc--yyHM*lN#F6~#*T`w1%1E=NbzcikJi*NXUD_`4)+gs$R71ZK55eJE` z3#@73Zy5LXh0!n@68Seq8p9wMt9`nOe_Gzb3X9!9G5Y z;p1(zUMrSg=jUp^L~1i~V85}bf4%wDk6%;ybx2#?oX+2i^e~UwjbH2N{Rs5oUujl8 z2eI7k$ycp{YukK%c)otB&{(EO2@$?x@fh2G{5V9l*j*{CUmbxSSqkO-lD4s&?J z_&cATVP5-hvE6}|VGb!g-Md&D^)uvXr*_Ag#weZ#;A>Oclk)f+{_f`Q>L^>j=PEB( zn^aG(h0%Kbs4;EdVckEpZ4O}_{3;fOHP~I@xLOk62iPT_Q*p8!3hlQS`{&^0Vz?cW zaPp#~lj9P`6eYx^z6&oc#79>+0vGr^55hWNPjP97OX9nvwk3&Dc!QIW;`7zG3-|oq zwf#lj%*NYs{5>L9F2Y#?bb!0yIXu3IyMV&`!&^S;>GC~m^DI^izGe>fcp;o5NxtcQ1S9G(vugV(d*3_d%=ROlc3?|{(8eaDYRaz2c|-r>t% zaz3n47{ZUqa-gFe=*5??cH@k=$hVy(9tMktu+~3?Pxs4%C*T(d>(kB_#jj!ZV%Q}v z-hoYQUoG~xv%g3@$U{kmay)jWYiW` z@#PRd!oBk;{8$U6?GL3_SRb}hoUaq-A=Veio`QgIpJX~;Ti8Cs-|+mx5%K=6qjo`9 z6aE;Vw?TFLN8oS%SJRgd;dA(RH!V|X*$lQp2W{Omco^-ddeD!qq3z$v`j_ZBO*=OS zF2~7vaVGyWjiIu%H(7Y9i?6!HbFtmz%7Hk?^{_P!@E~rQ;iw1vgQKtx`we>66=jLG zQ{y~i4Qt!98*S)63uhx?1m1?yeG}gPg(vV<3xCt$Buw&ow>+3AC+>mQaoH7*VVw6p z{>I~R4txateZCLo;54i+eitsb{{nt@;`e?`{E6S%@?t7}ztFaxhtqJ6K0I^r7JLQA zpq#iGjO#GZwFa-d;Q($w#^={Kyh0px7CTjnRj+}|8u~Bu*80wYaUJtanZ^`ZVm3>= zlcjC8)mel#hNDoM-yib(LH>qy^lytJt2gmi+B{p?;&o2P#ZfvxgMa7nHP5y& zOysMl8RTUJ@n6Yzm1ti{pHta&Vr6Y(W%*E5zh6}>S9RUtUY*ac;Ux{iJ*c)~t0RtP zh_CZ;vx3KA&2LAT0*AB{m&bL?BV=l?vYel-J;`y;v0l^^4vV2|xz=6GR1`Dc;jI?l z#>l(pdD1}pu?$|u)lQuK4NY;@S{(fee?eG_dl~K)YEycPqo!hLyBN9&Z{eQS0K9!8 zcADUB0nX-#k>St`XW=Q`vHrUNUo&vB3_lOyCH$KJpTG><6yRkiUcyu0ABc?`VIMBO z#mUup*oB8t+LxoSl)qD>I>vK#{5{Kfw65_)GvxQfd_KwN@LcyX z?d5HJZKoyrAAc`^kNEZ>-y%L;%%|{gq4;=`Zyov8+;$LL1M~Qq$j{AC-u^oNc7z+m z$SwR{%;)e1f|l?K-~SdbCHcMxz85zi#AUU6<$NkHKCPL=M;K=e;`=-N4r`ynGc8By ze6Ji>t#3F4b?xt_b36K$;_q~Muq)0HPq`S##f$kvI$JqsjNVYYU!Z>~(8m_4j=7_L8=4Pl;mi8f%b?FqTDoaVi0eXVx&J(`AJSp^?M5GOleM;JX{qSQB4a}O@CpI3SNeJ_zd?E z-jX-3;Ib(ztm4I=b*c_%W{I8pc{n+b_k`O=4-4Seg-KX)`<@BcAeL z2EGo9sZ85BxEg||x8pqXCDw{Um?J+|u6-1SGD0E+#Hj0;TfuAu_2T2TtoiU)KAtlzjBs-!5VAag!RYe z@e-b8xE7w}efWwb9h!BY4ZH%(wCZkFL@100GIT`wo$Fv&W;QsShPaa4+&@=heYtPv}_po#q+ za36j;;HE#^2~BbGF-~q&^XqVgr$fa_U9oWhFBiid^5zYwA#dj5rHaqu^Oj^L$Z+|ayv!{TRP=i{P+m?>j>7d(KEj##)D zC*gU=+r-IZa%qruxFufx!pmD?CO=N5d$Q~3WY+@8u9MuG7duagm059`^3sjj)A>}| zHH|(UAK{t%Tj59T%6+&9&tdGqMGZW>1fN4Y9Gr)bJK-4!&w(Go#}HhMz{4p1Kcz2v zODyemW_W(GHV($X!#FsGgDWD>*-`FC@e|g4eFPQlw-rNO#L{Qts2zMNe(K^SL+sSB zJ*Iux9_6_Yo3Bl7i2DL6dsa6t6a&_b!uzltPv_w51fJ%>bvQeQuQ1cF4cglOO)R~R zJFDP~%M;Cig;v{remH=`;W(TCui$P39&duh;%qGrZ-5WfoPkH9c4HUs(1L%V6+VB( z=id<4G9ScixD!+fo|KP99>x$n!f|Uk*&o8Y+}^}@H@FkuSKxWMGcJWIaN8K(#qBEb z)KM%g#A7eH8SWiyhF)^>2A(W~HLwv@;8M05gL%)77<&(g?oOE-%hf+vUlZ5oUjugM zxvt667v|gMYZtw%E-o;K8;MP0BHIJ_dxRH9W6v35>3(<@ci}w)hu|!+HyCFn@iiJ= zwZ9i{^KrHwXSYCG`Pmz9TcL#g^TpvDC4?1}H;cvP_?nKd@Lr^_-fRV~KEy{@Z}~ZF zmv49CVLu!bV|_3`L5!7x{%{Mt2RGp81*KMo-=P|=*5as}?H2ez{H>M=xj3o^AHZ30 zj=0Z}0cu;AM+An}wGz#9DY(@e|yvh7!1$h@0xPhoXzZ&(56;H3$)gK01y9>I|} zy6Jn$h=nrzE@KX`oIWE%o8diUFc-r68a@$Yy~NfBFbkK1#8m-i#^UmA`TC3gFs!5B zjLYM&*#15^qHkP*%fYt4;4?g@T?=82G z9H*YpC~nu@z;swY7uIprg3!i=^&I)QEQktmV;;~oT6o4LtYw)6kHd4Y23m^2=ipUb zx5sgF@wZO?UW?}ucumCfRA^}bYFLbCZ|u@OC5hi8eV2Oy+P}TxE*syWe|Qqty;y%a zgm>Wm4$E2k5AjKg0Ob!XMDbivzy?4ZQgvb zcUs(4gYmEt8pUU;Hf*m%Cs$uKL0A@Cwe#i_NK!Y`;0Qg;DSi4x3@GJ$wV<9&Q+u zHHLGcW1QqVDj5f!|AXIQAPxt?6fxULjQ)nP`*2nndc#IIBqruTa>i)ed+^pp94>{IMSl%wrymW^UoOXGxHs5A{4ElHHK3K)I~RvH!4G(x z7iAiwc*X*E;T=ei;B0-AV|jV&j0?(Z!^`Un%j2Yc@paD%Ia@<3z5?cmyL$Lr zsQ(>;w+#GMu>A#prA1kI-%tnW3KivSSo7UO%sm$34`=b(|NqkejZOUB@+aZC@Z)3$^Km+EmYM!VcJrpX*@_bivJiI7+d-9(KTCe1tVh z3t=`s9*mOkC=cN%gy&jH`}+y}+={1RI67NA8xD&h}u9{Yqv{Gy(zXrKYFK_`Uy|%@YEPj zeV|Be-GHY|Ez}|S#o7Dt6~-yy-Fe~pvq^Z`3eCmQ8frAtrd5Wc;%OD`YQXe3pREP* zzfikxz6oEoY|n?E;Ye(K9^OjY7Qzqs>Vm&(#a217bsBd`Vrz?ddJcb6@f6l+pAFaJ zYaYJ7!qa*@Jtvk1;Ash-+RDQR;h3Yd;9l4X=i_fC-u}bcPF(F4OOHT!*QF;Mw8NE* zSu2ac%6zM=9k5=(=Ud=WvGotWdO;Wb{iJWmfs^?A7I(La&RpAbA*}1a8;8Th+fdt` zxU7iB7Lhla;xRD$i5v~j)h5_hk*kAn`UgI{z!IDe#OeK_uLnMVm#4jKf5Yi7VloRR zipj-zeN!73+P8afJI;0&9`D2Hi?G`Mr*H_biFj>;&zX4q8?te^0gp@Y*bj$I;mat@ zAg8d>y8vN3KC>V^6&v2S)*j}1lX%wa?#+qzJ=9!}xq7M#?>O*07Z zH7$jo)nY9?RXqnk;W@Y!_<2;Ey@Q`Rupd7O+Wbq!TwS<2PSYNwJ0jhA>Fh3TE>fEo zYxCvj!*GNBHQKh9@fM!ry8(CI#aeiG+!Of9#8*SxJ8?D=M{^*o|J#eJ26)PV;dq({ zE5+6heZX>jy%~Gw6^Y7jFy1 z*=MrpPYB$Fchhu$i{LFdEZ#z&`2-G^!YA+}oX{@}gb1hIVZIm)>x+{i6QB8TH-vHV z-}1J)?Li#xl&xWn$`Wz;w748jt=X^{@0DN?3$Dlc7vk~(xEt42Wf)u6Gxt|t?ALdE z1OM{PG32{HXpOn}Y)glyXwV3L$GQKDF;9KrKFl>P!t*{j9VHselvI~@!QuA_D39ZH z@H&n||9T9!Q^e-W@B_5=xd)8G^WFG9X!{?YAHertFx38BoFB#W9Btxi@p?d$Qe6L|{4n$rO3sA9Cr?9sZ%kdY1i-_pz`T+R|jMIL@F$co**Txc>v* z!g+WXW-Gim63^pdt$1DrgT?eb+f+wdO^CPN7^e;58Qhx=);k_NBBP4-ve{YD!60e$Rmh12-I zALmtYUklnoxUQK5b?DF>=FuX&Co+v5t>`fXj?tqK?xDj#3j81sk}wSv2Ve z!{841hbGhMvjis5W~CgdZ2JoQs;vp{KNtloAiV$Q9TvS0ztd?SjVjrO_n9A{)t&GN zt!mLJ{O&+y`{%*Uv?`$4=kyA5V?*I9I8L{J=yeJ*yL9 zpGdDZ&3SXm%b47bIeLZpfT1u2Zj)ba zU=q!)rCpxw|7h1O%5i+IqjRG+(9r%$dA3%L?V?{t)=ZLHTj};=9JyB%o12KuGkA{| zO{&^Xp+{K965fHbktSQ=1Pwy_)>9sJ!0J-i1JC06ZTa&PRJQ*(yoUR5l`#}n;yn}Z z&)9Co^H+Gj2*)*Szl88R6DjyUmxoL7JsduhJB#3R_#gDea}n-GfB0vgQ0{|w62(6@wl z-BrZ2Sq7*NzdzK%CGz-svFNdkLdY^9R(d#K1Wz#5^M*qo; z@cd!}I$Z(J+drW!uA6?^@o8lS?o8G z2fb-@K0E+_(WblhDi5-06yAewg;CTDTG3)VE!whv0(2MOrSMw`-y3mUB?>Dv>bXzg zt+Mz&o5g3u_>)Qt>$;}I4KdnKy$!`~xKmr;I=Vof6}V4SsBN~E5!dx(UvKCK3*jF) z%iovNpeg<@f)C(BsOxX@@!yILx53x?qBD5khyMcGOW-RS+=KV);LbSN{l{dvS4w#) z#d)$8=SLunS@+;RyklY+oW%c7I{Zh2{&=4b8{x{hY!sgV3+q?Hd|6va#rcak->Pr9 z+;*KD*^K-9a6S=3zd?QbcjA2stkf>G7cmdf;3gW3fzXfqAy59s`?vU?kNf_3{|x6T zkQ%!OiT6oymVP*^cs)mJ`#;Em9&t9Ev$ZeT;y%Y1Cf7Aiu6D_@I<)bYFXMpl76koz zEbie|I^0f=f9TK-|DDBp3Jt=$<+k9p3VwG(=3g11_>Z9=2N&iD=MV?Kg#{aFc`!vOs5#cg;G%}?U`J?+dLV);!Revh|R_}h=SJY0>z z*&%I8b2zA7v6@jk;2pBiNSjg{Uvps{92I$K&?SUt|v z=Vh`#%N&0ed$JspElzWc&vRLni*vvE5PQEn-Xh!i;@I1Hv`2Zy?0IZ$=$ImHTjcMN zzAth;;Htsr_wZE$Z&}a;K826)748~@_u6lOcVVu-g?Y7sjy|H@2ya4PVY?6h!Ra_> zOoFgZ`3u~R$G{HQ3t!>57Yv2fP!jKd<9x7oB|M{@E4EAF{#kg7>LcKF3|$B_X%OBs zagYY5#P>Ga55)K#^r&DvLihG>|4uU>xj*vuV?YC6~aD z`0qi70g!9|YS={&cOB%e-|&D>VGlk2M}r!)$b|6T@z5XMng((EGJLv*^9UPov( z%TeL?wbp9udeH4By17z~&V~!&6^e&9WiEsF;a4~fVg1(x_#Yj^Jnm-!bd0ple?yV| ze`(qsZlGf?dM=0X?Ef2d94+U5fV%d#(lV?|Uu^p+O*hbVrM#Q0U%wQy17g1uM!&VT zAJgxy$a`sF3Y;B#|C`(!5!W*=^xIN&>P-_XjM!d4FYk<^PsgaxHMOxU3lpIvTmr8^ zd3tq#b73S*gH^Hl6Pg``py5{fET_{cc-ww>clFcq@g=3*N2hz07@k`^gT3&K^0!b* z+jp@%Tn1a|Rg+eq(W;16jo=#y_YgQc3?GO}IIdUq=ffR=Hte9x7u z-r*nD*ALg{YXkY_y}-8Dz|lD0ylnw)3tcM|x=u0gr|$}L&f%K=S$H14hpIF5g+42fO7<2j~bB zU@lC<{pYycE!J6KG83G79Q`N@E)GOY%f2C!*&S2QSvPQC%_H}_t@&eb?_+- z7SSRN|D{Cc5V*+xMmR)|JLvHORJK2rE^TOX13WKRmeQpoeXgcWFWL-)N1>fOnM|iP zbQ%r|;6++}M4zUx5IV%(!3nG4OxLJcVlIn+b&dDJ=p)-WHQQJ-TYqZ~j9xcJp3jZ^ z)>xcxERrwZ^3At4l!Fbee{1NwhWK@_iDolsb&yumXmc6-3V*>U+D(92^eRi6uv+h1 zxpGRLG^fvKInvYiLHewx)0cAONeJ^Tl_2~UN-McAmo`PVE#Mt`w5H3ma2%@m9M&C% zcX(BX+4NaMpReV|4bWTrRtH|7Q&`s=o)Hf3ArAA9UFi|#A|InkADVmseeH*PprO?m z1=HXWpV!c10SzkQ{{lMX;5_^m&j{NixbKR^@LNgC;dQz4FaB59p48vWl`kFRI`Xs* zt?Rhn%j9pSYxOMGK%S+CzVN2LXCqB+xBZ(ID>2rK9-HORLb)>y=F6WZFka5gqfuXY zk~YKX@hjA(#V545UdwU-Ud4Y0THFrnp^?v*(4(sS=?r7xS*S*rceG(Q+b)RBq0=V3 zA0mt&!uy+6(&a^eNwa-A=p%X;!9ECU>MPJ96E?#k=t7I|4oPp2H7+deUXeKxdK`r! z`zL6!MZZ%)9%ad+@EdArG#aW6J1U>fqthH3y&->|gqP_uK&_+UWw?YU;kmUBd6I5> z9ZaB43)+M??)b$HI@K`tt)cC#VXm*H_N}H^^NuFCi7vC~G9h-&PM0*>EEpYUMy+^q z4qQT$6Rc}Qk2*A%2$kf<%qUk()WcDI*IV_C3+lVx$}@+Rr!UVJKMko^;CiaixrMZ| zHkT$ZVB=~0XVc+E2xEXwxW63kfrC+k+)WTG3EI^J2o{$Nt^H-z-zP#iVcIipdy_vhkIcfy$a>hKyBV95c;0BG}|V3p4YCGfZu4+oi@W@ z6J36$&vDv3Ntf_!z*yK0AHc`(8jVKKDExj|4f{iBRYxvegZUe1bFX}9r5$_N_GQ{! zNt+XN*+2_-AN1Xxqoc=Qcm&qbVj8@O_o+1a8}~QjzPatY_-}&I?Y7^-pK{~nIENKE zG{_Z4xyCBi1JPz8+zT(p^|bGPr!#EnV}(XILcTQ6H+zqw=sX*m)2t8dqT3<5Ey8vi z+ojsIv*=fYZWHPDxa}d@m7&{o2y0{~`TQXbN6_vKIM;r7r|KT~3J%jSc-s@gZ=bzM zyWwz%esihyA^aUD@zeXqYZgY|Dgcl%*|_7yZ9Nb7ENUP+z;Wo(ld0=azuC^^a^-`{%zne zq=s{uM^iGR(V;hl6w+@60XmHjMLmA|1o~ zd^*9+^t??TAEICQeUo7{OksIV*yF!OwNVqeQ*J*()6uM(Pty}Ly-jZKq~8FTL%*){ z3-8c4mxeXu@gn*?7t3XuJuE+a#@0r{&$OB+Hw)v2^lzwN^`7#$klx+}0HzKc>z9tSU6|%&O zx7FF71%J}57X3~`CHo6#_;-};zD|y7`y73Q^$*a8o&#V9?4s-TxV|x519Jrp_?V|% z$dk{0V}-sStK}&g7sds}_lYLhE~aysvw9c~!M`;95)%*5@Iziyl4~6!<4HQ+K|{X) zYtAOgc-TAM=@;^-Ki!^yux@j-&$rO;F}j^kzYes!M?U=wFWCQ{e#7YaEex}N9W75| zCdKw)I78F$oZu+=H3(amX)i}Y_)XZCX!;I4pQoMGzuJdNQ7anWVf!Whexu!Oaw)u{ zF|6TVNwcfy7k*3lZ2P@nHoQc?)^vOYo~2`vTq|SSo1Rt;`>uvQ*Vmr_in}+(^H>$e|XYDQBK_M)<|(@E)fMG<}~2@2fMcaabd+yQ%LQyf0!`MKR%5Ns6C6 zDy_^n6n`oVR4dr}=g;i?v{d;0Fj*t^uV$E}nX-sQiMZ%Lsn=ikHELu|b^O2Az?8dot_ zS);fO9i{H@n_ffkGfzA5fHK0G*zm8Pya>PB`J4J$u;(f~hWVKA&WMjAYnZU!fFFrG zNOIh6tiLZ7!n0rp#cfx-e6IfR+loDK*ag3#9i6Y-YP4MKoCm~I2X<~|&mGPQ_v*{g zy1nl<`+HbJe;qx#;J+Te!h2WWb?mFkyn$cW(BWNWUKtlfVGYA(O0=YLc%Cf$HuF>* zg>hAQb}O52VU0*LF@1*eKdC$10XPrB`^t{1rvyH`v2(3F>Y%=p$|~abyL=hs`)%T@ zL+rj5`@;RdLGXz(uV!;Iemx;4zQe&md<}8#NOgocz)`_3b^M0wp13F-71AO_ewEfX zRmZ%xk^kY{LgCqpu-<+QUlQq6m7d`@s0JwGGe~hvc&_K5<9-!SAr`;FK@+iiVH_7n z)Hrr@Pnlf9b!;|H?_qCKwuay9 zuEO^pFh7JQKRTxXT0jqcZDak#{0;9L3cr0bKp9zh56>#iXX_ZbbA>qmJ1(JbF0GIC zJOZw72)47enYQ6MdY%y@Va}qiIJl7wkup3zDvn}v%L!sWmBsE}vHgI0yNZYV*<1xa z^WX5CVl!Hmc3xMJQ(l`pO$*jcT!b}Bp^Z79?y6!Vyf5k!_)&~}q`qI-yoS#`Sl%4= z;3_;75U!cRvn%1f+-qqc<^-C<)$DG<-Y)9AgH9pVTC1}bJjU;ATJ@z>bvkv{o}|PD z#dE#~BlpFXdstaN(XTyUx{9)JefuBGVp~6aG^bq?rTL{|{l7IZ_z3IU2Z@DE{JPzF z;hizzeQL+leXYL_!gXhD{$6LDWOH~{WRrT*@OGFt;hEu!a8*|M-SlYTT?_ZK{Q_7{ zz33gVVSR&s`;}zp?Yq@H;#& z{A--Yl05f18)}PV$_!8A)?&{D zx||aG|HsjtfLk@S0RTR`dA?DJesXVh?`^mPB@#lES)>q^Qb{QdXdq(~B8d_yq>w~P z8bxVNX;PF%O%zH}Df!>e)AP3X+2`DI_Fn5-Yp*@6^_=gnEM33%-L0hQN^#NCZ)}j3 zair5NDfHjHQt{lua^G8A`7S9gkNBo|PCB0LjVELN_8af`{-x6Oko=w^P4Bb*brWOR zraYr-d$*?Y)CUC|L&ecVN%wu~@5Z)Q6?3(1f84(jcLcxVZ=Sba*UIM(?69pHd`I-P zF-*G87IV9O@>TizoV0(%-}RQSx5~5H)@P}0eUY+F3D7tg@nL!6hKVB7Mf?_|I8KfiN<-`Q>Z;@WYXuX@9_m5`SM{Ps{=af9EB zy0C>DeKRSn4P4ag5~&@wE5;&tP+CSCt06H~{0`f9w>+LhXX|mT^@y`>H`>;?kEW+} z*v+}}ypwgg)HYvjt^13sTjc$7)@v5mS)cQ)%cs_(gZl9Y<);R(+TJ^*<0`*5S)CrE zjGwgq(Lek?VYAf#~tJi)QwgB-jlXvl<&Pqy2co6@7t!*)}pTOj{V75%EMaUUB-9E z)#eSf7bg>KN1R8Ba|;9IWt?~Y&hOOoJ4O9YeUWrGKPN>+by3$}ib=!b{;s%vWJ$m2 zeka?yS=`Q1&wXM0dRdQ)#O?j!cBu8(XdAan*C<(kOUpPv8TUk7>$e8_t%9~~ir@QK zI$rFz;u?HA4*Ap?`B|0A{MN6^YjJDw1^?LYJlj3dZ^wS6DAC_ZrnDX|PmQ3SihDs$ zT8}l-x-W4L!(Z}vFYz4Ox6062bQE_L#M@5swnsW$quy9Aebz~r1ojWTj(V%ukh;|q+ME)<-4=wMGftuno=)E1m@Y7=l@qn zd~Y429%^L$DoVe&17(slTq6y4NV_y?*VuZUkaoSS+u7Fbck6a8UF7X-QUA6Gj$U4N zt4B&%$9U%J9P72;r{lV79A`|H+LP!b*2;^Za;E9>I9J*2BX8@_z&ie79S0?~#GNr$ zO+S>jdS0+ziR(uCb+WE;?j?>uUX_+Jq~R>-x7jZbl)_uBV;C{&`MXK$Vzc<>Vfu>uN3~QNTeKN#B7294_-Cs=}XDVA+>VPbDK(_k0 zy6Z&M{gyG0t@}yo8h6FU{yokq)kzX zziyj%i=WZNIl~x3`(N95#5R?-O|5Llh5RaSy2zIrrgLn|Ys%tQ>sZV>A59te)_R?j zd&NlpxPO&av)ii=ztGwYPO$EM=lki+ij~zf=`% z$0}kr(^Bmgx{L2=(zL9&zSBBYR1SZ!UJb2RL+Km)ijmA@f66!yJZK#slIP=7g`F2F zqPS^p;T;SOav z#xZ}84a&yTzWGye@-R=!k9*`r0amFKJKEB=wj}y`iSsN=qm1(kW8}e$wt1p$o*~_% z&!ivOryMMj2Muj|9n<@4|Cy<5ZO`iV+s1B|2GJMHJ+^;@JbADD=28A{it%kbM7;-`ujbj{ngy)OU0HVx;U%Yr@rYd|KprZHJ^G;jIU<1^jgV3((HPBo#E1~gcvwi zytn5?^RuO0x_{NB)lg~nvh=!3dc`@i4(7j+UJt52cd%P}#V608zqE=z_`gfl6wkG! zX|A@c;|k^8k&YC}SNEozBlX;cn7KKXxYlL-L*_}t-qP@pG%W1jSJJ2wgNXip?&N-H z)K}UxR^AH}_Z;0MZQ{K5CDNm(w0PRK-(YPD+y0)mzr5`~@n6gEEU|Q!dd)u2H^0Q| z(qNl3h<*;DZ99q`VyBFBNTfl&wf;ky*e-Qeo9+`sd!$M9o%ODCcvksnEiK|+J-vF0 z?M&@b*C52w4br80Qe8e*cb-#!`Rc$Gez~e?Bk8h3x)hTx2`?&7S4)!~%2S4P*(WVZ zs2gI8fOwuH#>b53M&4qrbf`@sv2z9);^j=A?j&BGV4-p}PJNzlE8_g{*XB2|Tl@M> zu@mEJj}|}YORs;$%+=CqtTc*a<&~y?h>>f=#9h*10{@DK<EZ+?~~in#=hLP zmsDDEOxsF>c&5HOmy&0@jV$xuxq$n`!lOw|*FxRnE+&@Q-r2S@?$E2u3J%)l;k+gP z<2vSJL|@v6Y_A8!=N++GI$lYP?Q4ucKaFE^Z#MWg}TmS8?;3bW1nARNU0{SM%A*VdW|M6^U~= z=SsuJ#nHoJr=>KE`nI_FRm6RuN0Myop6&Ryx_GGm-}$A<((`4p^}gw9aWz-ko)Ak@ zrPP&lq!SNtJW0f(ev+i|XX0y*a`u$;jb{UYOchZ^ia3W;)IC&1#d|UP#A43f6t{0E zAyP`(PnA{=mG<{#9JA`rS{)tFi+{*!qE3pj>00vvd!^^i{&F}Aq-$ZZb)^^^FTTD} z2BW95uc&E$BA-du^5U$!c>7lxo<$xp9@lHqYMhvh=XviCZ{J9-PNrXTT)GXH3;U$q z5$SY=bQ{DR9+qBle7}ikhxAwe?xer8+N})Mk^3)7v*-(`H5)k~jb2wDHIYtVi^EZT zLR0bhf;7vMW-YA!chakTQbfDMwLWR~8v9ry-HuDYnNc>yWOZpbjW~wvB`&*(&jZBu zgbz3*J?DwbNY_JRa*3EMCniOdvQ^G8OnLiXW0y(8bH!xzf!~#L)N{SWXP$JuQ@U;= z`p9i3E!`C(z0bB^H8!pEyo!O+@^#;^k)NdLL)I>3`nPm#Cp|xqj&Y8@CvS6r!&H~1 zaU9r#Uefaw<@#Y|`eMo}(>~LMI3%8?uIa)_c%pk>HAhne-9CN z((euFSkm+n=@@<4xRPz1OZt3C`4VT^OIzbI)+MeGpXS*7bjLO2rA9koP2QC@-A(`Ch_s0jxqjt#^E0H)nZ7aZ)7nWqpSVKY-6g&5Y*SQQ;p3ksmamV%FsdVR5GSulA_D9BsV4c`}QM$e%4X3j*mFfCwmhaN< z8Qsj!lAh}=8KO*|EmHc}f5sW8bm>}7`c_G5DP!)h6R+KD$*1D=X=$A9SDW&uG+r%c zTT0g%JZJt3aa)c3;f;hpKqbp>f%S{iRvu9qbG8IjH}E7yNZ z<2b+3jzL6Ua8HQEOzGWHddFSv7Zb+@t7tC{yGhx}Oy^T+{ij$wQ~LfOo&TY?`FYa% z5#nBefwuT}aoIqcekBb@CFycH!#>oxP%+s|+BqAhEvg?hzkE!Zt+Iw6NT;rR%em5R zh&1aW%_f>2lx|l`uft3bkNu_B9nx!ybSf`>&SVZHrPK3?F}YKTeT8#Y>chp9Hs4n) zZWNQ*rtvIPPieJ{17dRl-$=Jj((Qm4y;rnOm3GCgc?aqDv-In$3^x_0e=Em7Nw>fF zOxl%Jj_X>BXsgBZ=KZAEW@{AZ)!Im(wyY(t=WZ5%e@UY|q|rR7-H`p#>Q8a^jNGrr z-_ofFop_dSq}i=0&&HA^_8yjYPZ8Hi<63k#-c!co&Z#uhHC{B?IUgfptSi4#vu zxxwpJ^k;yy-kvHbj|)kaLUOQ>JT2sWYhhcU-!NNuMEchevp1SPByO*i(uwr((nVjQ^dQZA;%bTy|(I3S9x$#WFHhFb{e45J_{3)NVRDNEQN0DFictBpA<(HNb{RX@u z&*Hg{9QpRRd|ED#=81r>d0sy0rOa{hS?+CeeT?Wsu!sEGp=@1b`XWo@S@a!HNExf- zFJ2+8=k?@mdD%oBE;3y#|GrQ+WhUviDBbsEh`|i|AmuKp@?SnaCeMmWfdTTZhkPp` zui|=KAFeXLMV>_)WIKE1R}%*EhJ1WSUVSUihRUy_sT`%rwH=1?7NzA^j6JhgYyLUD z;SBkAQodD{f3;}OJ|44dH1S-|T4n7X)AoEU4^Nk$xlCjs#VkvwKL_ON-SYRCGPhjb z{v&S-D0^|!R1@-AQAwUlpz<s{-7ZxX(!bY_yhFn4Xd^xg3%Qm&=cFZ0BaLf9H_A zxR!C!e2sdq8vU8VDS0qJ9(*qSZ?#1a@dSDDB1>M3BkIF?@*?iFe$n&?d9qr$+bdsg zR=(oj^&I6=`O{qf94DSvY@%#^ERXI?IR`I~mdKMhNA?gY`Laj3io0I*%IaKaImZL# zl*97Q?|F_#-b7zBx2q3lu|RzoqYTHL0rzot5q%O=UsH|6c`bTq$SzTTnCJ;zw$xvXoH zqPFt(cX?Y;-p)5&oD{J&MeIw9xZlQD#_}*v9`c&#16qMT;82YoHP5`n#47hn|MgR#hJ=H zV%(W-UPph0aoHZ?qXcsR~-lE6v!OUO@B`w=1 zPalxCBe~1`H%X3Cn4^uaPY?OkPCgyvZ25AFt+|^A)RQYY!YE~Ofjo-qevOH94~y8y z`IcWxd;XAzhveOCdDlta-D29BvG#x2@^QC3Jcp`0N^$vFg#k%n=RFH+{}%SW`i$iQ z%cjWRrgUcr+t{DDe!#12kk^;X^XSv=N#a_^ZuwnE*^N6;zT`*FNZeyBzaNy}GZ>Sk zX_KW%`?UY|m7mJ@7$f4OJYOl#56kmPrtwTDGIy@Ykja*K!s zCd&WlyE68lugm)UG&xS35{Z>ukM8d#%k#5LIC z;-{Y2$uYf|b>e1~_~{}xuJ+3}(vFvz#sij*V;R4Rom^`?fYTUK?J}H<=7gue?(h6&PcB+UxD5_>E>Uo8tj-!eyNBWDj>=yAcNa={DR^ndEt(+rX zdZ;5jX=po3X!m<&os&EwjwXnsDI#EUQrd6npMl;CVYhfH#!xXdNh}SrPl$f4pH+9n z*@H%G=2>wR_b6U2hQ=|S6D$==Q|uQu(%JkVri!QiYM*YV(ZB2-u{Bd%U1fR=F&0uU z^~_?TkLFzHU}O8J$Xco7Rb%lxfkVh~GChhdBF{-ePSOah>81F;`jr zl*2q4iMKZ5?lAY8e}aF--w;MIUA(>S+deamXZY8PvwAF7Hzic3lR7Hy-`Om#Jbfq5 z+>0rmI`WGcS|)B@7BjnuF@oALL99#@E1lF!tLddqDlHzq7aNB;$NV^c6dzs0#k0K2 zbT)91T<^WaOb&^Udx`$TCy0v}o4yf`uw6`CC)Wy^e#F1xBYFd>X!@-fc|dG@%kQ-I zI>sV?f5o3aGGlQHpu2Ca&wPwftxC^qsovUkaLU&eIGMTc!O~WA5WI;=Z337){)7 z^b4_2O=@IGn&?8hd>&%lS<__DVT# zMO;TbD*oO{6;~G*mo~-ir%Kqq63)4mPzFjU$0gJQCFN&H?Oyjb5#t{m5t~D;SuxXp z#AiD9Ci)6Yl~K2qQHPYVe=ei^xR+Y}xIm;fGCib@ETN><;SPp-{i>MWp@cnS`lGlm z$?0t2pct=0C%$GG`|O9W6zjKA(tIY-SJZuCzlgeYCwb;4DhK&INA%Mb_nCd7Of)dP zfw)gJo{f5%jb(C<9ve7}Lb!G|Q^O9xnFjpBVtGwLKA}V-2LwWd9)Sh7){ZN;r zC2f?2>(!fsOot@t;x|22pF-v@qpkALFO{KfnxPJN9?I)F%x42968EVoBWE)~nTX@# z+nLHDPFI)a5$*bqlPqmE{mF2S`F)9fdn((xr)=AqEiJNL1F5bqGA0HyDByK*>M9Rk zDI@zylA8WX-AhwCv5q)atIY}JAxY&pzpL*m$|)0bIHo*AEBtDP@_tZ0`YRt*Outta zE>R{Pp%4V9|xPNYp@|Dz98%YIN^GL(mGCMXYeh-Yc1aaN+Q ze11_bE>bRTG4011WudHyxJulgn=0zq+!)ki{6^ni!E|cky18h4hx5!g=PR)tSLE)d z9Yg4E*>-Urt^PXXng3C||0~X$F7YR z)5XN;js;Fv*Oaq0<*Z#f<++^iENA~#PQI0w-=5Q?x%r+P;iy<1q>XTw>HEwPzfW4b zarS;K#qR?3`}ag2bo<0?8S#4=mzZx(JM9E(>KN;P+T)(#^!jUY97l!2n8G*g6VG3Y z>t$q`Z_f&G-P%@kXBk&`{kHhNN(x`YO~iBA+mg6jKHYiybm`$aBe{1MoA`$_#CAMc zI)wL$v29`mmW$adwr`^!<2cC;Nj2wUTvy>qwut$M#Ce=s@KnKn=d(W$$3w+%Dbsx7 z`T94-b7S$_ind8D@7Gf2=g9XQ*AI=g%wNQ~r!&NHb1@zD`%#LD>5)tmzvqj}^SF*S zbml3ZW+rDPDfcWV>h8q#nMD2Pxp3O?y4anb6tq8e3}iYhaeg-CJO$U7KO$DI5~p!L z-8QC+*ZW1oi>xQwvy+ly(!7{`w6XBmEq*Je+{?pI;{MZjEE~rmas0lRZe$w$0(B9? zxu$jbnw3ekmYiu2!F0v2^F2Gn6~eO_-h( z+gW0|5fSIZI3T{=g(kYHTHmV9A$oRC9LJLm*HGVl3r2B7YuKGJyxy!Xp&R@&t+?~X**xo9ITigccA+27 zQBN7zBGxO5;M(lrT(9fXnh}ZqfR%ykS;R8xSvH*gNrvyvP(P^um4iyk#4~JW8;NC? z(15EM!WLzvz49@cWsLQDTH^d=s+x2#PBUxR&#%5cMG+q5;>7baEK_du$S-yEgBmI0 z^*ZI~K4qsdEm+Ji%vFxAQJ!M#nHfy*I{N8;P`O%9TnD|Erlbs~ zqm-)+VJu6pKVUj>e@;g!{xVCGfHROgx{d+_g0A&s1gaN#$;avew71J;R&C{f5if$Pda}8D;Qc zR#8xyyOB4QyLVHiohvIN^~;FoGTNPG#7>$2&cnTH`FqM(+}*y7`sTk--t=a#-Bnf^ z8OxMgl)d>JQr6-b&>Yj&3{%z)DtFyY2QZeC%H32Y^jkJFMOo9EvVDVoNja>X8F|?j zpJD%fhO~}7Xhr9AD*ktD@VoNn={9A|^>=nFSKE}i%BJyrX?rFyl|#x{ebGFIzp3GM zjN$M+8h&+bn4u$PmRwme(8OO(I5+{s32Sazqf*j!m$LFQ47yU;uFtfz8VkK0K1`X1#i zhgPg)EBlnUHp<;Nj?+=uds$gqo63|Ynd*p4_daE+hckVrds>yPescLo4$;8tHc7Vn zAlrGdY`ax?BDtTRD}~)DEqwtyW9B zsFwYIEp=KgdFmM*Wii(@pKDmE>|LiUPUb=LZ}MZ3XIt{L8}qag^2BVu^vPFl^R*T7 zt#7_|dcJ*YzWmE~&cJgd%Hz8%V--Cu`-G#3YwvP!1Yfb$>$unHA7%6WQ~~K=oOGtL zpLlL>v@)w?IG!q~j1{!swtrM+NAoG=y_Ykx_k3R_wY$)tqm;4i*Q8`B%^F@r zQ}cH!za4!07-HP|81v~i<+-oYkx6wfqn!ikb+0JSD>_cC=)6uvZJ3JMmd0;To)@u?`d%+%x$+x#2Mnc^`MB=(1kpFxC(7^J z%5(Hn*@*SZtX`|7|Nk5ly2q7%3}Ge(yLj;1FaZCJikAd7#rYU^?);*-mfk; zt7~((PG#9o%6@}X4acK3?BiAZ3gkI+_27 zzm@lr%KnS|%|BGK>{Dg^GG+c+X0w`WEW4KNP z56k$bGTM=4l&>=CnKEKlfBKB(ZPuwD;=Tqc?AR&Iej?32rmQt9tIU>l>{{0Lo+dv} zQ|Ic7!TiO%%X;-g6ME81Jz<0?<))lAP&xaP^75s;eQSAbi3+iXzNvy7G#(YFsAG9| z9-=of=0X|Y5?6Td=3)A=jw0Smqbw8EA5+yQUr^Hg*XoTy>W2(2<7#%RAD&THyuo6A zVu)oY)dv;T4bdx7^x?CR1DmPi5W+QrZZW-Gk%# zIO2No{j5{w%c=|RqytmPuNTkqcF9cNKL~ zT#c=&eyFNVRaJYpTBs!lk}Ci6cfaiuk}<%zU21{sYxNe%B^L`&j%*}-g6Csa^B z#4}q@@h_Q{tyM?7nDYEPOG)#(E~ArpGOds@U&yu>vY#)cj1=~n!eXYdbHT3E@&IFq zu>m%*i{nWV$1ldt;2R2geKFC`(oS_oT(^3b?}&TXD_C|e7jhe~Gdd|IGW6lYMO^On zd#q5WG*JWGWx6TRo{&5D@Ca`(i#TpC>b;sYa0jrae_I)L-e@SiQl8zTjiS1JI zTVLUvqt1!it}TO3El7Wl|S2hea&sHp>?!uIs1L)GcZ9G)?-YDVO>uQRgi7 z?Ilbb@~`@(pKs}5dP4mY{pZA0rkDAXlw~#eS-tbN`sO(K=3A+Enkl*6d4}lU@)+?9 z{o8z`p6McYpJNztZ*^;R%Ky|c?f8+RUe{xpx}~1l;uhW@`j&{X)bCW^6xU{nC&U}E znhm^Z`G@MBJG5QeGn+PEzf3%jR9qc&C4Csr`xLW$v^uDWHJD5ECG&*WPZHv`Vm*NJ|ZzEdZ~ zsCVs|%4`-;!SYPr<<}&h0Ljqy&2Ww;LtU3)U*yhd1~G$qY+`2;J#$sD)m7}XtGG|D zih4%hX1q%DTeN|n`9-~^#VAj!DmPW7i(?snV}SP#B-I=%Rn%S`K% zDep4HT&D7psou?$znPA)Gv#llV~9*;D@#gdi9ydJ(>RIV6|((pwt6vJ{AQ~o^^;}! z1ZK0E-#Mnvdqq9?4@JypFo9+2zJlsOtt!Vc`mJCltNDv^-hYsIMr;hTlA6xH*0f)( z>APy$*VlBsR?{(=`|Q+}6V;7-xxoAY^${-Bj*4-(I^cUGT1#`CoG`c-E0Jv);)6I)n$(#Hh9aG2=-xIT?}jvYx6ZLcEM zxrlW(E*L*?h2Nm%G-iv2M7AD5< zm5NJfYkme>II5o2)1Me9?Ve%tg4`vjcG(*2`gF4ekysdE$!&UF?DqlwZ; ziT7S%k~+JZ`a5M>k&e8=N%eI+G0~V9OZHoyS6APo2J6W{;-2_omfb`b#;{JEeUG}k z9Wjnkj0$nKy81(P^i68Zula*C^>XxJkxe5;F`3Pj@?I7r*sY$9f4%6<99B`x^5=Fqn_oz&2uRr0aQ!AJ~$lr_N8MTR-;zF@p0fYsh1)Bd!4+NHUcE4Er79 zpzs4n_?rseyNqtU&)59MF^(ryv~R138+W4-eK+;y1%~q(bCVc#P+#@jOg!sy2lw(m z@qBrGpJ~M~qKc zILT67S*&l%<{PS#ECU6(%x=ElIZ06j#!`sh=277MB2^#OH!HNOyda?U2*>nuhyJBcI4TB4?w^r&%*%=WfK+f+!!{LF-ivAn6xGHpDt(J;yA{>$E|#j2DrUQjNnhvt zILos6NpbfK=!b}rEM^;fxy*YFxsTUao0L$7J(p@4{X`AoWa2tn$}@v3V-J-q>%n6@ z$y1CbuD>lxjHw{C;%xo@_>K)6CCz&mP?ux&8+z4H=ap7}mUjKdH95=fU=AzT$o8a+ zy@h9Ed7M`n$3*V;UT4M8hhBwhA`3U`T!ivJo}p)?1LU;EOBfR<5|Vinw|NOv%H?l3gW%P_BDg;f4(A~@9*jL zG5els{_1&VGK&=ywY)96?T5~i+j>?PU&g}n`h4!>P2xVniuO(AO>bihN9}{s?UVZQ z0_!MfS@h8oeSyZENzq?lH$GqrpRt4JL!cVbSI`E2;1~O#g7!=CFZwKepC9;z+r1ah zm43z1Bzja%SLV`{iF9o;PkmZ8Ez$o#%JBum7(-LbUSJGAB~{$d?bKDxK0-I8X2u2Z_EPI(V-Ki#e9)3rCIe0KJGYoXT1j&-gyZbiU(AlIxx5 zy((9n8SlzmVH}&p5|4i zu!EM~YnK#E#lJGdk>t}n&t%T^USB@oQ+6eVQt{mAr8J@=eVND{DtLb>-zJ5{N?~<~ zzPahgGBz;Cd#jToYRMu>UXfH)E~gcP7{df+vWUGz9~3?Lm>DeR*QBTv(_bN7d505R zXZfw%PV^h~I0M;3JO}c7QcUW^C_KhR<8h{tZh1c@vnw&?Wy-x#yw6gWvx}3Q=lzEm zL-aQ=k2uC!$JWHTEBnZ^iL(P&ay#SM&v9P#nV*u9VxXkskCNKdCB;ce*LzDkW_3Ry z)mfL6^6662sFZvz<=k^AY3*Kb;+oqCzD)EFk#fw%b=<}ajAAqUDdaOZ5Pcq;C8zb; zt^_&vMHjl$pTT^@bbjC@1^rDeF6ViokCV6;<_8WW_MIYP6rXa6Y|EN3j(1sPA9{~1 z{fuw9)a!n{!veP2j~28~t;*%x!XSn+hUgRVM~e7d1umy4Eoe!MRS|c}-pTu1<#j9G z;5|O)E0)?1#}(6#3}ZB78AlQCH6`wqjGpG|a2>btBNtm9<7RHQPi<>I8~sA;qc`1V+)gW^@A2O`O!N_R9XE0_x6z8$ z_QTcfn>#UuS)^N5g%x)HC=hw>2pd6v;kS>Y&=&~lWvS; zJYVx8dpVKR6x}slhpDOVtEnET>D)j~+ga1GPfh!2Z53M5n_=wJ28mt^)e6?5mfx>s zzgWw8*jn`wd~>7Bv*RoI&YEdoQUUfDPmb!Do~vc^yD#y^BN2J zJ;}2!d5#J5+;5+!-pI56aSoWOTt$CgOFRdoB>c)zZuh!hl1kN;yL;$h{xzm2iP(rc zZS|u>YkD%1B_!T|mI*B4+oV9MxacstigYgESV`Zbyu~Nn?fqU%WG3sAg1%QDuiQuz z+Ax@H{7r42X~Y}6!yM+ZJ}IQlXIvS^FrEGU!D-&B#P!-cB~pdOpJRQZ|DD=QVl$<@ zUy(}OLwiQBfM`eUOX3L?V+c@(dh}xyllY$fqUCXO z(}uU1$%Z6a)6R2qKK*!(Nvz^0j*|FXajH|By0qjW9$_+Hu#xZCPxPmDB#Ean(v+Gs zahqlzktSx-)YWOu!8(sY0iP*CWv=2ShBKBaOlO<+*IC+R=Tnb{l(wt_IdtU-CUJlh z{G-j4uRYe1H<(LB%ewJ2FYqFtv4UgTULr&MJ0HSh+-TXyEYaqQo*OS zh#|zins65qeUOivsp{L{#E;{fW3~Mm*sCoxfpkD#Mz6A)=j?*+|A<*V>%`L&FP%W zwcJe`#`6i$-}g#3@wYac9-P#1XF5L7mSYn;sO`ORyvH2vwPrqfJD>9-yQyn=2cjR3 z_1b6`s|8!po*2{SI?Lwpv-VhAQM#3u^k4)ZF`wuQ`Z_ug{nz(rCF``sPP4Ta(vTRF zv>jcEv24b&ma0DY2rn^$m7LUGi_vJ}j-2SFJ9@9Zj=LGe1Xl1bC4Ke+>e857X~7X~ zxadjx656w!gPdXcWkkPWx6_$LY~z6TS1GNOe9ogT^|*miOy_HkbEeP6*Z@}%{g~X# za7Ge+(N1ADN0M~4x3(KMatn9TmR>wX^jkZY3Cv+R-?EPV9OoS0Q-kP(<6ipk1Y_9B zFZ`ZFEoclIT5vb7@&*%G!xql+xg0K`C2i@;6THYq4kT5SLF2LU3R~GuH_Q7lpCw6E z5m43dRaK`L8=J=&%M|9aki``7+4G2d0SEC3vsuL^eo3l{+-k~eHL+IBcj?2LZ5-nq z?^mTZkMSzeU)EH9U_a5H#Xu8yUvW#JEOvedb!mFpcO}?r0LN zlp*Fliw44LZoukcR7w@a>PfDI#1h_>Fj17XZu`j zE~Fc8C((mwt~AJ1_HrG+<*LJSWpY(qRL?tXC+um4ANa=41Y%-|bumKIqz(3DRjDwNWrZt8q4p*3CU$KyQ7AV#u^ zbbph zxPV-}0p#9sd5V&B)8n`llKhB7fJo>J4=_Z628#pRnmQRqbV)TP`&XL>SkL5w5&Jv+4ZJ#lOQQdT>wto1AF zJWE+^fU>R^mGv9W-Ex$FDd_X*)TJe@=*?i3@GWtDb(>>?l3I|L(vT0B$Sn5qg!kj# zo@W_JjIj~pZ2rPQPC0IfTGN%YR1P=Oou15R6>HhRPNGlMYW`**uQK0pL|j!j+J^e9 zoc2vQ?Oy%;@)L!;Uy6!6LyVg{oR5kAQ+7Fyh@Lag;7qc(kSl0Q^s6$7uUX9&_L2Cz z^SPSqiD&iO(~sedW)(lMjo%%A=uuK#UqKzFuO9|8i?!_M9G}S~pI(e$9OHSPS&A6W)^khBzIOG_ooD_~xWJYk9KS^1BIu$vSI2RscCB-;j zeTXquKjW0+pL%M+2YG}k?BxI_sp$RsG^I8Dc!@Xpm{X3Cq9=U4dP#xkt+J9juTtuK z`tu6E)86O$GlG%CbF~}T!ZycCF=kyW?&lfCvxZFX)!{xm5#yk4aEzpl`0NFTwS=jqbd}M|{i?{-KD^T}CsS)0qc&ma%-! zDmJp6-yB03v0A)T7L%3j|0+B7tSqmMbINeG@Fyp!>GQRCh#32ADqj%y1MhU~R6z6< zC$1j-j~Jome424L5AYl_Sj<8Gp^U#hjk7t27(et1Ze=W=vy2_==0su)KYQ-Z3}8H; zGL4!1#lMvHxw^EZ75$h*JQul;#q1@Zs;Bs>yR>!!nWIIYKd? zE62Ii;c~8{J)N1z&%`m{fyA+x5_CED@i;5_lT&1P|56&!nC7&jJ-z6|bG*+7e9C8h z#R7h0XA)N}vQqUK%S3)7u{@pgXhJ7uB-!FETfAi}BiXJ4W@}ID=bwKl;{6hwP6cYy zg1czV{rtp!P9||RuDVF7F1M=N8uvobh?{9j2Zrzh)0xE*zT=w?V}bb%?-4tFOTvuGx>_|I7&gEFF^)*H0DNbq7~hFlQF!_L?$sWsg+8p z8M25cePi6#`rJuZdh!S(d5yU&X9uVI+y7C8e4b<=gLr`$TmB8+Vk<|JoK#`0i8L;t z4X-egm7JoG&z#4D#B=>|Z~SX~!Zen#g19d-jw3y#r*6wtYvn3Go*&|2hBK2ztmk`< za-8D+t~?dFh-(u&gSTuH_vzF$=f_Jo+oXM@xo2K!C0m10!%M%=(H+(`@iGk_7ik;GjZai^eXB{>(Pwy1{^`?f@U>nn%8yw8Vx%2d8& z0cCtg1?q4CmvIf9=*x?I&N5CV1^k{qN{FkAb*M)}uHz;i;u(hXKJ&QM-`zo19^`S} z;ypfR6zfGLq3uRDRFBxisb)9_A6A;YHr!bJp+|iN7xs-@&=uN^2fx0B&#jR;^ZR_I7a&dK$dyMJNM@(S_ z-?E2a`Hh1dqpZKLMiv)PpDSrZV{YR~Mlza7%;gKVu!Ft)&L2q$^>YdB68&G%f~OeD ztHju;TiMQEl=AtDxrzRa;5Ei_EQuDAd$-8vJnGPZYiUATo?!sv`H;_;%@Vef_?~mf z z1$@mCq7Sf*?BD<=lQ6y$oH0{0_PCp zXI#qlG^07U(~H-LesCr;m3hqPYqqeJ<4M^lC8@Z}v^H09FOM*S3C!mJ$0^|RxwPS4 zI?QiJueq;A<@vBt|XqwD5obF7y92 zT+hSw<0+nDEVKE7HJl*B_vCRC_tTXg#2B5=F`f^JK1fd{4&A5|$i026&C;Etfp7FfTZ050w{X`#;fAWuW zNk-#SC%Hb(wKU;X#xjvvEMyy{d_Kmrtw2?BxtZSdh>2l$nP&Nand zJ2BEx8AP^tsGw|AkoOhTOBK}j`miR(4(!L{3}6Ifn8tKwGLQMJU?tzOne80mI4Qqb zoYT3C#@tME?jpud>cU7SF`XGKy&U9E{-K=TtxOfpr!E(9B`s*l zJv_>jJi|+j=1s;hjt`i^EM_y81*~N=KeC&j+0OxDT(SC&v~QyWkMKAHd7bf0V=sw#<>^)UR%ZaLORfyzC6Vq z_VXKQ{;n*iQ;8~6r8+gJ$pu`)Z8WC^j}v`{jAS%lu!1#g<#!GymDFdIQWwyGt7*X_ zOkfhTSw@W6zMWm1UxtlI@<0*zQf=SF_5iurEJnxfId!`X1vYk&)9;P>s@)-S@$$XZuf?tX8 zgo;p#GpRydF69cE(1jOxjY)jWEWTtrJDlf?(Wh&08CP=yJ$Z=Xj9?t^@gZNam;)T= zZwmRIvRuM_^x*}D@fvUN9v?8BHLPU=(U@(&b{2nbG*zu%x57hS;JO- zcH*$%iar9h=$72}=3=MKs_F8q$L zV-<$;3-yC4s7k9cs=TVBTB)vTfEuMntBGo=nx@vNo$8!=AC@xY-Fy5yCZ$Kvl>DXi zcMCu7R?SsUHB^mI6VzgLK%G&yR2=xkDwWEhGOH{quPUQzs9tKC+Nf@*JL+{<@Jg_6 zL!T)BdJg4Ql~ga)M-5lg)FQP-9aJaPS@lXqf{(9~r~;~-s-|kHUsO{yObu7l)eJR9 z{h9D$sr-#U_>|97rb_jH|7Gz1oNB6?tLCd^YJ=LU4yZ%wm^z`( zs9Wl>`cH*$$5&}pMwM3;P*v2=s*dWdCaWpxSG8QNQ2W$DbzGfQ=hSs|TisE?bHbjf z7wUCbYB{EMpGxhVrFKG6htjG%s*ozG%Bk|IhH9l+tG23x>Ztmtfoi;(t$tI>)lPLn zJyg$C1X@W{N|jONQiWAXRb4ey?NmqAO$D#p9HYjoDQbaQu2!qR)DCr29aF*Q8lO=& z)Lr!fcR&`T7)o?XlEmOa%Eo!gYr}nD@>YzHN zE~qQ&u6m#zsV6E1-FWJ2l}e>n8C7;wO;uMl)z7N2YNpz$ZmO>usm7>r>Q}W^9a6{D z?Xa}|HJ8?(P-#8tc+Ue>Sd~(hR4o;*8mVz=s+y*js7>mciizV#l~ol~B~(S#P_^t|9;HSpHGkAi3 zy?y6z7~i?CeCILzyHIA8UzJd$Rc+N$wNdTWKs8m(SBur3YKuCnE~rcDn!2HGs|V_# zdamB9Nc3Z=xGMN;%LFQs%BZrcY$~@Zrb?>{s+MY{TC4V|v+AaLsG(}48mlI$$!f0p zL;b0O-}&rRm(`81;2ov>&PA11;i`^mr#h%^s<-N=MyUyEzS^jEtBdNsdamN*N~OxE zvZ|JPMAdl~AQrUDZalRXtR1HA`(&+tprmH7tXiGRP@-#e(<5Q{Sq3s;O$LI;f7S ziyEj#sIh9ATB$Cn%PKamuT(OXNo7`*Rk*6BTB<&3yqd0Nse|g6x}YwFMf7(RKg$k9 zQNbth#87_r;@?Oi&O#m~e7cAVe!G=crBmrunE(E#K2e{l;1frJclC^^VyW0Fj`Ca| zCn3zg13$hicQTjB)i)}oN~`>yCiuA<%*R(M1@G(^JWDf)@|r;XAD^+A&SiQP5hAjR zsXW(;Mg)J>Md0n@ZWzyGNS%I_KUfBdwRL@v{-;I|XOf5C4kJl_nLcLr1m zRPgM^L@M}Hm1HWVN~O}O;FH(Vsq`v%MR(vPl8UNgs@N*7@>&gCegmNrEAK^w?=v1h zOS5`1#j_bPB0{FWh#im75NuM@!&+0S7j!}%?O3ZBWBPDRrBnTo1nsMso@ z3Z9{uRCzvtz9>9;HlnWJRYpN~bmix_9<8GC6II{G-vFd_`JGCqf_K!4=D!#!j`AEc z+#jE97<}@;*DAG2r$QJ%QIS;?6-~ufaaAJabGV!nuRc;KRce(^rB@kL@T|g!%I}x> z`uM2@pS%1@B~!sWGJmf!sNnPlDF+D!&nk?rVyQSPq4Ki~bH#*<3HR~z1@FQd{B$|^ zRHmRicn9|2nS!3%1@rM0MPIl~tbEohTr8Qyl7&|lDX$}>`|(xZpSlb-9Q^b+j*6>3 zR|%Ep`oMpDW%xHPJ)aytHvGq*Dt`hI`Tv{-NS`dNyv_#p<0lt?&SmzvX88D1-DEC3*TrK(Ts(1Q9+#iExZ?5? zSN7iNQ3dzR&s5C+o5D{~T?RjW4W23J^}_soeAgGBU5Ddyct2rP=~aaPJDKsElz4XW z|NrFF>)zqx@e_}qczEK$#-kq(HXf{J0IJ}beSYWb?8LYK;DpDwqxkqfzJqN7mkCuO z6}+>g*Ua$q@mYZW&Mcewc!HmLetcG6B$vTGF1m`TVyhtGARW)Yb1uB1N%@>M87Gi& z0=rJYR|0b-;4J~ZkN4Q%ohgG86`YV@yTO0SRB{zOQvOc`KWPgdB_b=Y?UR8|wN-)R z;6FdZa_+smzWPRetJ0`+DuW6h?=E@V!f~Hq1{-s2GFTRDCF)R2TOgHsu7F8D8% zN~gY8!J~%f+Bp-cXs2@B$5*+BoSxuck-?*X@M@u0DtKq}I4XFQ30~bE{L3rcHxd)$)5S&0ik27x?9A43`(y8FtVi}azXP77WQ_8?||$Ttg27X4Z1W`W5f=Pc&PLNALMvdSeZOjf;F&6?Govg*mo zUsic!wco5}&Z;*XU)kiJ%}m+kn9XjpnI#)cHr}(DJ?P19_Ut&b^O4Zqq;?4<|(^<=jJ11W`VRORegv$k&i=SNjbJ<5OGv|WI#ZNAK$YthSvdJaS;C+*F z%QQDkZkXKka?3S0TyD7Bc9C1Qx#4re=cb$6PIKeQZ7;cL=f)Fo7mC_zQ5r?zirQ&WeMRvX<) z4OiN%rR7&zc4eHQGIYwolrc*gm@>{#8NFqkgEG97(O*V?88er~TUNeh^_9h4Rz_v* zvn>9y?xSVxwJdB|I%WAQ%U@Z0Eo+{#w93JkGfz2ZuN;nY&O|vJ<>1Sis~nDUFy&y% z(Jn{3oVm)ItGwLG(=IQo@^s3Zxx9?a>nqP+d2>{daRuBJWLp7$1$-6ky#h}a@K(@U z0cQpM70g^we?|FJlxszgg%!&S=&NDA8hUHcuYs=y z&o#_eLw^mtHOyFpeoehKWm1#hn*7$3MNM97;;Kovrdew7SW9NL?5-A{we;1JS1lfE z@leZsu$JDS=Vv`XyZvW*{Y?93eLu_SXWBp0{aH>wI~zaa{~6|Ie#7yH^AOH+IM3lU z!f}M-2$xAXo^ZK@nrGlaVF}( z)Zwd+S?lms2eytgQ3rn=+0@~wj{8Ynv)1LKu3gr}Q`ekzanv%-PJ zM}4!_xAXe&_2KI~qxE^O&u@L&^_{2sbnC;_hid@Wz@8fL-2kqEGuVJ;1N{x;)PP5;;@i&%7WBA5)(wJ6bxiqE` z;G5`gqQ41WP4qU=*F;|vyKADa3I3*LXlm}J=4eW%Dc?=?Hsz-&-%ZWjl%HlYY9`xe z`kLWw#(y(Dn(^Pvyv^X6>2IdLnOU3h5MY|yWpn(^`ESlgbC~9`Z4T3%kLEl#x0B{( zZ|;86Tt?05wcx)6jTU-a=xbpIE%3L{(*l1BeJ%90)YsA+EuEK^@@(mhwB)^|+gdqK zt$1i<##U}`#dj;Qa~O!L{SB9j$h7?dY_#vv#oU^tUrtI~wg|&|dcK zouT%;wb$RCUVHuR<=URN_RdjzkBRMJ+tY0i+g{e~&Dfq^dpzxB(t+O&_TK?t2l^fG zb->qweh0og(C+}>fqn;GJIJX6jt=tafTx3uI`Yv`P95Pp!ghq~NWUX&NBSM%I>L0+ zAG}VXlfF*;c9KsgI-U6LM5B{zI??IGYbTgav^wGM%wK1|I^*uFzcc>M&Sz(HcDD1* zFrD>xrqP+N&a&(R*TsBY`04`N1-6Ubb+Pj@ZI3M$+8=KH~4O_-C(=%+ReVZ!E|#U z?WVsQpWU6|?)19Lq`Us^`nuEU&SQ5v-TCVd*B!3A`MTR%@R^1^%+>?82V4(lrU#!r zWYa?)J>Yuq*8`>pOb`BgI5R!rdcyUj*OOLHn4WgfQ!YJWd)h-!*q*RG>GqUsPxzkB zc2B$O$!{;`r5D{^?mxZkp%-tx^z?Fj@LHPQZtpFJ-g58FTW>k^rqdgLZ~VPsdds7? z8GD*2>cW?XaEtlSM3Euy#4_|%w>I2h9hJ9fA=@!Ods?0q20%Pec<}Q z^)**t*uHf7!u5sgi@UGhzWDp4&2qTt6B1hv^T~pJso({b}_#SARbH!}OPBfBE!>>rcNw?f$g;%V~i7!2mf8 zkmUfr2gqxHtOm$+0FD99=>XXbfFA%mfbIa;0sIbdz6R1C2s037ApL=S50uqFeh0!2 zgdYe$kcWZxIZ$2$oz;Q752QWFo(AC+pGZSb1lXZ^hw?a7HbZF*HP2A}L+K6GJCxo~9)~*P!{`m8GfXbS=nR7$ zW}m}o4Kw2~dc)v{(HdsPVerFf4Raob(Hq9|a6H4!Ivm$<84Wko>mj?z0yHly^8lFcX@qwH@KjZu7#GSg`1 zd^DcXypA^4XkJIt8;xVM^E(>HXnLc09PQkUwu{mBIU0U6{216VbjQ#g12@KuV`z_o z8AEpr%ov(uV8+N~tPID(jOBZ*{g1W#v9M$5jKo(-9Fyz+Z;hDtGBs`O3I?0)t#KR;! zlVm#y$0XTKGS4KLO>$l*$!3ySCexd2caz;V*}f;sZL*%pG^Ut$3SU$Bnxb!tY^RuG z3NKUacM8lDm??ZsftkYF6uzd)W2)Y%w5OV3s@|#krqZ2?cN*W*c%CM^X?myeJxzAg zXiS5f#`iRPmrt>{r4%7LW&d+qdgZD<7folei8E`Y;X6TurXND|hm}>^@8MJ5UooT+AG-uk| zOqiMc&y>+j`BZF^m6MvY2ItS-5B6o`rXo9n9i!7XDfIXW^g4-z*xzYfxv)XEvSL&ctlF%$DtJ zytD0Nwj5^Tm<>N0ezqKD%V9Q-**Iq7n9b81p62j02Y!zE=HQqk-#K{Z$Y~CbbId%4 zuQ`0pG4mX}b7VXR_Z<1n!9Rz;Ir!(`on!C6;{FxquQ-1-->>}q3i~V1bLBCY?p#`P zY0WkFT>NwK&&53#*Ie_>Guu2F&eJ~+-#k9&;hZPGd2sXa&V!xD-#l5(<7po3JU-^Z z&xfB+XFl9~v&=Wke0k4@n@?-LOy=ub!0!TkT0mz3odrBE;B5ho1@^Ll#sa<;;9dZ; zz*$)!!v(Ml&9hMc3+XK6Z6Te7G#1*yLNhHi_d;_ol+!}^g}g14QSf@sMKl-jw#Yn- zU>4C@#MdJIi_Ejgt`^Z+MCUiXzw!T@p5NsCo9h-kXN!4X%=2RUi*YUHVX@sU*1MSg zVtk8vSPZk6_F}#ld)!?Dy9DPFGcPgg5+0VoEivm7xFzOYBF`muyo8S>d@OOFTuOH- zy`{94!YzeaN@J=1rFxf|b1AK*a$0K6rTi_E(K35nCZ}a^%XnL62g~?bhI<)L%k(b8 z|GWO*_5RM=@3?>0_q&;Ym*?;Jf7k!J{@?Wnz0373*SlQba^9D_eYyMba<{FZw}RdZ znXIsv6*yM#w}S2peJk*+FwaW8D|uXrYo#1l>R)NLmGoEQT`7;1=37aBrP)@(uB5S2 zhO1y#!LEW`1-r`GStZL=aI4@}+2tx(tfIL}hO20)p@SJCp12tvB0xJnM0+hu>hP4gMV4!1D$?8*pr}#|?Nk;Mss@1MCKV zHt?|lZiBxwY?S3jxQ%AqsDGo`HtO9d(~bOZgxLtQk>*A-Y~**N9sGg!4;lPn_CL(~ zhn_$9`2**l&cmN_`qQj`%KlGUf5QDK^G&>MqO-~EoA}x!|4nAwWOtirZKAOWZj*dA znQJqx%{*>4-)25HyL`68ZHL=VcRTEM-nToG+j-n>f7@wor?Z334t+ah zvcsG^;Td8d8s#IuvuPCj?y+G)<6 zeD0LbPPm;gJ7I$N;@jo+T{L#dewW*K;oRl+UFO+ko?Ujn%ieaGX_t9+n`1ZL-Eg~Q zzZ-71Id;SB#<|=4yY=oi{~rB&%)f`u9%p$Eojvxp2WAgHd*rakEPHs`gJTcu9^UuL zV6XZ2ntv~iz5MSr|6cR&b+-1(V6XnYH23nhm)>5p?~~y^yV@s%eKhys*~jNTn)_(( zGv7WQ_tDvBu6=UfuYW&}`^~jq?|x@-Ki&Pj?x(#UZol)fA9lad_JCa-fICS0Ak0D89Hf2Fd6pNE{=Lwp_5dr03QvmAEYVcLgjAC}i)w;wk5VO)p# zJOY1&za!>5BHJUp9l>*i_7S>AoPi^J9+B@6XW$5*N9A)A?kLWqFh^x_6y~Vi9o2hO z?@<~@`8aAHM`dyh?ijseGC2l+%$&z~JBH(!Gkr`h$K-j8-Z31<;E%x_*L$45<1#tU z({Y}To8h>=-pR5f6M=GJ%98456}Pb_mAFx?C~Fd{?Yr7Isd`;4?q9t{m0%< z+xuz#r*WUgds_adc|VQ+H15-~KaKyi{?q!;=s!c_jO@?I=M3x_T4!W&My_XQp22ZO zwr6ObF?Y~^*6nBAc9zCjx1XhP*6n9$oTYJA_Gk5-!*xzJ=k%T9`Ow@pDf9dHv^kKacY~&*z<)^Z3s5dV$vq^1i_91^O4{eL>#^oEPX{kn;uiwF}P1 z1-utvE}G|}{am!yi|*SO?c}2Vi~29x%|-kd`MAi#Mf{g=Uo!V4^IoEV3D+gJU7~xL zzstN`Hp^vPm(6+^{<3*4>${BSvfj%$uF$>0;}ssS*x41kx?+wi{9Vy=1>aSCS9!Z? z=Bs+I@_5zxx$4|qHP=;MugdHypI4oqt2nOWxJv7)`L6PN4aYTl*Kl0JagDEQ@Yi6k z!Co`xH9oKLdCmRin%u9!U5C3)`?}p+hr4d(>t?&|++K&j4u2i?x*V>{;X3W>=Dd#M zI{oWrz0O0xbHlm0f$IjY8+dNuxnb@b&c_Y&+%U@xd^hBN1K$mMy5X$cFv|^D+`w^D zUN_Bu6aFUbP1u{TH)VHI-Z%NWiRY#)ZsNEpkDKPY$sxZU zh3^)QTQIlGa7zxi%zs<{x9Q&I{Wh)JdT!IXP3Jb9+h)Ga<87GRG;Y(l4R?pnJF>pR z;~ls=Jl@fN$4qzh-{I>H{X6*Y;=e1GyZZ0yy^H&<9PZl7UA=eB9^8JweO&kL^S-(73#dW&*y#G_swx&K$jp!EJTmJe z9v_+ak$E54=_7eQf`26c$Fv@s@3HwFJL`|>J?8B(&Bx|>tpBn8$NC@Ze?spGohL9) zXg%Tc39Tozp78rbzE5DE(0fAf3H%e-C+2#p?=I^E}1>R9;Wb^_0fH zH2#&-zq0w)b^qGQGuJdS=FFW_(8P8PCsXKBF1*KR4fVUZ2bV zIqm1>dk*s)=DGZz%k(+SbNbI|ypZJ!USG)Y1&tRpUYO|x{ug>*=zYQW3mPwAUckJB zdFiaYr1jEyc?tW{tS`;*68%<5GQ%tWU*UY^Y`il2D|`XRYhGT%zqZrY@UMAz?LPGy?ls(N zbG)Yi+PQyimeML2-r#ye z`;9!_nBy({Tlly1-_n1}|6Bgw+S^Nx3F(%zoq?_zyIX=A3y)O?myT4 zhwGhr-tqFzOz&{Llj}SB@67Vf?C+eVcQEgqqjzxc;NHpfJumP1cn|*`{yqGAv%JUg z9>;rL-pl1Zp7&;X&&zw6yf@2xKHlScFOLs+KJf9uZa%<&ki`dceK6Ap_z&{<0RKT2 zAI$Z^jz8GP2V5WUeBkSYY(DVy!M;Ob5tLVzghf~0zbGu0@;<;}o{b*%rSf{GuoNn# z`c|b@UiSnS7WSF)`X-pLFwfN28y4m@iDBu~_bP+>#B0^O>Lko-nZmq|3FrU9ynYGa z|H6ERg5Lk>`yZ|dVNp~x6^JAyGk;rElUh|24rXncb26Fi@o?Gw09%=HP}Cwe35jmURI{Sm{w zV;pZ0!@T|pHey%|<^7$)d|n`L5qXNpQ^YW@hr$;zEUEGgd!8cl5)o%aULx`l5qBiK zk?=;s8wqbDoRPx3T8O7e=8a_5NO&TJC03ppZ>~sT-pfY-HHTFp=>`)*qRl$YKBU9+MGNB;}JWVWXHS3TzbED72%fe-5%9WqUwoiFHxPNsCuKyEgFqzd_|L0G&_i9zG!gK_=_h0 zXzuOAqFoo@We1z3|Yr; zE@RkZOdK)!iHRpBPch9F6IV+NrdMMx?k}8 zg){I4KVO*Z3%mJ3K8bNBcArZui^Mn+<4lY*vHMM8{fXs~7++$1iD42uqe*Zkk!KRP zBzz^|D+&H2yd}Yt#B53MB*Bq{uOv83(SkU+Vo*j$hi} zmwLa{`=$HVmv-?b?Jwn#RF+9$lhRFU7fIognm;LAQW++tlT>D3@%9z|uke3`|0`a; zvYW5m{uR!zaDHX}WH^$!JsHo*cur>4WO|bEkW61PJ;|M|E~R=>JBZ z-@trhmXtJ7Iu|Kvq|~3%-cp)BC7qP^o)RXd98%&=iT7K+zs312&TsAWTRDB}wr}aB z!jlS5Dl?_RlgfQ672Q;LQpqEgbD0WXDl?|yAr;T5U{m2uZO+v2so_(@r#5S9xYTk< zZQj%{sm+_(K2r0ZT0UubPXm{Rr!=xm1Dl3!8s5{urQt7){xr1GIBRM3r`4ZU{%L8Y zwY#)1Y0aIMURrafg-vT8X=$hBIW50w>89m1t@}$_`F@A%JD$IDzP^*;cX+?U^&RYY zFyHZ(&RI$4`gF9?$srx?bZ$?FJDnWT$uu2b>D&*~;Z7%qbY}S8?BAQ=d-HzJ|M&9! z-n`%I`Ci89@ulZEy`J>;kRE4x8K=ja9wt3ZdYJUQrpK9{#|(UAFn0#n46qq!WPr^8 zn?bf2;4;8ukW~hA{lNbZ^7%ooKk)Pe%n$m1Fvkz(_<`OJX86GjKk)Sfjg0s+>dmM( zBX1dGgz zHyeC5_-wG*XlJ9H4L%!ucC%&2kzIz_WtbgDcAm52$xc5z&)MxQyPUG)$u86EX3lQr z?0B-vD2I%4;L3p`hnaK0=a6j<_m3QW~xrIpA~fkb{0s=RT(ya?;KT zn-eyt9CA84In9uheopwD{%p+&pOb!0a|G`zoeMsf*>lM|7hEp0=Yq>+?p*Y8nLC#~ z<$}pYGnedg!R6*Nw|R5Z%uO>lY;H5>rj^^AxoPEw%dJ0nAMZS7%)?_IT6y&6(VvIM zJpARMm50AP^z!J>!&@HN1n+sB*O|@BV_yF9(#i{ym%qHS$t#<@&PHB4&g*>SrJdLL z$ZN*DwDaQ0E2F%4^5MycBOl-S@Z_VPkKcUw@|igwd_KPO$u^%^^YM`nJ|Ap;*!<39 ze)#-w`Dy0oJ3rs~Y3GN@PcJ`AezO*UDF9P|Rsk6mfGI$$0G$H77vQ@9y#mfg0W%kn zaRGT1z)^tT0P=cGIn3gaqbt|GXK z;3{IKBD@qaOA-A=a27F35tt(8DFRo-Sues*5q^sB6X1*TQ`AgFWmD8SD9Te&Srp~H zC@)2MDN3)X9E#E`=4=-;dojJmWLV6+#q6mV&SLtC=_{tMIPT(RD9>y~Ul6;(Cia zSH*D`=chP7#c33WEpAUG;7YjfmVhZiqXbL|^OTTL3D^?mC?VGp@Fn;v0at=vfGH`f zl5|SKlyrMZyd~XU(wrsvDygp|jZ!?7GD9hIm!eY&u9W;r$*&ZhQgSOLw^FdB^q1CC znwQe@DXp)xj7!rf%~xqzm4+$JTWOhCm$GIqi?^&<%9^FDS<1?{9Besrl(WBb@a3G9 zay*r@|8lg-@m0?L%h4+*k8<$k_$jZqJgxG4m8VtSJmvM4cgD(_p}hX`&PjQg@-XG) zP{BMEU@O2@fUAJNg1!p+D(I_V){1Vg$bUtg70p}G{i342ie{+jj8rs7MVN{(70pl) zZzUO2GD9V}O1xL%p^~{PnYR+{N@lE#yRsQ8)2?g}mGM{Rx3VlNdTZKjy*ebkLvBxUrt|I3u@~VQbiXB#=T?JbzBl ztqxP2|LS_H>#eT0hQ1nfYdF_6WKx4>4gEFDP=js_{%g>y0b7IL8uV(w)pSN{n!P5! zHDPPQ)|5j{GuPy`CR|N(*EDNQn3_D+q+OGCP1-f>JNRsZTClZvtVOq$`%W!BYtgGE zr&_SV?LXVu&-j0K`_DLj<~f{pxa-1chRZ5k_ThBH?JS&jIPGv*h08qL84l+moQH7w z;qbq}|HA7pIDet@i`jm0`!D!^kzZ{+wfUl z>cG^&TZgYYFm;@#IyCC=Q-_B-xa-ibWA-{Stc$;{{<>zYD}%Z)b?MZFs|#0`x4O<@ zU3zut)x}X4M_p&H9<6%t_1qWg!Pb*sJ=l8AOFbFZlVLrWdc4)+t-k*H`s?ejZ+G?O zR-aydxcV^lkC!!)Mf7_Kqh#(Xx0Z4B4gnP?2x*uEOeurbe#X*QmD<)Il&GjlZ4(~OU1d^D3|GqW_~ry1R5 zbeqv_#(y)j2bktGnwz~j{^n+A4%-~Ax%r#(-`xDo`ECx|-2Bbqo6Dd%@6DZ~7V>LB zzXh%qJh$Mr1)di0E#O=5+5)ZxTnm^M=51ll7W4y5OZ_crx8%1a-InHS$!|;8mar|& z*-~~b?Yt$9mN;6Pxh0O4@U6_;3P&q5w8GPhb}L-1aJ91kR(!O=*UB8N$9-fZp7)?P;K?YKSO z_U3DkyFKpq@@bE|yWsJD9fvFCE}I%BUl3NBtdXc9cs;n2t0$>g~vTN1i+J+ewC<`0Yfa6R(}@q7#iy z=I>+&op|hI51r_BqSuK|Cv$hA)ye&*Gk=}oJM-6BhMnnkhU*O1nO-Obfq_T8O>?zpu7}Ke z=<8vI9&~%iwkO@5=IyDcr8$mXbx-^~&EJ!sUU0qO zdO35wX!PQ(m-|dFzI(y;g6-wZ^`hO2zg}>?X!hoptEWzOPyP^4phQKY8}U(+{?v?EAs>3%$XMg+Z&u4%4mHvG8r`g|J{pH%9UVnS(Pp?0o0lW>sJ-}@P za1Suc0JjZro(8&oAnrkY48l1G=OCPea1P>S5UxS?I|yzN&x83HZ0^Bk9xU6zybm_> zVDk=^>0laz?Pf6j!Db!|I~aB_ogwBKBI_aahL~jto*^kuA>@Gyjj zA?6x_dx(67m~V(VhR_*8V+f5Q_=n&hVxA##9co8I`5a1PDDI(nhT<7&=Ak?fqcu!^ z!*C6=|6#5l<_ryo9S%F3zv1x1Wj>sj;W8P{&v0`N$2HtOhtnMnH=O2h=YE77N5GAM z8$ox3xkfm|0uJJ!Z!-vD7lO>*C^SKlFKOlqdZoO=5Mr_M#GMl%V_w~ z@S}Mf&C_UFqw$Q!F&f8cxY2etMlNGu$H0z(8^hxmnq$l}Mjm76j-fY(-WZrM<{4wp zV`VYcOk?eNEX}d>#+rYuxyIV*Sej$aHI~*`xUq0!UMIj! zfSDkR32vVtg9&_1u;&T-Cg7jI-vn7qgqbLhiS#DQVxm1yq&1PwL>W%RJy9MLd7B6` z(QYTfO)}dgc}z0jBza771}DKx;%$;FCh<1Ojwd;bll4#5Kbh8KT9aWW!%U_(na9a+ zljSiPZZgfuyiTS$ndW3(gU@N7Vs}&Er`W?3_$hLk0zUvZ?M z8Spdc&Y(Mk=NYgwU}w;sVa6HyXYe>fMl)b$@ETxd%4Vj$&*XKcvpG{HGx?m!=S=s5 znR;jHor!mr3}@+^#oH{sv+&Q7$t-!ylIJX1v*2gJ&f;?x{4Bb&MDXXBd9>uh|pam~gx8_#U>&c-nt$87l7@U!7(+tY0G&XLs|9CLV|gJTZ-9QZk~ zbNHVFI|p_S>>Ri`W|(8%UuE>GnSZ7ItDJtN{VT0sW%R3jf0ga8JkHfSSMOXi&&4~} z-shTeuKDKToa??fSN?P5GuQc^%hNpPZ64e_{PXb7<82;q^JvWDZyt_$b~%sUJRaxq zIFH^uKIh?^hie|5`S9~)G#_?8+_3+Qyd@ta80o?_5vp|LmZyIlsq41O8xGCqEX`JIQ~<@mcyeuw+rJiqh)JB{D@ zUe5D!ear20x!&dcF6Vc-{^fF6PII|jmdj;1?d3R@+wF2QEH}e)K9<|Xa8^xdNpqz$u#(Irn%alR>QBRx!SC&&9~aV*Wh2HXAM7VaIfKG4em93tg*v2_}AcHgMSVF zwfNV{buG+Vn6-G<;#T;=M*16RZgkc+(%ndN zBb|->Zp5(>ek1NbaQ}ht4|;#{{-^K%l+mAN{nI)5lgB@ur9b)m6ZTIz{VCf&&Af^B zCfb{5Z=$ye-zNA?@SDuN3CAWJn|Rp7!zMFtGQ%d>ZNj+;=O#06!nw(=Hu10-*Jd+p z#<3a4X4!3q-^~AJ*v+t;Wwse+Gt6cl{*u#Q`v21V7tO!i{+C(*qVX4vzvQ$<&ldA- z!M%moEqrc)*+OFr+!k4F@wmOkezwrsLTd|-EgoyO$ZLy?w#aM?uB~>xmF`xvZYVNK0wwiq_&aM1!#km#VR(`g@Zj;+KxNSUa<9{3cHrQ?E z-p0>1yWa-8jh}61+GeJ0IJU`dn;f?Bw+-Jm{ft^ z&%1H%#D{MyACLR&dLN$sJnYA_-);Ndw%;uK z`Pq+aKR^5N?dNAd&i&59e!TlV?(OGizgZ5L<$xU?pm6~1fb)0&{{eeGV1@&F519P` z?+4_0fTsg+2WcO)gM;!vNdF*z2YEQi!$CX;<#~{YgLZO|hl4yEOi0;E%}t z2>cP)BX)a)mm|C!;pGVYQP`u-)=@JYwbP^WJSu~ubdNe4N9i7w?NOMcFh}KojOH~X%2!yo7SxGax53&)-36F5%D?F60^ay!BQ2|iB1pWyohjuSFEA)^z{!3p!8;Qa&- zLH|j9PtrX}_axnuaz5#Pdy?KsdMC|$QcfrB?Iezq_I?u2NjxWUoV34_e4OIp6c4B5 za0=%se5df8GXE*(_!OQ~yr05#3dboNr*NF&|8JPT_5aQD-+KS1`?sFI?c*OF|8d(t zvi!&G|M2yX+y61sKeYd$ecIVOt@kt!r_Fy_o~Pw`S`Me-PCJLE&2`!tI4zUYGC3{R z({eqH?=)|x?d3Gi)BK$_>uLI@`8#b-0rwd`&)`2}#xv$SgZB(SXK?yMQk@^zN4vpk(O&so`=mCackXPvRL=03~US)R_y>8$;p<>9QH&f+^O z>$AL^#e0^Yb9|h`dk*h8yywh&&K}S4bkIh^Ordd{5Z%z2K_b7nk;|D1iE z!+*|sIEVWjujg@|$9djP&f`6g_q=(}^Lbv5=lML3^E}S;e4dxtd0gkseICzw{?6OU z1s*Tpx&VK{tQX)fz+T|t0?iA0FX+3d?;_ocv@Yts$p1yX7kR%(=OUepyk9i`MIM9x zOZIRH_a#0r!Cm6-65J(uUV^=3CztHxl36eDbxDqwWO&IwF2i4jzYKfXjF;`>vfM7) z$z}M0H?GpYO8YA9t2nO0UxmHu3|w`_uhP0I&#OFNv+rwWz6Ns* z=9>O%`mV|4nx1R2yw1~gJ=f)N9p`nqU&nb}9@p_*=jA%i>$1NtiyL@v=)Ylx8?wIv zbHf}r%y2{QH+a6m?+qE=F!K!>H(_qd?Iz4kn4A3Egt;lVo4nsN_f6QFyxo-9P5y4e z-h{nn=3DT$_`GGFTeNTCxP{{upSN(_!f^}!7W^%F-NJE;=Ueu73)d}Nw{YF&|2Dqc z_-^C6jq5h9+h)FPkGFB%#&a9~HvDZf--f*{s{nV$40rV3G4mbTcl6((eFx@_tnR?v zp>r4SUA=ep-sSVI{<}2p>c1?9c;O&7dAG%LGr122PL-Rhw@z6PY zi02`Whd3UZ^C6yx&h7k4s%Jd=pBlt&Vdj$6g=8@SRne7qWBbZ0LJ~G=Qx{qiE z{g3H9hItJ0nAT&s$FzdajC~CA80Imp$Fv^v{n%_z^gPk?#O+U9|HSoAa6WM#e`1~| zGJ7KHr}&@Zev134%%1Y|)SjQ3;VJ&7a(l|pQ&~KPe+v858F?!2r)GM}<5TD4Up)Wf z`4`8(wExBPFa3Y*{a-x);`x_{f9?8T*ni>vrT@&id1k(6u+R8@X6Mi7KawRwS=XUv==5zBtcTQjEe?j*J zzb|ONfPDe?0`3LRFPzmEbYI|j!Sf5Vzi>uh$n6D=7reiq{es_@GJ9zsFX3OpzvT0! zUA(l5m-=7w`BGLd^}nR|%DH((^A+4HIlY2=Me7xeDw!Mwu%S{|?UzLv*p z^S{>f+8$oxeU0E7$gY%6!-r#$K>y6$wJijsb8+>ngexv`5^Y>QoTY105{TBCI z{@?QQR?ctX-tzty<}Ho4e7uEw%lm)!@t-XIv%~-F;6JzhC;xZ2-r;$N@%zq9@9f|m-|zT&Czp4y@9f|`FYnFyo|pGH-t+R_4DWHgw}X8FL^2lIb0`v*K9;6L#3fsYS%`N8@7Ade4tKA7tRFTrP~e!%lVCLf&N z5BNTWeI7d$>YpGKit}A4RLe&;v`HKawN-fw`}CbCq0ss`p-?*0lV>)gJ|3+?jyu(9zuLZQPhCx7ah=stKXRgqBWhm@hvAuqry0GHo~zzxK4!3*_9 zyDg*J9>>FvBOlx_74ky87(BB2vl*Z3PyBf()Ck{VFB%-oOB6Gg z$9WDe9gWg1??m-e>tf!WkVXVwU;HT)I`8`4d_8u3hR;Hwvv7TUA2VYpl$%a`S%uRm zS6pW1Yvi(k**BP}o!)-(sAAN@bkFM@fjht1o8rHK`+q!rl|B@z=*5jg_{v%$6dLMs zq`qM;FVUEkJQP|j_n79G==SH|xRl8jUzhzFuQ@NMB{Wi|vfGkyyr+{Lf`YA+`I{7x5f^m>nV!+675TTEH!hvGConjb^URSNf!#WhcDx{!@fU> z?-%>|T>oi(3*le5jQm3=6sxQ(WSMwyqZC}f%q5gidPr}nl*GG%w zeu2B7^Z30i5>~)rf0J^ALf`PWR$oKcwNZoB6qSPa=8;38tNM<>6o~6SM(c>ns1-w@ zZ1gLds{`M??Xfie)NW7wsmy(S(e-2P;1-_Abk@N1=P8e~)=y7aJui5@DvM0c^cR(6 z8Z#7nCDU?ln*x(2jvdOet=pys-{X%ekMv*oX}np>%Xx>}8kx62QuhVdKk)r46~i8? zMbm?;5})IoxfJr6m02D*I`Y>^y{9n^_ccB$=uL0dLUO8rI}zT;I4ZinGLA(!o60_~ z87jH1rthoZnxk)x`(|Odr}A80MZeptsMqHCo!1WLh!D{}oTZa6%gz4C+*RxxNV^=NZG+iED=njy@QFRQ*H?r$h-Fb)IW%uLk z_6PcNXx)a(Y?kUcQu_K_R_7_H#|_!ma9u?D7i1J(BNTclr@Dc698-ON#cUmU+v4kE zs)!6@%GE~;glf4Q;LP-KzM{#ZlAYP8EsgtGs3S!sC@WX5wssrw_h~ za{Lv?ubE+G-1%pC{Zn0TlF2DM-stNu^u*2>3azly@idZ#yN{^HvPx?|Ek2jGile8S z>s#p=?{cnMrxM~m>M|d_zj5Ywd5x!Lw0`n^bC;ETpVDr2n?DQQuD{4iKB?q0Tu*el zololQz!sHlLo?jha|8E>Le7Y}e=uW8dHrF|PIhs_bv5&NEYvgH?qlGn<@;GODCqk& zh0W=lA9oop*S5*zO8B2kjiiX!!oPd1Aw^;5R;e0(I26So^8-Agh1PIosSZ0)3d3c3GQ; z%fA1jxbyF}I=)Yv%AbLJ#K3>uUUs`rt_{w*+a}X*iZ8moTsGG&{u|?tL-#t4dU_fs z4TTQMAU<4Q*Jmr?x65WJ@48yP{#Q1gUG`N&R3X0i=)HpPirwyaT`D>M%1=xln&7XG zw}9JLn(dZdo^s}@mUKUJKUnA7jpMn!yINd&H-c!;@zQi@q*D=g@+iW*+wQ}7*&RP1SQGe%iOFj}9-8|4SK42f<(JMveb-=1;;G}hEx2yWZm``> z@%<2&^XR0LOY129?qZ&+uKUT36X{z7Kgm1^%K7s~-Q(*w*Ih{Jao+beRqFH}=ge6Y z{v8jWtM&5w+kCgpeMi0I^I3FffL21i$>^N>dV*bi!DqS(&Yn46nLV-ZAK))2$5?u6$MbkVX9SNE^`~`? zU+YW3Q!{y$w5NyqyST1MWRFi3{hbr0x!Zp6^$6EBv-j2d;?P{^y5_#$D$8$h3^dOz z*F9D~b|*BH-v#D;@d+PUbS5Dr2{&d8;g+Kjhoeb!YhK?=p!TPsEkIuOIV#%H?JCP$lLmJ)Y(~eTF-Q zS?=S{Th;lcAK7({eLWJszsoVI4Ie%1H>tC6?^E}4*VW_kH}j0-;p=KVI791k#nksL zPt9CcIid{pw1uw`&3ScQ68aJ8ZTIzkT$fya$?rN}XW;7xwb$%PUH7Hl8(G~y(uP9W zVJ;?>Np*kM$JfsMXU)3={{Z=A;^UhA4O7?Ed)GCmAGe14i@A=`&+qFp>N$V^IA57z zhWP%t$AOmSUghhmuD?n1imx-$2&?JuHtGYu96bDg6x{`w*VVQ+@S$VqW`>R_h6xzD zyJKjEp+fry*iIf#1#7zu|sgK|eE+HBk-Zdds;=FecdFD|CjF z`#(SvaFyMw{Li&_h%Af!p=@PEpIWExsE4pz|E|%C3o;|>Ue0%>nYzsr(}!ZsM!&oI z?4Wx?T>qexl48tlZtZ7@=1zTUcQRLIJ3IZT&xhioD0GL=Np&wcThzj?_o|t^`nJ5x`X~rb;!oXFge|_ULg1^gCMw;S}r_$nx`Hfmg< zuHWb4XoLQ1Z;~7uyW4}K6yL4LF_llNAhvsUOhEVEuQzh zK^gekoz`8H?_jpF%cq|EGK%h^dd@;NdAH+ZG}%-!ZDemsX5UY?a+sGd`MRMOmD#Lc z-}_0g3%ZHCt{N2L`x-wv>Aq`kUXWbHUtAr=Sl^?YkVh;H{NA!RL>!x)eNPYbi~kG$ z=ac79*Ll`s=so0nls@EfSJzn&6n`!CxX#aI>&MVU@4gpjVg9@E)q#(;;qnvH@uatj zjoHo=Qsc=D&A7Z*phNi|E3R8~r^)vgS)}|@HBukAss2US%%N5dt=mB&+lATrQflpVoX7V5hMfvH70!9o+FvY_*l|?Lcz% z`rMhf&Dcj`DemlZ`XS;EkD|l&ZSndCs(^4YZ&AO9q}b^Vlh1fI@1^s5Nqxqu<0o?c zy9vAaC+HpLH}iMFI+qx-it&@;?xMO6cYbJ7_XFmpvagR{v45HUon-H`U6$^L5!T-C zRL$gQor$l)@;wpaT993#U!J}FYBkn+8q9;g#avQOx!LXS%ywtm@DrjoA>yAU?@9La zG*>g2ZT&U>Gt{V=^~dbKfyR<=k;B0Bey3P3g^Tj}PkryG&3WrP5XxrmXut0=dqd60 z2EDoKE;Nf*cPN1$f{uc{8QDUYz-EXN$M0FZ1)21j?a9`uTbPGfxdi%~kj-=ULd@o` z>OE0So|uDd&SaL`6TWuS&%^%|Jw2&5&*?T3M^H;|R2si~&>C=8%>SrO7yNv-zf-r5 z@k=0@-*3=E@G(Cf?PX#AX=$;k+cR_`+eO8-K+P|h`DX=vN0RKda@S#_SnkPnigzbD zc}!A`#eY7l@26T`bmlz0o!QJC1d6j0KM}2cnCq_Q#b4c699I>CD{`lm6->_Mmtet!(R8@cVdrQ6_kX_Z=0`xv0-*0^ia}pib88!W&9sg=N4~@-jmiYe2*=u5ayMTX-fOGo%joHkq7YF3^ zlD(+*a-#FI7`Djw1^O+WtuW5sr_j&YZPq`ke}y1#e+hqwncFBa9OWlXf_q`V7@AA$ zdCgBIan{%S1o=-A$B0to>M%(R&CSz3x+BRZ+q;F|!`^+ld#+xbxCx$V%eV;X+E4UY5l<2`}AMZi>7y;y_WbA>Q>qJycvw}u8u%^s@HqL-c$T_ zxgSQKu{+qz{36C@)&tn6uf9{+Ule3U^tF9QImY_?S`H8S`%Qk4&X1BuyqtE5a~Hf) z*C2iV$o-n;y#IrK0ZsDBjn6kanP0JmiXlv2KP@f}vTbk}o{Qrneasf-y>zapc`GWe zpVaeI^=F`5c8hB|-E4KtKpA&f zUCZEWKs}hm=VCTqmlnJJh3oMkPVREBm8UVVDFK7Zsn_VJ#V`lo1h)_^;7&{{8*@=l%Jrt z#nw}uHsT+PXOG-Y;Qxes?#5#@Wqp6E>Q9)OmBjCGb}zm21@uo%AM^Kxx_xACHY|hb zeC+h(-~J4;|J3q3{6YLP zHtvV}&fsU#OVyBV7;Sd?rg5+E4d8Qov*6#{?uYgJ@D82X=xp*iFx4#lrZ#_&&9y$j z2NMA~8o3kJ<%)Zg(MHzoVF7*sx=npslr#@vl;hNdY@xm9V!MW4gx^l5SJKQi_8m9t z2iaas9)drFA3{DBCc+%(natdyzw>zlouXbh(8F{yH8BIQ#QB=&EkV9-KIcceGuD$~ zCM<^rW@0z`9rzF20{%gFh~>V1o@0BEm~%8$cQ#_t>d?@c3G9`OazD}i8GTpDmjwBJ zVm@}^m-3ymfc)@J;E1@gHgmVr`zGn%GW%b;IYM<@S#{^R6Bo zB0mm)LJoO8L|54VGm}1;gC@!S{chdG+uFo^Xx?0$e7sQKe)zP3?k&2CPBQgRG!Lus zGsJy?{7wnIKvOxBGv1wKuQXa6V#Sq+juFRDbTmwa1ocVI))xFhZGFktC45sk%?~vf zE%X3NLM2$k-&%MBKM11vI4IuBxy*uI?jgHLcBYW;sr7T~uvqo7zgbN$@KLj+-?ePC zL)W7#iuf(7MxR(u4l#3C+%59MDbY;ZPjtNb>RowN=YS7Wt$hELnt3_uCkY`;F*2Dab_pyOdWEP})G zdsob_**al;3BIED0)6Ja_$-6JfyLV|s~$v{GxONex4o@C{b3fL)7ikrPh!oR+59;B zo&D2r9=@^u7RHux2ip3L0HF{E-zqaog2 zeto!^+nupCo&gsM=w~i>#=%eW>c2C|>X*^O0VCbVXf+z5Ua7@D zkIpm+?GtO>VDsQETqNt|TxoC1VK&2kzyIZ|(A8eK|JJs+(`Q2xCtQUR=jDwS6j1gadv$xp#sPh-_xy^oj*lyznHVsR~Pf%E2o)ZYqr_?HI?5`a(dg|cXTG9xt;&P`O{(y=J!XkGqBwH zKKi4*5@snx?8O?$m5;C3d_x^p^53$gf6F5u0x8W!_As+)Ru<^xX|ep|d}F;mhF+!f z1X4G2*CAL;2gH%m{?3BlDfh4=c^^3AY)0`;!Oww6G2~0nj`&~jADG7dX7ic7iv`RD zJDtOQW7# z9q7)3(pwa1?#<8Ee12BZEyQlorvAQChZ<-a_7~GxCdb44&m@bD!m|^l zN4u?mg1_Z^#Qsx!s2*nGt5XJlGpXMI{d?`cq-Vc-GqVK`)bL~UH!=JrzcQijjrC!8 zQ_ufGm&vtNN&of=8Jeq4Mzf8^LM5mNz1eN)PQ9U5{jCec=nXm(mciF>2#)h}5zS}z zUZ82^l`G!-XK^p>)xeid>)(*P!RhP;iZuhwl*3GWOCg7T`d?NGIElXu9n|$HdP}W? zTDZ&3pG|EB@C9S^Ai~>#p9C4y_=@}8Mc%v2M|@^;OMVSrB<;8Ko+s6a?>KxR=L<)x zsXBMH9suvbJ$(tJvpkb;ue!XYz9-0Au=lq*{sUe4$k0lhX7Ok8z2f>>eSX2WB441^ z71eAszA!(36{o8P>#a|r2hlg_w|72B%+1WkjvVG!Porbq*K}g1S6VI3^LGITwpMqT z0gEB8y>D{5PyBqs?-DaKRXv7?C4=}EyN4y^@;`g`)x3^xdMoIx-t+0?)t6Ipn@096 zI~VyV6~%9M-vV)*;Xk0QcU~<&cV;V{FzXLW`7MVwheSv&)@gd4)Bb{_{y6t7`2qMJ zT!gz2(9W9z#b8J_@2$7CKE5No1;0BtP5jgG3*cf_zrWCJ5UfV$>`zz6KhUAhXYd{G zrVo*FD=vl)==3k(n<&<9_S)**K=e=gz0Lbh_5BOKm8`TpgXI6AT=I+kIa%5S^9M=) zzX=DhQxpF!+g;G#&D`J4{sY0zttI;!J{IqJvZ%a%msqz_^LG5sb?yRr)gZqC%tFRE zv*{hmpx$WHr$F}>ohYt2F=q1}Z{%EQw17C{(VKMd>+2+Z zw>;hwc5abnX7gwnb>sKG8a%gu37@Z{cOSmDSJr(>XXf6t9tlOF&AAxY;;W$_@p~M9 z9u|r_Lnk>F^7{e3%vO83wZ%`OmxaG~@GG1q#+*uR}dO zFf-fL;Re|R>#TJD#dk^VZv*QY?%YE)JA;OM$1=jN>NVB5L@`%1_Y3%`RNuD`ZaTY6 z-FA!fUF)oR@ekdr{C{ozP@N0vLwE6Cv3>yelFqX;EX>-sDyD(o3ufY`{NnNH3%Dof z-*kdf=`p^T8U>Q?)7NWkH%e#L_%0>xr}h){{Ww`Zy}E9{w|rLz`K`fjes%9_AhL<= zb{fBb!_Bd~)m)!8lMffi4QKwu|3tr#c(3w%oBe;uZ^AM18tjdr+q;ojfST5~_3XZU zzGg2spC9sn&feXEez&5j8haDOo}WA#e}~Ni_yqi2{3m>UCiZ*yg5>MTcf&0{U$S@D zJ^9xZS^E|cLW_It=OWjQDX!kt+?Lu8E zTGxe`Sl=Tx>sZpa%z6bIpJ#OU+q#w&8^-vA!`o>vgixl*<_zu=fd~MqxO6agP*-e zf093v(?fO_>2Z!Y-&`?|w6~W|*(h>#Imdo!_J_9e+s7kmb#(uKM^vTl+~d(AkQQQ-fPuqTo#IRhFWxQ zFSewbq0iZT!R9M*wosF6^z+*L&E9GAoVUDvI_VPB8CHw2H2RXPIa)>@LG;&<2eC1q zY_UFvt4mO3zo*Rqa<#Zi{$?J2oO@!%i+7MWG=*vA>oxhC`aRKlHay@vnBB7M|6u)} z{cGq$2$fqvHFq#k{qeEt_&wV#tcPdgSIm>~*Z5h5ZWmX1HU7b@oVU(t-uC4%LuBjK zD6%78&itxQPt-KM{*5BLs9y8YmleG=!M<S|XZ|4+O0icBjfcYd{a)1nck$ElbI}TH zrdFH2X5^@KF1aMH=8eQxaeoe(@!Vv=at>FU5n|~emKZT6)Nu}m!}~D9o6uCh*W**L z@l@^=)#NvI$fx#Q^e(@cGZt`1>iXvU?oQX!^l~bk)!(zuq=My<-V0}5C)J6spXr{k zzAcUu{Ek8s$_jM`diNXlF8Z6tWWW~9*s}yowk>fZbdyRsYfNm zkh3AG2M^?25&v0)IYlGwwQwdC{R!4H^f$Aa7)kefu)c{ay_f@GfjlnAcNY71@r4@6 z9SW<*NP2b5Vx)6}$l}aa24`lPqcZqiWT(YgKpmo?Xv zwF_6DVfc&o(#TqK`v55W@Wowc=Z;}6ANG|vt^m8>;f9OxQWj?=u^*=M)iACwLHOSepbb3su zh?+%YWDA|GHrL6XL7HZC^|+v%&p9(hO!dg>s?$Su2jDB|QDx|z$6Q$NRmaP0XW?rH z`+t+&kw-y!*K_W)oQsJ$w7Gto-3Rh$sekqD?V+zT^=JB-!I?bj7|iE7v4@H0UF)iHDx|L!Tbmj5 zih2~Y|A_6mrTk_Nc30K&89Up}Pf0N@Wc#kO_ni3S!n2$`(tw-b7j?LihlE(!#%I(ZhalFm%J+(WmHt}ukLoqeEp39xG zKg0S9=cCNUL40%aqHKrDEm=GLyuF|heXH9$l`OHeb zL7e~d#@!}+0k_Rxt_1S|LF`7dyVtywMBgThLd&r`Dw}>;FO%DM=xu>pfVXofk;JvZ;DLYNlc}Lh<>0VV4+dp)!+b=24Y~o9m$QM2YUEs`g zXWFMSFJjAXe<4|6K3Cf>p%xkBnS*?m8k}eAtvu$sqdS?2O|%NUO)mlM4WprqJ}*e( z`Kh4Sv-I_(byK~0g0>QSjGXS7t+(XW%Dm*ysiyihxRcq?`&j#}8%5ZPO38~%6l6>Scq+>b2Q)6{>Fnw_w}l+Op~BQw5D-V4-m zk(hFcr(Iq5N}U$dePpjoLEmPw=W4YB{ah^D-L1zd^u{dBP?OpCeCXz6<{+E5h5WJn z%abS8^P3qis>2+y78YY9`xV9Ty|-eMz1{Ew#{AYeJ8Opd5;cU+;?)NoY#l?SuJeY0?H7z2SRn8Y_D83wcvs}Y$w6y-* z*;LM6=A$URsbtY~W{TnW5O*|bUo2Tt`}U;KFS?z~#8i7B{JbsxDvkV&2p7rwpzFmj zQtTJS6BF$ok?$top`K++m;=6_SmzVp0lwObakZHG=)>py#;EHhXNEWSe)0Vdow@kZ zX73W4H3QAR^$|J!Yd@>=`Pn+*>?=NE=wFdfKYT=-nG$O`eml4~vte4W+0vUK=4mIL zC*gYI-aeFDBeBoGPv?KCe(Z9$-evD8op{`O<7Zq`%jCOqlPIx%-&? z40_z3?d@z7ZKe<6D5kzQ#oJn(J;-XwV?6o^%!hJnUDBD$?5>v6Yy3atYt*KeniY3G zV{@|uVNgaq1L#D`sRg@D3bS?Odm6P9V%S!1H zTV>RK7TQ={!`Q!~F3at0rn?Vzt8YfK8ClG}b*7f`Q^!zrQ&zn-N1^7d4!hmR!r7=H zmzq$D|Htb4EqRbL+341#S6)BvlSR-CP3bogf5lt5Gx}XnZn4qcAUVaEr(xtP?Z?LW z?q}C;`{|Q=@7Ng1|6D$H@KJ_tJ+`0IPuH41h%hg`)upg|@mwv-(kaYtIkX7+vD|-B z-@EGe1g)TM{pptC>slFh+IY*LQmDD%vmYCEtzV$m?O#!o`h0Fv&m;D4kd=_rz?}Mn z7Ukm({eQ_L%~xr5KXWE^F11#(iu|=v`%Z1uU+%@|@AvbrxV)B;ZG%SMwqMaFaxA4T zyV-t}TMgw8ki5(1v8E#P%JV!Fo3qR;y+IJnYEr8oh&b_KUl}dfdU+((@>Lv&7IqT{_cg zh*l_X&gE1`ja!?i$qC*tdVBC8a;RwkIN3|}7;7GO%5|+fHGr>mX!#D_bMZcwcTfDg z@DXfgvy!uk?!j&6s>-*TyK}={4Et&IB^Dhe&r)byqPGy*@H;!NdnTXC&X=~ghWu-Y z!=FY m1}QEIk0t(@sbb@bjlH(P#J_!%g^cgQ}1D`FX-4x90-%DO*jC?PW4!W$XCez-|My9C>wm zBk%{3nN@Ly!)kebL$)7ILV0>?$kNsIzN+bcHY?+sKm`2~kiCH40{XC)-dyqiqmQ@D ze|dF`R@+AAW4YK*izA?(dte>uT#UUq>yr3+&UM7!SJReQ3kDN_| zj#1Ml&Yz@HL2MOaj@if|h9dN%*cuq__irA(2z7_mHK)A-=6s#_qV3PA&nAo}kFh?6 zKMPkOPAy)c85{WhguhNFmQGdcy6|=?Z$vt8l(@?1S5v*1LS9q61^M^_A7$@qS~24H zq38Ri}w!}k*ZJ~NRz1-sUno4A+i6@k`5*Jw<$V=DR?q)sw`vKx)}PDe zpuMtUES*98_T!?>nemz*;d_PNXzF_aX`9)HVo;8LIsR7j)zbQg*?6dy@8gT4@!Ln8 z%Gxei0? z(cK`nrueqg<;mR}FM*HZh3(F!@P%rpJ%{nEWG}+yWQfhARl~|+HsFGUDwijmx#^xZ?za!*L zW7VsWceAw|vw9=Z{jk7nRCeYv{;65}LM(Hf|IW-6ZDTI&J+uCTpVrnz^PnFFylus4nE@V0UkPdT+oEA|el{N5DjIP2Q<+R32~ zx}DxjF;`>zE3wXHYmhsdqn-O}^D!l<_yOnjIh? z?ED4vF645yuGlJy^*1pzEU%|*%yDNMwfFZ{N&V5U!T9%}x;Vn*xE8++aywt2egha( zP^@GP@wpS6N8`|1P>KDnXi+^Jj!uCE@I|D!KGU2 zZy>S+vd4P37T=idtLoeY-5|Dh`OKu4()osehrgFxpYctg3L9^#&%5~QYS+}6LOFfU z)ps{pI{mnaK7&l1++8!>Of2Q`^`IRzr_;>2XL8%fUpje=7t1jB>^@&l)Zr4nKjiij z&DdEj)TE7>e6BA)s6p9if2TP!z}{G>UV>eIGTGm3{Ug5z&A|)t#*^PATg6X`I^JgQ zS3z_JS`MuRE$GAtc`M9gBlj|$H=`{7^X<4Qj_=eWupGCw_eTv_2P8-IcHs1-D>K0vU>6p>rQ(8X>Q>t8K?cR7<+t^ZKZ zitgu1dv)cy5B(Wlsp&Xx$o%|jV;*Wdldpl_qyhfTM*hw9u%2}pGgVGMBI$K9dtrRF zwm;L}La5{1p(Ng0a|*o%c?$TAC%;?`^_A{t`t?i<@ocstE6e6vXes_%kj+o!d!$}< z*{bYbea**@_Wy=@WR=vYKDtRC2BdSYkzT6TV7cApr@lMW!2SdE-6;OHd~Rm*xf%R{ zZXY$dfi6z(zNuGJZ$V+`Pk%F;4I8T;zr*=Gq~?iqCXvmDa$?~MNZHhES?5MW zpe8$OGdYisHxDDys&7H>ne!VIjQ zSHsOWABWgm&(;Uzv*15|26+P$*{*1R9{cymUqPyPGXfp$b%G`O(iwfIF5z-m6yaSZ z`!lt57Vjb5F7^kB=QO@rj3$@W4}AI-=G?q35O>v3bC>kqh^3Xj47Zorj0^}=hiKm$ z>t*;zvvC$*5C0^Ce`jjxEoVO*UC2%?v;zIt`jE_>Zq3$svgy!R{%g=EdI92mm+uHM z&u1q&-Q=y*3%aovBmeH`@0rz$tWA;}&YGoS^1IGNF7;hPmP|g|(8I7%-aVXa!dGtl zFUZojHY48ap5&GA(fn76qnE^IbS{VAr}iq7Z6>QF_BX8Gge%sA)$VXMv(M%{aX!)e z&F18N^nGXBwDCKV-@#(;?Mw`tpODRm&2Xl$`sXy0vF@Dp-CTa}(aGG_dkbyUs0>;I zV%ccMPj7q`@zmmPv07Ej?Kclu4s+I){$>0ZYnY~$9%UQiElD*`=2O4O;hoIvj*tsvv z?Gd%9#O5KgV!?E>d&l*yFWXh+5Md5O6XZ|+7TIm*Qi$`s^$7B%?&w$U<=60yb<&9b zoX1;$zYoRbx6*umtj2FUU$KMV^=vj*r%iHRAf^Pd{-CCqM!*ygj_ zp`#wjH4S?m#In$O9W1c-Dce2yZK?kS%uMD^?iK%O>F!~>zV$%s7UG@Her9)sHULo8<8_P`>=NWcM?EGI}F+*-KEb zU(f}7Ej0rVYJ1c5t0OFTCbW(@BF|UA_g;*P+?Th>rnpZN&|TtLfVPs)TJv}S|A({H z^yol7ITUx_)hM#AH;a!H)(QA;#oS6P>DjDgW?tJn;{LpcPgJvC=tQwsp8Y^N$?K`V zy&KLK!OxUK4KzTmozTkmrmO7?sIK<@Cm>q>t@)_!%uw~HWbU`}d!Bv@IW1JX|M;yO z?cUKFMrX17GVm9|&JeTTzrOd6e4SX9*qf|g-;i~7XMaI|hXC{WSUh*sC{qJ>ho6+} ze=nxF=DI#vMmAqLQ$%e>us4qE9v>Hyxx;*Kux_n}$Ivd$A58DglBaFx?x7=+>J=mh z>y+}TV=mU@ayP|Z$ND~7|I(SqUuEah+uJMtpXj$?_cj`3{u;UiH|Z5=qz`hciau~Q z0bLd9cdyvar15>CSC#GcNqYSMi2eZ;Qn6*;J0{Jw+7)l?_eKfdH|J){t%LOheN52% zPIN|-S9ggwkne>vDaWD}HbvPgGTotvdoPj#Cwr&;#CCI1TD z12@w9cR_T#8m~#|8-`ES)O^#O@6OK>V=3!i8T|IumjwJUNaU*>-<`!4YqmGn2#6)MUdTuVI2V}kAoz&{XPe=CV$a}Tgwc>lJ8F?ouXX}T- z>MY;K^xqff@PdAm$Zv>u{qGQdoBQn`w)WzA*O|}gk3qNV(_1CYjaXLD*$Q*{=%8Q! z6VC+u=^?Y)e4o+%6GvAu*NXRcs98t!2l6fAy_3bARok)bR*N>f_Odv42CbxSU(u-| zwnggn4AzLdR*c@st9c9G9vA`BAR*SfT+02B=XcgW!8mqj$fY{lWz?jjeB<rQBesdVIs~U--`aK1Ay#b3fEPUn}p8TKp)V^7!Xs z?BRR|u~ac5rJRjtce=d}W-fW8y!mdOT_4#Rnbo($-Xk$2+7E2)jS;i*1=N;jL+faD z?TOws4|m1)tymJ--$GVbjkdB`&G}Ax(9Z03wAY6HMfh>fjAyS(8*h^Sb>*`wegKSs znsMG5=NI4~vvCLwlY2}i`QaCmEoLVpdyDm{oAdqT+l1dHQR0&8{aoJuw*D;vK7+FA zeAbzj^8d}cW^Ug=zEiaGZbRlHW`G`|`@&-o2q$OCD{-Sjl>W*rOG2S%lxtIsM)#>m9RKlb^_}?DMmPzxUMg zJpBpmyh1-fmpk)Xy*sOA1M?m4oBmrKx!Ehq)>P|)Vcs2dEqe#hA@(n!t<+(zIoQj7 zo(}F|dGVmlATenTJH+)L`*ZNCpq$xU!RL?o({Mel8i;X1HsAY1@4H$r5c_dHs#>R1 zpKfSpHJF0Vff#rC4Zb?&b8oDh>dOsupE%NYbg$K?dnGl+cX4hR{XX~}WY6TXD6bhZ zQ_Y;)L*5*{gg%EXo%EPq3A6%iqt_m-V>Yv?YcG3~t!KkRvsNsb-=bm}pih^{{*hk| zcdx!)=j-gZH+=3sR7YDtS9oRbEj^otpIFV@pr z&SXZy@U``_H=if$b+8_WjzlNOZIoID>Px(wYKkMfm}h%)Ui0~o&O`mkTh}`awe<6Q zvsnguTaSbf@YB(y@CB@+b5Nbvv6)IQJ`(R^^7Ym&x&tt(Y>C^JE0c;sq}7Qp~>cv4-fit|%;&WHH@Xg^FGb>igQP@To`ro8s1W6QcTY^PJ#tV|ONlL39@ zzLI=|cyGw3g}ywYe}}BMvzt?UBN{ow=4kQlRJTa`sT_X7*I;|e#4%kA_X>F1(Y<6p z!JqIie4-9r&3RhBLmHbC_oq7A2)+}4Z}cf!abhUyo-eTOCzgF^;}U-3h_jBF`j~DS zzMsM_ar~Z!ya`)o??zgE2=w0Z-&wE9tH(-vOYFt!`AqT$>iMJE-yvTh{ zZ$_sOE7|Atp7Py+d?ndO=x)Adkso6B1>NED{e}E2{0n{5DP1$~2LGwq=&ELwtYksC*q!$8Baf=x$hPnHwPWfU<>>l zwo^sO!TxCL4`DW}gdAnvZ$58|WtqAh7I!b_?(0w9c<&QCWz}J*S&X;tlq8dD)ok7y zxlXd~;>?F~{T6>1ez4a?JgND7VIA1Q-*9X$XMe6(2FajE|pU2-R^lld4Z@qYg zPti(lbjxP&d(QrxU~ep&L1Z6f!`olS&lq$^B%A0`SU`7RNq%*N!$rbblgS%wD}1Gs}J*@^j=Dow*Om+R}q2?nO6#28lhujMXGd)YpP}KESs?d6zEI=AZ1qc!TSk56YaxC)A8m5^7U0v0 ze=_|C_!Z>!)v||Kcp#T@;y=aD5H{ZE;LfP+B(x0wtI>U>#O~dELGO(9w?XbAI!P=; zE16F*geA&J|Ei&_pgW9$UGm@Wd~-VATCe2$j(%Lg4|Dbf8Xjzp=|%gNf2Rhk=#=NH zuXS))Z@n0o@HIpaH>%-Y^1qH&FkImtXH|sCcoYAP0dyYwcIZDHS|^`?bF+h4i#5LwfzU* z)%l0Dy&>}LUtZnvd#l8`!?`)$hKFoy6;rGp=9AlWvQO&h7gQ6=R@i<`(P8i&WaaOeI7j9&8`jJ8L z{jDkIFX-(w2e*ssr@jBgvc>*BC@AlL(7W&)s`Hm1wvp<-!MSRU+&%j*ZTEmLHDSH_rv*ia(pcI5skg^?&)OoOZGyX zea9Sq=RU2B5HG%lSYF$&C%@n(-a1$)wo>B%T08@---RFfnT=+#cZiQda+;V!Tx6^C zZ3^9*Y;EEv9ozHGUMe*mCC9>ai&-Z&HA657x|^HLbcdB9t zBgf+Cf{flyeK{cZ*Yt)vw?Z!_*n2}=7NA+h_652F3ep|TcSHGKvu_A-qoL~D8Qm!M_ImLS{wsNf>suQ!1nAc!Ih-XgFOCE=x+;*kg*y#V@RFbI?pHj% z4=fbpC+OGgG}q^|Y)lYmYBm0yyso$A721~ld@a>FNned8^V8hAQH=MRy+Qc*VQ-}O z2mOY!GtbmRQY{~_ z{VG_`$hO%FY@_db&^k)Z@Ri)tdT3kd1_PlXoByi&r}%|XpYI}N&A+~W;7+aeo{SSm z`nKLw_qeuvo{0TmDfh>WEmo7^CDle92h;i8y1!Xnjvs0NDcuwJD^S$=RPDTvFgTq# zGZ$r(*6P^>UB%~ne0_x9B%d|rMEms9kOAHtP4E zIQrv9!gPE+v8RCr>U}1UJ1qap_8!7L@^|%nI1n^f+tx@;2X2z4$d&a?E!u)EXQT@aPk90i$&MBhv6I(ihY zHg(*|R_dH+Zx8u!IlZAjO*@jo`<2*1*YNit-JjHXFTH(az0AsGG#t%jZV!v=Gx2T~ zcY{uJ=)H*!gaAkIe}A^^SFKtuKmoo!R}M zjvVx1hZ=lU(7muO5v(WnimT;g>kq{+Qf-IRt5R2=_~;0%j`49b)$& z-v`MO%+dN(W>|kh*h*1PE!1(bJAG6h<;b%--$6cqnWbno`QH3BVdq1-$>ckV-z{po z*?KQDmBU2wq>=y6`nXT+-;;07`o6PlN24X>-5A|%eMl^aVXXD{@{8f)K7@*AKUq`0 zk|)iMTpn6a2{V^+zfO0BdWO-9l zov&LzgCc74k(hU?@c{MMuNR%vFp!NK1TH#H^S;bb40(Hy8SDc?Bd+aR$aZw*F;RPj-8fh zA7~rp4xz{RoP)kAt}XPti}{fCfwcae1nsL{Khs(6-RWfp-cipd;tiuWlFk9SEz6)! zO|A86hIMzgdXs&nc3;EjlJ>*BZ|uCJcS3w6)hDExxl^;s=>P1EH!mgi^tt|iK-N*6 z6ZtwT&kfdr&U7L>iLcM@B)y9w`!3RN96F=)uS9b(@>fDFM)H|UKi?2{Z}J25x~T06 zxpXF9MYau&!XNO9T8)y!pRRhO^Fi@`v#D!EJ;*GM7S_e|`(5_aslzv7_*LB7%w;7y zCG__Qod(7I+XT8q51#2`E%{{TJGzD6!(?^Q7EqFpG8ui#<*}4}w^}@>|Ct&VG4E5z z|D{(!EsK%oZ)w&t>pAF3_ZpR;i^7`~Q+MBcZ2AhqpS@`C^hx47|{B1~Q0y}rn1@_j8 zCqn&7w$@{|_UK7le2p}Ii^=5!eDfG}L2IkYZLwFEYr~{^kzLJh=GlKjKT2(m*pF`G z{V-1*<Vo8YJ0Uj$vmb6t#!oWG`jUJqakA34itLIGgq_}sXhv1IBKFlglvLW;ri8G{~8mMhKG>e#~B=@_>EM*ka zC*mopZ*QyFVsX6Zd<6P#9`&FdWfHi<=rE1)fZ%A z#I#V)9`P}qjc9!G4(vd?DBnl1R>A+Shg0R$8$Sjn!(51=zZpFvrk~KW@DcmZ&^u7W;%!P^U{}-(G`p`*i>%CFy4$uN^LG-z^C$JEBEAQm-;?6yr;3=ykY(m; zsrr6p#?IOM1I9V?qw^2#zfRKW?CwDXM3*(sXuO^^=4WC7??pAUNVWi0!!PQ+2mPy> zcMUxcFX=t1?cWXg{Y?Cm#r(JZy=pbyocx175Ap8Ds5<7#{--`Q|!5$^BYGHb8uL$ezJW zc{XLUXnpt0nHuim%G~aP8XRV8V^Y86{-HSM(s`CkuKCTQz0aNPX2$Zk!@cBw+4+FD=F_ayuus%XGRl@HU9~XLl&OIHp>cEyflfDdjv2%COx+ee23GI@lZD(7&(3 zV{>-R{p*=oyy`lO?)v znp(Xf_QiZPQOluZ*Vx&w4`tbU%M9-$I|k*f1NiyanU&(p-q<}AZ!8*<+?y!IMEtA} zy+y~U!JBL^!avXDH-sL{5!ZP2_@7?fBL5GvG%>pn4M)_e724amLFhqm$yD@z8Rdok zqIP}k9manmj`Cv48mY(5W#wZo`*{>76iUPY=vScA5&y2(SBZNTzO1+hnu7*`W=U@! zsHOix&VZlPa;6@>p(#`^fsJp4eG zuduwV|4r++o7hq`cQ4FQ5L!_@PtEuH)(xz$tM?7~7Y10*lS2#>%@~$fX zO*)s=_lTS)i6xtyBjW9eH#L8^?9X@pm^>=eyKOz${(XBh*xQPKqF2A8tr@H&meTU? z%ufCmY9ZFo=$*>pyuB)797+B(tG9#QED`Eszm2oMs_lEJ{AQ8=YdTq5`gb;1l+pWX zu9utFgZ7gCo2Gk`4}oLiSZx1}xcy7Vt%NXAUb*M>QnDOoV{#5Nfq$jehwQzR)0d3W{xQqmgch%L}}JPUlu?n63d@tg3g&1y$|?_)g_9@*;~ z=eHsLE%Y#Y5?+3ZiS&T1|epz9&0x*bKEilwbN zdCKMscO^rDe@}om?3G2^qAk!d?%mt+X|6Zz$ok={v)@VXN7H+Ao$ri)$wrC}ekbK+ z2YpUn6`fz+?VAGCBYUM-7)go`kp+|_e1=-&;`y%>g!`Q zcwf#v$bQwQ^x}Wld<@mc71{MQi@z0|y=ecYd!5z!z>ezRTnzd`JT;Qy#=iyaN^H?{5%u zcrxkE;5)MQ3BCW}4;A(9vb&OAvSPj|&Xi9r-v(yMtfaBm-Q2ab?h2dp>z!JpC7)(J z8y3SRI1H=UIgL)><2jnXq3=zSeL1B^t5`2g=RFJ-3*EP^1IhEs;a~T6p7mYcbnB^d+=#wch#hpe8r?*n!yjVE(OAFS zlTv66NYsOlXkXZbDagm*fzKuL&)o6O4q(P|^7|;-9P! znZ)%!_u&b@zuP+_p3C}{P(psq^bw+=3WSS)t(e}x4}x|0>1Y>z*W)v@b=LW5Vm(FP zTu%mz`3e3D`&sO-cJJ1TwNzGbU$lCuStz;#}8pQO}yXMaF*^^y{m)IW`Bw}*YJ}ap99{pKg-@q*ak=8`$*qf^idXbFSdGW zR-lC*KowXg#+GPK=SQLItvAy9!MVTl$kCkSvUkqjP52K&)5y*J8WF2M?$xjA8t+@4 zn_e$;C}em3OLmswbKrNNdF+3|{!lsfapw0@`rpcXTv$GZ{KjT0yBgOQV>vZwXYWU6 zGw5MnXD3<56>*2y-H6`~F9UqH(WB0e6i@LmbyAlM3Y{OisdZ&E$ocl@IQ1KXehBa3 zdz*m`Vme^m!7Q9Y59Rbb)BXLhq`EuzlNdtUs2Nm+2I7m?^B(v?uvt9$=;en3P@4b4 zTX@g+ zF4goXg*(;BUh}$URGv@G+-UsxRQ}#Tm%&p#Jy+N);(yC04tuxpPatJI@47R2&=4pG zL({Q`E|veAXbExstS8^FABpcI#>3|A2<)@}BYbP`G8$IE?=>_{eeXZJzhzTb{;OL5 zqXt{)m1Lt)sP}*@Ks@u%HSjGQ_7;pV`{(f2;1T=>SsM5@n)NWWwYpS88$c`QES?N( z?^2VsX}zC;zNKoi#NK9TV$RAJ^9{mB@!eAmchikf&*BZeGf)*8!XEN2=sGhrv51`I z@Uiuue6K;jf^XID7c`o!Hez{!??{$f+{GLD?z{hg(Qk%NgePi04E;eJmh!n2zn7ol z*1zDp=>0G1`vRZ3v2O#c=f5->7wq@8T$Z74S(ia0taI`a!~d6L`=N3MGw+)-!X3C{ zo%FwGbzE)t@l(N!*0pW}ec_qe8i&R@e_q{IlNkuBhDlGyrj7I zz+L<#%x@}opSwGQoB1ARQvbZ}6n+IX<7;ecbB2Fk{m!G;VYM2hZ0>$S3-L#yr6IRG zrilMZ7T+0muc&j*NcV${tN2@DKFHTjx<5b;^2_K0cnxXe*%50Av@C37Z(X21;oC;L zPg(R(>^;zmYIKl~?e^lFnH?fF{8(|mL^HPV*4mFnt3nw%hpl@S^4%3fKeZi}PM-FD z%joZ-SnsmF{b9Y$T?uIEoyw+Xd_~~PKpp65z1@CpaSgNX#okQxHNS`4ft~oH@G~F( zphx(8ijLBs%&pvYH4j54(65L#g!a%2R+8;tyR2A0vt9;Y!Vb8n&d1QxaFoq^=!MLB z%YT8^?iW;n=6aTZ_JX})NG`W;&D9wBC6nK$QY&GOtH}Qvod@t5%BppqHe%J2lj4|U zh8kFpj`N*yW`{E)ttZ0rU^9enfsFbwUJTFG<)QWWV*3HIwl%BHl&h`|rMv@ZPPRLf zzXikKJb5xPR=Jh^A@Bj~rJd?IovY-3+?I+wdblFU)I2=TpeD5*lCD zdt`4{d3m6h;Q<8rZg-1uU+D}=qjr4%a zdTY9(Uz@$xWIvmMAm8u^CW1OU+7&W ztA!uQb`g(PEB^jR{;R&#CTBjNsy>dhSI0VI9>06t>D|`fQNSG4@zy|VF_l0oKqmg9-T#jG z{xBMT74I9~$wl}KYFbyEU*z`PboZ{1J%zx!<^by1tA@ry@#l81%)o``vwIod*gQVj9- z&f;&t3kYne->Ka{^^e7`;J2E&c%cSety9Wz6naX29rgOJbYimqHQ99bual)0S`YXA zCD~YUW@hV8^{Hfz#_4&ET+TM~t#ChD@c%x31}uXgg80&#_VQ|JFK2{%@9bH*`X}_|BrGfLyTe{KK2m1zTSg5JsX-I6@!@LQlpXrF+^-U5y0i5t zmTvVHJacqe(QQjS)eYAVW#v(aHX7=uu@SCK$ znU~in@4+tok8oPt<%7NT`11k&T_~Qu-gFlC61mj1uE|GNv@cARM@}*HX0vowH6q&q zzeS3}dwv6dRPT;kKagX=7WyuZ^?F+OC@w8MoXFDh5XnjbnCy@1w@%zKvzEjNImh*Ep&dp|9wZF{fGdc!u}Lfq z@oz?n5A9l3@6gWlW}r(Uk2sE?-O9RS`k&XjJHA#LbL{(lRlE^sIj9Z2m6onlYIZnzgk|>lN07*e4IO=ucPeENwO)X!F(PH z@SBmWllq@ATZ`PEE7nKKi^~i~Iuq=>9VE`P_7@g*uhHgYd-bg^dX(<>@K9U}tv5it zJMcZaEQ`F@?#kae=T3^{4>S!~apx1Do4v7WJixboG<*V!;b>7k(WfK$lSy*Ak!CJc z2XBXZzchCt`0md4fZOKoE$i~?F%9iWwhsM4Y{BCHvyvFBbD8(8^4ur?JaqFWsyS41 zXKSIooa?QJ@8WytO-r>O#r8A$kL7iaykTCk(fg6@6=Z4bPtD;wqF&Aa&%s@Qd0lmR z0G}CP$blJ#W`G%x?oR1$7`juCMtKS8?(XjH1_?n#I#i@nr3FP1l-$qkKF@d0x#ymH z`rhAvaG!0dzHrmJuoyebVN5FDimcuY@}qDCIy?U!?V!#fdX>AC_Yzj{(+uqj{a|W( zcTK!A*?!`lAGQCvv$y3lQeR)u323b@=Ba^L?~r#9|K$j8nOqM9>9zggFdfc{xuCv0 z%3?0e$tdT3B){w2OY|NZuM!)4bcI~_2zH96b=RFc=X{X*ceB?WrrS>~u90kgjXw@o zpohKJnbpOe8>YX({FJTiH?zhFdKF8}%2xGuLk;rPe0+gVDb_3YdeKWNuIsV=J3oFjAGg?fLgzPeoHDb` z1H>zyO5_Eq=_%h`+{1?Ww&v|K`PGW_T^8>j=B6^e&(!sjb8XZyJjOoVRPOYAby(?a zo@~AsF|nmHr`5#<<;2!ipF6r!C*>T?=b!A|7sq=056}y0{e?I;#M3M1XYiYh{Zuu? zlfZ8i`(N6hu8yt9da>EZdKCVD=y>sNPU(G9#|2`!P2N1O?+CrPHQh74eZgl-eJO9< zgO4Zj8m(6?=sm?xCSMNo)#g0^#e?04Lh2=sKgn0heV6|CjSNT8#aL zd@sOvQoAAgw8T85kKIXiq&A-+=9P^r^a{{#LpIpl{+!e|Nl*Im zw}EU2^vC~z-hgNDQ0+R2H$@%yik($rj3c));%G#6FsnMDzsv8o{w={ja;B!4Ii(i^ z*n4bUPQUV6&x}+T>(ILHGXH-i)~A?0P454Zjdps7GbKHqq&r)7*&fIeB2Ty8+wHZ8vs4 zvsYK0^7ED3e6^<6&whKonvJdw)i3o)sz+(U>9Y4hpDwAvKyd{(FthAD6K8Q}4zX2| ztOkq|W6?0RGn?J=_M%X&f~-#?Hb!dfrEusAH_g&J~fXJSwo?MxFMeE7dvLUR$`$-XOll;fL_m z+xwG5tv+Y(Pqv1VT{Y+NnwTjVX5B?B6XjRe`dfAAYCXw%xV@QVFPuv+sjc{6v%ScC zT*eoto678TQn%B|^-&zp^&p=5Jk_Vv>NSFm66Pj@^Ha^-=%&6q$-Gng+(RB$*}H5n zo*Mn(>`*p8()rt)Gbxt7i@S@QlX$ZlHZ#j`hOa^BA@gusUG~XmE!~lPT_*2J|24W# zZQ^(*Mv-T2?p`=o7yYrA8lvySJq6t;hs|c{sl4`xwVU%d$!{fc|KwghskcFl)5M>% zh4&vSLU($9mh=wEXFp$y$Tq^)@Dm?B>{p32m)3j5F;ss?t4)EH?mHAwueNAyI`75u z1%Jz}uky9V91PaSFUcN}ZB@(B^62EQMA|Ra${PaT$p3@0Z{o5cha&h1?!e#bdD#B= z{Ct>`NBF21yY{cCSBBQ+j;;UHumZjzoK54+66Xl~T6)P-$t9im#nI8;82)3+p^@JF zL+4dRbD>_zEBSW`wlB%Kr<`A?Nha%7ye!Pz^qjQ-8X_?4%|C z+id(Tmd;{)nLw|qdeg`!qMhliN3+Fsr_k|y-9q2XFSU8go8G%b=dfCAN$Gc1HNSJ& zuZ}i?b`TQVZ!NPm0)IrVYZI9d{7yIl=ivrSV0(sqCZap|&r;p@SsyCUuZjP_=Sy++ zz)yCs+qf%}=uTn3mRjs4KL*ot`$nQ`&E9)7ehvEc*0J+Ep1&L8yFX;DVK|%p(E0)9 zR}L%j?e$~|`;XY)NdI>=eMg?c9Hp%3Z#H_LnWMC7d&a%)Eyj}km&m9NS)HZxk2trZ zQ`u>!?pN?}%-k1rkMO@HtvciL*OGUB-+w*oW1W(Z;bd_m)Ix3NWRsWm6!E`e^D_Rt z8H%4>J=~Lv3Hh*>AkyEAwY~jN0j80)MSGaDo&3+kZx`QD^`CCooooCh ztiwNCHb42$8T6{72i%Qn#;GfBw$h>> z`6z^zhPiSYrWUJ0^u=8CCJSO`7Mri-I2(OwCeM-ogFZy-scT906V&tlhTM=(eX5{6 z^QbXe$Q+DOm%;d|@*JXf_4u8`-o*UsWdEtXEV;Zl>SZXHqlv)-wofvdHB!%@9t+%w7NQ*0ZpUEMbJ3r;Z*_k588_>z-P?-lTfvNdo>R}l#hKLKXAQiS?o%nWJ=@LD+iKKPpZ;@ZqV+78B(M2m zI1sZI|5biUM4A_L0r`9KDqZfZw zlX>UF@)w(}W8w@ow_=$VV<)58ZREQo-@fXVj(oB6*R5AXnmv0Z9XQ8 zdyPDn+RNEkuKH97Z2;HUt>bL}7+%bC&@O88t)8C3|D#83#F1aV4cO?V|JUi|Y2tfo zZ&wO;osa3_UB>?B=B%0={>S$^I@@45d4*WsPCdCro+_2t^eRbH^T=L)^|(;4V|RMJ>~a-zaG{@VS~BZmYS_X?yr3Bs8!E= zVzZZq|2OCww)&a5fAzHq8*8mAwQ%3m@iTM?bXDgs+>?W5vXneelRaa*8~JnmJ4oEp z%)>h8%cHZz`O!@D#1Dnl{Fe!EcksKQepYjWK8BAa{k_-9cNOx$r!XhIzp>D!@$E%A zSV}Fd7r;6>cjtR-DswOH^?YuCz}9|8!wqwm8!ZAW<<~7SnUJ%q`Ax_LtJ%Mt%@ZKyU_qLqAQc_x!|m z?xMNc!p2=Qb1sd0g0EM_dy>ptqmR@jh4_bmqW9JV_*jKL%i=9^ev|n6^VdaBUaH4; z^x{=@20npO@(iZeLyb!Xduzz<@wMKWN%-RQxAFNEyLrei#mq5Vzr^TdR$F^v)%4!^ z1Z)kYH^tpfB)4pO)85`Jxg0Y~g#*MXpI@Er8`s?v`#nB4s!3ciRF_lQgwFbA$FrX8 z+|C?oXGUG(&NGYtTdTsfD(Jx$JLH zkFN6Gp(ZQHcECLH^XOIBB+nOUU@blq$c^50{&$$MreqO0_(Dg(Z)&s<{W7+1rg_*! zw?;B^sxHmr`R%24HO=Ka`$yyJh4-LO9&ecVcbNBdZ2iICz}oH->~{X0z3twJoz~x& zg_U}63!gogS(p1p@h4z^58btV2GglghfgtfMn8tR*W^Euzh|pHTmQ@BcY`w%+=&ll z3)D6uq5H@7Gj(b#&pP61$o8M&=}Ui*8A;?!>$?7);Oh%C1KoD^=izHQUqR32tK9+! zp*zOin5(BB`6#K+z4Dn0vD{au=z88|IdnsBp>_G2h_A)Ycy*eQ-yDay-(>04rMI)? zoP8sYl9A?AUla2&!@6O8zfH`>AT(Nji_kUjfS==NubloK6Z36;VnzFosQn#$PW%ME z_S;Jk;qMJJjdg$W(J%)B3aT;sJd5ubIx^DyqmQ9zs5`*_0q4>;5FFlrrG#_CT{3Eqvq3-x-0C|vVIXRKXw)t z61R1EF|9&(LnvDv)!-_=Kyr82Og~{ajC^)(@uc!yY-Ik$Fw87|A=ae!CXv_T<0;)7 zYO<2fE?A@Xo!rH6`E`-!628`>;f?)%)T3NzU_tMTSqVbJ+3HUACA-1qU@kru`AmIG z6ya`KKZF-b62FPN0FU|3$!7_CJt(igJKS5rB-b1vV8OY-f zeQW=EM!)ZhnRzkqFaxEVnst~+K21$WMf!eB|@9^oD)~H`iZ?>kec`XVYJ7 zf0@4YwC?9leIDrht;fg2|BK!obN|lUyTMlhcQP&8-`R`b!WcntA~P35-KB5%?u0*X ze}H*j!uDOZ50E`mhYaq_8Tpl>yV=|%Y3XlwNLk9a&zb#VysP$q1^D)*bzdX=?lA}F z^zI({w{%m{jifhJjLCf0r-~}zO@w-$WxzX5WXB}A-+w1kEncDnh{Tqa3RxdiI*+@~z zdzQ!<^mD!MWBzL5TSIR+L+72dU+T-B{G7Eu2Ysw7iTAd%JIzR<%Dx5WdubeV7U>Pp zpAl;JojeEI%WKY-W??I{cND&3GmP$ew(iv_6^5|^;%LuC7u@Jq*&n5fW#nOyUgVf#>I(_w|1OI=+a=E4xD?JIHrW z?pN#`B~MQ`1FR#P%V)frzUyrCRLjEn3Q!y7+3Swx^F5=zDRVb1sUHHO^gB_czwxiaSbd-#YuZxtx;BU88?W zU1pILvY%fZzmoly%Pi;AH$K+Yb=Tda^XTVnJwk)fk7!0dN{i{0x)iNPPw(zK+a5ne z9)tB}2>v%V=b}5ku?^Y0Z#~tle8uONvDBU4e)-fj)VJNG-ws zc`5dXbQ-afMDDNo`NLkBFmIijMMisX;1Qj;`d=U4m~LmZwmv4zuK(FaDu?=jYdR{2oXu_wM3&LZ=7X zU+fFBd6U@tjGsl~9H@>@?GMPKZ)&nxZg=PnYNY;h{N3InXLehkf?i2{OUUlfeUAQ^ z+${1Ls6P+v1_@%KP) z(y4W{^=2{6^JetLkA?2+EJbJO;X!o1y({Pg$jbMh;w(nzUo-L}UnQFP`%n$*(qGR{ zP)fBTn+^lypUs(@YLFehLbicU4*Vni>8IC|_3H(jgXC1Nxi})!6&(f_>LSHuNK}4sN@aEMYcihn^<>-H)0zdY2KY_Vt%J4^UWi_WPWzD zokac8t4E{M^vr(2mcEs2bmeb^e*VtiTk%gO&qE&9JfuzGw}|yWY_?F>5BLQ7@ya@` zn1_k2Rx59N8a9e}6V&jnI9}UdNWKZafirLg9>c%(7Nqy~#N55szD;b5boM{;f%@K& z>{I>Ru5Wq8aa+CL*!!2>VzYb7eq^xs+?|LI`RFWU?=7A0<9d^<2eC8MJPk%eBm5St zBo@7FZhyD8sTJRY^&y@dI?3@T`46#v$400;$3*I@dX(n#XgdE*v$ELraiN)C=*(>E zg!T&X72$j=vv85DK3b6MJv+fw)GM>M1ZK-&jGEU&>y!5r(;R$n{hK77+In?|&z0iZ zEY96@VyS6Yh^rRMvarQwI=;H{pQ@_21M)#MpI@+3n%;W7>_gVgxyk54cp~n7XbF2K z=oAuPnRsRq0;_p1psd(FvR~L5V z<@BCfFH)Z%dc~|SxD&xb&;%fO}=yRW0pqXAIf#O+N_XQLHh~GuaW--e?hET@`qq{x~b`IF*UbtBB#pw z^iGabte58Y9-*W8PQ=C-G!?tc$j9RM2D`r@epl4?O?T#!p5^1aqV+Mp+oOYE3e1NA ziTnm)zqt7*BKKoi%?x?x6z)v8IT24ou@tqpg-+=@@;57u(K_na7mY_JL@YnZYXw_d z$i|5!K6z92MyTymvIN!%VK3X2)4B`JC$cW7cZ>D7a5i?VKexYw?=p$~tzK80upScI znFfEMfSvLDCd@!rJW1d?Ip1+7hT5NCZyvf1lCX6Oy$Qd= zFeCaX33+{U&2IrT)cGxDr7^xMoYeP`a#@n!+b!2I*58v~fLpLP(A?_9C_1B|EIUP` z^+2rg)j5>UFz6z`G2|`c>Vq6U3-@=kb%5F>wZE0mC)VNkSNw!E@O?B}<-`0|l20FT z)+GDZ8`2q_D6iP+P(P{fkAC;hX_om;EtdRrE9=+xNZ&uULdZW=r=@a;XHN1p^tTs3 zdz{~*CuQW+k8EqG-^2;^n{IOZJH=SmUK#5Y)^GITBfUh8_<m>bch1>`H56(1=`L3}O(Nqn|tEl-7G!4H4)cIX8_UsMKbsYZ zu!8P|{LAV`33Ux=rWa}vLHE5o+D&|o$mWTq6`lW_ziYpOIuzo2uNY3*zY48m>zSH8 zg)wHVqPa@Seg=8(qgR;zMm-5aW&KO- z-P|d+$}Qv#Gt_7aopNIOHJO~ztZF-r-ClXUIcDfr_TSSTYc{X2G1iP^;HO?o@2Y-& zV1GLO+h)DDT#JbBxHDDM>X=$Sq4yEei)ki*0qiu*?Az7KU4mxbkJ#e>$j%h&Z2SjA zd0+TRCWotJMcMor0*cF9elyhKRIoVMtg5dw#QuT*YWR`9^N-c1j#wk@N5OGEGoq2s z-M2r`J*g_bOk{y<2El6PE0*$R>uaq_{+$?x>3JoxSoRak@v<0V{U!tU2}D(PK!>p0E?!yNodh&BgH)jcZM3|c2Jhv9nsA3n3QmHCXQ;%#=O zw?0(FM?*>dTjpFpd;;|ukIsPf`tXyS_TayTFPvGfcU9H@RFHRoowV{wUDe%z{i*dF ztqaYe1C&T_-sCq0UnHA+qugsXNw1E(%uE$AB~gR*WI@%;JNqBv$T!d%%+GAOXA%1x z^ekUJ>CB)%5{=J)JaNpmx7)g9T(zas(tZ8i`We$QP zK-evoRaxY!7FES`%Kpsk>S%9-d$l@&zfJT%d4w39&r-vkfvjRnAdUn2QowzfAipu> zvml$j?2rSF%jHY;%ZdLUA0mir?s?Xl>H?8sYnItJh^<_7#>95##FrcYjNLrwF?xB? ze6UiTqUE~XyydsvBmRQ5%&OR1paq@j&i*#<##HOEdaxtXTcK9pSf7Rzaw*{K3;fQ6 zzVA6fo(0LqdLx$TU8UOIKQR>|8^=cW9DXm86{1&|tU7<==vG$y+UD-Lz3=0D6X|}J zRSoL+cYiuX=tj9GO{|B@y;_vJV_lTIru`f0zSZ7AC`MKsO2AvXCDHwQP*e{i>Z%D8 zh8Ox(9c=&`)Z#0;<>R?$>`frst)_d$dkB94O0n}8{Rc9cvD);0bH_7^J6ccYtIel$ zR_6BGik$`4rL9}zCyTF|T6fpy)AjURZ%T%E57;O}C$zpELb8HtB$hsTg?{u_x5n<&V6)SoY@d2prnd&41HVuWpPR*N);X<1 z#khzpq>pgq)dDEaU_-ZHWVV(2T*)%Nzl8nISk{~jUm+iZ%e0L z6W=iU1JO5Z?_#q$Uv13AcCzN?{JOYnkPVB89ctnSt4+G5ew#$-v-?~CpIH5elOK0x zg1GvTje`Ydrj|2X@nLlL1?stY&+zdVSvCC_#7^2~>In73P@8@os0-nl)s4JB5_d@~ z^~m;FhZom-b$EdP7wVJGl2_X1`U7PkiJ0E;pNwv_{i$RXt!D>%7w|`5Rc<+pp_raG zpwkfA(`kezcW#ur>`&XI@6R+GnnZeX`^PPebb8tx}%x=Hgjel zKO@MR(>X1MP3qE=y%yHj$XlYVpp+huFmEFv6I;#bRE(MH)?$a&bVi}M)Nv^~LVr{7 zUn4E5N0-P}tIx@FzBSefDv1x)B=DX@ddKi_(&!i37S@{KL1>WundnmQ>?pnu;7`JJ zcnrex`6|_2QIR0+On~8B6HC$E{+n z^nHbUaz!k~tt&zGEMh{F+H1kz2=g{O*nJT9o^UaU@erG99=6k*P(x}TKIvZda`7!iI=)v|=v{D?uhpL%ZsH2xX z^sp@-Wvy#Ke13ivS1j?~;-^Qb-xTDD+?7{yhzCQfk#Q&ilQ|N_eaDUm!THQMX ze>uNW4Zc>t_SS)FltvyE%+{aItd;L}w$rF_kiEY0Y43dQ9Ae9^mNncRds)#U5P|Pt zf0I1>(fx{SEX;<@a@vM=w7=7=?1E?3?;#QW)HVH$pjSoFickmIKwS6ePF%lL^y!j# zTIx%Stol#)g8k3f>u*1;I>oM~chJO{Lg%IL-WNGzqWIXzG4v1y8dK6@1h*3Lj-W(73 zgZ_Jn)YsH?yaj9w$m{ou`i1F5Q?e*~$;`+&{6aZ=>fk*+h+uz%SvgPMOpGCV`#b(E zY$X3gf4Z<&D2aDf-MivvlZ7Nwx4LQ}&I#g7Z*QXY0@whD;nyg$%;r~ly>)U6sHd0k z9$&+`V)!sMYk(f0*GuhNse5afMAnV3^)Y+U!u~DvjQB2@y%A>aGqV=uJKlzV8T5bd zPg8WcSs#LScfJR#!|#O?FfyliO5QK<^OKsdXx|rc=0@AHb5cCf_{?(r9BoI|9uDeV zPqy-^X&kYiws#fQnUytq^9H{bP1?YJ7Z6)6biCRpW$&ZZn_G8*0WchTss9{wH6&Kc zX@O=(4STCW%#$qz}6O-$!nJg&(iE>z(O`P_|iElmnr8B=qcrVae;`vW3$+Elm zO}vqMRtieu>%*t`j_iL1ogiZhIfb%gZ@pOho5xf5-`IPEzJ!DB$2#$+Z|dL8)qG+; zvEX|KnlUt8LBC=3qd$H;{7)Ro)M6!Hhpan^=PLRTzE16(YNj_30e$I~M61B2{9;1~ zo5Nw~3|J14#hp*%-)7viYt~WL|Dls(-VJeXb}mw#$~Je;m+waF zL1t$#9OfgeCEH?bpiiC5SAFX*#GNITeDGVm4_{ixXDhk>94Ct{#{PW8{68%jfG)u1 z#eapbgAPQqI{yv(oA?_m{;6!|lTR->z78~R!M;^=H_Icawb&pE+Iin{h$V%XjyS(g z&V9&7$Ly=uGISK3o#-90eah}4v`Z;=$u^mVgq74ood=2It-h4Bt^{=;k^1#AJG=O( z5#_x#m&?dEJ9iL01r6l)1bqvKGPsMC{oaQ4=4mjSt?)fy2+UEFm1@349z*P{O7HJv zcjvXcG1NK^TSL@3UKPLX`Q0Oj1ZufP%_@`4Rj0jV#l_Lzx|vzK9fy6f&9`0+tMu&{ zdOyOOE6)e`6ykc1CaCJpK+?G011OKL2QT$|bvCs%7kkv8v=~!n)_;4Yy*J;XKS6DG zuQ{JHlX(-fs)K%|uja0bZ(N|?V`i+T^(W{SGuu+^^Tb?+y}9L&w=sJW`gPD8j3nzo)*H^LS2uaxw0}S>8_6=TwN*`j#CNxMki4N< zCaR(T&R&yKQG7W#Deet=GfNJm?d{fEVFawU7ZA@qW9tX}QS$_Yd&>ZkjF85i^4e*)OzcK!AeNPG7 z)b)XUciB4zKf)n3`c{r-*(~hrTYjpECx0FNf|}4X%A11rgYmE}Wms~j<8dP?j(B)VY|JobjIjQ zPq|%pwzNAESkJpa){86;ejL8D+BU?W)w5e{oYL2=`V-fghjQIu?;zBa-);1mTzfnJ z0e{F`?lhCF&`~+fwtN1y{O6n1q4nJdHM^&N84K!h5IC_8hba{os`czZ+{A8XduQEzLRJR zwV6sb4W>hVHO>_74KY7S^kIiu#dH23-7$%Mlf*QGUN-fcEXI%Ip}ygV<@lvo%HgZ? zIhnuD@kQdh|MHxR-;~?CEcYq)&RbuHnQRr7du!(>i1&2NeW#bHpu426p<-y(NFC+W zhy7XfKkCVBbPf4FctvlH^=du*$9fGqEU#GUM>f{){N6IO3IwawM1Hc1e>j~!a!cXv z+~eo4{hdYJ0lpvO|AlGpL)s?pgt)#dq_+5;`nW}mht1_6>+_$ehrD(giD$l)|u6Z6t0 zrQZeWI^X$`d}L=oS2J&g`@4XwzcZQT(3@;HJW6J!#oSU3`>kj4kxEY<;@?0<@q{+_ zHhQBbsZBY21@R4FXB7L>^?Z-mCfnO+Z*z!u3%vlhAxCj}piRYAMsL!!kh3_piMs*Y z&8_RD)d%ZlbQW80hJ$bx`kSL2_Wr~#a&D`8yHmcG#5^fxmegi3c_y~A$SWs*D}3`e z!(+L9YBnm#=c)A+y2)GVAuMs`n-uDTZw1|%b^B(7tL?R1fy5%W-pY+WHH=j@oM_pP$jMN|`x1ysl^#(|AYJ z;s~8QmDEO#4|lH> z^{GwXy0W?@@vfjl$j6$i9OfcEzrU#YN%9}z9(*s}p>%et`E@;i%}@3!Vi#K#^nx0; zLtDl5_gV%uk@3Pq?f;{JCNLL z%ddu5hk0*Sk?(hQt#|C2y&>wgpI*v9Jx6cDHo5#vw?%4wsOB!RH%Fa@M3_r))go&Q zoz<#ru)ELZDC-N>KZ!H9c=lRnH7^&?m*RXP*KF2#?2XCkyDhJu-|10mM)pF@ zz7|^vu}v0lCixZ@V|_V&f}bmfpXjgQ?{D;uUhI)u88llB`La_UO{gYE*q@hOkF7Ju z@s^mm;p({E`dj!B?!p&zOYm_*Us}pHTTL?wmEfXYv_*SB0=*pJ{51S3*a^oWERi>w z{psxgZGAxe)AZsC_o7HG-vW2IMP`31n2GIl_lkF|_!rsRhu@1Hfk|RmNB&ff3)Sj9 z*+I5~&CgM}B;>zHZS%xNL$m`7hLhqx1?!#dtPWp@r6qqq*~_V(+wmXpXY?se9r>^| zoz4vLj?3vyAlu;VJbq8JGs8MbVt{eSdub3DD&-&E@QF5Enr&+%+* zuvZ|5S*S~&ydl~a^6K>_XU}H!4$zs&=Vt3~>_zM88$L3na8BIk&2FN4e1?jJ&$#j) zsoq`i9mo^OKfC!ElF;4Ir}<=u)aMX-2BzmR5AymOzmc!MiYY!Y&aqpTpWAAcR_B`_Khb1u_#Lk{oAmjcBIYWOUh_LyzB}wU7XNAV8U(QWie9Xy?jqmY#a)5z)aIhA z-j*fr%f~D>W|_~LVd~6&F6SoX^v&mUwA!rV?+9Oy=#OD%EF@{>+sgJ2?1l3GwOMLR zR$5JW@bgKy*sLeAvjy$T#%Xk|nmj_^inl61JJmU&IqJ*;G1bSnf!?r3Z>O23Mfgpy zi*7CRdJ%sI;;UKnq`q-_mtP+z@ii!sdnT?bE!@Sl@-;>4CpWo5IR({XH1bg|=iRhOp z^cH`NowxeGK;6%ohs9(U)uTb6ITd%^*xo*}XL?^Uo0(U`UFwmiwce)ow=voTI>YMt z?r9qL2EUum9yqTKWAtgZ9!?Yg1AG5MZ2e4B$#*)Pcip+X_#6ej(|T2q-*(pJ(wl8G zgw49@|1L^?@+>XhGvrCtGqYa4!|%@Q4p$a4%;D=FwKy-Ip4P*lJX>?oRj?iA#MDR( zJJ`4=hPlqXA&iMg_ zU$cG)>Darch6$^>zi{0Cy|~^1ao?9uJ$0QRr+xgrr&m#p_L+^-&b3zG>DIsCGer2E zK^`km?W*}b2ZhurJNn38W<5K=<_-5^HGZ!A*6CMt7X4kMjk&|H6&#BVw4yYMFzXY*&e=lSVJ=Y!f^G+#%YYb?J*&h)P%W_NC%+;^jEqS!G% zhsdXk`6%7==Ht0_1$lglPoL4;)s=(2BIs+jZUw3tz8(CiX2Ie#aH%xq1|>#bJHc;cLCe=)3weQ*kXfP1izjg|@AeQ`%cduQ1xh8}09 z0XkGJUC=gW`z)RL_~~r_E54is-DUN@M)o0G9r*j+xws9)Ebq{mcS}A0mqI=GIp_R; ze08lFBh13yFTF-RuAe-;mnJ)hT-}A5K6^RyrB1ybyQFNV~pcXTz zWU%hb7qfWxgTM?q-I*Yi5q##^fVh<7?SEotuTF7*X`_{VlynUzxg8c z6Mw!4zv}Xn*aq4^W{%I0$8PMsg(*>LhkmM;BiXHr4`%Njz7&76$%*}t+=kOoo+AoFgNfWl90c?>+f3YuiA0p-QL`+ZNQ+E5))$)MD`Pqc?Y|PM` zd-O}&&(PFcEZz-jUI<@Wd=1e~upa*vI!u2riQyM#E;xUf?0&F0N-2N*MY2QsFoKPr z>Gch9Kh&aXGw0Ofm>9mmhmbe+maf46OupEBoWak_=li0TZ}D^FUNnc_8wK^6-6Q0Q z`ROgUFPeKl+`(z+YIHMtLEZniizL)5}Jgs2lxf#AJF#ZwM=m{;CtRsZ=%sUzFWP~vgA$D`hB5(b4#&l_D+yb zGq?HV7Szh$k?=FS`O)>>i45lSvKoCMpCkIQRqv+STLN3)2%Li(u+ZGkpdYKX?+#>y zyigQQn4RCu>gV{XZ2im5H>uT~&Glrv;9L0GTt`NE8}Ucg^0HVK=kYg7CHKc1%w&HS zdf$w+lY7PVzWMU|O}z^1=PY_F;EI^9I6FC~dn@niX5u~hJb7zXz(TWnSx^6ycbm#; zuWrN9i~93!|M9U@EdA}*Q^Vx( zy_06Sk39C172rEELY=G=y9<@rt`lVT#F(SH-@$?MKx;u0_!x&j`BtX)UhI|hV4?Ub z#*?QS9H(;wy^p?xoA`U`k*0?44*oV&z%O#(f3u18eGGR`H=pS2Uouz8}D8`-`SJK-Gv&HajOr6M1 zMFsr-|Fllt}KGH)*3Dty<+PfaezX5MGI$>jGXegs?=cbL46 zsYxl=@9w5nm&f+r!QtG#?eYp0_Y3yQ`<6GO^B4Y0{Ky3EfF3V#hqhQJkmI-LS82RO z=J1AC|E9aseP~Vhfw}sOt$F(Rk~~X7zYW+Q-a?G96b3qb7(E9k)%KT2XT)?a(EI65 z)|TIJ=LXSh(9&IoFJNsc-(2x76-!q8H)HBZp3}K!=>KBLUyj$zNjf@<#r=;MUO|sm z-c{J7UbmcEir)%V%;{0*Z{mN4*DxZ&@4#U1k6hEUvkbn`@7vCGY0VE5(zomU%$Mip z^m-NLo~nBtu`l6wDWs9pGJ3E1-$JK+sNePKQ{H@>)yn|6Y;(8s7vu{aTggn5cjK#x znjEseV6HmIt1i3sU>UpX)Zu~JRH$s8)az5d&!!$x=D$9jSk4|d2eb73h8R!Li>){9 zQ@g{?Mbn92#rKilP=5av-=O&BLLI($u9X-+h~szjID*~;2-n{$^6S9HUhD5*YFxj4 z#L|)MS3W;cw>R?rkG&mgSD>mI%R3rfsN2`>PCc^Hdh$>lBZ9pTalHNHYsh!PU2!!M>k0Sdd;ZFaC!w=d=-px~A9~W> zCcejsak-v<;Cnp2SCqF0zpJKrc39rS5} z*c#H?B8HCEpFvf3D;B#S>D5z*wekH;T*o`Af3eAH;g@IkZqZAdSx?0^lWYZ9g!K-5 zIp<3AcLjf2ZC0qo1-?&_T{VxN>sg+eZ;&0IC~j`-uzo0!;rr;|cF8_2uJVXL||qhC}Gf01P*s}ba$(l5&9 zQo2LrbW*Pei)E%b21I+8#JvFRES70{aRh$`f)jYB^!7)2yt5u^7S71?2f0*N?_vf0 zZfW3KZ+}Tr|9uqy6=YERGjd*M&i+hBht2-(S$*dhqtiWHJ>}HeeyKontrl7Mt*%zd8_UPtU#@n$68W7X_UsAORPL+I zUqZ1i75@Ty8(}w_dte8j*_?Yrc8mTVbC|7(yDQJq=x(+y>&u*sYNJkx*tv^e#QuG` zm9TffUB8{&+e?2`g!>?_*iH4H?I<*nT#ksnvb^{5*_dpbH|Aau?+JTz)FFX0XW36| z9he$V=NeR_S6uHF^OwY4x@JBj5Q^`lN6EzZne{$)JGq0uuvtrOw~{4GqZe_#-SP{g zzlUy8G|cyWv^XECT?PDOdfUunAN+8b4vS%wnc9VhJF}ntd-!u=oFtFAVm__U>09_l z>ES108N+5v>*VZ>K&Qp*!BufxgCO_qnqDzOWj6o>J^+3H8gtXPADh zF$=${(R1_jPGA0YCKVfpoXgbOTMXyrn3^n5Z13r`Dem{bGc)ZihYj!*TXzfj=F_`u z{RsA&oqsEO1Le7vpZm_Vb@pGeZq48g<)eKib#!hBy3*a7jqVh0BD1**UFY0dv-dnm zuU&ybYLk|af|b2Ba!JZ>TYN7V0rxVRYqcyX$DP*m^d-A9gZTT!`WrsK6;C-iHP1rE z_XBYqAwNc*jGqAYJEKxddb!IJ39Xh)ki)Cvl}AL<76+Y`0XOM2(iVs zPKO>$Yc9l7%pI(&cbUa-R?PKd>yP z$fu%Qm!e@bi#?a~DeV0w=i2NXK_cqflrM{m+PVuL$p~i47*f-Ys z^YQex<$R zY!uat8P=Kj?=02|&VGUqVY?68FYU#xrOsqgXffC$zE)@!vc6~wx>@lR)nudfeK|fA z#~J)JIIl(<6y%6=E7a_P{dBeE3B}-{^(ygn!dK*fsJoqwy_W2jR*z8$-A8vjGP~bb zeC#CuAhtwx{2oZGr{T`qi<4Vz$!eR2I=S3U`3#KfJ3`h)9m=p(F(ca%YRmpqdQq^# zETs(geXHvp$2UXh1u@qrE1{1ytiM;Q2YhzLchK)gV(y>Goo8n=*@k@HUa{?S=81KV z6l#&j9ZMoleBOF`X9k)=N9PBLYjs|CLr+hzJ=K0$d%vsOH~4-0-bKfX^`!m(?5&}n zslFPh<757xs8eTkStsV<e~`=G&JeRI-js4^ zrJjugz47Au%6){N5R}bDhpv^6Rix43|6_@A2^(xMy^qh~*D?J{3=Hwri8O zfK%c=4I9;_H6Q!g{6f8-%c(;g?=bs$=;zem2lQXUadDo~$0*>*%~Ozt!zuz<(WTcEnLtUe#i96I(uK^22iHva($Oze!&@Hv%?So15|Y z0doA@dO!Ymd5qM@$N1MUM-BIQPf|4ILk@GTOW-f4Z&!2mxgPYe9s~tr?8|o(z6*QL zi>O-+@7!p)>=Dx+@-Ia1nt0BrPs1i=PtH}{&ms6p@U)P*5@%t0mHGL_ymS-OH1rQP zC&cu#srNxW%agCumy7iNv{#P236v`+E_&;nnP}Zc4~nq24PVmCoI`KOdka5B$tTA1 z&2MJ@VTbyaLaVUZ1ieXbE4}h;)>Hqp;@(EyEu}iC?GN;JI#Wpxw$Y0#pLD+E>%}!r zzR~#Yg+@N(dtr2y^F`>thSZ`m3-IZiLC_Lm-x;3+&%OH`@gIC zZu}{DhVP=Uv0CYy`8@4R0em-p>Y;7mqB9%hmn=koGJ3PwU!fK|LZa(K`Be-Q4b^TT(1Pr;}THZ{Z8)kMMo4mR3OA1m0$~|IhlrigK|2 zf&V}F$XCg4J9fL!tBdadKiVtB?l^o8{*OoMAO0uuEponre+q$$lBBZx25qx=AJI>p zi)~hW<1f)KZ9NZvhL1n_`^w&F>mS|UU(s`FdoPY}N;1E5?7c99xvIzues;bQ+5y^S z@!y@r(hDw=pJne`{d>V?Z?awPZbGsz@PDy!SUw+}>y(-g^-Pyp4MOxy?Lzq(XurN$ ztZwc`RW#uc%I$MrT_cZ99# zGXO0k@2mC};47;A0rVtP!oOrAxmo;Z{SSVdGwH>b%dAwWZg%zOxp}?j{44uq+4@DD zl8b+_{Y|hJzK5TohJOEpwl+WI=tk5KAB^CmB3d8X!cS_RRqU_DU7p^h400kb9LWw^ zS!@l=(sh2b(TktJT*vZvhx_|ygu9{!?Q7}*Ot6lvwlndoVKEq4h%%A++J&Tc>SUwbpq#bWwPFZba$+B=JO&Fc36`X8MtVyosZZ&#}dVs9wE z;7@i)Z%hHm9}<~P0>aH-zLoVD&7s)lslS;seT4dvO_xg2%eO}OdoX?=We zzl-$%sG7q!5Ur`cAINv&tC8&#>s9=*a5YrZ_c6NSOPobasmzExDv9f(Gu7!guIq1) zAZO{^_(Xj6GP02Z-&d`U+CP`u{Ub|^zQVj@BcQ0?E%n?BY*+J94S!3VH4}N)!u&T0 zGctlK7TXKZjc^3|$*a9sp5rIU{kAh3^ZIR8U$3DoyiDdhiw=*mfu_>qK>jw-`!2OR z4?)(y*!vsqurXErzfa+}1^!7=Jr{c{I&1Z~YqUEB{rO&qrewFeeDBfy1&X?N=jGN- z9y{{si5h&xXI(aK=ySDDv)e!(a;a)>x4X7npGK1X>U@`sdcyWD>uUB-o3-lbkLtTw z%r)4mW^Q7OC!_EA1-{Co-T7;YMu~9{`WssV=}wH`pS|zNeq!SW`i*&+CdYx~HN@U5 z(r>*+cz(*EaoAoVru%FJ<7<-Ff?4+dDa4QSXYn_ns-C4c>v0;J136w4&*cPulLdKa z!oWji3#m){u|mmk6sdgPqOvq zw+4O|ei?i(mmc;Pt4logUdZ_gd2KQ8%Htc@)VBox8~gKfdaK2-2bz-qoL?`4?O#9K+iJae!uVGdmvi}Rd0r~f0 z9!@V|3-7V>8TEd*d>h((EsqfAM%rs+-59pf`NsM@B)0dcl(z`Kojkmy*?}TZ8E(3> z9ne1TXA<**UKQUKbPv3VG!yESL``byg>ea}ZGAr@e|q;b*<<#5wxXj}Bhe?;O~i3I zo%_xHd9quuOdkKCHO)r?hAQY~&v5;)uF_g^xsJjAE4LLfa<-!Qd&=$!{;!aQ zisucwS9~Fr%pCO9qh01|r1MGSU5{*!I)&nIx|5Ij`zM*&iF2F1!|-j8yXf3Pv+=?D zuLyUV?lm$sv&DM8{JXPP zstUchy}x+usKloV$!Zfbnd8M31C32jn?x(9I@qKBMvNpqN}I&?}z6TN6*e z`G3yOv@mw;?-$!HHE1iA1U3EJ5wv1&qBCjOsS(%rM9tgT&#pGr`!82O^`R+y1nx~(Cl*iga66)lIY1MvH|Sg=Ib1OulTEq^FT)P zrk%CePHG1JbmoS-y+EtfHuKITlyfz@f02DHkDKrfTTkV^S8bcuaYwDEI(s0Y-}>}FqqCNL8d*Di>|`zUfa%tMl7FTSaqEf=7KkG+ zS`-G$tmom<+Du@NLNH zT~)sWa#+LmM!Ki)d)fax!nfD@ALoLjo%J5KF*|wj#i1(1k>Bn7em^AkE?Lix)LXuy z`3;p%Py8(LJmce6`y0@Rk{5;=CoXW}a-zctmm0XxanAOjy6 zVtYGMnp-h`$!1_9fA`}1;MXeOMml56LQV4XQTj!;HpuTz`xD3}v$-Dq8fvlqmY<_B z_t5&a*#3o-js5!uKXt^jR6KL#cR>#A$$P?J$dK9JJ$kSOA78J}qBo!~KS|})51pA@ zKlppWR+%QgDbOBz!2xqWkKLSN-|wz8l;eMVAF?-}{3iMoUPFJj>WXCm`Ze7YMg1Pa z4}>=I>#HC8%z*L+6vEGG>hBYNx1)pTToF$nbs4HZ-7=b2-|z$MWI*%4r%)Z5!2-Ug zn~%ZH?Pqfcx}E&4zMn&HKs_JNGx5Bgj4R zP0t8LVTPVGMBDJ6QoS0f?{w>x@T-30O5t}YeoBNnXCt<6_i8a^Mi;6dluY8Rb7j@7uk~pB7>M~l zRGFdH)!pG#Vk~Z6>#5;bd)Lr?>X*2Z_Z$wIm;7ic2y}nW%b_FwOIX5JYUi&zGtSv; z;vJ8!6Z3vK{7&aeI(NLXJ3w9ttpp8XXhwGv{ir{`hTVKZJjPPITrs{3P_6 zJk#*Cg#XFbz4bEYhtpN|XUe5x6*C4+)gmpuiTK>?J!EGszDH(#i0eM#56S73`u~L= zsjq?Rk%#VUvIAzNLRCG^st)=58#EsY5~v;7Om+NTZBCQ5_r_nsFI9s(_MSlmzj3SS zt@sX_^HSagF}2zoX=&BTCmk3m-}ijj&p11_u}(gHan_EM}3=8SC7PUf&cqr`_a0V zoP$&QUf|DYN#c5bN1KhxY@X87>ekcb`x)9DddJi|q2IFnH4M`;vL$?8ms=~crTCfZ z)>15wGJ1#Vn<2g*aomL(X z*{Bw6MuNP@e6`}^Mh8gV#JO+sco*oz_hCr}QIH3&o2T;V zKIdv@av$)u`C7+Tq#C8;e+wHK<&h)Y?=JUbm6)&7>tKeLm3JS-_l&P#H80dqubi!m zwh>=9G%k5OIGE3#bvrqx)VHkiziFLJ-OA%XSBK^7J?Ae+BlT9Zcx1KkE67(utirxy z*74b!jc$NB$$ihz3g`{C{?)4uVz@7lRE_;MmwO&`BUyFy6ZP$YzD~v#S~;V>@_E;t zi!5qR(FFQBOg-M(Z!3>%;wf0@2pyUYDPPWaj3mRY;0mby?U(+P$RJ(jN@*R z{Y>^dofE0Wm(Kl^YsRMT5B`yU*2$(E-Mj3Z$KQlD&a{)?8g}cD_YC$s zrn$d|APF05y%GKJb)&p{)=}!S)jA9xZSH=+KbAu;=Z3`h`^a3^QMbqz-a~j|Z!O!c z@vF_vXtX_lL*(#)e(E&dFg099XMcM0fqtYDTMu%wxr^RSej2rOe_>b(?__pV^cQ42suvgalvL3WUf43j3F73@=Iy3u?*^O-uQt`8q?j|^_ zuG`p*-TMD3xCgLHQ{(g=dIf&%I@2@wfF>F!3l zyHgO5_t|sa_srb6bEoh9{g>l;=0vO=#F@IReRH!=Olj3*r#=m~pN70igxKhHg4AOB zK}@0fd`Io4wZELre=CvUSK!m3g)6I*S&HwS$*#89)%C}?Tgs{~g)3 zjA~ZZOv`nHKCBYMa%UsO)h~fwX7Zg=I>=qq{~#iNw*f^z5` z6s_*PieWoQjqtVXcUJqiVp(K{XQ!eor%HU=j`y>BI)?qr7M01;I6El`|8(oA{Vk|m zLqFgty$j;q!hh++=9<5z5#9`SSYdwy1e>2|>+6C3#!X`m)h-F25wYf44y(*gW&95E z_UK6+fUv9A@8e4bMz&umhUyh!pE>9f$u3=MxH<2p zPw96?Hqz8}@5H!D{7W3N*j?|K#@){yyZ>yYOV{ zsF&I6sWtrNu3e(P&0A4U&pOF_FnLUId(Q3^|8nai>h@Z1Hj(A9_t0!^#%B`q*c581 zpO?k)sF2?U^}Q+bPo$@F@QdM4D!*scuZsC>!rz}}`UQD#19uAQ$z^;}xwF~VOa`k* z)$HyXnN>a0ewQM?VGa2=caOy}UX6m;sz_EBy4r7N z7Sij{*8KiPAo~J-fPdN9YZgYU&olWn7H@rinwilAjnp5aV4r$i5mQK7zfG*K;xrX^$8NFuzrv6+JOAG76>Uv#mda#)-?wmQeDA#bZ+W4O29TV$;n#5Pn z>~ylL;e7Fhvk_?>;~uqW?w-PcG`_j$$8_3==Mg$wo>BBJkVm8cp^3ts!)NwA&kHT| zuOYfq9z)Q@>M#ebC)UJtdx>Kp+c}HNfqX&+zgNXHOg)att+)MxE&a_5&+O%3dwoeZ z^l=v1QpicR1^t6O7dy=(P_ni9v{)|N^{+%LwTD~$?h?yDe0g(FO}^jZU+|es%^C%Y zTZ|9MBJ?hOYu`CF+rdWevS!tKH@m;7|77+1hE4|k*e{n2W?(11mu&q559nrac3yI` zC7-A^dZJJ5`KX59Esjlk@gaMYtP|)%CpOcW$LIJw{Ou$EmX8(WZ}5AZIjZg%E3@N{ zZ8Ilj@ww&wi+8vaJ~F$y)5&I@^N@c_cGCCjj~MyL|AO^HXsI`w=)MwbA9b!;MGV1m z$mmT&H>l@SvKwOFEY6H_d86)W?M>8+oA#~+`KF8iH5)0asvUoO#aI@<4_z$ROwQ#a z|0rTTvc zgR7Z2dCWxXh;aovHk};MjpDzIUen`#?1WU8m-GAe{SyDhtoxJKMhlP+z}L|G0P#&A zyR7bkA?5@B7wm;y`g;W(NVi1-HIUne7&hhlR$QaW>egU4+S|*2Kl}*zC6PB2eX1Tk z^=2b}H-zLcljxMh?qRIo+whljN9Zk+e+~XOS$7~0uueqZ^APW-IDa5(n%j4#sNX5{ zmhn;Exp*~w`{j@k9i+xx#WyI8S&WjuK7_K9G~64P-OM}tBfSISUzy8{sPkFtZq5`` zlVR)~57U1!l&6fnU9zYZ54uhSaYrYoZ-_cpd&sO@otz(nBJ7`+5Y$Z=t$6UcS`-k}2j@GfR zuGhEu8;(ydx6S(StGo(|C z&a5Qs0{zK5sL4@#U*S{mQHbs1_>Otyh3}_FIrvObkN=eZ-l6l(Y{XjcD9AqAIe{6) zkHe?3KRJoE$#I+Yp8VcE{XNDTJkRpB$`7ZAA7PX^y(%#5MeygGNoIQUP&QcQE#;iTnIkr^Ew~P z6R8gy^VII9b1&8HTCl%EonK9_7@IBCKR^!QbXTZL!e;sm{}u8r660>M6|+9?%q=!9 zpv9eArVrD|=ff^}50gt6S+U~mlih=7Fq++M>e|-cBk`7Nt{-9#6K{IB<4#Q^n+N}c z@#?w(y~xiJ`pM*W37^)xk=A#-Ra$Q~x<7$;G(w&5drIQztl+DDPVWz!C;8~EzsKxf zhj;YQeN&QTe`Dw#ra<7MA)j%&)mhF@^8trA6@R8~N*3{%ChS}wQnCHc>) zrlrv0bRw$y#z1=URYRM?*6iLXbab$qkVi%PZN%3@_?diHLEi`d8YkpWjIZ^e3BG|D zNm@-^Atzk&E-@$Y;(eOy>w-GcaKXQiI+&Zx&^fol3#EXiw_1;_*C z*{_1;Q-_uAXc_kUTZhKDBT@QQJ z#B-m{GdiQ_dRr~PLo}O`07_m+@%xvKGxL3q}J$7`7}Up$zh*&YP%l9 z6X$a_uhLy_?!PZ3S)WzS)W^vnl-RTzq=13SURA`)Mo&iMvm38`@0W+SX|4*@`lgL)^E{& zoL#ObkN8Q?_ha<3SUyyjclK-X*{hyf*jq^VBk~l9^~8NU&PNjVBl!GXd|lN!etq-k zor>ZoCV{zTqpX^=CU0mx4m}ud#?Xpv?JuCV_~TH{e3XYBY^PAWfl#o4`Bbw?bf2X5 zJJvd#Tt}f(;465~$11d;`*A`{rTMN*|329YXQnCVN%HGrT_et74M8p$C@6_-th^%9okGX#E=-AB(dZy_CM+-PG<2eNR9y0A`3Q3;jTRPzv`hyDMNlkM1(~ z6Q<}>B;N<zs0Ujg;+w)oHSUX6DDJlsIuC;Bo>rsx;ush$9wK$>2s-zjgF@Xd2%&KC-FTT{L4dar^!shKq29oepAcDy~?$f1vL9ea9<9OZwx) zHWYu*OguIQXX$?^-)ibI!I|WIwTWO~3WLzT#Wv3E;+LHZY{HfO9;iKU_ZpU_QN+1;?Nss}~z=!@y#v}>`gJdf3uxeozwBZQM`xP4pOUy?&@B8cgcoTckkh*m=fB{Yd-Qh zb3+Y|lIO<{BMW2WKqC3o@J@-XC^}xwSJbc>zDtOEN~fvVCtF|fwtS7=$lTJxrWa` zf1qA{s)iZyk#cWX)4L*{zUTyT&X(Vg`j^T60d?F%e%;w#WDV4Gg1Jgu%S^%7Vl9K# zg!Yh0En>wvmcLKni2D9%R_@U|VwNM=JWuZygz)(u{WwBh+00T~uFjM}59L=MvVH{fGOTl9+e2jCsl8Tt~C;p>?=?db2lI4^t*qQ?jLUPpyZC z3wlefzp*}nW_9iXdeeH6JX6+FPj$(pPo?pP^esvsCZN^iaX@bxifyL7h44jkamSc< z{3K@{p{vwsGQJVL;byLQegBS;LfmK@NTjwS$mimhLN_(MEw7v6dzP9lHeZrYEWFzKi0nR$NYGyW~~YjJ9DTv)GI2>u`G0;f8#dqC45%j3yP| z1@u1TU?;#gytaPbHzNz3D{q~LzX<2L;J>st37ro$_4cS(x426WOZtY9Jwn^^`!O8X zqcrBPKnpX%Pb2gOB$iGt8c=Alh-Yl~J z!76ooBA$czsq&sp#-43gB0q47l=c#xO*gq(zllE$;)0;|u zXX&Sg;U|W@BJ!Cawj$N&TbEOt z>yi3FHiT@Vv$N5!;8*w)cC&k%ttMZZjeiLNvFyPMHhQaNEZt-7*}w9fqAx|oI@*3QbRPO6tc9KC ze786snCW_QiCXn7ur5KTqct zlyEjlE%y()>BlJfE@r=j^$`AcXZ9Azdo+3L-2N^kFX>Eox<}5s7V;eC?)^qQi?V%>w?KiPL z!RBl^7f`qH)}O;7SOMkuJ%nC?$56sGotVOVTHiZ@mOy*5Rh!SY_+BuKzw*x2Asa4^ zq3Tu_f5P5%sAfLq@Rv~yX38}fU#fvS!T%OETjDD?Kg}#m#n-m~ub!n)j}OV$sL46< z{A$#T?|N*{r4!oFZ##5UN--yrj~S{*))zcKqK)ohe?Ext0@|43YaTK_D@M(%lgW1bdOo>!*@mJx>|2>KQn}hxu`o<(072}y>u$sFC?dpx%kJ|m+P;o={E6> zSf3I{X?#s6pk7Nty_xd78R_pNJ^zZlFF#f3HYn~pVBHYCMX!(ge@`zHTDYk<*IevX zmt*ECg6vE8_qbR(@K-2}dnMQ5Iovb+LHY&7+EH!Cv2nnglG3@zX5K64L|({#Mf_H| zHR7|H7^*`mdS8$qN53%(=iJ4Ie9e|~VR_Dz?=48(-0yU6MaD3G#E{jT*Kj@-9wzjC zl;3rJYFcNuKPjCzv4H&m_eR|ts$WiXGl$*GYEsbmyo&y>gB#A>R)dB7ZWG@QwmXv* z7V|H9SVSEC%DN|F?5h^F_?XL1DzP?Vx2?Mr>-*jo`oahF$BOw+^h}6vt^G4%*+=&* z`kYQ}HipQhmRN?V^L~4;OVT47+{#>uwVjzwD*gqj{3aGtR!zVfNjfs&f0>`eXhFo8`6cN+$L%kzHlGju`4fF21I+|5~j7ilH6-l=5z@ zZX?*8>^r|F4d{72`fIG)*`FfzG~&5qJ;wfD@~v1 zQ((T>(-!sy=~n~i4v{rPAEEK9$Q2qF_f0{2vU4la`$o6Fbua58`m+e%$eCZW`1f6R z?WVq+iaUpYD~`tYl2rA!p_$_-dkgrjj&F)zrzXwNruO$m`|gm>hGuGBC)itzf5*-_ zefxmF=5!y&w@-f{nx&fh!A|w7j5dZ=&=~^N{I7&&4*vz8z0KD^Z{}(D^glM=!!7ZS zCQDr1-|R5P9oQMbw|x4SMaf$5nU8F;nZA_+B^zKbRPD}-YqNC;y(;GH5a*s)$E#tc z6X`elR)1TP2a849;=RIGFdtuvX{7yy$@MFe4nOJGY@&Ak=>2Uk#F^}J+pG7t`F=;9 zH`cclszO8PB8LW{?m9oytmi{3w$`D$U{fx!!DIXsIV7#gpICFFrJ=jr2FfR1Dc?oD zY6a<4j5}s;p%@!y<{y6qEB zx7t4hf9t^=w3YqBY50~y5wT^hZASI^he##F4`!86ZGDqbIzpYwc}?cGkt&YmfF5?-2eE zI@4(z89wneF(K(9yYvXnZwpZnjhTet)Wd zd*#tV`mOb=6P-3_z2v@A^lB&5*QCxD^xG>R|Ec{A;9Pb7K24&o^oNQsZGGQR=we;L zov4Zb7|OFh9vwz!H~BLBA28gyq`({)TRYNI9NIufqdapU1bwyH`7NPQMbG1>XS6S4bDv`{}OuQ@3<>_ z@JHbioR(W4TLF#DLQ&t|0%nk`q;*%e>Y%s9`G@$+^ATQHzGNS!Fn{cA!l#wbXt9lf z*VZ9T%$vQj*4gnT+3jPU$eFK-_#J|_6W>^SGvE_?zoL2dvbH;R34b5nK^gU5$nR7= zFVfUmm_XJ6{YkyYpj-636dz0RXU*U5=qBhEp_lCS<@=R&h{8xOUKDyZ%x^@!FOFZ7 z$!`+!(rolM+h2?EGxGAzeuJ)tWC7+~93!LUK>s;er_}C>Jo>TU9|G;iHg~rnNKG$s z_K_am;b$m+1L*u|?~M9pP{+mmytfYa&EKHDS(MsKq>(3wH7gzjSUgzT(|W^_Y!5%I>{(R}Of;s(LdnkqzeK z1vt?@pH)f0t~Z195VWj-(}~nSU$ur zL)&L_Cad3JW%-~pi+qK-dZd>ltw+ISvhnuolK<;IL`8c3_#WfVjJBVcofd7Z@k7wD zXo$Qs)A>XV3GJ9KIWG0;tH-NXKZEPfkAI%jV@`P9n3 z6YxKnV4Z~CkM^&!b2q~KQ^ju!$PVARw;i0D$X^Au7(g}w2E^!}n*XVO>#Qf)KY&g~ zm+Iq0cBkOK!JqWzl`(f^VG?=w2)~WwnNxlH=2g$Cexs}9JND+_&zha2V(J_28)kh? zTvYqgpUmC|)%<{>k`%73-%6~tCsk9q24^?KendRnH1n1&*?Y2y)Cd0mXcL)=I5;5 z)Ea){CUL*mIwzM8$*YppgYU#%I+r(7PMg&*op?TDa|YDZr18o%bDjIi_r8o+|H|Wz$fFydE5-Id=bq#?Gww-DEwzVvaemOF&JRT= zLl^a3jIM-!`aGY`Ilh{jsrtTiDaD@(7LY%d-*L82t526ies9?ST#UVId*}H79Gzu7 zNdKq#E{3UFYG<`3-c5VYAbuTx=g4PHfZu#<7PWq#PhFh5ApUsdBgy87YrFnUMk~?Z zq7U_g{pRpqq+#<*XVao{oC^}uHF?g(hm!4&HizO*hi_WfyP8~2*u9@a53J|WTPnsK z?1#}wFU|+{nmN-|edGR5AXEE}M?oQ2$k#?Wk7jqiv%Sd{pkGs6WbZ5M29e@-?g;)K zI~nM&hOGKDPfxy}U$Va6u6%A*tFN5vZ5>P}8=a5YT5i3?-mlsGR+jTfF})(IV(&}m zqZ{DKOQRdf8l$bCXgc>=Y+Kdlsh*F)r*rmeJ`UQysIFVp?;$=rU+u(`F4!BEO7H2c z#7D`yzHfSKh-D|ex@1f3ory3<`u-F7F0n76|8*vLT8GowD&B48?_g$iii@$4J_osX zMbwyXRq>uTPnX0{4Zny^2Io@neaDP^L+7$HJ;d~ty+GgXsq*=jJSNgTvi=ATyXURZ zZZH7GLOr#s&gORhH(MWp@0=@ZU4y){JC%{IO#0thE`QR0ral+-DL`K9%uE`x|2gx6 zc+ZJpRTFt@_1m~K8bjJw{wUmmfa8V-#gO`?Fc{M$D#w(cs`nm&mYOw;4jO2 zv3gy=U*h*U|6RoOKQrc(k_{rIAY&}Zo9dpleMydz-@9?v|!)PwD zU+^#RH@z9JKSc@-s3AI_I~?@$Cq#Q^yZoF z68^e5zb_>l=4%z%Dly+qCRe)ui20bAjuKB2e#6XfIelE}T&Y%k(fiGr)lgFp#*xi| z->ti|d*0a#{I0RSqUIsK;h9UBD|(TuPccG1awy(0gLrlfencK4JmR3L1COzBJ)bd+;J;bpN-2oltnMDkj)nPrGCj-=3 z?WUT!XnFoYHb|epWorYzOJ%jEe}#U3dg~#Z^>_9++ApFvf1;1+tySxL+1$_UemhoC zTkBr(?11l$US;D`{2RW$N7uteIsE1PPC4w?hfl;BBaR+&Oj(s({zim3r`Fe;|0bb1 zSgpTT%PQ5}Q}Whm zIyUcQ^&ZJ@vvUdfi^S&;&u;r$$gZJX+5M57hivwYP@n4N&w6)GXET`b%R|*HR|X2MlH}r@Z&Fu`Q4M>Ua<2k&A3O-D1|M+@t?e zcpK>KhXZgB4na9J%1w72z1-{{CVQRQ-BQCRWIvdz05uJ*YyKn1*}CSQeq^0ed`IZt z5Zfv~e}kjeFXXbsejavvJ97%(h0W<1_{-uQqO)DB_tfM#zDzy;27!L+-k(JKSa(&s zdj;G*HOnKem1NstH6K^eN%DCnu07^7w7z$WJd*t#?B);fn}zKIDgFJRKJ)DVE1x>n zdBxO1PJ5jjm{$!li;vH+2I{8wE%c=fJ};j`ojq-D4ciU%bQ*a^yCD><{vI(8gT%5@&gbYXQNQ(U zo^$>-1o4?W(%mTN9k%~QE`Qjs-zZ+Zwy;ATMxz_Z7NX1GFL)cL=j;{yH=+Ix;OkrY zZKT&k-qX~(QBnQnZQg?o9DgK-Muc3FtUVb*CeY+3p`98g#=KfZ;ZVhDu zh+S&2(j&FdRp*zhBsU zV(*T#!{jj1xghrMlHbs~*YtLnk%FbvK_9BN@GeE0@pR&|w?4o<&EWSCKB@U$Z~rb^ zyS;OB?X?hBLH-UWQkP)8q4&Lt@@_7|!56zj`j-c`O^($6gBrg~k7&%*F+sGQ<{ zJG3(MaF6{wk>(^l-}=#j{(Z8VY|XX*J*>frBsKaA4k@!dRQH0Iga{Q>ZeSS8V zRdYHI)u*laBE-6dYz*B8>?WkUlkH}7SGo)D<(o?lrmMkDwJGhpe%y>S6vu8o4U^{- zvbj(opSN8fKj3pLdj+k}k>6pnU$|MSk=~?6LK)^1Ds;1i#a}ltUZ^s+uc#ZE$y1*)PoYSL#tG$ZU$EWkR)(<3f7BK%WS8 zK=1HzEZFxs#@i*QLu#D7ntRJ`T4#&mi>gZ-Ha6(xZ|c5_?o+;pn$u2c`J>a)xjpE1 zc|Wr^E{7b&b4Q#@ee<$a=a0=t`uE)a5Vjjex_e^kXKy$ZGJ_4Bd0~GT+b_}I)u$N$ z&*cAEk3bW z9z>W0d|j~w>Gfg!8|ODBQ!o5(`IJ!mRJDEA=;uevLM&ug&n)yl!SA70T#bJ-H!tP) zmW{pSN8F{m`SJEAiX}^Ob5h5<3WbxXCptiUt@OS-{!mK&lzVFRDPpF6B73SA+t6c> zBZRFye&e&bkIf3|b2rlapa=KJ!r3Yzj>GIUtLq++_do|hUU7Ynz7yLjbdq>ZqxYO2 zCjWTu_5{A_%IhwiqVMsf5^f;19#TTw>vObrH2?d#H2M)g-c^xb6K;UI~9)`fbo3 zqvVoXe)z75*l?z(^E33S2)q|d4*fZA{|@vOQ@l#uOlhrK+}`shwXgn2ux6Pt+;^dY_WMg1Nz z|L62~pgFB%W_G!Ar{s}QFEjI3lAT`7^gZq@JR|?kdO9Cl&>_m%T&$1OrHpt3o0|zV z8(I)1v#~}`Pm1>xBw^zl_xCiuwL4Z!uBZ7D+so6d3Zokk)(8&Q&?D1Mbr2VE|&zYyr*YGq6zGI^)-8XbY+PJ^= z7doFzugX}*LTH>nG|XOln1>HXm!TQZooGd8GUD&z-@>`F{%y$mqS{5E*#q4J_hOxT zHL9%laWb`hAm3@$GsM$TuHE^{%+_JDOVBFH{8#ag(vL)E(yNXh*4x8|chMyO^Kf(3W}? zP3IJTc8LDrU*n7Ok-oZ^ARjzbqbTzlCZ5%5*4R68ncoriHnRD`yj4r+H@LXEs`mjk z%)wR$bDucByW@Py8fpmNW)dHIpFA+oyTRsNwg-}3VKb+5kLfS9&V|29HcO3jKv>; zzH>FrR6e>}#r8yg8|{~H2lJC%!2hnU&(V0b-2+Gs*!>jAl64n#2&m)RAhxP-G`Vk|cq@|Chd2E7KyNvh&E7oxQrMQoo5=2V z{PE2EiuD>kQx195b=RR9d)d&QW~LBd74bKc_@>}H;)|ez(c5g_P2@Xa{hM`B^l!AJ zyo=!j^0-TEEf80QH?x{Ch`@|Ok7d5qr+P|{vL{eLc&FYO(5rW?J5_!syD z^_``|yoGtRZj;kD672&MptSvO&>H-1K*yW)V}a_2e+_%G^J7NK(2J#WM-FB2t=yGz z=(ZI4i{1`&57AO|@6suxzegGXe>c!akVZf5*$Zpv{y;N5oD4PDZD1V`DopDs!tF|T z23ZR=zwgW^=H^58_p8xS@=}HQLH|*YkH{-Rjz($*M^gHIgH|F7q(9OOk1#8>_}M4F zMe4K9er@ZA`OPuDGTm2frD^O9GM{b5T_w)0bsc&~tq+VH;JM{4a8~fas-^jMYwJs;(i36v*3yBV&H9fd2=+Gzay`8bG1u?uWga z=)E{!X0ESzpOGzb{)Ap`mVZOK5o)tX?pv(Kc~?Sw%R^zWd0Xv{RKquggb`|}*5mM{ z&DIjMzFwX%54-Tk*uRB7f_N=`%h+xVXTp6u={3REB#%^&0cbt7TJB6!@|D&-)$|zp zTJQ6VBbPHv#o7RkY2}XSb#-*Sy>H@bAl?bq^WjJMmMlw@nZsX%r%<1dlFof+?|E7M z_APJ5--qhaz}{H%`CX{rQDlFZ_tp0A$$3D8ziT7a-a4U~O5tw(kBx@TY~#1A?|4Tx z3I@tw4nwV{!V5ln$s-;4QaIpjnM{6XDM|WlW-`hwRr2nWXR2%-AvwPN9ya zbN{uyV0k{X{#VVDR`t773|Y{EP~IIIlF+;D&c0y37ug`aY{^zRIUls&%Kj@cZL}9p zE$^GP@A*q=#v7^C2VzN4&6^BAimy0Y6`DeCb?Sx=gNe}0TzrkLh7a7iicGFnvq1DA z*}srQZ^End8ymn*Q4Ye=^x7D z_d#5IBab7{MRk4TH>ME#Cxs%h>$fn{zCudPMn~Ma(0~J3GIW z&h(^yv!L6}_!~2E5&uGc64FV)e|dGKInuwSaK_Zd?1(H!l@en^O(&~5FloJE$_w{11rQv7Go z-xHZ5{A>IN!E)25?#|sO|4#0;e9N0b8*%kTcdA7)arK}xmd{|lIVZLQa$LwqD}6hx z4gvMudpYDn7mh?UoH>82@jGjfNHXHoJ zS`Sq7&-AUQ7{*v%lw&G6e}_Nqd~5kMa=s1ze{6P7%zt6u1il{1V_u-U;!Jt9?Zw6l zJ)6PaN;J^iPV{YmBFB?*d5XRl?-sH)>>fiez&&Vdov@Mm^D)!;Bk~>;!8h5*&=W54 zGXq@&%fz$^?Q0GmiQ_8%8Tl*;m%Tigh`? zw*`DBQ;G@y5uLYaUuTjvanJNU8cj}K2CW4R%}hW0eenZ|n#(Y65dKb}9Fmw}{H`Ku zPA6VDdC0MXyz=^%4QpCoMfv+Gw&m`fxCfzU-I4Ea1VC&OR|w>U$)219v5t*w(O=f9YmZr7$<8T47)ll3`q zYcm8nU<{vA+>08%amAb;YyAmKw0A^|>)G3*-rKFaxcjX$n>)QuA)Y_%6=%O>pgUU5 z-GQ2I{62H1dh$7h|KsAHPxc*jl>fhKybFI2&cj1>8OL8@_wJ3otzsA-w`Qv-e-rTG z&ULHo`|ECm%Ihsp33_ zju7)R^rKvS)SwIbV2PUbQoB#-O<=PNSttI+pwq}Mhxna|?@X3C%=eNk1;6*m+NyI- zy-F1;4k!Sn;WIijU=h1NN0~da8L$xc#+b8&W=efWifKdK464ga`lGC~)bzVUty8%p z|H$W5PBHP_Jy6g1-AKQ5IyE;hUCH;;f3Mc>)a(xa6^v#ltd`l+lUH#yVLP?Do)&8h z`S#51EljWHY+RR_ ztb2&Fm-@anH@Gq#iSaA+NtAv$|A~5>q!%9Sz2>jB9#tvq z_YS?PXt@U7Ejgs8yM?TmbvXW>Ui9SmUZ8IodzHj{)ZSwJO>{||Opb%-rT5+LMYkGz zsZ*FW>#c0fQkOpXU!v3ooeAT_(<8OFI!*`n!g06=)t!HdX5c@xk^5<`H}O#w|G8eA z_vZA%kAa!6R1C}2V>AAV`5&E4z3>^G?=7Bh?7!nH!uR`x{bA03#$H|Pw$Kxb2JkPp zY4|VTKl8bd-{bg;@HhQ;XngW-*^6jmk8C(wW%0fBqz_+R$hOJ(6S7NeE=E_wz9Mw= z?mK70%L^s?gjwi;q|0>W&`-NzBXNV)y}{-AsKimLH(6JnoVGh)+wuFaiD*v@@Ao-&iv$Tzfyijll_d}uYO1Qi}oEa3!$>F54QS$$V zP9HtkE#~d)XOY7RdXesDI`;ar72x|G4mshmyBCW#g-$TrnMvpZh-UAhbD70jlg)#C zJtJSm-)pwcr}t$WAy%KD{uWr#EH$o^w%IYd+zH+%r` z3c1^86F5$PMWov5V=Z?3>CF%1d--1BTz~7U)>+DVE32p#M8jD*erf&&*e@=htoA3# z_cME6!?Gy)YCk&0Jj?NKvU{+S&0=b~CyN=b>Kyb)tPbR{_@qJl9jaIOk??5_Hqm)- zCs>W?)l>W0a{LW{QtoHe_qqLxQX3Pd?&_JRn(sH962oeGz0mv#)k?mLot8%Z9cQ2d0`v7 zC-bdC)s{F}pF zOzm5!Uf+{ms29h?b9&!CiCE9~}>~K>Onr zh&Jc=mdVu@J;Fz0F%5MO=Ih5S^&FGU_d$FmeA_>j`x6J-;5Ue26uJ>zfPMoh&9o7r+ zZ`32D*!P=L;q~wLsvT!mXKjPja&i>Xz;E^7j=Qqf~iFut(YKF)gtKAdyB?Q*z z2M(}XKbe@L#9YvCU9y8@zsO^X+DvkO3VW}`5T##->`habE&L{KASP#ysb@eT?@?~= z4p|du4ZYz{^%_jxlD+>~5B0W{5$|8*(d=zP3-dFb%_VX;%zkugZ%LT9LA*KT+7jBO z@=a*y?`V0Q%H=!l+#>6dYLT=BU`~| zdp5T^JBR!T{};^pF?)3in_2V~d>hCIoy?8(5seZ1X>ruEZW8LZzBxV4W+;2r_4iBi z6_CLkoFo1J6qMG7Rb9_3K8 zTKKwV;iUX?TJPmMp`2#PxvqN0=zS8p1_ln}EMfNcag#C2pqQmU}ioR8w za5z9dl3~Z&R5FO~PM}x(FAgQ+$|9vLkFf%dS`OD%em(O>CzoYivvGW{F(wc9w z{o?%qpAX*{ZJbTr_^HUoSb5JT`v%s)4!EK|Db%8gyK*j_8t8GjZ}$kk{?6#ll4ot} zpY&=h+n=xz9O^w3!&b3p;6GCQ6U{*Wa&pe>uB+vL&SVWU$7-KYj}M7qd6d{In-6?b z^gm}WhO>)*F0UqmewX1_!x1{O_-^2AeKvxts0aTsXbGqZ`?GuN&_S?X-DjXL%+^t7 zcgSO_^+7c}gWiW&vQzvVv!9E+r`!uy<%9j2XmcnfhQ8=1_(c3u^*yklzgMga$Zwqd zk0ap<{a!6X z_^zO?kGz{9YJ7wLLToe$_O?~`eS;bHTB8%4DMY`r``l3ch3Q>V%Mf$3-F_poow#7jqO)M6VX3muPgosGqOZXMcA5a-9;^bMNit_i=Km1_|uSx-W;+2 zjh`#OBIc)htlwS<#3SCCYCSAM-^|cg(MtTi&0+p(>b-dKp=a%- z(4!uK?pGpnLVk{Hg1vkp*5oVsI$)h3iG10wMSh;nT=7hGCTA`Go(1)x3w$7-rs`P~ z&8W`bv9XGLWNvS#b*>oSKC#{-dkG0@>$wO47(-kXnNY7m^# zZ(Mcz!u$47pg85aN-jC-nFaBbLu*5G=n7gIZ?gC&;;V~ol)9ayH-*jPWM}z!iv9yZ z^}S6=0SziBSF|>Cgm?0sgf4(=X8Q`if8zJUDafo&CDrw+I)*ne!}!Z`D34E4$jpg( zTSD)KIu)14cCnYD^S%AmaK`>IG%4LH&RvD4`g%q`!hN$x$-f|49IC+7GK^ z_?%4EF{gP&r^t7{eoc3$OYrrNdB4Wqb(m@|xRKvEFj+h$&}4j;q<7POj9dw$>K~~?ox%$i&+aY$wS?A8+P2m5O z+9%*6yL{55@V=_^d;7s?Y4x~C=Tk9$!Ctv0-e@`ABHIz@_l9PkE8_b_ekjQA3vmaq zb=dl>-abNa(@o{Ooelcx*=_f%3cf4bUC}Su9goh0!D5`oS6XqLF7EdPog?HIVLJI8 zwh}iJm%L-pe9$hlZ=Ss}WFK1x^533p6n+MjQG*TqPE)(>)<+?&{@i8jH9lc;I{efS zZ&@}%?VXp?J^RD;`*AYwGCn^WYuHFew#k_pe3qksA3qb{J{4IG->(+l5w(j&o5Sb! z5;;2ozg>;KMQ4$1M0Z1ExNn@ZPw??u`u!Q`{zv&nx@VW|71ZC_wBD)~B=Ars&ubi3fC${zYv(8>Z zU%(=^BUv l!hBzIdiU_HpeH;si__2rFk4NQqkq6|c<`TcXBMm+1ItTUhR*z)#Mly$t%4Wsc<>FDn`Red$1(NV={r8<6#>)AGrHaYf(s8V!B zoF8pd=GE%DJ5G+Cjw`S?FZwsC6IE4Kv#3hcC(4VOMrESQ1x|S8Gk7GZW^v~T(_tieyX#f5o_y2-zlSj{SACt%f@-i-HhA3 zxMtJ?kJtD<)9;@+_GY@w!_jH@>@H5)IVNZ^01n&FkL%;NZnOtiFRAY<$1fBo!}${Y z*w5eBlz*?^zmKa&chGCD^6s~vK$TqFbcr*fp+0w0@27mOiG>1oEQyl&S)HE^omWt_ z?XBEToYmLwuh<_S7e*b`e~_-1ilg?9t?ctlag}JI-&^^9qdG5*lA?iZI~T889CuKx z52e>g#~j9CJ-*z>r}1prt*(3M|FBrfbnH-^4xryt%3c@OiY^wPm(ry?<{!b^5x?Dx z^F6pcS2>S4ZcALQsGQb(=;8a9)wdfLm1r=Z{SEP%$lr-@$gzFIcz&GYyHq@+v8Q6R z6F0}J=REqq$e+RRDUQdoVPjMn9f#*P9Wg4(=5s-m9H&R=V)S3OG{gBF7=AaZp{}5M z{itqSHaeGv-JDZX43x(8I<}VKcL#M{DI)gb=w;fm-DEpHI_-QX*B|DcA3VrM&Fe&lanoF?y& z!)I1hiq=ily9yUEAI@O^%eX)6*o?R^x==fk$({_BEe>RNhU>c}}cV6*I>xW3Y1{kq;^A9M0DadY4n~Ea!da*kQQcVSkC= z%EYCk>3rUbgO_l71%F@kdpaxf@b#23hR|*SjQ9OcygjarZ^h+SpF!IDbhTra1OB<<6tlrR@3~SHG(J9sY%Q{Dw`l)Sbze z(X6>d8GDs=je7F@zR-CCY1xnur?b8mbY<6eKD_O`cFM`d`NMG%1uBc-hWsfLjiF6w z+Y7aA<=8R>|EG(Sm45%#k^T7cEc;ujV?tcj_jRIOboxeJsiJ=tfA`D#aq3D^&+qK{ zfTqpjvPFGUJI9C5GuU&JW8bGoOoNy4(wA-f`O{UMtJJkbJ$;nXl1-yv5I(OLH{0Mx z{tr^etI9ost&5}b;-!LfYT~RgYO9}9`Sz~< zt+m*>%rE7|`4x_NgEqVHeg_@q`RyC$A7}qMHeRRfT6q21euTq=`1_VUf2eDgW2?!p zwUCLwi~05#j+@Z(K77>^AMenypLlESyaW9EOs3XRUg+B!E8`LMT;}@>v6l}WonH#O zq5bJcoAZ?Ofmoi7hhBWFEe>b!P(Z1jj`c_?M>Mt?C>tKCi(2qd5AEFSoO?mO2vHk`^c8u5xt0 z<5r2K4{&!_{$K9+nt0nrQqD(w zzg3L?th`3@?Ez=(#^q@I75XMsj@?axDq=BH9DJvqb#WPal5W4UsHh&@ATA!4pPiI{ z6)(@F%=yq&J?A+l#W|meg|DF>UcTV(9^ZC~%0;*EX@MAS!S)?&Jtxi;TWRX8qD`$% zpA2=^XLGi;yg(n6rr)X)T_j%`;`#=5FJ|vYzH9Bf&g?$J@frHJ2jfERP=bCsCAwDm zPvEnMoDF^E60v)}^2#`7SXA16J(gz_wX;9^{u1`z%Z?^ITt~aXd^p{CS2!;XFY|D6 zE)MpKk3w3`=Eu2gUgfxQw40^8ipr})i>GKlRvjT`FHqKv{Awk?dg5yV9s2NN3Qp_t zP;7y-PQY*avx&v z7Jl3#N8U7&9q0Guyj;kK(7s$Nc5Bcgm4277rxNa-@>@gg)<}WH^n6NL18DRZRp;F4gXPpd$zZw z%j76oo}?JJr0I9c6}7ix+5G^UC-5h7Ud-p;<=*Xl-m3my>M4caT#>ied6%jua?BNU ze8BfT#L>rcuuqhvhH}xxes6=%Fh-t$o0r(X!7n-Z{F*0o*;iBS4_DU`^+b3%8SCxi z*cj#b=%l!Q)LUFUnL@<>b!L4jgmhj`5CT5uGJodG3rFIFwy5kKD@;93gY2! z_N*E`-DEdz*dB z*>wS%bHvej$3M)siQ1eOy0AvJT@@|*k|4JYxv2Rmjtel|4K2DB58A-5i4M=xAuDdUhbHi^fE~+ zLvr_8$DGaoh(+1rsxxb5;y6*g;hJiJ|9^vaL-$5cpE}{-emtMd-fPtRu=v`Hld*hU8Wk9u6)K@n>?N_Ng7~ZGjP&T8 z5LGm<&HjZrnoqYi{BFmllVLn-!`QBya<0bh?eK!~x+tp)KR$`exV}g=9!tkaWw}sA z=~d-YRp?Gt|3Y+&&h3pj&q}z3c3$aJsztncaQ){h{dkUOoHx+?(~w;tIwR#*j35j!&ET zv=-lemHVq>@@et6xC$O5sCO3r#{2DY=<4@f6zLb4^N15@ol(^EgLeQz+XCwd)Dd3HVDDQ@wC9ik+SLQeC{x z4RI}hu2V*u^L}U3x9T~JlSOze4Q1JP54^5jo2NP-sEdXuIBeUa^+c? z+)3Z&@Q0YGuH2@4m?#<^#aCIre66gfqgXqfWUP@Eor#mXl$D5=FQW?LtfDcHaWf9X zOu$z-zB4YZU9F&Pt{@iEd0W}FR%N_YF(#}j$Fj7G*)+o^_J16Vmlbb(CJm97KKBbJO*4vRB8E=qz$NGoL^OYSW`ULk?Y6d>3FqeU@ya%s=mHGFM1ME>9ZeC z&xxNbbL;EP5r>7V zV!gXM_Q;10271+ESC z@O2)<{Cx>NRPUp>zfJ6a#_zuPtR-Sr(<~CtSFj-+x3{aSIh-wTuaj#x%I z`ueCZv|qPH4WnoHv`-y7lwV%iGuYgdKht5B7#OG>+2s7!XwV%W&$9b?@o>Lf*zLRv z;41hLwkV@LO~$h)jI%2_{~|Gx#-=cP+Mg}git}ywpT^%y)W2T+Z;Iu)Y!A7(4Oc00 z=|6M}W83T1y+*V&$4P?YKY-_O(-Z%r)pY|NUZH)^?s_(-E2oU}N6@?)z2}InJo

|fY_0E%Yw2@0Ums&v6R{fR$Hwrbsd9(2?+N3!!e zsjsJ4?CbdR9eWjx->2_-UU#7FST=qTl{5A)N6U0MQc0g&Nt;ng?o^?N_LB5jlWo~%7CUo`K1Gv0rYg_rYxg*@A&ywS?r zCT<^tefTH?ht(Lf=PWqFmW+^>V&PLca5bAwq1VM?BFw=~pj$Hjx;Wrnu@z zvuEN${bDI|boxXZd<8#?g$i+sHnY5OTm^m_v%&Y-xXu@|1+F#9L?_TTTtoGOl`^~u`%cHn8ZmO8I(`)=AH#1j z4yVJ_aUSJdl zh=~jFJ(B$w^8YD$xQwmOJO4$#w~S-@m(fp|TcdfYI0>uC>Q2-5m(xy`r(^k|YuW;l z^F3~x!fy7@Wb+=Vi01`t-OJYbYz?F1$ML%|#4PL-V__`N89%$l?^?Q^q23pwn9Zf4 zQ`Mag>(#%A7M0^(DVwlfd&={I_O&xg0<%r7(;O5Pb~cZ1Vza|5-HYCicg~72Job z$ewgc*kw1ZnfLxOxE2=hH|&B+{e?j1WUT;BR&0sv{k;_%>NA4Cliw+PcN; zKZJ8L5&EJud6yoYrmRzN656l~m`v09&ijL&+t`ZpZ_Zc$DVE>zbrHm(;%yX96H z_>Vek()VK8mV=sb3CxH~%g-e9wJFB6Y5K5o+M5c-59XlNm!-ZY^80mXmr~0-S}fKN z457m$bv}ucW$LY|eLO)d{Y;0C)txS?d%$q0$j55Ven^YaN%<3Ld{{iD`nDz=`YN{o zu2F8bm^w|~wdT`5K2=h-QI)nS*LbV3F;<~ii?u_g#YTd0X(GN8jp>t&c~gwf(wtRM z|C7$2O4^pnwpH}qRkXh~^cl7ESGC1kUHwyCW3UXrWg53;u{28#=g6ZxeQ6|)Bk_=+ zJxY{|$&OCe4x}27mv=^Ga{^VP7RsE*?s?k1lN>QZUbj$A8Fuf5^2*DEhu~}FnHgkX zDfd*$m`_V|{huheQ;dC5@SSSxQciBAYh$b6ylTv;05A1)6H_Krdt~$xT(KJX`Um>0{#a2Fl8uF(pzJB4)h3X8dt+f3S zZ((e_jDPjmFqckYO0DX?22tgi&Ml{JKD74XBd~`GtDd@^7cuzKDOR{QH=H`_+Gz zHhK(;^6~u~mPWJTVjSP2?tXY~q<#4YgNtxEQ9aew@g_dg_%lTvXW?=uyoSRCBKKsz zg?j=g)3S+rPFBzJ(3HQSKN~BDzQthyJ@1S&X;HtZZ)~N$(&{^orVU^PO{?JZQ;he- z=^gA?g@>+qSS25(v1zv0Iwy+EJ;cUUrHoe-^pR!sPv*JAR2Khpp>AA(ZxxI|)Af1j z=BF#Ex2pE7n)0fN(`w@1eL`BiK<^>qdZ4x}+(TYP^98V+op;e?PL%K3zEDoZ+St;@ zgQZ=wCCcYS*BOb%If?qeMEzV@eMXvc$~ivWc()RLswk_fYr$%Aq?)#_deOc3p>${~ zr+!oSQgwI7&30PsRbNL4efcgiR-HGgc=;S3TiMYD`p5ZfFVr>_vLn{cm~)ED;Ld$* zcAX&J!(7@@{@;a%@7XZ`J{JoI92K(j9JV~9{wA;j4r*g3!{1^e68jIbX)3G~qtoTV zVSI%7u3+08@P3?whX!&X&s*NGvF$T$W)uDJSz>7@u1TPG>HKc=;W&*fN4Gk3xC+g*I`!_+E;qB>b$QTUgO>x9yAc z8z`Q`b$vVfUC)-+aP&4_HqxvmUQU4|-1KAD!YD;Pq-x*F^RGOyTu6rA56Y|aT7V;~`dnvJ68n>nSouEHX)c+)k zza+7o?AKKOr0Ty?-LJC_K^%RIpKx#XD>j6l>Kd_AMa=vP4aD)g_<2YiUxb_Y@p2jt zPR7AV_UwqWwWT@6a*g!Ox$f6AmcIpJt3Wx0=A~lS?xl_C6U>d^LJSXJ(@I=i06M58Yf%Z zlw*w4z%^8^@m#L!@kl!zyEZQ6IP;KfYO9oU*wl|r`*HM=c6MBxg2NPTVv6=Nl^vSD3Bu}EF*P(9pc@FT-DNG8p* z%wc8wtzpqzf4(@$kgby-nRcJxqL7cqYmz;^IVnv=l3C)cpWHX36KW^m$C&oND`^`m5un zE?#ESYL9q1jE}I&;2Wo=ZoL(;tHaisL`NyGW@Uu>R7xI0AxabHsz|*i=-SgC$ zgM;_wPG^1(rOz7vJt(TKfPy&BoL-)BN@LekuJ3LCQTKkHJ;=j(*l$ILGx)ery(iG1 zo0zx-9>aMJ_yDHU;8`)*6aUAHk3aQElf}kmayJ$4Tg6AF`ujj#ZQ2p}66*)=i!$(* z!PZRU%}n=6>$_&k(uZUlALMAi^6^_}uBOo3p8in|g&D86@p1_!j)P&e7{JaOada&` zX22$xE63l#S6e*ohsL;?2OGrFTl@%p!2vOp2!BCU_Upy2itR73`#Scvgz>R<2(Jm^d}Z87-m8T;l#4nBWm=SOUKOx;h4trOv2alIHv$I<&Dx%Lsw!#b@YY?{HQ#q@uP?oDu3 z9WKV%PIlZX&%%uT&+_bRsL!4-50(k-@%0Wo$(}Z_jpl8&4fE-JD&BsE^VxPT1s`Ht zHSyUL7SZ}%_?24SGTaKkWLNT|ueM zbo`Zu*T?zRPg)7WjtYLM2Yuj)C@z|_x{6IHaT#N~MCT?J&0#$&Z>HjF0-IhIZ~dVb z?xxeVi+p)Ht{|@}80^ikxfh3R9-IJAQ$FG+ER8s%&w7+ zI*nan4rd-T6L(jNyBTu2Bm1spPbS_n*>VYO8__hxSF-wFz|(E&4(qWN;%le+Td8{{ zp3cHkXE8O5hT#rnxF7H>{j%8<=CS9&yX?9TLR)_mdotN|k+!8d^ns2zn=BtLVplab zjbc+b{H?&>-8+j`RYEL_cj=f&cmY@KIZcfDF}q;)B=xEeN#$69jva@+6uaDo^Z zC=!Rz{R4c4YlSOh#0(ga4Ug_C(x=oO+UN`|&&swy^Ib{C=yw z>;kXBwz!_0%*3s=nYixF&fDex-?$#fzGfKg1x?s_4csh_8o{S{?;w`$#QSVEFU5Hk zF}0S>5maY)JMr`!WZM6aI2t0xhw~!^*I`cSZN9XFBVv4pcsf}oj}TA4;eQEV!nmd- zUs}=NDtHD?rbQPzEEHFFh^Ncw5LR}(W2V2Tt&OSe7;DHPYy0p&l0TbqUYAd|v%85S zpNBPU@4)tOt+1c%i`hI~4ut!y>+n5NoOHzTD)!6}+q>|(2tMRxU-A7ryVkMgRCo>6 zv*S}XTqq81Vn%_}+$3hI;P#NX$%YB=6*S>n0i46PnfMLsR+_=1arYiT?rC^NyBwv0J}MEzh{e3dmuD61V$HV#X1 z?U-f^lx8eh&f1%Dcq-?5#k?*R&u4!kTq6cYuxuf0w7*phj+YOm;Wk(TooKjHUWEIr zjcBz>9Olt$2ZY>sK>mNpuc7=3*F@>srOy1hSdJW`*Ae)ZM!(W1%Z{8AW5R8|l`?S%bvq2YkJopF{Y$grDuAM&i1WHXzqEK%RCq&%A~E1lr<{ zwGkO&u?Jln%g2zLvuJv=Xs9e6v-x{OEQU30VV?Xh8YaU)`rR$&hVX3#e=6~-hgiFo z4`EHxQucR*14_SyZQ~$ZyHCRTX>7@h^TkR&t@8Dkh3?}P%42=Dx(alEya0D?P*VE zEyD9Ks3m5e5HtPR`jC8Y2tVMu21WY7ck=&wHN4G_)A0QxKX&6etoeHi&#N)?D8J_8 z_GP@@%b)jPJFMeRsW>*4ie2NC(oU9gUZS{5G&W4sb|i_jB>iw%?Rm0xE?F)l<10;{ zmZm*PQ)juN`L>Vs6WuY<1rEyV@Vvm^tXv4|ao!oe<ex#8$E7#m;f!_=CZ;|VpSbHA3E-2-ixU~E(ZO%5qSgVY_ zG)Y^WgwwKYO)h%IAk3d0&-T06{5Tyh&~n|1|3NT@{b61GnQ*(jZb^@i>G1?V=0JD4 zwBbv>TzHlr<6#3d6-z5=^e~OW`qC@;c8={K8vR9^2WYaHHt*1-89&$2r8_^Xf|aA% zz--r5*UPh_-DbR-(vNtln9BZRWyY4QdjVWWE!+knheJ=N( zrBlfDuomVCepcgexYzlhTo2EJ)D&;O@O254mG}Mm{55|!^0&7fxtmqvoz)Q*!DRXJ z1AmwD@nktaly7Ar1qSgioo^q-+43q!o;TEfHZqsj$XGpCZjBFHE5Zu#^(Kw}&}LPF zS71M#zK_e`H_;J^Y$@x$kUmBJgnP<+A%_o}VK2QNg+qJ@YezqmJK>pzSLt`NZ5W3y z<-=I^pGK?uAgnEYlI@$t-Cnt~kj;0q_f}d=r9&C`gzZ*~n#awFnv1iod|1Q&i`hR^ zpYmwjI65iH=X*Xo3iXqP;-=7bc4VGBE*ih@VttsiYC(tQ{OJ_Cm&2dheEAhy*Yc+q z9m2CjLtrXDn$e^SP0U`JYpI~kt7vXFU7T0)du7)nRovUECcCS-?yF`EQ!U5TqH`^6 zUu|R3Ix@bFHsv^dM_smihNS45daeO7%yVS1k#BKBeM2L0)2L`p^evh!<<|$izRPws z|H2&3R&AZT655ACZGGgrBQl>_ims)N<&)TDzJMRiY0^h*S$*aDuskoy7yWBYj}O>i zk^N6o_$)p=!iO+3-i{^>_z>3LeGBPws)<~hp~i6CaV9^`=SR5LK9C=8YZpR)UzR?v zNA;sW`SLrBX3%JclEQkRy>wc@r%(8{k5&`-HQIJN{KTI#Wbz39%+XGS^(EmdYOMLB!!zaDaZd@%&yW zv_BG_$3UyL@Rak%Kp!t|FOU$!twAwn)WQnmz&e z`zYVTx}EoF-HMN4&2Shu{KmI``L!8x__b1dIhtQ*!kzqjLL0Kc_Fw*8&ZnDL74A7E z@@pucF5*vEBR7Zc3;1vvOouc+<-lL!c_}}_{6|HawxH>2bQ}iH!5U~v-}7J+ZA;O$ zBVD&Z7rqV`k+bDyL%0p5!g}%j7QgT2>ps3+#JAq`91&O0uU4dEx;i}{MbC@)dp+C_ zTj2-3zpnipMAtB{m}Ywj2T#!TcK&D3bsXO>=I>u2&iNc>aUP=CpW=H2&HB?Uw4aN# zKmWo`y7hE{>|p!YxLd9$4)eD&Bqw-6R+Xf(?7IFVg1R|v<&m>VRp6-Yp#V0_;&^W7VvE> zp9b^kZOlxF3;0)nrvko)yXA%QIFdtg(fX9nXd2dbq{8w1n?S$qd|b!J%V{`5Elq5P zL0GFYfu>XWS_;Cw`&r`nZ+`zPmMih|ISBpYRa&uI;VpVEr}rOzy#W4&bh_84^F}

8Gxo<2NJKUH8XQ>cUp zU$J60S{u<$FGBZjLHM%^|E?ItdR65ZdQ3hu4q4)94xIwf_{`?P(e2kiyfwv$at_ zLAG}4vpB;T#q$7sZEkx=9-qSB&-lA2%98In%FEFvHIQpzwB9gkO51l?_ZMwjKv)OA zkVRn)_L*=%Ev4Xl*dU)%a55hX?6(v9r{JY0+y;p_dCAd9acN_U(&AFzg_o1W$Ch-vhF+pQ(v*#AX&QR?`I*w+*Zs>uh zr*U(Vl569pG2g(fpY#jj(wCu|TG@4`y9FB1D}*$Xz3Ep6RHnWzmc>`0*l?v)`9qVSU&O z;(UoX53xQw_7nt!`y>=EzpIcfuhHQ|r)c@5OEzX$&0e@%V)C43J5 z?xJN1El+?o&|X{jG#-XHssWr!*U;F|i-N$IFYc`2AAbdInCzJ^Jv>#oO=|?1PHpst>NiJlA5peg->m`w2e3 z#$kVPbeh`8O8v6YjV!4Lv4)^MOejP8V5bi;>5nCN_biepI2R93N9M=4HfHAO3J8@ZD z&pbkg_A1l)S=y6q_Z%BU&0)6~%93lH#Y`13^DW-$;_X&>_Z&|eX+P${a$K#)*`LrH zXRXB1e)t2zTHH%<_pmmlt2k;dhSrLq8}JtHY4yU}H)5w5?q=X@vKScvC*UkRr90IB z&%oFHIGKl^hwu{qjet+#e%$2aWj$WPQ{W$pjq70xF1F+3N<3`9!(i>pUYN_@u~9wa zxq80OG#;&Qe3EU9pCc!7#B7f1RQC+|J)O^o_#B?=-lx61g|BV3ME~LMdGImcKH^)% zr=ENY{~i_}Pw=e+-&)vS2v@^YekSnqRj6ct34c4lbz06e+ z6Xd~$I9ojBU?2xC<`3y?<(x5kW9@#v{wZG{Q{Wn;P`qd#;8Fg*!QWLhZNlefv~0nn zuy)`{zJ~nY#GiNh(~duH!XB}IJ%8??<9X1UU+ehwIWKncDLkJ&6|S@YjCNpv{$vt_ zdEz3 zA)IG_lU!*nULKVzMvvy_I)TA-Tx3v*Pk9H!zk+(&p@-n@d#=B%)a7k}2ZVt*!% zO5wwkw)&79{ZEc@K?C{HQ2mX}tL4gBbER=(_fPZnrN$Hx#^W!-hwv{Rs^g$34sL|e z;;0rb-i#CJny7tCH1a=h9tPBP)1*uCz{vtUs=Vm+&mZHSi4oo4}a5z$9+)Q{++S+ zB=GTGcmUpu^P(2~JqOOSzfJC}=65>?b2T%?$0i(v@p=Ya25I;xuYI`z=E8Q|G=ufH znTMNYuq#e*y_|@{MC=p`oJ78Bv!XfMl^M!dK= zw%P+8hh_34Jo9^|?LT;X9B=*c_AivNe*kAs;4ZBB`xkewij`C3P*^#bg0GL_{Gx0A zuy%GQ`~?m1m5G-yTNBnG4iGobvJ*=Q;Y>u@S38& z6!Udy+JH3IVdbKRyg3X>_Qz>wD#O~iLeadyP}XdM#p3299EJB>^%I4&-~;*730GOR z`EWokZO7SZwx5Wlo${(VoQ%U$;A}jW7FQSG@dVrH@~a0v!}Fu};PM@9adY^LhqpqQ zx7>uIVelt@_TZ&M+}ON%|2C)|gR4p``klkmLbE#l-+xpbj+_+-5N zftR<%OkSKs_axWRNv;KwTqn6VFLoXmD-+{X<)s<3r}3$rYZ`qzKEgBiH^UFwm2tQT z&ta^?MQuF13|~N79GroVJK$*u&w=m3$0fKJh=;-ae^OucwpiNi%<%l=@i@2@rsH5A z4*Ex)v!mP(;wP;8`WUL%ZzG0I7fYXuqqeY7{M5%wy4b00yHESFHp+D$Hcy+}829;9 z_N;DPAO@@%g%4mYo=(BpK|D=`YjL&@Uty+U4YaZUt5|vicUHj}mnWG23az&L{ICOu z18_J3Ucuc!Jl+7a#o3EEybeB8b2=W4+KpYjLks?fmiYVypMOGF%e)h>;Z9IhctSoJ zc^E_R2*<7DWDf}Ma(fHko!}09_s8>mXIu>ZaoZH$!|g)x)IlsgjK?l=Gu%6P6}rgH z>v%E`7Q+fyfJ@nK4CXyQV(fh!x;tfVEJy!feNEhue~s9k>$)aaUzlf`r(N`}x;Wn) zZX`C1iEMY^ZzeDH#-20A(!KB=?!tQpcENFCuMf`3;A;rHW`8r@rr~TU&TfJ>^0O=6 zRzqp~=ZM3}N(d_`ZxoC3@ihTo;k`&eT0v&-tr4rE8p(G!&cZQ#=2pCgcvIe zJ>VvIAFjjE^GdA-zd=o0y@;buwyWSn@wZ4Oqm(y&Q;pa}R zMz}9O1~03`*+jg2Db~WXil5?U5tPQwNZfn}%kcB5SUL$elW=G8aE1 zaC0BaSKuN%>)9R`$BU;S;wh|-?FtLv6udNpwlE&1!Au-^qno~`yjUpD@ABpVE9x`S zwHe+s22&usui;ZM)XcV<;Z(=&Ep9|}_ z>OyGa!g`K8T;@jwxG@jt8ZA6y6V|d!gva1nSPUnN!Drz$T(`q<3-Px^{$7LUfp|^8 z^H^wX|4NvRXK(D%J|&9ZM17Zg0ouRK;w}r{p?`P+*IihD8H9J>{RZ=Kn}*wv!;{6~ z>te7i?wY_xI0#kb=b8BX5sy#VE`+dl=_h=?g|i#rUTxkqv3Eq=)r8@&0-D72#6~@B zN{042)A+*LEJ(3`18kJD>*VaGkR@-!b92|?@m4X}6fS`A&`kc0z+(d5!aFGsL3jMk z5tkd_GhCeu&%rA=t0XqZLX!O!&;|y>UpPDgd+p#G2={Qqn5-$B3LWA^*HKA0@cbYA z2EB23A&e2T9mVLc7#oMPYS0x{z%DV_0DqnEcZN7T&Gs>|SXKOmHLYPT=uT%0vAr8_ zr;Ee6uw3-lhPL|A@ciX`T!wpt?Zw|L@mCvKioH{DcmsTo$Ei_eGT0(KwGje|v z8rvU$tJg$Tx@~y&>3n=OusscWLSNi97K3YXwir*TP>8$h#a&qEdlTF*=0e}QL44K1 z(bG`{wp296t7u$MNgG~CUswqzm5Q!=7RcG!TJZ%iS==?i-^2RfOYoMCzsj~h^(U&uqdv*o>b+uozCq%{Uw-+YW+tup1v?jncy~2_FweiFlNUuouE} zE#-WF5I;BLsUMC`)EeuW+-NT7Jqr{XLfM_X`XML^UV!dklxV(Kli75dRT#MF;?YKy0)cUc)5>Lnl7h#nZE5sTZE+ z;Hix~d=U0IIuq`J^>7aU9>Ci_I9rda&&1M92=BV|goAdtsxfOd5m=3H)wBcFEBJg9 zJR-LK!dDkK9e+RS8?xaL{jiz`E%zi3I z!*jKzY^%%F3vv28K2L`^IPHzodqv+_`20N95#n-QJ6taW~Fxl!diT0LU<}R zyl<@?Oo2tX>n6r3imz7q3)kH@mN6Yl1;oZ*9M8RU3SCfZOA+$|z5} zo~OOY*ZzgsrULVV1-L8FE)<%>h(xw^p)d#zNHOlt(N5-wfm|_}uRXCwA0OdeX-PQQYOk#ASeOor;V;}g zC4R!Xt)C#Y`Qd#+8{i<+_BqU-hI!Lbup3XK#L^g;V1G8w>OoU&S##(C7r`fZYl6S4 zZ9l-><5arN_EsE*H+6-1;&Sr$Wax#n6Ck{Ya}mzM{pmdMHbb0!E}Qm4;4ZwIrafE$ zZ^Lf!7W&M`aX1$~g&*LcexWx+IPDD6#9&xooCF#8%!9iijEn!2x3z3{;&{Dm4Qo{9 zh|8zM-fg6=tzu5YE38m-oS4xV9?8*t&tazlLJJq2n9*muHS4&;3Db%*AIL zIy^~(Ch!~1{ZEW}>I?T_u5lKgx4@Ao!C0n@y1WYxzmG#D94~=4a2)#AeYhPXHkZTq z(8}ktU@)HV!uL+wfAD-CzW;!}_NU-{FP`%N8h1t-CCR4<5Ivv8haG%5d@9;Lx!@DqB;=PG@9u6;x z=XuaaOi#uAH?(Mq_i^~22uole{tFG(v0X}s@ZQ&O zPoOV+LW8k1m`{UKY{UCi!aex#9?Q4n!<;Bro8ldx^5R)Kbf>{us6~vabO_H>wbN1! zf`i(Ie`qioN{jhB@qZcIOM@3^@EY7;e<9wlr@`;g&HidQg8zGQULE&!p$&xVno&@X z4lQ6REy8;uQ|Zx?9+$vAdKAFjbm&ci@8v-vX2Tl6aDOe6CY_)k+zx-yWCDHWz$n_h zAV;d%z5>5!Yr^{v2Ezgf@Bew1MIXR#blO6rsM<~wM02h5~ZT{?xoJ5bI38E_-5 z@@e)3y~5mBU-$|R(CsgJ9foxIa;g#1=al#mu97!BY0Sy&0a4Z6|I$}r;z@13>Z5k}H% z6wT()>_s`YfqorWGfHl)rrQs3Y9$_6zcn8V~ znyiL{Gzjh6+4878R_DSdcm~(+$e$mfn*GP%b=-%mjK1&!-ZSw2wC!phu03%fpR3Zknz2(g?W5~4JcnoIu7!Uf zyw7J7j>8q(<**NGf1sK_zY7 zTG&C4Fi+Uq_GP++wb==D`4bM(B-}N53Z8~>K5wQ?4M)tiT}B(LQt|HHg7W0yDA9RI z=ID})sgm56D9a1$p5;dY&BF8F^Iv;Sv1O_(LZt{Jipk8PW|CI`vJH)=LD^Egxd<2Pfx`jpqZ9k+_59ke}#eNfc(3M8#zo%hv6?c&iC_a&>a64z=!Y= z)c4&q{I{gTE%3Fz=wG~V!GFH(Mer33?#BDIa7Ucv{$rBdE33TZqC9yK=X)TGSvTQ6 zyklY>9KwHJI{ZU}9(bPsE8z0DLKL3=3+q?Hd|4Yv!TC!#U#)Ms%yx+!c@_8La6S@4 zzd}R%ci?>vyr5lbCt@C=!3{LH6+%DqyFB?5@7wV|4fj3p{yEN*AtiPX67QqpO#N_X z(Rz+n_P>_{XT@1`&eFbQiTiA0m>kzQIoc)9>d?kpzKjFHTM+c?vABoV=x`f7{-Q%$ z{GTS)lW7p%Ew>7<)$#ioguXu9*M1E%>)-1`ke2Bf@9dD8CG;!?hJldmNWACU(-`u=rkM`=J=W zogS5K2WnsHiSzd2JX>5Zrp<2nn?4Wlwh?rNA0W&RpG2S2p}_u2;yUyJdui3&_7m77 z#s|_XrqjLR`%DP;cxH+5mu=U;2Krn=r^+;1LD_wfAl~c68BuFYX2|Y5R z=;5w|-1Rp+U?Xgz$A4&0n-&=m-aGy}oJNy&N#mVhT=U_36DTm`?uryT%7lW zN%mLZx-8C*gY)nm{tbZe{83rlhj*HJtDy`iZ!GO^(m*-iRe>|<(MG#d$@T-0&XRd}aRH5dV3!wa-31v$_gUYA?fh@3SvYX%!>_60WHpxa4$kv!WwVq@Fb z+9ddfjv`gt(@0z0$T-os2=>UYT~VHKy1&Z;FUI-W^L*RzcHqLI@zTS#RpeR+nhu9+ z=-QU1UEpT>>mcOeX4pa3-ZXvzX4(Ii#$g;)5l*D-aM}*Q`uz~zV;$D(Y@zKUdG;3z z&!yv3cr#8>Uy5s3YyIf?AuYpmq7@;$Pv$)c5zdEKC?4LFIS)R7U*HIY^f2vU%djqew(UlmE~DuS@@}+#{bJ1aiv681`n_oT3H|Pjyq6Zn zz=^T_W<(*OV=@1pTrZ$#kVFHwai{KTgM6dR6DqIEQ zVPS0kgl2moXtGv|t^6B*;TtKs{uz$Mt z?=hOJfzRkMO^!VbPr;QwPmpJA;XHVNE|1bDkuLYhw{Z~GYPO-p0Q`sNxt_-TUGiv{ z+zId68$g3s;mo+6w%GfY<4jCv>Z3F1P@k&xm67Fo*1KDw7n~gD$oCw%>K*=ZL;Y|= zzBZC?-V1DdHSCS^%-iPUw!pPQf$J3We)_I3=Nzu-pMmG#JE%c}Q)$s1|KZ<@@E^FA z7Kh|Ym;?Nq4k_~EM%9NzatVYOVc+(^b@3f*xI!{9Ud(jGd%2$%xnasLJG zf6x}jwrAqsUp~<Z>b95Z=tl53sY!QXnPX8OOIA`c?J$Zb)Un!!|)ETYA}gD zi|O;V{J0LfYTxR?^K=U9n!_`~;XTA*9`a0jgt^E^Y0`}*A3}Hg;T~vcH3q|YnCbIk zdd#3fRs5evhisgOzr{1qb`S2)#A5hcN%P?ix$-yu7uX)s-%OD&9pZZOv>vVNx!%j* zZ-#62OxHl3rHAhDmcC~NO>VROlNK*vtP4F}l|v89o$)YD{xpN(a%L)xy2BH+89(i>!r3(L7zWR8R$d!f+&L7J@6 z?^KpYner(74YgDn_0@*$l}~5TX)=x8ls`|va=P?V>kwED7ttg=9xrYDoj}zD{c!?bt<&}S4LNf*eQtyO zvx!vo&((Iqh0$_?0G|Y10o@(&Z=m z9H7k;bP3M}428AuA$$U_(`X=#!rw2eZND$A>dB?6F@GIx?vXDowPVw5m(%8Q+8m_I zGFrI%pzrn^9XQ&g+@3+zBJM|dyk^%JP}&ZtQ%~g+b+7z!gg!hx!Sek z=vSL=BkA^-?JnAtr`rSwYh*|H{1FWY((X+-)qZ%V>L&OKcGEC;dp3l>efAdZ2EZ=* zO`+CD@MoOJPwyYI?F!@JJqUlRq6r;;qn)R_v`_Bq#mUAeDKaiaPJ0d#K88=|*jUbu zmUE}jash-jTzAU5iSlnYyhq>rS^qY4wjb7K_os1hT6d!J3v_-dpK>9=-2@dm%hm`Y+ux9TDCKy#U^%e{0wc$>Jc) z+57_aw3!X%Wcd4df5^)p=wCVVo*0+{IX;gT2M;LieA`K4pogYfiq^g8dqa zFq+1dp)QSUQ?(6rgwZ}vq4ni7{})EupGfyaT5qHC?R5TxzTsKWu&VV62y0d<`1~ea zm(n%d0dECcw5efUI;^L9mX=jv53Awix!+mWB!ce;Okbk4#HF|XB-A>+y zzg_npEtb)uhV4|jw~-!e>5*caNrU@n@dq8&$+e%99^P&7ChUXK^ay_gwZpMZGS!UH`dq0VWmQ*c=5J6 z`x9Y5-Rjcs2voH{gNA=bS?=p(ySC5PM_B&=-RRj1*1-n4u8kWS!!~E~l_Y<`|N#lYzzvw>E2;13o4s%x1VHf;O(`}fzkA@%dqN-f$5E)O>@pc;e3$W&F z5{-ww3`@95W$j@zN1{cCA?1T)FD z)8Sv5hUWwa%dZQub*c99DhPiQ_GOyBOV8(MXZ5f4p=#8UhPT^pqu;NzyG1UAcQl4I z{4db#3i^e=C48d&E-(pRre7;Mz5>tCu~4p+x9v(#tA_nnTc2y~F@(Q?(vqfk((nm- zzD~D}l<5cE=y$e=m;&Ki?T$D@zn8%l?-Qb%mB{*z`ZTWZdLqm9RF>`7S zcGTKYJ#EL(t^!<7zq{mSc&GL?aTN{A7#I2f7XLS-b`;$R;bXOwC|X|0tfOO+-55K1 zOxO-{P70tAVwX6L1xcQc% zgridUd~R`>jf?Bp> z){D>6lzpJM%s-DQd-yS*Q}}kR!OYR~k8a1&_;oyfQP&f&#W6Fwn4dKj{(F3CF)y#+ z*_y6PfD@$k`jvrCX%|mP(R2)}(OPrUfoCoRIqeRRettV_77RTllIQy!E ztw)df()Snezeb6A&nWiuq+{YHoO5RJZvz~EH}2~c+mx30zkNSie%NNQZ&~qiCBLuF z_CX~y>QJ1g8;Z9ltN%QH-ExfGwY3AGoUe<&|3NuN%N@g}Q^eZc#m5}|{^+)l8dtLeot5{@lThu+2cd_Fs;7Pl?ow=`wk%D&s+pZ}wd za7_5GrgDEPF5?1a%q!NmU2!g6W+~DeG13PdIma@wO}8Dwg0a{MN7d_>YVA3g10cY#Ef*GRTfNxlfou-H zm;A3#OubUVk5S5-j>o5t@#p9{uN89=KG!O7%(N2m7kx(Zm%uw^TPKt)ql{%gVi{QuR*=`l4CQPt!9n=^|N5^Z z9xFPwA8xbN^FRqLN}h9eiCj6Oc#QCT2|XSt{_dp`$A$A(7jt>fQGbdHoltD=0XbUx z7>ybi>z(JEmz8@O8^_t-?zqqY54|ocVaK&4%IaU@`*izfmevQB(7%(Inr9%Ow+uzzZYjip9`Cj1r{_#7Xhim4{?AfvQ+I!B0&#iai-^mbMtR?H& zwSJ(lfQQpKa9}srwoA>dk)E?r^7A~Bzy3;sSg+$;LHy|jyf#q=zi|J3?~kM2SlG97 zu{k;&Gnl_zsQ1?7d1c@>9zG3(uW=yXRQO=$2EoUx;Oytb%jwDtZpVdumhk`Be;0bj zv6oE1{0p3!%NR!m#MBlsCzrUL?q=XgI@Xn2({U!_ILr7Z!d7nD{fHQ-e9-(u!7V*) z)f7ItGlLLA1>V;`NS@ukjTO)<#s!=$4U8j;KO4b~6cTS~uU_IH?%K35ls*mdczq4I zC?q@o;u* z+$(<8qm9z=={xi$+y7e59QWe3{b7tZ6XLV?P)~j84%)wfIPS5>w1=NRdijbD{6CVs zn~r*{@mh%Y25^0Yc(n${z7Sv1ivM;Ff8iA_1ATs*qpjQXCu^%Wa2&x{-ym0?OZ@fM zIljlHk9j2@CNoa@&_z$b)1hNZ;ngL`=Yzhj9#+4==gbn*TAZmVc1+(O*?0O!6Qkjt zAMMBqlRIzmXzz1}G43Nb7BM%n_4jnM*B`RLkDq`|GxVcA&^=iE8jn0xfzDIxRY!@p zDT_lq?~AP=UJaPsw;sQLB`+pG_cp;TtJob5I}K#d^yo7Yn4e=Eek?p&EWU)tdN$}x z0RPXjkDce4_2{Xs`OVOfb|zsj74kHeSYN<<5}ccdz8XIuF7?OFlR~~9YNL_odq&{* zF?-2n5eKYJ0oHm604>F87l2(Z)8+%YNXWpEZ{Qyv+h1_Rx-#kIx!?9e$P( zj@f=f;4nyhaA$1X6rv7P<`dY93tZ}>wz3lk+s`cD9QK|Sa%Thdb!A^QnZES%Dr0{L zy&b@r!^~A%=qN*cCnM$rlGoFW;TYn}#Pg;z)PsLu6s{ufyEkd`8~T=B^6Jh$lDy*| zE`@6jzNLm=2iOmsTtDg>NgoTq$HDOHBG+qDcWdOQ9b&O_qbTci_~R7j)?e56Edf5a zr+$;GEnH6sPKOV$86jXzZMb;@FR#V%yQ|x4PMq!q_~T^C3jfye>{)R6G<%?v3u}xU zg0FVwEAXi@b6Jqs;U7Bw?PFMab?+muuiD;k(nD3&=0m}SV;(k8@t^Jbdfj`A|A0^P zSN?5xNEmoshm)U?pU##W%L*s(5B);MQdqDmhco@DF9F7_R` zW-RuDmGXOQHgfqI>iURqSwPGT;9r`?8<+cMZ+229V14b`LbI>cs|eE+os zMkl3boBR^dujg%ESMYUv9Re_@Aox1D7md3F*C5$1QCmO5-!{D4%aM<@Bu|T|Bca!q zgY@azUg1D8;j)v7N8Z*JTW-(BRbF0CP{%uh^n@1F$XF}X>5c|Q< zxm@bVWE0{ZYzqC!flVaV)oS?kgO^*A)n`ONO~2)>9f!TI#AiDnIiPk`uRm`~K1>mw zzJ>eK$qj}7=K{VB+`f%}dcD={GA-Rv!dwRDCKdN%? zcitIq;r$$ND~p;nJtO2BWeqToBAAw=jLCIlJ$4klhLPV1L-cJsXHRC>C@lUyU;XE2 zMdl@q=>BK-|C}X4o)Uzp(@*4^`yA%+1A2vOYyl3>gs!i_t)t-jaIa?HpD*fZUtbjacBu++_+I}6WMX{8|Zo4vS2IFun`Smk%Ic2j>&P=gQ#5I^;AG z`E!eRN>0JT&ghd6d#j)$6YPu-yB$4@kI0^#Zv?z2Kqevb@;zf)ExA1j&My@nw+EJE zapqf*Q9r`N*I~%{8`z)d*;YB7cSOzoCjDIZ-2A-sq{Lp|?j$v9@@e410LXlT{_&N^ zC#P?{^gUZkL|!KK__`E$w)OhZPr|vx!e^&DONdyCNPS&GA9@XVM-|-_z{jj|_7C~4 za=Od2eyJt^StWwHBw^{>B*3*NR?0`E7F zvs;K~CG@=sn*+W6;LawM^<*+10b5^^v&m)M&OL#?UA%X45#KS8-GM*9f`b>NFF89l zR$!e@5}X?g-&NMlZxR>xZsSJamK3=h3?EX^)?CCqJ^-7hA?n}hn+Y9b;rB?73&Y6y zw8F`e;KAn}7EVtFuXnojps|z4ej^-kvWX>*3Sxf>@pE$i_yv922>rQGw>Jao)}6WH z#W7g=_k7O3liVa@?R+h3&4BGD*y@ZwhrD}uGIKl?K6k_3W{LTf*h&B{jgtQ1{Il^s z`RCTUAy>@<^4UiC|>W9sr!p~~J8>`RLA8 zdO^NTE{5=L6ml9JF{KQ_gA`sakJD~_te@pKN@2-E1FSCuKi06mw#4lAtI3(s$Zt$=q8ntdf@@vHuZf7`9OM1Ni>Vvr-USyk;hgEs#rhuu9_5x^ zV0%U2!~4ibb=Ku9>TE-u&wyP9^5;DFCJ>Kg9CHdkO{YD>#m?)(l_ccgR{HiT?B7C+ zshFQZQd220Uq^Y<`3YpJqaRK4^4}h3)3at$0kgN^Z|Z>B_(^iykv^3IPItUKv|``b z3>+`guXR$B$q{#H*1GxSp4*6{I`UTrdz*;OV2Q`c*;qsJJRF=VPkGzRXm~Sj{11}; zmqYxIlfuo1-tZX5FpgZnJr(rFL@cSu%MaoENB9##{573J;oC*AVQZ-o+Yp>j?QuGS z;CqQ@Mo2#GtUa$HhGXE(Y3R21N94&lsfDxDU6bDfDLk$g73@z*eoWt=BKxJ}Gcjz= zXI@Ose=Y%!x3GVgy00O|kBR46-tm(}_dw2XaU4TiZ=yd(^lwv{r#Ci+<<&pdr!Uf{R!f0UfHRee$F2nDmTO_CX zq<5IEZ$kJQWXG+wK^zz9TQ9`FRrEWV+u+hCklQLT>_h&`dVT#%_&5Xjc11mW!`QE* zo`#~noGu-6qb4%2XOx4kGy(q4=67Ef;i~Bvi?fluev=h4W8j1B=dpeG5({}b2|U+B zwjnu@7Is>bzb1R0>kHsRH>u^3Avk1bwu?U>A`fF}_X5vUh3%dqV`oaRhIS*S--L(9 zc{ZCTZ)fbOhr0P1e})mqj#8JmQMY#xTY30C8uL-KzX7%NDfRXAa+(G*OTg_rkbiFB zufI-%cf+CoHOaBtv+;NM`yTx)$vE78UiP+&Vc_LtZOl!7G9jjF#G#|~%XeI_VO`pp zPrVrKc|IhCZ*5tVnb~LT+-&rxoAl4#?YT#8wh<0I-+BJ?1mpY)V+C^jl84Ji;2Q&L zqXS}IidbTUN9h9iY!-5OcV8TGE}!Hh9eHo^osjS3Uca$*_nZMv8?YZ8eJek7bb|eH zo((5!WcIIbbc}04Ft!02um*L;r^zD22 zwoP=-0hXVTOIH~CR;f|bp-&^hCoN)n7Z|@Mci|?PyK}vT4{e3Fzrf#nUfj!wQ+?FI z4V)_yBHr#^oHb$hYv6M+j5|H8_&x>v&rI2Gq{iBbf0x3@nTzeej4|aKC7t+UI>t%8 zed%FRk#(%LMxhQ)gs7A6vFBvRi(a?39e1&thc#{v$fe20^|1dHdAtt(=Mx-F?xgr( z?=N~Lk#{qp>n_elVSn|{eFJX9gAb{Mr~8?=tHAXQ>?LCj>e(Ki-?nG}h7uZn)S0v4#(d;L6Xftd^F5XQsu%9QrXh0c&bx6M&eRe= zZ2b~=Jw`YbRdC!!%)SrM{|Vx(j9i+IZ>45COC4+hCnC$Y+cm+f0rIi~-z-z*zIniy zOy}?X2G|Z<5>WpY$WEqD*%9|;&VQrc?RX!HSa$GxAgB0Uo`1Ip_x{eI+3}w_v~^8t zE*1L9hXK7JqvYD@*~&V&Avn&$Jg0m&my>x(=N0gEoap+Uyt&TtYY&4(@IN2) zEheUQyt+3(;tThV$$j&U)NBkdU+E>rH|4yGE37efi2K<;^F3>D9Qgga*mH7YytV35 zYfi?-Jzl<35Q9g8V_grI%fgA;!s%_8|0&;ig+)g(VwXbtc{DE{xdS*|#gmD~I69-w zZo$@l;-E3*2PQSa$wyvZRL177bALbV?f?%Si@&pJCo=ojXs?e|1!u;CkL9ILna;;z zGq?2C=w2Kq8}xk6{Li8NCFEvG!7V?0T@Sf7i2o#Wa|5;Z1Lg(T?}{Ln4UqXp>=t0{ z{NTl+e*6d>S0uk~Zv)OG^sp`^v6@^T^pFYO_t7sQ?vhN9YY6V853uEIx-kK2=%)0? zF9UdDvN?FCO#s(Y30^Bb8wKS~XiF@{GxvXpo!h{7G5udAweS3z0uI&#UX{=bQsU1i z!hMtJ0=qkbd41-{owIQx?Z!YXOPS+-64xs6`8<0~HOL*{cbWZL;7m^;#PY>U^o)7D zo2!BkU8P@}4*&TGvVDQ&Rr*k!{`qqHo|D+mr=45GI5YBmn7AB-@7<+u>Un>@6Z#fV zm%nH4<0Xgw{#4$6m9C zH%eoEUuxduCwcYYbogTy%6$skZ^P#ao*tF|$>Y>~>?dw-8F}&<&&C$J*}Qwk&a{^r{R22>50G)XgT|uLul@b2 z~j7JAC7sJl~=x3|f|YV6N9 zXzwiFr+GD+4Y@red`-jnh6~1Hh1(OwCyV)Z81?cX1lQ8A7cLPS4!@s(K`;7Tj5u6} z{fyvl+81DNIz~cw^AL775npF(LKt~=G83We8f)FHx$!dl$Wp}dL~8d-*ond3S(kWC z6k92guLk%tL(U{&%#&F!L!_RZJvW|azPC%A+1^vJv08jC2+lmi-fq-SU*V6_=f*v< zH^#%N5cWm^+tJKjF7PI6Kp*{9>`t;gh)-r`3gb%)JsqV^ZEq^}1{1GFQnUD%J%+mn ze{K>ZleK3rd>Thy+xt_TNhP)LCv2RPyU%3Yv%}+3SNL7StKGT$R3*@-BmlvnQ#?114^Pb?Fj`|NHx5XhdPq4Og zJyBCd*jv6~+%;f(n%pOMfNLt&$!Ee{JMW&P;@g*A9-NGg<7nfE^g>(T3rSwsy{*g2b&be47lM9$G!T#PeD}KI~X-VA25b=0g5^3$AzI-%ZMy=fBGra(^&=$j&WswkYkB{-O#Jiz%! zoSDV=OW@pCvAqZQe}(s$(!WcJZqreezNeFOww|9lZSFs{i0LBft6SJ(&T%gj@tz8M zeLXByUnBC_){c?WsYCRuj@+*S9LA9cs_VbZuj2b+4Seh`^*9l}f9KVe>deNPZXo>8 zeSbb(dVuXsX5M|d|C@LFuhiMOwqBmfNUq1S9-RKC*!x&=X6q+WM@!h7z7`I2A`VmF z^XufuJLJxB?sXQdx&zBsJ$` zrWqdAPKS=|0(@x*8!-o)LJg z#Q2^UldtQ9;Oh3$iyyDjkIfzqPBxABU~AXdgYS#{GV~57--ET9iTn3y3;)9E|H=Fr zM*TV+Honf<*R==Km6~g^JMyY0`U{r3ekTX)ZQ)LWhY|Ln;vlziWZ-{Ym{ zIUNl=4!ZSF<8b47(?7@i1?ay^9!G=h0g)L){ij5S*>+Ep0DLlw zk0v)>aOmn`<@DG%5b?B;dT@Kmz?li6;|Ji|Tw--{Go`QoiM-^Xu7`5g$<{(|+Y0Wj zV&Aj%*TJKR;$KYgBPMZK3XDDp$lZ9{?cczb(_2&OXK;Z0ZN!on{(moaz9zQ0$z#)% z1AX%`d9<274x%s4*7O&Ak2<-VUOqAi9@|)d`Dn||^n;I6X=^bt>;N7-l5frs3)VPxiHJ(8|KR}XN~B5iv5$wXDs&KyRg|#;&^@n|NJJPR+fQRE9g^E zsXKQjXskt!%|fiRV6(RLF<-uqdP6HmrV$64?C)E;8n~vqUZ+c2Ub9(=xFMq$wq!{eg4Ty0C{QkwezvS5e zrk&weMXyetNj@v!d~L*dFCbrOaki)MZ$0M419CJ8zU6@bD+OEUM=b2^q>rET-k*)U z4S{S4(KP|wizGeaEsu*%pBtA1^hvj8<3#xLnOBE1yxRPj3p^VK<2{_s$^6BFjV@ASk5P*c!I6@PHH{b39QNyU>_KL;j_@r5^M?Nj%HRDp zAN;5bj?QMSIX|DkPE)C`NnS2~gpU63GcqwQ#cw%xzP=}K>&--;le2NM!wotcBktdE z7qy2!1BhX9_NfxmuiW`BnDdL&F^+df3Y6 zxDDM;y`DW$a!^p}^zXx%{3i8MhB(b4cjiey$&NaH2l3oz+@@oS^62($%uJr^ znww{Xjawe};bEt%#Fn11&m^9iq-QjddWjD1T%nIq1M=Z)rXn`$fpOb_zWTcKcz5QK z^u1qz%{t=L4)z)_S6L9}d+;HhSBKYu(Mb52-HWpx?4+hI?V)D^&JX1Kci_o9a&MCG z?zX36j@&c9A(puCa~X7%l3dIe4!d~V_$mBa4*90^dArAr8R$DLz=58?^FI7d54moX znNHqzW3HM)S1YbL{W>0pf0@}wHweE?CNpySNb04&&=*-hQuwC*gQcFKY*`Uq&D(-xpcbmA8h+S>e%i1V+U~K zePB1q*)BrJ67WqQJ+v#`X zRroj==dU1teSk%N=F0X;FnSGolHL-2XN*?=s!emHIcsI z{ObjuCLw2c8DCcB&+T`DuWlYRX7l7#W|Lrf5cM-ia+M~)*T00T3#3PFgm1lx-5ig@ z9|?ZWc5RW_$D031>eR{ESXg|T27Fh5V{X3>&g_?-;nsDmN-lmL5YOQNyq{36h}boK zn}Jn7-fODg&2u6Txx8GY!?#TeaD0p4+!sDS6pp&{IyQ!U1n3$peA0cJzbn1!I_yk< z&gjIjnP3!0?i)Kd2=z7BTeEflnT6ha92o4Pt_2eNLvTG8@;{1uO~A1PQo~MHUh(4; ze7oq$#P$4$h?qO$Pi}In6k^OOoVy2}T|xadN3B+p-f#9k2*HCBlE)S}=X?tqr@&T4 z`5xWMynP(NVLLyIIao-4v!Tb_XPr9zy##|tz|F1q10F}@&UfoJ?)Grn!CrDCfUkOP zInKov8@i|S+4MUP{Vwa};Q+C03A`3TX0Yey7g9@)$mMOq@jG76`3?2Ho-utQJbD0r zec6sC_4yxck;`{EAUF7ey? zWaerH___mo=Y?0cKZX0JS+lkl>Y0sNTuUE%$i3p^bE9V5{19W_BOEFl;ycRD`~VyZ zh^(!>FR{b}HwF`jr@(qA_IkpH#pI63+tVrxtXrdgvPm760iQO7=xs|x?^4X?NS}Ld zDfs7OSBe-|%Y6&OLrX-w)DlYkTB- z>Irp}4g;4b(gVh!UeW@Wg2=&C&z{~g#B2tzI1Ij)V_lSn z?{g&-C->8GkJczx8cBo$#}SS95x1KDFHiCJwNse4;zG0OBPE$*2HJzN>jPJmRXGp78<+G`-fp;5;&2=<3K{h z@S9h^s?X*>grU>PmH^&Ad9k}a9Z!>&Wxe~+$t(tzHw80W-}4gocN4ovUMwF;?(>J> z>k_H`3t_}i0OxDM|G~_|5!Azd>0=iI`h)qOA%y>p#jeQ>qpwv0cs_@9m&3bH-;?-E z2VDF6UbA<5@`Q-#M^APL{G2QIUQciTxwhv&?*{Y^-4B}og5EMrYTL=A!@V|-^|c#y zbu^57(Vg+f4+G9Lfv&j3IIm#v8ae3eu=gj7c@NII{U1Z*&+Y5jDu4q!aYrndJ6h%4 zylOzs-Cjvuifvv8+%M=hM; z`n-U9TXn92t$m&!T~Md5NPkoLu;z_I_#M_6-Ae?lJ_1*U$nQAih4U)=|JVHJ3vihS zf3tgdhVr!s@^n`2Gn30pF6P8tP!4#;#Gg%GUn#+F&3qwvJJz$`-}`O|<&UiofxXS1 zy+q($FE6)6nWKin{|Dq;Kj|Y6q#lkTw~=Mf?3MEP+llo*&ASUt_C-&6xr2{D?=jqC zdujjI01U!;eQ6hZL09jNdXY`re~r55A)oP4Z;u3%Q0L$F@Km|^;b&_N zuojtq&qn>*n*Zzr4%5VD8p(?icKJT4LEmC$h+^Xp!1#*<#|?D^w->EREkw;9F)9=7gW zZ}HQu1&wWhZD-coG4{3NUQG-|EDveZ>1fXPH^nd8dj`8ZQ3FoyMPvT}KW7E3IX!=R zINSO;uYWpO9m7k{eg*N3L=G~7$Av?1>j83d!t>)NoLMJzJtLq$J@RleA1cWidw(M! z->=|1!S+50;m=gWI1_t&0{VfSdrZDYkT`8Utf#&A_Z6iP-(|);6Z5p>ZFu_TbR}i~ zSxN34@;Kn+Y#ave6qCDL_jLX_?CE=asV!Tdi}ySMct?P5A29y|fO9%ekLvX2zlZ4a zd%e87bM-x+Z2b{>Sq8atoNPBw_fh7pKKv^pIgTNB(y9=(>~z|gmDs--;LBs_t9r)f zZwL5j>#xIp)-b-eUn6%H;y%b6PaCWTc4d9s5*ExXaPjTjl5WY?)k4^;mU^-W!51eEE zXTok#@Zz!8KfXtN4Up@fVdu2)<|@{Zj&RMS-eXc%LGmab@5N~W{c#`e!^Z)+`B7qd z$UYT;b_YQJPuMT`68*i4U=3+q3aH>MLUi zPJbtum~2DX`4#qtK>i82@{ql)5U`5K{x%iZuVCF4mRg*~oD|{O`|`a}4so6WCg+L0 z+4G;th#`~YAQIy0?)C9}~1aAKN^dZ*Lxyujs=^wvUh*6G!;JYs$XjNgRLTEJfSbY4>8a_i5C zZ(CrnUN{;H7+2tTsgreMSK*sm3pMV7@4Zm>vqIGRaokz%>{9yT-v7&3(&M~;_STE~ zX@Kq8!14lea@^y%)7=3!cQQ{6$??en_2cCIaWnGfuleuOQ4evb+xAY-mnc|EgY$K% zqjwnJ*iPpN$YvtvZ0(81=NsNV>dyINeDM8S#JR=8yr|T%o~aJ(yF>RwkuT@nBT=wF z)Z^!AyyvEm1H-6!vlT;lKV9tD-U!4zoO!v1I^XK;n~ZyE2v5F8o&Jhg*L(fLWOI6V zhdUo?OaeXmQ3JU`oJ$x+e>Rw=GyD~8@UQutu&8xU(2*ecAP(~(#e zkot{_y{O>dJIvPt<}wd*6+!Ob@>rk6`^Zo^qs8f4bJWxx;`>L48kmomThsq|)c>Av zWfXC2EAcu%7om@C^EfwK)xkf+JuLo%;d zwS!M3@TVz%f1d_iazV#Q$mJ0JoIuYmBz*Aw^WP&u&rJ9jDMV~POa3c>Gkb;W#gVV9 z@@~3^!~QMAbdm8_^m>VJ-`+1#*6kg}9o7tSZt;4c?Kc9qD?w-Z5cga)>}M07k}@B5 z_9JqwviP){wPgGDOlJ>YC^FA?p8s41-hM26&kkFey*f3?S-`Xe@QH&QB?9lOK;JXT zlgYFeJ@&3oUc_E8;Evx9T=HRWCeCifm>YKXAch_uk5+o`syku8eVPgNxtjGC4tDe7 zd|eM4({b6$m95PRBM#elPf6!#Z%MqrgwLzxo3sn;G!#rscAoHeIchu}YpN?`BTL>)_5=8MM=<%{8fDUZK0QTz z3Bk9l;D27oYyuXG1A3F`h=JPuRql*AVQ^uz#A9deDM|Tff~l=HW~~f|{)z$lo8$GM zxzKY1@lTaL(w91`q1RRN@_PmP7eM}ZVj7=)t`_84fJcc1+kyf1%}0Bhc)fKC?9UEy zAHPW(QP59D2izg1?+`ik40UoO#GPky?pfj4ck6dpTZ^IRN9l)d-;L|No^Vub9uv&^ z5z96n?*G%nzT|Y;I1{`XMqY#$Ju0K~f?oc!dHCd#Z&W+i3%RQ*c1L*o-4MrPe!H6d zm(Ul@>o*G}R+9Dhg#v*qOaJHWRjIrcs{HG=(Q3G1mTb^i>EJ_#dF zV>~>Zy*FTgk;HKl`hSp~<>Z3Kwy;+QIa*3z8o|$2=o8uD%g6v7Nneo1$LJ%O1NgB- z^7LbXUiIxRa?}`pWDkh9C-znXueoBQ6?j$_F?8~JSV{JeGgwa$@C~Cjb^e*hL)$wX zqOVs1Zry?XINB=U@k{j%ql~S+FLRZ#`B3A5#Aj>s!JDPT@O$=bTel~XSO~rS%`67mp=DgW57>v!mZAN{i0G=Zavia746rNzU<3>{u23_;oTjvEJo;D zM(&waH22c=?v_@fbVBva{>8P4lyMbuDY`}z6(CC3ZuuWtj%+HoVK;q@NYPC zQ<$+v4d|yEVe6#C*pc5K>3H@h_UeuQf#YU>q0~`VerNp@VBcij(*gM_LOzsd&g&uH z34vV&@bU@vw}RvIq#rCN-*1XNr*9d1#Aa||hSv|BjE!?(Gcx%S&H0EN7RCB^T(2fQ zThAsz4!c12b>LJBaaN{3e+2mB&LxLz6Yf_C!GW*6oEDQhae8d5LVFdVXAt{Fe#xiF z6vx>|j3YhsI-2i&$c^eD;rJD;ml5RdpJC*}^ff02Ii&wMJ3}O{KGJ)JV!fjDP$xf|*vCbDAM&1<$MKlt zI*a$ahdZz1YmSRp{8Q@hs`zB`B|>mRb=kZm<(hy4S)_)X+;WctBZ+Bk&sX~gj_!<& z`^l+x^!KRX*dA*Ofm>Db)Z{mz=M3{=u=P{Sm(8Qd-vQXWKCPr*pF{4pv$jvseqv8| zWceLzGi*Cu-Gt9>&BpzVZ&3hm*3!pY$j4mZ83Q&e1@It+#4_Cc5ucr{U6gqTdQJ!E zaB?;#k(j;{9;uAZQ$qJVjW394aT3B?L9(yjr zw;xcihhXm(bG?{ja`pt1`vChL$gP=TzmC*FQ=B^unIyuQ1K6*?d(8CcQ<3XAJq|9B z+}@@B!^Gn^a^MVo$pByKLZ$}l_7%u?JSjq|bgI(>9!>Uwc5fJ`>%xC398BIccl9HfgmF}?F0X-lG?SAC$qX6uDU+g_5V!uy2MFZ+5 ziqx^mS0Z=kL;qgj`Iz^D?Hz;G0y@FUj z6YLhUhTfC>I$buFk{)UA`w-{V0K2vq4ZUwQ^mK%eAA&~L-B+M0UCus+7gp6RhCA7zdSf2&YFyl}G{bA4Jcc&;Jz&(6fsmx#iCHIi}l zCSSJ%+_&?+yGV77V7(qEe_{f^WWb{)upA)V{|NT_A)mj)-ed7U3iyy%cv+vgMwjoH zKJ@K7;*%=?L-onc$BW#z0d;S3%b2&HiT4AMOCfie$xfiYUDC^XlTWdT^FrjZp6uH> zds-sK@8tU~7ULWRJc_{PR*$RCPvL)SN`I2zd<1f#nsCC&**F$43MLr!FGrA^_8mgmx;@2&5B6RI zI+IG@>EQ7!HZgq-7&MT2v$O6w%$i6hJ}(r0b)}A;VkfHfDyPTBk-+COp0#Y?vSVGz{Rc?%O;WaB^$?$&rtU_iMrnlZk|w zzXoRK$isZX|NNZSv3rer>`ZCF+}`zB$~gN1;}y6k769LGg`dTcj{>Zr?!@9LYV>3J z_YHh$2$_dC|10-%!S=WC!P#jfJ;>JHA+M4n?lq9h4fzGgc@LhS0nFMkCu?ZO&c)@v z&F%SKa{YYepP$9y(|KT&2;)Di&u8bElh47;lhl!%c2-J0e+JjvA=W(bZzs-_23Fex zungzj8D=G-$0_GSJjRe0I#1w_TOSQ9_6oN~!}n)$M_&a#l_lPtw3&k3>47>aBeh_9 zXYucqehl$FX)@o_hqa7C!0S*zO$ky?(p(_Kx~OePb1FD0_)byb9CgUpyW?=XAlmiqz}t+_6%*R?90yIlwA2|KJ{|p_EW+3C+vai zfZz9=_XxqOa>AwE>&O z?javNg#%SX#FYa*>QsO)W+T)S6Fxl=`zH4e-pA4hcdi^c;oh6dnf}0KZ3rH;AcrpU zd&!-D(O6Y-(Fi?d5OeCE`x1SmG4ajk`Rru(!H0z*a494?PR*Y5PC$;FE;p8!da<=V z9zL_BE+Ykei-hO?6tB-Z{Vy5^Nsk>vzc-*iJw`6}!k2LT9(dKO_mcs>I9vU|x3SFE zYV?nK@Nr{^yDcI3_z5t5U+Ul*_-DHMf*S=9S1Hf7?M;<<+fetrl7q_3rOCP{8tj~x zUerc@r@Ssa=}1oABp$KAZKoq>d<7UKLL4Q*tv_&P4&q%#-QlsZhnq&k7KR6bsP84pEQhH_k!kgg)46Vp!gQY^hu7KOk3vN z-d_inBgu(&z@QZHj3YJlEqX#!#{VmPUIk7MMju+k?;NL7$4?~RwqC*N0Q_BxGjXZ= ztCx&X_4W#g*Y;O|_e0=Qd2li(;vW)%6X%J2Mfmej{GX1T6_UIambvLqg?K)Z{?nCw zydWIjAhmj1F#kmEr`*i1>9i*vu$VzxEybU80l!y0Kk?7cPo)RC)gJ;lJk0tA{Y)YK zp$hKiO^7Q(01r*y8^Qs5FD*6qzxM!)H%OhBOk6KsTXRnh;c0vLotAu9jQ-F8@6kk8 zM9;_Zg0It;E(D+2BF6j3!B?#NMP3|D(BIw`Uz|Q0QxdQK?18_q|4x)XypUWd4V>-F zw?5;)((ShY`V)iv~`c`;Q7hL;U z?k%%BlX<%+{EqMOAr9jGl`*TY;lYd59tK6E-&8{F)q~A1y>G+8@b3$$6{oWSeE%jy zZkM{+?j@qZgCu8N(P>A-8FXZ#elja<6rQGlB_3|IU$pOUpBzpO2;mnT#ea-2d z2R%ha)~&noC!8(f^>erP0kJ)RoSmVLCCE<{oVhHqRtUgsBl}ZB=~L?v|2X05S>m@C zc{>H2W!N*fAkJT?GyLf`2CY})but?yx64H&0}k6(d5F`0*` zJUXEBGfL*&xQ z>DZC6oejasky0;u&Y!0bh|9@L@b;<$)fRm4_Ux#13S+)SQ(%VEA&hMj93X8wWBz$CBaU?1Mw(^kWPcshY+ z?*WS~;?E`4@L1aVDvZ16Rj)oS^K1mcXNi0-d=6f{4PHcp-E#q$yLf(*oNpE1Y`-Dy zn}p!WX4)#vDb@uy(#f8MJy*ETVL+p zuYgl8#NExi2a9>U_%1{ZU6WYN@7Fv`ZS7B~)nV{AHR8<=AFCmzw=nzahp6 zf>mVjXDecPmG;ktp~LjF7hf_8p8J`n_n~W)$k-Wsz7cF=vM+W3HmxMTi_n+uFuzZ5 zPdz{R@1Lzgtp8gI|9>guX=?KR5by8~!0j8@jKaG4koI3={Jp?~h9UCwi^O$HVn2hk zCB0s+{u~bYen`X~Jr4GFgY!wyV~XN#`w4kk10OO$e;2GZq0QH%4pq0C9|CSQ7;|s= zey9aqPG$u2{)+T}+v^D3!^n@@-21|d2d{tYc}sf9aGpy_o4e_6X86AmhgOoO2cfqZ z?EDO@63hMMd~@S%!PTw(h!}n+f1X0;1L-rx_&tzO`bIMN@~!l#4T!rD_BS&Aal*x4 z;7?!Z%L2PU30BUw8#j4)xb<*i*R8qnBXX-Vxn=7W*wZudTy5AK;N`=e*D;pZFG5V; z3Q-GlBUcJx4-T$lP%+4JG4g&-76p{I?1kCcuH-nvVPjTiW26R>l zz&{S~OcNrPcReghNN%0}UGQN!bR3bqB|(ivqHj*_Q|2wJ=zJYMrSUMF%{b1>w?$Iu zJ4+0mzO&%ghxEIu)YMbTtsyrW3BJiun>Dc>LB0o^zANk@>!I^h7`5o+{qdf}WqTKh z^;-d$+djhlyQkTELH37^z+)ok)#Q6`32UIY=i5ZueHtQ%ru$`{7`QtNvY#dwyOjm2 zIOwm7*<&}eC!E5Wk0CRc+}I}h+y>=8D zM(o_48;i2$R&ng^^`>OJ7fJ?j-03(YShzJCdr42y^`9Xh6C6+Kao1$lN*+@{?!Lz# z)zg%i>skiH{5fMzBf3;3)VzW8v|l`&dxwbSf#=_|0KUay-|8eBct~6Cg~*HgxWUu8 zPU6br<$W~kr!#E9?7wQsbaugFROumVCl2ya5&fwHa9u{dvBBr7>;Wd5g1Gd+{&w(K z&j-yTd;QLrxeVW{%UeD#g@GXJ(cC&s*QMhk`H~zy^-K;Hi^yYm={K$a567D?gn=* zcwF5q_g0XetsyWr*^A&v6U6=we5``_E GVQ`|H)O=2yoj|TW42WBOi4)*MV=q@G z-#8$a4ajp6_JnugXMHbzwG(QdUHm&QF_`>2^z8!fmeaVS>PRn1OAe1>Ep1}oc{d;) z(^VB1z9Dy%t>u926s*hFc{f*WT z@p%1NKyO}w_=*$bUvXabPeh$vlwRicE|AO51lJV&J}Ja+G$)&v{mQ*h^6=@!{rMsK zRTs&H)1l*}fP3pE*AQ;FrqW~ICmt6)A532}k+b)7sKLd+b{E#uvi@w{e^N5;$*iqZ zVdU<8@@=y81*b0xa_HuT1M>90J;A#PW7)%;jg#6QC$+tnx!s1d3k2i7@H2KmZr?}k z?#CbJOF5~L;m$9?qZD+eA_mPp44sbD9wvJj>lDEupT`qBdtJ_zhEH1=b2#R=G;CMH ze1yk2XV)LshN#_G;&)w<`7Fep)ZN2v1?r&@WG{hNEksvZ>FLgw9&&b%V0%z<))(h` zikv&^#&n`LBhPgT=-p1_iftekw6`<9CF~xTh;(P6ONTzT~}LMq+pZKd%s{)55`Y$XRsYwvM$^ z1Ub13Kl*`3rO==1NliIFcC${3!=G1iwlw=#HjI0?_l5X4Nam&^En;xG+;|3dOE9OY z(UY!A4xNk}JHpO7{JBg{)|GfnW-oixWMIA*{-lGw(#*#uaQb06ZYsX8Hhl;1lhcLcHJhwuBO zuZ4qOE%Bb))0>CyCB1JpwKqiWFkScO<%xA(>g>gsJ{24{)9(-9%Sh(J^h8EIbV6>A zNgVqT=TVPC-(i0;^Y7dL1Nkflon58QROSlLjS{JCc5chJ}OwXQi#*;Uj ztDx)q5PkL{Vk>~X^{DlqiAjIdOmpVw7uKce*#h57N!&}N$Jl;!;o~b3cUe!)Wc7*1 zSar?K&=?PP!U4bi#A<-#{0-vU ziM{Pp=#3FjLr(7o-g)^@gH@2bH2g-cCjI0$@O%~GT*5m440(B9dc>c^{1>ms?DJ|c zE^(;u-Lq+FGrX6(jgTvic-}!QS==|ha~xxQ#wtsHZ4X$>D0OZ{?eOg7m0Y+pFB(^ie=m~N{Vg~*-NQ`x-MqI~+x?}Mos1jHAfBz>x2@Z= zu?N2=Ka@I6&mQ9Brh`jw66XTKlWJHi5RgZAE@)gsJ#Vunpq(uN9(KMUYuz5oWFT_b9?@H12Hwj zUF_B~hRE+NaOIZg*Hrd~(X8D^!ryhMyWa4_>D6%>&ZT4B>Uz-pcf|3TS9hzWZ|a$k zgjc#AG#|iNZU*GkQXJ2&7=LE*WR9}>`8++L{hW5j<| z`t`d3->6@B@s<*q4d8icyzfMf#0TyNiDzNq$8hrcI`+TC_r-1Ykl2uIgW8WlezyQu z%RpBdo~_+w8ZSBJs*f1IYc||bNy@+1j>0Wnl?;x-}3(RLAFA2OoJr^|JCcS&5 zSG$`SPfGxBka(6P*0ErxhZk>gu{|ZA=c)Y~jJq3k#0{tsl@FS~A>X=gKNGo<5Ba&_ z^}}<(=?$;%x^v~3`$kerw%0GfMmqR@-;3Spv9SjGb9%@{gCD>1zWBqd!@+ou2yFH2 zA;D1B|21cyitBv`OcI|Kczf=AS-GR$3&A~;ErniG2t8^Ozr9N1y|eU0r$|p z5P7UXe4QUF=<9ajpWDli?*TX0aiGVkR1&X~s}itgdr^Dp;^o+# z`6VDu+iMN}zlXg;QU~)q{_a97ZLmLyxcnIqv$JDk5%{n=z>l6G`i`FU=Tk!X?c`eX zu6FMuh{iy2;&i?;3{^8v@L4JTP|Ek>}8MpR*Ku>ae(V?g$a`}nZ zb8>{iM?1Sa4Evp5kfW=xcb9#_`LPZB{ZP)`5>5?7&YawQ%DgG{bi?x{jnulGTkO5> zC!b#r=rs}GR|c`|>;{dEoL+KkMu5JcbLn65n?LAmbL3?=V*i|&%nSHtcIToAo^H*J z>5=bRh;>cCU4A&=+!pkp2LbC&pN;w9(=fRgMh5gFC*M$dT6yGSC^63fY|0|WdmcYN zgP(7Ed8tADvt1sjJcDpKW{8}Wp`R1Lor(b%I6q%S%w;7ex9-L}+?yfybNm2oA2h{r<8jl1MHtiJ~HupWw`h{%d3I6ne!&VZjRLQn3v$L`dl!Knja0U%Slhtb79R7 zg~6%x%=KLOyp}PX4C8ynx6|AEmh<=ReKPI0v*rr_e}MikL+~cEr*9O$6->`P;{O@* zngz1=B_}3hPg>ZB?)9^MSZf-_U2W$!fairIced6AzrXD9e6n?WJ{8}dul)Px0qnQo z_izIG{JY1S_^=rpYqRkuv-F}kct?4L`&I6;tnlM~aC9l}ye8!92LZp+HN{@sfV?jh zT&i>b6Z)71SpJCIjC~2tekXTwT==?2^0^oDg!pqE>;FdNpSLHd!2)KQeQ)pZ#k-xwDtt8tKKmfn0bm)P2F;?R@aZ)zU|zz;-$0<1zd?Nt`D0{Biod z5pgFbHl}Ng*CT!X+p+&Db<~z~zT8g%y`&N`P63RP3T|(sKl~v*yP5Rk>o15`?KVM- zZ%W;`y;cEr5;b7W$-D6@)a8}EDH&~Dwv=ppjD2|AYm+xEbsuyE9! zO$NN?l0zGK7n~r5yAjJN!Mmo+-T6Ub{=P3GbguMx(ueiIb_rhb_Zwd}3C1)do2MIhoCgYxFaxYDQPw{}sPRUcPfIPahVU3l+rB_~XM}>91 z9s2A$VfY<%_CWyWXJTy|Yq>CDUFYSYJ#fs5no3R#!;>S+1k(gKmll}3LB1U39dORO ztM(y(W64+b?_2O@fnc#&YIi;SX%xWebF6`ip1)tBb|WFCcVN3L&do=i_8~9dr9Yp` zU9NuIqTCkPFUVe8J4Ajjpw@Pv4{Qf^ud-kCk{EUn(~WX}Ia>?hQ~eM=n9N+@csjst zHt!yC`icPm*Whn^;@*xqa{F%F!I~)K)oE?ul^gL)^v*Sh-XqXihrC&Xb1jkQlahPW zU6=dL&WpxH@HG)?cn^AFw*Y)*dHN!I?>m9J*+|Sj5C=Xkga5bqp5z7n!Omv!z8Nwj zcL`BXuLbn}ZRiz6kh2zE+^Vf6rO3|>9;Z#F zztlt&4-eb7rvq_`0sXTfGYURck#i=~fV}Jf5*V4hJ&&MgFSyVf_}oT3HR*31kE8pD zN7MlS`gxo)y;G$}+xuPMyAO6o3zoM2CEn*_y(sV8Gq5|~!_?&LiOU*z{`T*mKYKNo zN%-CeyuRUm#~k5%X|Io&)`jq;TmX0fZSncrM%aB9*hL_Jj*5?$BtM5e?rj7QekR71 z$cg6U&MU+-6)^ldz(2Fxk$iZcIN!wBL+&o;Yc%G~z1Q>dH-$Jm+ljjE%m8w! zJMcIrvDw~j;n^53ueR@=x*nI@x{a3wqoUwJ8HqOrFnbTRpUKOeo_F)*!2b*4mk9aT zk66|T2b`RZU*VqGB>8O$8*YD-)Mt3)p#^lt2OsZv9A8XZcJ62Z?oOW@>v=LKc<&{U z`0IPQv9saPyLRHf$s+fUt#78EkAZtU>ny!pG&^n&wczxD-69oqy|Er9vh;r%TrIh7x+@v z%eC6s2rROZtK}%SLhyC6ExlZ1$C)Kq`+)1sJdEABpmB4Em|J>0`2u~tf_%>&ll*Q0 zubVRW&PG3Qd_Q@94*kd0e?Tm6yZ|4Q8{^##TUj@sLEpDx*JSM}j~HGN&Q7E6Td-c< z!#g|kw*Wotu2;{`wLSk?g*J7~<_&>EBi^lxa4+uSx39`o4RN*$IKCA=4M2}O0Dd=t zpYNb|qyR4KJYJi=n*qNIsGV=T*mO_lIl}0F9gvUy0hqe;7hq!%d&5`6?jCSE@8!Eb z@Q;8P5<+${zg?V;u*QV}-!^Xsto4T8ZNRlL_HNSO<6&^Azua|G1LAAIp1dD9_|@yR ziy5!^eZsTxRO;dr=BKD|&yV#sYq36U^kFUda`t|NygyHwd%*K`tbHv!h#qh!s;&mK zlSpFF{ZR8wUeCD6zTA-dQXr)MU2nWA%cs+SAeNb6{o!Z8b&`i zUm7BBGstV_+bQ&hZ^)nfz$_vBIVjkVr@ayZIdnR{$KD}1zTq zqk zr={fhp%?h&^p0kp+`El&7~`vwlVkyR%~EoDd4P^_lJ^n9^GG4;cM$V*0J6mq!$9!0 zL>Tp|{`&KZQcHD&-}hfoNB`0zKD+f6h&KUx!<$~t9(XbJAvX_uIauOx^OpDS?6|Q! za@c;K6aRB`MQuuMe<79Qg z)Xq#6T%3;b@Ut)M^~0Ydz;^?<@>_tMlYd`ox25>$_G5T`YP{D&OfCuWE$QVsFJdVI z{1SLE?_wMuv5&5hn4Mi4H+i{f&hNa0jAtxz7)!XLXTzFb04_Y*GUq48l0|V?mU>!dt&WU&rWJb&bynA*#`rT2i z?~&S%0q)Kve-DZ87EffvotSHx#Xs9C7La>e-vOKZh`YV(^WeYSnLN0OIMwkunGL;s z74i6;d->UG-T|*Va_w`^<`H0<7#JnNpSq&+bz z=n0*{=cmBw>wjvnS(9Om2-tZ>kf3@dK<+;&)kO(YMU2?Td_2?K~tO z2C*Jjcs)pEL(Lm|ea6Xs${IOLoLh-M2Se0LD#YT>Uqf9c1Xmu>$Nu2{rGOZcWB+y5 ztetHRnUBcd%{cP~&pW+t+)TfZp%1QP+!u+XlW}8X=ue6~HeoKiz^}=W{ggSX%yXB( ztzB3vi}48g?R40L^dyM~HJ!h`iXDox<~~ z#5O58YWvS%_bb@HB5Stpo*#kd`^ax)sd2ZbV=u%zh5BuMH}2L7h`YDopz>p+Kl}t6 zk(sYQaOODtIZEDDl-#)U$>{4maH)yllpVQ#Cj5O@IO5KJ4c%^T<0bJ~*ZukX)O%TS zq%tw#?>*!uIee?&$z)}$P3dbF@ z!vmz>9mV=%;eJKfN+SF7Js#A)4_#P z?4NG`E5v)*#f)B&7`QZt4=JQy+(0k;6Y*W3on+YmoH;9xZ<3Y3y@p_%jF^-{{=dba zi#T^hFxUazEvajP z|KFfzq=Zi?z_mD@4`%CYhmXDRiHh1O9l-A-)PIik;N%BG|3mRL7UHfceBFaQ?~!|} z3HArU_qg<@kMO|h{L*?U_FH=2g>K)C_ru_@lW9#m$tl|=00)!36b9Fvu9q3>0Q1TL zxZ2r2`2E*Oz5{m(A65TG`Hr*oPVzg7yh4djit4}`UDv#W? zws%E(k-bNOFG$<}`bshBtrG(7*7lT(iZiJY+g|4WsrYI-?fC#PRG0dBz8d^<5MxL! zzJ4U}4uP$TUazX@)s1O>1LseBy~4Lxg>kif!S6mfz+L+nPUQ>(U)4PYnDq7dews0S z5`vd{{{LpaN@`Qje}TOBWSvg)axgl?nY#Q283O$FB93NWPOB2PN~rJZ!s`^Qy|S=3 z#Ooi9~wqIe`9E+Peo{ zq`#Q>^hE$>dj1-HStvf~{^Ah#SQV^4l6vUF9v?k~9@W{D^<--UrGGn_!SY@FalpQl z3mUHpe*S(j4~r*Za9q!x@P6<52sTCq#QD_I*G_7#6yunRT)vL814Hx;wHq{lLv-E} z?gz;p;@uSvy?wdF{&@hV)4a88A?kIe7k7N|M}7D~?78)&SX;>47iQ1yj+$#Gd~mY< zn40=t3CM2`*6m~Pu|a@cUzdBID|O=5j>`ED0`NUWTh#)-drZek;PU};dnCkuXXJk< z?!`BW8&dPH-_=aV7%vyLW{>?3Ds3D?zrGOvs-hmt@h&$Rea;8O>GmhW&xM}7Y0_^J zNPb&My}R=^uJ<@$Ym)=wvUPhBL4HNRndiIz{ER}G_ruUtDqwHDtmWoCyfp+i|5*EH z79UvFW6dwY{4I&QJoTi+y%Iivi<5cp{G;u!zYfYb=xxcF$(#}_v-Nka(xxp4G!R(?_&qZtr47UKd0nowbZNaedxX0nmv!j*3S~de&nGaef^j* z{sDVEk3o$B?J>>=VhA$OZ^ z^A*8pBl5A9d^-=k1~O0XObc>(4ePlzeE8n`W^}Uub&QQVdKOUgq4Ycq(d*iPAI+sE zV};+;^NIaY=@II~RWGhw@IQV)zTzPt?%XlxDk>a!;&H&q`D4le zOo~gtO%4Aq!Tu^w?jUnMRO+{o_~YBr_qWI=w>Bxn_jonLu-dch&gl41Vm&52%Zoe= z#oikD^$a$fV*Z-P@3G?ZVSKZ+09U_;+-!2j*%^fQU!_+kgU_{v<1@U!qjTrhvhP$6 z5o37IPBih~={PKQY^^c;OB&+)s*aN-#>1rczXL~Vz>nR?>6>1EZzg$kc5Q6$_2HHQ z`S<{9@$lz+Pu9-20luH%@6oD>y|lRV)4;z-;B*Vt=ucRCd zv37E>UMdsc`oMLV#MqYo^D1l{6W>&SJTEuvkjv#B?k%Km>sgzpVeL=z?mXM;O5MZ7 zCtI(b zd*Ie`po)+5>nS;*IOkwnZz&W zs~fu`{&2iYsvxf6sQK_xJ5Da-h@RLoz+aWGDj3&*?q(tQ_z_}!&-3*L@)VJ|Qk_Bb zeA2IM|Bl$T_hEvGz1wqvHS+xJ`Oj9Y)7&^e0=de`e5GSwG^utk>66oYOniIKt5YYV zWBU;LkCInkd9mr)Q1g;r|1r5^)b|Up8NvA8fm}KCs}%uw`MST8xUQillDZo6I8hV) zKZX2u0#D!Z?$KnhmtXYwHtfAC@>CbT-VA}u7Xh(N_j>0^sW)e1B>DCo{oUqz7-Fx2 zc$y>kr&)W2(Faqojz2{2Y03V5PVRQ6*Nqp*9rqqI<_^*CI)sSTbVR~l4{+dP`1Y#w zrv!{+9DWaYggUF`{3Tz~3x}6Uj%GqfZRt^FV*<{{kX*g-5}Yd;qPBnLw^AyHuNmNz zDxi;l1Y5~M=yK=WSf1ZW?%l@8UMwR*@ar0T-SL3^p~(O1h(Aw2kH`}uXX=B^r+9wo z{s-hwefgd3OLEQR@&?#=3w#CL!mR*iH3Rxc8FHZ&@IJtNB?{pE45^KX$jK1syyo>@ z_0^w$4B17n@o5;`{{q;J48Zk-x0WK{F7@pt4v4{*vG=_Jm>flX%faL7<-;E#KcfF2MY?}OlOE6Ua*=KZ8cI{8rJk$}I;=t-{Z3#esZ&vAZ-`RndI zpXf+0Iqw+|vpZws53HS%-hF-sI`&DuoF`|W2Ixy2u;y%JfSy6jLn@EMCNqrn{k_Mh zv|`)odf9Qj^h`bT8*D`b=h}(C$6ruC?)(a=k;roQ79_W-A(zL=hhw7OowxBvU>#rj zLu$ceQHV2kE(6b0=Qu!eZTm0#Ps7&#z0}1?~UWwH98#o5lE4&xhoF6faJdv3Xbdj6yd5djE)bO%cjm zg1`8`aDxJJ@@@bJd>?)SK9QJr-TRH0)Mbq81HR8%@!Tr`J+nUQ^{A|;5dTsK*fZOm znA;kxpJJ37<@KCT<@~S^{Uy!p|4e7o_@#rOfPod zj(a}^-wq(g39!*1pwCVW(Wl*cHx`4v*dlvJ`b=u@HW@MK1k8s^-hM`mu^ID8Io}BL zS`Kn{_|F1JEi_NNt!YYLpts0if@*tF*RzM#p zTQ!j-ih{DpA|k5^rL+`zU`GX178Tke!3WeTf)ES=0TH6cQesR5VyJWhq{SWRDneoc ziQlp3kT1u%?H}($XYS0L^*iUA`vYv`s{A6yFBP!EXJef|O7$Z$M-lVk2wh{ePlnp!A3a*tc*GcW6Xe+l+-W8|*97|*dCnJ=aq_IvVv*~KhtE2#$LU7oRsw~lzYVKw}> zC)LSb!n+DMFLqdohHj3NO=~mPv$CG2x~r*&?0gEa%0xU%0x@sXn0CY%gXl~lj>)7R zE&|5A@q4~I{9YHpdA;^U8p)fFJZ7Nxrv>VLki+C%^i;OiRdEyk*SNeCk`Mg@-_%JO z*FcKnI?1rVGS4JT_Q0neid}2?xP-7>3Jfyg%O=MIUSB_|58f+mqTZ|V4X~Q-s`=EX zL%@ro%{XSnhYfXJ;VKy$CCl zv2k03-f2zLjmWoheLs(KHN@TLX3vjn2ou>Wg|E+Pjf~cN?ia*_wBI%pvB;TY$m12_ z3|R|3>Cn|vb;NNvrr*?Do z-}I{BdmpkT@?D_Y^vK7a&8yVjAdFe6r$jlR8vJ+=_7bR%OfHu32Gxx}e~9And-C2v zx#6k6H)H7h6|KQ5R3rY}aKyOC)oN?{PRggeuxwV0Jt;C@X+HCS#rXj5o8J0}`oF2R ze!%`R$Zu4ACf7fLpKQO(otO1utSwhxaukycI#&^)vz^v$Hr2J+@#8$Yk8E$I+9;(v zWov-9?7WPRDIP=dJtgn^RPTrIF_C^TT>_cc?Onp}ExH2cAoo6pEx@n`AsEE*}l!^==omAe?I+x zj_vPv@!EQ5JnimDJ5#QByn}qyXg;R9UTUxXHbM8xv7bZrF1o&fuWhJDW;h*YZ>83! zydMknv8;D?-;}cUu)};fVJZ9atRhUh1vq`9;#nJk`=_q2E>r)Ae!ze0UY$zc1+pj4 zR^XAZ`OG8#>K*Sioc{0MN|k9vxSpk)ibRp}=i?Y(jlfjSwR5rH=q=x6pPw_R*R$~Z zTk^A3F=(zfzEcjE7?Hbiu2)3=9*Sc<^!@F8_?r6UbReJXqs*tOzNO%{^avg)&=|S{ z|4yif7Ub_NmrK^)((y3w@%(mhN-1i8E^*6V%G=wz{v*}7$!{l4wKW?jk)KJ>H}n(<(C4hr=36MfPf+{Mz~52G^)mEZJk?MNVIi`^ojuccFEGkN zY;~^B`T*m}gwGwYn*jZz6?>82;ABkaWa83~NQdnw5|7AQTtuHuQw}>on3#?>fqJsN z#>OM?zk|D*pH+W918*I0dtTQ$S94ul4t62W>%g7KS|9P~sRfG5P~_rTM66~9kOsdK z)(7}lWKtc@rspQ%l#JTG7_d7CdHqTAIKkz(C44?coKpnf2h*8ysx`CuC*k^%>yHiU z|9SA<&;JvxrO{Ehmt26=cxu{caQ-05V@R~eVlxhcrcX83kW^O@ql_q!mTHpJH= zKZNoj@BAD`zP8idbc#5g&++_4VD^Xh_uPQ)#fo8D_2XdR4C|BmEV?6XZ#J!^BL_!e z_f4wfwho_kvXex6%Yk>Da%5lVI0PMcQw^9s8E3n`;k}kxt3`nvbw%D^AlV(tC%e&S zmDqbw@%@7Mww`#qqwD9SKp!NydvA*J!6T^Ih3Xso8=8*{)L8-izKVQ&slKhy+!sXgy4D2lzuJi2Hkm5Tg}na*&atcg From fbb8bfa3e256aa9cc3e83e398704f2206d2fe9c3 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 06:09:02 +0200 Subject: [PATCH 79/92] The argmax and the draft's top probability over many blocks a row The verify window's head ended with sampler_greedy_kernel, one 1024-thread block a token over the 248,320 logits, and each draft pass with it and row_top_prob over the 40,525-token draft head, one block each. argmax_rows scans a row in up to 128 blocks (4,096 logits each), the row's last block merging their picks (the larger value, on equality the lower index: the one-block kernel's order); row_top_prob_split computes row_top_prob's 32 warps' sums in 8 blocks, the row's last block adding them in order. Both keep their counters and partials in per-row slots of one scratch, whatever a launch's row count. Bitwise: verify_parity (ties, NaN, -inf, 1-8 rows on one scratch), and the window hashes on fixed text equal the previous build's (IQ3_S, UD-Q4_K_XL; thinking and code text). CUPTI, IQ3_S thinking text: the head's argmax 84-86 -> 9.7 us a window, a draft pass's argmax and top probability 15.1 + 10.1 -> 5.6 + 4.3 us; ~105 us a round (~0.7%). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/mtp.hpp | 2 + include/strata/core/verify.hpp | 1 + include/strata/kernels/verify_kernels.hpp | 11 ++ src/core/mtp.cpp | 9 +- src/core/verify.cpp | 6 +- src/kernels/cuda/verify_kernels.cu | 137 ++++++++++++++++++++++ src/kernels/verify_parity.cpp | 62 ++++++++++ 7 files changed, 219 insertions(+), 9 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index a44cad8a8c4..a8ac53dc4c7 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -134,6 +134,8 @@ class MtpDrafter { int64_t window_ = 0; // attention over the last window_ cells (0 = every cell) int64_t prompt_len_ = 0; float* probs_ = nullptr; + uint8_t* arg_scratch_ = nullptr; ///< argmax_rows' and row_top_prob_split's partials and counters + uint8_t* top_scratch_ = nullptr; // device int32_t *tok_ = nullptr, *step_ = nullptr, *pos_ = nullptr, *row_ = nullptr, *ident_ = nullptr; float *Rin_ = nullptr, *R_ = nullptr, *emb_ = nullptr, *en_ = nullptr, *e2_ = nullptr, *hn_ = nullptr, *h2_ = nullptr; diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 4129a79b5c0..37c4df1856c 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -314,6 +314,7 @@ class Verifier { float* hit_xs_ = nullptr; void* hit_scratch_ = nullptr; float *head_mixed_ = nullptr, *head_inj_ = nullptr, *head_logits_ = nullptr; + uint8_t* arg_scratch_ = nullptr; ///< argmax_rows' partials and counters float *sh_gate_ = nullptr, *sh_up_ = nullptr, *sh_g_ = nullptr; float* hist_snap_ = nullptr; // T * NG_HIST * NG_HC_DIM // the PLE block's projections (keys T * NG_HC_DIM, values T * n_embd) and the rest of the block's rows diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index bafc85afead..933198309cf 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -86,6 +86,17 @@ void gather_rows(const uint8_t* src, int64_t row_bytes, const int32_t* ids, int6 void map_ids(int32_t* ids, const int32_t* table, int n, void* stream); /// probs[t] = softmax(logits[t])[ids[t]] for n_rows rows of n_vocab (the probability of each row's argmax). void row_top_prob(const float* logits, int n_rows, int n_vocab, const int32_t* ids, float* probs, void* stream); +/// out[t] = the argmax of row t (n floats a row; ties to the lowest index, NaN never picked, 0 when no value is above +/// -inf): bitwise sample_tokens' greedy pick. Up to 128 blocks a row scan slices of it, the row's last block merges +/// their picks. `scratch`: argmax_rows_scratch_bytes(n_rows), zero before the first launch (each launch leaves it +/// so). Graph-capturable. +uint64_t argmax_rows_scratch_bytes(int n_rows); +void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream); +/// row_top_prob over 8 blocks a row, each computing 4 of its 32 warps' sums, the row's last block adding the 32 in +/// order: bitwise row_top_prob. `scratch`: row_top_prob_scratch_bytes(n_rows), zero before the first launch. +uint64_t row_top_prob_scratch_bytes(int n_rows); +void row_top_prob_split(const float* logits, int n_rows, int n_vocab, const int32_t* ids, float* probs, void* scratch, + void* stream); /// Dense-attention step records for `n` cells: [cell, cell+1, (cell+1)/4, cell+1] from cells[i] (device memory). void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream); /// A sliding attention window: for `n` step records (kStepCount ints each, n_kv at [1]) the selection becomes the diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 13885772a8c..f388be048c5 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -220,6 +220,8 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session x_bf16_ = b.take(N); out_ids_ = b.take(T + 4); probs_ = b.take(T + 4); + arg_scratch_ = b.take(strata::kernels::argmax_rows_scratch_bytes((int) T)); + top_scratch_ = b.take(strata::kernels::row_top_prob_scratch_bytes((int) T)); dummy_inj_ = b.take(HC); }; Bump count; @@ -417,11 +419,8 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri const bool sub = dhead_ != nullptr; const int64_t nv = sub ? n_dvocab_ : n_vocab_; native_mmvq(head_->type(), sub ? dhead_ : head_->weights(), xq_, head_logits_, (int) N, (int) nv, T, cs); - SamplerParams sp; - sp.greedy = true; - sp.temperature = 0.0f; - sample_tokens(head_logits_, T, (int) nv, nullptr, 0, sp, out_ids_, cs); - row_top_prob(head_logits_, T, (int) nv, out_ids_, probs_, cs); + argmax_rows(head_logits_, T, (int) nv, arg_scratch_, out_ids_, cs); + row_top_prob_split(head_logits_, T, (int) nv, out_ids_, probs_, top_scratch_, cs); if (sub) map_ids(out_ids_, dvocab_, T, cs); } catch (const std::exception& e) { err = std::string("mtp: ") + e.what(); diff --git a/src/core/verify.cpp b/src/core/verify.cpp index ae5d18d3944..19e6a48c197 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -273,6 +273,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sh_gate_ = b.take(T * (uint64_t) g.n_ff); sh_up_ = b.take(T * (uint64_t) g.n_ff); sh_g_ = b.take(T + 4); head_logits_ = b.take(T * (uint64_t) n_vocab_); + arg_scratch_ = b.take(strata::kernels::argmax_rows_scratch_bytes((int) T)); hist_snap_ = b.take(T * HS); const uint64_t PD = (uint64_t) strata::kernels::NG_HC_DIM; ple_key_ = b.take(T * PD); ple_val_ = b.take(T * N); ple_nkey_ = b.take(T * PD); @@ -959,10 +960,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { return false; } } - SamplerParams sp; - sp.greedy = true; - sp.temperature = 0.0f; - sample_tokens(head_logits_, T, (int) n_vocab_, nullptr, 0, sp, m_out_, cs); + argmax_rows(head_logits_, T, (int) n_vocab_, arg_scratch_, m_out_, cs); } if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // a plain copy kernel, here to host stamp(g.n_layers, 3, cs); diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index 03fe04fb929..c0386ffef89 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -7,6 +7,7 @@ #include +#include #include #include @@ -402,6 +403,142 @@ void row_top_prob(const float* logits, int n_rows, int n_vocab, const int32_t* i check("row_top_prob"); } +namespace { + +constexpr int kArgMaxBlocks = 128, kArgThreads = 256, kTopBlocks = 8; + +// the larger value, on equality the lower index: sampler_greedy_kernel's order +__device__ __forceinline__ void arg_take(float ov, int oi, float& bv, int& bi) { + if (ov > bv || (ov == bv && oi < bi)) { bv = ov; bi = oi; } +} + +// a block's (value, index) pick, in thread 0; `none` is the index of no candidate +__device__ __forceinline__ void arg_block(float& bv, int& bi, int none) { + __shared__ float sv[32]; + __shared__ int si[32]; + for (int off = 16; off > 0; off >>= 1) { + const float ov = __shfl_down_sync(0xffffffffu, bv, off); + const int oi = __shfl_down_sync(0xffffffffu, bi, off); + arg_take(ov, oi, bv, bi); + } + const int warp = (int) (threadIdx.x >> 5), lane = (int) (threadIdx.x & 31); + if (lane == 0) { sv[warp] = bv; si[warp] = bi; } + __syncthreads(); + if (warp == 0) { + const int nw = (int) (blockDim.x >> 5); + bv = lane < nw ? sv[lane] : __int_as_float(0xff800000); + bi = lane < nw ? si[lane] : none; + for (int off = 16; off > 0; off >>= 1) { + const float ov = __shfl_down_sync(0xffffffffu, bv, off); + const int oi = __shfl_down_sync(0xffffffffu, bi, off); + arg_take(ov, oi, bv, bi); + } + } +} + +// A row's scratch, whatever the launch's row count (one scratch serves windows of every size): its counter, then +// the blocks' values and indices. +struct ArgRow { + unsigned counter; + unsigned pad[3]; + float v[kArgMaxBlocks]; + int i[kArgMaxBlocks]; +}; + +// grid (blocks, rows): block b scans [b * per_block, +per_block) of its row +__global__ void argmax_rows_kernel(const float* __restrict__ logits, int n, int per_block, ArgRow* __restrict__ rows, + int32_t* out) { + const int row = (int) blockIdx.y, b = (int) blockIdx.x, nb = (int) gridDim.x; + unsigned* counter = &rows[row].counter; + float* pv = rows[row].v; + int* pi = rows[row].i; + const float* l = logits + (size_t) row * n; + const int lo = b * per_block, hi = min(n, lo + per_block); + float bv = __int_as_float(0xff800000); // -inf + int bi = n; + for (int v = lo + (int) threadIdx.x; v < hi; v += (int) blockDim.x) { + const float s = l[v]; + if (s > bv) { bv = s; bi = v; } + } + arg_block(bv, bi, n); + __shared__ bool last; + if (threadIdx.x == 0) { + pv[b] = bv; + pi[b] = bi; + __threadfence(); + last = atomicAdd(counter, 1u) == (unsigned) (nb - 1); + } + __syncthreads(); + if (!last) return; + __threadfence(); + bv = __int_as_float(0xff800000); + bi = n; + if ((int) threadIdx.x < nb) { + bv = ((volatile const float*) pv)[threadIdx.x]; + bi = ((volatile const int*) pi)[threadIdx.x]; + } + arg_block(bv, bi, n); + if (threadIdx.x == 0) { + out[row] = bi < n ? bi : 0; // no value above -inf: 0, as the one-block kernel + *counter = 0; + } +} + +// a row's top-probability scratch: its counter, then its 32 warps' sums +struct TopRow { + unsigned counter; + unsigned pad[3]; + float parts[32]; +}; + +// grid (kTopBlocks, rows), 1024 / kTopBlocks threads: thread j of block b is row_top_prob's thread b * blockDim + j +__global__ void row_top_prob_split_kernel(const float* __restrict__ logits, int n_vocab, const int32_t* __restrict__ ids, + float* __restrict__ probs, TopRow* __restrict__ rows) { + const int row = (int) blockIdx.y; + float* parts = rows[row].parts; + const float* l = logits + (size_t) row * n_vocab; + const float m = l[ids[row]]; + const int vt = (int) (blockIdx.x * blockDim.x + threadIdx.x); + float s = 0.0f; + for (int i = vt; i < n_vocab; i += 1024) s += __expf(l[i] - m); + for (int o = 16; o > 0; o >>= 1) s += __shfl_xor_sync(0xffffffffu, s, o); + if ((threadIdx.x & 31) == 0) parts[vt >> 5] = s; + __threadfence(); + __syncthreads(); + __shared__ bool last; + if (threadIdx.x == 0) last = atomicAdd(&rows[row].counter, 1u) == (unsigned) (gridDim.x - 1); + __syncthreads(); + if (!last || threadIdx.x != 0) return; + __threadfence(); + float tot = 0.0f; + for (int w = 0; w < 32; ++w) tot += ((volatile const float*) parts)[w]; + probs[row] = 1.0f / tot; + rows[row].counter = 0; +} + +} // namespace + +uint64_t argmax_rows_scratch_bytes(int n_rows) { return (uint64_t) n_rows * sizeof(ArgRow); } + +void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream) { + if (n_rows <= 0) return; + const int nb = std::min(kArgMaxBlocks, std::max(1, (n + 4095) / 4096)); + const int per_block = (n + nb - 1) / nb; + argmax_rows_kernel<<>>( + logits, n, per_block, (ArgRow*) scratch, out); + check("argmax_rows"); +} + +uint64_t row_top_prob_scratch_bytes(int n_rows) { return (uint64_t) n_rows * sizeof(TopRow); } + +void row_top_prob_split(const float* logits, int n_rows, int n_vocab, const int32_t* ids, float* probs, void* scratch, + void* stream) { + if (n_rows <= 0) return; + row_top_prob_split_kernel<<>>( + logits, n_vocab, ids, probs, (TopRow*) scratch); + check("row_top_prob_split"); +} + namespace { __global__ void window_ids_kernel(int32_t* steps, int window, int32_t* ids, long long stride) { const int q = blockIdx.y; diff --git a/src/kernels/verify_parity.cpp b/src/kernels/verify_parity.cpp index 045e2f819ac..d233fad2c2e 100644 --- a/src/kernels/verify_parity.cpp +++ b/src/kernels/verify_parity.cpp @@ -17,6 +17,7 @@ #include "strata/kernels/native_qsa_indexer.hpp" #include "strata/kernels/native_rope.hpp" #include "strata/kernels/qsa_decode_attn.hpp" +#include "strata/kernels/sampler.hpp" #include "strata/kernels/native_router.hpp" #include "strata/kernels/s2_expert_grouped.hpp" #include "strata/kernels/shared_expert.hpp" @@ -1233,6 +1234,66 @@ int test_prefill_gdn(std::mt19937& rng, cudaStream_t s) { return bad; } +// argmax_rows against sample_tokens' greedy pick and row_top_prob_split against row_top_prob, bitwise: rows of both +// vocabularies with ties at the maximum (the lowest index wins), NaN, -inf, a row with no value above -inf, and +// launches repeated on one scratch (each must leave its counters at zero) +int test_argmax(std::mt19937& rng, cudaStream_t s) { + using namespace strata::kernels; + int bad = 0; + std::normal_distribution nd(0.0f, 4.0f); + uint8_t* sa = dev(argmax_rows_scratch_bytes(8)); // one scratch for every row count, as in the engine + uint8_t* st = dev(row_top_prob_scratch_bytes(8)); + check(cudaMemset(sa, 0, argmax_rows_scratch_bytes(8)), "memset"); + check(cudaMemset(st, 0, row_top_prob_scratch_bytes(8)), "memset"); + for (const int n : {248320, 40525, 4097, 33}) { + for (const int rows : {4, 8, 1, 3, 8, 2}) { + std::vector h((size_t) rows * n); + for (float& v : h) v = nd(rng); + for (int r = 0; r < rows; ++r) { + float* row = h.data() + (size_t) r * n; + const int a = (int) (rng() % n), b = (int) (rng() % n); + row[a] = row[b] = 40.0f; // a tie at the maximum + row[(int) (rng() % n)] = std::nanf(""); // never picked + row[(int) (rng() % n)] = -INFINITY; + if (r == 2) // no value above -inf: 0 + for (int i = 0; i < n; ++i) row[i] = i % 7 ? -INFINITY : std::nanf(""); + } + float* d_l = dev(h.size()); + up(d_l, h); + int* d_ref = dev(rows); + int32_t* d_out = dev(rows); + float* d_p1 = dev(rows); + float* d_p2 = dev(rows); + SamplerParams sp; + sp.greedy = true; + sp.temperature = 0.0f; + sample_tokens(d_l, rows, n, nullptr, 0, sp, d_ref, s); + row_top_prob(d_l, rows, n, d_ref, d_p1, s); + for (int rep = 0; rep < 3; ++rep) { + argmax_rows(d_l, rows, n, sa, d_out, s); + row_top_prob_split(d_l, rows, n, d_out, d_p2, st, s); + check(cudaStreamSynchronize(s), "argmax"); + const std::vector ref = down(d_ref, rows); + const std::vector out = down(d_out, rows); + const std::vector p1 = down(d_p1, rows), p2 = down(d_p2, rows); + for (int r = 0; r < rows; ++r) + if (ref[r] != out[r] || std::memcmp(&p1[r], &p2[r], 4) != 0) { + if (bad < 5) + std::printf(" argmax n %d rows %d row %d: %d / %d, p %.9g / %.9g\n", n, rows, r, ref[r], + out[r], p1[r], p2[r]); + ++bad; + } + } + cudaFree(d_l); cudaFree(d_ref); cudaFree(d_out); cudaFree(d_p1); cudaFree(d_p2); + } + } + cudaFree(sa); + cudaFree(st); + std::printf("argmax_rows, row_top_prob_split: %s\n", + bad ? "MISMATCH" : "bitwise equal to the one-block kernels (ties, NaN, -inf, 1-8 rows on one scratch)"); + return bad; +} + } // namespace int main(int argc, char** argv) { @@ -1260,6 +1321,7 @@ int main(int argc, char** argv) { bad += test_ple_tokens(rng, s); bad += test_decode_attn(rng, s); bad += test_prefill_gdn(rng, s); + bad += test_argmax(rng, s); cudaStreamDestroy(s); std::printf("verify_parity: %s\n", bad ? "FAIL" : "PASS"); return bad ? 1 : 0; From 22b46b647aad4a64dd6064faae73d1fb0751c10b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 06:17:44 +0200 Subject: [PATCH 80/92] The draft round graph runs the draft layer's front once MtpDrafter::capture_round ran the draft layer's front (the embedding, the fc projections, the attention hyper-connection read, the K/V projections and appends) over the window's T cells, then the whole layer again for row a, repeating row a's front. record_forward is now record_front (T rows) and record_rest (one row: the attention from the q projection, the MLP, the final mixer, the head); the round graph runs the front once, copies row a's residual, injection and mixed input to row 0 (copy_row_to_first, the row read on the device) and runs the rest there. The multi-row front is per row the one-row front's arithmetic: the window hashes on fixed text equal the previous build's (IQ3_S, UD-Q4_K_XL; thinking and code text). CUPTI, IQ3_S thinking text (--adapt-every 0): the round graph 721 / 747 / 789 / 784 / 898 -> 655 / 662 / 689 / 692 / 812 us after windows of 1 / 2 / 3 / 4 / 8 tokens. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/mtp.hpp | 7 +++ include/strata/kernels/verify_kernels.hpp | 4 ++ src/core/mtp.cpp | 69 ++++++++++++++++------- src/kernels/cuda/verify_kernels.cu | 18 ++++++ 4 files changed, 77 insertions(+), 21 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index a8ac53dc4c7..18c1650bc4b 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -92,6 +92,13 @@ class MtpDrafter { private: bool record_forward(int T, int step_row0, cudaStream_t cs, std::string& err); + /// The layer's front for T rows at step rows [row0, +T): the embedding, the fc projections, the attention + /// hyper-connection read (R_, inj_, mixed_) and the K/V appended. + bool record_front(int T, int row0, cudaStream_t cs, std::string& err); + /// The rest for row 0 at step row `step_row`: the attention from the q projection, the MLP, the final mixer, + /// the head, the draft and its probability. Needs row 0's R_, inj_ and mixed_ (its q8_1 rows in xq_). + bool record_rest(int step_row, cudaStream_t cs, std::string& err); + void norm_rope(float* data, const float* gamma, int rows, int cols, const int32_t* p, cudaStream_t cs); bool capture_round(int T, std::string& err); bool capture_step(int j, std::string& err); cudaGraphExec_t step_exec_[9] = {}; diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index 933198309cf..72eaf2ddc61 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -80,6 +80,10 @@ void ident_hits(const int32_t* ids, int n, int32_t* slot, int32_t* dst, int32_t* void mtp_select(const float* R_src, int64_t R_stride, const int32_t* ids, const int32_t* row_dev, float* R_dst, int32_t* tok_dst, int32_t* out, int j, void* stream, const float* probs = nullptr, float* out_p = nullptr); +/// Row *row_dev of a, b and c (a_n, b_n, c_n floats a row) copied to their row 0 (the draft layer's rest runs on +/// row 0). Graph-capturable: the row is read on the device. +void copy_row_to_first(const int32_t* row_dev, float* a, int64_t a_n, float* b, int64_t b_n, float* c, int64_t c_n, + void* stream); /// dst row i = src row ids[i] (row_bytes each, a multiple of 4), for n rows. void gather_rows(const uint8_t* src, int64_t row_bytes, const int32_t* ids, int64_t n, uint8_t* dst, void* stream); /// ids[t] = table[ids[t]] for n entries (a subset index back to a token id). diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index f388be048c5..b050455e64e 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -284,16 +284,18 @@ bool MtpDrafter::bind(const WeightTable& wt, const NativeHead* head, const float // The layer for T rows. full = false stops after the K/V append (the prompt only needs the cache). bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::string& err) { - using namespace strata::kernels; - const ModelGeometry& g = *g_; - SessionState& ss = *ss_; // step_row0 >= 0: the full layer on step rows [step_row0, +T); step_row0 < 0: K/V only on rows [-1 - step_row0, +T) const bool full = step_row0 >= 0; - const int row0 = full ? step_row0 : -1 - step_row0; if (full && T != 1) { err = "mtp: the full layer runs one row at a time (its attention scratch is sized for one)"; return false; } - const int64_t N = g.n_embd, HC = g.hc, K = ss.k, NH = g.n_head, HD = g.head_dim, NKV = g.n_head_kv; + if (!record_front(T, full ? step_row0 : -1 - step_row0, cs, err)) return false; + return !full || record_rest(step_row0, cs, err); +} + +bool MtpDrafter::record_front(int T, int row0, cudaStream_t cs, std::string& err) { + using namespace strata::kernels; + const ModelGeometry& g = *g_; + const int64_t N = g.n_embd, HC = g.hc, NH = g.n_head, HD = g.head_dim, NKV = g.n_head_kv; const QsaShapes s = shapes_of(g); - const GrShapes gs{g.n_embd, g.hc, g.hc_lr}; const int32_t* step = step_ + row0 * 4; const int32_t* pos = pos_ + row0 * NH; try { @@ -323,17 +325,12 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri } fused_gr_read_multi(fa, T, grs_, cs); } - // ---- attention: K/V into the layer's own cache, then (full) dense attention over every cell - auto norm_rope = [&](float* data, const float* gamma, int rows, int cols, const int32_t* p) { - native_qsa_rms_norm_weighted(data, gamma, data, cols, rows, EPS, cs); - if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), p, cs); - else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st_.cos_tab, st_.sin_tab, p, cs); - }; + // ---- attention: K/V into the layer's own cache native_quantize_q8_1(mixed_, xq_, (int) N, T, cs); native_mmvq(GGML_Q8_0, q8("self_attn.k_proj.weight"), xq_, kcur_, (int) N, (int) (NKV * HD), T, cs); native_mmvq(GGML_Q8_0, q8("self_attn.v_proj.weight"), xq_, vcur_, (int) N, (int) (NKV * HD), T, cs); for (int t = 0; t < T; ++t) { - norm_rope(kcur_ + t * NKV * HD, f32("self_attn.k_norm.weight"), (int) NKV, (int) HD, pos + t * NH); + norm_rope(kcur_ + t * NKV * HD, f32("self_attn.k_norm.weight"), (int) NKV, (int) HD, pos + t * NH, cs); if (st_.kv_int8) kv_append_q8_step(st_.k_q, st_.v_q, st_.k_scale, st_.v_scale, st_.page_table, step + t * 4, kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); @@ -341,7 +338,33 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri kv_append_step(st_.k_pool, st_.v_pool, st_.page_table, step + t * 4, kcur_ + t * NKV * HD, vcur_ + t * NKV * HD, s, cs); } - if (!full) return true; + } catch (const std::exception& e) { + err = std::string("mtp: ") + e.what(); + return false; + } + return true; +} + +void MtpDrafter::norm_rope(float* data, const float* gamma, int rows, int cols, const int32_t* p, cudaStream_t cs) { + using namespace strata::kernels; + const QsaShapes s = shapes_of(*g_); + native_qsa_rms_norm_weighted(data, gamma, data, cols, rows, EPS, cs); + if (native_rope_enabled()) native_rope_apply(data, data, rows, cols, (int) s.n_rot, (float) qsa_freq_base(), p, cs); + else rope_neox_apply(data, data, rows, cols, (int) s.n_rot, st_.cos_tab, st_.sin_tab, p, cs); +} + +bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) { + using namespace strata::kernels; + const ModelGeometry& g = *g_; + SessionState& ss = *ss_; + const int T = 1; + const int64_t N = g.n_embd, HC = g.hc, K = ss.k, NH = g.n_head, HD = g.head_dim; + const QsaShapes s = shapes_of(g); + const GrShapes gs{g.n_embd, g.hc, g.hc_lr}; + const int32_t* step = step_ + step_row * 4; + const int32_t* pos = pos_ + step_row * NH; + try { + // ---- dense attention over every cell native_mmvq(GGML_Q8_0, q8("self_attn.q_proj.weight"), xq_, qfull_, (int) N, (int) (NH * 2 * HD), T, cs); for (int t = 0; t < T; ++t) { float* qc = qcur_ + t * NH * HD; @@ -350,7 +373,7 @@ bool MtpDrafter::record_forward(int T, int step_row0, cudaStream_t cs, std::stri err = "mtp: q split failed"; return false; } - norm_rope(qc, f32("self_attn.q_norm.weight"), (int) NH, (int) HD, pos + t * NH); + norm_rope(qc, f32("self_attn.q_norm.weight"), (int) NH, (int) HD, pos + t * NH, cs); } QsaAttnPools pools; pools.page_table = st_.page_table; @@ -461,16 +484,20 @@ bool MtpDrafter::capture_round(int T, std::string& err) { copy_i32_from_mapped(pos_, m_pos_, (int64_t) 2 * T * g_->n_head, cs_); copy_i32_from_mapped(row_, m_row_, 2, cs_); copy_from_mapped(Rin_, window_R_, (int64_t) T * HCN, cs_); - // the catch-up: K/V for the window's T cells, then the full layer for row a only (its cell's K/V is written - // again, identically), staged by the host in step row 2*max_t - 1; the draft chain is one graph per step - // (`capture_step`) so the host can stop it when a draft is unlikely + // the catch-up: the layer's front for the window's T cells (their K/V), then its rest for row a only, on row + // a's intermediates copied to row 0, at the cell the host staged in step row 2*max_t - 1; the draft chain is + // one graph per step (`capture_step`) so the host can stop it when a draft is unlikely const int ra = 2 * max_t_ - 1; - ok = record_forward(T, -1, cs_, err); - if (ok) mtp_select(Rin_, HCN, tok_, row_, Rin_, tok_, nullptr, 0, cs_); + const int64_t N = g_->n_embd; + ok = record_front(T, 0, cs_, err); if (ok) { + if (T > 1) { + copy_row_to_first(row_, R_, HCN, inj_, g_->hc, mixed_, N, cs_); + native_quantize_q8_1(mixed_, xq_, (int) N, 1, cs_); + } copy_i32_from_mapped(step_ + ra * 4, m_step_ + ra * 4, 4, cs_); copy_i32_from_mapped(pos_ + ra * g_->n_head, m_pos_ + ra * g_->n_head, g_->n_head, cs_); - ok = record_forward(1, ra, cs_, err); + ok = record_rest(ra, cs_, err); } if (ok) mtp_select(R_, HCN, out_ids_, row_ + 1, Rin_, tok_, m_out_, 0, cs_, probs_, m_prob_); return finish_capture(cs_, ok, round_exec_[T], "round", err); diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index c0386ffef89..beaa660c59e 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -375,6 +375,24 @@ void ident_hits(const int32_t* ids, int n, int32_t* slot, int32_t* dst, int32_t* check("ident_hits"); } +namespace { +__global__ void copy_row_to_first_kernel(const int32_t* __restrict__ row_dev, float* a, int64_t a_n, float* b, + int64_t b_n, float* c, int64_t c_n) { + const int64_t row = *row_dev; + if (row == 0) return; + const int64_t i0 = (int64_t) blockIdx.x * blockDim.x + threadIdx.x, st = (int64_t) gridDim.x * blockDim.x; + for (int64_t i = i0; i < a_n; i += st) a[i] = a[row * a_n + i]; + for (int64_t i = i0; i < b_n; i += st) b[i] = b[row * b_n + i]; + for (int64_t i = i0; i < c_n; i += st) c[i] = c[row * c_n + i]; +} +} // namespace + +void copy_row_to_first(const int32_t* row_dev, float* a, int64_t a_n, float* b, int64_t b_n, float* c, int64_t c_n, + void* stream) { + copy_row_to_first_kernel<<<16, 256, 0, (cudaStream_t) stream>>>(row_dev, a, a_n, b, b_n, c, c_n); + check("copy_row_to_first"); +} + void mtp_select(const float* R_src, int64_t R_stride, const int32_t* ids, const int32_t* row_dev, float* R_dst, int32_t* tok_dst, int32_t* out, int j, void* stream, const float* probs, float* out_p) { mtp_select_kernel<<<16, 256, 0, (cudaStream_t) stream>>>(R_src, R_stride, ids, row_dev, R_dst, tok_dst, out, j, From 235784c2798c7f349103ebf950ee2dd1ec02b55c Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 06:36:32 +0200 Subject: [PATCH 81/92] A one-token window commits itself A one-token window always keeps its token, so its graph now advances the sequence state itself: the GDN step kernel writes the state (n_keep = 1, the arithmetic the commit's replay used), the conv kernel writes the history it read (gdn_conv_l2_multi's commit flag), and its indexer append and PLE history were final already. Verifier::commit launches no commit graph after it: the draft round graph then runs alone instead of beside the commit's 98 kernels, which read and write each GDN layer's 3 MB state. Window hashes on fixed text equal the previous build's (IQ3_S, UD-Q4_K_XL; thinking and code text); cache_test.py's regenerated turn and stash round trip reproduce their output. CUPTI, IQ3_S thinking text (--adapt-every 0), the 35% of rounds after one-token windows: the window 11,270 -> 11,380 us, the time from its end to the next window's start 3,706 -> 3,325 us (the round graph 655 -> 450 us, the draft starting 150 us sooner): -270 us a round there, ~-95 us over all rounds. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/verify.hpp | 1 + include/strata/kernels/verify_kernels.hpp | 6 +++--- src/core/verify.cpp | 26 +++++++++++++++++------ src/kernels/cuda/verify_kernels.cu | 18 ++++++++++------ 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 37c4df1856c..27c240aded6 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -315,6 +315,7 @@ class Verifier { void* hit_scratch_ = nullptr; float *head_mixed_ = nullptr, *head_inj_ = nullptr, *head_logits_ = nullptr; uint8_t* arg_scratch_ = nullptr; ///< argmax_rows' partials and counters + int32_t* one_ = nullptr; ///< device {1}: the n_keep of a one-token window, which commits itself float *sh_gate_ = nullptr, *sh_up_ = nullptr, *sh_g_ = nullptr; float* hist_snap_ = nullptr; // T * NG_HIST * NG_HC_DIM // the PLE block's projections (keys T * NG_HC_DIM, values T * n_embd) and the rest of the block's rows diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index 72eaf2ddc61..fd9444ca53d 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -21,10 +21,10 @@ namespace strata::kernels { inline constexpr int kVerifyMaxT = 8; /// For token t of T: conv over [history(3) | qkv_0 .. qkv_t] -> SiLU -> L2 norm of the q/k heads -> h[t]. -/// `history` is NOT written. Bitwise `fused_gdn_conv_l2` per token. Up to 65535 tokens (the prompt path runs a -/// whole chunk through it). +/// `history` is NOT written, but with `commit` (one token): it then keeps the token, as gdn_conv_commit. Bitwise +/// `fused_gdn_conv_l2` per token. Up to 65535 tokens (the prompt path runs a whole chunk through it). void gdn_conv_l2_multi(const float* history, const float* qkv, const float* conv_w, float* h, int channels, - int qk_heads, float eps, int n_tok, void* stream, int t_begin = 0); + int qk_heads, float eps, int n_tok, void* stream, int t_begin = 0, bool commit = false); /// history <- the last 3 entries of [history | qkv_0 .. qkv_{n-1}], n = *n_keep (0 leaves it as it was). void gdn_conv_commit(float* history, const float* qkv, int channels, const int32_t* n_keep, void* stream); /// alpha/beta for T columns of x (T, n_embd): gate (T, h_v), beta (T, h_v). Bitwise `fused_gdn_ab` per column. diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 19e6a48c197..c46be3df039 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -274,6 +274,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sh_up_ = b.take(T * (uint64_t) g.n_ff); sh_g_ = b.take(T + 4); head_logits_ = b.take(T * (uint64_t) n_vocab_); arg_scratch_ = b.take(strata::kernels::argmax_rows_scratch_bytes((int) T)); + one_ = b.take(4); hist_snap_ = b.take(T * HS); const uint64_t PD = (uint64_t) strata::kernels::NG_HC_DIM; ple_key_ = b.take(T * PD); ple_val_ = b.take(T * N); ple_nkey_ = b.take(T * PD); @@ -289,6 +290,13 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& Bump real; real.base = (uint8_t*) arena_; carve(real); + { + const int32_t one = 1; + if (cudaMemcpy(one_, &one, sizeof one, cudaMemcpyHostToDevice) != cudaSuccess) { + err = "verify: the arena could not be set"; + return false; + } + } sink_.staging = (unsigned long long) staging_; sink_.staging_cap = kStagingBlobs; (void) TS; @@ -615,11 +623,13 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { proj(wqkv, qkv + (size_t) tb * C, N, C, cs); proj(wg, z_ + (size_t) tb * ZV, N, ZV, shs_); if (!mark(shs_, pjoin_)) return false; - gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb); + // a one-token window keeps its token: it advances the conv history and the state itself (commit) + gdn_conv_l2_multi(conv, qkv, (const float*) wc->data, hb, (int) C, (int) (2 * HK), EPS, n, cs, tb, + T == 1); if (!wait(bjoin_) || !wait(pjoin_)) return false; // the recurrence from the untouched state over tokens [0, te); outputs only for this group's gdn_step_norm_multi(state, hb, (int) C, gate, beta, z_, (const float*) wnm->data, EPS, y_, (int) HK, - (int) HV, te, nullptr, cs, tb); + (int) HV, te, T == 1 ? one_ : nullptr, cs, tb); quant(y_ + (size_t) tb * ZV, ZV); proj(wout, bo_ + tb * N, ZV, N, cs); } else { @@ -1403,11 +1413,13 @@ bool Verifier::commit(int n_keep, std::string& err, bool wait) { h_commit_[0] = n_keep; h_commit_[1] = n_keep - 1; for (int t = 0; t < max_t_; ++t) h_commit_[2 + t] = t < n_keep ? (int32_t) (last_pos0_ + t) : -1; - std::atomic_thread_fence(std::memory_order_seq_cst); - const cudaError_t le = cudaGraphLaunch(commit_exec_, cs_); - if (le != cudaSuccess) { err = std::string("verify: commit launch: ") + cudaGetErrorString(le); return false; } - (void) cudaStreamQuery(cs_); // submit now, not at the next driver call - commit_pending_ = true; + if (last_t_ > 1) { // a one-token window has advanced the state itself (record_window) + std::atomic_thread_fence(std::memory_order_seq_cst); + const cudaError_t le = cudaGraphLaunch(commit_exec_, cs_); + if (le != cudaSuccess) { err = std::string("verify: commit launch: ") + cudaGetErrorString(le); return false; } + (void) cudaStreamQuery(cs_); // submit now, not at the next driver call + commit_pending_ = true; + } for (int t = 0; t < n_keep; ++t) { ss_->ple_prev[0] = ss_->ple_prev[1]; ss_->ple_prev[1] = last_tokens_[t]; diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index beaa660c59e..f6d350631d4 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -23,10 +23,10 @@ void check(const char* what) { if (e != cudaSuccess) { std::fprintf(stderr, "%s: %s\n", what, cudaGetErrorString(e)); std::exit(1); } } -__global__ void __launch_bounds__(S) gdn_conv_l2_multi_kernel(const float* __restrict__ hist, - const float* __restrict__ qkv, +// `commit` (one token): the history then keeps it - [hist1, hist2, x_0], each thread its channel's, after reading them +__global__ void __launch_bounds__(S) gdn_conv_l2_multi_kernel(float* hist, const float* __restrict__ qkv, const float* __restrict__ w, float* __restrict__ h, - int C, int qk_heads, float eps, int t_begin) { + int C, int qk_heads, float eps, int t_begin, bool commit) { __shared__ float part[S / 32]; const int t = t_begin + blockIdx.y; const int c = blockIdx.x * S + threadIdx.x; @@ -38,6 +38,11 @@ __global__ void __launch_bounds__(S) gdn_conv_l2_multi_kernel(const float* __res win[j] = src < 3 ? hist[c * 3 + src] : qkv[(size_t) (src - 3) * C + c]; } const float v0 = win[0], v1 = win[1], v2 = win[2], x = qkv[(size_t) t * C + c]; + if (commit) { + hist[c * 3] = v1; + hist[c * 3 + 1] = v2; + hist[c * 3 + 2] = x; + } float sum = v0 * w[c * 4] + v1 * w[c * 4 + 1] + v2 * w[c * 4 + 2] + x * w[c * 4 + 3]; float y = sum / (1.0f + __expf(-sum)); if ((int) blockIdx.x < qk_heads) { @@ -582,13 +587,14 @@ void dense_steps(const int32_t* cells, int n, int32_t* steps, void* stream) { } void gdn_conv_l2_multi(const float* history, const float* qkv, const float* conv_w, float* h, int channels, - int qk_heads, float eps, int n_tok, void* stream, int t_begin) { - if (!history || !qkv || !conv_w || !h || channels % S != 0 || n_tok < 1 || n_tok > 65535) { + int qk_heads, float eps, int n_tok, void* stream, int t_begin, bool commit) { + if (!history || !qkv || !conv_w || !h || channels % S != 0 || n_tok < 1 || n_tok > 65535 || + (commit && (n_tok != 1 || t_begin != 0))) { std::fprintf(stderr, "gdn_conv_l2_multi: invalid arguments\n"); std::exit(1); } gdn_conv_l2_multi_kernel<<>>( - history, qkv, conv_w, h, channels, qk_heads, eps, t_begin); + const_cast(history), qkv, conv_w, h, channels, qk_heads, eps, t_begin, commit); check("gdn_conv_l2_multi"); } From d3d48b14e89ede6d29deafecb6382f3f5f103266 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 06:49:05 +0200 Subject: [PATCH 82/92] The draft pass's kernels in the main layer's forms - quantize_q8_0_scaled runs a warp a 32-value block (it ran a thread a block: 80 threads for the draft layer's input): the block's |max| over the same values, then each value's bytes as before. 6.2 -> 2.4 us and 9.2 -> 1.6 us for the Q2_0 draft experts' two quantizations (CUPTI). - The draft graphs' inputs from mapped memory in one copy kernel (copy_from_mapped_multi): 7 launches in the round graph, 2 in a step, became one each. - The router in one kernel (verify_router: the logits, the top 10, and the native draft experts' q8_1 rows), bitwise the gemv, top 10 and quantizer it replaces; as fast as the gemv and top 10 were. - The shared expert on a branch of the draft graphs beside the router and the routed experts, joined before the combine. Window hashes on fixed text equal the previous build's (IQ3_S, UD-Q4_K_XL; thinking and code text; IQ3_S with the IQ4_NL draft experts too). CUPTI, IQ3_S thinking text (--adapt-every 0): the round graph 450 / 661 / 687 / 693 / 824 -> 411 / 592 / 620 / 627 / 760 us after windows of 1 / 2 / 3 / 4 / 8 tokens, each step 435 -> 405-410 us. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/core/mtp.hpp | 4 ++ include/strata/kernels/elementwise.hpp | 8 +++ src/core/mtp.cpp | 82 ++++++++++++++++++-------- src/kernels/cuda/elementwise.cu | 31 ++++++++++ src/kernels/cuda/quantize_act.cu | 38 ++++++------ 5 files changed, 119 insertions(+), 44 deletions(-) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index 18c1650bc4b..ffb9a53e68d 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -115,6 +115,10 @@ class MtpDrafter { int64_t n_vocab_ = 0; uint64_t vram_ = 0; cudaStream_t cs_ = nullptr; + cudaStream_t side_ = nullptr; // the shared expert's branch of the draft graphs + cudaEvent_t sh_fork_ = nullptr, sh_join_ = nullptr; + uint32_t* rseq_ = nullptr; // verify_router's sequence word and counter (device) + unsigned* rcount_ = nullptr; cudaGraphExec_t round_exec_[9] = {}; struct Tensor { std::string name, kind; int64_t rows = 0, cols = 0; uint64_t off = 0, bytes = 0; }; diff --git a/include/strata/kernels/elementwise.hpp b/include/strata/kernels/elementwise.hpp index 0136176c47c..6db5abd9dfa 100644 --- a/include/strata/kernels/elementwise.hpp +++ b/include/strata/kernels/elementwise.hpp @@ -116,5 +116,13 @@ void doorbell_publish(const float* x, const int32_t* ids, const float* weights, /// Plan v0.3 P3: copy `n` int32 from mapped pinned host memory into device memory with a kernel (the QSA /// per-token step and positions), instead of a host-to-device memcpy node in the middle of a layer. void copy_i32_from_mapped(int32_t* dst, const int32_t* src, int64_t n, void* stream); +/// Up to 8 copies from mapped host memory in one launch: `words` 4-byte words each (16 bytes a load where both +/// ends are 16-byte aligned and the count a multiple of 4). Graph-capturable. +struct MappedCopy { + void* dst = nullptr; + const void* src = nullptr; + int64_t words = 0; +}; +void copy_from_mapped_multi(const MappedCopy* copies, int n, void* stream); } // namespace strata::kernels diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index b050455e64e..36d79b492ba 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -83,6 +84,9 @@ MtpDrafter::~MtpDrafter() { for (auto& e : round_exec_) if (e) cudaGraphExecDestroy(e); for (auto& e : step_exec_) if (e) cudaGraphExecDestroy(e); if (cs_) cudaStreamDestroy(cs_); + if (side_) cudaStreamDestroy(side_); + if (sh_fork_) cudaEventDestroy(sh_fork_); + if (sh_join_) cudaEventDestroy(sh_join_); if (dense_) cudaFree(dense_); if (experts_) cudaFree(experts_); if (state_arena_) cudaFree(state_arena_); @@ -213,6 +217,8 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session grp_counts_ = b.take(4); hit_xq_ = b.take(T * (N / 32) * 34); hit_xs_ = b.take(T * (N / 32)); nat_xq_ = b.take(T * (N / 32) * 36); + rseq_ = b.take(4); + rcount_ = b.take(4); hit_scratch_ = b.take(std::max( strata::kernels::moe_hit_grouped_scratch_bytes((int64_t) (T * K), g.n_embd, g.n_ff), strata::kernels::native_expert_scratch_bytes((int64_t) (T * K), g.n_ff))); @@ -238,7 +244,13 @@ bool MtpDrafter::load(const std::string& rt_dir, const ModelGeometry& g, Session for (int64_t i = 0; i < cap_; ++i) id[(size_t) (t * (uint64_t) cap_ + (uint64_t) i)] = (int32_t) i; cudaMemcpy(ident_, id.data(), id.size() * 4, cudaMemcpyHostToDevice); } - if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess) { err = "mtp: stream"; return false; } + if (cudaStreamCreateWithFlags(&cs_, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&side_, cudaStreamNonBlocking) != cudaSuccess || + cudaEventCreateWithFlags(&sh_fork_, cudaEventDisableTiming) != cudaSuccess || + cudaEventCreateWithFlags(&sh_join_, cudaEventDisableTiming) != cudaSuccess) { + err = "mtp: streams"; + return false; + } char fmts[48] = "Q2_0 blobs"; if (nat_.gu_type >= 0) std::snprintf(fmts, sizeof fmts, "ggml types %d/%d", nat_.gu_type, nat_.d_type); std::fprintf(stderr, "strata mtp: draft layer loaded, %.0f MiB of VRAM (experts %.0f as %s, dense %.0f)\n", @@ -398,16 +410,45 @@ bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) { } fused_gr_read_multi(fa, T, grs_, cs); } - // ---- MoE: router, the 512 resident experts, the shared expert, the combine, the write + // ---- MoE: the shared expert on a branch beside the router and the 512 resident experts, then the combine + // and the write + NativeSharedWeights nsw; + nsw.gate_type = GGML_Q8_0; nsw.gate_data = q8("mlp.shared_expert.gate_proj.weight"); + nsw.up_type = GGML_Q8_0; nsw.up_data = q8("mlp.shared_expert.up_proj.weight"); + nsw.down_type = GGML_Q8_0; nsw.down_data = q8("mlp.shared_expert.down_proj.weight"); + nsw.q8_1 = xq_; // the routed experts read nat_xq_ / hit_xq_ + const SForm none{}; + if (cudaEventRecord(sh_fork_, cs) != cudaSuccess || cudaStreamWaitEvent(side_, sh_fork_, 0) != cudaSuccess) { + err = "mtp: the shared expert's branch"; + return false; + } for (int t = 0; t < T; ++t) { - bf16_gemv_fp32_mmvf(mixed_ + t * N, bf16("mlp.gate.weight"), logits_ + t * g.n_expert, (int) N, (int) g.n_expert, cs); - if (native_router_enabled()) native_router_top10(logits_ + t * g.n_expert, ids_ + t * K, w_ + t * K, cs); - else router_top10(logits_ + t * g.n_expert, 1, (int) g.n_expert, (int) K, ids_ + t * K, w_ + t * K, cs); + f32_to_bf16_bulk(mixed_ + t * N, x_bf16_, N, side_); + shared_expert(nullptr, nullptr, x_bf16_, none, nullptr, nullptr, nullptr, none, nullptr, nullptr, nullptr, none, + nullptr, nullptr, nullptr, bf16("mlp.shared_expert_gate.weight"), sh_scratch_, shared_ + t * N, + N, g.n_ff, 32, side_, mixed_ + t * N, &nsw); + } + if (cudaEventRecord(sh_join_, side_) != cudaSuccess) { err = "mtp: the shared expert's branch"; return false; } + bool xq1 = false; // the router kernel wrote the native experts' q8_1 rows + if (native_router_enabled()) { + VerifyRouterArgs ra; + ra.x = mixed_; ra.w = bf16("mlp.gate.weight"); ra.logits = logits_; ra.ids = ids_; ra.weights = w_; + ra.xq1 = nat_.gu_type >= 0 ? nat_xq_ : nullptr; + ra.seq = rseq_; ra.counter = rcount_; + ra.n_tok = T; ra.n_embd = (int) N; ra.n_expert = (int) g.n_expert; + verify_router(ra, cs); + xq1 = ra.xq1 != nullptr; + } else { + for (int t = 0; t < T; ++t) { + bf16_gemv_fp32_mmvf(mixed_ + t * N, bf16("mlp.gate.weight"), logits_ + t * g.n_expert, (int) N, + (int) g.n_expert, cs); + router_top10(logits_ + t * g.n_expert, 1, (int) g.n_expert, (int) K, ids_ + t * K, w_ + t * K, cs); + } } moe_group_resident(ids_, (int) (T * K), (int) K, experts_, (int64_t) blob_, grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, cs); if (nat_.gu_type >= 0) { - quantize_q8_1_rows(mixed_, T, N, nat_xq_, cs); + if (!xq1) quantize_q8_1_rows(mixed_, T, N, nat_xq_, cs); native_expert_grouped(nat_, grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, (int64_t) T * K, (int64_t) T * K, nat_xq_, hit_scratch_, parts_, cs); } else { @@ -415,17 +456,8 @@ bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) { moe_grouped_s2(grp_ptr_, grp_start_, grp_counts_, hit_dst_, hit_slot_, (int64_t) T * K, (int64_t) T * K, hit_xq_, hit_xs_, hit_scratch_, parts_, cs); } - NativeSharedWeights nsw; - nsw.gate_type = GGML_Q8_0; nsw.gate_data = q8("mlp.shared_expert.gate_proj.weight"); - nsw.up_type = GGML_Q8_0; nsw.up_data = q8("mlp.shared_expert.up_proj.weight"); - nsw.down_type = GGML_Q8_0; nsw.down_data = q8("mlp.shared_expert.down_proj.weight"); - nsw.q8_1 = xq_; - const SForm none{}; + if (cudaStreamWaitEvent(cs, sh_join_, 0) != cudaSuccess) { err = "mtp: the shared expert's join"; return false; } for (int t = 0; t < T; ++t) { - f32_to_bf16_bulk(mixed_ + t * N, x_bf16_, N, cs); - shared_expert(nullptr, nullptr, x_bf16_, none, nullptr, nullptr, nullptr, none, nullptr, nullptr, nullptr, none, - nullptr, nullptr, nullptr, bf16("mlp.shared_expert_gate.weight"), sh_scratch_, shared_ + t * N, - N, g.n_ff, 32, cs, mixed_ + t * N, &nsw); if (native_moe_combine_enabled()) native_moe_combine(parts_ + (size_t) t * K * N, w_ + t * K, shared_ + t * N, y_ + t * N, N, K, cs); else @@ -479,24 +511,21 @@ bool MtpDrafter::capture_round(int T, std::string& err) { const int64_t HCN = g_->hc * g_->n_embd; if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { err = "mtp: begin capture"; return false; } bool ok = true; - copy_i32_from_mapped(tok_, m_tok_, T, cs_); - copy_i32_from_mapped(step_, m_step_, (int64_t) 2 * T * 4, cs_); - copy_i32_from_mapped(pos_, m_pos_, (int64_t) 2 * T * g_->n_head, cs_); - copy_i32_from_mapped(row_, m_row_, 2, cs_); - copy_from_mapped(Rin_, window_R_, (int64_t) T * HCN, cs_); // the catch-up: the layer's front for the window's T cells (their K/V), then its rest for row a only, on row // a's intermediates copied to row 0, at the cell the host staged in step row 2*max_t - 1; the draft chain is // one graph per step (`capture_step`) so the host can stop it when a draft is unlikely const int ra = 2 * max_t_ - 1; - const int64_t N = g_->n_embd; + const int64_t N = g_->n_embd, NH = g_->n_head; + const MappedCopy in[7] = {{tok_, m_tok_, T}, {step_, m_step_, (int64_t) 2 * T * 4}, {pos_, m_pos_, 2 * T * NH}, + {row_, m_row_, 2}, {Rin_, window_R_, T * HCN}, {step_ + ra * 4, m_step_ + ra * 4, 4}, + {pos_ + ra * NH, m_pos_ + ra * NH, NH}}; + copy_from_mapped_multi(in, 7, cs_); ok = record_front(T, 0, cs_, err); if (ok) { if (T > 1) { copy_row_to_first(row_, R_, HCN, inj_, g_->hc, mixed_, N, cs_); native_quantize_q8_1(mixed_, xq_, (int) N, 1, cs_); } - copy_i32_from_mapped(step_ + ra * 4, m_step_ + ra * 4, 4, cs_); - copy_i32_from_mapped(pos_ + ra * g_->n_head, m_pos_ + ra * g_->n_head, g_->n_head, cs_); ok = record_rest(ra, cs_, err); } if (ok) mtp_select(R_, HCN, out_ids_, row_ + 1, Rin_, tok_, m_out_, 0, cs_, probs_, m_prob_); @@ -511,8 +540,9 @@ bool MtpDrafter::capture_step(int j, std::string& err) { const int64_t HCN = g_->hc * g_->n_embd; const int row = max_t_ + j - 1; if (cudaStreamBeginCapture(cs_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { err = "mtp: begin capture"; return false; } - copy_i32_from_mapped(step_ + row * 4, m_step_ + row * 4, 4, cs_); - copy_i32_from_mapped(pos_ + row * g_->n_head, m_pos_ + row * g_->n_head, g_->n_head, cs_); + const MappedCopy in[2] = {{step_ + row * 4, m_step_ + row * 4, 4}, + {pos_ + row * g_->n_head, m_pos_ + row * g_->n_head, g_->n_head}}; + copy_from_mapped_multi(in, 2, cs_); bool ok = record_forward(1, row, cs_, err); if (ok) mtp_select(R_, HCN, out_ids_, row_ + 1, Rin_, tok_, m_out_, j, cs_, probs_, m_prob_); return finish_capture(cs_, ok, step_exec_[j], "step", err); diff --git a/src/kernels/cuda/elementwise.cu b/src/kernels/cuda/elementwise.cu index a2e9e53b5c7..60c26e8b186 100644 --- a/src/kernels/cuda/elementwise.cu +++ b/src/kernels/cuda/elementwise.cu @@ -293,6 +293,37 @@ __global__ void copy_i32_from_mapped_kernel(int32_t* __restrict__ dst, const vol for (int i = threadIdx.x; i < n; i += blockDim.x) dst[i] = src[i]; } +struct MappedCopies { + MappedCopy c[8]; +}; + +__global__ void copy_from_mapped_multi_kernel(MappedCopies a) { + const MappedCopy c = a.c[blockIdx.y]; + const int64_t i0 = (int64_t) blockIdx.x * blockDim.x + threadIdx.x, st = (int64_t) gridDim.x * blockDim.x; + if ((c.words & 3) == 0 && ((uintptr_t) c.dst & 15) == 0 && ((uintptr_t) c.src & 15) == 0) { + const volatile float4* s = (const volatile float4*) c.src; + for (int64_t i = i0; i < c.words / 4; i += st) ((float4*) c.dst)[i] = const_cast(s)[i]; + } else { + const volatile int32_t* s = (const volatile int32_t*) c.src; + for (int64_t i = i0; i < c.words; i += st) ((int32_t*) c.dst)[i] = s[i]; + } +} + +void copy_from_mapped_multi(const MappedCopy* copies, int n, void* stream) { + if (n <= 0) return; + if (n > 8) { std::fprintf(stderr, "copy_from_mapped_multi: at most 8 copies\n"); std::exit(1); } + MappedCopies a; + int64_t most = 0; + for (int i = 0; i < n; ++i) { + a.c[i] = copies[i]; + most = copies[i].words > most ? copies[i].words : most; + } + const int64_t units = (most + 3) / 4; + const unsigned bx = (unsigned) ((units + 255) / 256 < 32 ? (units + 255) / 256 : 32); + copy_from_mapped_multi_kernel<<>>(a); + check_launch("copy_from_mapped_multi"); +} + void copy_i32_from_mapped(int32_t* dst, const int32_t* src, int64_t n, void* stream) { if (n <= 0) return; copy_i32_from_mapped_kernel<<<1, 128, 0, (cudaStream_t) stream>>>(dst, (const volatile int32_t*) src, (int) n); diff --git a/src/kernels/cuda/quantize_act.cu b/src/kernels/cuda/quantize_act.cu index ddd14dfbb0a..0e05a6c3546 100644 --- a/src/kernels/cuda/quantize_act.cu +++ b/src/kernels/cuda/quantize_act.cu @@ -97,33 +97,35 @@ __global__ void quantize_q8_0_kernel(const float* __restrict__ x, uint8_t* __res /// `moe_hit_parity`. This function's job is different: it must match **this engine's own CPU reference**, /// because a hit and a miss for the same expert on the same layer have to produce the same number. The CPU /// path is the reference - C1 passes on it - so the hit path is brought to it, not the reverse. +// A warp a block, lane i on value i: the block's |max| is a max over the same values (0 for none above it, as the +// serial scan from 0 through fmaxf), so every byte is the per-thread loop's. __global__ void quantize_q8_0_scaled_kernel(const float* __restrict__ x, uint8_t* __restrict__ blocks, float* __restrict__ scales, long long n_blocks) { - const long long b = (long long) blockIdx.x * blockDim.x + threadIdx.x; + const long long b = ((long long) blockIdx.x * blockDim.x + threadIdx.x) >> 5; + const int i = (int) (threadIdx.x & 31); if (b >= n_blocks) return; - const float* xb = x + b * QK8_0; + const float xi = x[b * QK8_0 + i]; uint8_t* out = blocks + b * 34; - float amax = 0.0f; - for (int i = 0; i < QK8_0; ++i) amax = fmaxf(amax, fabsf(xb[i])); + float amax = fmaxf(0.0f, fabsf(xi)); + for (int o = 16; o > 0; o >>= 1) amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, o)); // VERBATIM from `cpu/expert.cpp:144-145`, including the `amax > 0` guard, so the fp32 value written here // is bit-identical to the `s` the CPU path used. const float s = amax > 0.f ? amax / 127.f : 0.f; const float inv = s > 0.f ? 1.f / s : 0.f; - scales[b] = s; - - const uint16_t d16bits = f16_from_f32(s); - out[0] = (uint8_t) (d16bits & 0xFF); - out[1] = (uint8_t) (d16bits >> 8); - for (int i = 0; i < QK8_0; ++i) { - // VERBATIM from `cpu/expert.cpp:159-162`: reciprocal multiply, then `t + copysign(0.5, t)` truncated - // toward zero, which is `lround`'s rule - round half away from zero. - const float t = xb[i] * inv; - const float r = t + (t >= 0.f ? 0.5f : -0.5f); - int v = (int) r; - v = v < -127 ? -127 : (v > 127 ? 127 : v); - out[2 + i] = (uint8_t) (int8_t) v; + if (i == 0) { + scales[b] = s; + const uint16_t d16bits = f16_from_f32(s); + out[0] = (uint8_t) (d16bits & 0xFF); + out[1] = (uint8_t) (d16bits >> 8); } + // VERBATIM from `cpu/expert.cpp:159-162`: reciprocal multiply, then `t + copysign(0.5, t)` truncated + // toward zero, which is `lround`'s rule - round half away from zero. + const float t = xi * inv; + const float r = t + (t >= 0.f ? 0.5f : -0.5f); + int v = (int) r; + v = v < -127 ? -127 : (v > 127 ? 127 : v); + out[2 + i] = (uint8_t) (int8_t) v; } __global__ void dequant_q8_0_kernel(const uint8_t* __restrict__ blocks, float* __restrict__ x, @@ -254,7 +256,7 @@ void quantize_q8_0_scaled(const float* x, uint8_t* blocks, float* scales, int64_ } const long long nb = n / QK8_0; const int threads = 128; - const unsigned grid = (unsigned) ((nb + threads - 1) / threads); + const unsigned grid = (unsigned) ((nb + threads / 32 - 1) / (threads / 32)); // a warp a block quantize_q8_0_scaled_kernel<<>>(x, blocks, scales, nb); const cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { From 96af377a19ac1914ab331bad2c967242f9778d6b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 07:55:19 +0200 Subject: [PATCH 83/92] The verify head across both GPUs After the last layer the head streamed the whole output matrix on the 3090 (IQ3_S's Q6_K 521 MB, UD-Q4_K_XL's Q8_0 675 MB). With --second-gpu the 5070 Ti now holds its last rows (--head-split, default 0.45 of them: 234 / 290 MiB, which its expert tier gives up and the 3090's cache gets back): the window's graph writes the head's input rows to mapped memory and raises a count, the host launches the 5070 Ti's part once it sees it (the same quantization, projection and argmax kernels, so every logit is the one-GPU head's), and keeps each token's larger logit, on equality the 3090's. argmax_rows can return the pick's logit. A sampled request or --window-logits gets whole rows: the 5070 Ti's part then copies its logits to mapped memory and the 3090 joins them to its own (join_rows). The drafter gathers its draft head's rows before the 3090 keeps only its part (MtpDrafter::bind_head); without a draft head the head stays whole. split_head.hpp/.cpp. With both GPUs' caches held at one size (--expert-cache 3000 --second-gpu-gib 10) the window hashes and the --window-logits output equal the previous build's (IQ3_S); with the configs the caches' sizes change with the head's. Fixed text, 2 runs each in both orders, t/s against d3d48b1 (at 0.5 / 0.45 of the rows on the 5070 Ti): IQ3_S code +5.3 / +5.8%, thinking +2.9 / +2.5% UD-Q4_K_XL code -6.3 / -0.4% (the 0.5 build ran second of three in both rounds, where UD's code runs are slowest: 106 against 110-117 t/s), thinking +1.4 / +2.7% The 5070 Ti's part starts ~30 us after the 3090's (the host sees the count, launches under WDDM): balanced near 0.46. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- CMakeLists.txt | 3 +- include/strata/core/mtp.hpp | 4 + include/strata/core/native_head.hpp | 6 + include/strata/core/split_head.hpp | 65 +++++++++ include/strata/core/verify.hpp | 13 ++ include/strata/kernels/verify_kernels.hpp | 8 +- src/core/mtp.cpp | 13 ++ src/core/native_head.cpp | 19 +++ src/core/split_head.cpp | 164 ++++++++++++++++++++++ src/core/verify.cpp | 54 ++++++- src/kernels/cuda/verify_kernels.cu | 30 +++- src/program/generate.cpp | 31 ++++ 12 files changed, 398 insertions(+), 12 deletions(-) create mode 100644 include/strata/core/split_head.hpp create mode 100644 src/core/split_head.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index d0314b3a6d2..e5e216b1094 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -242,7 +242,8 @@ if(STRATA_ENABLE_CUDA) # about arithmetic, and this one knows the ORDER the arithmetic happens in. add_library(strata_engine STATIC src/core/layer.cpp src/core/session.cpp src/core/expert_source.cpp src/core/expert_cache.cpp src/core/native_head.cpp src/core/native_dense.cpp src/core/verify.cpp src/core/mtp.cpp - src/core/prompt_cache.cpp src/core/adaptive_tier.cpp src/core/second_gpu.cpp) + src/core/prompt_cache.cpp src/core/adaptive_tier.cpp src/core/second_gpu.cpp + src/core/split_head.cpp) target_include_directories(strata_engine PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include) target_link_libraries(strata_engine PUBLIC strata_core strata_kernels strata_kernels_cpu CUDA::cudart) diff --git a/include/strata/core/mtp.hpp b/include/strata/core/mtp.hpp index ffb9a53e68d..0480c668076 100644 --- a/include/strata/core/mtp.hpp +++ b/include/strata/core/mtp.hpp @@ -80,6 +80,10 @@ class MtpDrafter { float* probs = nullptr, float min_p = 0.0f, int* n_drafts = nullptr); /// The first round: one cell (`cell`) from `R_row` (device) and `token` -> T-1 drafts. + /// The draft head's rows gathered from the main head now (before the main GPU keeps only part of it); `bind` + /// then takes them as they are. + bool bind_head(const NativeHead* head, std::string& err); + bool has_draft_head() const { return dhead_ != nullptr; } bool draft_first(int T, const float* R_row, int32_t token, int64_t cell, int32_t* drafts, std::string& err, float* probs = nullptr, float min_p = 0.0f, int* n_drafts = nullptr); diff --git a/include/strata/core/native_head.hpp b/include/strata/core/native_head.hpp index cf2bc09988c..8a02843027f 100644 --- a/include/strata/core/native_head.hpp +++ b/include/strata/core/native_head.hpp @@ -27,6 +27,11 @@ class NativeHead { int type() const { return type_; } /// Bytes of one vocabulary row. size_t row_bytes() const { return n_out_ > 0 ? (size_t) (bytes_ / (uint64_t) n_out_) : 0; } + /// The rows held: all of the vocabulary, or the first ones after keep_rows. + int64_t rows() const { return n_out_; } + /// Keeps rows [0, rows) (the rest of the head runs on another GPU: split_head.hpp) and frees the others' VRAM. + /// `run` then refuses: it would give part of a token's logits. + bool keep_rows(int64_t rows, std::string& err); private: void* weights_ = nullptr; @@ -34,6 +39,7 @@ class NativeHead { uint64_t bytes_ = 0; int n_in_ = 0, n_out_ = 0; int type_ = -1; + bool part_ = false; }; /// Plan v0.3 P6: `token_embd.weight` in its GGUF form (the IQ model files), in mapped pinned host memory: a row diff --git a/include/strata/core/split_head.hpp b/include/strata/core/split_head.hpp new file mode 100644 index 00000000000..3d45522c2b2 --- /dev/null +++ b/include/strata/core/split_head.hpp @@ -0,0 +1,65 @@ +// include/strata/core/split_head.hpp - the verify window's output head across two GPUs. +// +// The head streams the whole output matrix after the last layer (IQ3_S's Q6_K 521 MB, UD-Q4_K_XL's Q8_0 675 MB): the +// main GPU keeps rows [0, split) and a second GPU holds rows [split, n_vocab), so each streams part of it. The main +// GPU's window graph writes the head's input rows (the final hyper-connection mix, T x n_in floats) to mapped memory +// and raises a count; the host then launches this GPU's part (the same quantization, projection and per-row argmax +// kernels, so every logit is the one-GPU head's), which writes each token's pick and logit to mapped memory and raises +// a count of its own. The host keeps the larger logit, on equality the main GPU's (the lower index). It never waits +// on host memory on this GPU (a display may draw there). With `logits`, a window's rows are copied to mapped memory +// too, for sampled requests and --window-logits. +#pragma once + +#include + +#include +#include +#include + +namespace strata::core { + +class SplitHead { +public: + SplitHead() = default; + ~SplitHead(); + SplitHead(const SplitHead&) = delete; + SplitHead& operator=(const SplitHead&) = delete; + + /// Rows [split, n_vocab) of `output.weight` in the model's shards on `device`, for windows of up to `max_t` tokens. + /// `main_device` is current again after every call. + bool init(int device, int main_device, const std::vector& shards, int64_t n_in, int64_t n_vocab, + int64_t split, int max_t, std::string& err); + bool on() const { return dev_ >= 0; } + int64_t split() const { return split_; } + uint64_t bytes() const { return bytes_; } + /// Mapped: the head's input rows the main GPU writes (max_t x n_in floats) and the count it raises after them. + float* input() const { return h_in_; } + uint32_t* handoff() const { return h_hand_; } + /// Launches this GPU's part for a window of T tokens (after the handoff count has risen); `logits` copies its + /// rows to logits() too. + bool submit(int T, bool logits, std::string& err); + /// Until this GPU's count reaches `n` (the submits so far). + bool wait(uint32_t n, std::string& err); + /// Per token: the pick's row within this part, its logit; T x (n_vocab - split) logits after a `logits` submit. + const int32_t* idx() const { return h_idx_; } + const float* val() const { return h_val_; } + const float* logits() const { return h_logits_; } + +private: + bool capture(int T, bool logits, std::string& err); + + int dev_ = -1, main_ = 0, max_t_ = 0, type_ = -1; + int64_t n_in_ = 0, split_ = 0, rows_ = 0; + uint64_t bytes_ = 0; + cudaStream_t s_ = nullptr; + void* w_ = nullptr; // the rows' GGUF blocks + float *mixed_ = nullptr, *out_ = nullptr; // device: the input, the logits + uint8_t *xq_ = nullptr, *xil_ = nullptr, *scratch_ = nullptr; + float* h_in_ = nullptr; // mapped (portable) + uint32_t *h_hand_ = nullptr, *h_done_ = nullptr; + int32_t* h_idx_ = nullptr; + float *h_val_ = nullptr, *h_logits_ = nullptr; + cudaGraphExec_t exec_[9][2] = {}; +}; + +} // namespace strata::core diff --git a/include/strata/core/verify.hpp b/include/strata/core/verify.hpp index 27c240aded6..c8e42118c52 100644 --- a/include/strata/core/verify.hpp +++ b/include/strata/core/verify.hpp @@ -32,6 +32,7 @@ #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" #include "strata/core/session.hpp" +#include "strata/core/split_head.hpp" #include "strata/kernels/sampler.hpp" #include @@ -117,6 +118,12 @@ class Verifier { const float* final_R_host() const { return h_rows_; } /// The head's logits of the last window's first `T` tokens, T * n_vocab floats, copied to host memory `out`. bool copy_logits(int T, float* out, std::string& err) const; + /// The head's rows [split, n_vocab) on another GPU (`head` then holds rows [0, split)): the window hands the head's + /// input over, and `run` keeps each token's larger pick. Set before `init`. + void set_split_head(SplitHead* sh) { shead_ = sh; } + /// Whole logits rows after every window (copy_logits, --window-logits): with a split head the second GPU's part + /// then copies its rows back. Set before `init`. + void set_logits_wanted(bool on) { logits_wanted_ = on; } /// Where the pool publishes each layer's PCIe share of the misses; give it to the dispatch /// (`ExpertDispatch::plan`) before the first `run`. @@ -203,6 +210,12 @@ class Verifier { int32_t* h_commit_ = nullptr; int32_t* m_commit_ = nullptr; // [n_keep, n_keep-1, pos_0 .. pos_{T-1}] float* h_ple_ = nullptr; float* m_ple_ = nullptr; // T * n_embd int32_t* h_out_ = nullptr; int32_t* m_out_ = nullptr; // T argmax ids + float* h_val_ = nullptr; float* m_val_ = nullptr; // their logits (a split head's merge) + SplitHead* shead_ = nullptr; + bool logits_wanted_ = false; + int64_t head_rows_ = 0; // the head's rows here: n_vocab, or the split + float* head_full_ = nullptr; // T whole logits rows (a split head, when wanted) + uint32_t hand_n_ = 0, done_n_ = 0; // the split head's counts so far float* h_x_ = nullptr; float* m_x_ = nullptr; // doorbell payload: T * n_embd int32_t* h_ids_ = nullptr; int32_t* m_ids_ = nullptr; // T * k float* h_w_ = nullptr; float* m_w_ = nullptr; // T * k diff --git a/include/strata/kernels/verify_kernels.hpp b/include/strata/kernels/verify_kernels.hpp index fd9444ca53d..b837a103d4f 100644 --- a/include/strata/kernels/verify_kernels.hpp +++ b/include/strata/kernels/verify_kernels.hpp @@ -94,8 +94,14 @@ void row_top_prob(const float* logits, int n_rows, int n_vocab, const int32_t* i /// -inf): bitwise sample_tokens' greedy pick. Up to 128 blocks a row scan slices of it, the row's last block merges /// their picks. `scratch`: argmax_rows_scratch_bytes(n_rows), zero before the first launch (each launch leaves it /// so). Graph-capturable. +/// With `out_val` each pick's logit too (-inf when no value is above it). uint64_t argmax_rows_scratch_bytes(int n_rows); -void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream); +void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream, + float* out_val = nullptr); +/// *flag += 1 once the stream's earlier work is visible to the host (mapped memory). Graph-capturable. +void mapped_bump(uint32_t* flag, void* stream); +/// dst row t (n floats) = a row t (na floats) then b row t (n - na floats), for `rows` rows. +void join_rows(float* dst, int64_t n, const float* a, int64_t na, const float* b, int rows, void* stream); /// row_top_prob over 8 blocks a row, each computing 4 of its 32 warps' sums, the row's last block adding the 32 in /// order: bitwise row_top_prob. `scratch`: row_top_prob_scratch_bytes(n_rows), zero before the first launch. uint64_t row_top_prob_scratch_bytes(int n_rows); diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index 36d79b492ba..f3f42a3e29f 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -267,11 +267,24 @@ bool MtpDrafter::bind(const WeightTable& wt, const NativeHead* head, const float if (!wo) { err = "mtp: output.weight is missing"; return false; } n_vocab_ = wo->ne1; if (head == nullptr || !head->loaded()) { err = "mtp: the draft layer needs the native head (--native)"; return false; } + if (dhead_ == nullptr && !bind_head(head, err)) return false; + if (dhead_ == nullptr && head->rows() != n_vocab_) { + err = "mtp: the main head holds part of the vocabulary and the draft layer has no draft head of its own"; + return false; + } if (head_logits_ == nullptr && cudaMalloc((void**) &head_logits_, (size_t) max_t_ * (size_t) n_vocab_ * sizeof(float)) != cudaSuccess) { err = "mtp: the draft logits do not fit"; return false; } + return true; +} + +bool MtpDrafter::bind_head(const NativeHead* head, std::string& err) { + if (head == nullptr || !head->loaded()) { + err = "mtp: the draft layer needs the native head (--native)"; + return false; + } // the draft head's token subset, when tools/draft_vocab.py wrote one if (dhead_ == nullptr) { std::vector raw; diff --git a/src/core/native_head.cpp b/src/core/native_head.cpp index e5fb95fa3d7..e77dff0d753 100644 --- a/src/core/native_head.cpp +++ b/src/core/native_head.cpp @@ -15,6 +15,24 @@ NativeHead::~NativeHead() { if (weights_) cudaFree(weights_); } +bool NativeHead::keep_rows(int64_t rows, std::string& err) { + if (!loaded() || rows <= 0 || rows >= n_out_) { err = "native head: keep_rows out of range"; return false; } + const uint64_t bytes = (uint64_t) rows * row_bytes(); + void* w = nullptr; + if (cudaMalloc(&w, bytes) != cudaSuccess || + cudaMemcpy(w, weights_, bytes, cudaMemcpyDeviceToDevice) != cudaSuccess) { + if (w) cudaFree(w); + err = "native head: keep_rows: " + std::string(cudaGetErrorString(cudaGetLastError())); + return false; + } + cudaFree(weights_); + weights_ = w; + bytes_ = bytes; + n_out_ = (int) rows; + part_ = true; + return true; +} + bool NativeHead::load(const std::vector& shards, int64_t n_in, int64_t n_out, std::string& err) { if (loaded()) { err = "native head is already loaded"; return false; } if (n_in <= 0 || n_out <= 0 || n_in > INT_MAX || n_out > INT_MAX || n_in % 256) { @@ -66,6 +84,7 @@ bool NativeHead::load(const std::vector& shards, int64_t n_in, int6 } bool NativeHead::run(const float* mixed, float* logits, void* stream, std::string& err) const { + if (part_) { err = "native head: it holds part of the vocabulary (the rest is on the second GPU)"; return false; } if (!loaded() || !mixed || !logits || !stream) { err = "native head requires loaded weights, device buffers and an explicit stream"; return false; diff --git a/src/core/split_head.cpp b/src/core/split_head.cpp new file mode 100644 index 00000000000..db45c9e7e72 --- /dev/null +++ b/src/core/split_head.cpp @@ -0,0 +1,164 @@ +// src/core/split_head.cpp - see include/strata/core/split_head.hpp. +#include "strata/core/split_head.hpp" + +#include "strata/artifact/gguf_reader.hpp" +#include "strata/kernels/elementwise.hpp" +#include "strata/kernels/native_mmvq.hpp" +#include "strata/kernels/verify_kernels.hpp" + +#include + +#include +#include + +namespace strata::core { +namespace { + +struct DeviceScope { // this GPU current for one call, the main one again afterwards + int main; + DeviceScope(int dev, int main_dev) : main(main_dev) { cudaSetDevice(dev); } + ~DeviceScope() { cudaSetDevice(main); } +}; + +} // namespace + +SplitHead::~SplitHead() { + if (dev_ < 0) return; + DeviceScope scope(dev_, main_); + if (s_) cudaStreamSynchronize(s_); + for (auto& e : exec_) + for (auto& x : e) if (x) cudaGraphExecDestroy(x); + void* dev[] = {w_, mixed_, out_, xq_, xil_, scratch_}; + for (void* p : dev) if (p) cudaFree(p); + void* host[] = {h_in_, h_hand_, h_done_, h_idx_, h_val_, h_logits_}; + for (void* p : host) if (p) cudaFreeHost(p); + if (s_) cudaStreamDestroy(s_); +} + +bool SplitHead::init(int device, int main_device, const std::vector& shards, int64_t n_in, + int64_t n_vocab, int64_t split, int max_t, std::string& err) { + if (split <= 0 || split >= n_vocab || max_t < 1 || max_t > strata::kernels::kVerifyMaxT) { + err = "split head: bad arguments"; + return false; + } + dev_ = device; + main_ = main_device; + max_t_ = max_t; + n_in_ = n_in; + split_ = split; + rows_ = n_vocab - split; + DeviceScope scope(dev_, main_); + try { + const strata::GgufModel model(shards); + size_t at = 0; + const strata::TensorInfo* t = model.find("output.weight", &at); + if (!t || !strata::kernels::native_mmvq_supported((int) t->type)) { + err = "split head: output.weight is missing or not a native format"; + return false; + } + type_ = (int) t->type; + const uint64_t row = strata::kernels::native_mmvq_weight_bytes(type_, (int) n_in, 1); + bytes_ = row * (uint64_t) rows_; + const unsigned pm = cudaHostAllocPortable | cudaHostAllocMapped; + const size_t T = (size_t) max_t; + cudaError_t e = cudaSuccess; + const char* step = nullptr; + auto run = [&](const char* what, cudaError_t r) { + if (e == cudaSuccess && r != cudaSuccess) { e = r; step = what; } + }; + run("context", cudaFree(nullptr)); + run("stream", cudaStreamCreateWithFlags(&s_, cudaStreamNonBlocking)); + run("its rows", cudaMalloc(&w_, bytes_)); + run("its rows", cudaMemcpy(w_, model.shard(at).tensor_data(*t) + (uint64_t) split * row, bytes_, + cudaMemcpyHostToDevice)); + run("buffers", cudaMalloc((void**) &mixed_, T * (size_t) n_in * 4)); + run("buffers", cudaMalloc((void**) &out_, T * (size_t) rows_ * 4)); + run("buffers", cudaMalloc((void**) &xq_, strata::kernels::native_q8_1_bytes((int) n_in, max_t))); + run("buffers", cudaMalloc((void**) &xil_, strata::kernels::native_q8_1_il_bytes((int) n_in, max_t))); + run("buffers", cudaMalloc((void**) &scratch_, strata::kernels::argmax_rows_scratch_bytes(max_t))); + run("buffers", cudaMemset(scratch_, 0, strata::kernels::argmax_rows_scratch_bytes(max_t))); + run("mapped", cudaHostAlloc((void**) &h_in_, T * (size_t) n_in * 4, pm)); + run("mapped", cudaHostAlloc((void**) &h_hand_, 64, pm)); + run("mapped", cudaHostAlloc((void**) &h_done_, 64, pm)); + run("mapped", cudaHostAlloc((void**) &h_idx_, T * 4 + 64, pm)); + run("mapped", cudaHostAlloc((void**) &h_val_, T * 4 + 64, pm)); + run("mapped", cudaHostAlloc((void**) &h_logits_, T * (size_t) rows_ * 4, pm)); + if (e != cudaSuccess) { + err = std::string("split head: ") + step + ": " + cudaGetErrorString(e); + return false; + } + *h_hand_ = 0; + *h_done_ = 0; + } catch (const std::exception& x) { + err = std::string("split head: ") + x.what(); + return false; + } + return true; +} + +// One window's part: the input from mapped memory, quantized as the main GPU's head does, the rows' projection, the +// per-token argmax (and logit) to mapped memory, the count raised. +bool SplitHead::capture(int T, bool logits, std::string& err) { + using namespace strata::kernels; + if (exec_[T][logits]) return true; + if (cudaStreamBeginCapture(s_, cudaStreamCaptureModeThreadLocal) != cudaSuccess) { + err = "split head: begin capture"; + return false; + } + try { + copy_from_mapped(mixed_, h_in_, (int64_t) T * n_in_, s_); + if (T >= 2) { + native_quantize_q8_1_il(mixed_, xq_, xil_, (int) n_in_, T, s_); + native_mmvq_il(type_, w_, xq_, xil_, out_, (int) n_in_, (int) rows_, T, s_); + } else { + native_quantize_q8_1(mixed_, xq_, (int) n_in_, T, s_); + native_mmvq(type_, w_, xq_, out_, (int) n_in_, (int) rows_, T, s_); + } + argmax_rows(out_, T, (int) rows_, scratch_, h_idx_, s_, h_val_); + if (logits) copy_from_mapped(h_logits_, out_, (int64_t) T * rows_, s_); + mapped_bump(h_done_, s_); + } catch (const std::exception& x) { + cudaGraph_t g = nullptr; + cudaStreamEndCapture(s_, &g); + if (g) cudaGraphDestroy(g); + err = std::string("split head: ") + x.what(); + return false; + } + cudaGraph_t g = nullptr; + if (cudaStreamEndCapture(s_, &g) != cudaSuccess || cudaGraphInstantiate(&exec_[T][logits], g, 0) != cudaSuccess) { + if (g) cudaGraphDestroy(g); + err = "split head: capture failed"; + return false; + } + cudaGraphDestroy(g); + cudaGraphUpload(exec_[T][logits], s_); + cudaStreamSynchronize(s_); + return true; +} + +bool SplitHead::submit(int T, bool logits, std::string& err) { + if (T < 1 || T > max_t_) { err = "split head: window size out of range"; return false; } + DeviceScope scope(dev_, main_); + if (!capture(T, logits, err)) return false; + if (cudaGraphLaunch(exec_[T][logits], s_) != cudaSuccess) { + err = std::string("split head: launch: ") + cudaGetErrorString(cudaGetLastError()); + return false; + } + (void) cudaStreamQuery(s_); // submit now + return true; +} + +bool SplitHead::wait(uint32_t n, std::string& err) { + const auto t0 = std::chrono::steady_clock::now(); + while (*(volatile uint32_t*) h_done_ < n) { + _mm_pause(); + if (std::chrono::steady_clock::now() - t0 > std::chrono::seconds(5)) { + DeviceScope scope(dev_, main_); + err = std::string("split head: its part never finished (") + cudaGetErrorString(cudaStreamQuery(s_)) + ")"; + return false; + } + } + return true; +} + +} // namespace strata::core diff --git a/src/core/verify.cpp b/src/core/verify.cpp index c46be3df039..8acf3e58f47 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -110,7 +110,8 @@ Verifier::~Verifier() { if (e) cudaEventDestroy(e); if (arena_) cudaFree(arena_); if (stamps_) cudaFree(stamps_); - void* hosts[] = {h_tok_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, h_ymiss_, + void* hosts[] = {h_tok_, h_val_, h_step_, h_pos_, h_commit_, h_ple_, h_out_, h_x_, h_ids_, h_w_, h_seq_, h_flag_, + h_ymiss_, h_flagA_, h_plan_, h_flagB_, h_pids_, h_pw_, h_pseq_, h_pleflag_, h_rows_, h_res_, h_flag2_, h_list2_, h_xk_, h_xq1_}; for (void* h : hosts) @@ -158,6 +159,11 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& attn_scratch_floats_ = (int64_t) strata::kernels::qsa_decode_attn_scratch_floats(cap_, s); const uint64_t T = (uint64_t) max_t, N = (uint64_t) g.n_embd, HC = (uint64_t) g.hc, K = (uint64_t) ss.k; + head_rows_ = (head != nullptr && head->loaded()) ? head->rows() : n_vocab_; + if (head_rows_ != n_vocab_ && (shead_ == nullptr || shead_->split() != head_rows_)) { + err = "verify: the head holds part of the vocabulary without its other part"; + return false; + } const uint64_t C = (uint64_t) g.ssm_conv_channels, ZV = (uint64_t) g.ssm_value_dim, HV = (uint64_t) g.ssm_v_heads; const uint64_t NH = (uint64_t) g.n_head, HD = (uint64_t) g.head_dim, NKV = (uint64_t) g.n_head_kv; const uint64_t IQ = (uint64_t) g.idx_q_heads, ID = (uint64_t) g.idx_key_dim; @@ -181,6 +187,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& mapped((2 + T) * 4 + 16, (void**) &h_commit_, (void**) &m_commit_) && mapped(T * N * 4, (void**) &h_ple_, (void**) &m_ple_) && mapped(T * 4 + 16, (void**) &h_out_, (void**) &m_out_) && + mapped(T * 4 + 16, (void**) &h_val_, (void**) &m_val_) && mapped(T * N * 4, (void**) &h_x_, (void**) &m_x_) && mapped(T * K * 4, (void**) &h_ids_, (void**) &m_ids_) && mapped(T * K * 4, (void**) &h_w_, (void**) &m_w_) && @@ -274,6 +281,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState& sh_up_ = b.take(T * (uint64_t) g.n_ff); sh_g_ = b.take(T + 4); head_logits_ = b.take(T * (uint64_t) n_vocab_); arg_scratch_ = b.take(strata::kernels::argmax_rows_scratch_bytes((int) T)); + if (shead_ != nullptr) head_full_ = b.take(T * (uint64_t) n_vocab_); one_ = b.take(4); hist_snap_ = b.take(T * HS); const uint64_t PD = (uint64_t) strata::kernels::NG_HC_DIM; @@ -958,19 +966,24 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { } if (head_ != nullptr && head_->loaded()) { try { + if (shead_ != nullptr) { // the second GPU's part starts once the host sees the count + copy_from_mapped(shead_->input(), head_mixed_, (int64_t) T * N, cs); + mapped_bump(shead_->handoff(), cs); + } if (T >= 2) { native_quantize_q8_1_il(head_mixed_, xq_, xil_, (int) N, T, cs); - native_mmvq_il(head_->type(), head_->weights(), xq_, xil_, head_logits_, (int) N, (int) n_vocab_, T, cs); + native_mmvq_il(head_->type(), head_->weights(), xq_, xil_, head_logits_, (int) N, (int) head_rows_, + T, cs); } else { native_quantize_q8_1(head_mixed_, xq_, (int) N, T, cs); - native_mmvq(head_->type(), head_->weights(), xq_, head_logits_, (int) N, (int) n_vocab_, T, cs); + native_mmvq(head_->type(), head_->weights(), xq_, head_logits_, (int) N, (int) head_rows_, T, cs); } } catch (const std::exception& e) { err = std::string("verify head: ") + e.what(); return false; } } - argmax_rows(head_logits_, T, (int) n_vocab_, arg_scratch_, m_out_, cs); + argmax_rows(head_logits_, T, (int) head_rows_, arg_scratch_, m_out_, cs, m_val_); } if (host_rows_) copy_from_mapped(m_rows_, R_, (int64_t) T * HC * N, cs); // a plain copy kernel, here to host stamp(g.n_layers, 3, cs); @@ -1255,14 +1268,39 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, _mm_pause(); } settle(true); + const bool sampled = head_sampling_ && ((!sampling_.greedy && sampling_.temperature > 0.0f) || hist_d_ != nullptr); + const bool whole = shead_ != nullptr && (logits_wanted_ || sampled); + if (shead_ != nullptr) { // the head's input is handed over after the last layer: the second GPU's part then + ++hand_n_; + const Clock::time_point h0 = Clock::now(); + while (*(volatile uint32_t*) shead_->handoff() < hand_n_) { + _mm_pause(); + if (ms_since(h0) > 5000.0) { + err = std::string("verify: the head's input never came (") + cudaGetErrorString(cudaStreamQuery(cs_)) + + ")"; + return false; + } + } + if (!shead_->submit(T, whole, err)) return false; + ++done_n_; + } const cudaError_t se = cudaStreamSynchronize(cs_); if (se != cudaSuccess) { err = std::string("verify: ") + cudaGetErrorString(se); return false; } cudaStreamSynchronize(copy_); // no host function of this window may raise flag B in the next one + if (shead_ != nullptr) { // each token's larger logit; on equality this GPU's, the lower index + if (!shead_->wait(done_n_, err)) return false; + for (int t = 0; t < T; ++t) { + const float v1 = ((volatile float*) h_val_)[t], v2 = ((volatile const float*) shead_->val())[t]; + if (v2 > v1) ((volatile int32_t*) h_out_)[t] = (int32_t) (shead_->split() + shead_->idx()[t]); + } + if (whole) join_rows(head_full_, n_vocab_, head_logits_, head_rows_, shead_->logits(), T, cs_); + } // a sampled or penalized request: the head's sampling again, after the graph (see set_sampling) - if (head_sampling_ && ((!sampling_.greedy && sampling_.temperature > 0.0f) || hist_d_ != nullptr)) { + if (sampled) { SamplerParams sp = sampling_; sp.counter = (uint64_t) pos0; - sample_tokens(head_logits_, T, (int) n_vocab_, hist_d_, hist_len_, sp, m_out_, cs_); + sample_tokens(shead_ != nullptr ? head_full_ : head_logits_, T, (int) n_vocab_, hist_d_, hist_len_, sp, m_out_, + cs_); if (cudaStreamSynchronize(cs_) != cudaSuccess) { err = "verify: the head sampling failed"; return false; } } for (int t = 0; t < T; ++t) out[t] = ((volatile int32_t*) h_out_)[t]; @@ -1302,7 +1340,9 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool, } bool Verifier::copy_logits(int T, float* out, std::string& err) const { - if (cudaMemcpyAsync(out, head_logits_, (size_t) T * (size_t) n_vocab_ * sizeof(float), cudaMemcpyDeviceToHost, + if (shead_ != nullptr && !logits_wanted_) { err = "verify: the logits need set_logits_wanted"; return false; } + const float* src = shead_ != nullptr ? head_full_ : head_logits_; + if (cudaMemcpyAsync(out, src, (size_t) T * (size_t) n_vocab_ * sizeof(float), cudaMemcpyDeviceToHost, cs_) != cudaSuccess || cudaStreamSynchronize(cs_) != cudaSuccess) { err = std::string("verify: reading the logits back: ") + cudaGetErrorString(cudaGetLastError()); diff --git a/src/kernels/cuda/verify_kernels.cu b/src/kernels/cuda/verify_kernels.cu index f6d350631d4..786937f07f3 100644 --- a/src/kernels/cuda/verify_kernels.cu +++ b/src/kernels/cuda/verify_kernels.cu @@ -470,7 +470,7 @@ struct ArgRow { // grid (blocks, rows): block b scans [b * per_block, +per_block) of its row __global__ void argmax_rows_kernel(const float* __restrict__ logits, int n, int per_block, ArgRow* __restrict__ rows, - int32_t* out) { + int32_t* out, float* out_val) { const int row = (int) blockIdx.y, b = (int) blockIdx.x, nb = (int) gridDim.x; unsigned* counter = &rows[row].counter; float* pv = rows[row].v; @@ -503,10 +503,24 @@ __global__ void argmax_rows_kernel(const float* __restrict__ logits, int n, int arg_block(bv, bi, n); if (threadIdx.x == 0) { out[row] = bi < n ? bi : 0; // no value above -inf: 0, as the one-block kernel + if (out_val != nullptr) out_val[row] = bv; *counter = 0; } } +__global__ void mapped_bump_kernel(uint32_t* flag) { + __threadfence_system(); + *(volatile uint32_t*) flag = *(volatile uint32_t*) flag + 1u; +} + +__global__ void join_rows_kernel(float* __restrict__ dst, int64_t n, const float* __restrict__ a, int64_t na, + const float* b) { + const int64_t t = blockIdx.y; + const int64_t nb = n - na; + for (int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; i < n; i += (int64_t) gridDim.x * blockDim.x) + dst[t * n + i] = i < na ? a[t * na + i] : ((const volatile float*) b)[t * nb + i - na]; +} + // a row's top-probability scratch: its counter, then its 32 warps' sums struct TopRow { unsigned counter; @@ -543,17 +557,27 @@ __global__ void row_top_prob_split_kernel(const float* __restrict__ logits, int uint64_t argmax_rows_scratch_bytes(int n_rows) { return (uint64_t) n_rows * sizeof(ArgRow); } -void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream) { +void argmax_rows(const float* logits, int n_rows, int n, void* scratch, int32_t* out, void* stream, float* out_val) { if (n_rows <= 0) return; const int nb = std::min(kArgMaxBlocks, std::max(1, (n + 4095) / 4096)); const int per_block = (n + nb - 1) / nb; argmax_rows_kernel<<>>( - logits, n, per_block, (ArgRow*) scratch, out); + logits, n, per_block, (ArgRow*) scratch, out, out_val); check("argmax_rows"); } uint64_t row_top_prob_scratch_bytes(int n_rows) { return (uint64_t) n_rows * sizeof(TopRow); } +void mapped_bump(uint32_t* flag, void* stream) { + mapped_bump_kernel<<<1, 1, 0, (cudaStream_t) stream>>>(flag); + check("mapped_bump"); +} + +void join_rows(float* dst, int64_t n, const float* a, int64_t na, const float* b, int rows, void* stream) { + join_rows_kernel<<>>(dst, n, a, na, b); + check("join_rows"); +} + void row_top_prob_split(const float* logits, int n_rows, int n_vocab, const int32_t* ids, float* probs, void* scratch, void* stream) { if (n_rows <= 0) return; diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 4bdbca4c4b3..c03fded4742 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -18,6 +18,7 @@ #include "strata/core/adaptive_tier.hpp" #include "strata/core/expert_cache.hpp" #include "strata/core/second_gpu.hpp" +#include "strata/core/split_head.hpp" #include "strata/core/expert_source.hpp" #include "strata/core/layer.hpp" #include "strata/core/layout.hpp" @@ -68,6 +69,7 @@ #include #include #include +#include #include #include #include @@ -240,6 +242,7 @@ struct Options { int second_gpu_reserve_mib = 2048; double second_gpu_min_mb = 4.0; ///< a layer's misses from which it takes its share (smaller: the CPU is quicker) double second_gpu_prefetch_mb = 12.0; ///< per layer, the likeliest experts no GPU holds copied to it ahead (0 = off) + double head_split = 0.45; ///< the second GPU's share of the head's rows (0: all on the main GPU) /// Plan v0.3 P6: the share (0..1) of each layer's distinct missed experts the GPU reads over PCIe from the /// pinned arena while the CPU computes the rest (verify windows). double pcie_frac = -1.0; ///< < 0: the model's default (0.2 direct for the Q2_0 pack, 0.55 DMA for native packs) @@ -370,6 +373,8 @@ void usage() { " --second-gpu-prefetch-mb M per layer, copy its likeliest experts that no GPU holds (the next\n" " layer's router on this layer's input), up to M MiB, to it while the RAM is idle\n" " (default 12: ~0.28 ms at its ~48 GB/s; 0 = off)\n" + " --head-split F with --second-gpu: its share of the output head's rows (default 0.45;\n" + " each GPU streams its part after the last layer; 0 = all on the main GPU)\n" " --no-prompt-offload with --second-gpu: the prompt path streams the experts the main GPU's cache\n" " lacks to the main GPU instead of computing them on the second\n" " --spec-follow PATH benchmarks: emit this continuation (token ids) instead of the argmax and\n" @@ -889,6 +894,7 @@ int main(int argc, char** argv) { else if (a == "--second-gpu-reserve-mib") o.second_gpu_reserve_mib = std::atoi(next("--second-gpu-reserve-mib")); else if (a == "--second-gpu-min-mb") o.second_gpu_min_mb = std::atof(next("--second-gpu-min-mb")); else if (a == "--second-gpu-prefetch-mb") o.second_gpu_prefetch_mb = std::atof(next("--second-gpu-prefetch-mb")); + else if (a == "--head-split") o.head_split = std::atof(next("--head-split")); else if (a == "--pcie-frac") o.pcie_frac = std::atof(next("--pcie-frac")); else if (a == "--adapt-every") o.adapt_every = std::atoi(next("--adapt-every")); else if (a == "--spec-min-p") o.spec_min_p = std::atof(next("--spec-min-p")); @@ -1413,6 +1419,7 @@ int main(int argc, char** argv) { if (wo == nullptr) { std::fprintf(stderr, "strata generate: output.weight is missing\n"); return 1; } const int64_t n_vocab = wo->ne1; strata::core::NativeHead native_head; + std::unique_ptr split_head; if (!o.native_head_gguf.empty()) { std::vector head_shards; try { @@ -1427,6 +1434,27 @@ int main(int argc, char** argv) { } std::fprintf(stderr, "strata generate: native head, type %d, %llu bytes\n", native_head.type(), (unsigned long long) native_head.weight_bytes()); + // the head's last rows on the second GPU (split_head.hpp): each GPU streams its part after the last layer. + // The drafter gathers its draft head's rows first; without one it needs the whole head here. + if (o.second_gpu >= 0 && o.head_split > 0.0 && o.head_split < 1.0 && o.spec >= 2) { + const int64_t split = (int64_t) ((1.0 - o.head_split) * (double) n_vocab) / 256 * 256; + auto sh = std::make_unique(); + std::string e2; + const bool ok = (o.mtp.empty() || mtp.bind_head(&native_head, e2)) && + (o.mtp.empty() || mtp.has_draft_head()) && + sh->init(o.second_gpu, o.main_gpu, head_shards, g.n_embd, n_vocab, split, o.max_window(), + e2) && + native_head.keep_rows(split, e2); + if (ok) { + std::fprintf(stderr, "strata generate: the head's rows %lld-%lld on the second GPU (%.0f MiB), the " + "first %lld here\n", (long long) split, (long long) n_vocab, + (double) sh->bytes() / 1048576.0, (long long) split); + split_head = std::move(sh); + } else { + std::fprintf(stderr, "strata generate: the head stays on the main GPU: %s\n", + e2.empty() ? "the draft layer has no draft head of its own" : e2.c_str()); + } + } } // ---- R4's slot storage. Allocated AFTER the weights, the session and the head, so `cudaMemGetInfo` inside // `open` sees the memory this process actually has left rather than the card's idle figure - and refuses with @@ -2432,6 +2460,7 @@ int main(int argc, char** argv) { vh.slot_off = xcache.slot_offsets(); vh.slots = xcache.slots(); vh.blob = thits.blob; + ver.set_split_head(split_head.get()); if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.max_window(), err) || !mtp.bind(wt, &native_head, ver.final_R_all(), err)) { std::fprintf(stderr, "strata serve: %s\n", err.c_str()); @@ -3285,6 +3314,8 @@ int main(int argc, char** argv) { vh.slot_off = xcache.slot_offsets(); vh.slots = xcache.slots(); vh.blob = thits.blob; + ver.set_split_head(split_head.get()); + ver.set_logits_wanted(!o.window_logits.empty()); if (!ver.init(wt, g, ss, vh, native_head.loaded() ? &native_head : nullptr, o.max_window(), err)) { std::fprintf(stderr, "strata generate: %s\n", err.c_str()); return 1; From 2ec7a80688a0901edbb7752267f026ea6c55bede Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 08:53:50 +0200 Subject: [PATCH 84/92] The final mixer in one fused read The hyper-connection read before the head ran gr_read's kernels once per token of the window (lm_head_mix), and once per draft pass in the draft layer. Both now go through fused_gr_read_multi, the layers' read (no injection weight: the final mixer), the window's tokens in one launch. Its sums are gr_read's: window hashes on fixed text and the distributions along llama.cpp's texts (scratch/parity_new.sh: IQ3_S and UD-Q4_K_XL, code, thinking, after 16K) equal the previous build's. CUPTI, IQ3_S thinking text (--adapt-every 0): the time between windows 3,679 -> 3,578 us a round, each draft step 6-9 us shorter. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- src/core/mtp.cpp | 16 +++++++++++----- src/core/verify.cpp | 25 ++++++++++++++++++------- 2 files changed, 29 insertions(+), 12 deletions(-) diff --git a/src/core/mtp.cpp b/src/core/mtp.cpp index f3f42a3e29f..5b443cdbf7a 100644 --- a/src/core/mtp.cpp +++ b/src/core/mtp.cpp @@ -478,11 +478,17 @@ bool MtpDrafter::record_rest(int step_row, cudaStream_t cs, std::string& err) { gr_write(R_ + (size_t) t * HC * N, y_ + t * N, inj2_ + t * HC, gs, R_ + (size_t) t * HC * N, cs); } // ---- the final mixer and the main model's head - for (int t = 0; t < T; ++t) - gr_read(R_ + (size_t) t * HC * N, f32("hyper_connection_mixer.hc_norm.weight"), - bf16("hyper_connection_mixer.input_mix_weight_down.weight"), - bf16("hyper_connection_mixer.input_mix_weight_up.weight"), nullptr, EPS, gs, ss.block.gr, - sample_ + t * N, dummy_inj_, cs); + { + FusedGrArgs fa[kFusedGrMaxT]; + for (int t = 0; t < T; ++t) { + fa[t].R = R_ + (size_t) t * HC * N; fa[t].R_out = R_ + (size_t) t * HC * N; fa[t].apply = false; + fa[t].w_norm = f32("hyper_connection_mixer.hc_norm.weight"); + fa[t].w_down = bf16("hyper_connection_mixer.input_mix_weight_down.weight"); + fa[t].w_up = bf16("hyper_connection_mixer.input_mix_weight_up.weight"); + fa[t].eps = EPS; fa[t].lo = lo_ + t * g.hc_lr; fa[t].rs = rs_ + t * HC; fa[t].mixed = sample_ + t * N; + } + fused_gr_read_multi(fa, T, grs_, cs); + } native_quantize_q8_1(sample_, xq_, (int) N, T, cs); const bool sub = dhead_ != nullptr; const int64_t nv = sub ? n_dvocab_ : n_vocab_; diff --git a/src/core/verify.cpp b/src/core/verify.cpp index 8acf3e58f47..ddbcd88906b 100644 --- a/src/core/verify.cpp +++ b/src/core/verify.cpp @@ -954,15 +954,26 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) { const WeightRef *hn = wt.find("output_hc_norm.weight"), *hd = wt.find("output_hc_down.weight"), *hu = wt.find("output_hc_up.weight"); if (!hn || !hd || !hu) { err = "verify: an output_hc_* weight is missing"; return false; } - for (int t = 0; t < T; ++t) { - BlockBuffers bb = ss.block; - bb.R = Rt(t); - bb.mixed = head_mixed_ + t * N; - if (head_ != nullptr && head_->loaded()) { - if (!lm_head_mix(wt, g, bb, cs, err)) return false; - } else if (!lm_head(wt, g, bb, head_logits_ + (size_t) t * n_vocab_, cs, err)) { + if (head_ != nullptr && head_->loaded()) { // the final mixer, the window's tokens in one read + if (hn->kind != WeightKind::F32 || hd->kind != WeightKind::Bf16InF32 || hu->kind != WeightKind::Bf16InF32) { + err = "verify: the output_hc_* weights have the wrong engine forms"; return false; } + FusedGrArgs fa[kFusedGrMaxT]; + for (int t = 0; t < T; ++t) { + fa[t].R = Rt(t); fa[t].R_out = Rt(t); fa[t].apply = false; + fa[t].w_norm = (const float*) hn->data; fa[t].w_down = (const uint16_t*) hd->data; + fa[t].w_up = (const uint16_t*) hu->data; fa[t].eps = EPS; + fa[t].lo = lo_ + t * g.hc_lr; fa[t].rs = rs_ + t * HC; fa[t].mixed = head_mixed_ + t * N; + } + fused_gr_read_multi(fa, T, grs_, cs); + } else { + for (int t = 0; t < T; ++t) { + BlockBuffers bb = ss.block; + bb.R = Rt(t); + bb.mixed = head_mixed_ + t * N; + if (!lm_head(wt, g, bb, head_logits_ + (size_t) t * n_vocab_, cs, err)) return false; + } } if (head_ != nullptr && head_->loaded()) { try { From a06a7b340dde9b476f590763a63c7da65750bec0 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 11:21:52 +0200 Subject: [PATCH 85/92] Prompt path: the hyper-connection reads without their FP32 rows, the mixer's write with the FFN norm Upstream's F-1 and F-2 (882bb6d, b046845) in this prompt path. A read normalized its rows into an FP32 copy for the mix and a BF16 one for the projections: gr_norm_rs now writes the row scales and the BF16 image only, and gr_mix_r recomputes each normalized value from the residual rows, in gr_norm's order. The mixer half's write runs with the FFN half's norm (gr_write_norm_rs: one pass over the rows, the reduction as gr_norm's), so the FFN read starts at its projections. The FFN half's write covers the whole chunk after the experts while the next layer's reads run per 2048-token sub-chunk, so it stays apart; the draft layer's prompt pass keeps the FP32 rows (its up projection overwrites its source rows). Window hashes after the 16K prompt equal the previous build's (both models). 16K-token prompt (bench strata_gen.sh, alternating builds): IQ3_S with both GPUs 7,816 -> 7,638 ms (-2.3%, 3 runs each), on the 3090 alone 11,691 -> 11,633 ms (-0.5%: there the dense steps overlap the expert copies); UD-Q4_K_XL with both GPUs 8,399 -> 8,163 ms (-2.8%, 2 runs each). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/prefill/kernels.hpp | 8 +++ src/prefill/kernels.cu | 80 ++++++++++++++++++++++++++++++ src/prefill/prefill.cpp | 19 ++++--- 3 files changed, 100 insertions(+), 7 deletions(-) diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index bcf078cb97b..288e1f539c0 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -12,6 +12,14 @@ namespace strata::prefill { // ---- hyper-connection (n_embd 2560, hc 4, hc_lr 320) /// xn[t, c*2560 + d] = R[t,c,d] * rsqrt(mean_d R[t,c,:]^2 + eps) * w_norm[c*2560 + d]; also its BF16 image. void gr_norm(const float* R, const float* w_norm, float eps, float* xn, uint16_t* xn16, int64_t T, void* stream); +/// gr_norm without the FP32 rows: the row scales rs[t*4 + c] and the BF16 image; gr_mix_r reads R instead. +void gr_norm_rs(const float* R, const float* w_norm, float eps, float* rs, uint16_t* xn16, int64_t T, void* stream); +/// gr_mix with xn recomputed from R, rs and w_norm as gr_norm computes it (the same bits). +void gr_mix_r(const float* R, const float* rs, const float* w_norm, const float* gated, float* mixed, uint16_t* mixed16, + int64_t T, void* stream, uint16_t* mixed_h = nullptr); +/// gr_write (no partial), then gr_norm_rs of the written rows with the next read's norm weights, R read once. +void gr_write_norm_rs(float* R, const float* bo, const float* inj, int64_t inj_ld, const float* w_norm, float eps, + float* rs, uint16_t* xn16, int64_t T, void* stream); /// lo16[t, k] = bf16(silu(lo[t, k] / hc)) void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream); /// mixed[t, d] = mean_c xn[t, c, d] * sigmoid(gated[t, c, d]); FP32, BF16 and FP16 (any of them may be null). diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index c086a3a1da5..95a2a3683f4 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -71,6 +71,70 @@ __global__ void gr_norm_kernel(const float* __restrict__ R, const float* __restr xn16[row * N + d] = bf(v); } } +// gr_norm_kernel's row scale and BF16 image; gr_mix_r_kernel recomputes r * rs * w in the same order, so the FP32 +// copy of the normalized rows is neither written nor read +__global__ void gr_norm_rs_kernel(const float* __restrict__ R, const float* __restrict__ w, float eps, + float* __restrict__ rs_out, uint16_t* __restrict__ xn16) { + __shared__ float sh[32]; + const int64_t row = blockIdx.x; // t * 4 + c + const int c = (int) (row % HC); + const float* r = R + row * N; + float ss = 0.0f; + for (int d = threadIdx.x; d < N; d += blockDim.x) ss += r[d] * r[d]; + const float rs = rsqrtf(block_sum(ss, sh) / (float) N + eps); + if (threadIdx.x == 0) rs_out[row] = rs; + for (int d = threadIdx.x; d < N; d += blockDim.x) xn16[row * N + d] = bf(r[d] * rs * w[c * N + d]); +} +__global__ void gr_mix_r_kernel(const float* __restrict__ R, const float* __restrict__ rs, const float* __restrict__ w, + const float* __restrict__ g, float* __restrict__ mixed, uint16_t* __restrict__ mixed16, + int64_t T, uint16_t* __restrict__ mixed_h) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (i >= T * N) return; + const int64_t t = i / N, d = i % N; + float s = 0.0f; +#pragma unroll + for (int c = 0; c < HC; ++c) { + const int64_t j = t * D + c * N + d; + const float x = R[j] * rs[t * HC + c] * w[c * N + d]; // gr_norm_kernel's value, bit for bit + s = fmaf(x, sigm(g[j]), s); + } + s /= (float) HC; + if (mixed) mixed[i] = s; + if (mixed16) mixed16[i] = bf(s); + if (mixed_h) mixed_h[i] = hf(s); +} +// gr_write_kernel for one row (t, c), then gr_norm_rs_kernel's reduction over it with the next read's norm weights: +// the same thread-to-element mapping (256 threads, stride 256) and block_sum, so rs and the BF16 image are the same +// bits, and R is not read back +constexpr int GRW_PER = N / 256; +__global__ void __launch_bounds__(256) +gr_write_norm_rs_kernel(float* __restrict__ R, const float* __restrict__ bo, const float* __restrict__ inj, + int64_t inj_ld, const float* __restrict__ w, float eps, float* __restrict__ rs_out, + uint16_t* __restrict__ xn16) { + __shared__ float sh[32]; + const int64_t row = blockIdx.x; // t * 4 + c + const int64_t t = row / HC; + const int c = (int) (row % HC); + float* r = R + row * N; + const float sc = 2.0f * sigm(inj[t * inj_ld + c] / (float) HC); + float v[GRW_PER]; + float ss = 0.0f; +#pragma unroll + for (int k = 0; k < GRW_PER; ++k) { + const int d = threadIdx.x + 256 * k; + const float x = fmaf(bo[t * N + d], sc, r[d]); + r[d] = x; + v[k] = x; + ss += x * x; + } + const float rs = rsqrtf(block_sum(ss, sh) / (float) N + eps); + if (threadIdx.x == 0) rs_out[row] = rs; +#pragma unroll + for (int k = 0; k < GRW_PER; ++k) { + const int d = threadIdx.x + 256 * k; + xn16[row * N + d] = bf(v[k] * rs * w[c * N + d]); + } +} __global__ void gr_silu_kernel(const float* __restrict__ lo, uint16_t* __restrict__ lo16, int64_t n) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= n) return; @@ -460,6 +524,22 @@ void gr_norm(const float* R, const float* w_norm, float eps, float* xn, uint16_t gr_norm_kernel<<<(unsigned) (T * HC), 256, 0, (cudaStream_t) stream>>>(R, w_norm, eps, xn, xn16); check("gr_norm"); } +void gr_norm_rs(const float* R, const float* w_norm, float eps, float* rs, uint16_t* xn16, int64_t T, void* stream) { + gr_norm_rs_kernel<<<(unsigned) (T * HC), 256, 0, (cudaStream_t) stream>>>(R, w_norm, eps, rs, xn16); + check("gr_norm_rs"); +} +void gr_mix_r(const float* R, const float* rs, const float* w_norm, const float* gated, float* mixed, uint16_t* mixed16, + int64_t T, void* stream, uint16_t* mixed_h) { + gr_mix_r_kernel<<>>(R, rs, w_norm, gated, mixed, mixed16, T, + mixed_h); + check("gr_mix_r"); +} +void gr_write_norm_rs(float* R, const float* bo, const float* inj, int64_t inj_ld, const float* w_norm, float eps, + float* rs, uint16_t* xn16, int64_t T, void* stream) { + gr_write_norm_rs_kernel<<<(unsigned) (T * HC), 256, 0, (cudaStream_t) stream>>>(R, bo, inj, inj_ld, w_norm, eps, + rs, xn16); + check("gr_write_norm_rs"); +} void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream) { gr_silu_kernel<<>>(lo, lo16, T * LR); check("gr_silu"); diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 8651ce3e038..5ce43a92c5b 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -60,9 +60,9 @@ struct Bufs { float *R = nullptr, *bo_moe = nullptr, *inj_f = nullptr, *w = nullptr, *ple_emb = nullptr; uint16_t* mixed_h = nullptr; int32_t *ids = nullptr, *steps = nullptr; - // a sub-chunk's: the mixer's BF16 input, its output, the mixer half's injection + // a sub-chunk's: the mixer's BF16 input, its output, the mixer half's injection, a read's row scales uint16_t* mixed_bf = nullptr; - float *bo = nullptr, *inj_a = nullptr; + float *bo = nullptr, *inj_a = nullptr, *rs = nullptr; // the union: what one step needs at a time float* emb = nullptr; float *xn = nullptr, *lo = nullptr, *gated = nullptr; @@ -100,6 +100,7 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.mixed_bf = a.take(p * N); b.bo = a.take(p * N); b.inj_a = a.take(p * HC); + b.rs = a.take(p * HC); const uint64_t u0 = a.used; uint64_t u1 = u0; auto next = [&] { u1 = std::max(u1, a.used); a.used = u0; }; @@ -477,7 +478,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e uint16_t* mixed_h = b.mixed_h + t0 * N; for (int half = 0; half < 2; ++half) { mark(half == 0 ? kPsHcRead : kPsHcFfn); - // ---- the hyper-connection read of this half + // ---- the hyper-connection read of this half (the FFN half's norm came with the mixer's write) const char* pre = half == 0 ? "hc_attn_" : "hc_ffn_"; const std::string sn = std::string(pre) + "norm.weight", sd = std::string(pre) + "down.weight", su = std::string(pre) + "up.weight", si = std::string(pre) + "inject.weight"; @@ -485,12 +486,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wu = need(v, su.c_str(), err), *wi = need(v, si.c_str(), err); if (!wn || !wd || !wu || !wi) return false; float* inj = half == 0 ? b.inj_a : b.inj_f + t0 * HC; - gr_norm(b.R + t0 * D, (const float*) wn->data, EPS, b.xn, b.xn16, P, m.cs); + if (half == 0) gr_norm_rs(b.R + t0 * D, (const float*) wn->data, EPS, b.rs, b.xn16, P, m.cs); if (!bf16_proj(m.gemm, wd, b.xn16, b.lo, P, sd, err)) return false; gr_silu(b.lo, b.lo16, P, m.cs); if (!bf16_proj(m.gemm, wu, b.lo16, b.gated, P, su, err)) return false; if (!bf16_proj(m.gemm, wi, b.xn16, inj, P, si, err)) return false; - gr_mix(b.xn, b.gated, nullptr, b.mixed_bf, P, m.cs, mixed_h); + gr_mix_r(b.R + t0 * D, b.rs, (const float*) wn->data, b.gated, nullptr, b.mixed_bf, P, m.cs, + mixed_h); if (half == 1) break; if (!qsa) { @@ -574,9 +576,12 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e gate_attn(b.attn, b.Qf, b.attn_h, P, m.cs); if (!native_proj(wo, b.attn_h, b.bo, P, v.name("attn_output.weight"), err)) return false; } - // ---- the hyper-connection write of the mixer half + // ---- the hyper-connection write of the mixer half, with the FFN half's norm mark(kPsHcFfn); - gr_write(b.R + t0 * D, b.bo, b.inj_a, HC, P, m.cs); + const core::WeightRef* wfn = need(v, "hc_ffn_norm.weight", err); + if (!wfn) return false; + gr_write_norm_rs(b.R + t0 * D, b.bo, b.inj_a, HC, (const float*) wfn->data, EPS, b.rs, b.xn16, P, + m.cs); } // ---- the FFN half's input is ready: its trip to the second GPU starts here if (m.offload) { From e8ade6634bac6aeb7728b81f3e4086f5f2b59fb0 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 12:00:41 +0200 Subject: [PATCH 86/92] Prompt path: the second GPU's sums back a sub-chunk at a time, each layer's FFN write deferred to the next layer's read A layer's expert phase ended with the second GPU's FP16 sums for the whole chunk crossing the 3090's x4 link (80 MiB for 16K tokens, ~13 ms a layer) and one write over the chunk, with the 3090 idle meanwhile. The sums now come back in 2048-token pieces on a stream of their own, and each sub-chunk's FFN write runs at the start of that sub-chunk in the next layer, after its piece has landed and with the next read's norm in the same pass (gr_write_norm_rs takes the partial sums): the next layer starts once the first piece is in, and the rest land beside its sub-chunks. Before the PLE block at layer 1 and after the last layer the writes run without the norm. The sums buffer leaves the buffers' union for the chunk's (the next layer's steps would overwrite it): the prompt borrows 80 MiB more of the caches for a 16K chunk. Window hashes after the 16K prompt equal the previous build's on the 3090 alone, and with both GPUs against it with the sums buffer moved the same way (IQ3_S, UD-Q4_K_XL); against the previous build as it was, the borrowed slots differ, so which GPU computes some experts, and with it the rounding (1 of 100 argmaxes differs on IQ3_S, a near-tie). 16K-token prompt (bench strata_gen.sh, alternating builds): both GPUs, IQ3_S 7,623 -> 7,104 ms (-6.8%, 3 runs each), UD-Q4_K_XL 8,237 -> 7,656 ms (-7.1%, 2 runs each); the 3090 alone 11,755 -> 11,638 ms (-1.0%: the FFN write with the next norm). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/prefill/kernels.hpp | 4 +- src/prefill/kernels.cu | 9 ++-- src/prefill/prefill.cpp | 71 ++++++++++++++++++++++-------- 3 files changed, 60 insertions(+), 24 deletions(-) diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index 288e1f539c0..5a3d99a4387 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -17,9 +17,9 @@ void gr_norm_rs(const float* R, const float* w_norm, float eps, float* rs, uint1 /// gr_mix with xn recomputed from R, rs and w_norm as gr_norm computes it (the same bits). void gr_mix_r(const float* R, const float* rs, const float* w_norm, const float* gated, float* mixed, uint16_t* mixed16, int64_t T, void* stream, uint16_t* mixed_h = nullptr); -/// gr_write (no partial), then gr_norm_rs of the written rows with the next read's norm weights, R read once. +/// gr_write, then gr_norm_rs of the written rows with the next read's norm weights, R read once. void gr_write_norm_rs(float* R, const float* bo, const float* inj, int64_t inj_ld, const float* w_norm, float eps, - float* rs, uint16_t* xn16, int64_t T, void* stream); + float* rs, uint16_t* xn16, int64_t T, void* stream, const uint16_t* partial = nullptr); /// lo16[t, k] = bf16(silu(lo[t, k] / hc)) void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream); /// mixed[t, d] = mean_c xn[t, c, d] * sigmoid(gated[t, c, d]); FP32, BF16 and FP16 (any of them may be null). diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index 95a2a3683f4..f0fc096d91a 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -110,7 +110,7 @@ constexpr int GRW_PER = N / 256; __global__ void __launch_bounds__(256) gr_write_norm_rs_kernel(float* __restrict__ R, const float* __restrict__ bo, const float* __restrict__ inj, int64_t inj_ld, const float* __restrict__ w, float eps, float* __restrict__ rs_out, - uint16_t* __restrict__ xn16) { + uint16_t* __restrict__ xn16, const uint16_t* __restrict__ partial) { __shared__ float sh[32]; const int64_t row = blockIdx.x; // t * 4 + c const int64_t t = row / HC; @@ -122,7 +122,8 @@ gr_write_norm_rs_kernel(float* __restrict__ R, const float* __restrict__ bo, con #pragma unroll for (int k = 0; k < GRW_PER; ++k) { const int d = threadIdx.x + 256 * k; - const float x = fmaf(bo[t * N + d], sc, r[d]); + const float b = partial ? bo[t * N + d] + __half2float(__ushort_as_half(partial[t * N + d])) : bo[t * N + d]; + const float x = fmaf(b, sc, r[d]); r[d] = x; v[k] = x; ss += x * x; @@ -535,9 +536,9 @@ void gr_mix_r(const float* R, const float* rs, const float* w_norm, const float* check("gr_mix_r"); } void gr_write_norm_rs(float* R, const float* bo, const float* inj, int64_t inj_ld, const float* w_norm, float eps, - float* rs, uint16_t* xn16, int64_t T, void* stream) { + float* rs, uint16_t* xn16, int64_t T, void* stream, const uint16_t* partial) { gr_write_norm_rs_kernel<<<(unsigned) (T * HC), 256, 0, (cudaStream_t) stream>>>(R, bo, inj, inj_ld, w_norm, eps, - rs, xn16); + rs, xn16, partial); check("gr_write_norm_rs"); } void gr_silu(const float* lo, uint16_t* lo16, int64_t T, void* stream) { diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 5ce43a92c5b..adf215e86c2 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -56,9 +56,10 @@ struct Alloc { struct Bufs { uint16_t* w16 = nullptr; void* gemm_ws = nullptr; - // the chunk's: residual stream, the MoE's input (FP16) and output, the FFN half's injection, routing, steps, PLE rows + // the chunk's: residual stream, the MoE's input (FP16) and output, the FFN half's injection, routing, steps, PLE + // rows, the second GPU's sums (FP16) float *R = nullptr, *bo_moe = nullptr, *inj_f = nullptr, *w = nullptr, *ple_emb = nullptr; - uint16_t* mixed_h = nullptr; + uint16_t *mixed_h = nullptr, *sums = nullptr; int32_t *ids = nullptr, *steps = nullptr; // a sub-chunk's: the mixer's BF16 input, its output, the mixer half's injection, a read's row scales uint16_t* mixed_bf = nullptr; @@ -76,7 +77,6 @@ struct Bufs { int32_t* sel_ids = nullptr; float *logits = nullptr, *sgate = nullptr, *sup = nullptr, *sg = nullptr; uint16_t* sh_h = nullptr; - uint16_t* sums = nullptr; // the second GPU's sums (FP16) uint8_t* local = nullptr; // this GPU's expert runner uint64_t local_bytes = 0; }; @@ -97,6 +97,7 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.w = a.take(t * K); b.steps = a.take(t * strata::kernels::kStepCount); b.ple_emb = a.take(t * N); + b.sums = offload ? a.take(t * N) : nullptr; // read by the next layer's sub-chunks b.mixed_bf = a.take(p * N); b.bo = a.take(p * N); b.inj_a = a.take(p * HC); @@ -124,7 +125,6 @@ Bufs layout(Alloc& a, int64_t T, int64_t cap, int64_t max_blocks, bool offload) b.logits = a.take(p * NE); b.sgate = a.take(p * 640); b.sup = a.take(p * 640); b.sh_h = a.take(p * 640); b.sg = a.take(p); next(); // the routed experts - b.sums = offload ? a.take(t * N) : nullptr; b.local_bytes = ExpertRunner::bytes_needed(T, NE, false, !offload); b.local = a.take(b.local_bytes); next(); @@ -153,9 +153,10 @@ struct Prefill::Impl { strata::kernels::QsaShapes s; int64_t cap = 0, max_blocks = 0; cudaStream_t cs = nullptr; - // the second GPU's share: the input goes out and the sums come back on `xfer` - cudaStream_t xfer = nullptr; - cudaEvent_t ev_mixed = nullptr, ev_input = nullptr, ev_sum = nullptr; + // the second GPU's share: the input goes out on `xfer`, the sums come back on `xsum` a sub-chunk at a time + cudaStream_t xfer = nullptr, xsum = nullptr; + cudaEvent_t ev_mixed = nullptr, ev_input = nullptr; + std::vector ev_piece; // a sub-chunk's sums have landed Gemm gemm; void* owned = nullptr; // the buffers' own allocation (no region) Bufs b; @@ -176,8 +177,11 @@ Prefill::~Prefill() { Impl& m = *impl_; if (m.cs) cudaStreamSynchronize(m.cs); if (m.xfer) cudaStreamSynchronize(m.xfer); - for (cudaEvent_t e : {m.ev_mixed, m.ev_input, m.ev_sum}) if (e) cudaEventDestroy(e); + if (m.xsum) cudaStreamSynchronize(m.xsum); + for (cudaEvent_t e : {m.ev_mixed, m.ev_input}) if (e) cudaEventDestroy(e); + for (cudaEvent_t e : m.ev_piece) if (e) cudaEventDestroy(e); if (m.xfer) cudaStreamDestroy(m.xfer); + if (m.xsum) cudaStreamDestroy(m.xsum); if (m.ids_host) cudaFreeHost(m.ids_host); if (m.w_host) cudaFreeHost(m.w_host); if (m.owned) cudaFree(m.owned); @@ -195,10 +199,14 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co m.s = shapes_of(g); m.cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, m.s); m.max_blocks = ss.qsa_states[0].max_cells / m.s.idx_block + 2; - if (cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || + m.ev_piece.assign((size_t) ((max_chunk + SUB - 1) / SUB), nullptr); + bool pieces = true; + for (cudaEvent_t& e : m.ev_piece) + pieces = pieces && cudaEventCreateWithFlags(&e, cudaEventDisableTiming) == cudaSuccess; + if (!pieces || cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || + cudaStreamCreateWithFlags(&m.xsum, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&m.ev_mixed, cudaEventDisableTiming) != cudaSuccess || cudaEventCreateWithFlags(&m.ev_input, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&m.ev_sum, cudaEventDisableTiming) != cudaSuccess || cudaHostAlloc((void**) &m.ids_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess || cudaHostAlloc((void**) &m.w_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess) { err = "prefill: streams, events and pinned routing buffers"; @@ -425,6 +433,20 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e for (int64_t t = 0; t < T; ++t) strata::kernels::qsa_step_fill(m.steps_host.data() + t * SC, p0 + t, s); cudaMemcpyAsync(b.steps, m.steps_host.data(), (size_t) T * SC * 4, cudaMemcpyHostToDevice, m.cs); + // ---- a layer's FFN write, deferred to the sub-chunk that reads its rows next: the second GPU's sums come back + // a sub-chunk at a time while this GPU works on the sub-chunks before (with `norm`, the next read's norm in + // the same pass) + bool pending = false; + auto ffn_write = [&](int64_t t0, int64_t P, const float* norm) { + const uint16_t* part = nullptr; + if (m.offload) { + cudaStreamWaitEvent(m.cs, m.ev_piece[(size_t) (t0 / SUB)], 0); + part = b.sums + t0 * N; + } + float *R = b.R + t0 * D, *bo = b.bo_moe + t0 * N, *inj = b.inj_f + t0 * HC; + if (norm) gr_write_norm_rs(R, bo, inj, HC, norm, EPS, b.rs, b.xn16, P, m.cs, part); + else gr_write(R, bo, inj, HC, P, m.cs, part); + }; int64_t qsa_index = 0, gdn_index = 0; for (int64_t l = 0; l < g.n_layers; ++l) { const core::LayerView v(*m.wt, l); @@ -445,6 +467,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e const strata::kernels::PleWeights& pw = ss.ple.w; for (int64_t t0 = 0; t0 < T; t0 += SUB) { const int64_t P = std::min(SUB, T - t0); + if (pending) ffn_write(t0, P, nullptr); const float* ple_emb = b.ple_emb + t0 * N; to_bf16(ple_emb, b.mixed_bf, P * N, m.cs); if (pw.key_bf16 != nullptr) { @@ -464,6 +487,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e {b.ple_key, b.ple_q, b.inj_a, b.ple_gated, b.ple_q, b.R + t0 * D}, (int) P, m.cs); } catch (const std::exception& e) { err = std::string("prefill PLE: ") + e.what(); return false; } } + pending = false; new_layer(); } const core::WeightRef *wr = nullptr, *wgi = nullptr, *wsg = nullptr, *wsu = nullptr, *wsd = nullptr; @@ -486,7 +510,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e *wu = need(v, su.c_str(), err), *wi = need(v, si.c_str(), err); if (!wn || !wd || !wu || !wi) return false; float* inj = half == 0 ? b.inj_a : b.inj_f + t0 * HC; - if (half == 0) gr_norm_rs(b.R + t0 * D, (const float*) wn->data, EPS, b.rs, b.xn16, P, m.cs); + if (half == 0 && pending) { + mark(kPsCombine); + ffn_write(t0, P, (const float*) wn->data); + mark(kPsHcRead); + } else if (half == 0) { + gr_norm_rs(b.R + t0 * D, (const float*) wn->data, EPS, b.rs, b.xn16, P, m.cs); + } if (!bf16_proj(m.gemm, wd, b.xn16, b.lo, P, sd, err)) return false; gr_silu(b.lo, b.lo16, P, m.cs); if (!bf16_proj(m.gemm, wu, b.lo16, b.gated, P, su, err)) return false; @@ -602,6 +632,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e cudaMemcpyAsync(m.ids_host + t0 * K, b.ids + t0 * K, (size_t) P * K * 4, cudaMemcpyDeviceToHost, m.cs); cudaMemcpyAsync(m.w_host + t0 * K, b.w + t0 * K, (size_t) P * K * 4, cudaMemcpyDeviceToHost, m.cs); } + pending = false; if (!qsa) ++gdn_index; else ++qsa_index; // ================ the routed experts of the whole chunk ================ @@ -648,16 +679,20 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e if (m.offload && !m.offload->run_layer(l, T, m.ex2, m.off2, m.src2, m.w2, m.ev_input, nullptr, nullptr, err)) return false; if (!m.local.run_layer(l, T, m.ex1, m.off1, m.src1, m.w1, nullptr, b.mixed_h, b.bo_moe, err)) return false; - // ---- the MoE output (+ the second GPU's sums) into the residual stream - mark(kPsCombine); + // ---- the MoE output (+ the second GPU's sums) goes into the residual stream as the next layer reads it if (m.offload) { - cudaStreamWaitEvent(m.xfer, m.offload->done(), 0); - cudaMemcpyAsync(b.sums, m.offload->host_sum(), (size_t) T * N * 2, cudaMemcpyHostToDevice, m.xfer); - cudaEventRecord(m.ev_sum, m.xfer); - cudaStreamWaitEvent(m.cs, m.ev_sum, 0); + cudaStreamWaitEvent(m.xsum, m.offload->done(), 0); + for (int64_t t0 = 0; t0 < T; t0 += SUB) { + const size_t bytes = (size_t) std::min(SUB, T - t0) * N * 2; + cudaMemcpyAsync(b.sums + t0 * N, m.offload->host_sum() + t0 * N, bytes, cudaMemcpyHostToDevice, + m.xsum); + cudaEventRecord(m.ev_piece[(size_t) (t0 / SUB)], m.xsum); + } } - gr_write(b.R, b.bo_moe, b.inj_f, HC, T, m.cs, m.offload ? b.sums : nullptr); + pending = true; } + mark(kPsCombine); // the last layer's write + for (int64_t t0 = 0; t0 < T; t0 += SUB) ffn_write(t0, std::min(SUB, T - t0), nullptr); if (draft != nullptr && !draft_pass(tokens + c0, T, p0)) return false; if (profile) { mark(kPsPle); From 7706c1b8c5b828d6a874ebfd0c16990d2ddf7a6b Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 12:39:57 +0200 Subject: [PATCH 87/92] Prompt path: the experts through llama.cpp's MMQ kernels (upstream's step 2b) on both GPUs The prompt path dequantized each batch of experts to FP16 (iq_dequant_experts_f16) and multiplied them with cuBLAS. Layers whose expert types llama.cpp's MMQ covers now keep the weights quantized: a batch's experts' GGUF blocks are gathered side by side in the dequantization buffers' place (mmq::gather_native, a zeroed tail after them), its rows are gathered in FP32 (gather_rows16_f32) and rounded to q8_1, and gate/up and down run as one MMQ launch each over the batch (int8 tensor cores), with SwiGLU between; the weighted scatter-add is as before. Upstream's moe_mmq.cu, and its host shim for ggml-cuda, are built into strata_mmq with the instances of every expert type of ISTA-DASLab's and unsloth's files (the K-quants, Q5_1 and IQ1_S added to its switch; IQ1_M keeps FP16). A single expert with more rows than a batch keeps the FP16 path in pieces; STRATA_PREFILL_MMQ=0 takes it for every layer. The identity row map MMQ writes through is filled with each layer's uploads: the 3090's runner shares the prompt buffers' union with the dense steps, which overwrite it. The activations are rounded to 8 bits, as the decode path's expert kernels already take them. Along llama.cpp's text after the 16K prompt (--adapt-every 0), against llama.cpp: IQ3_S 99.33% / KL 1.11e-3 -> 99.67% / 1.05e-3, UD-Q4_K_XL 98.00% / 3.80e-3 -> 98.67% / 3.95e-3; MMQ against FP16 99.00% / 9.2e-4 and 99.33% / 1.3e-3, the size of llama.cpp's own CPU-against-CUDA difference. 16K-token prompt (bench strata_gen.sh, 2 runs each, alternating): IQ3_S with both GPUs 7,100 -> 6,759 ms (-4.8%), on the 3090 alone 11,626 -> 11,047 (-5.0%); UD-Q4_K_XL with both GPUs 7,772-8,500 -> 7,178 ms (-7.6% or more), alone 17,662 -> 16,828 (-4.7%). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- CMakeLists.txt | 21 ++++++++ include/strata/prefill/kernels.hpp | 2 + include/strata/prefill/moe_mmq.hpp | 3 +- src/prefill/experts.cpp | 79 +++++++++++++++++++++++++++--- src/prefill/kernels.cu | 18 +++++++ src/prefill/moe_mmq.cu | 13 +++-- 6 files changed, 125 insertions(+), 11 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e5e216b1094..75364617d9a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -526,6 +526,27 @@ if(STRATA_NATIVE_EXPERTS) target_link_libraries(strata_kernels_cpu PUBLIC ggml-cpu ggml-base) target_compile_definitions(strata_kernels_cpu PUBLIC STRATA_NATIVE_EXPERTS=1) if(STRATA_ENABLE_CUDA) + # llama.cpp's MMQ kernels (ggml-cuda, MIT) for the prompt path's experts, from the same checkout as ggml: mmq.cuh's + # instances for the expert types of ISTA-DASLab's and unsloth's files and the quantizer, with the few host + # symbols of ggml-cuda.cu they use in src/prefill/ggml_cuda_host.cu + if(STRATA_GGML_DIR) + set(_strata_gg ${STRATA_GGML_DIR}/ggml) + else() + set(_strata_gg ${strata_llamacpp_SOURCE_DIR}/ggml) + endif() + set(_strata_mmq ${_strata_gg}/src/ggml-cuda) + add_library(strata_mmq STATIC src/prefill/moe_mmq.cu src/prefill/ggml_cuda_host.cu ${_strata_mmq}/quantize.cu) + foreach(_t q2_0 iq1_s iq2_xxs iq2_xs iq2_s iq3_xxs iq3_s iq4_nl iq4_xs q4_k q5_k q6_k q5_1 q8_0) + target_sources(strata_mmq PRIVATE ${_strata_mmq}/template-instances/mmq-instance-${_t}.cu) + endforeach() + target_include_directories(strata_mmq PRIVATE ${_strata_gg}/include ${_strata_gg}/src ${_strata_mmq} + PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include) + target_compile_options(strata_mmq PRIVATE $<$:--extended-lambda -use_fast_math>) + if(MSVC) + target_compile_options(strata_mmq PRIVATE $<$:-Xcompiler=/bigobj>) + endif() + target_link_libraries(strata_mmq PUBLIC ggml-base CUDA::cudart) + target_link_libraries(strata_prefill PUBLIC strata_mmq) add_executable(native_expert_parity src/kernels/native_expert_parity.cpp) target_include_directories(native_expert_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) target_link_libraries(native_expert_parity PRIVATE strata_kernels strata_kernels_cpu strata_artifact CUDA::cudart) diff --git a/include/strata/prefill/kernels.hpp b/include/strata/prefill/kernels.hpp index 5a3d99a4387..1f4ea38de73 100644 --- a/include/strata/prefill/kernels.hpp +++ b/include/strata/prefill/kernels.hpp @@ -55,6 +55,8 @@ void swiglu_interleaved(const float* gu, uint16_t* h16, int64_t n, void* stream) void swiglu_pair(const float* g, const float* u, uint16_t* h16, int64_t n, void* stream); /// Gather rows: dst16[i, :] = x16[src[i], :] (n rows of `width` BF16). void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int64_t n, int64_t width, void* stream); +/// The same from FP16 rows into FP32 ones (MMQ's quantizer reads floats). +void gather_rows16_f32(const uint16_t* x16, const int32_t* src, float* dst, int64_t n, int64_t width, void* stream); /// y = FP16 bits of x, saturated at +-65504 (the second GPU's expert sums on their way back: below ~20 measured) void sums_to_f16(const float* x, uint16_t* y, int64_t n, void* stream); /// In place: x[t, :] *= sigmoid(sg[t]) (the shared expert's scalar gate) diff --git a/include/strata/prefill/moe_mmq.hpp b/include/strata/prefill/moe_mmq.hpp index 4efa1baf857..48e882fda15 100644 --- a/include/strata/prefill/moe_mmq.hpp +++ b/include/strata/prefill/moe_mmq.hpp @@ -12,7 +12,8 @@ namespace strata::prefill::mmq { /// This build has the MMQ path (the ggml sources were available to the build). bool built(); -/// MMQ covers this ggml type (the i-quants and Q2_0 the packs use; IQ1_M is not covered). +/// MMQ covers this ggml type (the i-quants and Q2_0 of ISTA-DASLab's files, the K-quants, Q5_1 and Q8_0 of +/// unsloth's; IQ1_M is not covered). bool supported(int ggml_type); /// Bytes of one expert's gate+up ([2*n_ff, n_embd]) or down ([n_embd, n_ff]) weights in `ggml_type`. size_t matrix_bytes(int ggml_type, int64_t rows, int64_t cols); diff --git a/src/prefill/experts.cpp b/src/prefill/experts.cpp index d778bfb1f13..d69d2c2fdcf 100644 --- a/src/prefill/experts.cpp +++ b/src/prefill/experts.cpp @@ -5,9 +5,12 @@ #include "strata/kernels/iq_kernels.hpp" #include "strata/prefill/gemm.hpp" #include "strata/prefill/kernels.hpp" +#include "strata/prefill/moe_mmq.hpp" #include #include +#include +#include namespace strata::prefill { namespace { @@ -22,6 +25,7 @@ constexpr int SLOTS = 2; // batches in staging: the copies of on constexpr size_t GEMM_WS = 32u << 20; // cuBLAS workspace constexpr int NPTR = 7; // per-expert device pointers: blob, xs, gate/up, gu, h, down, d constexpr int PRE_GROUP = 16; // prefetched blobs per copy event +constexpr size_t MMQ_TAIL = 4096; // zeroed after a batch's gathered experts: MMQ reads past the last one struct OnDevice { // a remote runner's GPU current for one call, the main one again afterwards int main; @@ -45,8 +49,23 @@ struct Alloc { size_t blob_cap() { return ((size_t) strata::kernels::cpu::expert_layout().max_blob + 255) & ~(size_t) 255; } -// the pinned upload block: NPTR x n_expert pointers, then the adds (at most 3 per row, 2 per expert), rows, weights -size_t adds_cap(int64_t chunk, int64_t n_expert) { return (size_t) (chunk * K * 3 + n_expert * 2); } +// the pinned upload block: NPTR x n_expert pointers, then the adds (at most 3 per row, 2 per expert) with each +// batch's row bounds (2 per expert), rows, weights +size_t adds_cap(int64_t chunk, int64_t n_expert) { return (size_t) (chunk * K * 3 + n_expert * 4); } + +// a native layer whose expert types llama.cpp's MMQ covers takes it (STRATA_PREFILL_MMQ=0: FP16 for every layer) +bool mmq_layer(const strata::kernels::cpu::NativeFmt& f) { + static const bool on = [] { + const char* e = std::getenv("STRATA_PREFILL_MMQ"); + return e == nullptr || std::atoi(e) != 0; + }(); + return on && strata::kernels::cpu::expert_layout().native && mmq::supported(f.gu_type) && mmq::supported(f.d_type); +} +bool mmq_any() { + const auto& lay = strata::kernels::cpu::expert_layout(); + for (const auto& f : lay.fmt) if (mmq_layer(f)) return true; + return false; +} size_t up_bytes(int64_t chunk, int64_t n_expert) { return (size_t) NPTR * (size_t) n_expert * sizeof(void*) + (adds_cap(chunk, n_expert) + (size_t) chunk * K * 2) * 4; } @@ -65,11 +84,14 @@ struct ExpertRunner::Impl { cudaEvent_t t0 = nullptr, t1 = nullptr; // the previous layer's GPU time (remote), read at the next one bool timed = false, live[SLOTS] = {}; Gemm gemm; + std::unique_ptr mmq_ctx; // the MMQ layers' launches (null: none) void* owned = nullptr; // the buffers' own allocation (no region) uint16_t *input = nullptr, *xs = nullptr, *hh = nullptr, *dq_gu = nullptr, *dq_d = nullptr; float *sum = nullptr, *w = nullptr, *gu = nullptr, *d = nullptr; int32_t* rows_src = nullptr; - int32_t* adds = nullptr; // per batch: its tokens, their row lists' starts, the row lists (moe_gather_add) + int32_t* adds = nullptr; // per batch: its tokens, their row lists' starts, the row lists (moe_gather_add), bounds + uint8_t* xq = nullptr; // MMQ: a batch's rows as q8_1 (gate/up's input, then down's) + int32_t* ident = nullptr; // MMQ: the identity row map std::vector mark, first, count; uint8_t* stage[SLOTS] = {}; void** ptrs = nullptr; // NPTR arrays of n_expert device pointers @@ -96,7 +118,7 @@ struct ExpertRunner::Impl { float* sum = remote ? a.take(t * N, ok) : nullptr; int32_t* rows_src = a.take(t * K, ok); float* w = a.take(t * K, ok); - int32_t* adds = a.take(t * K * 3 + (size_t) n_expert * 2, ok); + int32_t* adds = a.take(adds_cap(T, n_expert), ok); uint16_t* xs = a.take((size_t) ROWS * N, ok); float* gu = a.take((size_t) ROWS * FF2, ok); uint16_t* hh = a.take((size_t) ROWS * FF, ok); @@ -108,7 +130,11 @@ struct ExpertRunner::Impl { for (auto& p : stage) p = a.take((size_t) G * blob_cap(), ok); void** ptrs = a.take((size_t) NPTR * (size_t) n_expert, ok); *ws = a.take(GEMM_WS, ok); + const bool q = mmq_any(); + uint8_t* xq = q ? a.take(mmq::q8_bytes(ROWS, N), ok) : nullptr; + int32_t* ident = q ? a.take((size_t) ROWS, ok) : nullptr; if (m == nullptr) return; + m->xq = xq; m->ident = ident; m->input = input; m->sum = sum; m->rows_src = rows_src; m->w = w; m->adds = adds; m->xs = xs; m->gu = gu; m->hh = hh; m->d = d; m->dq_gu = dq_gu; m->dq_d = dq_d; for (int i = 0; i < SLOTS; ++i) m->stage[i] = stage[i]; @@ -188,6 +214,7 @@ bool ExpertRunner::init(int device, int main_device, void* stream, core::ExpertS if (err.empty()) err = "prompt experts: streams or events"; return false; } + if (mmq_any()) m.mmq_ctx = std::make_unique(); // the input and the sums: pinned and portable, both GPUs copy them if (m.remote && (cudaHostAlloc((void**) &m.h_input, (size_t) max_chunk * N * 2, cudaHostAllocPortable) != cudaSuccess || cudaHostAlloc((void**) &m.h_sum, (size_t) max_chunk * N * 2, cudaHostAllocPortable) != cudaSuccess)) { @@ -333,6 +360,9 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector int32_t { return m.res && m.cache ? m.res[(size_t) layer * (size_t) m.n_expert + (size_t) e] : -1; }; @@ -378,7 +408,7 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector adds_at(batches.size()); + std::vector adds_at(batches.size()), bounds_at(batches.size()); std::vector ntok(batches.size()); size_t n_adds = 0; for (size_t b = 0; b < batches.size(); ++b) { @@ -409,6 +439,8 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0) last_grp = std::max(last_grp, m.pre_grp[(size_t) experts[j]]); if (last_grp >= 0) cudaStreamWaitEvent(m.s, m.pre_ev[(size_t) last_grp], 0); - if (lay.native) + const int64_t r0 = off[j0], rows = off[j1] - r0; + const bool q = use_mmq && rows <= ROWS; // an expert with more rows is multiplied in pieces, in FP16 + if (q) { + // the batch's GGUF blocks side by side in the dequantization buffers' place (MMQ reads the experts at + // one stride), a zeroed tail after them + for (size_t j = j0; j < j1; ++j) { + const uint8_t* bl = (const uint8_t*) P[0 * E + j]; + mmq::gather_native(bl, bl + f.up_off, gub / 2, bl + f.down_off, db, (uint8_t*) m.dq_gu + (j - j0) * gub, + (uint8_t*) m.dq_d + (j - j0) * db, m.s); + } + cudaMemsetAsync((uint8_t*) m.dq_gu + (size_t) n * gub, 0, MMQ_TAIL, m.s); + cudaMemsetAsync((uint8_t*) m.dq_d + (size_t) n * db, 0, MMQ_TAIL, m.s); + } else if (lay.native) strata::kernels::iq_dequant_experts_f16(f.gu_type, f.d_type, (const uint8_t* const*) (m.ptrs + 0 * E + j0), n, f.up_off, f.down_off, f.n_ff, f.n_embd, m.dq_gu, m.dq_d, m.s); else @@ -480,7 +525,6 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector= 0) ++experts_resident; - const int64_t r0 = off[j0], rows = off[j1] - r0; if (rows > ROWS) { // one expert: its rows in pieces for (int64_t p = 0; p < rows; p += ROWS) { const int64_t nr = std::min(ROWS, rows - p); @@ -492,6 +536,27 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector(maxr, off[j + 1] - off[j]); + mmq::Product p; + p.w = m.dq_gu; p.type = f.gu_type; p.w_rows = FF2; p.w_cols = N; p.expert_bytes = gub; p.n = n; + p.xq = m.xq; p.bounds = m.adds + bounds_at[b]; p.ids = m.ident; p.total_rows = rows; p.max_rows = maxr; + p.dst = m.gu; p.ld_dst = FF2; + m.mmq_ctx->run(p, m.s); + float* h = (float*) m.xs; + mmq::swiglu(m.gu, h, rows, FF, false, m.s); + mmq::quantize(h, nullptr, m.xq, f.d_type, FF, FF, rows, m.s); + p.w = m.dq_d; p.type = f.d_type; p.w_rows = N; p.w_cols = FF; p.expert_bytes = db; + p.dst = m.d; p.ld_dst = N; + m.mmq_ctx->run(p, m.s); + const int32_t* a = m.adds + adds_at[b]; + moe_gather_add(out, m.d, r0, m.w, a, a + ntok[b], a + 2 * ntok[b] + 1, ntok[b], m.s); + continue; + } gather_rows16(in, m.rows_src + r0, m.xs, rows, N, m.s); m.gemm.f16_grouped(dp + 1 * E + j0, dp + 2 * E + j0, (float* const*) (m.ptrs + 3 * E + j0), (const uint16_t* const*) (P + 1 * E + j0), (const uint16_t* const*) (P + 2 * E + j0), diff --git a/src/prefill/kernels.cu b/src/prefill/kernels.cu index f0fc096d91a..5908d7ad7e2 100644 --- a/src/prefill/kernels.cu +++ b/src/prefill/kernels.cu @@ -412,6 +412,19 @@ __global__ void gather_rows16_kernel(const uint16_t* __restrict__ x, const int32 const int64_t r = i / per, j = i % per; reinterpret_cast(dst)[r * per + j] = reinterpret_cast(x)[(int64_t) src[r] * per + j]; } +__global__ void gather_rows16_f32_kernel(const uint16_t* __restrict__ x, const int32_t* __restrict__ src, + float* __restrict__ dst, int64_t n, int64_t width) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; // 8 values + const int64_t per = width / 8; + if (i >= n * per) return; + const int64_t r = i / per, j = i % per; + const uint4 v = reinterpret_cast(x)[(int64_t) src[r] * per + j]; + const __half2* h = reinterpret_cast(&v); + float4* o = reinterpret_cast(dst + r * width + j * 8); + const float2 a = __half22float2(h[0]), b = __half22float2(h[1]), c = __half22float2(h[2]), d = __half22float2(h[3]); + o[0] = make_float4(a.x, a.y, b.x, b.y); + o[1] = make_float4(c.x, c.y, d.x, d.y); +} __global__ void moe_shared_gate_kernel(float* __restrict__ x, const float* __restrict__ sg, int64_t T) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i >= T * N) return; @@ -610,6 +623,11 @@ void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int gather_rows16_kernel<<>>(x16, src, dst16, n, width); check("gather_rows16"); } +void gather_rows16_f32(const uint16_t* x16, const int32_t* src, float* dst, int64_t n, int64_t width, void* stream) { + if (n <= 0) return; + gather_rows16_f32_kernel<<>>(x16, src, dst, n, width); + check("gather_rows16_f32"); +} __global__ void sums_to_f16_kernel(const float* __restrict__ x, uint16_t* __restrict__ y, int64_t n) { const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; if (i < n) y[i] = hf(fminf(fmaxf(x[i], -65504.0f), 65504.0f)); diff --git a/src/prefill/moe_mmq.cu b/src/prefill/moe_mmq.cu index ac2daee4f56..d43f5b5889d 100644 --- a/src/prefill/moe_mmq.cu +++ b/src/prefill/moe_mmq.cu @@ -91,9 +91,11 @@ bool built() { return true; } bool supported(int t) { switch ((ggml_type) t) { - case GGML_TYPE_Q2_0: case GGML_TYPE_IQ2_XXS: case GGML_TYPE_IQ2_XS: case GGML_TYPE_IQ2_S: - case GGML_TYPE_IQ3_XXS: case GGML_TYPE_IQ3_S: case GGML_TYPE_IQ4_NL: case GGML_TYPE_IQ4_XS: - case GGML_TYPE_Q8_0: // the draft layer's dense matrices (E-9) + case GGML_TYPE_Q2_0: case GGML_TYPE_IQ1_S: case GGML_TYPE_IQ2_XXS: case GGML_TYPE_IQ2_XS: + case GGML_TYPE_IQ2_S: case GGML_TYPE_IQ3_XXS: case GGML_TYPE_IQ3_S: case GGML_TYPE_IQ4_NL: + case GGML_TYPE_IQ4_XS: + case GGML_TYPE_Q4_K: case GGML_TYPE_Q5_K: case GGML_TYPE_Q6_K: case GGML_TYPE_Q5_1: // unsloth's files + case GGML_TYPE_Q8_0: return true; default: return false; @@ -143,6 +145,11 @@ void Context::run(const Product& p, void* stream) { case GGML_TYPE_IQ4_NL: mul_mat_q_case(ctx, a, s); break; case GGML_TYPE_IQ4_XS: mul_mat_q_case(ctx, a, s); break; case GGML_TYPE_Q8_0: mul_mat_q_case(ctx, a, s); break; + case GGML_TYPE_IQ1_S: mul_mat_q_case(ctx, a, s); break; + case GGML_TYPE_Q4_K: mul_mat_q_case(ctx, a, s); break; + case GGML_TYPE_Q5_K: mul_mat_q_case(ctx, a, s); break; + case GGML_TYPE_Q6_K: mul_mat_q_case(ctx, a, s); break; + case GGML_TYPE_Q5_1: mul_mat_q_case(ctx, a, s); break; default: std::fprintf(stderr, "prefill mmq: type %d is not covered\n", (int) t); std::exit(1); From 6f3ef7f948e76dd32487c67f651567d276e0d253 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 12:53:17 +0200 Subject: [PATCH 88/92] Prompt path: the second GPU's input in pieces as it reaches host memory, its sums back in pieces The second GPU copied a chunk's MoE input in (80 MiB for 16K tokens) once the first GPU's last sub-chunk had reached host memory and the host had grouped the layer's experts, and started computing ~5.6 ms after the dense steps' end (CUPTI). Each sub-chunk's rows now go on to it as they arrive (ExpertRunner::stage_input, queued behind the previous layer's sums on its stream, which pass through the same buffer), so the input is there when its plan is. Its sums leave in the same pieces (converted to FP16 and copied to host memory piece by piece, piece_done(k)), so the first piece starts over the 3090's link before the last one leaves the 5070 Ti. Window hashes after the 16K prompt equal the previous build's (IQ3_S, UD-Q4_K_XL). 16K-token prompt, both GPUs (bench strata_gen.sh, alternating builds): IQ3_S 6,849 -> 6,686 ms (-2.4%, 3 runs each), UD-Q4_K_XL 7,233 -> 7,044 ms (-2.6%, 2 runs each). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- include/strata/prefill/experts.hpp | 12 ++++++--- src/prefill/experts.cpp | 42 +++++++++++++++++++----------- src/prefill/prefill.cpp | 30 ++++++++++++--------- 3 files changed, 52 insertions(+), 32 deletions(-) diff --git a/include/strata/prefill/experts.hpp b/include/strata/prefill/experts.hpp index 8e32d4f7306..68263da0eb7 100644 --- a/include/strata/prefill/experts.hpp +++ b/include/strata/prefill/experts.hpp @@ -66,16 +66,20 @@ class ExpertRunner { /// Remote: pinned [max_chunk, n_embd]: the MoE input goes in here, the sums come out here (both FP16). uint16_t* host_input() const; uint16_t* host_sum() const; + /// Remote, before run_layer of the same layer: rows [t0, t0 + rows) of the input, copied in once `ready` (an + /// event of the first GPU, recorded once host_input holds them) has passed; a chunk's rows in pieces, as they + /// reach host_input. + void stage_input(int64_t t0, int64_t rows, cudaEvent_t ready); /// Layer `layer` of a chunk of T tokens: expert experts[j] serves rows [off[j], off[j + 1]), row r being token /// src[r] with routing weight w[r]; each row times its weight adds to its token's sum. Local: reads `input` /// [T, n_embd] (FP16) and adds into `sum` [T, n_embd], queued on the prompt path's stream. Remote: `input` and - /// `sum` are null; its work waits for `input_ready` (an event of the first GPU, recorded once host_input holds - /// the input), and done() is recorded once host_sum holds the sums. Returns once the work is queued. + /// `sum` are null, the input came through stage_input, and the sums reach host_sum in pieces of `piece` rows, + /// piece_done(k) recorded as piece k lands. Returns once the work is queued. bool run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, - const std::vector& src, const std::vector& w, cudaEvent_t input_ready, + const std::vector& src, const std::vector& w, int64_t piece, const uint16_t* input, float* sum, std::string& err); - cudaEvent_t done() const; + cudaEvent_t piece_done(int64_t k) const; int64_t experts_resident = 0; ///< expert-layer groups served from its cache int64_t experts_streamed = 0; ///< expert blobs copied from the arena diff --git a/src/prefill/experts.cpp b/src/prefill/experts.cpp index d69d2c2fdcf..392e035029d 100644 --- a/src/prefill/experts.cpp +++ b/src/prefill/experts.cpp @@ -80,7 +80,8 @@ struct ExpertRunner::Impl { const int32_t* res = nullptr; int64_t n_expert = 0, max_chunk = 0, T = 0; // T: the chunk the buffers are bound for cudaStream_t s = nullptr, copy = nullptr; - cudaEvent_t done = nullptr, copied[SLOTS] = {}, used[SLOTS] = {}; + cudaEvent_t copied[SLOTS] = {}, used[SLOTS] = {}; + std::vector piece_ev; // remote: a piece of the sums is in host_sum cudaEvent_t t0 = nullptr, t1 = nullptr; // the previous layer's GPU time (remote), read at the next one bool timed = false, live[SLOTS] = {}; Gemm gemm; @@ -154,8 +155,9 @@ ExpertRunner::~ExpertRunner() { if (m.copied[i]) cudaEventDestroy(m.copied[i]); if (m.used[i]) cudaEventDestroy(m.used[i]); } - for (cudaEvent_t e : {m.done, m.t0, m.t1, m.pre_used, m.up_done}) if (e) cudaEventDestroy(e); + for (cudaEvent_t e : {m.t0, m.t1, m.pre_used, m.up_done}) if (e) cudaEventDestroy(e); for (cudaEvent_t e : m.pre_ev) if (e) cudaEventDestroy(e); + for (cudaEvent_t e : m.piece_ev) if (e) cudaEventDestroy(e); if (m.h_input) cudaFreeHost(m.h_input); if (m.h_sum) cudaFreeHost(m.h_sum); if (m.h_up) cudaFreeHost(m.h_up); @@ -193,8 +195,7 @@ bool ExpertRunner::init(int device, int main_device, void* stream, core::ExpertS m.first.resize((size_t) max_chunk); m.count.resize((size_t) max_chunk); OnDevice on(device, main_device, m.remote); - bool ok = cudaEventCreateWithFlags(&m.done, cudaEventDisableTiming) == cudaSuccess && - cudaEventCreateWithFlags(&m.up_done, cudaEventDisableTiming) == cudaSuccess && + bool ok = cudaEventCreateWithFlags(&m.up_done, cudaEventDisableTiming) == cudaSuccess && cudaEventCreate(&m.t0) == cudaSuccess && cudaEventCreate(&m.t1) == cudaSuccess && cudaHostAlloc((void**) &m.h_up, up_bytes(max_chunk, n_expert), cudaHostAllocDefault) == cudaSuccess; if (m.remote) ok = ok && cudaStreamCreateWithFlags(&m.s, cudaStreamNonBlocking) == cudaSuccess; @@ -336,10 +337,17 @@ bool ExpertRunner::prefetch(int64_t layer, const std::vector& experts, uint16_t* ExpertRunner::host_input() const { return impl_->h_input; } uint16_t* ExpertRunner::host_sum() const { return impl_->h_sum; } -cudaEvent_t ExpertRunner::done() const { return impl_->done; } +cudaEvent_t ExpertRunner::piece_done(int64_t k) const { return impl_->piece_ev[(size_t) k]; } + +void ExpertRunner::stage_input(int64_t t0, int64_t rows, cudaEvent_t ready) { + Impl& m = *impl_; + OnDevice on(m.dev, m.main, m.remote); + cudaStreamWaitEvent(m.s, ready, 0); // behind the previous layer's sums, which pass through `input` + cudaMemcpyAsync(m.input + t0 * N, m.h_input + t0 * N, (size_t) rows * N * 2, cudaMemcpyHostToDevice, m.s); +} bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector& experts, const std::vector& off, - const std::vector& src, const std::vector& w, cudaEvent_t input_ready, + const std::vector& src, const std::vector& w, int64_t piece, const uint16_t* input, float* sum, std::string& err) { Impl& m = *impl_; const size_t n_exp = experts.size(), n_rows = src.size(); @@ -442,17 +450,13 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector 0) cudaMemcpyAsync(m.adds, h_adds, n_adds * 4, cudaMemcpyHostToDevice, m.s); cudaMemcpyAsync(m.ptrs, P, NPTR * E * sizeof(void*), cudaMemcpyHostToDevice, m.s); - if (m.remote) cudaMemcpyAsync(m.input, m.h_input, (size_t) T * N * 2, cudaMemcpyHostToDevice, m.s); if (n_rows > 0) { cudaMemcpyAsync(m.rows_src, h_rows, n_rows * 4, cudaMemcpyHostToDevice, m.s); cudaMemcpyAsync(m.w, h_w, n_rows * 4, cudaMemcpyHostToDevice, m.s); @@ -568,10 +572,18 @@ bool ExpertRunner::run_layer(int64_t layer, int64_t T, const std::vector 0 ? piece : T; + for (int64_t t0 = 0, k = 0; t0 < T; t0 += pc, ++k) { + const int64_t n = std::min(pc, T - t0); + if ((size_t) k == m.piece_ev.size()) { + m.piece_ev.push_back(nullptr); + cudaEventCreateWithFlags(&m.piece_ev.back(), cudaEventDisableTiming); + } + sums_to_f16(m.sum + t0 * N, m.input + t0 * N, n * N, m.s); + cudaMemcpyAsync(m.h_sum + t0 * N, m.input + t0 * N, (size_t) n * N * 2, cudaMemcpyDeviceToHost, m.s); + cudaEventRecord(m.piece_ev[(size_t) k], m.s); + } cudaEventRecord(m.t1, m.s); m.timed = true; } diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index adf215e86c2..7966096d731 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -153,10 +153,10 @@ struct Prefill::Impl { strata::kernels::QsaShapes s; int64_t cap = 0, max_blocks = 0; cudaStream_t cs = nullptr; - // the second GPU's share: the input goes out on `xfer`, the sums come back on `xsum` a sub-chunk at a time + // the second GPU's share, a sub-chunk at a time: the input goes out on `xfer`, the sums come back on `xsum` cudaStream_t xfer = nullptr, xsum = nullptr; - cudaEvent_t ev_mixed = nullptr, ev_input = nullptr; - std::vector ev_piece; // a sub-chunk's sums have landed + cudaEvent_t ev_mixed = nullptr; + std::vector ev_in, ev_piece; // a sub-chunk's input is in host memory, its sums on this GPU Gemm gemm; void* owned = nullptr; // the buffers' own allocation (no region) Bufs b; @@ -178,7 +178,8 @@ Prefill::~Prefill() { if (m.cs) cudaStreamSynchronize(m.cs); if (m.xfer) cudaStreamSynchronize(m.xfer); if (m.xsum) cudaStreamSynchronize(m.xsum); - for (cudaEvent_t e : {m.ev_mixed, m.ev_input}) if (e) cudaEventDestroy(e); + if (m.ev_mixed) cudaEventDestroy(m.ev_mixed); + for (cudaEvent_t e : m.ev_in) if (e) cudaEventDestroy(e); for (cudaEvent_t e : m.ev_piece) if (e) cudaEventDestroy(e); if (m.xfer) cudaStreamDestroy(m.xfer); if (m.xsum) cudaStreamDestroy(m.xsum); @@ -199,14 +200,15 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co m.s = shapes_of(g); m.cap = strata::kernels::qsa_selection_width(strata::kernels::kTopkMaxCells, m.s); m.max_blocks = ss.qsa_states[0].max_cells / m.s.idx_block + 2; - m.ev_piece.assign((size_t) ((max_chunk + SUB - 1) / SUB), nullptr); + m.ev_in.assign((size_t) ((max_chunk + SUB - 1) / SUB), nullptr); + m.ev_piece.assign(m.ev_in.size(), nullptr); bool pieces = true; - for (cudaEvent_t& e : m.ev_piece) - pieces = pieces && cudaEventCreateWithFlags(&e, cudaEventDisableTiming) == cudaSuccess; + for (auto* v : {&m.ev_in, &m.ev_piece}) + for (cudaEvent_t& e : *v) + pieces = pieces && cudaEventCreateWithFlags(&e, cudaEventDisableTiming) == cudaSuccess; if (!pieces || cudaStreamCreateWithFlags(&m.xfer, cudaStreamNonBlocking) != cudaSuccess || cudaStreamCreateWithFlags(&m.xsum, cudaStreamNonBlocking) != cudaSuccess || cudaEventCreateWithFlags(&m.ev_mixed, cudaEventDisableTiming) != cudaSuccess || - cudaEventCreateWithFlags(&m.ev_input, cudaEventDisableTiming) != cudaSuccess || cudaHostAlloc((void**) &m.ids_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess || cudaHostAlloc((void**) &m.w_host, (size_t) max_chunk * K * 4, cudaHostAllocDefault) != cudaSuccess) { err = "prefill: streams, events and pinned routing buffers"; @@ -617,7 +619,10 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e if (m.offload) { cudaEventRecord(m.ev_mixed, m.cs); cudaStreamWaitEvent(m.xfer, m.ev_mixed, 0); - cudaMemcpyAsync(m.offload->host_input() + t0 * N, mixed_h, (size_t) P * N * 2, cudaMemcpyDeviceToHost, m.xfer); + cudaMemcpyAsync(m.offload->host_input() + t0 * N, mixed_h, (size_t) P * N * 2, + cudaMemcpyDeviceToHost, m.xfer); + cudaEventRecord(m.ev_in[(size_t) (t0 / SUB)], m.xfer); + m.offload->stage_input(t0, P, m.ev_in[(size_t) (t0 / SUB)]); } // ---- the router, and the shared expert (its gated output starts the MoE sum) mark(kPsRouter); @@ -636,7 +641,6 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e if (!qsa) ++gdn_index; else ++qsa_index; // ================ the routed experts of the whole chunk ================ - if (m.offload) cudaEventRecord(m.ev_input, m.xfer); if (cudaStreamSynchronize(m.cs) != cudaSuccess) { err = std::string("prefill: ") + cudaGetErrorString(cudaGetLastError()); return false; @@ -676,13 +680,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e } of.push_back((int32_t) sr.size()); } - if (m.offload && !m.offload->run_layer(l, T, m.ex2, m.off2, m.src2, m.w2, m.ev_input, nullptr, nullptr, err)) + if (m.offload && !m.offload->run_layer(l, T, m.ex2, m.off2, m.src2, m.w2, SUB, nullptr, nullptr, err)) return false; - if (!m.local.run_layer(l, T, m.ex1, m.off1, m.src1, m.w1, nullptr, b.mixed_h, b.bo_moe, err)) return false; + if (!m.local.run_layer(l, T, m.ex1, m.off1, m.src1, m.w1, SUB, b.mixed_h, b.bo_moe, err)) return false; // ---- the MoE output (+ the second GPU's sums) goes into the residual stream as the next layer reads it if (m.offload) { - cudaStreamWaitEvent(m.xsum, m.offload->done(), 0); for (int64_t t0 = 0; t0 < T; t0 += SUB) { + cudaStreamWaitEvent(m.xsum, m.offload->piece_done(t0 / SUB), 0); const size_t bytes = (size_t) std::min(SUB, T - t0) * N * 2; cudaMemcpyAsync(b.sums + t0 * N, m.offload->host_sum() + t0 * N, bytes, cudaMemcpyHostToDevice, m.xsum); From 0bbd8f540c992aa9ecf10d89f495d613c3eebd65 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 13:12:48 +0200 Subject: [PATCH 89/92] Prompt path: a chunk's PLE rows read on a thread, the next chunk's while the one before runs Each prompt chunk began with its PLE rows read from the SSD (one batched request, ~0.4 s for 16K tokens) and dequantized on the host, both GPUs idle meanwhile (CUPTI: two stalls of 0.36-0.52 s in a 16K prompt). The read now runs on a thread into one of two pinned buffers: the first chunk's from the prompt's start while its embeddings and layer 0 run, each next chunk's from the layer 1 of the chunk before; the PLE block at layer 1 waits for it and the rows go up asynchronously. The rows' token history for the reads runs ahead of the one the prompt cache keeps, which still advances chunk by chunk. Window hashes after the 16K prompt equal the previous build's (IQ3_S, UD-Q4_K_XL). IQ3_S with both GPUs (bench strata_gen.sh, alternating builds, 2 runs each): 16K prompt 6,785 -> 6,357 ms (-6.3%), 64K 25,391 -> 24,372 ms (-4.0%). UD-Q4_K_XL 16K 6,723 ms (7,044 before). The two buffers pin 336 MB more (UD's 71.7 GiB arena stays pinned whole). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- src/prefill/prefill.cpp | 71 +++++++++++++++++++++++++++++------------ 1 file changed, 51 insertions(+), 20 deletions(-) diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp index 7966096d731..aa712066478 100644 --- a/src/prefill/prefill.cpp +++ b/src/prefill/prefill.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include namespace strata::prefill { @@ -167,8 +168,9 @@ struct Prefill::Impl { std::vector steps_host, cnt, off, entry_of; std::vector ex1, off1, src1, ex2, off2, src2; // this GPU's experts, the second GPU's std::vector w1, w2; - std::vector ple_emb_host; - std::vector ple_rows; + // a chunk's PLE rows, read on a thread while the chunk before runs: two pinned buffers, their row indices + float* ple_host[2] = {}; + std::vector ple_rows[2]; }; Prefill::Prefill() : impl_(new Impl) {} @@ -183,6 +185,7 @@ Prefill::~Prefill() { for (cudaEvent_t e : m.ev_piece) if (e) cudaEventDestroy(e); if (m.xfer) cudaStreamDestroy(m.xfer); if (m.xsum) cudaStreamDestroy(m.xsum); + for (float* h : m.ple_host) if (h) cudaFreeHost(h); if (m.ids_host) cudaFreeHost(m.ids_host); if (m.w_host) cudaFreeHost(m.w_host); if (m.owned) cudaFree(m.owned); @@ -219,8 +222,13 @@ bool Prefill::init(const core::WeightTable& wt, const core::ModelGeometry& g, co m.cnt.resize(NE); m.off.resize(NE + 1); m.entry_of.resize(T * K); - m.ple_emb_host.resize(T * N); - m.ple_rows.resize(T * strata::kernels::PLE_N_HEADS); + for (int i = 0; i < 2; ++i) { + m.ple_rows[i].resize(T * strata::kernels::PLE_N_HEADS); + if (cudaHostAlloc((void**) &m.ple_host[i], T * N * 4, cudaHostAllocDefault) != cudaSuccess) { + err = "prefill: pinned PLE row buffers"; + return false; + } + } int dev = 0; cudaGetDevice(&dev); return m.gemm.init(stream, err) && @@ -390,6 +398,37 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e return true; }; + // ---- a chunk's PLE rows (one batched SSD request) on a thread: the first chunk's while its layer 0 runs, each + // next chunk's while the one before runs; waited for before the PLE block at layer 1 + struct PleRead { + std::thread th; + bool ok = true; + std::string err; + bool wait(std::string& e) { + if (th.joinable()) th.join(); + if (!ok) e = err; + return ok; + } + ~PleRead() { if (th.joinable()) th.join(); } + } ple_read; + int32_t pa[2] = {prev[0], prev[1]}; // the rows' tokens before the next chunk to read + int ple_cur = 0; + auto ple_start = [&](int64_t c, int buf) { + const int64_t Tc = std::min(m.T, n - c); + uint32_t* rows = m.ple_rows[buf].data(); + for (int64_t t = 0; t < Tc; ++t) { + const int32_t tok = (int32_t) tokens[c + t]; + strata::kernels::ngram_rows(&tok, pa, 1, ss.ple.consts, rows + t * strata::kernels::PLE_N_HEADS); + pa[0] = pa[1]; + pa[1] = tok; + } + ple_read.ok = true; + ple_read.th = std::thread([&m, &ss, &ple_read, rows, Tc, out = m.ple_host[buf]] { + ple_read.ok = ss.ple.table->gather_batch(rows, (size_t) Tc, out, ple_read.err); + }); + }; + if (ss.ple.ready() && n > 0) ple_start(0, 0); + static const bool trace = std::getenv("STRATA_TRACE") != nullptr; processed_ = 0; int64_t c0 = 0; @@ -414,23 +453,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e } gr_broadcast(b.emb, b.R + t0 * D, P, m.cs); } - // ---- the PLE rows of the whole chunk, one batched SSD request const bool ple_on = ss.ple.ready(); - if (ple_on) { - const auto tp = Clock::now(); - for (int64_t t = 0; t < T; ++t) { - const int32_t tok = (int32_t) tokens[c0 + t]; - strata::kernels::ngram_rows(&tok, prev, 1, ss.ple.consts, m.ple_rows.data() + t * strata::kernels::PLE_N_HEADS); - prev[0] = prev[1]; - prev[1] = tok; - } - if (!ss.ple.table->gather_batch(m.ple_rows.data(), (size_t) T, m.ple_emb_host.data(), err)) return false; - // before any expert copy of the chunk takes the link - cudaMemcpyAsync(b.ple_emb, m.ple_emb_host.data(), (size_t) T * N * 4, cudaMemcpyHostToDevice, m.cs); - stats_.ms_ple += ms_since(tp); - } else { - for (int64_t t = 0; t < T; ++t) { prev[0] = prev[1]; prev[1] = (int32_t) tokens[c0 + t]; } - } // ---- the QSA step records of every position in the chunk for (int64_t t = 0; t < T; ++t) strata::kernels::qsa_step_fill(m.steps_host.data() + t * SC, p0 + t, s); cudaMemcpyAsync(b.steps, m.steps_host.data(), (size_t) T * SC * 4, cudaMemcpyHostToDevice, m.cs); @@ -466,6 +489,13 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e // ---- the PLE block at layer 1: the key and value projections, then the decode path's per-token // arithmetic (the conv reads the previous tokens' normalized rows) if (l == 1 && ple_on) { + // the chunk's rows, then the next chunk's read on the thread (the other buffer) + const auto tp = Clock::now(); + if (!ple_read.wait(err)) return false; + cudaMemcpyAsync(b.ple_emb, m.ple_host[ple_cur], (size_t) T * N * 4, cudaMemcpyHostToDevice, m.cs); + if (c0 + T < n) ple_start(c0 + T, 1 - ple_cur); + ple_cur = 1 - ple_cur; + stats_.ms_ple += ms_since(tp); const strata::kernels::PleWeights& pw = ss.ple.w; for (int64_t t0 = 0; t0 < T; t0 += SUB) { const int64_t P = std::min(SUB, T - t0); @@ -710,6 +740,7 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e ev.clear(); } stats_.tokens += T; + for (int64_t t = 0; t < T; ++t) { prev[0] = prev[1]; prev[1] = (int32_t) tokens[c0 + t]; } } ss.ple_prev[0] = prev[0]; ss.ple_prev[1] = prev[1]; From 61521be6943e19105bace49b1695d710e21b1d63 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 15:40:05 +0200 Subject: [PATCH 90/92] docs: this fork against llama.cpp and upstream Strata on an RTX 3090 + RTX 5070 Ti (docs/COMPARISON.md) docs/COMPARISON.md holds the current numbers for IQ3_S and UD-Q4_K_XL: speed through the API (prompts 16K-128K, generation on code, thinking, an edit and after each prompt, a follow-up turn, the time to the API, VRAM and RAM), generation on fixed text, agreement with llama.cpp's next-token distributions with their rounding baselines, llama.cpp's perplexity of the files, and the three engines' features. llama.cpp's and upstream's columns were measured once; this fork's are kept current. The README's fork note and DETAILS' second-GPU and unsloth sections point to it instead of carrying numbers of their own. data/draft_vocab.bin is this fork's 40,525-token draft subset again (the merge took upstream's 106,299-token one with the Chinese, Japanese and Korean tokens: a 2.6x larger draft head, English text 1-2% slower by upstream's measure). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- README.md | 4 ++ data/draft_vocab.bin | Bin 425196 -> 162100 bytes docs/COMPARISON.md | 149 +++++++++++++++++++++++++++++++++++++++++++ docs/DETAILS.md | 12 ++-- 4 files changed, 159 insertions(+), 6 deletions(-) create mode 100644 docs/COMPARISON.md diff --git a/README.md b/README.md index 14a3aa1167e..2ff67406137 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,10 @@ of a word): faster than you can read. - **Free and open source.** +> **This fork** (branch `custom`) is tuned for a PC with an RTX 3090 and an RTX 5070 Ti: the second card works as another +> expert tier, unsloth's K-quant files run too, and prompts and answers are faster. How it compares with llama.cpp and +> upstream Strata on that PC: [docs/COMPARISON.md](docs/COMPARISON.md). + > **Jump to:** [How fast?](#how-fast-is-it) · [Which model?](#which-model-should-i-pick) · [Install](#install) · > [Using it](#using-it) · [Problems?](#something-went-wrong) · [How it works](#how-does-it-work) · > [All the details](docs/DETAILS.md) diff --git a/data/draft_vocab.bin b/data/draft_vocab.bin index 217f2745362b68df717f0f2b1b0eb3256abd9cb5..c588b567f6e35fac0e9472a0772d2d03898a93ef 100644 GIT binary patch delta 13 UcmaF!Q*z5E&W0AoElehr05a|d761SM literal 425196 zcmWjM(-I)t5(LmYwr$(CZQHhO+qP}nwr$(?%w6YUeL_X=h_1@~-~ayizW@Xz5P=Co zP=XPh5QHQYp$S7+!V#VbL?jZCi9%GO5uF&sBo?uWLtNq!p9CZ%5s67cQj(FJ6r>~- zsYydx(vhAFWF!-r$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#Q zRHPD>sX|q%QJospq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We! z(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT z!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^ z5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)675P}kn;DjI~p$JVF z!V-?~L?9xOh)fis5{>A@ASSVhO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3 zeBdLW_{h2uUbH6Na#aBRmm^NF*W? zg{VX$Ix&bzEMgOfxWpqq2}npH5|f0aBqKQ~NJ%PElZLdUBRv_&NG39qg{)*FJ2}Wn zE^?EHyyPQ41t>@%3R8rl6r(sLC`l}a>$Rs8+g{e$qIy0EbEM_x@xy)le3s}e^7PEw-V?7(#$R;+kg{^F3J3H9PE_Snrz3gK@2RO(f4s(Q~9OF1AILRqabB42=<2)C* z$R#dwg{xfSIybnDP6JlYEp~Z)S)i*s80hL(ul@1p()L1 zP77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW= zEaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9N zZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE z_+Mc2Pe1|@m>>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9c zm?R`68OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_; zNlH=yOIp#IHngQ3 z?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bB zv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bhfil%qTq zs7NI$Q-!KjqdGOHNiAwqhq~0GJ`HF{BO23$rZl5DEoezATGNKMw4*&8=tw6z(}k{d zqdPt5NiTZShraZqKLZ%ZAO&aK$t-3w zhq=sSJ_}gLA{MiRr7UAPD_F@YR>(8$u4%YhrR4$KL+=(3WeG#AU83dBtnq@RoPH z=K~-4#Am+nm2Z6K2S546Z~pL?fBY|)`6nQO2uu)y5{%%4AS9s(O&G!wj_^bvB9Vwp z6rvK1=)@oeQenwWv)U>QayT zG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd73}7IG7|alc zGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkGj`w`vBcJ%p7rye1 z@BH8=zxd4`{_>Cie((n%Ab|)>5P}kn;DjI~p$JVF!V-?~L?9xOh)fis5{>A@ASSVh zO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3eBdLW_{25Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHY zBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQh zP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXr zM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku z3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9n zUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X z9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fBDCMi+uqINFV|e zgrEc?I3Wm0C_)p4u!JK#5r{}6A`^wEL?b#eh)FDB6Nk9OBR&a8NFoxGgrp=RIVng< zDpHe%w4@_F8OTT`GLwa@WFtE{$Vo18lZU+IBR>TwNFfSSgrXFqI3*}aDN0j@vXrAd z6{tuhDpQ53RHHgIs7WnqQ-`|LqdpC2NFy54gr+p3IW1^OD_YZrwzQ)?9q33WI@5)& zbfY^x=t(bn(}%wFqdx-}$RGwYgrN*$I3pOzC`L1ev5aFp6PU;(CNqVpOk+ATn8_?= zGl#j%V?GO5$RZZAgrzKFIV)JnDps?GwX9=38`#JuHnWATY-2k+*vT$-vxmLxV?PHt z$RQ4MgrgkeI43yCDNb{Svz+5R7r4kJE^~#eT;n=7xXCSUbBDX!<30~~$Ri%}gr_{? zIWKt0D_--4x4h#$ANa^8KJ$gIeB(Pm_{lGR^M}9u zBomp*LRPYoogCyO7rDtpUhrl%y1;DMMMxQJxA^q!N{>LRG3! zof_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-;q!XR#LRY%cogVb07rp62 zU;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X)of*tz7PFbdT;?&K1uSF{ zi&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWRUiPt{103WKhdIJgj&Yn5 zoa7XzIm21bah?lYUG8z82R!5vk9opVp7ER)yyO+HdBa=Y z@tzNSDP6JlYEp~Z)S)i* zs80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$p zVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qt ziq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S z1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4 z-}ufCe)5ao{NXSE`0tn}009X^V1f{oU<4-wAqhoj!Vs2lgeL+Ki9}?g5S3^|Ck8Qz zMQq{_mw3b{0SQS&Vv>-QWF#jADM>|Y(vX&Pq$dLz$wX$dkd z00k*TVTw?cViczYB`HN|%21Yal&1m}sYGR}P?c&_rv^2tMQ!R(mwMEv0S#$HW17&E zW;CY-Eont-+R&DEw5J0d=|pF`(3Ngq#cl3zmwVjj0S|e^W1jGoXFTTxFL}jl-td-pyypWS`NU_w@Re_T=LbLe#c%%b zmw)_s$QOWs1R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi;*)@c zBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJk zQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3 zOFP=rfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5ioHny{a zo$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800yceu+v z?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk{yX9gKtKW! zm>>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdO zN>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)F zz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5b{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKG2jl8*Fb zAS0Q`Oct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRepdyv1Ockn9 zjq22(Cbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+jqdcIC%x!R zANtad{tRFsgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$9Og2Q`7B@| zi&)GOma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9S zoZuv;3J>-%oo1$jqm*6C%^d3AO7->|H3;05RgCwCI~?ZMsPw9l2C*u3}FdJcp?yy zNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~ zsYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pV zc`k5~OI+p(SGmS@Zg7)Z+~y8hfil%qTqs7NI$Q-!KjqdGOHNiAwqhq~0GJ`HF{BO23$ zrZl5DEoezATGNKMw4*&8=tw6z(}k{dqdPt5NiTZShraZqKLZ%ZAO&aK$t-3whq=sSJ_}gLA{MiRr7UAPD_F@YR>(8$u4%YhrR4$KLP^DMC?- zQJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1c zLtEO>o(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W83 z5|f$2RHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O7 z9qeQmyV=8D_OYJ>9OMv(Il@tnahwyJh{PlzDalAq z3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+br zs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>G zbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAl zGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtP zDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=LAFRU{F0SQE4f)JEo1SbR` z2}Nka5SDO+Cjt?PL}a26m1smK1~G|6Y~m1?c*G|G2}wj^l8}^SBqs$aNkwYXkd}0$ zCj%MDL}s#(m26}u2RX?_Zt{?qeB`G91t~;ficpkd6sH6wDMe|@P?mC(rveqJL}jW_ zm1+=(3WeG#AU83dBtnq z@RoPH=K~-4#Am+nm2Z6K2S546Z~pL?fBYBJ8-RcWA}~P+N-%;Gf{=tFG+_u!IKmTw zh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@ zvXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2% zVlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)= zInHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`Y zU--&5zVm~h{Ngu%_{%^33+WC(Kmrk%AOs~C!3jY~LJ^uUge4r|i9kdm5t%4NB^uF* zK}=#1n>fTJ9`Q*)LK2afBqSvn$w@&^$t zANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVT zCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_Oy zMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4teQenwWv)U>QayTG@v1kXiO8D(v0S`pe3zn zO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)j zB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6 z?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>Ci$~giM zkU#_`2tf%(a6%B0P=qE7VF^cgA`p>CL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyt za#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib` z2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}Q$A85<0SHJS0uzLw1S2>h z2uUbH6Na#aBRmm^NF*W?g{VX$Ix&bzEMgOfxWpqq2}npH5|f0aBqKQ~NJ%PElZLdU zBRv_&NG39qg{)*FJ2}WnE^?EHyyPQ41t>@%3R8rl6r(sLC`l}a>$Rs8+g{e$qIy0EbEM_x@xy)le z3s}e^7PEw-V?7(#$R;+kg{^F3J3H9PE_Snrz3gK@2RO(f4s(Q~ z9OF1AILRqabB42=<2)C*$R#dwg{xfSIybnZ7q7j`K#3UB6i9=lC5uXGkBoT>8LQ;~EoD`%a6{$%>TGEl83}hq|naM&{ zvXPw}F`or2WD$#5!cvy8 zoE5BO6{}gpTGp|i4Qyl+o7uuvwy~WZ>|__a*~4D;v7ZAR` zP7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WP zFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ z#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8 zMly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3|RNkn3jkd$O3Cj}`YE-8NHK|2y>QI+@)TaRrX+&e1(3EC0rv)u( zMQhs7mUgtK10Cr^XS&dpZgi&yJ?TYn`p}nt^k)DA8N^_QFqB~oX9Ob|#c0MbmT`<{ z0u!0UWTr5cX-sDZGnvI~<}jCe%x3`$S;S(Nu#{yiX9X)+#cI~DmUXOW0~^`IX11`E zZER-;JK4o<_OO?I?B@UnImBU(aFk;l=L9D?#c9rPmUEov0vEZ&Wv+0QYh33BH@U@a z?r@iT+~)xgdBkI$@RVmf=LIi$#cSU1mUq1810VUsXTI>2Z+zzmKl#OP{_vN7{8z*q zfPe%dFhK}PFoF|;kc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^D zGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw} zWhhHI%2R=gRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rK zbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M! zGlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu( z%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNd zF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rl}r;SNAR0uh)X1SJ^3 z2|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5 zB^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1 znJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$a zPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw z@tQZht)-vZQHhO+qP}nwr$(CZPyO_goP#yVF^cgA`p>CL?#MRiAHo{5R+KMCJu3l zM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu z2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy z<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}QM}P$W z6OcdzCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW z3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjO zPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p z*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y82 z5Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvx zM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi z3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1 zUi799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh|9Oei| zImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I@RC=& z<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%ME(Qjn5Vq$Uk%Nk@7zkdaJeCJR}~ zMs{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK z4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+ zK@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNp zR)oEPH>V_oaPK? zImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^ z<_ll>#&>@3lVAMi4}bYbfW-b2kU#_`2tf%(a6%B0P=qE7VF^cgA`p>CL?#MRiAHo{ z5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZ zM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_ zxyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs z<_~}QM}Q>$6OcdzCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`A zkdQ@0trU*qT zMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){) z4Q**hdpgjOPIRUVUFk-5deDAZhTiM2T zcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y8< zxyOAT@Q_D5<_S-E#&cfql2^Ru4R3kJdp_`yPkiPJU-`y&e(;lD{N@jT`A2}H{u7Wu z1SSYU2}W>25Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~ zkdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2% zM|mnxkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1` z3tj0(cY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J< zS-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+ z4seh|9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{c zdB$^I@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%Wd0M7Km;ZTK?z21LJ*Qr zgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7z zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5 zMs;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u z4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{f zMJ#3sOIgNpR)oE zPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvW zdB=M`@R3h^<_ll>#&>@3lVAMi4}bYbfaLxYkU#_`2tf%(a6%B0P=qE7VF^cgA`p>C zL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^ zkds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z z`NnsC@RMKs<_~}QM}QRm6OcdzCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9 z#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}H zMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@ zZg7)Z+~y825Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHY zBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQh zP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXr zM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku z3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9n zUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X z9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%RQ?l?Km;ZT zK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5V zq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuK zP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^ zMt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW z4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3 zUhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bYbfYkmIkU#_`2tf%(a6%B0P=qE7 zVF^cgA`p>CL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr> zWF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoAT zM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXi zKJbxGeC7*Z`NnsC@RMKs<_~}QM}Rc`6OcdzCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(I zQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp z(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~ zOI+p(SGmS@Zg7)Z+~y825Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{ zafwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j z6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee z(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E) z$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu7 z3tQR7c6P9nUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8Ez zTioUjce%%X9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p% zbp8{NKm;ZTK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS`d=ik5L?k8& zNl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwg zl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH z(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8 z#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?T zM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bYbfb{+okU#_`2tf%( za6%B0P=qE7VF^cgA`p>CL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;i zX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y z(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0 zSG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}QM}Q3e6OcdzCI~?ZMsPw9l2C*u3}FdJ zcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9} z)TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K z3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y825Ry=YCJbQ-M|dI-kw`=)3Q>th zbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJX zdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZvb*V>v8qknN zG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C83}y&J8OCr% zFp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf!u##1*W({ju z$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W z3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4b zU;O3|fB8p%O#Ty)Km;ZTK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS` zd=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5 zMJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bK zw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}g zFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;8 z4tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bYbfXx0A zkU#_`2tf%(a6%B0P=qE7VF^cgA`p>CL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyt za#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib` z2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}QM}RE;6OcdzCI~?ZMsPw9 zl2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_ zRjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ z#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y825Ry=YCJbQ-M|dI- zkw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oV zc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZv zb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C8 z3}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf! zu##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>! z$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW) z3t#!hcYg4bU;O3|fB8p%Z2l9FKm;ZTK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8j zlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N) zehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}g zO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05 zjAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2 z#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi z4}bYbfb9MgkU#_`2tf%(a6%B0P=qE7VF^cgA`p>CL?#MRiAHo{5R+KMCJu3lM|={H zkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GI zaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc z$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}QM}Qpu6Ocdz zCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=t zc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUV zUFk-5deDAZhTiM2TcCeFO>}C&p*~fkk zaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y825Ry=Y zCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`l zkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi3RS5_ zb!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1Ui799 zed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J- zEM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh|9Oei|ImU5L zaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I@RC=&<_&Lo z$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%T>cY~Km;ZTK?z21LJ*QrgeDAO2}gJ$5Rphk zCJIrBMs#8jlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+L zlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$V zeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UH zLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY} zaFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll> z#&>@3lVAMi4}bYbfZYBQkU#_`2tf%(a6%B0P=qE7VF^cgA`p>CL?#MRiAHo{5R+KM zCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)W zkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&F zaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~}Q zM}R#36OcdzCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3| zl2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**h zdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO z>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y825Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoS zCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnx zkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0( zcY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh| z9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I z@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%eEt)VKm;ZTK?z21LJ*QrgeDAO z2}gJ$5RphkCJIrBMs#8jlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJe zCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dX zlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJ ze+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3s zOIgNpR)oEPH>V_ zoaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M` z@R3h^<_ll>#&>@3lVAMi4}bYbfc*XwkU#_`2tf%(a6%B0Q2!aGr)?n+004lUT$^p% zwr;j<+qP}nwr$(CZ5ule^9e&3!V-?~L?9xOh)fis5{>A@ASSVhO&sD9kN6}YA&E## z5|WaPSnGn&(amb9WZZD>n7 z+S7rKbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%q zn94M!GlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-N zVmEu(%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP% zfQLNdF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rd4I_)lPh5R_m9 zCj=o0MQHvb3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E z-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N z%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^< zahM|<fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*2k3i?lAf)JEo1SbR`2}NlBBMe~) zM|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ z3t7oVc5;xDT;wJXdC5n93J^d+3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*m zO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@ z{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E` z4|&N)ehLskK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJ zLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0 z!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tCL?#MRiAHo{5R+KMCJu3l zM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7(rKtT#o zm?9LV7{w_;NlH=y zOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv# z;I&HLPVF>)F6YHnEv4 zY-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0t zahp5bMQr5Vj>K}%ZEnl`kh9qs8r zM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP= z)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1 z>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+ z@t7w({N*2kiuzArf)JEo1SbR` z2}NlBBMe~)M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvx zM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93J^d+3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1 znJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$a zPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw z@tQZhQjn5Vq$Uk%Nk@7zkdaJeCJR}~ zMs{+LlU(E`4|&N)ehLskK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8= zn>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7? zKn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb z%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$ z@tH4tCL?#MRiAHo{ z5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZ zM}7(rKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLC zNJcT5F^pv#;I&HLPVF z>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PF zT;&?qxxr0tahp5bMQr5Vj>K}%ZE znl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5 zL?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q z+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC z+~pqkdB8&+@t7w({N*2kO8QS= zf)JEo1SbR`2}NlBBMe~)M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~ zkdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93J^d+3Q?FM6r~u&DM3j}QJON8 zr5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>! znJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`Tqb zJmneBdBICw@tQZhQjn5Vq$Uk%Nk@7z zkdaJeCJR}~Ms{+LlU(E`4|&N)ehLskK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@K zr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+o zn?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jki zLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eY zyyYG5`M^g$@tH4tC zL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^ zkds{GCJ%YZM}7(rKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58u zm>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld z=efW|E^(PFT;&?qxxr0tahp5bMQ zr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3i znlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^ zMmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo z*SWz>ZgHDC+~pqkdB8&+@t7w( z{N*2k%KA@Wf)JEo1SbR`2}NlBBMe~)M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHY zBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93J^d+3Q?FM6r~u& zDM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2 zr5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S z_j$lW9`TqbJmneBdBICw@tQZhQjn5V zq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehLskK?+frA{3<<#VJ8aN>Q3Jl%*Wy zsX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rE zr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc` zn>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD z&w0U1Uh$eYyyYG5`M^g$@tH4tCL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr> zWF`w)$wqc^kds{GCJ%YZM}7(rKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI? zr62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@um zNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9? zWf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%K znl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^f zMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*2kD*8`gf)JEo1SbR`2}NlBBMe~)M|dI-kw`=)3Q>thbYc*bSi~j{ zafwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93J^d+ z3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8M zX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehLskK?+frA{3<<#VJ8a zN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G z=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$ zWg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{` zLmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tCL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;i zX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7(rKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edi zm?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD z8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++ zWf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*2ks`^i0f)JEo1SbR`2}NlBBMe~)M|dI-kw`=)3Q>th zbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJX zdC5n93J^d+3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu z8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@ z8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehLskK?+fr zA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1peP zTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0- znZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*) zn>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tCL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyt za#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7(rKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>A zI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^Y zWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3 zdeNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<) zS-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*2kYWh!Lf)JEo1SbR`2}NlBBMe~)M|dI- zkw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oV zc5;xDT;wJXdC5n93J^d+3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_ zI@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO z1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N) zehLskK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVT zCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_Oy zMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tCL?#MRiAHo{5R+KMCJu3lM|={H zkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7(rKtT#om?9LV z7{w_;NlH=yOIp#I zHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU z*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5b zMQr5Vj>K}%ZEnl`kh9qs8rM>^4& zE_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3c zW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4PO zIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w( z{N*2k>iSP$f)JEo1SbR`2}NlB zBMe~)M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`l zkxXPJ3t7oVc5;xDT;wJXdC5n93J^d+3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ z8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPat zKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZh zQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+L zlU(E`4|&N)ehLskK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5` z9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|G zAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQun zR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4t zwQbwBZQHhO+qP}n zwr$(CyBzX_gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{Nk~dEl9Pgz zq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`KMJP%!ic^A;l%h0cC`&oY zQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=ow4ya_XiGcV(}9k3qBC9S zN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@uGl7XrVlq>h$~2}kgPF`? zHglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVRH+$I2KK65f zgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^>hdkmjPk72R zp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLRxHpMV4+FhK}PFoF|;kc1*M zVF*h&!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygxX-G>t(vyLV zWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`e zQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b(}SM$qBni$ zOF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M!GlQATVm5P_%RJ_@fQ2k# zF-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu(%RcsVfP)<3Fh@AbF^+SB zlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNdF;95PGoJH;m%QRNZ+Oc) z-t&QveBv`-_{ulF^MjxK;x~Wz%Rd4%^q+tPA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbX zQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@vXg_H zI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2%VlYD( z$}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)=InHx| zi(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`YU--&5 zzVm~h{Ngu%_{%>6H1eN-1R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGB zafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76< z6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h z(}I??qBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfp zGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1m zo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk z0yOrYfCM5iK?q7Pf)j#}gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{ zNk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`KMJP%!ic^A; zl%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=ow4ya_XiGcV z(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@uGl7XrVlq>h z$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVR zH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^> zhdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLRxIpMV4+FhK}P zFoF|;kc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygx zX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=g zRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b z(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M!GlQATVm5P_ z%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu(%RcsVfP)<3 zFh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNdF;95PGoJH; zm%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rd4%^`C$QA}~P+N-%;Gf{=tFG+_u! zIKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+ zS;$H@vXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS z)S@Q6^rAO?=u1EP zGk}2%VlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$ zG-o)=InHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnw zk9^`YU--&5zVm~h{Ngu%_{%>6H1nT;1R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1 zG@=uOn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=m zdB{sX@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGOR zG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6 z%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(? zGFQ0DHLi1mo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?O zpZwxCfB4Hk0yOuZfCM5iK?q7Pf)j#}gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkc zJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`K zMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=o zw4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@u zGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV z$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvq zHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLWJy zpMV4+FhK}PFoF|;kc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^D zGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw} zWhhHI%2R=gRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rK zbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M! zGlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu( z%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNd zF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rd6N^q+tPA}~P+N-%;G zf{=tFG+_u!IKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpo zI?|JYjASA+S;$H@vXg_HI4f|8V?G-W7DIm%Okid3R9 zRj5ies#AlS)S@Q6 z^rAO?=u1EPGk}2%VlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z z$}x^}f|H!$G-o)=InHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDF zHE(#!JKpnwk9^`YU--&5zVm~h{Ngu%_{%>6wDO;T1R^j&2ud)56M~S0A~azLOE|(4 zfrvyRGEs<1G@=uOn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(= zHnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5; zb*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY% zvx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC z%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIR zGhg`1H@@?OpZwxCfB4Hk0<`v@fCM5iK?q7Pf)j#}gd#Ly2unD^6M=|CA~I2kN;IMq zgP6o3HgSkcJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQ zKJrt5f)t`KMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrv zO=wCpn$v=ow4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZ zjAArn7|S@uGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%r zvw@9lVl!LV$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb z$~CTYgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83 zH-GrcKLWJzpMV4+FhK}PFoF|;kc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXz zF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZ zZD>n7+S7rKbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5 zOky%qn94M!GlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAx zvxA-NVmEu(%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1< z%RTP%fQLNdF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rd6N^`C$Q zA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@ zf|R5pHEBpoI?|JYjASA+S;$H@vXg_HI4f|8V?G-W7D zIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2%VlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmfl zbAW>!;xI=z$}x^}f|H!$G-o)=InHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn z$}^txf|tDFHE(#!JKpnwk9^`YU--&5zVm~h{Ngu%_{%>6wDX^U1R^j&2ud)56M~S0 zA~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh% zfsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7Nb zHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIPH@ee zEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}O zbApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0 z%RAolfscIRGhg`1H@@?OpZwxCfB4Hk0<`y^fCM5iK?q7Pf)j#}gd#Ly2unD^6M=|C zA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~S zgPi0dH+jfQKJrt5f)t`KMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{= zJ?hhdhBTrvO=wCpn$v=ow4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0W zLm0|1hBJbZjAArn7|S@uGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GD ztYS55Sj#%rvw@9lVl!LV$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTC zbAgLo;xbpb$~CTYgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P z$~V6AgP;83H-GrcKLT{{pMV4+FhK}PFoF|;kc1*MVF*h&!V`grL?SX#h)Oh~6N8w< zA~tb|OFZI}fP^F>F-b^DGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^b zfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->Sn zGn&(amb9WZZD>n7+S7rKbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$A zV;IXg#xsG5Oky%qn94M!GlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@blYgo%V*0X_) zY+^H8*vdAxvxA-NVmEu(%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_T;eiUxXLxI zbAy}Q;x>1<%RTP%fQLNdF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz z%Rd5i^q+tPA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRI zA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@vXg_HI4 zf|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2%VlYD($}omAf{~13G-DXcIL0%9iA-WL zQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)=InHx|i(KL|SGdYGu5*K%+~PKOxXV56 z^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`YU--&5zVm~h{Ngu%_{%>6bn>5o1R^j& z2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_ zA~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@ zfr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIP zH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P z9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$JmN7=c*--L z^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk0(ADDfCM5iK?q7Pf)j#}gd#Ly z2unD^6M=|CA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6 zA~RXYN;a~SgPi0dH+jfQKJrt5f)t`KMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;Rre zgPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=ow4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?! zKl(F(fed0WLm0|1hBJbZjAArn7|S@uGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw* zOIXS>ma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*O zoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9 z^MQ|i;xk|P$~V6AgP;83H-GrcKLT{|pMV4+FhK}PFoF|;kc1*MVF*h&!V`grL?SX# zh)Oh~6N8wF-b^DGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y( zA~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`eQ-hk+qBeD?OFin- zfQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b(}SM$qBni$OF#NEfPoBRFhdy1 zForXNk&I$AV;IXg#xsG5Oky%qn94M!GlQATVm5P_%RJ_@fQ2k#F-us=GM2M~m8@bl zYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu(%RcsVfP)<3Fh@AbF^+SBlbqr-XE@6_&U1l_ zT;eiUxXLxIbAy}Q;x>1<%RTP%fQLNdF;95PGoJH;m%QRNZ+Oc)-t&QveBv`-_{ulF z^MjxK;x~Wz%Rd5i^`C$QA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbXQHV-3q7#Fd#3D9v zh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@vXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2%VlYD($}omAf{~13G-DXc zIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)=InHx|i(KL|SGdYGu5*K% z+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`YU--&5zVm~h{Ngu%_{%>6 zbn~Bp1R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi;*)@cBqA|M zNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF| zqBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=r zfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*a zd)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$ zJmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk0(AGEfCM5iK?q7P zf)j#}gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{Nk~dEl9Pgzq#`wG zNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`KMJP%!ic^A;l%h0cC`&oYQ-O+9 zqB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=ow4ya_XiGcV(}9k3qBC9SN;kUG zgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@uGl7XrVlq>h$~2}kgPF`?HglNE zJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVRH+$I2KK65fgB;>8 zM>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2 zyy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLYgdpMV4+FhK}PFoF|;kc1*MVF*h& z!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygxX-G>t(vyLVWFj+J z$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`eQ-hk+ zqBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b(}SM$qBni$OF#NE zfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M!GlQATVm5P_%RJ_@fQ2k#F-us= zGM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu(%RcsVfP)<3Fh@AbF^+SBlbqr- zXE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNdF;95PGoJH;m%QRNZ+Oc)-t&Qv zeBv`-_{ulF^MjxK;x~Wz%Rd72^q+tPA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbXQHV-3 zq7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@vXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2%VlYD($}omA zf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)=InHx|i(KL| zSGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`YU--&5zVm~h z{Ngu%_{%>6^zxs81R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi z;*)@cBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOi zC`vJkQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I?? zqBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5io zHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800y zceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk0`&Hu zfCM5iK?q7Pf)j#}gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkcJmQmpgd`#{Nk~dE zl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`KMJP%!ic^A;l%h0c zC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=ow4ya_XiGcV(}9k3 zqBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@uGl7XrVlq>h$~2}k zgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVRH+$I2 zKK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^>hdkmj zPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLYgepMV4+FhK}PFoF|; zkc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygxX-G>t z(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=gRH8Cf zs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b(}SM$ zqBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M!GlQATVm5P_%RJ_@ zfQ2k#F-us=GM2M~m8@blYgo%V*0X_)Y+^H8*vdAxvxA-NVmEu(%RcsVfP)<3Fh@Ab zF^+SBlbqr-XE@6_&U1l_T;eiUxXLxIbAy}Q;x>1<%RTP%fQLNdF;95PGoJH;m%QRN zZ+Oc)-t&QveBv`-_{ulF^MjxK;x~Wz%Rd72^`C$QA}~P+N-%;Gf{=tFG+_u!IKmTw zh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+S;$H@ zvXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2% zVlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)= zInHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`Y zU--&5zVm~h{Ngu%_{%>6^z)y91R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uO zn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX z@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0 zXi77h(}I??qBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1Jx zfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0D zHLi1mo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxC zfB4Hk{_F2PK?p~9A`zJwBql8x{-@|3!{s{L1prU>q)D6fYtu9~8{1AA+g4-SwvEQN zZQHh!#~Okf(*nawg* zu!?nTVH-Qx%|Q-xf|H!$Ea$k)H6HVgAB5>+ydoTbkeKA8AQc(NPeBS%nDSJl61Dk< z1~jAzz3D?=1~P)N%wr)-SjIXwv6Wry;Ru(x&qJQ@ln;F38^ONP%`f~;WTFt27{nny znaD~`a#4s9l%gEvsY7#G(1igE_{Ey#>NL1pGki;Y*C27e{Axct)iqxhK4QNUWTGELg^rb(; z7|v*Bu$r~3X9K&~%VExPo(o*#7Pq;_L!R@J&wSw`Rq%Mng+o|`=8 z32%7Edp`4(AA}lcO~UU)AR>{8N_G4@D_X3Cd8NwzQ)?9qGXUMlgzTOlB(6 znZZovGLNNfW;;9B%RWwXk;`1=F86uBBOddDH+<&@!9mtiMBoqpBoa}HL2QzelU(HG zZ%R;_@>HY}m8ro$G@=>JY0UtJFp6 zNG5WVj{+2=DCMb06>3q3dNiadt!PJky3w5>3}Y<+GLu=%VIfOc!AjP!jty+%7?-%r z4Q_FtS9~MvQ28JZ>B&c7icpHul%*n-s7_rP(TtX~r5l47%rJ&ChH*?}3R79YVpg(> zZJgo^XF1OWu5gtb+~gh)dCXJZ@PUtfCDbtU7~zOWBq9@sxWp$hsYpi#ic*p))S@=^ zXi9Th(UuN$rW-xz&p-w-nQ6>m9t&8=YSy!deH`T!r#Ztd{^J?XdC7MI!;Po>%>Rf+ zbYc^azeq|7Qjv+wWT5~BDMB$yQ-b*M`{>eGPMbf5=48OR9! zWdf6##cbxWjP-0{D<`?gC9ZIjTRh`4U-(9-k>+cD=MN$hndrnO4)I7qDzcN4+~lPo zC8d=@rbf7D}=*uX^@Gn!C!!lN~nl)@>6MHzsVUBQ?t32c>&v?lz-tnH#d?VB- zc_Iuy6OJgvAOUH}P7ZQYhKkgt9v$dPcY4r=QOsuno7uu14s(PvoaF+SxXFJ!OTN>PTgl%qTqsZ15>(wd(1 zr9Yz>%Os{Ti`mR)5ldOYcJ_0S6P)D&m$=UhUh#(SgdVHxiAHRal8jVjBokT4K`!!9 zm?9LXBxNW|1u9dW`ZS>heHqGd#xa>$%w{eN*~~T$aDp?O=Q6js!)xC0iSPU%%sA!G zFN7x&k%>w)5|WHGq$3;oDL_GrQ=V$nq#g}uNOM}!nqKr_6yq4rROT>``7C5BJK4=% z4seh|oZ}LgxyCy_@PioRjg!P8Az8^m9{#2Pg(yn}s!@Y~s7nKy(~?%SrY-GgPj`AS zhS|(vDa%;HI@Yt9ZS3R#hd9h}PH>C6JmUo)`AqO%<TC$RZ zQk10}<*7+C1~Gw!tYS6m*~)fyu$NO@;Vut&%zHlYnZN|&A7S{JKZr(j;t-$Yq$3lV z$w5wvQIaZDr5d$qL~~lvfo=?ED$|(5T;{Qi)ofxb+u6-t4setcT;&G;afiD+;2E!Y z!#h6lgOG{lS;7*5NJJw#2}nY6(vX>~{q8EMXM}Gz}jFC)aABvOfoW)m2Bjq0EH<= zHL6pSTKq!?y3miYOk@gkna={2u#BzjU@wO`!3$pUnm4@V10mClRs2Q-q7#nTJ>mJCNJJ$u>BvbgN>hOv)TS;CXh|D7(v`uCW-K$9#ax!KoHeXt16$b3aZYlV zhx|0dJWP20Mmoa)vA1v{Y+y4xIKVm1bCdtL%X41yj*omM zFjspZ3NcAYB2tr{tmLC4rKv<^s?dPu3}7r1n8H++v4X7};W!t$#7%B~Q-E$n7L7rDkQ{^KF< z_)6IM#wB79kAx&9C0WQpAxcq+D%7AJ&1gXn3^kWFa8OguQVHqn}#cH;)ot^CC z0H-<2Ij-=CSNyTS-i27iCjkjbLP| z*v4^AbA>zH=K&9S$~!*rolpz)bABc~zY>8ziA*dKk(gwpCMS6)LLKVUjLvkWJ3Z-5 ze}*uYiA-k}D_G4sHn5eGoZ<`@xx)jV@|KT$;yYm%Ni)BZh}`6-AjK$787fnQnzW!b zT^PV1hBA^dOlAhNSj1A6vzB#iU^Cm;&LK|okS9Fj4etrH*xH!T{6rYyl7wWWAw8MN zN>1`nit;q1CGF`zSGv=O!3`=>e7au3}ih2GMSmoVLq!_&rYs!j|V*F6W{r1nem0+i9~EtlYu-Gq!6X4 zKuucGjR{O-7PDEvLe{d5^=x1thd9X%-v57}~5|EYLRG=dDs82(> z(1V`zrVoAT#~_99qi>0r#Q_S?r@KXJmooW`M_s>T`9dJAq52} zLuLM}4NEIL-;K@{lJy=Pe)j$XC7-Sf!uw6OlWEu|`&O zxXfK%^NBF)q>117ok+wYDalAlX0nit+~lDY<)}sj+R%f(^k)D=7|l%Pu!J>iU=usp z&q0oHmaE+4HurhTGrsbTu^4$K@4RiWBHdU z%wQgi*}`s)a-Lh<k-9XbIc@1l z9|kgv(adKBYgo@NF7ceVyyF8O3A;%?dZx7 zMlzaNEMf&KS<4U>L(0!FZ;#fMu*?BilI2NzQYPJG|mMp|(gXVF*WfA`p?NL?aFfNJvsrk%OG% zqXcCrPerOzoBA}NDJ^M7dpgjKAq-f_Hr8 zE8mE?Rl6WMu}DBdl98M&WFXhAwoc7Xui~P{uNzY0O{_ z^I5<`ma>tZ9N-|QxxjT^@tSvhAk;Qv6u%LXNF*UE*~mdo@=%hBRHiD`X-pG3FohW` zVi~Jg!&=s{fdic3EcbZGBi{0zU$7m5}$PBASb!VOEHR5n(|blE)8i)OIp#6 zE_A0ieHqFqrZS&pEN2aeILs;T@r>tu;4@$NO1NFpNkpQOkkn)%2YJX(MJn+RjcH9A z+R=qx^koo(8Nx6|GoGo;VG+w%&Q=a_j{7|3DX;iM*xkk@ej^fT$wW4Ck(YurrxV@i zK~IJ-j4@1L8cSKmat?8tGo0f*m$}X@9`KYmd?L8VdYZ6=CnAwZLNYRwg`5Za#E1O6sHBPX-8*9 zGK#TGVJg#^%{&&fj8$x46Pww?UXF5tQ{3P-_jtey-t(C+gxatEh(%J;l9OVTq$aIs zPj`CIpCJrqJd>Et3O2HXW1Qg@|M7$`1P*A!{7H0T5R2HPATI@}LVeoOp3Zb(4hvY# zMz*tu{T$^q=efmy+~E=L_`qj=5Im?26NMNgCl`N{p8`~-Dh+8wZ~8Kj5sYFqV;IlB zOlJnmSiws6ag0-(uLe{d4GhF5c5sxSj5|N$^L-?1uEN3-q*u+WhaF3_F;Va*X za7?=-4)KUjQZkT-!W5?jWhhHq+S7}^jA0xLSj=+Pvw__l;S6_pz(bz!oR@qh%yD@q z0`W;eLK2aLf|Q{ged)&_Mlyq0EMo;LSS;SJ-vXM<}VJrJNz7o7llF4so3qyx}ch`Nntt zcS^e@3bBYyd=iq1)MOzm*~v{YN>Q6S)TJdI7|C=Nu#nyC;UYJ9!f&V57n#UHZVFO} zqLiRAwP;3jS}};R%w{=jImU4=afN$4;RB)0XrF}Ve?%b>iAhR2(vzLM^$ZPk75WemkrDiAF*ak(6X4Cmrd@Kvwcl zfFcy9G-WAIC2CWLwzQ`M0~x_0wzG$`T;o3;@Py|C&si%Gg*e0`KFLTyYO;`rzbQo( z>eG~Fw4fEOX-ikS(T^bvXB-n)#A?>Dk-hBa2*aF<6s<_RD9M)*tGCF#jbNlMX}=Cq+L?dZjD#xRv-tYa(tILRd*^Nqk| zc_u8e$wprCQ-R7L`Una4b&1_>ApNV$EJWmYbkdP!4rYNPUOidcl zoGHv=K8sk!Hg<886I|sc4|&RS-t&ncM7XI=NJuX7@;9X^Pem$GjfQlfGrbwX7{)W1 zSd~0GLe%46rwn#DMJOS(wJtnq&34B!!+iwmYtmB0vEZ$ zBcAh?@BDV#{fR&nqLPH1g z%qn(rjho!z0Z$3uw{{>oDagm)l%zcMX~tkiGK)DZVikut!g;Q6lec^%^aJw+VfmTp zBqKQ)$U_l|QJhlLq!x9kOC$O*gyBqL2J>0UdbY5Go$O*aCpgDd?(%?_d?vy}ZHhP~ zCL`I%O+E@xn4%P;Bo(Ma1Dey4E_7uCW0=51Hn59JT;Uq`dB|g467i8XM?C%_87W9Z zMsibtid3OGHEBQ>`ZJIr3}q}+n87UOvX66I<~k2}%oAP{>aqGKIx&b%Qj(LKzo|-H z>d}SKjAtS~uQ@YZPp7drgW0}fy=CF_rY~d(( zdBPjs@}6){-G|gW%E=AuoT^gqC!n2Sb^{3>LAL4Qyv0$GE{0Uh;;|eBm2E`2D%OP>hl^q&Y2U$wa2H zlI?7k+-Be)yf}{6$J~QHZkCr3tO*NpGgGiOn412*v5+~OV&c+MvRuhcz7s7Os}QJ01^ zp%dL0#t6nRfsO3r3>Ue@J)ZK0&;0pXdm|xQj&@cWFkA&_=h?)r4yr=%rs`Ol3ncM07p5&Bi{0zAB4O$UK5_g zq$L-XXhT;9FoJQ+WDZMM%K@%(g9kk15nRRD_PGr zwzG#bTq5wnctA8_lYk_oBsFPBOFHsVl;TvOBi$LyP{uQxrEKQ_XSm2UuJeM>AKjB! z#3K)XQ-DGgr3%%lLpwUqkr9k%5ldOm1~zknC%oVd?|4tjPx3$(vXhU(RHQBK=|p#W zF_0k)Wdx&{$7YUjndiLYE8qFwXKN#plbSSSCJTiqPE{Jxg}(G>00SA#SSB)^SuAHQ z>p99}z7X=oT7}poAvxK|K`!!8h~ku?91ZD6C%Vyt;ml$I3t7x|c5#419ODF6xyB7{ z@|ur);m@z;O;VAXbmXBVr72HkYS4%-bfp{rvVeswVHKO%#xC}-kF#9h25))KC&GR+ z?vR39v~(KVz82OqQ{d^=xD_hdIi5E^?ixJR@|7 zJQJN5B&INpXhKK2(2cAn(JmQm#RHPv-Imk^x3R8q? zw5JQh7{wT-vWiU{<``$W%R`>KqTT2mn3AN9)0P@NG5QEM||Q7-}&_? z?BvS!YEz$vw4)CI4v zF`V&i<`5@2%O#%hhIfR74HXE(pQIrhr724_YEhRaw4@WA8N@VZFq@U^W*>(*%@xA^ ztc>^{k%>w){vru!$U;5}P?K78rzg{x%OX~@jqU8@FgJ+zi}EHb*(gFuDp8gCv}6E- z8Nqy3u#ydIWDol}&OILRlJ|TkR5)#h$iyZNX~{!=3Q~l+w59|7n94NPa+o8W;yjnQ z!(ATml92GqjcEKuN>Y)T+!Ui6O=(4U`Y@DfOlKu)Im&siaf91@B+Rd_Ph66cf|OLI z9u4Wh00uISsmx#|vsuhCHnEp|oZz(=C|CO@PgC%MSW-&CPGt>{EI`Y@VtjAsHd}JMbfyQr=|g{pv6bB%;4mk-#B<&dDS|dmCUR1U;#8z2 zwW&*c`ZJg@9ONt)2>xL_CmeqeoAl(M3iW71Ck8Tw6|84Bm$=IlzVS;${hC;$CKEX+ zKuIdnlCBJ8EaREWO7?Jz7knVppVG#k#3CU{Nlj+*P>Qlt;~(nNgl4p+E$!$?7rHZ$ zp^RcI)0n|Lmb048oaHkA@ru{H<>yH9N<^X(i-aU1HEGCBF7i`^(p07%4QRp~=COp0 zT;MJ*dB<0Rk@Zb}Ar1*hL|XDtgbFmF7sDCBWM;9R!(8Ax_j$+@Uh;vj1fnQsVv(A3 zWFR+%C{0Zo(3B2zr8gs(!~za*it9Y%CEp1hRk`sCzY~R+Bp?gf$V~wXQ>MIkCsnYwgk8Z(%~2KIA+!`$RPPk6(3 zLPrl32unC35R+7-qZfl1%nW9;m}RVCE$i6AR<3ZBTRi0*p9mAfctR{Pl8t;+p(;&j zPe-~kg=OsG80Wdn4Q}$4U`%yDbmEeVtmLK?rKv$p8q$PT^kE`1n9mwEvWe~N=N|6~ z6HD2Un}U?26lJML1DZ2{#Vltn>)FUIj&O#j{2V(}AUu)DN>Q5AmM-*U07IC}3^sF| zvz+5RS9nAqj&dM0;fY8bl9QTrWFjj?DMcOX)1F=oV>qK2%~H0pogLif1>gB8t~EQ6 ziAy|El7n32p(N$0LIXO`jqdbiBHKC4ajx@-XM~QYjS!ct6r~s?s6ic?(~dpW>CPVkVIeBmom5-4A?Q<}1rqYD4fk#6*6 z6w{f>9OkiuBOK)zXSv8--td)h3EhKuBqkYI$WAG$Q-_9hWGLg9%yj0nlFjVkIM;bj z@xHGm@3;W)CO1!c$)IhW{njuZc-~l9HPA)S@MA=|ESy(VLNs zVl@AQJ9{Oky%qn8R{*aezx);X1c?%^N-ul2ZN& zODu9wfC@CDJv|x9ForXhNlalM$2iYb?r@g}Jmn1^_(H5y+BIp(%->X|KCPI@EM~Kw zZJgvZXF11tZu5*cyypWy2%Xwql88hm7Rks%Y06TO=5(ejz39(SCNhIL9Ond=cu2-H z=2`NRpHh^g1>G3VSSGT9Ega@3clkk>wE7=0NyJ~IB0bs2O#$lBly3B99CKO7N>;O% zeH`Homw8SgoxV+6;*o>`6r(KFXhaj*(w>fVW)6#4%^EgxkmH=;GS|4pOJ4Jd@BH9@ z>8&4#Ndi)kp7K*etk#3A&G*g+z95!&7n*=j!FT^4tsYpvsDp8;IbY(cB8Ovm5v4)+T<|6-bm#4ht zGe7t;%$7GhXhAkZA z9GAGwT^{g?U^ewfJQ9H^yH-o)%b@V z^kfN}*vf1?BEU$ct%JbX(s~FNl9+<(vEH{XD53(%?BB%qFqy@yWHY%B=LYxq$`8W*Z5>QJ3R0BvRG}uds6!LFF_|eWVFyPz&N-g*k?(}dZ>}XJ zsVG1l+R~eTjA1fUn8OB+ah4~fD4-vck>Zr2CUt2*OIp*L!3<#m3t7xkwsMf0yy7*X z3Tm_bN>q}Oiu@F&2-WCEe+Dy?*{t9I=eW*0A{R17l9&{vCo9>=Lw*WSl{z$~6@wYV z7`AYb6I|pgKL}e`-SZczNKFxHQJ=vKVFoi<#Ad}_zY-TTqxz0Vl5vrIulPJWZ z5T&R?L#8sFSY#jOEIM;7u?j2g6K7?WAT9u9DvlRV-baZ4y?@{*5&6rwx* zSjv8G@PxMnN}6-{lWY{E8VzVc2c~j{tK8!g-w0Jo-S9sWl9WtTrv^1?LuZz=mIu7z zJ>g0lyZDpXq$UH^Xh|1_GKR6tXCph<&v~wJm76@`F>eW!(YA?8YSNOG`gCR(6PU_o zwsDfPT;n{mG^0B`n94$ybAZDf;S^7K&rcPU8QCdB z4eHZ@&h(={1DMVNRHB9^k8Eo|c@9|>1o|0fk0DN1oF zQ=K04V-!d=zTj9@ZL*~b+g^McoetEJ3|LK2dbhHPXfA0;SFS*p{V7R+Q5+t|fk zPH~CPd?Tc`Jd=zRq#{4X8P61!vWu5Q{73zhncNhlC4CvqSY|Vay#(v%JH#X&f02d^ zWT6b@s6cHdGnF|kW(^zJ!CuaDn~(fdSDzsvDak{A3Q&mhRHZt#7{P27aGEE)B27K- zp)$1@&Q#{JfaPps2m3k8EpBs7mBXCi3~!0vKpm5t zVw9l@4QN6uMlq2oOl2v%ImLCJ^PNCLbxIa;(~y?*V==qA##mLJ zU?78-#ZFFhmq$D%Y*S+(k%&f4@>7^%l%gvmnZyE?v6k)Z=Nwn~POzE05uengAsw}8 zN=Jq26l3h5B%Ix+a(&w$xc3+(SpH@V;0L;!7ZK=x|KR2IVI`K0ERG>nJncL zSGdUoo|CS%{!ea7P?Hg1+RHWpp7*h@ySaGN>Phu^k5_tn8b9} zv4JyOvS>)4TCtG8jp8AJZY4tk=}U(MN)xGENZ?V$kic)1LIT%ogan$`3<{3t zO$zZ{_#uI2Q$qqBrib{B@Q}a`ex2ntgr6M}D8C>iFlk{(pwXg`z`4aCfw{{<0vnfy z1jeoi2^3rz68LFVNT4c1*Gl_3dEFWkIJ!-`c7z0a?+XdsrQ8Ak9u5h7VbIZ#K;)CI zeI_K3ffM|5PMMxp23Mu=hHKmk3B3F-B+%+^NTBn*kib$FJq!_lB0rBq0>3;738W?8 z(~!U?M!pCM?0*>&*!;$Q-iHLndzvy1~G*_(6XU2?kOo4+e6i2nKqj3 zWVWC)vtS@^u3(^HZm%g2w3iPC#up9-4i=?&Fp$4wFz}{KFfgl9Fp#ryFi@~cFi^Z^ zFi@5SjHwe0yd`SgpgB1h$X?$)8U+L28~a@gKer49lC};8GPVf@hII%A&UW>l9zo9z z4|)!C&@-2Vp35o^#Of0aZ0#o<1A?A|=N^NCKFAyNe5GJu!suWi+?b%}ZFuj*V4&R8 zVBo{FV4&CZU?ASiV4(HfpzmJ}25QZh&&9z&1FA0#2Es0rp5?(njTOPbZ!6_}buch} zO)!vYeK0U;Lol$1o12uumSEuUj$q*B&S0S3u3(_U9%ZvP=zBkcf!*w3FZ;3J>-%oo1$jqm)x2QmYp2q8#lej*HE`I%n`M|ghafBeSpMBoo1 z@+XmqOcbILjp)Q6Cb5W39O4p>_#_}9iAc;}Bq1ruNKOh;l8V%%AuZ`hPX;oQiOggn zE7{0S4sw!<+~grI`S_ds6rdo5C`=KGQjFr1pd_UzO&Q8kj`CEXB9*926{=E=>eQen zwW!TM)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P z=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@zf52vlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG z%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c( zB&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+_>bG%;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^ z?|9D#KJtmreBmqK_|6aR_3?T90b+#)iVtEnhq+DqLE<6?``u8n4a0oqoQc!szP9?ul-Ldsb_zduK8 z0l&->`#~M%vU0xIBjPQR778r(+GS$0m_~^e?!f~-l5dqb0+y}zzBOVV){4&2FA5RbxbhHmt`&ElW7h@so69N5mx?PAafw?iD;PH`W*Tx+*j2WsvS<3Z=W;)D1? z>-~=T4vJmp$RXeBa9HdIO^=9=VIdihid$g{D~Wr|(J6l%7k|RL6JkzJiv8m}DbIKv zzn(QkvVjxiJ12I6Z)7?z9T${4yGeXe+yz5fN0v+C{m61foCmdNOq8q2?V9)%vR;>V zI&gs;HOJrk-&K)_xL@b z^w5f3oZ%AD64@`ZlQW5(5Bw$lEKFkmk<6J?a(lLv_INCzbt?Oh)UHpXH1a~dV#Z9*lR zy1$9t%wEAvW|7q1(t+s&ZRw6=H1%8Cfb0Yf+uAC*#SeFPwGMV6=JTPWt*(<@m1*wL z^V}q27tawT*~wc%xI4oUfrvyQHi<|>269r6A{3<nZsP>v4BM^VJXX4&I(qs znl-Ft0~^`IX123~o$O{0d)d!H4s(=a9OnclIn5c)bAgLo;wsm;&JAvIi@QAF5s!Jo zQ=ajP*Sz5^?|9EgKJ$gI{N@k;_^+GB2}}@z5{wXpBov_uLj)obnJ7dh8qtYCOkxv< zc*G|GNk~d^Qjn5Vq$UmN$w($LlZEW$A~$)+OFjxvkU|uu2t_GIaY|B((v+bb<*7hL zDp7^1RHHgIs7WnqQ-`|LrvVLVL}QxJlxDP`C9P>oJKEEMj&!0kUFc2^deVzN^ravD z8NfgWF_<9?V+11^#c0MbmT`<{B9oZHRHiYV8O&rBvzfzO<}sfIEMyT&SjKWzvWnHL zVLcnz$R;+kg{^F3J3H9PZuYR3eeCA|hd9g;j&h9SoZuvQHV-3q7#Fd#3D9vh(~-9kdQ-nkC`cg+Q-q=vqc|lfNhwNGhH{jr0#&F=HL6pCn$)5; zb*M`{>eGORG@>z0Xi77h(~8!#p)KubPX{{Ei7s@d8{O$iFM895zVxF%0~p9)hA@<2 z3}*zR7|j^QGLDH%Vlq>h$~2}kgPF`?HuIRz0v57}#VlbtD_F@YR>(9#cuYnmwoK#00%k5VUBQ= z%RTP%fQLNdG0%9;3tsYy*Sz5^?|9D#KJtmreBmqK`N2-n zYE-8NHK|2y>QI+@)TaRrX+&e1(3EC0rv)u(MQhs7mUgtK10Cr^XS&dpZuF!Vz3D?= z`q7_(3}y&J8OCr%Fp^P>W(?yP&jcniiOEc1D$|(G4CXMGdCX@43t7Zsmavp%EN2C) zSj`&NvX1p^U?ZE@%oet?gPrVRH+$I2KK65fgB;-~$2iUjPI8LVoZ&3zIL`$xa*4}a z;VRd-&JAvIi`(4g9`|{`Lmu&%Cp_gD&w0r!Uh{^xyyHC|_{b+d^NsKP;3vQM%^&{q zj{rS&N>itB%ugR7{U^c@I)XYk%&wbq7seh#2_ZIh)o>g5|8*KAR&oJOcIil zf|R5pHEBpoI?|JYjASA+S;$H@vXg_H@0trU*qTMsZ3|iqe#!EafOq z1u9aB%2c5$)u>JlYEp~Z)S)i*s80hL(ul@1p()L1PAgi|hPJe$Jss#sCpy!Gu5_b2 zJ?KdC9jzvzW~s<}#1@ zEMOsvSj-ZZvW(@dU?r9OMv(Il@tn zahwyJ{7 zOcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKIPOk)8}>Bomp*LRPYoogCyO z7rDtpUhrl%y1;DMMMxQJxA^q!N{>LRG3!hq~0GJ`HF{BO23$ zrZl5DEoezATGNKMw4*&8=tw6z(}k{dqdPt5NiTZShraZqKLZ%ZAO&aK$t-3whq=sSJ_}gLB9^d}Wh`d}D_O;A*07d!tY-ro z*~DhHu$66WX9qjk#UA#ukNq6rAcr{25sq?<W_xyE&FaFbiy z<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`N2zIH9 zA}~P+N-%;Gf{=tFG+_u!IKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIAt}j7P6|?z ziqxbbE$K*41~QU~%w!=e*~m@~a*~VODP6Jl>QI+@)TaRrX+&e1(3EC0rv)u(MQhs7mUgtK10Cr^XS&dpZgi&y zJ?TYn`p}nt^k)DA8N^_QFqB~oX9Ob|#c0MbmT`<{0u!0UWTr5cX-sDZGnvI~<}jCe z%x3`$S;S(Nu#{yiX9X)+#cI~DmUXOW0~^`IX11`EZER-;JK4o<_OO?I?B@UnImBU( zaFk;l=LDxX%^A*ej`LjLBA2+#6|QoP>)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk z%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>9ieRNJh0uh)X1SJ^32|-9g5t^`sBRmm^ zNF*W?g{VX$Ix&bzEMgOfxWpqq2}npH5|f0aBqKQ~NJ%PElZLdUBRv_&NG39qg{)*F zJ2}WnE^?EHyyPQ41t>@%3R8rl6r(sLC`oC`P?mC(rvjC!LRG3!of_1n7PYBEUFuPv z1~jA*jcGztn$esVw4@cSX+vAu(Vh-;q!XR#LRY%cogVb07rp62U;5FX0SsgigBik5 zhB2HGjARs}8N*n{F`fxbWD=8^!c?X)of*tz7PFbdT;?&Kg)Cw*OIXS>ma~GDtYS55 zSj#%rvw@9lVl!LV$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo z;xbpb$~CTYgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$`5|> zi{Jd=FaHS8SK|aE5P=CoP=XPh5QHQYp$S7+!V#VbL?jZCi9%GO5uF&sBo?uWLtNq! zp9CZ%5s67cQj(FJ6r>~-sYydx(vhAFWF!-r$wF4Lk)0gmBp12KLq76TfPxgFFhwXz zF^W@yl9Zw}WhhHI%2R=gRH8Cfs7f`eQ-fO6rVe$fM|~R5kVZ772~BB6b6U`nRY(34*DrVoATM}Gz|kU*>T;VF$xXul3a*NyC;V$>M z&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP&`Z7q7j`K#3UB6i9=lC5uXGkBoT>8LQ;~EoD`%a z6{$%>TGEl83}hq|naM&{vXPw}ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5 zdeDAZhTiM2TcCeFO>}C&p*~fkkaF9bB z<_JeQ#tBYxiqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}h zir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2+&{i1SAlF2|`eU5u6Z&Bov_uLs-HQ zo(M!F5|N2QRH6}`7{nwNv57-m;t`(&BqR}uNkUSRk(?ByBo(PiLt4_2o(yCp6Pd|E zRP^DMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~ zwW&j0>QSGDG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIPH@eeEMhTBSjsY% zvx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9_H%%P9O5uXILa}ObApqc;xuPC z%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$JmN7=c*--L^MaSW;x%u0%RAolfscIR zGhg`1H@@?OpZwxCfB4Hk0u1;sz<&fJ5P=CoP=XPh5QHQYp$S7+!V#VbL?jZCi9%GO z5uF&sBo?uWLtNq!p9CZ%5s67cQj(FJ6r>~#X-P+VGLVr>WF`w)$wqc^kds{GCJ%YZ zM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_ zxyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs z<_~}QM}UF4Cm?|cOb~(+jNpVIB%ugR7{U^c@I)XYk%&wbq7seh#2_ZIh)o>g5|8*K zAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p z8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J37{)S=@l0SM zlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w z>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+xXm5z za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w@{a(6^iMzn z5ttwZB^bd8K}bRonlOYV9N~#TL?RKHC`2V1(TPD!ViB7-#3df_NkBppk(6X4Cj}`< zMQYNJj`U<8Bbmrd7P69!?BpOPxyVf(@{*7I6rdo5C`=KGQjFr1pd_UzO&Q8kj`CEX zB9*926{=E=>eQenwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o z?)0E1z35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s z<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a z;3cnk%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_+oh7!=??0uqS81R*HF2u=t>5{l4- zAuQntPXr}a>$Rs8+g{e$qIy0EbEM_x@xy)le3s}e^7PEw< zEMqw-V?7(#$R;+kg{^F3J3H9PE_Snrz3gK@2RO(f4s(Q~9OF1AILRqa zbB42=<2)C*$R#dwg{xfSIybn`*Kmrk%AOs~C!3jY~LJ^uUge4r|i9kdm5t%4N zB^uF*K}=#1n>fTJ9`Q*)LK2afBqSvn$w@&^$tANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJ zLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0 z!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tF-b^DGLn;ml%ygxX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgF z2t_GIaY|5@0trU*qTMsZ3|l2VkW3}q=t zc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUV zUFk-5deD|rna*v|nD za)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8} z&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE_;0xG2}EFm5R_m9Cj=o0MQFkh zmT-h80uhNsWTFt2XhbIlF^NTN;t-d3#3um>Nkn3jkd$O3Cj}`YE-8N zHK|2y>QI+@)TbehXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd7 z3}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZ zvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkGj`w`v zBcJ%p7rye1@BH8=zxd4`{_>9iBeYLI0uh)X1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh z8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f( zJme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu z8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@ z8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhVdfCM5iK?q7Pf)j#}gd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkc zJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`K zMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=o zw4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@u zGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9lVl!LV z$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvq zHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-GrcKLU)> zKLH6uV1f{oU<4-wAqhoj!Vs2lgeL+Ki9}?g5S3^|Ck8QzMQq{_mw3b{0SQS&Vv>-Q zWF#jADM>|Y(vX&Pq$dLz$wX$dkd00k*TVTw?cViczYB`HN| z%21Yal&1m}sYGR}P?c&_rv^2tMQ!R(mwMEv0S#$HW17&EW;CY-Eont-+R&DEw5J0d z=|pF`(3Ngq#cl3zmwVjj0S|e^ zW1jGoXFTTxFL}jl-td-pyypWS`NU_w@Re_T=LbLe#c%%bmwyBpt$zX%h`h{PlzDalAq3R04a)TALD z>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ z7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@Q6^rAO?=u1EPGk}2% zVlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$G-o)= zInHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnwk9^`Y zU--&5zVm~h{Ngu%_{%>6jMYB@2}EFm5R_m9Cj=o0MQFkhmT-h80uhNsWTFt2XhbIl zF^NTN;t-d3#3um>Nkn3jkd$O3Cj}`YE-8NHK|2y>QI+@)TaRrX+&e1 z(3EC0rv)u(MQhs7mUgtK10Cr^XS&dpZgi&yJ?TYn`p}nt^k)DA8N^_QFqB~oX9Ob| z#c0MbmT`<{0u!0UWTr5cX-sDZGnvI~<}jCe%x3`$S;S(Nu#{yiX9X)+#cI~DmUXOW z0~^`IX11`EZER-;JK4o<_OO?I?B@UnImBU(aFk;l=L9D?#c9rPmUEov0vEZ&Wv+0Q zYh33BH@U@a?r@iT+~)xgdBkI$@RVmf=LIi$#cSU1mUq1810VUsXTI>2Z+zzmKl#OP z{_vN71Q@4(0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcni ziOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOpBomp*LRPYoogCyO7rDtpUhrl%y1;DMMMx zQJxA^q!N{>LRG3!of_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-;q!XR# zLRY%cogVb07rp62U;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X)of*tz z7PFbdT;?&K1uSF{i&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWRUiPt{ z103WKhdIJgj&Yn5oa7XzIm21bah?lYUG8z82R!5vk9opV zp7ER)yyO+HdBa=Y@tzNShfil%qTqs7NI$Q-!Kj zqdGOHNiAwqhq~0GJ`HF{BO23$rZl5DEoezATGNKMw4*&8=tw6z(}k{dqdPt5NiTZS zhraZqKLZ%ZAO&aK$t-3whq=sSJ_}gL zA{MiRr7UAPD_F@YR>(8$u4%YhrR4$KLC zL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^ zkds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz|kUW_xyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxGeC7*Z z`NnsC@RMKs<_~}QM}SHCCm?|cOb~(+jNpVIB%ugR7{U^c@I)XYk%&wbq7seh#2_ZI zh)o>g5|8*KAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>Y zjOMhUC9P;p8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J3 z7{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1x zo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj* z+~6j+xXm5za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w z@{a(M^-n+o5ttwZB^bd8K}bRonlOYV9N~#TL?RKHC`2V1(TPD!ViB7-#3df_NkBpp zk(eYTB^k*{K}u4Qnlz*(JsHSICNh(StYjlQImk&aa+8O=lxi$tXrMhOvxeJQJA6BqlS3 zsZ3)!GnmONW;2Jm%ws+aSjZw4vxKEAV>v5W$tqT}hPA9?Jsa4_CN{H$t!!gEJJ`uC zcC&}Q>|;L%ILILmbA+QD<2WZc$tg~AhO?aGJQujgB`$M?t6bwcH@L|yZgYpb+~YnE zc*r9j^Mt27<2f&Q$tzy-hPS-qJsKlsTne)EUF{3E~=?Gund1SSYU z2}W>25Ry=YCJbQ-M|dI-kw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoS zCJkvxM|v`lkxXPJ3t7oVc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnx zkxEpi3RS5_b!t$PTGXZvb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0( zcY4s1Ui799ed$Mk1~8C83}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf!u##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh| z9Oei|ImU5LaFSD;<_u>!$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I z@RC=&<_&Lo$9q2TkxzW)3t#!hcYg4bU;O3|fB8p%srn}%fe1_xf)b42gdilL2u&Em z5{~dhAR>{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKG2jl8*FbAS0Q` zOct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRepdyv1Ockn9jq22( zCbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+jqdcIC%x!RANtad z{tRFsgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$9Og2Q`7B@|i&)GO zma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9SoZuv< zIL#T(a*p#{;3Ai}%oVP3jqBXtCbziF9qw|E`#j(wk9f@g4AVmb7zO|Uu&%ak+qP}n zwr$(CZQHhO+jf^jo{%Rz{N*13 zruk1m0uh)X1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlB zBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j} zQJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|* zKu0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW z9`TqbJmneBdBICw@tQZhfTJ9`Q*)LK2afBqSvn$w@&^$tANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4 zQJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMp zK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy7 z9`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1 zUh$eYyyYG5`M^g$@tH4t>it7{LiaNJ0^sFoY!> z;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdON>Y)UG^8aR>B&GwGLe}q zWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tn zz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^ zGo0ld=efW|E^(PFT;&?qxxr0tahp5b-n zMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5 z!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft z9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{of zD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*13X8BJ*0uh)X1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$ z@ku~J5|NlBBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM z6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX` z(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|U zJKW_S_j$lW9`TqbJmneBdBICw@tQZhfTJ9`Q*)LK2afBqSvn z$w@&^$tANeUjK?+frA{3<<#VJ8aN>Q3J zl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1 z(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63& z!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&% zCp_gD&w0U1Uh$eYyyYG5`M^g$@tH4t>it7{Lia zNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdON>Y)UG^8aR z>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ z7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5b-nMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgW zF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^ z!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*13=J`)R0uh)X1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYC zOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G z`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0 zG^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhfTJ9`Q*) zLK2afBqSvn$w@&^$tANeUjK?+frA{3<< z#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&| zw51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQ zF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a) z9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4tDP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP; z&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ- zW-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna z*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg z;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2(Zw90uqS81R*HF2u=t> z5{l4-AuQntPXrvz z-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;9 z7P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL) zP7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwo zF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM z!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3| zRh{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJ zPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iOb zDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB z(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQ zHnNG$Y+)*>T;VF$ zxXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6 z;V=IPu*81?5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk#33&6h))6% zl8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VOD zP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^ED zE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x z$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(y zcCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?w zxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2(Z+D0uqS8 z1R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm z+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbb zl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p= zP7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91 zFa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57} z#Vlbd%UI3|Rh{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J& zl8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2P zPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hD zD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd; z)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK z_|6Z0@{8a6;V=IPu)===5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk z#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VODP6JlYEp~Z)S)i*s80hL(ul@1p()L1 zP77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW= zEaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9N zZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE z2(Z$B0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1 zD$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_) zq#-ToNKXbbl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{ zQi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|! zP7iw0i{A91Fa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR! zF7uer0v57}#Vlbd%UI3|Rh{PlzDalAq3R04a)TALT=}1ooGLnhR zWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)P zQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD z&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^n zDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D# zKJtmreBmqK_|6Z0@{8a6;V=IPu*QD^5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3 z(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VO zDP6JlYEp~Z)S)i*s80hL z(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G z&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))P zE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l z%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufC ze)5ao{NXSE2(Z?F0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq z&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp z$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_ zl%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^ z(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@ z&J1QUi`mR!F7uer0v57}#Vlbd%UI3|Rh{PlzDalAq3R04a)TALT z=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM z(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES z&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy z*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IPu)%)<5{SSAAt=EJP6$F0iqM21Ea3=G z1R@fN$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e z*~m@~a*~VODP6JlYEp~Z z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KK zGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4 z&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9o zEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN z&wSx4-}ufCe)5ao{NXSE2(ZzA0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYr zGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e z&jvQKiOp`P7Goai`c{=F7b#@ z0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w> z#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQv zw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SI zGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3|Rh{PlzDalAq z3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+br zs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>G zbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAl zGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtP zDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IPu*H7@5{SSAAt=EJP6$F0 ziqM21Ea3=G1R@fN$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*4 z1~QU~%w!=e*~m@~a*~VODP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D z^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+h zvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A z&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1 zE$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2(Z5{l4-AuQntPXrvz-t?g_{pimC1~Q1j z3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4Mo zvWnHLVJ+)e&jvQKiOpP^DMC?-QJfN#q!gtoLs`mE zo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@Wo zSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rB zvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ> z9OMv(Il@tnahwyJ zBomp*LRPYoogCyO7rDtpUhrl%y1;DMMMxQJxA^q!N{>LRG3! zof_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-;q!XR#LRY%cogVb07rp62 zU;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X)of*tz7PFbdT;?&K1uSF{ zi&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWRUiPt{103WKhdIJgj&Yn5 zoa7XzIm21bah?lYUG8z82R!5vk9opVp7ER)yyO+HdBa=Y z@tzNS~-sYydx(vhAFWF!-r$wF4Lk)0gm zBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}o zp9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=Q zP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_x zt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a# zT;vj$xx!Vhah)67Z7q7j`K#3UB6 zi9=lC5uXGkBoT>8LQ;~EoD`%a6{$%>TGEl83}hq|naM&{vXPw}F`or2WD$#5!cvy8oE5BO6{}gpTGp|i4Qyl+ zo7uuvwy~WZ>|__a*~4D;v7ZARP^DMC?-QJfN# zq!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO> zo(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2 zRHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQm zyV=8D_OYJ>9OMv(Il@tnahwyJBomp*LRPYoogCyO7rDtpUhrl%y1;DMMMxQJxA^ zq!N{>LRG3!of_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-;q!XR#LRY%c zogVb07rp62U;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X)of*tz7PFbd zT;?&K1uSF{i&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWRUiPt{103WK zhdIJgj&Yn5oa7XzIm21bah?lYUG8z82R!5vk9opVp7ER) zyyO+HdBa=Y@tzNS~-sYydx(vhAFWF!-r z$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJosp zq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQ zp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*E zQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A} zr#Zt}&T*a#T;vj$xx!Vhah)67Z7 zq7j`K#3UB6i9=lC5uXGkBoT>8LQ;~EoD`%a6{$%>TGEl83}hq|naM&{vXPw}F`or2WD$#5!cvy8oE5BO6{}gp zTGp|i4Qyl+o7uuvwy~WZ>|__a*~4D;v7ZARP^ zDMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~v zq!q1cLtEO>o(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^K zo(W835|f$2RHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww> zR<^O79qeQmyV=8D_OYJ>9OMv(Il@tnahwyJBomp*LRPYoogCyO7rDtpUhrl%y1; zDMMMxQJxA^q!N{>LRG3!of_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-; zq!XR#LRY%cogVb07rp62U;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X) zof*tz7PFbdT;?&K1uSF{i&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWR zUiPt{103WKhdIJgj&Yn5oa7XzIm21bah?lYUG8z82R!5v zk9opVp7ER)yyO+HdBa=Y@tzNSg5|8*KAR&oJOcIikjO3&sC8HNA zm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p8`{#2_H>{lo#;##y3&pA z^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J37{)S=@l0SMlbFmDrZSD`%wQ(7n9UsK zGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M z%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+xXm5za*z8w;31EA%oCpSjOV=I zC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w@{a%q{U;!S2uu)y5{%%4AS9s(O&G!w zj_^bvB9Vwp6rvK1=)@oeQen zwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd7 z3}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZ zvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkGj`w`v zBcJ%p7rye1@BH8=zxd4`{_>9ihx{iXfe1_xf)b42gdilL2u&Em5{~dhAR>{7OcbIL zjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKG2jl8*FbAS0Q`Oct_|jqKzgC%MQ? z9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRepdyv1Ockn9jq22(Cbg(d9qLk#`ZS;+ zjc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+jqdcIC%x!RANtad{tRFsgBZ*ZhBA!d zj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$9Og2Q`7B@|i&)GOma>fHtY9UpSj`&N zvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9SoZuv;3J>-%oo1$jqm*6 zC%^d3AO7->0Ehi2Ab|)>5P}kn;DjI~p$JVF!V-?~L?9xOh)fis5{>A@ASSVhO&sD9 zkN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3eBdLW_{g5|8*KAR&oJOcIik zjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p8`{#2_H>{l zo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J37{)S=@l0SMlbFmDrZSD` z%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w>|iIm*v%gH zvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+xXm5za*z8w;31EA z%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w@{a&V{U;!S2uu)y5{%%4 zAS9s(O&G!wj_^bvB9Vwp6rvK1=)@oeQenwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1 zz35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@ zEMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk z%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>9i$NVQCfe1_xf)b42gdilL2u&Em5{~dh zAR>{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKG2jl8*FbAS0Q`Oct_| zjqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRepdyv1Ockn9jq22(Cbg(d z9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+jqdcIC%x!RANtad{tRFs zgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$9Og2Q`7B@|i&)GOma>fH ztY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9SoZuv;3J>- z%oo1$jqm*6C%^d3AO7->0LT3&Ab|)>5P}kn;DjI~p$JVF!V-?~L?9xOh)fis5{>A@ zASSVhO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3eBdLW_{g5|8*K zAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p z8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J37{)S=@l0SM zlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w z>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+xXm5z za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w@{a%~{U;!S z2uu)y5{%%4AS9s(O&G!wj_^bvB9Vwp6rvK1=)@oeQenwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13> z7rN4o?)0E1z35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jz zvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-? z@{H%a;3cnk%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>9ir~D@%fe1_xf)b42gdilL z2u&Em5{~dhAR>{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}!NKG2jl8*Fb zAS0Q`Oct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRepdyv1Ockn9 zjq22(Cbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+jqdcIC%x!R zANtad{tRFsgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$9Og2Q`7B@| zi&)GOma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9S zoZuv;3J>-%oo1$jqm*6C%^d3AO7->0H^&YAb|)>5P}kn;DjI~p$JVF!V-?~L?9xO zh)fis5{>A@ASSVhO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3eBdLW_{g5|8*KAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>Y zjOMhUC9P;p8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J3 z7{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1x zo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj* z+~6j+xXm5za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w z@{a&#{U;!S2uu)y5{%%4AS9s(O&G!wj_^bvB9Vwp6rvK1=)@oeQenwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+M zj`nn*Bc13>7rN4o?)0E1z35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR z6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX& zJm4XZc+3-?@{H%a;3cnk%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>9i=lmxife1_x zf)b42gdilL2u&Em5{~dhAR>{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQml9G(%q#z}! zNKG2jl8*FbAS0Q`Oct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~C`}p4QjYRe zpdyv1Ockn9jq22(Cbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Ypd+2=Oc%P+ zjqdcIC%x!RANtad{tRFsgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_Fjp@u_CbO8$ z9Og2Q`7B@|i&)GOma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#R zhd9g;j&h9SoZuv;3J>-%oo1$jqm*6C%^d3AO7->0O$QDAb|)>5P}kn;DjI~p$JVF z!V-?~L?9xOh)fis5{>A@ASSVhO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0uXxQH-tvz3 zeBdLW_{g5|8*KAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvH zpdpQDOcR>YjOMhUC9P;p8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$ zjNy!6B%>J37{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D? z8rHIo^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH} zm$=Lou5yj*+~6j+xXm5za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl z{NN|Q_{|^w@{a%){U;!S2uu)y5{%%4AS9s(O&G!wj_^bvB9Vwp6rvK1=)@oeQenwWv)U>QayTG@v1kXiO8D(v0S` zpe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd73}7IG7|alcGK}GjU?ig$%^1cq zj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZb zx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkGj`w`vBcJ%p7rye1@BH8=zxd4`{_>9i zm;5Infe1_xf)b42gdilL2u&Em5{~dhAR>{7OcbILjp)Q6Cb5W39O4p>_#_}9iAYQm zl9G(%q#z}!NKG2jl8*FbAS0Q`Oct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z?l%OP~ zC`}p4QjYRepdyv1Ockn9jq22(Cbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1(vJ3Y zpd+2=Oc%P+jqdcIC%x!RANtad{tRFsgBZ*ZhBA!dj9?_A7|j^QGLG>~U?P*4%oL_F zjp@u_CbO8$9Og2Q`7B@|i&)GOma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q z9`>@2{T$#Rhd9g;j&h9SoZuv;3J>-%oo1$jqm*6C%^d3AO7->0GItIAb|)>5P}kn z;DjI~p$JVF!V-?~L?9xOh)fis5{>A@ASSVhO&sD9kN6}YA&E##5|WaP>6Q1&n=e*!0 zuXxQH-tvz3eBdLW_{g5|8*KAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{ zs7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWO zU?77S%n*h$jNy!6B%>J37{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3Ke zjODCgC97D?8rHIo^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd z8P0N!^IYH}m$=Lou5yj*+~6j+xXm5za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?Z zpZLrdzVeOl{NN|Q_{|^w@{a&l{U;!S2uu)y5{%%4AS9s(O&G!wj_^bvB9Vwp6rvK1 z=)@oeQenwWv)U>QayTG@v1k zXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ`qGd73}7IG7|alcGK}Gj zU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsvSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm7zZ=^+6O0{{S6SKGF2+qP}nwr$(CZQHhOyUQU@$U`3S zm?u2t8P9paOJ4DsH@xK?@A<$-KJl3^eB~S8`N2-nMQr5Vj>K}%ZEnl`kh9qs8rM>^4&E_9_E-RVJ3 zdeNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3cW-*&N%w-<) zS-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w({N*13uKQ0w0uh)X1SJ^32|-9g5t=ZBB^=?2 zKtv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_ z8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_ zI@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO z1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhfTJ9`Q*)LK2afBqSvn$w@&^$t zANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5`9`$KJLmJVT zCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|GAq-_0!x_Oy zMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4t>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8of zKtd9cm?R`68OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV z7{w_;NlH=yOIp#I zHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU z*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5b z-nMQr5Vj>K}%ZEnl`kh9qs8rM>^4& zE_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$trDNJP=)0x3c zW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZRcCnj1>}4PO zIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqkdB8&+@t7w( z{N*13Zu?I_0uh)X1SJ^32|-9g z5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L> zKt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ z8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPat zKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZh zfTJ9`Q*)LK2afBqSvn$w@&^$tANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4QJE@Kr5e?#K}~8=n>y5` z9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMpK~H+on?CfVAN?7?Kn5|G zAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg63&!Axc`n>oy79`jkiLKd-@B`jqb%UQun zR$y!A)*)n>*a)9`|{`Lmu&%Cp_gD&w0U1Uh$eYyyYG5`M^g$@tH4t z>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a z5t}%~B_8ofKtd9cm?R`68OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`I zKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5 zF^pv#;I&HLPVF>)F6Y zHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?q zxxr0tahp5b-nMQr5Vj>K}%ZEnl`kh z9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5L?$tr zDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q+u6ZR zcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC+~pqk zdB8&+@t7w({N*13?)y(b0uh)X z1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|d zk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{ zKt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt z8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneB zdBICw@tQZhh{PlzDalAq3R04a)TALT=}1oo zGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+br zs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~o zp)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>* zh{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^ z?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP@X&t(5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN z$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~ za*~VODP6JlYEp~Z)S)i* zs80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$p zVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qt ziq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S z1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4 z-}ufCe)5ao{NXSE2=K^%0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fR zVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQK ziOp`P7Goai`c{=F7b#@0uqvl z#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe* zN>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8S zXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAd zVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3|Rh{PlzDalAq3R04a z)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV z=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;F zVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$ z3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP@YH_-5{SSAAt=EJP6$F0iqM21 zEa3=G1R@fN$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~ z%w!=e*~m@~a*~VODP6Jl zYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P z=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUej zVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL( ziqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G z2R`zN&wSx4-}ufCe)5ao{NXSE2=L5*0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn4 z7|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHL zVJ+)e&jvQKiOp`P7Goai`c{= zF7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x z!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;w zTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8Mly=gjA1O} z7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3|Rh{Plz zDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m z(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2 zI?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1 zn9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLv zh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP@X~(*5{SSAAt=EJ zP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7P6|?ziqxbb zE$K*41~QU~%w!=e*~m@~a*~VODP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP;&UB$G-RMpa zdeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ-W-^P}%waC` zn9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm z;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg;VI8}&I?}h zir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2=K~(0uqS81R*HF2u=t>5{l4-AuQnt zPXrvz-t?g_{pimC z1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0j zSk4MovWnHLVJ+)e&jvQKiOp` zP7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL)P7ZRCi`?WP zFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwoF7>ES0~*qZ z#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM!3<$2!x+v8 zMly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3|Rh{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1 zD8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O z*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38b zCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC z;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP@Ya6< z5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk#33&6h))6%l8D44At}j7 zP6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VODP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^EDE$wJe2RhP; z&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x$xLA?)0oZ- zW-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(ycCw4z>|rna z*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?wxX%L~@`%Sg z;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2=LB-0uqS81R*HF2u=t> z5{l4-AuQntPXrvz z-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm+00=s^O(;9 z7P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL) zP7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwo zF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM z!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3| zRh{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJ zPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iOb zDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB z(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQ zHnNG$Y+)*>T;VF$ zxXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6 z;V=IP@X>z)5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk#33&6h))6% zl8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VOD zP6JlYEp~Z)S)i*s80hL(ul@1p()L1P77Mniq^ED zE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW=EaMo@1ST?x z$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z&1_*S+t|(y zcCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9NZgPv;+~F?w zxX%L~@`%Sg;VI8}&I?}hir2j1E$?{G2R`zN&wSx4-}ufCe)5ao{NXSE2=K{&0uqS8 z1R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq&jcniiOEc1D$|(G3}!Nm z+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOp`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbb zl8MY@AuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p= zP7P{Oi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91 zFa7Ax00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57} z#Vlbd%UI3|Rh{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J& zl8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2P zPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hD zD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES&jJ>*h{Y^nDa%;S3Rbd; z)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy*Sz5^?|9D#KJtmreBmqK z_|6Z0@{8a6;V=IP@YR0;5{SSAAt=EJP6$F0iqM21Ea3=G1R@fN$V4G3(TGkAViJqk z#33&6h))6%l8D44At}j7P6|?ziqxbbE$K*41~QU~%w!=e*~m@~a*~VODP6JlYEp~Z)S)i*s80hL(ul@1p()L1 zP77Mniq^EDE$wJe2RhP;&UB$G-RMpadeV#D^r0{P=+6KKGKj$pVJO2G&Im>_iqVW= zEaMo@1ST?x$xLA?)0oZ-W-^P}%waC`n9l+hvWUejVJXX4&I(qtiq))PE$dj%1~#&Z z&1_*S+t|(ycCw4z>|rna*v|nDa)`qm;V8#A&IwL(iqo9oEay1S1uk-l%Ut0q*SO9N zZgPv;+~F?wxc@%~_n-vGmP7$?-q^Nn+qP}nwr$(CZQHhO+p`%P9oc_y;;O26z(XGK zm?u2t8P9paOJ4DsH@xK?@A<$-KJl3^eB~S8`N23)1u02IYSNIF zbfhN(8OcOuvXGT*WG4qX$whARke7VqrvL>hL}7|hlwuU81SKg&Y06NRa+Ie66{$pJ zs!)|`RHp_tsYPw-P?vhtrvVLVL}QxJlx8%i1ubbsYueD3cC@Dh9qB}8y3mzwbf*VB z=|yk)(3gJnX8;2k#9)Rnlwk~K1S1*6XvQ#>ag1jI6Pd(hrZAOhOlJl&nZ<18Fqe7E zX8{XY#A24Plw~Yu1uI#_YSyrpb*yIt8`;EWwy>3LY-a~M*~M=5u$O)8=Ku#e#9@wb zlw%y{1SdJgY0hw#bDZY_7rDe`u5guWT;~Qixy5bnaF=`B=K&9S#ABZDlxIBW1uuEU zYu@mdcf98VANj;*zVMZAeCG#0`NePk@RxrC_%8nhBoKiKLQsMcoDhU06rl-2Si%vW z2t*_jk%>Z7q7j`K#3UB6i9=lC5uXGkBoT>8LQ;~EoD`%a6{$%>TGEl83}hq|naM&{ zvXPw}F`or2WD$#5!cvy8 zoE5BO6{}gpTGp|i4Qyl+o7uuvwy~WZ>|__a*~4D;v7ZAR@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zzn zrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p( zSGmS@Zg7)Z+~y8-nMQr5Vj>K}%ZE znl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5 zL?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q z+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC z+~pqkdB8&+@t7w({N*13e#t)p z2}EFm5R_m9Cj=o0MQFkhmT-h80uhNsWTFt2XhbIlF^NTN;t-d3#3um>Nkn3jkd$O3 zCj}`YE-8NHK|2y>QI+@)TaRrX+&e1(3EC0rv)u(MQhs7mUgtK10Cr^ zXS&dpZgi&yJ?TYn`p}nt^k)DA8N^_QFqB~oX9Ob|#c0MbmT`<{0u!0UWTr5cX-sDZ zGnvI~<}jCe%x3`$S;S(Nu#{yiX9X)+#cI~DmUXOW0~^`IX11`EZER-;JK4o<_OO?I z?B@UnImBU(aFk;l=L9D?#c9rPmUEov0vEZ&Wv+0QYh33BH@U@a?r@iT+~)xgdBkI$ z@RVmf=LIi$#cSU1mUq1810VUsXTI>2Z+zzmKl#OP{_vN71o$og1SAlF2|`eU5u6Z& zBov_uLs-HQo(M!F5|N2QRH6}`7{nwNv57-m;t`(&BqR}uNkUSRk(?ByBo(PiLt4_2 zo(yCp6Pd|ERP^DMC?-QJfN#q!gtoLs`mEo(fc?5|yb! zRjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@WoSGv)i9`vLa zz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rBvzfzO<}sfI zEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ>9OMv(Il@tn zahwyJ25Ry=YCJbQ-M|dI- zkw`=)3Q>thbYc*bSi~j{afwHK5|EHYBqj+-Nk(!~kdjoSCJkvxM|v`lkxXPJ3t7oV zc5;xDT;wJXdC5n93Q&+j6s8D8DMoQhP?A!VrVM2%M|mnxkxEpi3RS5_b!t$PTGXZv zb*V>v8qknNG^PnnX-0Ee(2`cPrVVXrM|(QZkxq1`3tj0(cY4s1Ui799ed$Mk1~8C8 z3}y&J8OCr%Fp^P>W(;E)$9N_%kx5Ku3R9WJbY?J-EM^HyS;lf! zu##1*W({ju$9gufkxgu73tQR7c6P9nUF>ELd)dc+4seh|9Oei|ImU5LaFSD;<_u>! z$9XPrkxN|W3Rk(tb#8EzTioUjce%%X9`KMyJmv{cdB$^I@RC=&<_&Lo$9q2TkxzW) z3t#!hcYg4bU;O3|fB8p%zw%E&0uh)X1SJ^32|-9g5t=ZBB^=?2Ktv)DnJ7dh8qtYC zOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L>Kt?i=nJi=_8`;T0PI8f(Jme)G z`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e^=Uvu8qt_0 zG^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZh+= z(3WeG#AU83dBtnq@RoPH=K~-4#Am+nm2Z6K2S546Z~pL?e*_2+Fo5*` z{|n?Fm>>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`6 z8OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>A zI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^Y zWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bh{PlzDalAq3R04a)TALT z=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV=uQuM z(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;FVJ`ES z&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$>M&jTLvh{rtPDbIM$3tsYy z*Sz5^?|9D#KJtmreBmqK_|6Z0@{8a6;V=IP5K#UJNFV|egrEc?I3Wm0C_)p4u!JK# z5r{}6A`^wEL?b#eh)FDB6Nk9OBR&a8NFoxGgrp=RIVngTwNFfSSgrXFqI3*}aDN0j@vXrAd6{tuhDpQ53RHHgIs7Wnq zQ-`|LqdpC2NFy54gr+p3IW1^OD_YZrwzQ)?9q33WI@5)&bfY^x=t(bn(}%wFqdx-} z$RGwYgrN*$I3pOzC`L1ev5aFp6PU;(CNqVpOk+ATn8_?=Gl#j%V?GO5$RZZAgrzKF zIV)JnDps?GwX9=38`#JuHnWATY-2k+*vT$-vxmLxV?PHt$RQ4MgrgkeI43yCDNb{S zvz+5R7r4kJE^~#eT;n=7xXCSUbBDX!<30~~$Ri%}gr_{?IWKt0D_--4x4h#$ANa^8 zKJ$gIeB(Pm_{lGR^M}9uL+AtiM?eA*m>>it7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0 zi9t+a5t}%~B_8ofKtd9cm?R`68OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^ zB_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLC zNJcT5F^pv#;I&HLPVF z>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edim?IqJ7{@umNltN^Go0ld=efW|E^(PF zT;&?qxxr0tahp5bQjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5 zMJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bK zw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}g zFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;8 z4tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bZG&3=IY z2uL6T6NI1yBRC-lNhm@ShOmSqJQ0XUBq9@qs6-<=F^EYlViSkB#3MclNJt_QlZ2!s zBRMHZNh(s4hP0$3JsHSICNh(StYjlQImk&aa+8O=lxi$tXrMhOvxeJQJA6BqlS3sZ3)! zGnmONW;2Jm%ws+aSjZw4vxKEAV>v5W$tqT}hPA9?Jsa4_CN{H$t!!gEJJ`uCcC&}Q z>|;L%ILILmbA+QD<2WZc$tg~AhO?aGJQujgB`$M?t6bwcH@L|yZgYpb+~YnEc*r9j z^Mt27<2f&Q$tzy-hPS-qJsKlsTne)EUF{KG*%z<&fJ5P=CoP=XPh z5QHQYp$S7+!V#VbL?jZCi9%GO5uF&sBo?uWLtNq!p9CZ%5s67cQj(FJ6r>~-sYydx z(vhAFWF!-r$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD> zsX|q%QJospq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUj zq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Su zp9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO! zQI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)67P^DMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~ zwW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@WoSGv)i9`vLaz3D?=`q7^O z3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rBvzfzO<}sfIEMyUjS;A75 zv78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ>9OMv(Il@tnahwyJ~-sYydx(vhAFWF!-r$wF4Lk)0gmBp12K zLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}op9VCf z5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_ z5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_xt69TZ z*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$ zxx!Vhah)67P^ zDMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~v zq!q1cLtEO>o(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^K zo(W835|f$2RHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww> zR<^O79qeQmyV=8D_OYJ>9OMv(Il@tnahwyJ~-sYydx(vhAFWF!-r$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGA zr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4 zbfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2 zF`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H z!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)67P^DMC?-QJfN#q!gtoLs`mEo(fc? z5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@WoSGv)i z9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rBvzfzO z<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ>9OMv( zIl@tnahwyJ~-sYydx(vhAFWF!-r z$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJosp zq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQ zp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*E zQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A} zr#Zt}&T*a#T;vj$xx!Vhah)67P^DMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEt zG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8 zF`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf z!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ>9OMv(Il@tnahwyJ~-sYydx(vhAFWF!-r$wF4Lk)0gmBp12KLtgTcp8^!5 z5QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}op9VCf5shg=Q<~A7 z7PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLMqZz|k z#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIg z*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)67 zP^DMC?-QJfN# zq!gtoLs`mEo(fc?5|yb!RjN^)8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO> zo(^=R6P@WoSGv)i9`vLaz3D?=`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2 zRHiYV8O&rBvzfzO<}sfIEMyUjS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQm zyV=8D_OYJ>9OMv(Il@tnahwyJ{z<&fJ5P=Co zP=XPh5QHQYp$S7+!V#VbL?jZCi9%GO5uF&sBo?uWLtNq!p9CZ%5s67cQj(FJ6r>~- zsYydx(vhAFWF!-r$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#Q zRHPD>sX|q%QJospq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We! z(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT z!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^ z5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)67P^DMC?-QJfN#q!gtoLs`mEo(fc?5|yb!RjN^) z8q}l~wW&j0>QSEtG^7!YX+l$)(VP~vq!q1cLtEO>o(^=R6P@WoSGv)i9`vLaz3D?= z`q7^O3}g_48NyJ8F`N;MWE7(r!&t^Ko(W835|f$2RHiYV8O&rBvzfzO<}sfIEMyUj zS;A75v78mGWEHDf!&=s{o(*hd6Pww>R<^O79qeQmyV=8D_OYJ>9OMv(Il@tnahwyJ z~-sYydx(vhAFWF!-r$wF4Lk)0gm zBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}o zp9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=Q zP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_x zt69TZ*0G)qY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a# zT;vj$xx!Vhah)67lxi$tXrM zhOvxeJQJA6BqlS3sZ3)!GnmONW;2Jm%ws+aSjZw4vxKEAV>v5W$tqT}hPA9?Jsa4_ zCN{H$t!!gEJJ`uCcC&}Q>|;L%ILILmbA+QD<2WZc$tg~AhO?aGJQujgB`$M?t6bwc zH@L|yZgYpb+~YnEc*r9j^Mt27<2f&Q$tzy-hPS-qJsKlsTne)EUF z{KLSW0RIt?Km;ZTK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS`d=ik5 zL?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYyc zN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APh zX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2 zW(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7 zeID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bZGVO;_KBOrkY zOb~(+jNpVIB%ugR7{U^c@I)XYk%&wbq7seh#2_ZIh)o>g5|8*KAR&oJOcIikjO3&s zC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQDOcR>YjOMhUC9P;p8`{#2_H>{lo#;## zy3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6B%>J37{)S=@l0SMlbFmDrZSD`%wQ(7 zn9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7 z;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lou5yj*+~6j+xXm5za*z8w;31EA%oCpS zjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q_{|^w@(+Xh0{lln0uh)X1SJ^32|-9g z5t=ZBB^=?2Ktv)DnJ7dh8qtYCOkxq6IK(9$@ku~J5|NlBBqbTiNkK|dk(xB5B^~L> zKt?i=nJi=_8`;T0PI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ z8r7*mO=?k_I@F~e^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPat zKJ=v@{TaYO1~Hf+3}qO@8NoS|UJKW_S|6}ML62L$JAP8n-+qP}nwr$(C zZQHhO+qP|I>rhYV>&$%~@Q_D5<_S-E#&cfql2^Ru4R3kJdp_`yPkiPJU-`y&e(;lD z{N@jT`G=9t0RIt?Km;ZTK?z21LJ*QrgeDAO2}gJ$5RphkCJIrBMs#8jlUT$i4snS` zd=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5 zMJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bK zw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}g zFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;8 z4tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bZGf!+ZB z5s*LxCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW z3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjO zPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p z*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y8Qjn5Vq$Uk% zNk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1Vh zrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9 zlV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5u zd={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAv zyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bZGVg3OB5s*LxCI~?ZMsPw9l2C*u3}FdJ zcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9} z)TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K z3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y8Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E` z4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^ zMl_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?! zMlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~ z<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3 zlVAMi4}bZGL!JQt5s*LxCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}L ziAQ`AkdQ@0t zrU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%w zl2){)4Q**hdpgjOPIRUVUFk-5deDAZh zTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z z+~y8Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwg zl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH z(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8 z#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?T zM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bZGBfbFt5s*LxCI~?Z zMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tm zN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5 zdeDAZhTiM2TcCeFO>}C&p*~fkkaF9bB z<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y8Qjn5Vq$Uk%Nk@7zkdaJe zCJR}~Ms{+LlU(E`4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dX zlUmfK4t1$VeHze^Ml_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJ ze+Dp+K@4UHLm9?!Mlh05jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3s zOIgNpR)oEPH>V_ zoaPK?ImdY}aFI(~<_cH2#&vFRlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M` z@R3h^<_ll>#&>@3lVAMi4}bZG1Kt4t5s*LxCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(I zQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=tc`8tmN>ru_RjEdGYEY9})TRz~sYiVp z(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUVUFk-5deDAZhTiM2TcCeFO>}C&p*~fkkaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~ zOI+p(SGmS@Zg7)Z+~y8Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E`4|&N)ehN^K zLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^Ml_}gO=(7R zTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?!Mlh05jAjgD z8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNpR)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFR zlUv;84tKf7eID?TM?B^UPkF|3UhtAvyygvWdB=M`@R3h^<_ll>#&>@3lVAMi4}bYb zfQ0%dAb|)>5P}kn;DjI~p$JVF!V-?~L?9xOh)fis5{>A@ASSVhO&sD9kN6}YA&E## z5|WaP>6Q1&n=e*!0uXxQH-tvz3eBdLW_{>it z7{LiaNJ0^sFoY!>;fX**A`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdON>Y)U zG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLCNJcT5F^pv#;I&HLPVF>)F6YHnEv4Y-JnU*}+bBv70^YWgq)Fz(Edi zm?IqJ7{@umNltN^Go0ld=efW|E^(PFT;&?qxxr0tahp5bz{xGA}~P+N-%;Gf{=tFG+_u! zIKmTwh(sbXQHV-3q7#Fd#3D9vh)X=;lYoRIA~8uwN-~m@f|R5pHEBpoI?|JYjASA+ zS;$H@vXg_HI4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS z)S@Q6^rAO?=u1EP zGk}2%VlYD($}omAf{~13G-DXcIL0%9iA-WLQ<%y$rZa|!^2*vmflbAW>!;xI=z$}x^}f|H!$ zG-o)=InHx|i(KL|SGdYGu5*K%+~PKOxXV56^MHpu;xSKn$}^txf|tDFHE(#!JKpnw zk9^`YU--&5zVm~h{Ngu%_{%>6B+)+s2}EFm5R_m9Cj=o0MQFkhmT-h80uhNsWTFt2 zXhbIlF^NTN;t-d3#3um>Nkn3jkd$O3Cj}`YE-8NHK|2y>QI+@)TaRr zX+&e1(3EC0rv)u(MQhs7mUgtK10Cr^XS&dpZgi&yJ?TYn`p}nt^k)DA8N^_QFqB~o zX9Ob|#c0MbmT`<{0u!0UWTr5cX-sDZGnvI~<}jCe%x3`$S;S(Nu#{yiX9X)+#cI~D zmUXOW0~^`IX11`EZER-;JK4o<_OO?I?B@UnImBU(aFk;l=L9D?#c9rPmUEov0vEZ& zWv+0QYh33BH@U@a?r@iT+~)xgdBkI$@RVmf=LIi$#cSU1mUq1810VUsXTI>2Z+zzm zKl#OP{_vN71W2lX0uqS81R*HF2u=t>5{l4-AuQntPXrvz-t?g_{pimC1~Q1j3}Gn47|sYrGK$fRVJzbq z&jcniiOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKiOpBomp*LRPYoogCyO7rDtpUhrl%y1; zDMMMxQJxA^q!N{>LRG3!of_1n7PYBEUFuPv1~jA*jcGztn$esVw4@cSX+vAu(Vh-; zq!XR#LRY%cogVb07rp62U;5FX0SsgigBik5hB2HGjARs}8N*n{F`fxbWD=8^!c?X) zof*tz7PFbdT;?&K1uSF{i&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWR zUiPt{103WKhdIJgj&Yn5oa7XzIm21bah?lYUG8z82R!5v zk9opVp7ER)yyO+HdBa=Y@tzNShfil%qTqs7NI$ zQ-!KjqdGOHNiAwqhq~0GJ`HF{BO23$rZl5DEoezATGNKMw4*&8=tw6z(}k{dqdPt5 zNiTZShraZqKLZ%ZAO&aK$t-3whq=sS zJ_}gLA{MiRr7UAPD_F@YR>(8$u4%YhrR4$KLCL?#MRiAHo{5R+KMCJu3lM|={HkVGUV2}wyta#E0zRHP;iX-P+VGLVr>WF`w) z$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5Y(34*DrVoATM}Gz| zkUW_xyE&FaFbiy<_>qc$9*2~kVib`2~T;(b6)V0SG?v8Z+XXiKJbxG zeC7*Z`NnsC@RMKs<_~}QM}U<2Cm?|cOb~(+jNpVIB%ugR7{U^c@I)XYk%&wbq7seh z#2_ZIh)o>g5|8*KAR&oJOcIikjO3&sC8HNAm8eV=s#1;W)SxD{s7)Q}QjhvHpdpQD zOcR>YjOMhUC9P;p8`{#2_H>{lo#;##y3&pA^q?ob=uIE`(vSWOU?77S%n*h$jNy!6 zB%>J37{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo z^=x1xo7l`2wz7@w>|iIm*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH}m$=Lo zu5yj*+~6j+xXm5za*z8w;31EA%oCpSjOV=IC9inR8{YDc_k7?ZpZLrdzVeOl{NN|Q z_{|^w@{a(i^iMzn5ttwZB^bd8K}bRonlOYV9N~#TL?RKHC`2V1(TPD!ViB7-#3df_ zNkBppk(eYTB^k*{K}u4Qnlz*(9qGwHMlz9^EMz4c*~vjpa*>-nMQr5Vj>K}%ZE znl`kh9qs8rM>^4&E_9_E-RVJ3deNIc^ravD8NfgWF_<9?Wf;R5!AM3inlX%J9OIe5 zL?$trDNJP=)0x3cW-*&N%w-<)S-?UTv6v++Wf{v^!Ae%Knl-Ft9qZY^MmDjTEo@~Q z+u6ZRcCnj1>}4POIlw^fMJ{ofD_rFo*SWz>ZgHDC z+~pqkdB8&+@t7w({N*13QtO|9 z1R^j&2ud)56M~S0A~azLOE|(4frvyRGEs<1G@=uOn8YGBafnMi;*)@cBqA|MNJ=u2 zlY*3_A~k79OFGh%fsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF|qBLbF zOF7C@fr?b3GF7NbHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=rfsS;d zGhOIPH@eeEMhTBSjsY%vx1eZVl``6%R1JxfsJfpGh5ioHny{ao$O*ad)Ui9 z_H%%P9O5uXILa}ObApqc;xuPC%Q?<-fs0(?GFQ0DHLi1mo800yceu+v?(=|$JmN7= zc*--L^MaSW;x%u0%RAolfscIRGhg`1H@@?OpZwxCfB4Hk0;JJD0SQE4f)JEo1SbR` z2}Nka5SDO+Cjt?PL}a26m1smK1~G|6Y~m1?c*G|G2}wj^l8}^SBqs$aNkwYXkd}0$ zCj%MDL}s#(m26}u2RX?_Zt{?qeB`G91t~;ficpkd6sH6wDMe|@P?mC(rveqJL}jW_ zm1+=(3WeG#AU83dBtnq z@RoPH=K~-4#Am+nm2Z6K2S546Z~pL?e*{RYe*zMSzyu*E!3a(WLK2G5gdr^92u}ne z5{bw}Au7>`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@AuHL) zP7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{Oi`vwo zF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax00uIM z!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&J1QUi`mR!F7uer0v57}#Vlbd%UI3| zR~-sYydx(vhAFWF!-r$wF4Lk)0gmBp12KLtgTc zp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJospq!zWQLtW}op9VCf5shg= zQ<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQp8*VH5Q7=QP=+y_5sYLM zqZz|k#xb4=Ok@(1nZi`2F`XIAWEQiT!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)q zY-AIg*}_(~v7H_4WEZ>H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vh zah)67TwNFfSSgrXFq zI3*}aDN0j@vXrAd6{tuhDpQ53RHHgIs7WnqQ-`|LqdpC2NFy54gr+p3IW1^OD_YZr zwzQ)?9q33WI@5)&bfY^x=t(bn(}%wFqdx-}$RGwYgrN*$I3pOzC`L1ev5aFp6PU;( zCNqVpOk+ATn8_?=Gl#j%V?GO5$RZZAgrzKFIV)JnDps?GwX9=38`#JuHnWATY-2k+ z*vT$-vxmLxV?PHt$RQ4MgrgkeI43yCDNb{Svz+5R7r4kJE^~#eT;n=7xXCSUbBDX! z<30~~$Ri%}gr_{?IWKt0D_--4x4h#$ANa^8KJ$gIeB(Pm_{lGR^M}9uBR~fI6Ocdz zCI~?ZMsPw9l2C*u3}FdJcp?yyNJJ(IQHe%$Vi1#9#3l}LiAQ`AkdQ@0trU*qTMsZ3|l2VkW3}q=t zc`8tmN>ru_RjEdGYEY9})TRz~sYiVp(2zznrU^}HMsr%wl2){)4Q**hdpgjOPIRUV zUFk-5deDAZhTiM2TcCeFO>}C&p*~fkk zaF9bB<_JeQ#&J$?l2e@K3}-pVc`k5~OI+p(SGmS@Zg7)Z+~y8eQenwWv)U>QayTG@v1kXiO8D(v0S`pe3znO&i+Mj`nn*Bc13>7rN4o?)0E1z35FJ z`qGd73}7IG7|alcGK}GjU?ig$%^1cqj`2)jB9oZR6s9tb>C9jzvzW~s<}#1@EMOsv zSj-ZZvW(@dU?r)hZbx46w6?sAX&Jm4XZc+3-?@{H%a;3cnk%^TkG zj`w`vBcJ%p7rye1@BH8=zxd4`{_>9ineKt?i=nJi=_8`;T0 zPI8f(Jme)G`6)m_3Q?FM6r~u&DM3j}QJON8r5xp{Kt(E1nJQGJ8r7*mO=?k_I@F~e z^=Uvu8qt_0G^H8MX+cX`(V8~2r5)|*Ku0>!nJ#pt8{O$aPkPatKJ=v@{TaYO1~Hf+ z3}qO@8NoS|UJKW_S_j$lW9`TqbJmneBdBICw@tQZhlYxw6A~RXYN;a~SgPi0dH+jfQKJrt5 zf)t`KMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCp zn$v=ow4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn z7|S@uGl7XrVlq>h$~2}kgPF`?HglNEJm#~2g)Cw*OIXS>ma~GDtYS55Sj#%rvw@9l zVl!LV$~LyMgPrVRH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTY zgPYvqHg~woJ?`^>hdkmjPk72Rp7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-Grc zKLTXYKLH6uV1f{oU<4-wAqhoj!Vs2lgeL+Ki9}?g5S3^|Ck8QzMQq{_mw3b{0SQS& zVv>-QWF#jADM>|Y(vX&Pq$dLz$wX$dkd00k*TVTw?cViczY zB`HN|%21Yal&1m}sYGR}P?c&_rv^2tMQ!R(mwMEv0S#$HW17&EW;CY-Eont-+R&DE zw5J0d=|pF`(3Ngq#cl3zmwVjj z0S|e^W1jGoXFTTxFL}jl-td-pyypWS`NU_w@Re_T=LbLe#c%%bmwyDvs(%6!h`h{PlzDalAq3R04a z)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2h{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k33tG~O*0iB5?PyO2I?{>GbfGKV z=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg%;~38bCNhc1Okpb1n9dAlGK<;F zVJ`ES&jJ>*h{Y^nDa%;S3Rbd;)vRGH>sZeQHnNG$Y+)*>T;VF$xXul3a*NyC;V$?7XSkjM?Mk*NfaWFX zj&0kvZQHhO+h)hMZQHhSlib+W8z1BIoZ3}u&9!UufQLNdF#$Z`DbEPxIWKt0D_--4 zx4h#$ANa^8KJ$gIeB(Pm_{lGR^M}6#$szv)B^bf^hkpq{NJ0^sFoY!>;fX**A`zJ= zL?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdON>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wX zk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h z7{eLCNJcT5F^pv#H!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a# zT;vj$xx!Vhah)67h{PlzDalAq3R04a)TALT=}1ooGLnhRWFafr$W9J&l8fBrAusvJPXP*2 zh{6=1D8(pF2})9m(v+brs7?)PQj6Nup)U2PPXij#h{iObDa~k3 z3tG~O*0iB5?PyO2I?{>GbfGKV=uQuM(u>~op)dXD&j1E8h`|hDD8m@e2u3oB(Trg% z;~38bCNhc1Okpb1n9hI9U?#Je%^c=3kNGTMA&Xed5|*-z<*Z;Ot60q%*0PTEY+xgs z*vuBTvW@NRU?;oS%^vo$kNq6rAcr{25sq?<CL?#MRiAHo{5R+KMCJu3lM|={HkVGUV z2}wyta#E0zRHP;iX-P+VGLVr>WF`w)$wqc^kds{GCJ%YZM}7)WkU|uu2t_GIaY|5< zQk13)WhqB_Do~M1RHh15sYZ2bP?K8JrVe$fM|~R5kVZ772~BB6b6U`nRY(34*DrVoATM}Gz|kUq#cl3zmwVjj z0S|e^V*+@>Q=SpXb6)V0SG?v8Z+XXiKJbxGeC7*Z`NnsC@RMKs<_~`fl3V@>N-%=+ z5C0N^kc1*MVF*h&!V`grL?SX#h)Oh~6N8wF-b^DGLn;ml%ygx zX-G>t(vyLVWFj+J$VxV{lY^Y(A~$)+OFr^bfPxgFFhwXzF^W@yl9Zw}WhhHI%2R=g zRH8Cfs7f`eQ-hk+qBeD?OFin-fQB@pF->SnGn&(amb9WZZD>n7+S7rKbfPm|=t?)b z(}SM$qBni$OF#NEfPoBRFhdy1ForXNk&I$AV;IXg#xsG5Oky%qn94M!^B*&q$t-3w zhq=sSJ_}gLA{MiRr7UAPD_F@YR>(8$u4%YhrR4$KL~-sYydx(vhAFWF!-r z$wF4Lk)0gmBp12KLtgTcp8^!55QQm1QHoKV5|pGAr71&M%2A#QRHPD>sX|q%QJosp zq!zWQLtW}op9VCf5shg=Q<~A77PO=lt!YDB+R>g4bfgoV=|We!(VZUjq!+#ELtpyQ zp8*VH5Q7=QP=+y_5sYLMqZz|k#xb4=Ok@(1nZi`2F`fUI!Axc`n>oy79`jkiLKd-@ zB`jqb%UQunR$y!A)*)n>*a)9`|{`Lmu&%0G{xaX9V(`7rf*ZuX)2; z-tnFfeB=|K`NCJe@tq(1A@ASSVhO&sD9kN6}YA&E##5|WaP-QWF#jADM>|Y(vX&Pq$dLz$wX$dkd00k*T zVTw?cViczYB`HN|%21Yal&1m}sYGR}P?c&_rv^2tMQ!R(mwMEv0S#$HW17&EW;CY- zEont-+R&DEw5J0d=|pF`(3Ng)oEPH>V_oaPK?ImdY}aFI(~<_cH2#&vFR zlUv;84tKf7eID?TM?5BgCp_gDfjs90FL}jl-td-pyypWS`NU_w@Re_T=LbLe#c%%b zmmvA&pP&RIIREf3AqYt*LKB9tgd;o=h)5(N6NRWmBRVmNNi1R$hq%NeJ_$%jA`+8? zq$DFbDM(2wQj>hfil%qTqs7NI$Q-!KjqdGOHNiAwqhq~0GJ`HF{BO23$rZl5DEoezATGNKMw4*&8 z=tw6z(}k{dqdPt5NiTZShraZqKLZ%ZAOma~GDtYS55Sj#%rvw@9lVl!LV$~LyMgPrVR zH+$I2KK65fgB;>8M>xtcj&p*OoZ>WRILkTCbAgLo;xbpb$~CTYgPYvqHg~woJ?`^> zhdkmj0X*R;&j{o>FL=o-Uh{^xyyHC|_{b+d^M$W`<2yh2$uEBMhra|VApZm<7{U36 ze+fZILJ^uUge4r|i9kdm5t%4NB^uF*K}=#1n>fTJ9`Q*)LK2afBqSvn$w@&^$tANeUjK?+frA{3<<#VJ8aN>Q3Jl%*WysX#?4 zQJE@Kr5e?#K}~8=n>y5`9`$KJLmJVTCN!lP&1pePTG5&|w51*G=|D$1(U~rEr5oMp zK~H+on?CfVAN?7?Kn5|GAq-_0!x_OyMlqT(jAb0-nZQIQF_|e$Wg64@j~UEl7PFbd zT;?&K1uSF{i&?@_ma&`_tYj6dS;Jb^v7QZVWD}d&!dAAiogM6C7rWWRUiPt{103WK zhdIJgj&Yn5oa7XzIm21bah?lYUG8z82R!5vj|t!jPkBZl z&w0U1Uh$eYyyYG5`M^g$@tH4t`P7Goai`c{=F7b#@0uqvl#3Ugp$w*ELQj&_)q#-ToNKXbbl8MY@ zAuHL)P7ZRCi`?WPFZsw%0SZ!x!W5w>#VAe*N>Yl_l%Xu;C{G0{Qi;k`p(@p=P7P{O zi`vwoF7>ES0~*qZ#x$WR&1g;wTGEQvw4p8SXio<^(uvM=p)1|!P7iw0i{A91Fa7Ax z00uIM!3<$2!x+v8Mly=gjA1O}7|#SIGKtAdVJg#@&VS5cCbO8$9Og2Q`7B@|i&)GO zma>fHtY9UpSj`&NvX1p^U?ZE@%oet?jqU7UC%f3q9`>@2{T$#Rhd9g;j&h9SoZuv< zIL#T(a*p#{;3Ai}%oVP3jqBXtCbziF9qw|E`#j(wk9bS~Pk72R0(s60Uh<09yx}eH zc+Uqu@`=xU;Va+x&JTX_i{Jd=FF^{)KS2pbaQ@+6LJ*QrgeDAO2}gJ$5RphkCJIrB zMs#8jlUT$i4snS`d=ik5L?k8&Nl8X>Qjn5Vq$Uk%Nk@7zkdaJeCJR}~Ms{+LlU(E` z4|&N)ehN^KLKLP5MJYycN>Gwgl%@=2DMxuKP?1VhrV3T5Ms;dXlUmfK4t1$VeHze^ zMl_}gO=(7RTF{bKw5APhX-9iH(2-7brVCx^Mt6G9lV0?u4}IxJe+Dp+K@4UHLm9?! zMlh05jAjgD8OL}gFp)`2W(rf8#&rH;1~Zw(Z00bRdCX@43t7Zsmavp%EN2BPS;cDB zu$FbKX9FAA#Addzm2GTi2RqrtZuYR3eeCA|2RX!Hj&PJ?9OnclImKztaF%nN=K>eG z#AU8 z#&>@3lVAMi4}S?#SpEr0FoN?B{}O_bgd#Ly2unD^6M=|CA~I2kN;IMqgP6o3HgSkc zJmQmpgd`#{Nk~dEl9Pgzq#`wGNJ~1>lYxw6A~RXYN;a~SgPi0dH+jfQKJrt5f)t`K zMJP%!ic^A;l%h0cC`&oYQ-O+9qB2#eN;RregPPQ$Hg%{=J?hhdhBTrvO=wCpn$v=o zw4ya_XiGcV(}9k3qBC9SN;kUGgP!!FH+|?!Kl(F(fed0WLm0|1hBJbZjAArn7|S@u zGl7XrVlq>h$~30)A2XQAEM_x@xy)le3s}e^7PEw-V?7(#$R;+k zg{^F3J3H9PE_Snrz3gK@2RO(f4s(Q~9OF1AILRqabB42=<2)C*$R#dwg{xfSIybn< zEpBs%yWHbG4|vEU9uvS5p7M-9p7Vm2yy7))c*{H9^MQ|i;xk|P$~V6AgP;83H-Gp` zkRtL=P=XPhfB2UWgd`N92}4-I5uOM{BodK{LR6v=ofyO<7O{y#T;dU*1SBL8iAh3I zl98Mgq$CxoNkdxFk)8}>BomqGNnZvrks0jZDk+Nw4N{nrRHZuY>BS%>GKtAdVLJ0z z!+JKbjqU8=D91R#Y0hww0G{!gFZ|&z;fvWnafwetl8}^SBqt9=DMmTUQ-O-KrVZ`r zOfUK{gh|X}IV;)1PIj}0z3k%v$2iU{9`l5EeBvwL`9a9yYDyH6l8p3ZAS0Q`N;Zm7 zf|8V`4CUxZC%VytUi4-ZqZ!8pwy>35?B*bcxW#i`5Tt}2B^bf^hfste98pL{3R04a z)TAXnnaM&qD$$m9w5J0-=t(d7(3cs^VlE3<%x11~hkHEWF#$Z`Ghg__UxJkM%mgPS zVTn#$5|EtK%@N>Y)^RHHieX-qdpGM+iiV-@S! z&jF5bg43MgEa$nvMJ{oft6bv-H@U?f?(vYvJmD!X_(X`(<` z4)Rfq;*_Ecb*N8sI?|oq%wRh^*vT$-vxmLx;{b;^$|=rqk(<2XFA>U^sU#*TDM(2w z(vp$VG@=QI;ZG^7ztXhBQb z(vePdqc6jm#!A+)k!|eaAcr{2F-~)yi(KImfjs90?|9EAzVd^g{3b|w^ORtOARG~h zNnDbUlvJcAD>*1YAqrEJ;#8m}wWvcw8qt(yw4gIR=}SNQGms$+V=UuYz-{jGm}dm? zmhctil}JP;3Q>th3}Od=^`G@~_bXiGPGFpwb(XFBs)%5qk*p3Q7!8~ZuNZSL@p7rf*(ANa%rYDBqIgc z$w_%CQk(j;r89jP$a*%jgFD>i9`|{`BLaBBGXio?$Gh5ltF7|Vf!yMrlr#R0gu5q0k+~f|Ac+3+5dCnW&@_|qM zr#ZuUZgYo6Jm)nZ_{bN2@|O@*%>`1ChV*118+j-|aY|5@8q}gT4QWnSx-*F3OkpYu z*v&o;a*oSf<1If3TFne11*yqMF{;vriA-WL^I6S74snS`Jmx8ZyyO+H`OFu7@RMNG z)sBDomym=a3=xP#RH6}sIK(9($w*EL(vXR)l%fi?=|V3?FqMTYW(iAK$p&_^n*$u; z2*)|YMJ{unr@Y`L@A<$dKJ%9#HSC3G#3D9{$w6N7QIJBEpe*I6LRG3$mqs+D1HBo? z1ZK04)okJbXE@I_?(u*C-teAJeBn32Yw9mT5|;2pA_)a4M0ILXn>sY16&>ipa3(Q@ zsZ3)gv)IB8_H&dIoZ>!@3E&y8iCD|KgD6BJ7IDc$c8XGhl9Zwjb*V=~TF{P;^q>y| z8O~Vdu#AqeCH>>`9qL;W)1%m zl2Ak>5>bdsG-42!_#_}9iAh3IQjn4~q$2|v$wp3ck&i+YrU*qTMhQw%nlhB90#&F^ z4Qf)0I@F~RjcG=6TG5UUbfz0U=|vy<(Vs!AWdmC{#aXTsz!N_6m5}xAofyOadHQU+2KK65%BOK=xXSl#cE^&=Op7WC5{I8MsE1?NTcp?#l&309XhvH)(}kY&W-vn-##qKPoteyI32WKRRt|B5lbq)v&w0rkKJlIK zjm<@(6N5w~At@=zLq76TfWlOw3e9LiCpt5b>C9#YYuUjr4so65yy7(<`9{zta!5?# zkcrIXp)f@$K}pI{jT+RX5pC$ha7Htp$?W1hm$}YOo)E}8KJb~JglMW(Bp?SlDNT8* zP@Ot7rac|##26+po24vgJ)79VHg>az3tS_Bcl_cv5u2$2QHVi&5|fl<@;qEw~^wP;Qo+S85R^kD=e8N)axFr7v0VlVqRz%fp9 zi7VXXE`hw_EuZ*bOXopil97Q-w6z&TSfUV>l%ypC#i>Cp8qtfs z^k*njS;#szvYC@yC4grH@`eAku@2#gOls1Qks{Qf7DE`$Xby6OqXhDr&xC4gUJ{uo z#3UB6i9=lClYm4dCMn5CP6|?ynlz**0~yIg7P66@9ONPodC5ls3R0LN6r(sLDMe|@ zP?mC(r!rNkP7P{Pm-;lKF->Vp2RhP;&h(%U{pimC1~ZgljAk4Yn8(vq1Hl%Xs&s6#y((vm*(We`Id#z;mpnJG-=KV~zBMXX>g>)6O^ z9OD#cxx#&(@{BKhxJRGttX2LkL0Yk)RG|(HXiO7Y(w>2g zVFJ^b&VS5g9-G+AA&zjFE8OA^4|vEk0{KbsuD-1aOE@AEpM)eQC27b^R`OAjid3fo zO=(Um+R&a3bfhai=*cK1F_|eWU>WP!#%}g=h{K%YJeRmZ08e@CN=+Kmnx6EcFN2xDTo$sJO>E~d$2rLbu5p9g z+#!HxeBdh~y6Yh#5ubEqBomp*LRNB;o8pwGKK&TRL?$tf8O&im3)#gHj&qVzT;Lkl zxyyZ?@{H%a;RB(2=nWzfnJ7dfH5ti5R!UHp`ZT5qZD>bldeV>n3}P6=8O=l%u$ZN+ zVGDaX#9>ZwhRa;%1~<9KeV*~04}9hqe+k;t9*9bG5|NBdWF{X4DNIpHP=SinrvVLV zL}QxMhPJe)GXogHI3_ZgX{={6`#8cGE^?VWJmMW+_{J~(*ULGRgtVk116j#IUh+|# zlGLCc^=V00`Z0u|jAA--SixG>ahT(r;X1dt%^lwJi=e&bh){$lB9VzkEE1BLB2=U@ zHK|Qq>eGpy^kzIWna@%-bBe3nCxE8}@`f;dtV;yq5SN7HB_D+-PHD|pbE`sL3esEh`G#T8Ee_b zP7ZQ}laBP{Bo8GiLuIPbm?bP_C3`r)Meg#LS9~Vu z0RK%yVi1pHOI+m!H@U?<9`KS^yx}Kt2I+YclZ4FVpeQ9M zOF3#$pP8&=6Nfm*eO~a1@BH8wK?dt1Vi1=EBq0Tz1dNiXIZRtQ~y3vnujAsJVn9X_)a)=`w=PGabLC}%DM+ixHA`p>i#3C-q$U%N; zQkzD!p%Vj`!cQ?cu}MWbvXGNJl&2aEX+&$fGnDZxU@up= z$s?W-$P3=_iQuF41W|}aOcIcYlw>0h6{t>Q+R%kQ3}i6F7{@yHagbvi=R9`^;5Bdf zM94ALB`PUNMJ|d_g7Q?R8O>=)H~KJ);f!J|D_O@zcC(*zJmMK2`O0?^jMeL8BR}P7 zKtl$woYkyjJv-RL0SBC@#GL4z+;}lo9#T}mWo{t2XD6ga!nAASLO^K`!!AfP$2y5>=_iP{uHU87yKYJ2}Bk?r@K1yyP`+_`p{pPS#JP zAQwd`OEX%~j`s9t6jNBtKF;%um%QRV-v~BE4Jb!N>e7T}w5B~h>B|5HGn^5OVItF* z&vDLhl@EO6Ghg`1H+~ajs(vOY!T6Vugd#Ly2~R{K6P*~uBo?uWOFZI}kVGUVDalAd zN>Y=SbYvhSnaD!;xI=z%5hF`lGB{!JQumd6>e~oTioRV4|zlYPkGKuUh#%^eBdLW z_{L9u^M}6#nWldTMhHR?n(#y*B2kD=3}O+(vyLVWFb2_$wht&P>@0t zp(w>DK}kwcnsQX2B2}nLO=?kxx-_COO=wCpTGEQvw51&#=tw6z(}k{dqdPt5NiTZS zmwpUlFhd!^XvQ#>aZF?~Q<%yOW-^Pp%ws+aS;P{SvW(@dWEC6P%ocXCpJSZjG-o)^ z1uk)w8{FbHceu|39`cx{yy7kI`M@VW^MhXmnQnd%lwbtsUqTU{NJJ$X(TPPI;u4Se zBp@+KNJc8sl7Wn5CM(&=NiOn`m%S|UJKW;|j|ku?fjs9GuX)FNzVMUZ{NaE9sV%|zhtPx}9Fd4j z3}O?H_#`6*DM>{d(vpt!WF#|L$Vv`!k&pZor5MF2L0QUCp2}3E4h?BU6Q1yfpM;v} zj>=`i&eC(VV-j;%#wK3#hVZlXC^1MyCW=szDm0-vJsH77R&bE3yyu@ee#a!1u$J8% z;vClqsE;~swqI$xcL zNpcEMm1@+Z6@3`POct_&Rcv4r`#H>U9`J+zEzk?Zr4S{kNGrP1jR8z&E=yR;UaoP2 zcYNh95f?fOl2L?8RHYN$7{_#Gu#n}fX9K%9z!6S!h6~)}F86rG7lJHu#~}^r$xKf2 zlArR_pat#eNDoFZjycR{DQh^$NiGq{TfXp@e->Md2t+3i@fge)CbOGMT;&$GxzBU{ zw?xi}Lr$vEkftt)-5lTyXSu{<0(il{ zE98yn#3ltf$VEPiQkF_op*nTxOm7A;gh?!C3%fbPE$;D~m@D0}NkwWhk((lvq!N{> zLJiu|j-Cu*7!z2`5;n4(b6nsux4FY>J`#SFGbIg~$V@KsQIs-NrV34IPHToRg3(N8 zHglQBTDEhB4}2r&YWE2ek%HXhp(^!hNNYOMg|76V57U^%a@MnvogCyiC%DLEuJVDO z#8@L|#3LbzNkJ+yk(t7jr5yEXK`YwPjb037FvA(mST?bj{aoZW_jtf-z7TY+^+`r9 z%2A#=G^G^-7|v*>GM82C(SZ^m=y#Qj?a9WFr@OC`DPSQjJ>FrYGZ>%4`;~jXm7v9X|-M!5NXB zBDA0b-59`h7O;@ztm6QOIm&TPaFRPb<{5#!PSd)TcFV=|g{pGm&{LWI5~D%SG<h>9ONV~g(*f&>eG&n z^kO)(na={Yv6q7!NKP=y&1?TMl+8^Y~~P$Im$6EafL@b zCXly$AkH4|Hqw)m!jz>v9T>)Brn8t$9N-jZxylW`@$X*sBN~~=MnMWuk}6cC1`TLS z2YS+<;f!G{6Pd(x{$nny*v?_jae<3m<32BW%{%@QZJ%Bu3CYMoPHIt?dNiUbeHp?; zrm~rx>|#F$IKgG!@P$zO^$AHyNqVx8m!g!UG&QMD1NtzKnXF_r+t|fEj&OqWT;MWy zc*0k{5#oTnkccFtBo}!oPbI2Rof%|fBRr9aK^zj0gfyfj6S>Go zK?+fV3RI^)jcCpQ1~G;S%w#T$*~~tUbB=pFA&^(R<0lc0m{X*tHg%{+ABHoMu}tJY z=CYP8oZ~7__{~2@^%miYPZ5eyl5*6h9?j{?0ERGQ4sRHq)zXiGmvGl|KpU>Ey2#dRJMz|&Fth5CppDU?(&*n{O_!65`oA>Ar1*hOA+ zi{0$!00%k7aZVEYivA@&Dak=D{@;*21=wM20RWw#ZP3@)wr$(CZQE93+csyy#Z9ew4ob4>BRtsGLosxVI`|r!#Xw*$X@nwfWuti5;u9y z3;yF90awg2v4}%F5|EIjq#`@{C`?gG(vU{9VkjdS#aJdXgPHupPL6Yei`?a3p7EMD zd?UBA)EF`tD5ae+I$;x*s-!B0Y6_xU6yBU#8t0cui<+H_zf^H|7YHnD}R z?BxJQIK~xj@`NA!=LCoeKPH=`h+~*CS2zygckeEzlr8L!O zM;E#=jNy!98mkB-h&>$P7?-%rRUYz)r-Z)colPX75}P(# zC?+zMC9Gg2LF{A~d)d!1j&qW;+~gsT2;sNe-e$xhE-A=FUP@4gid3RCUFpw2Ml*%E z%wq{lS;lg9vY+Fe;x6}i!V7}=z;ActgE+(`5s67cQc{tIOk^f2C8$jkn$wxC3}q5? zSj&30u#JD%%@JF@>4TV?L|dL?A(I<`73Y&N;4egGU7O zl6Qo=Z$A==$iyKL$;eI)3R8qql%_ltsX{ZF(~j=+U@()J$^sU%obBvkFNZkKRc>&T z2RtO0_x#5Pei7<{y6`_@lA0Wppfu&FLw9=6lYWe07IT=(26l0b3%nqhSNtHxBk_=y0+gf*jcGzF+R~2B z^rRQP=|_LYGL8vMWg2r?%qk9Wk{jISU!L=V(2v!g$iyKoNy$eMN>Q1r)SwZKX+|r0 zGlV6q;UK3u&o%CHk5_!=3t^tfEq@Z7G~}i<<*7nds?ms6bfyaf8N^`zW)$O@!XEZ> zjGNry9bXCfSFD5~Dapx2KJrt6%2cNojc7?*y3&pAjAR@$*~DHBaELS9;6J_*>Z#r# zJsHSCRti#*Qq-g+9qC1HhBK9EEMo^(xy>E^rru3mN{TR*! zW;2I*EMXZd*gz0FILRr_aEE)m<2~OAc;Q|n1~G|4Qc{tcv=pWs6{tuhs#1r#)T1Mv z8ORJ~GoMAQVjDX+#tANSg}dD60q^;b4}2y{uzn{#3CKqQ%F~d}j9?;@n8_k`vxkE` zBdbFS=?deG``p}oj%w!3hxWavcdB=A`_~VV55Ss)fCmXq` zNiFI!lwk~K3e#A~V)k;BE8OM|k9a~bAISJt3>2aW<*7(Rx-yBGEMx_12x2qaxWGj& zafAE3;{!hk;rDlPMFP^2hrAS_3>BzDUE0u&K}=vStJum3PI8ee+~g6#eBdLW2=(4Q zL}U_@h?Ha^8x^QXP5RK6ehgzg6PU&f=CPPH9Ooq0xxq~y@`ShiAcWulGn>RB9_h(R zOg};bPA`+9G9F(I16{$@f>e7!1%x4Gx zu$#kN<_cf=M);5Vk7z_E7Rg9W8q$)DoD`ub#VJcgD$|e_bfPoe7{^++aFdt3;x%u1 z&qqG-nO}tZBvv93nb^c5E%_)#bsEx+z3gK@2RP0t&U2kdJmDk12>9#_LKA_ABqR}O zNJ}2_l8;i9p)&31Kv%jkgsCiI2}@Z{AVF+q563vpLmm;#OWyE{fG_%pKL|%Ol8}^i zq$ei@C`VzkNB4t1oM+$g#Rj6#2_Jw zNJUBN(2D*HVlcxP%QP0SkwErygo|9|KF@hU-fwbEY0A)oRA@q0kr6}d7K}{Obm?ku%B|Ygy9|kjt8O&xbOIXS>ma~ySg4oPn zPH>V_T;UoIdCXH@5X=|C{_wu#526s8%;cg7r6@-=YSNwl3}r40Sjid!31SxqILK+v zaF=@o^M$W`<0nx=^dIqvPXaQLox&8MB6Vm+TiVl$k&I&+vspqAdpXG|PIHf^yyPRl z|I}loARDD9O&Q8kjuy0|1N|Am1ST?-Svn$wfs^k*zHnav!wvy0vA z;S^W6$2&s!$uIsFDons%#3U7I$x05Y(v9v6VGfJf$R@UNkkg#wJeRn~3;yFLF+z)p zRHPv-IVehby3m!ujAb0-naeVkvx!~o=M-1CPY992gb9d398!~p^kk+CwP{9Yy3n0I z3}-Ckn8Xs+vw@B5=PcK`#cl5Ml6QP0Qds8`l_Vr31zE^W0SZ!vO4Ox3Eoem#1~Z-s zOk@G;SkGae@tRM3xaY)0oaORuIG%j&OymydapjyyF``NbpCP zfQ00s2qmdXHL6paR`h2eBN@dsmavLLT;@Ixc)}~*5+R%%5}z~_rUb1S&opMSip?D7 zA`f`TD?SiHK=?2Lp@>6VGLng06r%>M7{N#;Fqs7`WCa`9&0bD(mK(e$gufz$35ZTo zGLe~r6r~s?C`l!%(ST-jqBC6>&ItZyEYn%UGFG#Owd`acM>xSLu5**;eC7)=BZdix zMQl=&hKv-W9ObD@6PnVS{tRFOYgkJV2RY0c9`J}zf7&7BqAca9Nef!hhyDy;AcL8} z6c(|HjqG7BM>)+o?(>Z2ydap@eB(PIg#BNbfZqsD1fmg>eGORbfY^x=|yk)FpANPXFhA$%yy1) zpAUQ?AeOfTk%>Y=GLnhRWFZ@csZS%?(U*QqW+6fB;5aw9!(CqSnm4@V9lyo)rXV^o zi9L}rRoj9RpyC2eU(CwkGFehgqN zzY`OFt0BvwxA%& zfFx#z{}F|#Bqs%_$VgW5k)H}wqzR)K%^1cqj>RltIa}DyLC$i8>pb8!eRa}GX*G25h_!adbFlD!x+O%W-*^d1QNt%_OO?I9Oe?2xxzJ` z@|_?2BuomuN=y=yf{c`>8g*zwOIp#L9`vCv!x_O;Rr5yWOra-N5L=LgYKiIF(OB_k!NKt-z1m}WGm z8$IYrebqcKhBO&|I*fQd|EGE-T_KOE-- zSGmS*J`g6YbNP*k{7D)zP>OOirwwiC$Pk7yjG4@4A#2&q9xif;TLcq2oiiv$B`VX2 ziA-V&t69e;4se9y+~6jUdCNP#5H`IxI+2J%TGEk$jAWrGB`HH)>d}CPG@~P(=u8*- zFph~VVkdb$_ri*!cTq?C!^mgq$Ml)DMe|T(}I@trXRx?&Q#WMkV72i1SdJqeLnG- z@BAQypTy1NZxf^^0~yImZt78=1~jG#O=&?_deNJ4OlAtxS;u-du%E+R;3C(!&P!es zI_G`6xg^3Q?T;bfE{!Sk7A3v7X(W+_1aX3soZ&2wc*74u_&uxo5ud~)Aq`n5NFfSSof_1n7ES2D7`C#5i(Kah zuXxQH-tmcu+4L-tNkn>zP=b2YrvVLVL}Pl>p8*VH7*m+aG-fl0`7CDzdpXD<&T^ew z+~yUpdBX?NX4gODCJ%YZM`0>apH{S?Edv?CP$sdKL)_&a|M7u<9NwZtAR>Peg=8cr z1u02G0SZ!tPV`^|qnW@w=Cgv^+~E_S`NB`4=CsSnK~8d$hmw@00gV{U5T-Gm8O&xm z8`#KxZu5XId?!XO_XP(S{!MW-6;$!+9?8kxzW)Cz12( zabgmSA{3<&uQ-Qkf*%lBcJ$8;sSOUNl8u$ zGLVn_6reWsXiO7Y(2}0?q7Q?a$r{!YNDv1(#9_{Ho?E=&6|Z?in1X6ac#@NXViczY zWob+kn$ny8j9@C$n9UqE5y(!C^NQaJ+1-RAJQ0XYDpHeyjFh1)<)};*YSElFv}FWy zna4sFvy1&4;5K*o$R|GYg&2j^ns_86C;6yOEo#$__H>{Vo#{eXdNGwHEM*->dBQV3 z6S|085|-cio$w?hHR;GtVTw?clGLUF4QWJUTGE?=jAJ~LS;A75v4*t-vY$iT=0862 ziBLs-FQOBJn8YGBaVSeU%2R87}djk3=e=kBLH5Vi1#9B&Svic*Zyw5J|rklILa{|@h`9WLI^+k zMUryjq9BDROc9DwjH)!F4LzC66lSx7o$O&Rhq=WA9`c2+eB%cp{3J|yIUo-4Nkmq% zk%zqGBR_>GOi3Ejh{iObDWjOoY&NiwK!VuJP7ZRN8{Fg*pZP)vKZ#wz-X}f@NK6{C zlY`vkrz{nzL_6BknJ)azNJcT5@vLAgySd0EuJMpZJmwbx6`fCVQqr833}P@t7{h#) zu$GN%;wUFL$$4H7v65X!8j4bkrnICN{TRw9CNZCdY-JnU+07mfahM~V7PDE&TGp|i4Qyd2XSv8FE^~#u+~YN&s>%TgNJt{` zl8^iprU>a_y zz(RJims{NC4)?gvFH%;w?@3EK(vyMg{1q zg^$ZZwRRCI}n}-#HTnVDMb^S(VP~HU?ihh z!&-tk!&%O8o@?CZHD8ER&#oc~rKwH}TGEbgbZ0s_u$>+3WEZ=+#C6{Cj`w^e zetrExYD!R&R&=HdUFpLxCNPz0OlJmjSY)U zjAWuXWvN3Ky3&n4^ratz8N%O;WE8WR!vYqvjC~yD2(Jj+(BF0Wi}Yk5BU#8wMXFMb z>eQenwWv*RhBBR5%w`UAS;S(Nu$tXm<2pCE&3(QTsgc~1ki?`W3+1UmMJiF5>a?L9 z?dd>AI?<2*jAuGCn8`x6v6DSKy1i3k*-C@pA7JG#(^iA-V|yE)AnF7lOceCG!t z{3L2qziUW93JTDO#x$WR&1p+J#xa43OkytU*vMusa*gZU;ywTIfiHaJ8=;%YDJe)v zS_)HGjAk-(S;IE=aFuIZ=N8}iPC!e$ zfzU)J5lKl#a#E0zR1}~DC8g4bfGI_naCuzvV(uP$SrOY%uC)8(8|swF7Zh~ zDpHe%+!UoWWhhT)y3>P+Oky%KnZ;t3aGEomqivB^ws@{pG@G^7!Y zX-0Ee(2n7ZWE_iF%o0|ymW>?aG|vd;CI9h(k9^`Y3ERj$i787t%2R<_^q@ZjSjaLq zu#qDith z9`aIzrnI0Xt!T{%#xR!EtYIUY2xJRqImbn=@SLyw9`Kg;eB}rK>*)IvlUQUS7r7}&aVpY)=Cq(K?dZW6#xjoaOky6331S=D*}*=} zaGwV}<2fJrL8wl?3(<&864Fqbdeo-@Lm0|1hBJauOkpbP*}^3*bB#OP<2yeHAyj8I zB^=>NLJCrniX0TBB&}#o8+y`<-b`XLQ<=sLX0nMuwz7?#+~y81__K?536Y3Kbh43~ zy0oAr9qB}8rZaG=PV$kTqBNv2Eg8a4#xRy~jAsJV znaeU(vYj35W)FMW$9@iQl4m^U1uuEUe|#WLH~W`_BqBK}NlQ7(Q-O+9VkpBH&M3yQ zm?bP_8OvG0YSyrpb?o3S_qfjk9`c9?-R%kfBqbThN&$*doto65Hg#xB6IwEYzZuI; zcCnj19OE{3c*JM^=%LSuN=#xAn;hgM7rDtpUg}Vndeo-@qZrK?#xsG5%wQ(7*u_4M za*Xp_Aei_3NA#Y~CJqTnL}HSWoLbbT4)tk3Lt4^`o{VQQQ<%zZ7P5!EoF^ zjs`TLDXkg9SSB-%`7C2QJK4o<4sw$FJmv|({MpCdKop`9jXdNfAH^t62})9mseG!d31QrJq=dMj6Udjs|q48{O$cU;5FX0Ssgr%UQ=34se)F6YwsMItgdX5`EMfVL-}!@ZL?;H> zC`EmmF_O_79F_|gMVJ-_; z$Z`&HoC{p!5@80*F~9LU8OTT`GLwZ|6s0k(XiXd1(w(u4V=_~i$u@RzmSA2JbBH}e zY~oOq3RI*fwP;Ik`p}obEMyVOSk5}uvzhZ;;3Ai}#cl3zmpA-5RL>BDid3UIHK<8L zS~HB1jAArn*v58tu#4U7;SMi(#cSU2l`zBf42elaYBG?Kvecq20~y3%hB2HS{KHQ6 zv7cj{;Tg|)K``(6$~R&Tms8@9kR&7}2RSK9W!ln_PIRUR0~pUzma&{o1agoQoFs&w zgc{)-;u4SeBqR}uNkVcmkdf@uW`Vi1#9 z#3ljBNKOh;k(#0uqcjz$NFVyrj}a_oC97D?8Une-b#8Nq2Ye;sC_T>qh)FDRkeebD zr43V<%1mZ4n>oy7IV(8BSuS#e5PlMWv{=YRRccY2Iy9ms0~x^tCNhbC*vWoQaFX|Y zB*7THLn0EBoD`%a6`9FGP8!mbAq-^COz3GMsZ40 zfr@mY2R-RUZw4@sK@4X(D_F@I*7AgZ3FalQ2qD2ZF_MI&BqKQ~NK00-QJCt~pcb{M zLtUECoso=UG-DXcIL0%TS?pmy2RO?sUh{?z{NT6o`j+U#CJu3lM`DtYlvI?Z995}C zBN{V=p^Rh{yEw#g-tvwZ6Z9jA$W0|`P?JWqqBSEK#b_q6o{elGkRUd5fs0(?GBT~ic*YHbfgoV=}9m8(VyY0Vl`{n zz()4)h{rtV6Q4;mMcqk8a%xhK`ZS;=lbFVIW-y1j%ws+$IKx>U^Mrr-P6$8wMZi>d z2%!l>cp?yyKgmiq@==&_l&1pKs7@mq(}bq9q6=M_&LS4GgrzLw3Rk(tW1jGh7}NA9 zv4}@fl97(ml%WC@sYGRJQJXr{r5-I9#RR5vh@%|iIM)f`C%*`oZVrh+4)W8KW;CZQ zT^Y$DmavpntmYJ#xx!UG5_X2qBqDzjjhy5nHw7s~VM@`LehgwTLm0|1hBJ>qg4oO! zwz8W&9Ond2_?KY9&$LU3M|={Hlw_o&7WHUEV>&U3$;@CTvzW~s7O{#g9N-{_ILrku z@`Im*o~5UVKo0UyoD!6!C%x!Le+Dp+K@4UTqZz|SHW5e=+t|(l4swXIyx}eH_{o1Nhq)|bHETG;Wv+0Q+uY$U_qfknV$RXSB%m@isYP9y z@HZnF#RMiYiODQvA18UkTR!lSPlTE)w}c@qQOQIb7PFc&oaG$nxx!Vhahp5bCC)tk zK>`wzh{U8JBblg94Qf)0+BBpQjcGzF+B1g9Okpb1Sj0Z|bBMzn;V8#A&INArkxzUl z!hClkF(^V&N>Q3Jl%*W~=+6L#GK}Hu=Ku#e#3@d5hIhQ@J3mOYK-?rDDalAdN>Y)U zG^8a5mFY|ux-yiptY-r|*~NYiaE7NmBbb-G;x%9RN|J@nCKaj4Kt}3PkH)m3HNBX^ z3}!Nm+00=vOIXbs&T)y$+~W5|e!CNn@I)XIu}Mr0a*~VOqiOEc1Dzli)VwSL!LmcJ^XF11tE^v`cT;>iDm$_?*L}a26m1ra;2}wytUFy+* zhRk3lvzW~y7PEw7|k?-*vvodWEVgAMZg-n zf;<$WFoPJ(cJ{HK103Wer?|j7-t!+n2qD&5H6|sg$VNk&(}K3NqX+X@z*5$;fxA59 zC9n9(HxjKAFG)yBMlz9^rnIFU?ODwl*0PSRY-2k+_=orWN2v8ai`?WPFNG*UD_YZ@ z9`s=ofdsLc9lYTqp9tY6NjJy?$;m)1YSVy*G@>#o)4R86#CqDCy z@BASACi6`$^3#Ft^q?R8Sxq28Y-S64*~f8C@QZ*zbs{_wh)6DSlZU+2q7Qu;!cayt zhJEbk1SdJgX)bY@D_rFo5rgbu{>NX$ATBw`PkE|Qox0Sc2~BB6bNVrdMJ#3sOIgNE zZgH1;+~+aDyd>IYdzcu+AudVCPE%UZnk6h{E$dj%1~zhnle{39m%Qa2@A*!IEzTe# znaE6ba*&f;Y(}%wFqd!9!!QZTA4Vwrgh|O$aC&&1~Pks@wRc#4P7$Q)B zf|R8k)#=ADhBJa$oZ%vuxXew$ZPO=2AQF*@LTpM>iqe#!ELEvSUFuPvX-sD!i&)Gi zc5sgC+~Yp4c+DHWl5)GfM{3fLm26}u2l*&T3tG~N&TL>KTiD7rwsU}koZ&2Y3A@9) zjNkc#aQw;t_=}jtA~tbIMjh%>kLI+XCH_togCyPAC>7wcX~5~QEVcR?d;$mc5;-neB~S8`9TOj3A0lU_>G99pb#af$|NQ; zg{jP8E~^M)Gh4XN10M2{FuT-^G^8aXnaE5QvXYPd6rc=csl_mcGlB`MWiwZ~#!YT< zn>*a)5s!I7+}+*{Bqs$0Xi76$(VDJwqdSAx&MtPdhoc`p}nt^k*I4f|8V?G-W7DIm%Okid3R9Rj5ies#AlS)S@J37{)S=@l0SMlbFmDrZSD`%wQ(7n9UsKGLQKzU?GcG%o3KejODCgC97D?8rHIo z^=x1xn+PO`&1_*S+t|(y{$VG(*v%gHvXA{7;2?)M%n^=qjN_c(B&Rsd8P0N!^IYH} zm$=Lou5yj*+~6j+xXm5za*z8w;31EA%oG0QDbIM$3xavcD_--4x4h#$|M7v3eBv`- z_{MjB5W-J>5pcw=BQ#+M%WwS7AA}=35s1j2{ExqgL}a26m1smK1~G|6Y~m1?c*G|G z2}wj^l8}^SBqs$aNkwYXl8*FbAS0Q`Oct_|jqKzgC%MQ?9`cfp{1l)dg(yrBic*Z? zl%OP~C`}p4QjYRepdyv1Ockn9jq22(Cbg(d9qLk#`ZS;+jc800n$nEsw4f!eXiXd1 z(vJ3Ypd+2=Oc%P+jqdcIC%x!RANtad{tRFsgBZ*ZhBA!djNorZGK$fRVJzbq&jcni ziOEc1D$|(G3}!Nm+00=s^O(;97P5%NEMY0jSk4MovWnHLVJ+)e&jvQKi9mwb%oet? zjqU8KM3I`zX&*L z_6bcG!txux^9SJwPXr?JC;#IwA`zJ=L?s&0i9t+a5t}%~B_8ofKtd9cm?R`68OcdO zN>Y)UG^8aR>B&GwGLe}qWF;Hf$w5wXk()f^B_H`IKtT#om?9LV7{w_;NlH=yOIp#IHngQ3?dd>AI?r62tnz(58um>~>h7{eLC-;87wqZz|k#xb4=Ok@(1nZi`2F`XIA zWEQiT!(8Sup9L&r5sO*EQkJot6|7_xt69TZ*0G)qY-AIG1hJVdY-JnU*}*^TWEZ>H z!(R5Wp937^5QjO!QI2t(6P)A}r#Zt}&T*a#T;vj$xx!Vhah)67#&>=Y!cTq?aLoJ@nlOarH-6_2 z!V#VbMC4EY$6rJuGEs<1G@=uOn8YGBafnMi;*)@cBqA|MNJ=u2lY*3_A~k79OFGh% zfsAA#Gg-(=HnNk0oa7=mdB{sX@>76<6rwOiC`vJkQ-YF|qBLbFOF7C@fr?b3GF7Nb zHL6pCn$)5;b*M`{>eGORG@>z0Xi77h(}I??qBU)3OFP=rfsS;dGhOIPH@eev5W$tqT}hPA9?Jsa4_CISg!Gh5ioHny{af7r<`cC&}Q>|;L%ILILm zbA+QD<2WZc$tg~AhO?aGJQujgB`$M?t6bwcH@L|yZgYpb+~YnEc*r9j^MrqS$}^tx zf?!_qir2j1E$?{Ge|+F0pZLrdzVeOl{2+v%{377E`6o1C2+ME$&L4y$JQ0Y-QWF#jADM>|Y(vX&Pq$dLz$wX$dkd00k*TVTw?cViczYB`HN|%21Yal&1m}sYGR}P?c&_rv^2tMQ!R( zmwMEv0S#$HW17&EW;CY-Eont-+R&DEw5J0d=|pF`(3Ng0L%jAjgD8OL}gFp)`2W(rf8#&l*dlUdAW4s)5ud={{fMJ#3sOIgNp pR +| | llama.cpp, both GPUs | upstream Strata, 3090 | upstream Strata, both GPUs | this fork, both GPUs | this fork, 3090 | +|---|---|---|---|---|---| +| Prompt, 16K tokens (t/s) | 581 | 1029 | 1060 | 2466 | 1605 | +| Prompt, 32K tokens (t/s) | 562 | 1120 | 1119 | 2666 | 1768 | +| Prompt, 64K tokens (t/s) | 551 | 1139 | 1148 | 2688 | 1791 | +| Prompt, 128K tokens (t/s) | 527 | 1111 | 1143 | 2650 | 1787 | +| Generation: code, 28-token prompt (t/s) | 31.0 | 74.1 | 84.8 | 167.7 | 113.9 | +| Generation: thinking, 3,000 tokens | 31.3 | 77.1 | 89.0 | 130.2 | 111.2 | +| Generation: edit, the prompt's file returned (2.4K tokens) | 31.1 | 99.4 | 117.2 | 234.8 | 171.3 | +| Generation after the 16K prompt | 29.8 | 75.7 | 85.4 | 151.2 | 108.1 | +| Generation after the 32K prompt | 26.9 | 80.6 | 94.2 | 150.2 | 122.1 | +| Generation after the 64K prompt | 22.0 | 73.3 | 85.3 | 139.6 | 113.9 | +| Generation after the 128K prompt | 16.4 | 67.6 | 76.8 | 129.2 | 110.2 | +| Follow-up turn after the 16K prompt: new tokens, prompt time | 34 in 0.62 s | 34 in 0.35 s | 440 in 1.12 s | 34 in 0.21 s | 34 in 0.34 s | +| Start to the API, the model file cached (s) | 21 | 16 | 21 | 21 | 20 | +| VRAM taken: 3090 / 5070 Ti (GiB) | 22.9 / 13.0 | 22.9 / 0.0 | 21.9 / 14.6 | 22.5 / 13.1 | 22.6 / 0.0 | +| RAM taken (GB) | 57 | 56 | 60 | 53 | 52 | + + +### UD-Q4_K_XL + +Upstream Strata cannot run this file: it has no kernels for K-quant experts. + + +| | llama.cpp, both GPUs | this fork, both GPUs | this fork, 3090 | +|---|---|---|---| +| Prompt, 16K tokens (t/s) | 467 | 2341 | 1099 | +| Prompt, 32K tokens (t/s) | 453 | 2645 | 1184 | +| Prompt, 64K tokens (t/s) | 444 | 2662 | 1209 | +| Prompt, 128K tokens (t/s) | 426 | 2627 | 1221 | +| Generation: code, 28-token prompt (t/s) | 23.5 | 121.3 | 76.8 | +| Generation: thinking, 3,000 tokens | 23.7 | 114.3 | 74.3 | +| Generation: edit, the prompt's file returned (2.4K tokens) | 23.7 | 160.0 | 113.4 | +| Generation after the 16K prompt | 22.6 | 126.7 | 67.6 | +| Generation after the 32K prompt | 21.7 | 138.4 | 75.7 | +| Generation after the 64K prompt | 18.6 | 119.7 | 66.4 | +| Generation after the 128K prompt | 14.5 | 127.5 | 69.2 | +| Follow-up turn after the 16K prompt: new tokens, prompt time | 34 in 0.69 s | 34 in 0.26 s | 34 in 0.48 s | +| Start to the API, the model file cached (s) | 68 | 41 | 36 | +| VRAM taken: 3090 / 5070 Ti (GiB) | 22.3 / 12.9 | 22.7 / 13.1 | 22.7 / 0.0 | +| RAM taken (GB) | 86 | 81 | 80 | + + +## Generation on fixed text (this fork) + +Each run emits the model's reference text (`--spec-follow`), so runs and builds compare on the same tokens and +experts: two runs, averaged, tokens a verify pass in parentheses. The code text's reference repeats its answer after +the end of the turn, which prompt lookup drafts and a server never generates. + + +| Text | IQ3_S t/s (tokens a pass) | UD-Q4_K_XL t/s (tokens a pass) | +|---|---|---| +| code (400 tokens) | 181.5 (4.02) | 114.6 (3.69) | +| thinking (1,200) | 137.8 (2.07) | 105.4 (1.71) | +| after the 16K prompt (400) | 145.2 (2.71) | 124.7 (2.65) | +| edit (2,397) | 243.7 (7.51) | 160.8 (7.51) | + + +## Agreement with llama.cpp + +Argmax agreement and mean KL divergence (nats) of the next-token distributions along llama.cpp's greedy text, which +this fork follows (`--spec-follow`, `--window-logits`), with `--adapt-every 0`. The two baselines differ from each +other only in rounding: llama.cpp's CPU backend against its CUDA backend, and this fork in one-token windows against +its four-token windows. Any change of rounding moves 2-4% of the thinking text's argmaxes, all near-ties. + + +| | Code, ~260 tokens | Thinking, 1,200 | After the 16K prompt, 300 | +|---|---|---|---| +| IQ3_S: this fork vs llama.cpp | 100.0%, 1.3e-04 | 98.0%, 2.1e-03 | 99.3%, 1.0e-03 | +| llama.cpp's CPU vs CUDA backend | 100.0%, 2.2e-04 | 96.9%, 3.2e-03 | 99.0%, 1.4e-03 | +| this fork, one-token vs four-token windows | 100.0%, 1.8e-04 | 98.3%, 1.9e-03 | 99.7%, 3.8e-04 | +| UD-Q4_K_XL: this fork vs llama.cpp | 99.3%, 5.3e-04 | 95.8%, 6.7e-03 | 97.7%, 3.5e-03 | +| llama.cpp's CPU vs CUDA backend | 99.6%, 1.3e-04 | 96.8%, 3.3e-03 | 100.0%, 1.6e-03 | +| this fork, one-token vs four-token windows | 99.6%, 1.0e-04 | 97.2%, 2.3e-03 | 98.7%, 5.4e-04 | + + +UD-Q4_K_XL's CPU baseline after the 16K prompt covers 181 positions: llama-server's greedy text itself differed between +two runs there, and the CPU backend was forced along the other one, which parts after 180 tokens. + +## The two files + +llama.cpp's perplexity over 28 chunks of 4,096 tokens of llama.cpp's own docs, C++ and Python sources (lower is +better): UD-Q4_K_XL 1.8849, IQ3_S 1.8946 (+0.5%), and ISTA-DASLab's IQ3_XXS, which IQ3_S replaced in this fork's +configs, 1.9215 (+1.9%). IQ3_S's experts take 46.8 GiB, UD-Q4_K_XL's 71.7: on this PC IQ3_S generates 16-58% +faster on fixed text, and reads prompts about as fast. + +## Features + +| | llama.cpp | Upstream Strata | This fork | +|---|---|---|---| +| Model files | any GGUF of the model | ISTA-DASLab's GSQ-RCO sizes (Q2_0 to IQ3_S), their Coder, Swift 1.5: i-quant and Q2_0 experts | those, and every quant of unsloth's (K-quant, Q5_1 and Q8_0 experts) | +| Speculative decoding | none: no graph for this model's MTP layer | the MTP layer's drafts, up to 3 a pass; prompt lookup (up to 5 tokens where it pays) | the MTP layer's drafts; prompt lookup (up to 7 tokens after a 16+-token match) | +| Several GPUs | a layer or row split | a layer split, each card holding its layers' experts | the second card as an expert tier: its share of each layer, the next layer's likely experts copied ahead, the prompt's experts the first card lacks, part of the output head | +| Experts in VRAM | a fixed placement | ranked by use, updated every 4 verify passes | ranked by use on both cards, updated every pass | +| KV cache | 8-bit, in VRAM | 8-bit, in RAM past 32K cells | 8-bit, in VRAM | +| Prompt cache | the live slot, states saved to RAM | checkpoints at turns and every 16K tokens | the live sequence, checkpoints at turns and chunks, replaced conversations kept in RAM with their K/V | +| Sampling, APIs | full; OpenAI and Anthropic | per request; OpenAI and Anthropic, a web app, MCP tools | upstream's | diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 2f0b6f80320..dc49db39476 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -182,17 +182,17 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are and start it with `--pack packs/ --native

/-00001-of-0000N.gguf` plus the usual flags (`--spec`, `--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. -UD-Q4_K_XL on an RTX 3090 with 160 GB of RAM: prompts 1004-1134 tok/s, generation 63-73 tok/s. +UD-Q4_K_XL's speeds on an RTX 3090 with 160 GB of RAM, with and without a second GPU: [COMPARISON.md](COMPARISON.md). ### A second GPU With `--second-gpu N`, CUDA device N (the engine runs on `--main-gpu`, default 0) holds more experts and computes its share of each layer while the CPU computes the rest, copies each next layer's likeliest missing experts to itself while the RAM -is idle, and computes the prompt's experts the first GPU lacks; it may be the card that drives the display. Build for -both GPUs' architectures (`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). On an RTX 3090 plus an RTX 5070 Ti (12.6 GB of its -VRAM), 16K-131K prompts: UD-Q4_K_XL 1856-2121 tok/s instead of 1004-1134, generation 94-112 tok/s instead of 63-73; -IQ3_XXS 2028-2256 instead of 1548-1744, generation 115-147 instead of 107-121. When the card has no free VRAM, the -engine runs on the main GPU alone. +is idle, computes the prompt's experts the first GPU lacks, and holds the last rows of the output head (`--head-split`, +0.45 of them); it may be the card that drives the display. Build for both GPUs' architectures +(`-DCMAKE_CUDA_ARCHITECTURES="86;120"`). When the card has no free VRAM, the engine runs on the main GPU alone. An RTX +3090 with and without an RTX 5070 Ti as the second GPU, against upstream Strata and llama.cpp: +[COMPARISON.md](COMPARISON.md). ## Before you start From d3003427549307dd869e86994b35995756e026c0 Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 17:27:09 +0200 Subject: [PATCH 91/92] examples/: this fork's configs for IQ3_S and UD-Q4_K_XL, the paths in setup's layout The engine arguments of the configs COMPARISON.md measures, with paths relative to the Strata folder: the engine in CMake's build/ (setup's engine/ holds upstream's ready-made engine), the model files, packs and MTP draft layer in setup's data folder ../Strata-data. The README says how to build and start them. Both start verbatim from such a layout and answer (checked through junctions to this PC's files). Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- .gitignore | 2 ++ README.md | 3 ++- docs/COMPARISON.md | 8 +++---- docs/DETAILS.md | 3 ++- examples/README.md | 39 ++++++++++++++++++++++++++++++++++ examples/iq3_s.json | 44 ++++++++++++++++++++++++++++++++++++++ examples/ud-q4_k_xl.json | 46 ++++++++++++++++++++++++++++++++++++++++ 7 files changed, 139 insertions(+), 6 deletions(-) create mode 100644 examples/README.md create mode 100644 examples/iq3_s.json create mode 100644 examples/ud-q4_k_xl.json diff --git a/.gitignore b/.gitignore index 16f92c6b471..eca1ac7dff4 100644 --- a/.gitignore +++ b/.gitignore @@ -34,6 +34,8 @@ td.log /run-*.sh /strata-*.json /strata-*.log +# the web app's settings shared with other apps, written next to the config the server runs (examples/) +*.shared-settings.json /build-strata.bat # image encoder (setup.py --vision): build output and the generated build script diff --git a/README.md b/README.md index 2ff67406137..5224330c43e 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,8 @@ of a word): faster than you can read. > **This fork** (branch `custom`) is tuned for a PC with an RTX 3090 and an RTX 5070 Ti: the second card works as another > expert tier, unsloth's K-quant files run too, and prompts and answers are faster. How it compares with llama.cpp and -> upstream Strata on that PC: [docs/COMPARISON.md](docs/COMPARISON.md). +> upstream Strata on that PC: [docs/COMPARISON.md](docs/COMPARISON.md). Its configs, and how to run them: +> [examples/](examples/README.md). > **Jump to:** [How fast?](#how-fast-is-it) · [Which model?](#which-model-should-i-pick) · [Install](#install) · > [Using it](#using-it) · [Problems?](#something-went-wrong) · [How it works](#how-does-it-work) · diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 0c7b7ace839..c55136e16b6 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -31,10 +31,10 @@ The llama.cpp and upstream columns were measured once (2026-09-30). This fork's KV kept in RAM past 32K cells, the expert cache sized by the free VRAM. On the 3090, and on both GPUs as its setup recommends here (a layer split with the newer 5070 Ti first: layers 0-25 there, 26-47 on the 3090). It cannot run UD-Q4_K_XL: it has no K-quant expert kernels. -- **This fork** with its configs: the 3090 runs the model and the 5070 Ti is a second expert tier (12.2-12.4 GiB of - experts; its share of each layer's experts, the next layer's likely experts copied ahead, the prompt path's experts - the 3090 lacks, and with IQ3_S part of the output head). MTP drafts and prompt lookup, 8-bit KV in VRAM, 16K-token - prompt chunks. And on the 3090 alone. +- **This fork** with its configs ([examples](../examples/README.md)): the 3090 runs the model and the 5070 Ti is a + second expert tier (12.2-12.4 GiB of experts; its share of each layer's experts, the next layer's likely experts + copied ahead, the prompt path's experts the 3090 lacks, and with IQ3_S part of the output head). MTP drafts and + prompt lookup, 8-bit KV in VRAM, 16K-token prompt chunks. And on the 3090 alone. ## Speed through the API diff --git a/docs/DETAILS.md b/docs/DETAILS.md index dc49db39476..5192c8df1fe 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -181,7 +181,8 @@ Pack a model once (seconds: the tokenizer and the float tensors; the experts are python tools/iq_pack.py --gguf /-00001-of-0000N.gguf --out packs/ and start it with `--pack packs/ --native /-00001-of-0000N.gguf` plus the usual flags (`--spec`, -`--mtp`, `--prefill`, ...). The other shards, and the one holding the n-gram table, are found by name. +`--mtp`, `--prefill`, ...; this fork's config: [examples/ud-q4_k_xl.json](../examples/ud-q4_k_xl.json)). The +other shards, and the one holding the n-gram table, are found by name. UD-Q4_K_XL's speeds on an RTX 3090 with 160 GB of RAM, with and without a second GPU: [COMPARISON.md](COMPARISON.md). ### A second GPU diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 00000000000..9cd1f9e37b8 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,39 @@ +# This fork's configs + +The configs this fork runs with on a PC with an RTX 3090, an RTX 5070 Ti and 160 GB of RAM, for `serve/server.py` +(their speeds: [COMPARISON.md](../docs/COMPARISON.md)): + +- `iq3_s.json`: ISTA-DASLab's GSQ-RCO IQ3_S (~53 GB of RAM taken). +- `ud-q4_k_xl.json`: unsloth's UD-Q4_K_XL (~81 GB of RAM taken). + +Their paths are relative to the Strata folder, where the server starts: the engine in `build/`; the model files, the +packs and the MTP draft layer in setup's data folder, `../Strata-data`. + +1. Set up IQ3_S once: `START-HERE.bat --setup --model IQ3_S --no-start` downloads the model and prepares its pack and + the MTP draft layer. +2. Build the engine from this branch with code for both cards (the ready-made engine setup puts in `engine/` is + upstream's, without this fork's options), with MSVC, CUDA 13, CMake and Ninja (setup puts the last two in + `.venv\Scripts`): + + cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF ^ + "-DCMAKE_CUDA_ARCHITECTURES=86;120" + cmake --build build --target strata + + `86;120` is an RTX 30 card and an RTX 50 card. The engine needs the CUDA runtime DLLs (`cudart64_13.dll`, + `cublas64_13.dll`, `cublasLt64_13.dll`) beside it, on the PATH, or their folder in the config's `lib_dirs`. +3. For UD-Q4_K_XL, download unsloth's four `Qwen3.8-Flash-Next-UD-Q4_K_XL-0000N-of-00004.gguf` files into + `../Strata-data/models/UD-Q4_K_XL` and pack them (seconds): + + .venv\Scripts\python.exe tools\iq_pack.py --out ../Strata-data/packs/ud-q4_k_xl ^ + --gguf ../Strata-data/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf + +4. Set the GPU numbers, as `nvidia-smi` numbers the cards (here the 5070 Ti is 0 and the 3090 is 1). `--main-gpu` + runs the model, and nothing else may use that card: a display or another program on it slows generation to a few + tokens per second. `--second-gpu` holds more experts and may drive the display. +5. Start a config from the Strata folder: + + .venv\Scripts\python.exe serve\server.py --engine strata --config examples\iq3_s.json --port 8081 + + (`START-HERE.bat` starts setup's own config.) + +On Linux the engine is `./build/strata` and Python `.venv/bin/python`. diff --git a/examples/iq3_s.json b/examples/iq3_s.json new file mode 100644 index 00000000000..b4cccab107b --- /dev/null +++ b/examples/iq3_s.json @@ -0,0 +1,44 @@ +{ + "exe": "./build/strata.exe", + "args": [ + "--pack", + "../Strata-data/packs/iq3_s", + "--native", + "../Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf", + "--expert-profile", + "./data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "16384", + "--feed-max", + "80", + "--spec", + "4", + "--spec-min-p", + "0.5", + "--spec-lookup", + "16", + "--adapt-every", + "1", + "--mtp", + "../Strata-data/mtp/rt", + "--max-context", + "200192", + "--kv", + "int8", + "--pcie-mode", + "kernel", + "--pcie-frac", + "0", + "--second-gpu", + "0", + "--main-gpu", + "1" + ], + "cwd": ".", + "tokenizer": "../Strata-data/packs/iq3_s/tokenizer", + "model_name": "qwen3.8-flash-next-iq3_s", + "log": "./strata-iq3_s.log", + "lib_dirs": [] +} diff --git a/examples/ud-q4_k_xl.json b/examples/ud-q4_k_xl.json new file mode 100644 index 00000000000..6a516293ea7 --- /dev/null +++ b/examples/ud-q4_k_xl.json @@ -0,0 +1,46 @@ +{ + "exe": "./build/strata.exe", + "args": [ + "--pack", + "../Strata-data/packs/ud-q4_k_xl", + "--native", + "../Strata-data/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf", + "--expert-profile", + "./data/expert-profile.bin", + "--expert-cache", + "auto", + "--prefill", + "16384", + "--feed-max", + "52", + "--spec", + "4", + "--spec-min-p", + "0.5", + "--spec-lookup", + "16", + "--adapt-every", + "1", + "--mtp", + "../Strata-data/mtp/rt", + "--max-context", + "200192", + "--kv", + "int8", + "--pcie-mode", + "kernel", + "--pcie-frac", + "0", + "--second-gpu", + "0", + "--main-gpu", + "1", + "--head-split", + "0" + ], + "cwd": ".", + "tokenizer": "../Strata-data/packs/ud-q4_k_xl/tokenizer", + "model_name": "qwen3.8-flash-next-ud-q4_k_xl", + "log": "./strata-ud-q4_k_xl.log", + "lib_dirs": [] +} From 25e86c762b569f8d4a5f434e3cb31b4b916ee50d Mon Sep 17 00:00:00 2001 From: Eddoursul Date: Wed, 30 Sep 2026 17:41:09 +0200 Subject: [PATCH 92/92] examples/: a launcher per config; the README says where the packs and the MTP layer come from run-iq3_s.bat (port 8081) and run-ud-q4_k_xl.bat (8082) start the server from the Strata folder with their config, as this fork's own launchers do. Both start and answer from a setup-style layout (checked as copies on another port beside them). The README: a pack is tools/iq_pack.py's output from the model's GGUF files (setup makes IQ3_S's, step 3 UD-Q4_K_XL's), the MTP draft layer setup's conversion of the original checkpoint's MTP tensors. Claude-Session: https://claude.ai/code/session_019XxRxQQNyk7Ycdr4ComSs1 --- examples/README.md | 17 +++++++++++------ examples/run-iq3_s.bat | 10 ++++++++++ examples/run-ud-q4_k_xl.bat | 10 ++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-) create mode 100644 examples/run-iq3_s.bat create mode 100644 examples/run-ud-q4_k_xl.bat diff --git a/examples/README.md b/examples/README.md index 9cd1f9e37b8..51cdfc09cdc 100644 --- a/examples/README.md +++ b/examples/README.md @@ -7,10 +7,14 @@ The configs this fork runs with on a PC with an RTX 3090, an RTX 5070 Ti and 160 - `ud-q4_k_xl.json`: unsloth's UD-Q4_K_XL (~81 GB of RAM taken). Their paths are relative to the Strata folder, where the server starts: the engine in `build/`; the model files, the -packs and the MTP draft layer in setup's data folder, `../Strata-data`. - -1. Set up IQ3_S once: `START-HERE.bat --setup --model IQ3_S --no-start` downloads the model and prepares its pack and - the MTP draft layer. +packs and the MTP draft layer in setup's data folder, `../Strata-data` (unless setup was given another: `--data-dir`). +A pack is what `tools/iq_pack.py` makes from a model's GGUF files in seconds (~1.5 GB): the tokenizer with the chat +template, the model's float tensors in the forms the engine takes, and an index of the rest; the engine reads the +quantized tensors, the experts and most projections, from the GGUF files at start. + +1. Set up IQ3_S once: `START-HERE.bat --setup --model IQ3_S --no-start` downloads the model into + `../Strata-data/models/IQ3_S`, makes its pack in `../Strata-data/packs/iq3_s`, and makes the MTP draft layer in + `../Strata-data/mtp/rt` from the ~5 GB of MTP tensors of Qwen's original checkpoint. 2. Build the engine from this branch with code for both cards (the ready-made engine setup puts in `engine/` is upstream's, without this fork's options), with MSVC, CUDA 13, CMake and Ninja (setup puts the last two in `.venv\Scripts`): @@ -22,7 +26,7 @@ packs and the MTP draft layer in setup's data folder, `../Strata-data`. `86;120` is an RTX 30 card and an RTX 50 card. The engine needs the CUDA runtime DLLs (`cudart64_13.dll`, `cublas64_13.dll`, `cublasLt64_13.dll`) beside it, on the PATH, or their folder in the config's `lib_dirs`. 3. For UD-Q4_K_XL, download unsloth's four `Qwen3.8-Flash-Next-UD-Q4_K_XL-0000N-of-00004.gguf` files into - `../Strata-data/models/UD-Q4_K_XL` and pack them (seconds): + `../Strata-data/models/UD-Q4_K_XL` and make its pack: .venv\Scripts\python.exe tools\iq_pack.py --out ../Strata-data/packs/ud-q4_k_xl ^ --gguf ../Strata-data/models/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf @@ -30,7 +34,8 @@ packs and the MTP draft layer in setup's data folder, `../Strata-data`. 4. Set the GPU numbers, as `nvidia-smi` numbers the cards (here the 5070 Ti is 0 and the 3090 is 1). `--main-gpu` runs the model, and nothing else may use that card: a display or another program on it slows generation to a few tokens per second. `--second-gpu` holds more experts and may drive the display. -5. Start a config from the Strata folder: +5. Start a config with its launcher, `run-iq3_s.bat` (port 8081) or `run-ud-q4_k_xl.bat` (8082), or from the Strata + folder: .venv\Scripts\python.exe serve\server.py --engine strata --config examples\iq3_s.json --port 8081 diff --git a/examples/run-iq3_s.bat b/examples/run-iq3_s.bat new file mode 100644 index 00000000000..22992850a63 --- /dev/null +++ b/examples/run-iq3_s.bat @@ -0,0 +1,10 @@ +@echo off +title Strata IQ3_S +rem Qwen3.8-Flash-Next, ISTA-DASLab's GSQ-RCO IQ3_S, on http://127.0.0.1:8081/v1 with iq3_s.json (README.md). +rem The config's --main-gpu runs the engine: another program on that card stalls its per-layer spin-waits. +rem Its --second-gpu holds more experts and may drive the display. +rem A PYTHONPATH set for another Python would mix that Python's packages into the venv's. +set "PYTHONPATH=" +cd /d "%~dp0.." +".venv\Scripts\python.exe" serve\server.py --engine strata --config examples\iq3_s.json --port 8081 +pause diff --git a/examples/run-ud-q4_k_xl.bat b/examples/run-ud-q4_k_xl.bat new file mode 100644 index 00000000000..48e1c2e6afa --- /dev/null +++ b/examples/run-ud-q4_k_xl.bat @@ -0,0 +1,10 @@ +@echo off +title Strata UD-Q4_K_XL +rem Qwen3.8-Flash-Next, unsloth's UD-Q4_K_XL, on http://127.0.0.1:8082/v1 with ud-q4_k_xl.json (README.md). +rem The config's --main-gpu runs the engine: another program on that card stalls its per-layer spin-waits. +rem Its --second-gpu holds more experts and may drive the display. +rem A PYTHONPATH set for another Python would mix that Python's packages into the venv's. +set "PYTHONPATH=" +cd /d "%~dp0.." +".venv\Scripts\python.exe" serve\server.py --engine strata --config examples\ud-q4_k_xl.json --port 8082 +pause