[code-martin]

Summary

Implements continuous, non-blocking MoE expert cache adaptation between VRAM resident slots and host RAM during multi-turn generation.

Motivation

Static expert placement profiles calibrated on generic corpora often yield low hit rates (~25–35%) when operating in specialized domains or conversations in other languages. Continuous adaptation enables the cache to dynamically follow conversation context.

Key Changes

• Usage Frequency Tracking:
  • Tracks expert routing activation counters per layer during token generation.
• Exponential Decay Smoothing:
  • Applies a gentle 0.985 decay factor per round instead of abrupt reset (0.7), maintaining stable long-term resident profiles while remaining responsive to topic shifts.
• Hysteresis & Thrashing Prevention:
  • Implements candidate and victim thresholds (gain +1.0) to avoid ping-pong thrashing between VRAM and RAM.
• Engine & Server Integration:
  • Adds the ADAPT protocol command to generate.cpp.
  • Automatically triggers non-blocking adaptation rounds in serve/server.py at the end of each request.

Testing

• Tested on multi-turn conversations in other languages: observed over 5,600 successful expert swaps, with cache hit rate climbing from 25.8% to 63.0%.

[Niko1221]

Thanks for the work and the write-ups. I went through all three carefully, and I'm not merging them as they are - here is why, so the useful parts can come back in a shape that fits:

#37 (VMM cache + on-demand vision):

• The cache can only shrink and grow back to its startup size, and the slot gain in the description comes from loading vision after the engine rather than from VMM.
• RESIZE/GROW copy the cache's startup residency table over the live one (xcache.residency_table()), while adaptation (on by default) has already swapped slots - after a resize those slots are read as the wrong expert.
• Since 0.1.13 the prompt path borrows the top of the cache for its buffers; shrinking unmaps exactly that region, and a partial GROW leaves those pointers on unmapped memory.
• The 5.4 GB / 10 GB targets are hard-coded, and every image pays a full encoder load (~7.5 s).

#38 (adaptive eviction): it mostly retunes the adaptation the engine already has (0.7 was already exponential decay), the new ADAPT command blocks after every request, and it reads usage even when adaptation is off (--adapt-swaps 0), which crashes. A per-swap stderr line floods the log.

#39 (telemetry): it changes adapt_every 4 -> 1 and adapt_swaps 96 -> 1 without saying so, checkpoints at every turn boundary (for a 40-turn agent prompt that is ~40 borrow/refill cycles and ~40 x 118 MB saves, which undoes much of 0.1.13's prompt speedup), and /api/stats skips the API-key check.

What I'd gladly take as small separate PRs against current main: the per-request decode hit rate (the DONE fields, the Monitor column), keeping the root checkpoint (only the first turn boundary), the Vision.close() temp-dir cleanup, and flags for the adaptation thresholds - each with a before/after measurement. Also please drop tools/test_vision_client.py and the CMakeLists BOM change from them.