You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Persist the runtime expert heat map across restarts; optionally select a profile per working directory #477
What the engine already does
At startup, GPU expert residency is seeded from the shipped static profile (data/expert-profile.bin), then refined at runtime by --adapt-every as real token traffic is observed.
Measured on my side (RTX 4090 24 GB, IQ3_S, 262K context): the decode cache hit rate climbs 63% → 73% → 85% within the first requests, and long agent sessions then hold a much higher sustained decode rate (65-75 tok/s, MTP draft acceptance 65-84%).
The gap
Everything learned since engine start lives only in RAM. On restart, the engine falls back to the generic shipped profile, and the first minutes of every session are spent re-learning my usage pattern from scratch.
Proposal — three levels, each independently useful
Persist the learned residency. Save the runtime heat map on clean exit (or periodically) and reload it at start instead of the shipped profile — e.g. --expert-profile-save paired with the existing --expert-profile . A cold start becomes warm.
Per-context selection. The engine already accepts a profile file at start, so launchers could pick one per working directory (code vs prose vs a specific codebase — the token distribution of a project is a signature, and different projects heat different experts). Only the save side from (1) is missing to make this possible.
Later: opt-in sharing/blending of profiles. They are small (KB-scale), so per-user or per-domain profiles ("Python coder", "FR writer"…) could be exchanged. A profile is a usage fingerprint, though, so this should stay strictly opt-in.
Why it fits the current design
The load path exists today; the runtime measurement exists today. This is only save + selection built around them.
Staleness risk is minimal: --adapt-every still refines at runtime, so an outdated profile is simply a better-than-generic prior, never a wrong answer.
Great fit for agent workloads (Claude Code and friends), where engines get restarted often and the first minutes matter.
Note
Keeping the save feature off by default (or local-only) seems right, since profiles reflect what a user works on.
It’s a good idea, but I’d look at it from a different angle. Essentially, this is a ready-made expert importance file for LLM pruning, generated specifically based on your tasks.
Added in 0.1.36 as an opt-in: "expert_profile_save": "path" in strata-<model>.json (or --expert-profile-save PATH, saved every --expert-profile-save-every minutes, default 10, and on exit). The saved profile ranks the experts in VRAM first, then what the adaptive tier learned, then the shipped profile's order, in the same format --expert-profile reads. The next start begins from it, and per-project profiles are just different paths. For now it learns from decoding, not from the batched prompt pass. Thanks for the well-argued proposal!
Activity
ladno64 commentedon Oct 2, 2026
It’s a good idea, but I’d look at it from a different angle. Essentially, this is a ready-made expert importance file for LLM pruning, generated specifically based on your tasks.
Opt-in: keep what the expert cache learned across restarts (--expert-…
Engine 0.1.36: a cancelled prompt logged as read so far (#471), the d…
Niko1221 commentedon Oct 2, 2026
Added in 0.1.36 as an opt-in:
"expert_profile_save": "path"instrata-<model>.json(or--expert-profile-save PATH, saved every--expert-profile-save-everyminutes, default 10, and on exit). The saved profile ranks the experts in VRAM first, then what the adaptive tier learned, then the shipped profile's order, in the same format--expert-profilereads. The next start begins from it, and per-project profiles are just different paths. For now it learns from decoding, not from the batched prompt pass. Thanks for the well-argued proposal!Adaptive tier: --adapt-decay
Opt-in: keep what the expert cache learned across restarts (--expert-…
Engine 0.1.36: a cancelled prompt logged as read so far (#471), the d…
Adaptive tier: --adapt-decay
pool: the split learns from its own timings; split nodes keep what th…
Opt-in: keep what the expert cache learned across restarts (--expert-…
Engine 0.1.36: a cancelled prompt logged as read so far (Niko1221#471)…
Adaptive tier: --adapt-decay
Opt-in: keep what the expert cache learned across restarts (--expert-…
Engine 0.1.36: a cancelled prompt logged as read so far (Niko1221#471)…
Adaptive tier: --adapt-decay
プレーンなモデルの serve: 学んだ置き場を --expert-profile-save-every 分ごとにも保存する
--expert-profile-save (upstream's Niko1221#477) for this engine: the …
Fork 0.1.36-nvfp4.1: README, docs/NVFP4.md and the bundle version