Skip to content

Persist the runtime expert heat map across restarts; optionally select a profile per working directory #477

Closed

Description

@AXKore

What the engine already does
At startup, GPU expert residency is seeded from the shipped static profile (data/expert-profile.bin), then refined at runtime by --adapt-every as real token traffic is observed.
Measured on my side (RTX 4090 24 GB, IQ3_S, 262K context): the decode cache hit rate climbs 63% → 73% → 85% within the first requests, and long agent sessions then hold a much higher sustained decode rate (65-75 tok/s, MTP draft acceptance 65-84%).
The gap
Everything learned since engine start lives only in RAM. On restart, the engine falls back to the generic shipped profile, and the first minutes of every session are spent re-learning my usage pattern from scratch.

Proposal — three levels, each independently useful
Persist the learned residency. Save the runtime heat map on clean exit (or periodically) and reload it at start instead of the shipped profile — e.g. --expert-profile-save paired with the existing --expert-profile . A cold start becomes warm.
Per-context selection. The engine already accepts a profile file at start, so launchers could pick one per working directory (code vs prose vs a specific codebase — the token distribution of a project is a signature, and different projects heat different experts). Only the save side from (1) is missing to make this possible.
Later: opt-in sharing/blending of profiles. They are small (KB-scale), so per-user or per-domain profiles ("Python coder", "FR writer"…) could be exchanged. A profile is a usage fingerprint, though, so this should stay strictly opt-in.
Why it fits the current design
The load path exists today; the runtime measurement exists today. This is only save + selection built around them.
Staleness risk is minimal: --adapt-every still refines at runtime, so an outdated profile is simply a better-than-generic prior, never a wrong answer.
Great fit for agent workloads (Claude Code and friends), where engines get restarted often and the first minutes matter.
Note
Keeping the save feature off by default (or local-only) seems right, since profiles reflect what a user works on.

Environment: engine 0.1.34, Windows 11, RTX 4090, Ryzen 9 5950X, 128 GB RAM, IQ3_S packed install.

Activity

  1. ladno64 commented on Oct 2, 2026

    @ladno64

    It’s a good idea, but I’d look at it from a different angle. Essentially, this is a ready-made expert importance file for LLM pruning, generated specifically based on your tasks.

  2. Niko1221 commented on Oct 2, 2026

    @Niko1221
    Owner

    Added in 0.1.36 as an opt-in: "expert_profile_save": "path" in strata-<model>.json (or --expert-profile-save PATH, saved every --expert-profile-save-every minutes, default 10, and on exit). The saved profile ranks the experts in VRAM first, then what the adaptive tier learned, then the shipped profile's order, in the same format --expert-profile reads. The next start begins from it, and per-project profiles are just different paths. For now it learns from decoding, not from the batched prompt pass. Thanks for the well-argued proposal!

  3. added a commit that references this issue on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions