Skip to content

Experiment: choose between prompt lookup and MTP drafts by measured acceptance and cost #37

Closed

Description

@Piggidragon

Idea from Strata by @Niko1221 (MIT): when the reply repeats the context (code edits, quoted text), prompt lookup drafts up to 5 tokens from the earlier copy, but only where its measured acceptance and cost say it pays. Strata reports code edits 6-11% faster and other text unchanged (HOW_IT_WORKS.md, "Guess, then check").

Where we are

  • The fork has n-gram drafters (ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache), and --spec-type takes a comma-separated list.
  • With several drafters the first one that produces a draft wins (common_speculative_draft, common/speculative.cpp). Nothing tracks acceptance or verify cost.
  • Coding-agent traffic copies a lot: edits of quoted files, tool outputs, diffs. MTP alone already gets acceptance 0.80-0.91 on the edit-code task of mtp-bench.py, against 0.53-0.64 on prose.

Experiment

  1. Baseline: draft-mtp alone vs ngram-mod,draft-mtp vs draft-mtp,ngram-mod (first wins) on the 4 tasks of mtp-bench.py. Add an agent-like task that rewrites a long quoted file with small changes. Both temperature 1.0 and greedy.
  2. A per-sequence chooser: keep a running estimate of each drafter's accepted tokens per draft and the cost of its draft plus verify window. Take the drafter with the most expected tokens per ms for the next step; an n-gram draft can be longer than --spec-draft-n-max.
  3. Keep the MTP context in sync when an n-gram draft was used (catch-up of the accepted tokens).

Done when

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions