Skip to content

[Docs] does --tail-skip cost quality? Eight prompts, both ways - #9

Merged
RandoTeam merged 1 commit into
mainfrom
docs/tailskip-quality
Nov 3, 2026
Merged

RandoTeam merged 1 commit into
mainfrom
docs/tailskip-quality

Conversation

@RandoTeam

Copy link
Copy Markdown
Owner

[Docs] does --tail-skip cost quality? Eight prompts, both ways

The flag is in Clover's config and its effect on answers had never been
measured. The server's OpenAI endpoint, greedy, the same eight prompts with
--tail-skip 6 and with it off:

  • arithmetic, primes, JSON, the capital of Australia: byte-identical.
  • code: the same function, one answer in a fence and one without.
  • Russian: the same three reasons, other wording.
  • summary and Fibonacci: the same content, different spacing and phrasing.

So every answer is correct in both runs, four of eight byte-identical, and
the rest differ in formatting; the reasoning block differs each time, which
is expected because the cache changes where each expert is computed.

It is not a quality gate - eight short prompts are a smoke test. The page
says so, and names what is still to come: long context, tool calls, code
that has to run.

Validation:
Passed: two full server runs (one per setting), same config otherwise,
temperature 0, max_tokens 1200 (smaller budgets stopped inside the
reasoning block and returned empty answers, which is a budget artifact,
not a quality difference).
Not run: more than eight prompts, long context, tool calls.

The flag is in Clover's config and its effect on answers had never been
measured. The server's OpenAI endpoint, greedy, the same eight prompts with
--tail-skip 6 and with it off:

- arithmetic, primes, JSON, the capital of Australia: byte-identical.
- code: the same function, one answer in a fence and one without.
- Russian: the same three reasons, other wording.
- summary and Fibonacci: the same content, different spacing and phrasing.

So every answer is correct in both runs, four of eight byte-identical, and
the rest differ in formatting; the reasoning block differs each time, which
is expected because the cache changes where each expert is computed.

It is not a quality gate - eight short prompts are a smoke test. The page
says so, and names what is still to come: long context, tool calls, code
that has to run.

Validation:
Passed: two full server runs (one per setting), same config otherwise,
  temperature 0, max_tokens 1200 (smaller budgets stopped inside the
  reasoning block and returned empty answers, which is a budget artifact,
  not a quality difference).
Not run: more than eight prompts, long context, tool calls.
@RandoTeam
RandoTeam merged commit 91e9618 into main Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant