Repository navigation
[Docs] does --tail-skip cost quality? Eight prompts, both ways - #9
Merged
Merged
Conversation
The flag is in Clover's config and its effect on answers had never been measured. The server's OpenAI endpoint, greedy, the same eight prompts with --tail-skip 6 and with it off: - arithmetic, primes, JSON, the capital of Australia: byte-identical. - code: the same function, one answer in a fence and one without. - Russian: the same three reasons, other wording. - summary and Fibonacci: the same content, different spacing and phrasing. So every answer is correct in both runs, four of eight byte-identical, and the rest differ in formatting; the reasoning block differs each time, which is expected because the cache changes where each expert is computed. It is not a quality gate - eight short prompts are a smoke test. The page says so, and names what is still to come: long context, tool calls, code that has to run. Validation: Passed: two full server runs (one per setting), same config otherwise, temperature 0, max_tokens 1200 (smaller budgets stopped inside the reasoning block and returned empty answers, which is a budget artifact, not a quality difference). Not run: more than eight prompts, long context, tool calls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Docs] does --tail-skip cost quality? Eight prompts, both ways
The flag is in Clover's config and its effect on answers had never been
measured. The server's OpenAI endpoint, greedy, the same eight prompts with
--tail-skip 6 and with it off:
So every answer is correct in both runs, four of eight byte-identical, and
the rest differ in formatting; the reasoning block differs each time, which
is expected because the cache changes where each expert is computed.
It is not a quality gate - eight short prompts are a smoke test. The page
says so, and names what is still to come: long context, tool calls, code
that has to run.
Validation:
Passed: two full server runs (one per setting), same config otherwise,
temperature 0, max_tokens 1200 (smaller budgets stopped inside the
reasoning block and returned empty answers, which is a budget artifact,
not a quality difference).
Not run: more than eight prompts, long context, tool calls.