Skip to content

hyperloom custom workload: the quality-gate verdict does not name the reference it was measured against, so a KLD number cannot be compared across sessions #954

Closed

Description

@bong-water-water-bong

Class: the artifact does not describe the measurement (same as #892, #930, #932, #947 — all fixed).

benchmark.py decides pass/fail by comparing a candidate against two references kept in HL_STATE
(~/lb/hl-gate/<model-stem>): kld-base.kld (1.0 GB of base logits) and texts.json (the baseline's three greedy
replies). The result JSON records the candidate's own numbers only:

$ python3 -c "import json; print(sorted(json.load(open('arm_pair2_c20-A.json'))['quality_gate']))"
['kld_max_mean', 'kld_mean', 'kld_ok', 'kld_recorded', 'passed', 'reason', 'texts_differing', 'texts_identical']

So a row that reads KLD 0.000e+00 <= 0.0001, texts identical cannot be tied to the reference it is a distance to.
That state is per box, outside the checkout, and kld() re-records the base whenever kld-base.kld is absent
(recording = not base.is_file()), so the same KLD number can be PASS against one base and meaningless against
another — while the reports of the two series look identical. ref.json does carry a model fingerprint and the
baseline's levers, but neither reaches the result.

Wanted: every compared result names the reference it was compared against.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions