Skip to content

conformance: wire the Phase 2–2.8 scenario bodies as their features land #444

Open

Description

@tommasobolis

Summary

Split out from #9. The M1 conformance kit has shipped: the scenario
declaration (tests/conformance/suite.py — CONFORMANCE_SCENARIOS +
KNOWN_LIMITATIONS), the LangGraph conformance run, the customer-facing
pytest plugin (donkey_kit/conformance/*, #191), the contract fixtures, and
the nightly matrix (nightly-matrix.yml). This epic tracks the remaining,
still-open work: wiring the scenario bodies for the Phase 2–2.8 scenarios,
each of which is gated on the feature it exercises.

The scenario names already exist in CONFORMANCE_SCENARIOS; what's missing is
each scenario's executable body plus its LangGraph pass and any asserted
KNOWN_LIMITATIONS exemption. Never a silent skip (§8.1).

Motivation

Working instruction #5: build the conformance kit before the second adapter —
done. But a declared scenario with no body is not coverage. Each Phase 2
feature (tools discovery, scanner/publish, adapter descriptor) must land its
matching conformance scenario in the same slice, or the kit's credibility
erodes into a list of names. This epic keeps that lockstep visible.

Scope — scenario bodies to wire, grouped by the feature that gates them

Each item wires the body, adds the LangGraph pass, and records any framework
exemption in KNOWN_LIMITATIONS with a stated cause.

Cross-surface impact

Out of scope

Depends on

Related

Activity

  1. added
    epicTracking issue spanning several child issues
    area:testingConformance kit, contract fixtures, nightly matrix
    priority/p1Must-have: blocks a real adapter or CI gate
    size/l3+ days, cross-layer or verification sign-off
    on Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testingConformance kit, contract fixtures, nightly matrixepicTracking issue spanning several child issuespriority/p1Must-have: blocks a real adapter or CI gatesize/l3+ days, cross-layer or verification sign-off

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions