Repository navigation
test(conformance): retire 13 body-less scenarios, pin README (#749) - #992
Merged
Merged
Conversation
…ons (#749) Trims CONFORMANCE_SCENARIOS from 21 to 8 names, keeping only the ones with a real executable body (proven by the adapter contract suite, #742, and test_typed_refusal_bridge.py) and retiring the 13 that described registry/tools/governance/publication surfaces still blocked by _verify.blocked(...) per the verification discipline (§0.3) — there is no defensible scenario body to write for a surface that unconditionally refuses. Adds a lock test (test_conformance_scenarios_are_exactly_the_executable_set) so the retired names can't silently drift back in without a real body, and a new test_readme_exemptions_table.py that cross-checks every README "Conformance exemptions" row against KNOWN_LIMITATIONS in both directions, so the published table can't drift from the code it claims to summarize. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the review-time prose mapping and the self-referential set check with SCENARIO_BODIES: each CONFORMANCE_SCENARIOS name lists the pytest node ids that run it, and test_every_scenario_has_an_executable_body checks the keys match and that every listed test function exists. A renamed or deleted body now fails CI instead of leaving a bare name. Corrects the mapping: jwt_token_refreshed runs in tests/unit/test_adapter_jwt_mode.py (per-send JWT, CrewAI refusal), not the shared-client kwargs test; simple_completion is the buffered async call plus the sync call. Narrows the retirement note to the per-adapter surfaces that are blocked and points at the unit-tested pure logic. The README exemptions test now checks both READMEs (repo and PyPI), and the parsing is in one helper. CONTRIBUTING notes that a scenario lands with its body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#749) Plain pytest does not put python/ on sys.path, so importing examples.crewai.main failed in CI's test job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
python/tests/conformance/suite.py) from 21CONFORMANCE_SCENARIOSto 8. The 13 retired names (tools, registry, governance, publication) need a per-adapter surface that still raises_verify.blocked(...): MCP binding intools/session.py, and Exchange discovery, publication and introspection inregistry/. No adapter can run a scenario against a surface that always refuses. The pure logic beneath them (ToolFilter, collision prefixes, criteria, digests) is already unit-tested intests/unit/test_pure_logic.py.SCENARIO_BODIES, which maps each remaining scenario to the pytest node ids that run it.test_every_scenario_has_an_executable_bodychecks that the map's keys equalCONFORMANCE_SCENARIOSand that every listed test function exists, so a name can't outlive its body.jwt_token_refreshedruns intests/unit/test_adapter_jwt_mode.py.typed_refusal_bridgedruns intest_typed_refusal_bridge.py.tests/conformance/test_readme_exemptions_table.py. It checks the "Conformance exemptions" table in both the repoREADME.mdandpython/README.md(PyPI) againstKNOWN_LIMITATIONS, in both directions. The one public-only row (budget refusal not retried) is checked againstexamples/crewai/main.py.Closes #749
Done-when amended (maintainer decision)
KNOWN_LIMITATIONSkeys may stay internal scenario names (CrewAI'sgateway_identity_observed,jwt_token_refreshedandtyped_refusal_bridged, ADK'styped_refusal_bridged), as long as each has an executable body.last_callreportsUNAVAILABLE, CrewAI raisesConfigErrorin jwt mode. One exception:typed_refusal_bridgedfor ADKmodel()and CrewAI is asserted only through thetyped_refusalscapability flag, with no call run.Test plan
pytest -q: 2328 passed, 199 skipped, 13 deselectedpytest -q -m local_gateway: 9 passed, 5 skippedmypy(strict): no issues in 79 source files; per-example mypy (adk, agent_framework, anthropic, crewai, llamaindex, openai_agents, quickstart, strands) all cleanruff check .: all checks passedlint-imports: 9 contracts kept, 0 brokenvulture,scripts/check_verification_claims.py,scripts/check_doc_links.py,scripts/scrub_fixtures.py --check: exit 0scripts/check_new_dependencies.py --base <develop pyproject>: no new direct dependenciesSCENARIO_BODIESentry at a missing testKNOWN_LIMITATIONSentryPost-deploy steps
donkey-development-kit-ai-config/CLAUDE.mdand theddk-testingskill.swallows_pii_as_generic,works_without_budget_headers).🤖 Generated with Claude Code