Skip to content

ci: add informational downstream compatibility - #24

Merged
leoromanovsky merged 1 commit into
mainfrom
agent/informational-downstream-compatibility
Aug 10, 2026
Merged

ci: add informational downstream compatibility#24
leoromanovsky merged 1 commit into
mainfrom
agent/informational-downstream-compatibility

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Fixture pull requests need early evidence from the tracer implementations without assuming that either the SDK or the shared expectation is automatically correct. PR #23 demonstrated the fan-out while stacked on the regex proposal; this PR establishes the same feedback loop directly from main and independently of regex conformance.

Changes

  • Add blocking static validation for ufc-config.json and every evaluation-cases JSON file, including duplicate keys, required envelope fields, map-key consistency, and unknown flag references.
  • Run canonical base/head evaluation tests in Go, Java, JavaScript, Python, Ruby, .NET, PHP, and libdatadog.
  • Classify each consumer as compatible, new regression, improvement, or existing drift and upload both logs.
  • Document the local validation command and the different merge policies for static and downstream checks.

👉 Example JSON fixture failure: #25

Decisions

  • Base this PR directly on main; it includes no regex contract or fixture-data changes.
  • Keep downstream compatibility advisory and allowed to fail. These jobs must not be configured as required checks.
  • Keep static fixture validation blocking because malformed fixture files are unambiguously repository errors.
  • Do not automatically update submodule pins, open issues, or decide whether a mismatch belongs to the consumer or fixture.
  • Continue invoking consumer-owned test entrypoints from purpose-built branches until those entrypoints land on the repositories default branches.

Validation

  • python3 ci/validate-fixtures.py: 258 cases across 33 files pass.
  • Duplicate-key and missing-field negative paths reject invalid input.
  • bash -n and shellcheck pass for the runner.
  • actionlint v1.7.7 and YAML parsing pass for both workflows.
  • A real Go base/head run classifies current main as compatible.
  • A synthetic failing consumer classifies existing drift and exits successfully, proving the compatibility result is informational.

Run blocking static validation and advisory base/head SDK comparisons.

Environment: Datadog workspace
@leoromanovsky
leoromanovsky marked this pull request as ready for review August 10, 2026 19:27
@leoromanovsky
leoromanovsky requested a review from a team as a code owner August 10, 2026 19:27
@leoromanovsky
leoromanovsky requested review from btthomas and typotter and removed request for a team August 10, 2026 19:27

@btthomas btthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#25

Nice! 🚀

@leoromanovsky
leoromanovsky merged commit 256920d into main Aug 10, 2026
10 checks passed
@leoromanovsky
leoromanovsky deleted the agent/informational-downstream-compatibility branch August 10, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants