DO NOT MERGE - test the converting-agent PR against all system-tests by ajgajg1134 · Pull Request #7339 · DataDog/system-tests

ajgajg1134 · 2026-07-17T17:36:00Z

Motivation

Changes

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-07-17T17:36:31Z

CODEOWNERS have been resolved as:

binaries/agent-image                                                    @DataDog/system-tests-core
run-all-jobs                                                            @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_async_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_inner_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_multiframe_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_recursion_3_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_recursion_5_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_rockpaperscissors_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/approvals/php/1.18.0/exception_replay_simple_spans_expected_v1.json  @DataDog/debugger @DataDog/system-tests-core
tests/debugger/test_debugger_exception_replay.py                        @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils.py                                                 @DataDog/debugger @DataDog/system-tests-core
tests/test_config_consistency.py                                        @DataDog/system-tests-core
utils/dd_types/_datadog_agent_trace.py                                  @DataDog/system-tests-core

datadog-official · 2026-07-17T17:37:00Z

✨ Unblock PR with BitsAI

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

DataDog/system-tests | mirror_images — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/system-tests | Ubuntu_20_amd64.CHA: [test-app-php] — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

Testing the test | System Tests (java, dev) / Build end-to-end (uds-spring-boot)

View more details · View in GitHub Actions

Non-resolvable parent POM for com.example:myproject:0.0.1-SNAPSHOT: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:2.6.0, status code: 429, reason phrase: Too Many Requests.

View all 4 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 32d7312 | Docs | View more details | Give us feedback!}

…de origin The trace agent's v1 conversion feature made three debugger test groups fail, all because the tests still assumed the legacy agent payload shape. Exception replay (java): a converted span carries a `_dd.convertedv1` provenance marker, and v0.4 `meta` values stay strings where a natively emitted v1 payload keeps real booleans. Drop the marker and canonicalize booleans on both sides of the approval comparison, so a single set of approval files stays valid whether or not conversion is enabled. Normalizing is preferred over re-recording the baselines, which would bake the current conversion path into them. Code origin (python): `_wait_for_code_origin_span` indexed the raw payload with legacy-only keys (`tracerPayloads`, `resource`, `type`, `meta`), so under v1 it never matched a span and timed out into a misleading "code origin not enabled" assertion. Iterate `interfaces.agent.get_spans()` instead, whose typed wrappers resolve those fields for both formats. Exception replay (php): the `*_spans_expected_v1.json` approvals were never generated, so the tests raised FileNotFoundError. Recorded them with DI_OVERRIDE_APPROVALS so they land in the existing 1.18.0 folder alongside the legacy files, rather than in a new version folder. Validated locally per language with the converting agent and with the default agent, confirming the payload format in each run via the captured agent interface files: java/spring-boot DEBUGGER_EXCEPTION_REPLAY 8 passed both ways python/flask-poc DEBUGGER_INPRODUCT_ENABLEMENT 5 passed both ways php/apache-mod-8.0 DEBUGGER_EXCEPTION_REPLAY 9 passed both ways Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Re-run CI after widespread SSI matrix and PHP e2e job failures.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e3f04c641

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-21T20:02:06Z

+    if isinstance(data, bool):
+        return "true" if data else "false"


Limit boolean normalization to span attributes

When a v1 payload incorrectly encodes a schema field such as the top-level error flag as the string "true", this recursive conversion changes the approval's Boolean and the received string to the same value, so the approval test passes despite the malformed span. The format difference described here applies to legacy meta values carried into v1 attributes; restrict canonicalization to those attributes so envelope and span-field types remain validated.

Useful? React with 👍 / 👎.

datadog-official

All seven new PHP v1 approval files keep _dd.p.tid, but the scrub code removes this tag before comparison. Each affected exception replay approval comparison therefore fails.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 2e3f04c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

…vertingAgent # Conflicts: # tests/debugger/test_debugger_exception_replay.py

_dd.p.tid was added to the scrub-removal set in test_debugger_exception_replay.py after these approval files were generated, so they still carried the key as "<scrubbed>" instead of having it removed, causing comparison failures.

The efficient trace payload format (idxTracerPayloads) merges meta and metrics into a single typed `attributes` map, so a tag a tracer reports both as a string tag and as a numeric metric collides, and the numeric value is the one that survives. The Rust tracer does exactly this for http.status_code, which made Test_Config_HttpServerErrorStatuses_Default fail against the converting agent with `assert 400.0 == '400'`, while every other library passed. Add DataDogAgentSpan.get_tag(), which normalizes integral numbers back to their string form for that format only. The legacy format keeps meta and metrics separate, so values are returned untouched there and a numeric value in meta remains a failure, as it should. Verified in replay mode against the rust/axum CI artifact: the two failing tests pass, the full DEFAULT scenario matches CI exactly (67 passed / 386 skipped / 43 xfailed / 3 xpassed) with no other change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cbeauchesne · 2026-09-02T08:42:30Z

@@ -0,0 +1 @@
+datadog/agent-dev:andrew-glaude-convert-by-default-on-55244-py3


You can achieve the same by adding [agent@andrew-glaude-convert-by-default-on-55244-py3] in your PR title.

### What does this PR do? Enables the "v1.0" trace payload conversion by default. `handleTraces` now routes through the V1 (idx) pipeline unless the new `disable-convert-traces` feature flag is set, inverting the previous opt-in `convert-traces` flag. **Note that much of this has been previously reviewed but was reverted due to unexpected breakages in system-tests which have since been resolved** OTLP receiver is left out to minimize changes and reduce review burden. Three latent bugs that the default flip exposes are fixed here: - `Get128BitTraceID` parsed all 32 hex characters of `otel.trace_id` as a `uint64`, which always overflowed. Only the first 16 hold the upper bits. Reachable on any span the datadog exporter sends to the normal HTTP endpoints, since `ConvertToIdx` now runs on that traffic. - `TraceWriterV1` dropped the `_dd.otel_gateway` payload tag that the legacy writer sets. - `idx.InternalSpan` had no `String()`, so a log line interpolating one rendered as `%!s(*idx.Span=&{...})`. The rest is test migration to the idx format: the trace-agent testsuite, new `idxTracePayload` helpers in the fakeintake client, and the apm, ssi, installer and agent-platform e2e suites. ### Motivation The conversion path has been opt-in behind `convert-traces` since #53339. This makes it the default so it gets real traffic, while keeping `disable-convert-traces` as the escape hatch. We will remove the disable flag once we have sufficient success after release and following from that will remove the old code paths entirely :) ### Describe how you validated your changes - `dda inv test --targets=./pkg/trace/api,./pkg/trace/agent,./pkg/trace/writer,./pkg/proto/pbgo/trace,./pkg/serverless/trace,./comp/trace/payload-modifier/impl,./cmd/serverless-init/trace` — 1300 pass, 5 skipped - `bazel build //cmd/trace-agent/test/testsuite:all //comp/otelcol/otlp/integrationtest:all //test/fakeintake/client:all //pkg/trace/api:all //pkg/trace/writer:all` — success - `bazel run //:gazelle` — no remaining BUILD.bazel churn - New unit tests for each fix: `span_utils_test.go` (32-hex `otel.trace_id`), `tracev1_test.go` (`otel_gateway` tag), `internal_span_test.go` (`String()`) - We've been running this enabled internally for quite some time without incident - I manually triggered the APM System-tests DataDog/system-tests#7339 and validated that all the system-tests pass. ### Additional Notes Co-authored-by: andrew.glaude <andrew.glaude@datadoghq.com>

cbeauchesne

Another not-well-known feature : the run-all-scenarios label. So you don't need to add a dummy file to trigger them.

TEMP force using the converting agent to run the full CI

b10d85c

ajgajg1134 and others added 10 commits July 21, 2026 08:50

add file to trigger all jobs

3ca30f1

Merge branch 'main' into andrew.glaude/TestConvertingAgent

9de4d13

Merge branch 'main' into andrew.glaude/TestConvertingAgent

151684b

ci: retrigger pipeline

e457539

Re-run CI after widespread SSI matrix and PHP e2e job failures.

Merge branch 'main' into andrew.glaude/TestConvertingAgent

459bc0b

Merge branch 'main' into andrew.glaude/TestConvertingAgent

8de6179

Merge branch 'main' into andrew.glaude/TestConvertingAgent

d4c79c6

Merge branch 'main' into andrew.glaude/TestConvertingAgent

b5d80e4

Merge branch 'main' into andrew.glaude/TestConvertingAgent

2e3f04c

ajgajg1134 marked this pull request as ready for review August 21, 2026 19:57

ajgajg1134 requested review from a team as code owners August 21, 2026 19:57

ajgajg1134 marked this pull request as draft August 21, 2026 19:57

chatgpt-codex-connector Bot reviewed Aug 21, 2026

View reviewed changes

datadog-official Bot reviewed Aug 21, 2026

View reviewed changes

Comment thread tests/debugger/utils/approvals/php/1.18.0/exception_replay_simple_spans_expected_v1.json Outdated

ajgajg1134 and others added 5 commits August 21, 2026 16:10

Merge remote-tracking branch 'origin/main' into andrew.glaude/TestCon…

facfd69

…vertingAgent # Conflicts: # tests/debugger/test_debugger_exception_replay.py

Merge branch 'main' into andrew.glaude/TestConvertingAgent

eecab75

new img to test

7e909fe

ajgajg1134 mentioned this pull request Sep 1, 2026

APM: Enable convert-traces feature by default DataDog/datadog-agent#55544

Merged

cbeauchesne reviewed Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

DO NOT MERGE - test the converting-agent PR against all system-tests - #7339

DO NOT MERGE - test the converting-agent PR against all system-tests#7339
ajgajg1134 wants to merge 16 commits into
mainfrom
andrew.glaude/TestConvertingAgent

ajgajg1134 commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Jul 17, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Uh oh!

Uh oh!

datadog-official Bot left a comment

Uh oh!

Uh oh!

cbeauchesne Sep 2, 2026

Uh oh!

ajgajg1134 Sep 2, 2026

Uh oh!

cbeauchesne left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if isinstance(data, bool):
		return "true" if data else "false"

		@@ -0,0 +1 @@
		datadog/agent-dev:andrew-glaude-convert-by-default-on-55244-py3 No newline at end of file

Uh oh!

Conversation

ajgajg1134 commented Jul 17, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jul 17, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cbeauchesne Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ajgajg1134 Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 17, 2026 •

edited

Loading

datadog-official Bot commented Jul 17, 2026 •

edited by datadog-prod-us1-3 Bot

Loading