Skip to content

build(native): align libdatadog and PyO3 dependencies - #21618

Open
hhan2024 wants to merge 1 commit into
helen/dependent-flags-python-rcfrom
helen/python-native-upgrade-prerequisite
Open

hhan2024 wants to merge 1 commit into
helen/dependent-flags-python-rcfrom
helen/python-native-upgrade-prerequisite

Conversation

@hhan2024

@hhan2024 hhan2024 commented Nov 9, 2026

Copy link
Copy Markdown

Description

Updates the shared libdatadog revision and aligns PyO3, its build dependencies, and the minimum source-build Rust version. Adapts trace-model integration and Python extraction annotations to the new APIs while preserving the existing Python-facing native interfaces.

The FFE binding continues to advertise no dependent-flag capability. Capability activation, prerequisite telemetry, and exposure finalization belong to the feature branch layered above this prerequisite.

Testing

  • Native formatting, all-features Clippy, all 31 Rust tests, and repository lint checks pass.
  • An optimized CPython 3.11 Linux x86_64 release wheel builds successfully.
  • An isolated installation of that wheel passes native imports, independent FFE resolution, trace encoding with Unicode attributes, and crashtracker binding checks. The FFE capability reporter remains empty.

Risks

This revision changes shared native dependencies used by tracing, profiling, crash tracking, and Remote Configuration. It requires their supported-platform CI coverage and an approved native release before merge. Source builds require Rust 1.91.1 or later.

Adapt the shared trace model and Python wrappers to the upgraded native baseline while keeping dependent-flag capability advertisement disabled.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
@hhan2024
hhan2024 marked this pull request as ready for review November 9, 2026 20:38
@hhan2024
hhan2024 requested review from a team as code owners November 9, 2026 20:38
@hhan2024
hhan2024 requested review from brettlangdon and gyuheon0h and removed request for a team November 9, 2026 20:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T20:43:26.981606Z d345b05 Draft marked ready
🔒 Security Review ✅ Completed 2026-11-09T20:42:21.455969Z d345b05 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@pr-commenter

pr-commenter Bot commented Nov 9, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-11-09 20:41:44

Comparing candidate commit d345b05 in PR branch helen/python-native-upgrade-prerequisite with baseline commit 12e0e8c in branch helen/dependent-flags-python-rc.

📊 Benchmarking dashboard

Found 13 performance improvements and 0 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:coreapiscenario-core_dispatch_exception_no_listeners

  • 🟩 execution_time [-44.432ns; -36.583ns] or [-19.774%; -16.281%]

scenario:coreapiscenario-core_dispatch_no_args_no_listeners

  • 🟩 execution_time [-46.028ns; -36.538ns] or [-20.315%; -16.126%]

scenario:coreapiscenario-core_dispatch_no_listeners

  • 🟩 execution_time [-46.082ns; -36.024ns] or [-20.396%; -15.945%]

scenario:coreapiscenario-core_dispatch_with_results_no_listeners

  • 🟩 execution_time [-42.206ns; -34.655ns] or [-19.180%; -15.749%]

scenario:coreapiscenario-has_listeners_no_listeners

  • 🟩 execution_time [-45.663ns; -37.430ns] or [-21.675%; -17.767%]

scenario:coreapiscenario-has_listeners_with_listeners

  • 🟩 execution_time [-49.044ns; -40.169ns] or [-21.812%; -17.865%]

scenario:httppropagationextract-empty_headers

  • 🟩 execution_time [-116.494ns; -105.204ns] or [-15.079%; -13.617%]

scenario:httppropagationextract-wsgi_empty_headers

  • 🟩 execution_time [-108.631ns; -100.320ns] or [-14.151%; -13.069%]

scenario:httppropagationextract-wsgi_invalid_span_id_header

  • 🟩 execution_time [-112.164ns; -97.156ns] or [-14.530%; -12.586%]

scenario:msgpackencoderscenario-simple_one_span

  • 🟩 execution_time [-361.828ns; -292.470ns] or [-8.861%; -7.163%]

scenario:otelspan-add-tags

  • 🟩 execution_time [-12.394ms; -11.421ms] or [-8.901%; -8.202%]

scenario:rand-rand128bits

  • 🟩 execution_time [-22.386ns; -19.678ns] or [-11.504%; -10.112%]

scenario:rand-rand64bits

  • 🟩 execution_time [-15.039ns; -13.210ns] or [-12.268%; -10.776%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-766.393ns; +658.397ns] or [-7.158%; +6.150%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-72.957ns; -2.891ns] or [-10.609%; -0.420%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1883.711ns; +1810.022ns] or [-9.416%; +9.048%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-2331.243ns; +1248.487ns] or [-11.826%; +6.333%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-422.020ns; +299.075ns] or [-9.750%; +6.909%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-252.676ns; +187.021ns] or [-9.326%; +6.903%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-179.599ns; -6.307ns] or [-12.635%; -0.444%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-6.079µs; +3.146µs] or [-12.082%; +6.253%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-1327.412ns; +542.904ns] or [-12.433%; +5.085%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-62.835µs; +48.672µs] or [-10.665%; +8.261%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:packagespackageforrootmodulemapping-cache_off

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d345b05922

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/native/py_string.rs
Comment on lines +359 to +360
} else {
Self::from_box(Box::<str>::from(self.deref()))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Benchmark the new native string cloning path

When trace-model cloning is exercised for Python-backed span text, this new Clone implementation allocates and copies every non-static value, potentially affecting hot trace/event processing. The commit provides functional tests but no benchmark results for this native runtime change, so please measure representative Python-backed string workloads and report the performance impact before merging.

AGENTS.md reference: AGENTS.md:L16-L18

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code, working with Mark:

The same Clone implementation has been on main since #21576. This PR's benchmark run shows no regressions (13 improvements, 0 regressions; msgpack span encoding −7% to −9%). libdatadog's stats path copies &str with to_owned/to_string rather than calling TraceText::clone.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The dependency pins are consistent across the manifest and lockfile, and FFE continues to advertise no dependent-flag capability. Supported-platform build and runtime compatibility remain unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit d345b05

@mark-caii mark-caii left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code, working with Mark:

CI is green at d345b05. The main blocker is the libdatadog pin (see the inline comment on src/native/Cargo.toml). Other notes:

[P2] Much of this PR overlaps with work that main already has. #21576 (libdatadog v48.0.0, merged Nov 6, not in a release yet) has the same py_string.rs changes (Clone, From<String>, TraceText, ValueTypes), the same RUST_MINIMUM_VERSION in setup.py, and parts of the Cargo.toml and Cargo.lock changes. Four files will conflict when the stack reaches main. Suggestion: merge main into the stack first and keep the py_string.rs from main, because its Clone comment is more complete. This PR then shrinks to the libdatadog bump, PyO3 0.30, the #[pyclass] cleanup, and the ffe.rs doc comment.

[P2] The release note releasenotes/notes/native-toolchain-upgrade-08a2e3fdd11cbcc4.yaml can land in the wrong release. main raised the Rust minimum to 1.91.1 in #21576 without a release note, and that change is not in a release tag yet. Suggestion: add the note on main and drop it here, unless this stack lands before the next release.

[P3] Stale test comment. tests/openfeature/test_agentless_poller.py:259-260 is outside this PR's diff, so this note is here and not inline. The comment says "The native dependency does not implement prerequisite evaluation yet." The linked evaluator now evaluates prerequisites; only the binding does not surface them. #21346 removes this test, so this matters only if this PR lands without #21346. In that case, suggestion: reuse the updated wording from src/native/ffe.rs: "Prerequisite records and exposure eligibility are not yet surfaced by this binding, so transports must not advertise support."

Heads up: with this evaluator, dependent-flag configs change behaviour even when the capability is not advertised.

  • Before (libdatadog v46.1.0): a condition with flagEvaluation and no attribute failed to parse. The binding returned ErrorCode.ParseError with do_log=False, and the provider returned the caller's default value.
  • Now: the prerequisites are evaluated, and Python gets the root flag's assignment and its do_log. Prerequisite exposures are dropped, because get_assignment discards the prerequisite records. A failed prerequisite maps to ErrorCode.General.

This matters only if the backend sends dependent flags to clients that do not advertise support. Please confirm that the backend does not do this before #21346 advertises the capability.

Comment thread src/native/Cargo.toml
libdd-live-debugger = { git = "https://github.com/DataDog/libdatadog", rev = "v46.1.0" }
libdd-remote-config = { git = "https://github.com/DataDog/libdatadog", rev = "v46.1.0", features = [
# Keep libdatadog crates on one revision to share native types.
libdd-ffe = { git = "https://github.com/DataDog/libdatadog", rev = "cb3bfba0d51adba06f9afca32841573fa7bad0b3", optional = true, features = ["pyo3"] }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code, working with Mark:

[P0, before merge] Every libdatadog rev in this file is pinned to cb3bfba0d5. This covers lines 38-91, including build_common and the Linux-only target dependencies. Please re-pin all of them to a libdatadog release tag before merge.

The PR body already says that an approved native release is needed before merge. This comment makes the crash risk at the current pin concrete.

Fix: after DataDog/libdatadog#2748 is merged and released, set every libdatadog dependency to that tag. Then regenerate Cargo.lock and name the tag in the setup.py comment.

Comment thread setup.py
# DEV: update this accordingly when src/native upgrades libdatadog dependency.
# libdatadog v46.1.0 requires rust 1.87.0.
RUST_MINIMUM_VERSION = "1.87.0"
# The mainline libdatadog dependency requires Rust 1.91.1.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code, working with Mark:

[P3] This comment says "The mainline libdatadog dependency requires Rust 1.91.1", but the pin cb3bfba0d5 is not on libdatadog main. main in this repo already uses the release form of this line, from #21576:

# libdatadog v48.0.0 requires rust 1.91.1.

When you re-pin to a release, please keep that form and set the version to the tag that you pin.

Comment thread src/native/py_string.rs
Comment on lines +359 to +360
} else {
Self::from_box(Box::<str>::from(self.deref()))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code, working with Mark:

The same Clone implementation has been on main since #21576. This PR's benchmark run shows no regressions (13 improvements, 0 regressions; msgpack span encoding −7% to −9%). libdatadog's stats path copies &str with to_owned/to_string rather than calling TraceText::clone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants