ActivitySource copies creation tags and links into activities sampled as PropagationData

[martincostello]

Description

When listeners sample an activity as ActivitySamplingResult.PropagationData, ActivitySource.CreateActivity() and ActivitySource.StartActivity() still copy the tags and links passed at creation into the new Activity. The documentation for PropagationData says:

The Activity object needs to be created. It will have Name, Source, Id and Baggage. Other properties such as Links, Tags, and Events are unnecessary and will be ignored by this listener.

This is also supported by the discussion in #91070.

Thus it seems like this is redundant work and allocations - with 9 creation tags, each activity allocates twice as much memory (848 B instead of 416 B) and takes about two-thirds longer to create, start and stop (215 ns instead of 130 ns on .NET 11 RC1).

This affects services that sample their traces - the OpenTelemetry .NET SDK returns PropagationData when its sampler drops a root span or a span with a remote parent, so that the trace ID is still propagated (PropagateOrIgnoreData), which is the case for every incoming request that is sampled out. Server instrumentations pass their tags at creation so that samplers can use them. ASP.NET Core passes up to 11 for each HTTP request (CreateInitializeActivityTags).

With sampling configured at 10%, 90% of requests pay for the overhead of copying tags that aren't used.

Note

I was assisted in creating this issue by Claude Code.

Reproduction Steps

Save the following as repro.cs and run dotnet run repro.cs with the .NET 10 SDK or later.

#!/usr/bin/env dotnet
#:property PublishAot=false
#:property Optimize=true

// ActivitySource.CreateActivity copies the creation tags and links into activities that a listener sampled as
// ActivitySamplingResult.PropagationData, although the documentation says those properties are unnecessary.
//
//   dotnet run repro.cs

using System.Diagnostics;
using System.Runtime.InteropServices;

Console.WriteLine($"{RuntimeInformation.FrameworkDescription} ({RuntimeInformation.ProcessArchitecture})");
Console.WriteLine();

using var source = new ActivitySource("Repro");
var samplingResult = ActivitySamplingResult.PropagationData;

using var listener = new ActivityListener
{
    ShouldListenTo = s => ReferenceEquals(s, source),
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => samplingResult,
};
ActivitySource.AddActivityListener(listener);

// The tags an HTTP server instrumentation provides at creation so that samplers can use them.
KeyValuePair<string, object?>[] tags =
[
    new("client.address", "192.0.2.1"),
    new("network.peer.address", "192.0.2.1"),
    new("network.peer.port", 50000),
    new("server.address", "localhost"),
    new("server.port", 5000),
    new("http.request.method", "GET"),
    new("user_agent.original", "Mozilla/5.0"),
    new("url.scheme", "https"),
    new("url.path", "/api/items"),
];

ActivityLink[] links = [new(new ActivityContext(ActivityTraceId.CreateRandom(), ActivitySpanId.CreateRandom(), ActivityTraceFlags.None))];

using (var activity = source.StartActivity("Request", ActivityKind.Server, parentContext: default, tags, links))
{
    Console.WriteLine($"PropagationData: IsAllDataRequested={activity!.IsAllDataRequested}, tags={activity.TagObjects.Count()}, links={activity.Links.Count()}");
}

Console.WriteLine();
Console.WriteLine($"{"Sampling result",-20} {"Tags",4} {"Time (ns)",10} {"Allocated (B)",14}");

foreach (var result in new[] { ActivitySamplingResult.PropagationData, ActivitySamplingResult.AllDataAndRecorded })
{
    samplingResult = result;

    foreach (var creationTags in new[] { null, tags })
    {
        var (ns, bytes) = Measure(() =>
        {
            var activity = source.CreateActivity("Request", ActivityKind.Server, parentContext: default, creationTags);
            activity!.Start();
            activity.Stop();
        });

        Console.WriteLine($"{result,-20} {creationTags?.Length ?? 0,4} {ns,10:F1} {bytes,14:F0}");
    }
}

static (double Ns, double Bytes) Measure(Action action)
{
    const int Iterations = 1_000_000;

    // Run long enough for tiered compilation to optimize the code being measured.
    var warmup = Stopwatch.StartNew();
    while (warmup.Elapsed < TimeSpan.FromSeconds(2))
    {
        action();
    }

    var bestNs = double.MaxValue;
    var bytes = 0.0;

    for (var run = 0; run < 5; run++)
    {
        var allocated = GC.GetAllocatedBytesForCurrentThread();
        var start = Stopwatch.GetTimestamp();

        for (var i = 0; i < Iterations; i++)
        {
            action();
        }

        bestNs = Math.Min(bestNs, Stopwatch.GetElapsedTime(start).TotalNanoseconds / Iterations);
        bytes = (GC.GetAllocatedBytesForCurrentThread() - allocated) / (double)Iterations;
    }

    return (bestNs, bytes);
}

Expected behavior

An activity sampled as PropagationData doesn't copy the creation tags or links, so it costs the same as an activity created without them (416 B here). Samplers still see the tags and links through ActivityCreationOptions<T>.Tags and ActivityCreationOptions<T>.Links.

Actual behavior

The activity keeps the tags and links even though IsAllDataRequested is false:

PropagationData: IsAllDataRequested=False, tags=9, links=1

The time and allocations for PropagationData are within noise the same as for AllDataAndRecorded. Time and allocations to create, start and stop an activity, with the process pinned to one core:

Sampling result	Tags	.NET 10.0.12	.NET 11.0.0-rc.1.26425.128
PropagationData	0	141.9 ns, 416 B	129.5 ns, 416 B
PropagationData	9	237.9 ns, 848 B	215.4 ns, 848 B
AllDataAndRecorded	0	137.1 ns, 416 B	117.5 ns, 416 B
AllDataAndRecorded	9	242.6 ns, 848 B	210.6 ns, 848 B

Regression?

No - ActivitySource has copied the creation tags and links regardless of the sampling result since it was added in .NET 5 by #35220.

Known Workarounds

None that keep tags available to samplers.

Configuration

• .NET 10.0.12 and .NET 11.0.0-rc.1.26425.128. The code is the same on main (514e6a8).
• Windows 11 Pro 10.0.26200, x64, 13th Gen Intel Core i7-13700H.

Other information

Activity.Create copies the links and the tags before it uses the sampling result to set IsAllDataRequested. ActivitySource calls it for any sampling result other than None.

The likely fix would be for Activity.Create to skip copying tags, samplerTags and links when request is PropagationData.

Assuming the current behaviour is an oversight and not intentional, I'd be happy to open a PR to make the changes.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @dotnet/area-system-diagnostics-tracing
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-activity
See info in area-owners.md if you want to be subscribed.