Repository navigation
Address memory consumption when Tracing is enabled with OTel #3550
Description
We've had multiple users report either memory leaks or high memory consumption when tracing is enabled in the SDK. Conversations with users exchanging logs etc. has not led to a successful resolution of the problems and none of the SDK maintainers has been able to reproduce a scenario that results in a memory leak.
We really need some way to be able to reproduce the problem to be able to investigate and find a solution.
Related Issues
Metadata
Metadata
Assignees
Projects
- Status
Activity
ericsampson commentedon Aug 22, 2024
Maybe Julian from the other issue would be willing to do some experimenting/data gathering, since it seems like they have a good repro situation…
jamescrosswell commentedon Aug 22, 2024
cc: @Julian-Robinson @cliedeman
jamescrosswell commentedon Nov 25, 2024
Closing this as we're unable to reproduce. If anyone is able to provide a minimal reproducible example, we can reopen and look into this again.
13 remaining items
danpaul88 commentedon May 4, 2026
Looks like
dotMemoryis now able to open the dump files from thenet10.0containers, so I can more clearly see what's going on again.Using the following versions of Sentry and OpenTelemetry
We still see the memory usage issue. I've taken a memory dump from one of our problematic pods, which currently reports about 1.6gb of memory consumption.
However, interestingly,
dotMemoryshows that only ~600mb of this is actually used bydotnet... around 90% of which is retained by theSentrySpanProcessorA lot of the memory usage seems to stem from very high fragmentation, such as this example of a 33mb pinned object heap (one of several pinned object heaps) with only 4% utilisation
EDIT: Correction - it's actually in a Large Object Heap, which makes sense. I think DATAS affects how
dotMemorysees the LOH and causes it to lump it in with the POH.Which just contains a concurrent dictionary of
ActivitySpanIdtoISpanmappingsBecause this object is pinned it's holding the entire 33mb pinned object heap in memory, when it could otherwise be released back to the OS - this seems to be at least one of the causes of the large amount of memory consumption I'm seeing on our pods.
There is definitely a lack of heap compaction occurring on the pod I've sampled since there are around 100 Gen2 heaps of 4mb each, which is definitely contributing to the total memory usage, but I still wouldn't expect to see over 550mb of memory being directly retained by the
SentrySpanProcessordanpaul88 commentedon May 4, 2026
Looking at an example of one of the retained spans, I see it has zero spans in
_spans, started at2026-04-30T05:29:32.691Z(based on the backing field forStartTimestamp) and was for anMicrosoft.AspNetCore.Hosting.HttpRequestInactivityGiven my memory dump was taken on
2026-05-04, it seems quite unlikely that an activity that started 4 days ago is still ongoing with 0 child spans - I wonder if somehow spans are not being cleaned up with the associated activity ends, perhaps due to sampling decisions made byOpenTelemetry? It would be odd that we don't observe the same behaviour across all of our clusters if that is the case thoughLooking in
_contextsI see thatIsSampledappears to have no value, but I'm not sure enough about the internal data structures inSentryto know if this is relevant or notLet me know if there's anything specific I can dig out of the memory dump that could help
danpaul88 commentedon May 4, 2026
I'm wondering if it might be related to;
#3198
This works around an OTel bug whereby
OnEndisn't called if a span is filtered byOpenTelemetry.Instrumentation.AspNetCore.AspNetCoreTraceInstrumentationOptions.FilterLooking at the source for
PruneFilteredSpans(which that PR introduced), it looks like it tries to get theActivityassociated with the span via a weak reference and, if that activity is{Recorded: false, IsAllDataRequested: false}, it prunes it from the map.What it doesn't seem to handle, if I'm reading it right, is the case where the associated
Activityhas already been GC'd beforePruneFilteredSpansnext gets called. SincePruningIntervalis 5 seconds a filtered activity lasting less than 5 seconds could easily have already been GC'd before the next pruning occurs. In this case the weak reference cannot be resolved, can never match the condition{Recorded: false, IsAllDataRequested: false}, and thus can never get prunedI may be misunderstanding what
GetFused<Activity>is doing mind you@jamescrosswell what do you think?
jamescrosswell commentedon May 4, 2026
@danpaul88 that sounds possible yeah... the condition on this line basically:
sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs
Line 305 in 55c220b
... will only match if the associated
Activity is not null.We could change that to:
That way it would filter and remove any Activities that had already been cleaned up by the GC.
I think we'd need to update this logic as well:
sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs
Line 143 in 55c220b
That one is less about memory consumption and more about the accuracy of the visual representation in the trace, but still.
linear-code commentedon May 4, 2026
NET-670
jamescrosswell commentedon May 5, 2026
@danpaul88 fix coming in the 6.5.0 release.
Flash0ver commentedon May 5, 2026
Thank you very much, @danpaul88, for your detailed analysis!
[-]Address memory consumption when Tracing is enabled **Cannot reproduce***[/-][+]Address memory consumption when Tracing is enabled with OTel[/+]danpaul88 commentedon May 5, 2026
@jamescrosswell fantastic - thanks for the quick fix. I'll get our services updated once it's out and will report back to confirm
@Flash0ver no problem at all 👍 good to get to the bottom of it
Flash0ver commentedon May 5, 2026
@danpaul88
Sentry@6.5.0https://www.nuget.org/packages/Sentry/6.5.0Sentry.OpenTelemetry@6.5.0https://www.nuget.org/packages/Sentry.OpenTelemetry/6.5.0Sentry.OpenTelemetry.Exporter@6.5.0https://www.nuget.org/packages/Sentry.OpenTelemetry.Exporter/6.5.0