Repository navigation
[MAUI] Breadcrumbs occasionally applying to event out of order #4139
Description
Tracking issue for:
Environment
SaaS (https://sentry.io/)
Steps to Reproduce
Filing this issue on behalf of a user in this Zendesk support ticket
SDK - dotnet MAUI 5.5.0
OS - iOS 17.1.1
The Breadcrumbs array seems to sometimes be out of order and is not sorted according to the timestamps of the breadcrumb. Here is an example from the user's event JSON -
Note how the breadcrumb I highlighted for device.connectivity is timestamped before the started Breadcrumb Tracking timestamp. This example here was captured by the Cocoa SDK.
Expected Result
They should be in timestamp order
Actual Result
They are not in timestamp order. This causes the UI in Sentry to display them out of order as well.
Product Area
Unknown
Link
No response
DSN
No response
Version
No response
Activity
bruno-garcia commentedon Apr 24, 2025
Is there an example event? Curious if this is coming from a native SDK, such as
sentry.cocoa.dotnetorsentry.java.android.dotnet.In which case my be related to how we bind the .NET breadcrumb scope to native.
bruno-garcia commentedon Apr 24, 2025
@Fwang36 said:
bruno-garcia commentedon Apr 24, 2025
Reading the code in the cocoa SDK, it comes from here:
https://github.com/getsentry/sentry-cocoa/blob/bfcfc62c15b1531674c50fb306b57fbc7fbf776f/Sources/Sentry/SentryBreadcrumbTracker.m#L158-L162
This specific breadcrumb doesn't come from the .NET SDK itself though: https://github.com/search?q=repo%3Agetsentry%2Fsentry-dotnet%20device.connectivity&type=code
So it seems to me it could be an issue in the cocoa SDK. (they are on sentry.cocoa: 8.39.0)
@philipphofmann are you aware of any reason why this could happen?
bruno-garcia commentedon Apr 24, 2025
Note, the Breadcrumb tracking crumb's timestamp is actually after the 2nd and 3d
device.connectivityones:philippsuess commentedon Apr 25, 2025
These breadcrumbs stem from the SentryReachability. The SentryReachability posts connectivity changes on a BG thread, which uses a serial dispatch queue, meaning that if such chance occurs shortly after each other the breadcrumbs should be in order, but there could be an edge case we're missing of course, but I didn't find one after a quick investigation. I'm actually surprised that the Sentry UI doesn't automatically sort these by timestamp.
philippsuess commentedon Apr 25, 2025
A positive side effect of investigating this issue was finding slightly blocking the main thread on init in the HttpTransport of the Cocoa SDK getsentry/sentry-cocoa#5138.
bruno-garcia commentedon Apr 25, 2025
I'm aware Sentry doesn't order by timestamp by default.
I have mixed feelings about doing that, because in cases where the device travels to a new timezone, things would become incorrect and misleading.
I find that in its current form, at least it lets us find areas where the order doesn't match the timestamp and we can assess what we want to do.
jamescrosswell commentedon Apr 27, 2025
Isn't that why we have UTC? If all events were recorded using UTC timestamps, they could be sorted automatically by the UI right?
philippsuess commentedon Apr 28, 2025
UTC timestamps would cover traveling to a new time zone, but not when the device adjusts its time, which all devices have to do regularly. I'm not sure how often that happens, but I know it's tough to build a watch that's less than a second off after 24 hours. So, my guess is that it happens a few times a day. If that occurs between two breadcrumbs and the adjusted time is 100 ms, it could already mess things up. What the user posted up there could actually be due to what I described.
The solution to this problem is to use a combination of wall time and UTC time. Basically, we would need to use wall time for breadcrumbs and then somehow map that to UTC time when sending them. Doing that properly isn't trivial.
philippsuess commentedon Apr 28, 2025
Another idea would be to subscribe to https://developer.apple.com/documentation/uikit/uiapplicationdelegate/applicationsignificanttimechange(_:) and add a breadcrumb.
jamescrosswell commentedon Apr 29, 2025
Even then though, what do we do with that information? Let's say the clock on the device was changed from 13:07:64 to 13:06:94... so basically shifted back 60 ms. How would we use that information to change the timestamps for old or new breadcrumbs? It seems like a bit of a rabbit hole.
A possible alternative (not saying it's a good idea) would be to record the time the process started (UTC) and then start a timer. The timestamp used on breadcrumbs could be start_time + elapsed_time. That way all of your breadcrumbs would be sequential and, presumably, as accurate as the system clock was when the process was started.
It seems like quite a lot of effort (especially making this change across all the SDKs)... but theoretically it could work.
philippsuess commentedon Apr 29, 2025
Another idea would be to have an extra field for sending the wall time alongside the UTC timestamp. We can then take appropriate action in the UI or on the backend. We could do recalculations or just show in the UI that the UTC time shifted.
jamescrosswell commentedon Apr 30, 2025
I don't have any strong opinions. I guess whatever we do it makes sense to pilot it in the Cocoa SDK (where we're seeing this behaviour)? Should I create an issue in the Cocoa repo and link to it from here?
philippsuess commentedon May 2, 2025
I created an issue for the time change notification getsentry/sentry-cocoa#5161. The topic with CPU time is a bigger one that would also require changes to the product and backend. That's not something we can ship quickly.
jamescrosswell commentedon May 4, 2025
OK, I'll leave this issue in the .NET SDK as a tracking issue until getsentry/sentry-cocoa#5161 is completed then. Thanks @philipphofmann !
alexsohn1126 commentedon Sep 15, 2025
getsentry/sentry-cocoa#5161 was merged and is included in the 8.56.0 release which was just created!
Thanks again @philipphofmann!