dotnet-dsrouter not connecting on iOS - "Failed connecting 9191 over usbmux, wait 500 ms before retrying."

[kcoop]

I'm attempting to create a .nettrace file to track down memory leaks in a Maui iOS app running on an iPhone tethered via USB to a Mac, but haven't yet been able to capture any telemetry with dotnet-trace and dotnet-dsrouter.

Following the instructions here:

https://github.com/xamarin/xamarin-android/blob/main/Documentation/guides/tracing.md

Install dotnet-dsrouter and dotnet-trace:

$ dotnet-dsrouter --version
WARNING: dotnet-dsrouter is a development tool not intended for production environments.

7.0.435501+b56bae8476fb1a944de6618b4aa2b602db05b8b3

$ dotnet trace --version
7.0.430602+c9c987585e1fa9a8a8a5e1ffb60a70f85b0da4ca

Launch the app:

$ mlaunch --launchdev MyApp.app --wait-for-exit --argument --connection-mode --argument none '--setenv:DOTNET_DiagnosticPorts=127.0.0.1:9191,suspend,listen'

It starts up and waits as expected:

Press enter to terminate the application
>The runtime has been configured to pause during startup and is awaiting a Diagnostics IPC ResumeStartup command from a Diagnostic Port.
DOTNET_DiagnosticPorts="127.0.0.1:9191,suspend,listen"
DOTNET_DefaultDiagnosticPortSuspend=0

Start dotnet-dsrouter:

$ dotnet-dsrouter server-client -ipcs /tmp/maui-app-port -tcpc 127.0.0.1:9191 --forward-port iOS -v trace -rt 100

It seems healthy at this point too:

info: dotnet-dsrouter[0]
      Starting IPC server (/tmp/maui-app-port) <--> TCP client (127.0.0.1:9191) router.
trce: dotnet-dsrouter[0]
      Calling AMDeviceNotificationSubscribe.
dbug: dotnet-dsrouter[0]
      Trying to create a new router instance.
dbug: dotnet-dsrouter[0]
      Waiting for new ipc connection at endpoint "/tmp/maui-app-port".
trce: dotnet-dsrouter[0]
      Start dispatching notifications.
trce: dotnet-dsrouter[0]
      AMDeviceNotificationInternal callback, device=5274562304, action=Connected
info: dotnet-dsrouter[0]
      Successfully connected new device, id=4.

Start dotnet trace to get a dump (unlike the tracing instructions I added ',connect' to the port, omitting it throws an Address already in use exception):

$ dotnet-trace collect --providers Microsoft-DotNETRuntimeMonoProfiler:0xC900003:4 --diagnostic-port /tmp/maui-app-port,connect

At this point, the dotnet-trace just sits, and these flow by in the dotnet-dsrouter output until I quit:

trce: dotnet-dsrouter[0]
      Failed connecting 9191 over usbmux, wait 500 ms before retrying.

Which looks to be emanating from here:

diagnostics/src/Tools/dotnet-dsrouter/USBMuxTcpClientRouterFactory.cs

Line 343 in f1de6c2

_logger?.LogTrace($"Failed connecting {_port} over usbmux, wait {TcpClientRetryTimeoutMs} ms before retrying.");

I have seen #3134 by @rolfbjarne which sounded possibly relevant, but we are on a much later version, so I imagine that fix is already in 7.0.435501.

Any ideas on how to get this working? It would really help see this data.

[hoyosjs]

cc: @lateralusX

[lateralusX]

Sounds like something has broken on later iOS SDK's related to usbmux. Will need to debug this a little deeper to see if I can reproduce the issue on my end. You should be able to use iossumulator instead of device if that could be a way forward.

[kcoop]

Timely, I was just writing this, struck me as really odd that it referred to usbmux? Could that be a clue?

Oddly, I've now also repro'd this against a simulator:

mlaunch --launchsim MyApp.app --device :v2:udid=BCBF3C73-A566-44CB-96AB-D537EF55DD58 --wait-for-exit --stdout=$(tty) --stderr=$(tty) --argument --connection-mode --argument none '--setenv:DOTNET_DiagnosticPorts=127.0.0.1:9000,suspend,listen'

dotnet-dsrouter[0]
      Failed connecting 9000 over usbmux, wait 500 ms before retrying.

[lateralusX]

When running against iOS simulator you should not use port forwarding, see https://github.com/dotnet/runtime/blob/main/docs/design/mono/diagnostics-tracing.md#application-running-diagnostics-and-connecting-over-loopback-interface for examples on how to run against iOS simulator. You can also find docs from iOS team doing the same thing when using iOS SDK here, https://github.com/xamarin/xamarin-macios/wiki/Profiling.

[kcoop]

Ah, RTFM indeed, missed that. Thank you very much! I'm up and running now.

[lateralusX]

Did you still have issues with device over usbmux or did you get that working as well?

[kcoop]

Yes, but I can work with the simulator.

[lateralusX]

OK, good, when I finished my current work item I will see if I can track down what change around usbmux, but that won't be until next week.

[kcoop]

Great. Appreciate the quick response.

[mdh1418]

Hi @kcoop, @lateralusX improved some of the experiences surrounding dotnet-dsrouter (including some default scenarios for iOS where now dotnet-dsrouter ios will automatically setup IPC server-TCP client (127.0.0.1:9000) #4090, would you mind trying out the updated versions of diagnostics tooling again and see if you're still hitting issues?

I was able to run through a diagnostics scenario using my iPad building on Mac, building a MAUI iOS sample using dotnet version 8.0.300-preview.24107.20 and diagnostics tooling version 8.0.510801 (though I think some older versions would work as well, this was just what I built locally).

---

Scenario that I was able to get working

Building the MAUI iOS app

dotnet new maui -o sample
Setting <RuntimeIdentifier>ios-arm64</RuntimeIdentifier> in the csproj
dotnet build -f net8.0-ios -c Debug (debug configuration should set _BundleDebug to true in the sdk https://github.com/xamarin/xamarin-macios/blob/b0c3ae5fec1e69f036d5abb243e393fdb2c72075/dotnet/targets/Xamarin.Shared.Sdk.targets#L447

Spawning dotnet-dsrouter in another terminal window

dotnet-dsrouter ios -v trace -i (-v trace and -i will output more diagnostics about whats happening)

Install the App on iPad with DOTNET_DiagnosticPorts and run

Opened XCode > Debug > Debug Executable > (app built earlier)
Set scheme Environment Variable (Product > Scheme > Edit Scheme)

Ran the app through xcode

Run dotnet diagnostics tool

dotnet-trace collect -p <pid logged by dotnet-dsrouter>

[kcoop]

Thanks, @mdh1418, appreciate the follow through. I'm unfortunately no longer working on that project (or with Maui), so can't provide you with any feedback at this point.

[mdh1418]

Thanks for trying out the tools! Will close this issue as it may have been fixed. If another repro occurs, we can open this up again.