Repository navigation
Mono crash in System.Net.Requests tests on linux-arm64 #74667
Description
It appears that there is some trouble trying to reap child processes and that leads to a crash. Could it be possible we are timing out waiting for a request to finish?
Example log
Process terminated.
Error while reaping child. errno = 10
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
Id Target Id Frame
* 1 Thread 0x7f92ba5ff0 (LWP 25) "dotnet" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
2 Thread 0x7f91bff1c0 (LWP 26) "SGen worker" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f926008c8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3 Thread 0x7f8fe0f1c0 (LWP 27) "dotnet" 0x0000007f92841ef8 in __GI___poll (fds=0x7f88003af0, nfds=547913473925, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
4 Thread 0x7f8fc0e1c0 (LWP 28) "Finalizer" 0x0000007f92b69a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f925f19c0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
5 Thread 0x7f8ccd51c0 (LWP 29) "dotnet" 0x0000007f92b6bd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f8ccd22b0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
Lot of other threads are waiting similar to
Thread 14 (Thread 0x7f76bf91c0 (LWP 40)):
#0 0x0000007f92b675bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f00000000, expected=0, futex_word=0x7f5c00547c) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1 __pthread_cond_wait_common (abstime=0x7f76bf3fc8, mutex=0x7f5c005420, cond=0x7f5c005450) at pthread_cond_wait.c:533
#2 __pthread_cond_timedwait (cond=0x7f5c005450, mutex=0x7f5c005420, abstime=0x7f76bf3fc8) at pthread_cond_wait.c:667
#3 0x0000007f8f768490 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f5c005420, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4 0x0000007f8c1f958c in ?? ()
#5 0x0000007f8cbc52c8 in ?? ()
#6 0x0000007f8fe7b1a0 in ?? ()
#7 0xf90047bda907f3bb in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
{ "ErrorMessage":"Error while reaping child. errno = 10" } Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 0 | 0 |
Activity
Skip System.Net.Requests tests on mono linux-arm64
Skip System.Net.Requests tests on mono linux-arm64 (#74668)
wfurt commentedon Aug 30, 2022
this is not mono specific (even if mono failures seem prevalent) see #74795
steveisok commentedon Aug 30, 2022
Thanks! I wondered but couldn't find any evidence.
wfurt commentedon Aug 30, 2022
is there some way how to run the tests with Mono and dump the managed objects @steveisok? I would like to see if this is happening during some particular test....
steveisok commentedon Aug 30, 2022
I'm not sure. @akoeplinger would you happen to know?
akoeplinger commentedon Aug 31, 2022
If you have the crash in lldb/gdb then https://www.mono-project.com/docs/debug+profile/debug/#debugging-with-gdb might help, maybe @BrzVlad knows a better way.
The only other quick suggestion I have to pinpoint the exact test is to write a custom xunit v2 assembly runner which logs when a test starts by subscribing to
runner.OnTestStartingand setting theparallelparameter to false in therunner.Start()method: https://github.com/xunit/samples.xunit/blob/main/TestRunner/Program.cswfurt commentedon Aug 31, 2022
We could disable parallelization as workaround for now. Any thoughts on that @karelz? Its seems better than disabling the tests....
karelz commentedon Sep 1, 2022
I think this is showing in CI now quite a bit (I am behind on Test Monitor duty), in which case it is blocking CI and therefore any action to make it not a problem is good.
We can go after the root cause in parallel.
System.Net.Requests: Pick correct RemoteExecutor overload in test (#7…
akoeplinger commentedon Sep 5, 2022
Turns out this is still happening, will re-disable the tests and let's continue investigating in #74795 since it's the same issue