Skip to content

May 9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6 #86183

Closed
Closed
May 9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6#86183
Labels
area-GC-coreclrneeds-further-triageIssue has been initially triaged, but needs deeper consideration or reconsideration
Milestone
Future

Description

@stefanschnabs
stefanschnabs
opened on May 12, 2023

Description

We are seeing segmentation faults as seen in the kernel logs:

May  9 21:11:41 server883 kernel: [218937.760135] .NET Server GC[1427719]: segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6
May  9 21:11:41 server883 kernel: [218937.760141] .NET Server GC[1427720]: segfault at 4 ip 00007f7440940652 sp 00007f74395bb9c0 error 4
May  9 21:11:41 server883 kernel: [218937.760143]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760149] Code: a2 91 0f 00 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 ff 74 18 66 48 8d 3d b7 33 0c 00 66 66 48 e8 a7 0c 0a 00 48 8b 00 <f0> 83 48 08 40 48 8b 05 c8 33 0c 00 48 8b 38 48 85 ff 74 09 48 8b
May  9 21:11:41 server883 kernel: [218937.760152]  in libcoreclr.so[7f74406a9000+4e2000]
May  9 21:11:41 server883 kernel: [218937.760156] Code: c8 22 00 00 48 3b 8f c0 22 00 00 77 0e 4c 89 29 49 8b 8f c8 22 00 00 49 8b 45 00 48 83 c1 08 49 89 8f c8 22 00 00 48 83 e0 f8 <8b> 48 04 83 38 00 0f 88 d0 03 00 00 45 31 f6 49 01 ce 49 8b 97 b0

We have seen it on multiple servers now running dotnet 7, we have not identified the issue on any servers running dotnet 6

We have identified one server where this is happening quote often:
image

What addition information could help with your investigation? For example, turn on a extra reporting, or we may be able to privately share a mini dump. If so please specify what you need and the correct way to enable and share privately.

Reproduction Steps

Unfortunately we have no steps identified to repro, but as seen above we do have a server experiencing the problem often.

Expected behavior

No segfaults

Actual behavior

segfaults occurring causing the container to crash and restart

Regression?

We have not seen this occur in dotnet 6

Known Workarounds

N/A
Update May 17th 2023: Can workaround this issue using the dotnet 6 GC in dotnet 7 by setting ENV VAR COMPlus_GCName=libclrgc.so. Note: this can only work in .NET 7 until such a moment when libclrgc.so is removed from the distribution

Configuration

  • mcr.microsoft.com/dotnet/sdk:7.0-jammy (7.0.203)
  • Ubuntu 20.04.6 LTS
  • x64

Other information

  • We know there has been big changes in GC in dotnet 7 compared to dotnet 6 and have yet to pin the problem any other change as of yet

Activity

  1. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on May 12, 2023
  2. ghost added
    untriagedNew issue has not been triaged by the area owner
    on May 12, 2023
  3. teo-tsirpanis
    added
    area-GC-coreclr
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on May 13, 2023

  5. danmoseley commented on May 13, 2023

    @danmoseley
    danmoseley
    on May 13, 2023
    Contributor

    Not on GC team, but they may ask you to try COMPlus_GCName=libclrgc.so. This will swap in the 6.0 GC - if it doesn't segfault, it's evidence the new GC is implicated.

  6. Maoni0 commented on May 15, 2023

    @Maoni0
    Maoni0
    on May 15, 2023
    Member

    yeah, if you could the env var that @danmoseley mentioned it would be great. if that doesn't repro, do you have a dump or at least a callstack that you could share with the default GC? and would it be possible to run with a private clrgc.dll if that would dramatically help with the investigation?

  7. stefanschnabs commented on May 17, 2023

    @stefanschnabs
    stefanschnabs
    on May 17, 2023
    Author

    Hi @Maoni0,

    • We have not been able to reproduce the issue by:

      • Reverting some instances back to dotnet 6
      • Using the dotnet 6 GC in dotnet 7 (COMPlus_GCName=libclrgc.so)

    • We have been able to capture 2 native thread call stacks, hopefully this can help investigate

    • I am not familiar with what you mean by private clrgc.dll. A custom GC with addition debugging enabled? If the 2 stack traces above are not fruitful we can look at that next.

    Let me know if you need something else,
    Thanks for you help

  8. Maoni0 commented on May 17, 2023

    @Maoni0
    Maoni0
    on May 17, 2023
    Member

    the callstacks just say that GC detected a heap corruption but this is a very generic thing. is it possible to share a couple of dumps?

    yes, by a private clrgc.dll (or in your case, libclrgc.so) I just meant we enable some instrumentation so it helps with debugging.

  9. stefanschnabs commented on May 17, 2023

    @stefanschnabs
    stefanschnabs
    on May 17, 2023
    Author

    How can we send you dump(s) privately using a secure channel?

  10. stefanschnabs commented on May 17, 2023

    @stefanschnabs
    stefanschnabs
    on May 17, 2023
    Author

    Have a couple other data points:

    • We think a line in our code return new ABCClass() as T ?? new List<double>() as T; ended up with this stack
      bt: server883btallCast.txt
    * thread #106, stop reason = signal SIGSEGV
  * frame #0: 0x00007f86bd43cc7f libc.so.6`wait4 + 95
    frame #1: 0x00007f86bd1bb143 libcoreclr.so`PROCCreateCrashDump(argv=<unavailable>, errorMessageBuffer=<unavailable>, cbErrorMessageBuffer=<unavailable>) at process.cpp:2486:22
    frame #2: 0x00007f86bd1bc287 libcoreclr.so`::PROCCreateCrashDumpIfEnabled(signal=<unavailable>, siginfo=<unavailable>) at process.cpp:2700:9
    frame #3: 0x00007f86bd1849ab libcoreclr.so`invoke_previous_action(action=<unavailable>, code=11, siginfo=0x00007f0e68687bf0, context=0x00007f0e68687ac0, signalRestarts=<unavailable>) at signal.cpp:416:5
    frame #4: 0x00007f86bd184067 libcoreclr.so`sigsegv_handler(code=11, siginfo=0x00007f0e68687bf0, context=0x00007f0e68687ac0) at signal.cpp:623:5
    frame #5: 0x00007f86bd8b2420 libpthread.so.0`__restore_rt
    frame #6: 0x00007fb2531fd817 libcoreclr.so`___lldb_unnamed_symbol2490$$libcoreclr.so + 39

00007E20B8FF70F0 00007fb253860c7f [HelperMethodFrame_1OBJ: 00007e20b8ff70f0]

System.Runtime.CompilerServices.CastHelpers.ChkCastAny(Void*, System.Object) [/_/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs @ 374]
    • verifyheap and dumpobj results
    verifyheap                                                                                                                                                                                               
Heap      Segment       Object Failure                          
   4 7f0f48f5d698 7f103e404428 InvalidObjectReference           Object 7f103e404428 has a bad member at offset cee8: 7f106210cf58
   4 7f0f48fda068 7f12f3cf00a8 InvalidObjectReference           Object 7f12f3cf00a8 has a bad member at offset 16ccd8: 7f12d8ce5568
   8 7f0f48f6bec0 7f108f1a1348 InvalidMethodTable               Object 7f108f1a1348 has an invalid method table 7f8647dad500
   9 7f0f48f63070 7f105d891310 InvalidMethodTable               Object 7f105d891310 has an invalid method table 7f8647dad6e8
  13 7f0f48fad488 7f11fc906160 InvalidObjectReference           Object 7f11fc906160 has a bad member at offset 16ccd8: 7f12d8ce5568
  15 7f0f48f63d60 7f106210cf58 InvalidMethodTable               Object 7f106210cf58 has an invalid method table 7f8647daced8

101,216,252 objects verified, 6 errors.

> dumpobj 7f103e404428                                                                                                                                                                                     
Name:        System.Object[]
MethodTable: 00007f863e82a508
EEClass:     00007f863e82a480
Tracked Type: false
Size:        58640(0xe510) bytes
Array:       Rank 1, Number of elements 7327, Type CLASS
Fields:
None

> dumpobj 7f12f3cf00a8                                                                                                                                                                                     
Name:        Company.Product.ObjectModel.DTOParent.DTOChildren[]
MethodTable: 00007f864a75ea00
EEClass:     00007f863e82a480
Tracked Type: false
Size:        4194328(0x400018) bytes
Array:       Rank 1, Number of elements 524288, Type CLASS
Fields:
None

> dumpobj 7f11fc906160                                                                                                                                                                                     
Name:        Company.Product.ObjectModel.DTOParent.DTOChildren[]
MethodTable: 00007f864a75ea00
EEClass:     00007f863e82a480
Tracked Type: false
Size:        2097176(0x200018) bytes
Array:       Rank 1, Number of elements 262144, Type CLASS
Fields:
None

  11. mangod9 commented on May 17, 2023

    @mangod9
    mangod9
    on May 17, 2023
    Member

    hi @stefanschnabs, you could upload the dumps to a share and send details to my email (should be in my profile). Also since you seem to have been able to narrow down the issue to something around casting, is it possible to create a smaller repro?

    @VSadov as fyi.

  12. VSadov commented on May 17, 2023

    @VSadov
    VSadov
    on May 17, 2023
    Member

    Looks like corrupted heap issue. CastHelpers are some of the most called methods and thus tend to trip over GC holes more often. Seeing such crash in CastHelpers is typically (and sadly) not very helpful with figuring the root cause.

  13. jeffschwMSFT commented on May 17, 2023

    @jeffschwMSFT
    jeffschwMSFT
    on May 17, 2023
    Member

    Our preference for receiving dumps is via the Visual Studio Developer Community. https://developercommunity.visualstudio.com/home

  14. stefanschnabs commented on May 24, 2023

    @stefanschnabs
    stefanschnabs
    on May 24, 2023
    Author

    As an update, the dump had been uploaded using Secure File Exchange

  15. 14 remaining items

  16. ghost removed
    needs-author-actionAn issue or pull request that requires more info or actions from the author.
    on Aug 19, 2023
  17. ghost removed
    no-recent-activity
    on Aug 19, 2023

  18. pieter-venter commented on Sep 15, 2023

    @pieter-venter
    pieter-venter
    on Sep 15, 2023

    As we are approaching the release of .Net 8 and we are still dependent on the workaround of using
    COMPlus_GCName=libclrgc.so (.Net 6 GC) - will this workaround be compatible/available with .Net 8.0?

  19. mangod9 commented on Sep 15, 2023

    @mangod9
    mangod9
    on Sep 15, 2023
    Member

    yeah it should still be supported in 8, though would be good to try without falling back to clrgc.

  20. pieter-venter commented on Sep 15, 2023

    @pieter-venter
    pieter-venter
    on Sep 15, 2023

    Thanks @mangod9, yes - we certainly want the problem fixed and run the latest 8.0 without falling back. But if the same issue we saw in .Net 7.0 is going to happen in .Net 8.0, it would be very important for us to enable the 6.0 GC behaviour until the root cause can be identified and fixed.

  21. stefanschnabs commented on Nov 1, 2023

    @stefanschnabs
    stefanschnabs
    on Nov 1, 2023
    Author

    Hi @mangod9 we have captured a second full dump caused by GC. It looks like the channel that @baal2000 used to upload the first full dump has expired, see Secure File Exchange Case 2305190040006263 . Hopefully we can get this channel reopened and the second full dump uploaded.

  22. mangod9 commented on Nov 2, 2023

    @mangod9
    mangod9
    on Nov 2, 2023
    Member

    hello @stefanschnabs, so you are still seeing the crashes with latest .NET 7 servicing build?

  23. baal2000 commented on Nov 3, 2023

    @baal2000
    baal2000
    on Nov 3, 2023

    @mangod9 It was 7.0.12 (7.0.13 is the latest security patch as of Oct 24)

    .NET Server GC[1221056] 
segfault at 8 ip 00007f0cf5cf935c sp 00007f0cef33edf0 error 6
in libcoreclr.so[7f0cf58ba000+4e2000]

    vs. May 12 report

    segfault at 8 ip 00007f7440ae835c sp 00007f74395bfdf0 error 6

  24. baal2000 commented on Nov 18, 2023

    @baal2000
    baal2000
    on Nov 18, 2023

    @mangod9
    The full process dump generated after GC segfault had been uploaded on Nov 10.

  25. stefanschnabs commented on Nov 20, 2023

    @stefanschnabs
    stefanschnabs
    on Nov 20, 2023
    Author

    @mangod9 The full process dump generated after GC segfault had been uploaded on Nov 10.

    If its not clear, @baal2000 is talking about the second full dump, so you can look for patterns in them.

  26. mangod9 commented on Nov 20, 2023

    @mangod9
    mangod9
    on Nov 20, 2023
    Member

    sorry must have missed that message. I looked at the dump from 11/10, and I dont see any crash in it. Is it the right dump? I see most threads waiting and a few doing json deserialization with this stack:

            Child SP               IP Call Site
00007DE11FFFB970 00007F71C437986D System.Text.Json.JsonDocument.Parse(System.ReadOnlySpan`1<Byte>, System.Text.Json.JsonReaderOptions, MetadataDb ByRef, StackRowStack ByRef) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.cs @ 1011]
00007DE11FFFBB10 00007F71C43F8531 System.Text.Json.JsonDocument.Parse(System.ReadOnlyMemory`1<Byte>, System.Text.Json.JsonReaderOptions, Byte[], System.Text.Json.PooledByteBufferWriter) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs @ 692]
00007DE11FFFBBC0 00007F71C4378E90 System.Text.Json.JsonDocument.TryParseValue(System.Text.Json.Utf8JsonReader ByRef, System.Text.Json.JsonDocument ByRef, Boolean, Boolean) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs @ 625]
00007DE11FFFBE10 00007F71C43F75AC Geotab.Checkmate.Serialization.DeviceConverter2.Read(System.Text.Json.Utf8JsonReader ByRef, System.Type, System.Text.Json.JsonSerializerOptions)
00007DE11FFFC020 00007F71BF27D616 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].TryRead(System.Text.Json.Utf8JsonReader ByRef, System.Type, System.Text.Json.JsonSerializerOptions, System.Text.Json.ReadStack ByRef, System.__Canon ByRef) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs @ 226]
00007DE11FFFC070 00007F71BF27D135 System.Text.Json.Serialization.JsonConverter`1[[System.__Canon, System.Private.CoreLib]].ReadCore(System.Text.Json.Utf8JsonReader ByRef, System.Text.Json.JsonSerializerOptions, System.Text.Json.ReadStack ByRef) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs @ 60]
00007DE11FFFC0F0 00007F71C43657A0 System.Text.Json.JsonSerializer.ReadFromSpan[[System.__Canon, System.Private.CoreLib]](System.ReadOnlySpan`1<Byte>, System.Text.Json.Serialization.Metadata.JsonTypeInfo, System.Nullable`1<Int32>) [/_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Helpers.cs @ 44]
00007DE11FFFC310 00007F71C436553B System.Text.Json.JsonSerializer.ReadFromSpan[[System.__Canon, System.Private.CoreLib]](System.ReadOnlySpan`1<Char>, System.Text.Json.Serialization.Metadata.JsonTypeInfo)
00007DE11FFFC390 00007F71C437C834 System.Text.Json.JsonSerializer.Deserialize[[System.__Canon, System.Private.CoreLib]](System.String, System.Text.Json.JsonSerializerOptions)
00007DE11FFFC3D0 00007F71C437C9AC Geotab.Serialization.BaseJsonSerializer2.JsonToObject[[System.__Canon, System.Private.CoreLib]](System.String)
00007DE11FFFC400 00007F71C43F5639 Geotab.Checkmate.Database.Bridge.DeviceBridge.DeviceFromReader(System.Data.Common.DbDataReader)

  27. baal2000 commented on Nov 20, 2023

    @baal2000
    baal2000
    on Nov 20, 2023

    @mangod9
    Sorry we have been under impression this is the correct dump because it was recorded right after the GC segfault exception.

    Later I have loaded the dump into lldb and could not find an exception there. The main thread's stop reason = signal SIGSTOP.

    Not sure what it really means and what triggered the dump collection but agree the dump is probably not useful.

  28. stefanschnabs commented on Apr 30, 2024

    @stefanschnabs
    stefanschnabs
    on Apr 30, 2024
    Author

    Hi @mangod9 @Maoni0 @VSadov @jeffschwMSFT @danmoseley, We have not seen this issue on .NET 8.
    I think we can close out this issue, thanks all for your help

  29. stefanschnabs
    closed this as completedon Apr 30, 2024
  30. github-actions
    locked and limited conversation to collaborators on May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-GC-coreclrneeds-further-triageIssue has been initially triaged, but needs deeper consideration or reconsideration

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions