System.Net.* and System.Security.* crash on NativeAOT arm64 #74710

Activity

1. karelz

   added

   arch-arm64

   area-System.Net.Security

   on Aug 27, 2022
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Aug 27, 2022
5. jkotas commented on Aug 27, 2022

   

   jkotas

   on Aug 27, 2022

   Member

   More actions

   Stacktrace:

   0x0000007f`84350000
   libcrypto_so_1!X509_get_pubkey_parameters+0xf18
   libcrypto_so_1!X509_verify_cert+0x78
   System_Net_Security!System_Security_Cryptography_Interop_Crypto___CryptoNative_X509VerifyCert_g____PInvoke_275_0+0x3c [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 7264] 
   System_Net_Security!System_Security_Cryptography_Interop_Crypto::CryptoNative_X509VerifyCert+0x38 [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 5839] 
   System_Net_Security!System_Security_Cryptography_Interop_Crypto::X509VerifyCert+0x1c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.X509.cs @ 196] 
   System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::BuildWorkingChain+0x98 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 653] 
   System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::Finish+0x58 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 688] 
   System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChainCore+0x320 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 198] 
   System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChain+0xb4 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 49] 
   System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_X509Chain::Build+0x308 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Chain.cs @ 126] 
   System_Net_Security!System_Net_Security_System_Net_Security_SslStreamCertificateContext::Create+0x184 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamCertificateContext.cs @ 56] 
   System_Net_Security!System_Net_Security_System_Net_Security_SslAuthenticationOptions::UpdateOptions+0x174 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.cs @ 123] 
   System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x24 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 426] 
   System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x8c [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 419] 
   System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_AsyncSslStreamSystemDefaultTest::AuthenticateServerAsync+0x48 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 231] 
   System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest___c__DisplayClass9_0::<ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails>b__0+0x70 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
   System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync_d__95::MoveNext+0x38 [/_/src/xunit.assert/Asserts/Record.cs @ 104] 
   System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
   System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync+0x30 [/_/src/xunit.assert/Asserts/Record.cs @ 75] 
   System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync_d__67_1<System___Canon>::MoveNext+0x48 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 132] 
   System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x64 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
   System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync<System___Canon>+0x44 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 119] 
   System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails_d__9::MoveNext+0x1ac [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
   System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
   System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails+0x38 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 30] 
   
6. jkotas commented on Aug 27, 2022

   

   jkotas

   on Aug 27, 2022

   Member

   More actions

   A function pointer in libcrypto data structures is bad. Use-after-free bug?

   84350000 is the corrupted function pointer. 8490... are the good function pointers around it.

   0000007f`7c0dbaf0  0000007f`8490f1b0 0000007f`84350000
   0000007f`7c0dbb00  0000007f`8490b8e0 0000007f`8490e290
   
7. wfurt

   self-assigned this

   on Aug 29, 2022
8. karelz

   added this to the 7.0.0 milestone on Sep 1, 2022
9. karelz

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Sep 1, 2022
10. karelz commented on Sep 1, 2022

    

    karelz

    on Sep 1, 2022

    MemberAuthor

    More actions

    Triage: We should treat it as high pri for 7.0 until we have further details about root cause.
11. karelz

    changed the title [-]System.Net.Security.Tests crash on NativeAOT arm64[/-] [+]System.Net.* crash on NativeAOT arm64[/+] on Sep 9, 2022
12. karelz commented on Sep 9, 2022

    

    karelz

    on Sep 9, 2022

    Last edited by karelz

    MemberAuthor

    More actions

    cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...
13. jkotas commented on Sep 9, 2022

    

    jkotas

    on Sep 9, 2022

    Member

    More actions

    I have taken a look at the two most recent core dumps. It is still same pattern - function pointer overwritten in libcrypto
    data structure.

    My guess is that this crash is another manifestation of the problem that is behind other mysterious crashes in networking tests: #74795 and #72830 . There seems to be a stray memory write in networking that corrupts random memory.
14. MichalStrehovsky commented on Sep 9, 2022

    

    MichalStrehovsky

    on Sep 9, 2022

    Member

    More actions

    I was looking at the System.Security.Cryptography.Tests crash in #75306 that also looks similar (we end up in some location that is not valid code) - https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75306-merge-171bd00681b64e6184/System.Security.Cryptography.Tests/1/console.108589f8.log?helixlogtype=result.

    That one is not a networking test, but libcrypto is the common theme. Maybe a crypto bug instead of networking bug?

    cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...

    I'm not aware of any active ARM64 issues with NativeAOT. The one we had was fixed two weeks ago.

    ❤️React with ❤️1Reacted by Karel Zikmund
15. 16 remaining items

    Load more
16. ghost added

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 10, 2022
17. jkotas

    closed this as completedin #75393on Sep 10, 2022
18. jkotas

    added a commit that references this issue on Sep 10, 2022

    Flush instruction cache after thunk pool allocation (#75393)

    ...

    This issue will close once commit 1f94e11 is merged into the 'main' branch.

    1f94e11
19. ghost removed

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 10, 2022
20. github-actions

    added a commit that references this issue on Sep 10, 2022

    Flush instruction cache after thunk pool allocation

    ...

    This issue will close once commit 2f16613 is merged into the 'main' branch.

    2f16613
21. ghost added

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 12, 2022
22. karelz commented on Sep 12, 2022

    

    karelz

    on Sep 12, 2022

    MemberAuthor

    More actions

    Reopening to track the 7.0 backport ...
23. karelz

    reopened this on Sep 12, 2022
24. karelz

    added

    tenet-reliabilityReliability/stability related issue (stress, load problems, etc.)Reliability/stability related issue (stress, load problems, etc.)

    on Sep 12, 2022
25. carlossanlop

    added a commit that references this issue on Sep 12, 2022

    [release/7.0] Flush instruction cache after thunk pool allocation (#7…

    ...

    This issue will close once commit d920ddc is merged into the 'main' branch.

    d920ddc
26. ghost removed

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 12, 2022
27. jkotas commented on Sep 12, 2022

    

    jkotas

    on Sep 12, 2022

    Member

    More actions

    Backport has been merged.
28. jkotas

    closed this as completedon Sep 12, 2022
29. karelz commented on Sep 12, 2022

    

    karelz

    on Sep 12, 2022

    MemberAuthor

    More actions

    Fixed in main (8.0) in PR #75393 and in 7.0 (RC2) in PR #75403.
30. ghost locked as resolved and limited conversation to collaborators on Oct 13, 2022