Skip to content

FreeBSD tests crash with HRESULT: 0x8007FF02 from #106724 #106892

Closed
#107100
Closed
FreeBSD tests crash with HRESULT: 0x8007FF02 from #106724#106892
#107100
Labels
area-VM-coreclrin-prThere is an active PR which will close this issue when it is mergedos-freebsdFreeBSD OS
Milestone
10.0.0

Description

@Thefrank
Thefrank
opened on Aug 23, 2024

Description

From #106724 forward tests on FreeBSD now all creash with HRESULT: 0x8007FF02

These do not show up in /var/log/messages on either the jail or the host. dmesg also free of errors from this. Nothing SIGXXXX.
Builds otherwise complete without issue.

Reproduction Steps

As initially discovered:
./build.sh -ci -c Release -subset Clr+Mono+Host.Native+Host.Tools+Host.Pkg+Libs+Libs.Tests+Packs --test
Double checked with (requires #106302):
src/tests/build.sh Release /p:LibrariesConfiguration=Release -rebuild -runtests

Expected behavior

Tests to run either pass/fail

Actual behavior

"exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill."

or

"Failed to create CoreCLR, HRESULT: 0x8007FF02"

Regression?

Yes

Known Workarounds

None?

Configuration

FreeBSD-x64 both 13.3 and 14.1
System has about ~200G of free RAM during this. Jails run without resource quotas.

Other information

Public log for both FreeBSD 13.3 and 14.1: https://dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_build/results?buildId=1625&view=results BinLog are published to artifacts

commit listed is 6df7807 (after PR listed)

git bisect log

git bisect start
# status: waiting for both good and bad commits
# good: [21fe57174af159f7bcb39f591cd9fbb96a36bfcb] COMObject invoke through reflection (#106677)
git bisect good 21fe57174af159f7bcb39f591cd9fbb96a36bfcb
# status: waiting for bad commit, 1 good commit known
# bad: [6df7807617cff3e4b4eba7bb33c8ad00c2b1cf1c] Add missing .alt_entry to CoreCLR *_FakeProlog methods (#106744)
git bisect bad 6df7807617cff3e4b4eba7bb33c8ad00c2b1cf1c
# good: [a3cbf8c2534040faa4664e45c19a0ba877bb5c5c] Grab back some performance in `RuntimeTypeHandle::GetElementType` (#106730)
git bisect good a3cbf8c2534040faa4664e45c19a0ba877bb5c5c
# good: [4e8892ea65fe85593b1853907d965fb49d324ce9] JIT: Add Non-Faulting behaviour for Sve.LoadVector*NonFaulting*() (#106648)
git bisect good 4e8892ea65fe85593b1853907d965fb49d324ce9
# good: [cfa0cc5b0282c29597f10b6c2fdceab159e0be81] Inject IJW Copy Constructor calls in the JIT instead of in the IL stubs (#106424)
git bisect good cfa0cc5b0282c29597f10b6c2fdceab159e0be81
# good: [cfa8da560f674d035a26e9b3b07067933d46e44c] Avoid signed overflow in DBG_FlushInstructionCache (#105918)
git bisect good cfa8da560f674d035a26e9b3b07067933d46e44c
# bad: [92e41b6b2cf7aea845fcfc9a439dc4e191c07428] Add callback-based Encode to AsnWriter
git bisect bad 92e41b6b2cf7aea845fcfc9a439dc4e191c07428
# bad: [27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d] pal init: InitializeFlushProcessWriteBuffers() before first thread to improve start time (#106724)
git bisect bad 27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d
# first bad commit: [27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d] pal init: InitializeFlushProcessWriteBuffers() before first thread to improve start time (#106724)

Activity

  1. ghost added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Aug 23, 2024
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Aug 23, 2024

  3. harisokanovic commented on Aug 23, 2024

    @harisokanovic
    harisokanovic
    on Aug 23, 2024
    Contributor

    @Thefrank , can you recommend a guide for building the v9 SDK on FreeBSD? https://github.com/dotnet/dotnet/ fails with ./prep-source-build.sh: line 199: /dotnet-dotnet/.dotnet/dotnet: No such file or directory.

  4. Thefrank commented on Aug 23, 2024

    @Thefrank
    Thefrank
    on Aug 23, 2024
    ContributorAuthor

    Building the SDK requires bootstraping from Linux or using an old SDK to build the new one. If you just want a prebuilt we have those too.

    Building:
    Bootstraping will require: #105004 #105587 and optionally #105587. Native builds only need #105587. If you want a collection of scripts for native building, there is this

    I have not tried to build from VMR for net9 as the VMR tends to require more work than "bootstrap from tags" or "use old SDK to build new SDK" :(

    Downloading:
    Both I and @sec have net9p7 SDKs prebuilt: Here or Here

    The former repo is what I use for my builds; the later by @sec also has ARM64 versions and will work just as well for either AMD64 or ARM64. I cherry-pick missing/needed PRs into my SDKs (e.g., the ones listed above). If you want an SDK with 0 changes @sec's are better.

    Please let me know how else I can help with this!

  5. JulieLeeMSFT
    added
    os-freebsdFreeBSD OS
    and removed
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    on Aug 23, 2024

  6. JulieLeeMSFT commented on Aug 23, 2024

    @JulieLeeMSFT
    JulieLeeMSFT
    on Aug 23, 2024
    Member

    Not a codegen issue. Removing codegen label.

  7. JulieLeeMSFT
    added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Aug 23, 2024
  8. jkotas
    added
    area-VM-coreclr
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Aug 24, 2024

  9. cperciva commented on Aug 26, 2024

    @cperciva
    cperciva
    on Aug 26, 2024

    FreeBSD dev here. Just to confirm, this is running in native FreeBSD, not via the Linux emulation layer?

  10. cperciva commented on Aug 26, 2024

    @cperciva
    cperciva
    on Aug 26, 2024

    Does this process ever call execve? The "process has called MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED" flag gets cleared when we execve and that would result in MEMBARRIER_CMD_PRIVATE_EXPEDITED returning EPERM later.

  11. Thefrank commented on Aug 26, 2024

    @Thefrank
    Thefrank
    on Aug 26, 2024
    ContributorAuthor

    It is running on native FreeBSD (no linux kernel modules, linux service not running) on both host and jail. allow.mlock=1 is set for the jail where the build system is located.

  12. cperciva commented on Aug 27, 2024

    @cperciva
    cperciva
    on Aug 27, 2024

    @Thefrank And does is call execve at any point?

  13. Thefrank commented on Aug 27, 2024

    @Thefrank
    Thefrank
    on Aug 27, 2024
    ContributorAuthor

    @cperciva Yes. I have attached a truss.txt (truss -fadeH -o truss.txt /root/runtime/artifacts/log/repro/ilasm/PortablePdb/IlasmPortablePdbTests/repro_IlasmPortablePdbTests.sh). Test selected at random.
    truss.txt

  14. 5 remaining items

  15. harisokanovic
    added a commit that references this issue on Aug 28, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    5bc42ed
  16. dotnet-policy-service
    added
    in-prThere is an active PR which will close this issue when it is merged
    on Aug 28, 2024
  17. harisokanovic
    added a commit that references this issue on Aug 28, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    86b2026
  18. mangod9
    removed
    untriagedNew issue has not been triaged by the area owner
    on Aug 28, 2024
  19. mangod9
    added this to the 10.0.0 milestone on Aug 28, 2024
  20. janvorli
    closed this as completedin #107100on Aug 28, 2024
  21. janvorli
    added a commit that references this issue on Aug 28, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    c655529
  22. github-actions
    added a commit that references this issue on Aug 28, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    c91139f

  23. jkotas commented on Aug 28, 2024

    @jkotas
    jkotas
    on Aug 28, 2024
    Member

    @Thefrank The fix is in. Could you please verify that FreeBSD is functional again?

    Also, it would be useful to find out why MEMBARRIER_CMD_QUERY path does not work on FreeBSD. MEMBARRIER_CMD_QUERY is important for Arm64. If this path does not work on Arm64, the runtime will have reliability issues. I would not be opposed to disabling the fallback path for Arm64, so that we do not have to chase the Arm64 intermittent crashes caused by it.

  24. jkotas
    added a commit that references this issue on Aug 29, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    0a5cccb

  25. Thefrank commented on Aug 29, 2024

    @Thefrank
    Thefrank
    on Aug 29, 2024
    ContributorAuthor

    @jkotas Looks good on my end!

    I don't have an ARM64 hardware setup to test on but @sec might have some insight on FreeBSD-ARM64 with .NET.

    Under FreeBSD-x64, I have not noticed any consistent errors*

    Not sure what to do about the MEMBARRIER_CMD_QUERY issue either. There is no manpage for either membarrier(2) or rseq(2) on FreeBSD and source/differential might be a pain to dig through. If reliability is a concern then it sounds like disabling the fallback path might be for the best.

    *I have noticed an inconsistent error when restore is run during the initial build of runtime. I will open an issue about this when I can better reproduce it.

  26. sec commented on Aug 29, 2024

    @sec
    sec
    on Aug 29, 2024
    Contributor

    I will try to find some time and do the checks/tests on arm64.

    edit: fix looks fine also on arm64.

  27. jtschuster
    added a commit that references this issue on Sep 17, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    f2c3044
  28. github-actions
    locked and limited conversation to collaborators on Oct 2, 2024
  29. mikelle-rogers
    added a commit that references this issue on Dec 10, 2024

    pal init: Move InitializeFlushProcessWriteBuffers() after VIRTUALInit…

    016303c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-VM-coreclrin-prThere is an active PR which will close this issue when it is mergedos-freebsdFreeBSD OS

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions