Repository navigation
Cross built ILCompiler NuGet contains HostOS (Linux) ELFs not TargetOS (FreeBSD) ELFs #104497
Description
Activity
Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.Ok, here is the problem:
The regular build uses "last known good" toolchain (the one that comes from https://github.com/dotnet/runtime/blob/main/global.json#L3) for aot compilation of most managed tools and components. The one exception is crossgen2 that uses live built aot toolchain that leads to interesting problems every once in a while.
When cross-compiling from Linux to FreeBSD, there is no "last known good" aot toolchain that is able to cross-compile from Linux to FreeBSD. We end up doing Linux->Linux publish instead due to some incorrect build logic and produce binaries for wrong architecture as you have observed.
I can think about two options for addressing this:
- Enable use of live built aot toolchain for compilation of managed tools and components. The current prior art with crossgen2, is complicated and fragile. We had several attempts to extend it to other tools and components (e.g. Crossgen2 should publish with NativeAOT at the beginning of the build #71557) that we ended up not pursuing. I think that the proper solution would be a multi-stage build: We need to first build the compiler that is then going to be used to compile the shipping compiler. It should be opt-in option and we should keep using "last known good" to keep the default build simple. The source build bootstrapping does similar multi-stage build already, perhaps we can use some of the same controls.
- Disable aot compilation of the managed tools and components when there is no "last known good" toolchain. It may be simpler, but it would mean that the cross-compiling Linux to FreeBSD build does not have full fidelity with the native FreeBSD build.
Would disabling AOT for managed items still give FreeBSD (TargetOS) ELF's on components like ILCompiler?
If yes, that might be the simplest solution for the time being.The multi-stage build sounds like the best solution for this issue. But, unless there is something I am missing, this issue seems limited to a niche case of cross compiling from Linux to FreeBSD/Illumos/Haiku/Other. If it helps other build scenarios for officially supported platforms then there is a better case for this but I am not familiar enough with intricacies here.
unless there is something I am missing, this issue seems limited to a niche case of cross compiling from Linux to FreeBSD/Illumos/Haiku/Other
I think having the recipe for how to use live-built AOT compilers would be generally useful. We keep having long discussions about whether it is better to use live-built AOT compiler or last-known-good AOT compiler. So having both options and use the one that's more appropriate for given situation would be best.
Would disabling AOT for managed items still give FreeBSD (TargetOS) ELF's on components like ILCompiler?
It would require some extra build logic. Nothing impossible.
Reacted by Thefrank12 remaining items
Which FreeBSD version and which LLVM/Clang version?
13.3-RELEASE-p3 and llvm18 (had the same with llvm15). Tried to native build runtime with
-p:LinkerFlavor=bfdswitch, but this does not changed anything. Also there's one version of ilc that does work from native build:sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % find . -type f -name ilc -exec ls -al {} + -rwxr--r-- 1 sec sec 16606596 Aug 19 16:56 ./.packages/runtime.freebsd-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.7.24405.7/tools/ilc -rwxr-xr-x 1 sec sec 14560696 Aug 20 11:58 ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc-published/ilc -rwxr-xr-x 1 sec sec 72480 Aug 20 11:57 ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/ilc -rwxr-xr-x 1 sec sec 14560696 Aug 20 11:58 ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/native/ilc sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % ./.packages/runtime.freebsd-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.7.24405.7/tools/ilc|head -n 2 Required argument missing for command: 'ilc'. Description: .NET Native IL Compiler sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc-published/ilc|head -n 2 Segmentation fault (core dumped) sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/ilc|head -n 2 Required argument missing for command: 'ilc'. Description: .NET Native IL Compiler sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/native/ilc|head -n 2 Segmentation fault (core dumped)edit: hm, looks like ilc is build without symbol exported from #105587 when doing native build, will look into that.
./artifacts/bin/coreclr/freebsd.arm64.Release/ilc/ilcmight be the same from./.packages/runtime.freebsd-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.7.24405.7/tools/ilcwhich might explain why it works but not why there is a difference between the crossbuilt and native build ILCThe size of working ones are different - native one is 72480 bytes and the one from cross nuget is 16606596. the other two, that don't work don't have symbols exported that added them on crossbuild - have build running with symbol export added without
Condition="'$(_targetOS)' == 'freebsd' and '$(IsNativeExecutable)' == 'true'"to verify if this is the issue or is it somewhere else, will know if few minutes.edit: it sill segfault and there's no progname and environ symbol exported
sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % nm --dynamic ./.packages/runtime.freebsd-arm64.microsoft.dotnet.ilcompiler/9.0.0-preview.7.24405.7/tools/ilc|grep prog 00000000009efaf8 D __progname sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime % nm --dynamic ./artifacts/bin/coreclr/freebsd.arm64.Release/ilc-published/ilc|grep prog sec@amper:/mnt/sec/dotnet-core-freebsd-source-build/runtime %@am11 maybe you could point in right direction, why those symbol exports could be missing when doing native build of ILC?
Could anyone explain why with the same target WriteIlcRspFileForCompilation, arguments list is different?
this one is missing export of __progname and environ:
while this one have them:
@sec, I haven't tested it with freebsd-arm64, but if all things being equal (to x64) it should work the same way. I think the first run of the target is getting targetOS == linux for some reason in your environment.
I have changed those conditions to
<IlcArg Include="--export-dynamic-symbol:__progname" />so they should always be included, but I'm missing something.I have changed those conditions to
<IlcArg Include="--export-dynamic-symbol:__progname" />so they should always be included, but I'm missing something.Have you changed it in all
find ~/.nuget -name '*Unix.targets'?Hm, based on your PR, you only added those to
src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targetsI tried to look where
--export-dynamic-symbol:DotNetRuntimeDebugHeaderis added and it's only also in this file and it's added in both runs.Microsoft.NETCore.Native.Unix.targetsdon't contains any IlcArgsAh I meant find ~/.nuget -name '*Native.targets'. I'll create a VM on osx arm64 and try to debug it on the weekend.
Ok, I think I've found the issue, indeed I had some old nuget's cached - will confirm that in few minutes.
edit: yes, I had old
Microsoft.DotNet.ILCompiler.9.0.0-previ ew.7.24405.7insideruntime/.packagesthat didn't had those args added (that was from the crossbuild before ilc patch...) - thanks @am11 :)full build of runtime/aspnet/sdk was completed under freebsd-arm64 :)
Reacted by Adeel MujahidOne of those
IlcArgitems in both projects might(should?) contain a--targetos:value. If it does readlinuxor that entry is not there, then you might have to fish around the log and find where it got misassigned.- locked and limited conversation to collaborators
on Feb 16, 2025
Overview:
When cross compiling runtime, the ILCompiler (and NuGet) that is produced contains HostOS (Linux) and not TargetOS (FreeBSD) ELFs.
Reproduction:
Using a docker container on Linux pulled from: https://raw.githubusercontent.com/dotnet/versions/master/build-info/docker/image-info.dotnet-dotnet-buildtools-prereqs-docker-main.json containing a FreeBSD ROOTFS and the most recent net9 preview tag from runtime:
v9.0.0-preview.5.24306.7Expected behavior:
ELFs should be like other items generated for TargetOS:
Actual behavior:
The resulting
runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.9.0.0-preview.5.24306.7.nupkgcontains Linux ELFs and FreeBSD libs.Regression:
To the best of my knowledge, it has always been this way. This was uncaught until recently when I tried to use a cross built package to bootstrap a native build.
Known Workarounds:
None?
Other info:
TargetOS=linuxonly appears three places in the .binlog for the build that are after Evaluation. All three are from ILCompiler.cspoj : The first seems to come as a return from ResolveReadyToRunCompilers and the other two (_PrepareForReadyToRunCompilation) and (_CreateR2RImages) use it.NativeAotSupportedis reassigned here:runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
Line 17 in a5cc707
ilcbinary and the process does not error fromruntime/src/tasks/Crossgen2Tasks/ResolveReadyToRunCompilers.cs
Lines 116 to 120 in a5cc707
There is no "Property reassignment" note in the binlog when ResolveReadyToRunCompilers changes(?) the
TargetOS=freebsdtolinux.binlog is not an allowed attachment type so hopefully a screenshot from the MSBuild Structured Log Viewer is enough to help explain what I am seeing.
The Crossgen2 project seems to avoid the ReadyToRun part during packaging:
runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj
Lines 52 to 53 in a5cc707