Microsoft.NETCore.App.Crossgen2.sfxproj in source-build

@ericstj

This project publishes crossgen as a self-contained app.

For source-build this needs to use the runtime that got built from the runtime repo.

The builds also needs to be aware of the rid that source-build added via AdditionalRuntimeIdentifiers by using #69455.

cc @ericstj @omajid @MichaelSimons @jkotas @ViktorHofer

For source-build this needs to use the runtime that got built from the runtime repo.

It may be easiest to publish it as framework dependent app for source-build.

@hoyosjs

@hoyosjs, @trylek, @jkoritzinsky (@agocke) just changed labels as this isn't related to the libraries infrastructure. Can you please take a look?

For source-build this needs to use the runtime that got built from the runtime repo.

When we're source-building .NET, runtime gets built twice: once as runtime-portable, and then as (non-portable) runtime.

When the non-portable runtime build happens, it is using packages that got source-built by runtime-portable. So all output artifacts from runtime were built from source, and we meet the built-from-source requirement.

The runtime-portable is taking up a large chunk of the build time, and it would be a huge improvement if we can eliminate it.

To meet the built-from-source requirement, the Microsoft.NETCore.App.Crossgen2 package may only contain things that were built from source. We can't use a pre-built Microsoft.NETCore.App.Runtime.<rid> package and should change to use artifacts that are built as part of runtime.
Then runtime can produce built-from-source artifacts in a single pass.

We can then eliminate the runtime-portable build by making builds use the non-portable packages built by runtime (like #75597 and dotnet/aspnetcore#43937)

Is this needed for .NET 7? From what I see this depends on #69455 which is not merged. We are already past RC2 snap.

Once that's in, I wonder if we need a task to update KnownFrameworkReference et al to account for AdditionalRuntimeIdentifiers in RuntimePackRuntimeIdentifiers and have the runtime packs be produced in a place visible to it.

@ericstj

Is this needed for .NET 7? From what I see this depends on #69455 which is not merged. We are already past RC2 snap.

I think we should set our goal to .NET 7 if that is feasible for everyone involved.

source-build's timing doesn't make it feasible to change something before rc.

From what I see this depends on #69455 which is not merged.

In this case, the rid we're publishing would be an exact match with a known rid.
I don't know if it needs to be in the graph for this match to work ... may be.

Once that's in, I wonder if we need a task to update KnownFrameworkReference et al to account for AdditionalRuntimeIdentifiers in RuntimePackRuntimeIdentifiers and have the runtime packs be produced in a place visible to it.

I don't know about the Task. In #75597 I'm making some changes so source-build can pick up non-portable rids from a previous build.

It would be good if we could eliminate the known references to things we shouldn't be using when DotNetBuildFromSource == true. That would be a good guard for the built-from-source requirement.

cc @ericstj

We are already past RC2 snap.

Infrastructure changes are OK to go into 7.0 so long as they stay constrained to infrastructure and feed into the source-build requirements. We can take them to Tactics if we think there is some product risk - typically we don't and instead work to mitigate the risk through some PR validation (manual checks, test cases, etc). I closed out that PR since I'm not actively working on it and my time for making fixes (especially those that others are dependent on) is pretty small at the moment. I think it makes more sense for that to be driven by folks who need it and can test the end-to-end with the change.

@ViktorHofer

@ViktorHofer @MichalStrehovsky can someone take a look at this?

fyi, this is what my latest attempt looks like:

diff --git a/eng/targetingpacks.targets b/eng/targetingpacks.targets
index 744a0cd20b4..b4d35aab5f7 100644
--- a/eng/targetingpacks.targets
+++ b/eng/targetingpacks.targets
@@ -26,6 +26,7 @@
 
   <!-- Add Known* items if the SDK doesn't support the TargetFramework yet. -->
   <ItemGroup Condition="'$(UseLocalTargetingRuntimePack)' == 'true'">
+    <KnownFrameworkReference Remove="$(LocalFrameworkOverrideName)" />
     <KnownFrameworkReference Include="$(LocalFrameworkOverrideName)"
                              DefaultRuntimeFrameworkVersion="$(ProductVersion)"
                              LatestRuntimeFrameworkVersion="$(ProductVersion)"
@@ -36,6 +37,7 @@
                              TargetingPackName="$(LocalFrameworkOverrideName).Ref"
                              TargetingPackVersion="$(ProductVersion)"
                              Condition="'@(KnownFrameworkReference)' == '' or !@(KnownFrameworkReference->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))" />
+    <KnownRuntimePack Remove="$(LocalFrameworkOverrideName)" />
     <KnownRuntimePack Include="$(LocalFrameworkOverrideName)"
                       TargetFramework="$(NetCoreAppCurrent)"
                       RuntimeFrameworkName="$(LocalFrameworkOverrideName)"
@@ -44,12 +46,14 @@
                       RuntimePackRuntimeIdentifiers="linux-arm;linux-armv6;linux-arm64;linux-musl-arm64;linux-bionic-arm64;linux-loongarch64;linux-musl-x64;linux-bionic-x64;linux-x64;osx-x64;rhel.6-x64;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64;maccatalyst-x64;maccatalyst-arm64;browser-wasm;ios-arm64;ios-arm;iossimulator-arm64;iossimulator-x64;iossimulator-x86;tvos-arm64;tvossimulator-arm64;tvossimulator-x64;android-arm64;android-arm;android-x64;android-x86"
                       RuntimePackLabels="Mono"
                       Condition="'@(KnownRuntimePack)' == '' or !@(KnownRuntimePack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))"/>
+    <KnownAppHostPack Remove="$(LocalFrameworkOverrideName)" />
     <KnownAppHostPack Include="$(LocalFrameworkOverrideName)"
                       AppHostPackNamePattern="$(LocalFrameworkOverrideName).Host.**RID**"
                       AppHostPackVersion="$([MSBuild]::ValueOrDefault('$(_AppHostBaselinePackVersion)', '$(ProductVersion)'))"
                       AppHostRuntimeIdentifiers="linux-arm;linux-armv6;linux-arm64;linux-musl-arm64;linux-bionic-arm64;linux-loongarch64;linux-musl-x64;linux-bionic-x64;linux-x64;osx-x64;rhel.6-x64;tizen.4.0.0-armel;tizen.5.0.0-armel;win-arm;win-arm64;win-x64;win-x86;linux-musl-arm;osx-arm64"
                       TargetFramework="$(NetCoreAppCurrent)"
                       Condition="'@(KnownAppHostPack)' == '' or !@(KnownAppHostPack->AnyHaveMetadataValue('TargetFramework', '$(NetCoreAppCurrent)'))" />
+    <KnownCrossgen2Pack Remove="$(LocalFrameworkOverrideName).Crossgen2" />
     <KnownCrossgen2Pack Include="$(LocalFrameworkOverrideName).Crossgen2"
                         TargetFramework="$(NetCoreAppCurrent)"
                         Crossgen2PackNamePattern="$(LocalFrameworkOverrideName).Crossgen2.**RID**"
@@ -87,6 +91,14 @@
                        Condition="'$(UsePackageDownload)' == 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Runtime'))" />
       <PackageReference Remove="@(PackageReference)"
                         Condition="'$(UsePackageDownload)' != 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Runtime'))" />
+      <PackageDownload Remove="@(PackageDownload)"
+                       Condition="'$(UsePackageDownload)' == 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Crossgen2'))" />
+      <PackageReference Remove="@(PackageReference)"
+                        Condition="'$(UsePackageDownload)' != 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Crossgen2'))" />
+      <PackageDownload Remove="@(PackageDownload)"
+                       Condition="'$(UsePackageDownload)' == 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Host'))" />
+      <PackageReference Remove="@(PackageReference)"
+                        Condition="'$(UsePackageDownload)' != 'true' and $([System.String]::Copy('%(Identity)').StartsWith('$(LocalFrameworkOverrideName).Host'))" />
     </ItemGroup>
   </Target>
 
diff --git a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
index 12864a2341e..bd45dd2dd41 100644
--- a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
+++ b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
@@ -13,6 +13,7 @@
     <RuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
     <SelfContained>false</SelfContained>
     <SelfContained Condition="'$(RunningPublish)' == 'true'">true</SelfContained>
+    <UseLocalTargetingRuntimePack>true</UseLocalTargetingRuntimePack>
   </PropertyGroup>
 
   <Import Project="crossgen2.props" />
@@ -28,6 +29,7 @@
   </PropertyGroup>
 
   <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
+  <Import Project="$(RepositoryEngineeringDir)targetingpacks.targets" />
 
   <PropertyGroup Condition="'$(NativeAotSupported)' == 'true'">
     <IlcToolsPath>$(CoreCLRILCompilerDir)</IlcToolsPath>

And dotnet pack Microsoft.NETCore.App.Crossgen2.sfxproj gives me:

/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018: The "ResolveRuntimePackAssets" task failed unexpectedly. [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]
/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]
/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ResolveRuntimePackAssets.ExecuteCore() [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]
/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]
/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]
/home/tmds/repos/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(440,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/home/tmds/repos/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]

@ViktorHofer

@ViktorHofer @MichalStrehovsky can someone take a look at this?

@mangod9 I Cc'd @dotnet/crossgen-contrib but there's no volunteer, could you nominate one? I assume this is blocking source build in .NET 7, but I don't know anything about source build or "packs".

@trylek

Is this required for 7? @trylek @AntonLapounov should be able to help here.

Strictly speaking it is not required as we are capable of building .NET 7.

We should still aim to fix this for .NET 7 as it enables reducing source-build's build time by 25% and more, see dotnet/installer#14549 (comment).

@trylek

@trylek @AntonLapounov should be able to help here.

@trylek @AntonLapounov can one of you ptal?

Are you seeing that failure in the ResolveRuntimePackAssets task when doing the following sequence of commands with the diff given above? I have no prior knowledge of that part of SDK; however, it should not be difficult to debug this.

./build.sh /p:DotnetBuildFromSource=true
dotnet pack Microsoft.NETCore.App.Crossgen2.sfxproj

@ViktorHofer

I'll try to capture what happens when source-build runs this.

@ViktorHofer @AntonLapounov it doesn't look like the top-level verbosity ripples down to the MSBuild tasks. How can I get more verbose logging of what happens in these MSBuild tasks:

runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj

Lines 32 to 48 in c325a5a

    
           <MSBuild Projects="$(RepoRoot)src/coreclr/tools/aot/crossgen2/crossgen2.csproj" 
        
                        Targets="Restore" 
        
                        Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid()) 
        
                         ;RunningPublish=true 
        
                         ;RuntimeIdentifier=$(PackageRID) 
        
                         ;CoreCLRArtifactsPath=$(CoreCLRArtifactsPath) 
        
                         ;R2ROverridePath=$(MSBuildThisFileDirectory)ReadyToRun.targets" /> 
        
               <MSBuild Projects="$(RepoRoot)src/coreclr/tools/aot/crossgen2/crossgen2.csproj" 
        
                        Targets="Publish;PublishItemsOutputGroup" 
        
                        Properties="RunningPublish=true 
        
                         ;RuntimeIdentifier=$(PackageRID) 
        
                         ;CoreCLRArtifactsPath=$(CoreCLRArtifactsPath) 
        
                         ;R2ROverridePath=$(MSBuildThisFileDirectory)ReadyToRun.targets"> 
        
                 <Output TaskParameter="TargetOutputs" 
        
                         ItemName="_RawCrossgenPublishFiles" /> 
        
               </MSBuild>

@AntonLapounov

@AntonLapounov if you want to take a stab at this.

You can perform a build like this:

./build.sh /p:DotNetBuildFromSource=true --outputrid banana.10-x64 /p:RuntimeOS=linux /p:AdditionalRuntimeIdentifierParent=linux

The Crossgen2 package should be built using Microsoft.NETCore.App.Runtime.banana.10-x64 package, instead of the pre-built Microsoft.NETCore.App.Runtime.linux-x64 package.

@agocke

Maybe this is already consuming the runtime pack that got built?

What does this do:

runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj

Lines 56 to 58 in f156fb9

    
           <ResolvedRuntimePack Update="Microsoft.NETCore.App.Runtime.$(RuntimeIdentifier)"> 
        
             <PackageDirectory>$(MicrosoftNetCoreAppRuntimePackDir)</PackageDirectory> 
        
           </ResolvedRuntimePack>

And, what is the contents of MicrosoftNetCoreAppRuntimePackDir? Is this a package that got extracted, or was this built from source?

cc @agocke @am11

Yes, that pulls the runtime pack that just got built

Btw, <NativeAotSupported property is set in two places with different conditions:

runtime/eng/Subsets.props

Line 102 in d6fdb3a

    
           <NativeAotSupported Condition="('$(TargetOS)' == 'windows' or '$(TargetOS)' == 'linux' or '$(TargetOS)' == 'OSX') and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')">true</NativeAotSupported>

runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj

Line 8 in d6fdb3a

    
           <NativeAotSupported Condition="'$(DotNetBuildFromSource)' == 'true'">false</NativeAotSupported>

maybe it's better to merge the condition in crossgen2 into the top-level Subsets.props to avoid any unexpected build state. It is basically controlling "after compiling crossgen2 assembly, whether to publish crossgen2 as nativeaot app (NativeAotSupported=true) or a singlefileapp (NativeAotSupported=false)".

Yes, that pulls the runtime pack that just got built

Great! I can just close this then.

Navigation Menu

Repository navigation

Microsoft.NETCore.App.Crossgen2.sfxproj in source-build #74721

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	<MSBuild Projects="$(RepoRoot)src/coreclr/tools/aot/crossgen2/crossgen2.csproj"
	Targets="Restore"
	Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid())
	;RunningPublish=true
	;RuntimeIdentifier=$(PackageRID)
	;CoreCLRArtifactsPath=$(CoreCLRArtifactsPath)
	;R2ROverridePath=$(MSBuildThisFileDirectory)ReadyToRun.targets" />

	<MSBuild Projects="$(RepoRoot)src/coreclr/tools/aot/crossgen2/crossgen2.csproj"
	Targets="Publish;PublishItemsOutputGroup"
	Properties="RunningPublish=true
	;RuntimeIdentifier=$(PackageRID)
	;CoreCLRArtifactsPath=$(CoreCLRArtifactsPath)
	;R2ROverridePath=$(MSBuildThisFileDirectory)ReadyToRun.targets">
	<Output TaskParameter="TargetOutputs"
	ItemName="_RawCrossgenPublishFiles" />
	</MSBuild>

	<ResolvedRuntimePack Update="Microsoft.NETCore.App.Runtime.$(RuntimeIdentifier)">
	<PackageDirectory>$(MicrosoftNetCoreAppRuntimePackDir)</PackageDirectory>
	</ResolvedRuntimePack>

Microsoft.NETCore.App.Crossgen2.sfxproj in source-build #74721

Description

Activity

jkotas commented on Aug 28, 2022

ViktorHofer commented on Sep 13, 2022

tmds commented on Sep 14, 2022

hoyosjs commented on Sep 20, 2022

tmds commented on Sep 21, 2022

ericstj commented on Sep 21, 2022

9 remaining items

tmds commented on Sep 29, 2022

tmds commented on Sep 29, 2022

MichalStrehovsky commented on Sep 29, 2022

mangod9 commented on Sep 30, 2022

tmds commented on Sep 30, 2022

tmds commented on Oct 4, 2022

AntonLapounov commented on Oct 6, 2022

tmds commented on Oct 10, 2022

tmds commented on Oct 10, 2022

tmds commented on Oct 10, 2022

agocke commented on Oct 10, 2022

am11 commented on Oct 10, 2022

tmds commented on Oct 11, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions