source-build on unknown linux distros

@ericstj

For source-build we want to be able to build .NET using source-build on a system where the rid is not known to .NET.

We want this build to succeed, and the rid to be known to .NET.

For the build to succeed, we need to build using another rid which is known and compatible. I think this may be possible already (to some extent) by using the DOTNET_RUNTIME_ID envvar.

For the rid to be known to .NET, it needs to know its place in the graph.

Example pseudo build commands:

Build on Fedora 35 (rid derived from os-release is fedora.35):

./build.sh --build-rid linux --parent-rids fedora,linux

Build on RHEL 8 (rid derived from os-release is rhel.8):

./build.sh --build-rid linux --parent-rids linux

Build on CentOS 8 (rid derived from os-release is centos.8):

./build.sh --build-rid linux --parent-rids rhel.8,linux

Relates to NuGet/Home#5862, dotnet/source-build#297.

cc @ericstj @omajid @crummel

@tmds

Thanks for filing this issue, @tmds ! I think this captures our issues with RID well: we just need a way for a random Linux distro to get an RID automatically, and assign some parents RIDs (mostly linux, but maybe musl too).

cc @dagood @MichaelSimons @dleeapho

For the build to succeed, we need to build using another rid which is known and compatible. I think this may be possible already (to some extent) by using the DOTNET_RUNTIME_ID envvar.

It would be good to understand if this is sufficient or if you would also need to modify the shared-framework's deps file (where the RID fallbacks live) in the tool shared framework. Doing the latter is possible but grows the complexity of this.

runtime/src/libraries/Microsoft.NETCore.Platforms/pkg/runtimeGroups.props

Lines 248 to 258 in 09a1256

    
             <Target Name="GenerateRuntimeJson" BeforeTargets="CreatePackage"> 
        
               <!-- Generates a Runtime graph using RuntimeGroups and diffs it with the graph described by runtime.json and runtime.compatibility.json 
        
                    Specifying UpdateRuntimeFiles=true skips the diff and updates those files. 
        
                    The graph can be visualized using the generated dmgl --> 
        
               <MakeDir Directories="$(PackageReportDir)" /> 
        
               <GenerateRuntimeGraph RuntimeGroups="@(RuntimeGroupWithQualifiers)" 
        
                                     RuntimeJson="runtime.json" 
        
                                     CompatibilityMap="runtime.compatibility.json" 
        
                                     RuntimeDirectedGraph="$(PackageReportDir)$(Id)$(NuspecSuffix)-runtime.json.dgml" 
        
                                     UpdateRuntimeFiles="$(UpdateRuntimeFiles)" /> 
        
             </Target>

can be used to generate, today it writes to the repo location but that could be overridden. We'd need to expose the right set of options to extend the graph. Then when building make sure that the generated runtime.json was used everywhere instead of the checked in one.

Also, once you produce such a shared framework, are you OK with nuget restore not knowing about? NuGet still gets the runtimegraph from the Micrsoft.NETCore.Platforms package which the user's package graph brings down. This is what is used for resolving and publishing self-contained apps.

Also, once you produce such a shared framework, are you OK with nuget restore not knowing about?

Is there a fallback here? To something like linux-x64? Or is it the case that if nuget doesn't know about the RID, restore wont work at all for RID-specific assets?

This is what is used for resolving and publishing self-contained apps.

Does that mean self-contained apps wont build/work? What about normal Framework-Dependent apps?

@dsplaisted

I may be mistaken. It looks like the SDK did the work to pull out the runtime.json, bundle it in the SDK and provide it to NuGet. Thanks @dsplaisted, @nkolev92
NuGet/NuGet.Client@04166a6
dotnet/sdk@2723589

Assuming we flow the built content of Microsoft.NETCore.Platfroms from sourcebuild up through the SDK/installer then it looks like this should work. NuGet will "merge" the contents of the SDK's runtimegraph with whatever is defined in packages. So even if someone added the public shipped platforms package it would still have your source-build defined RIDs.

It would be good to understand if this is sufficient or if you would also need to modify the shared-framework's deps file (where the RID fallbacks live) in the tool shared framework. Doing the latter is possible but grows the complexity of this.

If it doesn't work already, probably we can pick up DOTNET_RUNTIME_ID in a few more places and avoid the complexity.

@tmds

@tmds can you better specify the inputs of what you'd like to see happen?

Describe what needs to be specified, vs what is calculated from the system, and how that is used by the build system. Also talk about how things interact with existing properties / parameters to the build

Interesting considerations:

Is cross-targeting relevant, or perhaps required? Both by arch and distro. IOW will you always be building on the system that exposes the RID you're trying to add or are you sometimes building on a different RID?
Does this also need to impact the configuration system in dotnet/runtime libraries for selecting which configurations to build?

@ericstj

I don't know how this would interact with the existing properties/parameters, and configurations. We need to figure that out.

When we use source-build there are two steps:

First https://github.com/dotnet/source-build is used on a system to generate a tarball. The rid may be not known to .NET so we want to be able to specify a compatible rid (SourceBuildRid).
Then that tarball is used on another system (TargetRid, e.g. fedora.32-x64) to generate a .NET installer for that system.

We would like the tarball to be usable on a range of target distros. So, the tarball would target something common (BuildRid, like linux-x64).

TargetRid can be a unknown rid to .NET. As part of the build, .NET needs to learn how that rid connects to known rids (TargetRidGraph, e.g. fedora.32 -> fedora -> linux).

For example. We run source-build on linux-x64 to generate a tarball that is usable on linux-arm64, that is used on fedora.32-arm64, with rid graph fedora.32 -> fedora -> linux, gives us these values:

SourceBuildRid=linux-x64
BuildRid=linux-arm64
TargetRid=fedora.32-arm64
TargetRidGrah=fedora,linux

Describe what needs to be specified, vs what is calculated from the system

TargetRid is calculated when building .NET. The other values can be specified.

Is cross-targeting relevant, or perhaps required?

We don't have a need to generate cross-target tarballs. So SourceBuildRid = BuildRid for our use-cases.
We do want the source-build tarball to be usable on multiple platforms (TargetRid is compatible with BuildRid).

@ericstj is it more clear?

cc @MichaelSimons @dleeapho @omajid

@ericstj

@ericstj are you, or is someone else looking at this?

Yeah, I'm actually preparing a draft PR right now. Should be available in a hour or so.

FYI: draft PR #50157
Arcade change to map RID additions into the RID graph honoring rules defined in existing RuntimeGroups: dotnet/arcade#7141

Please review user experience here to make sure it meets your needs.

If I understand the PRs correctly, they infer the current OS's RID, guess the parent RID, and add the current RID to the RID graph (if missing).

Going back to the initial examples:

Build on Fedora 35 (rid derived from os-release is fedora.35):

./build.sh --build-rid linux --parent-rids fedora,linux

The parent-rids part is inferred correctly, and the build-rid is controlled via --portableBuild?

Build on RHEL 8 (rid derived from os-release is rhel.8):

./build.sh --build-rid linux --parent-rids linux

Same?

Build on CentOS 8 (rid derived from os-release is centos.8):

./build.sh --build-rid linux --parent-rids rhel.8,linux

This wouldn't work, would it? It will say centos.8 inherits from centos (which inherits from linux), right? (I am okay with that, just want to make sure I understand this correctly).

they infer the current OS's RID

Infer was probably bad naming on my part: we infer the place in the graph but the RID is specified. There are already mechanisms today for determining what RID is being built. It can be specified or inferred. We leverage these rather than introducing anything new.

I've also added a property called InferRuntimeIdentifiers which can be use to add an arbitrary set of RIDs, where they belong, in the graph.

guess the parent RID

There is no guessing at parent, this follows the rules which are described in the checked in RuntimeGroups. So rather than let the caller specify a different hierarchy they add the new RID to the existing hierarchy where it belongs. As much as I can I'm trying to make it behave "as if" the change was checked into RuntimeGroups.props.

The above cases you describe should all work correctly. In those cases it looks like you're trying to build the product as if it is a linux-<arch> build but then also add RIDs to the graph for some new version of a known distro. In those cases you would need to specify the InferRuntimeIdentifiers (which I think is behaving like the --parent-rids command). So InferRuntimeIdentifiers=centos.10 would add centos.10 RIDs to the graph as if the change was made here, inheriting from rhel:

runtime/src/libraries/Microsoft.NETCore.Platforms/pkg/runtimeGroups.props

Lines 51 to 57 in 366cd23

    
           <RuntimeGroup Include="centos"> 
        
             <Parent>rhel</Parent> 
        
             <Architectures>x64;arm64</Architectures> 
        
             <Versions>8;9</Versions> 
        
             <ApplyVersionsToParent>true</ApplyVersionsToParent> 
        
             <TreatVersionsAsCompatible>false</TreatVersionsAsCompatible> 
        
           </RuntimeGroup>

I think I'll rename InferRuntimeIdentifiers to AddRuntimeIdentifiers. I'll also refine some of the tests in arcade to better illustrate how this is working. I

build but then also add RIDs to the graph for some new version of a known distro. In those cases you would need to specify the InferRuntimeIdentifiers (which I think is behaving like the --parent-rids command)

--parent-rids was meant to specify the known parent. The GenerateRuntimeGraph implementation is actually deferring this from the input provided. For our use-cases the deferring works fine since the distros we care about are already known.
For general source-build, you may want to support an unknown distro (which, for example, should inherit from linux-musl).

The InferRuntimeIdentifiers/AddRuntimeIdentifiers allows to specify rids to add. For our use-cases, where we build on the same OS that we target, we will want this value to default to what dotnet considers the rid for the build platform (derived from /etc/os-release on Linux).

Independent of the GenerateRuntimeGraph, we also want to be able to override what .NET considers the rid of the build platform. This may be done by using/extending DOTNET_RUNTIME_ID. When the default from the previous paragraph is determined we want to get the actual platform value, and not the overridden value.

For general source-build, you may want to support an unknown distro (which, for example, should inherit from linux-musl).

Who's going to compute the correct parent? The end user who's running the source build? Might you need to add more than one? Supporting this is pretty trivial: I added a sample here that assumes linux, we can come up with scheme to encode arbitrary parents either as item metadata or some format in the string or a separate parameter, but I want to make sure we understand the end scenario. At some point we'll need to say that we don't need to support more complex additions and those require a change in source: IMHO that point should be driven by our current customer scenarios for source build.

where we build on the same OS that we target, we will want this value to default to what dotnet considers the rid for the build platform

I see, I was making the default be the RID of the Packages we build I can change that to the computed RID (though the OutputRID is also derived from the computed RID). I think it's very important to be crisp about the purpose of these properties: we have a number of different RID values in the system already and adding new ones is going to cause confusion and misuse. Today the "computed" RID is retrieved here:

runtime/Directory.Build.props

Line 103 in 6496624

    
           <_parseDistroRid Condition="'$(_parseDistroRid)' == '' and '$(MSBuildRuntimeType)' == 'core'">$([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier)</_parseDistroRid>

Is this the value you'd prefer to be the default?

Independent of the GenerateRuntimeGraph,

I believe this should work correctly.

runtime/src/libraries/System.Runtime.InteropServices.RuntimeInformation/tests/RuntimeIdentifierTests.cs

Lines 24 to 30 in 6496624

    
           RemoteInvokeOptions options = new RemoteInvokeOptions(); 
        
           options.StartInfo.EnvironmentVariables.Add("DOTNET_RUNTIME_ID", "overridenFromEnv-rid"); 
        
           RemoteExecutor.Invoke(() => 
        
           { 
        
               Assert.Equal("overridenFromEnv-rid", RuntimeInformation.RuntimeIdentifier); 
        
           }, options).Dispose();

Who's going to compute the correct parent? The end user who's running the source build? Might you need to add more than one?

The source-build user could specify it. It would be a single value.

At some point we'll need to say that we don't need to support more complex additions and those require a change in source: IMHO that point should be driven by our current customer scenarios for source build.

Yes, this is a scenario we don't need for the distros we target.

Is this the value you'd prefer to be the default?

It's not clear for me what the differences are between these rids. When we build the source-build tarball, then the output packages would be for the rid of /etc/os-release.

Now when we run source-build to create the tarball, we'd like to be able to create a tarball that works across a range of linux-x64 distros. I'm not sure if packages are built for the tarball at this stage. If they are, I assume this means the output packages would be for the linux-x64 rid.

@ericstj

@ericstj does the rid logic make sense?

we'd like to be able to create a tarball that works across a range of linux-x64 distros

As future steps, this tarball could be the output of source-build, produced by Microsoft.

The tarball can be compiled with a pre-built SDK from Microsoft. Or with an SDK that was already source-build compiled for a previous patch version, or for a previous (compatible) version of the distro.

cc @crummel @dseefeld @MichaelSimons @omajid

The source-build user could specify it. It would be a single value.

This makes sense, we can easily add the ability to allow the source-build user to specify a new parent. I'll add that to the current changes. I'm in the propcess of moving this over from dotnet/arcade to dotnet/runtime to make it easier to maintain. Next steps on my side are to finish this port and add it to the draft PR in dotnet runtime and move out of draft.

Is this the value you'd prefer to be the default?

It's not clear for me what the differences are between these rids. When we build the source-build tarball, then the output packages would be for the rid of /etc/os-release.

I was pointing to the existing property reads from $([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier) which should derive it's value from /etc/os-release with an override of DOTNET_RUNTIME_ID.

Here's what I'm imagining the interface with the user. During a source build:

Automatically include $([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier) in RID graph
Automatically include PackageRID|OutputRID in RID graph
Both of above will be added to AdditionalRIDs property
When considering any unknown RID family, choose the parent based on single user-specified parent RID, lets call it AdditionalRIDsParent

This sounds good to me. When we say build, we need to think about two different builds:

When source-build produces a tarball.
When the source-build tarball is used with an SDK to produce a source-built SDK.

For our use-cases. The first build is for linux-x64/linux-arm64. The second build is for the RID of /etc/os-release.

Navigation Menu

Repository navigation

source-build on unknown linux distros #48507

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	<Target Name="GenerateRuntimeJson" BeforeTargets="CreatePackage">
	<!-- Generates a Runtime graph using RuntimeGroups and diffs it with the graph described by runtime.json and runtime.compatibility.json
	Specifying UpdateRuntimeFiles=true skips the diff and updates those files.
	The graph can be visualized using the generated dmgl -->
	<MakeDir Directories="$(PackageReportDir)" />
	<GenerateRuntimeGraph RuntimeGroups="@(RuntimeGroupWithQualifiers)"
	RuntimeJson="runtime.json"
	CompatibilityMap="runtime.compatibility.json"
	RuntimeDirectedGraph="$(PackageReportDir)$(Id)$(NuspecSuffix)-runtime.json.dgml"
	UpdateRuntimeFiles="$(UpdateRuntimeFiles)" />
	</Target>

	<RuntimeGroup Include="centos">
	<Parent>rhel</Parent>
	<Architectures>x64;arm64</Architectures>
	<Versions>8;9</Versions>
	<ApplyVersionsToParent>true</ApplyVersionsToParent>
	<TreatVersionsAsCompatible>false</TreatVersionsAsCompatible>
	</RuntimeGroup>

	RemoteInvokeOptions options = new RemoteInvokeOptions();
	options.StartInfo.EnvironmentVariables.Add("DOTNET_RUNTIME_ID", "overridenFromEnv-rid");

	RemoteExecutor.Invoke(() =>
	{
	Assert.Equal("overridenFromEnv-rid", RuntimeInformation.RuntimeIdentifier);
	}, options).Dispose();

source-build on unknown linux distros #48507

Description

Activity

omajid commented on Feb 19, 2021

ericstj commented on Feb 19, 2021

ericstj commented on Feb 19, 2021

omajid commented on Feb 19, 2021

ericstj commented on Feb 19, 2021

tmds commented on Feb 23, 2021

ericstj commented on Mar 4, 2021

tmds commented on Mar 9, 2021

7 remaining items

tmds commented on Mar 23, 2021

ericstj commented on Mar 23, 2021

ericstj commented on Mar 24, 2021

omajid commented on Mar 24, 2021

ericstj commented on Mar 24, 2021

tmds commented on Mar 25, 2021

ericstj commented on Mar 25, 2021

tmds commented on Mar 25, 2021

tmds commented on Mar 30, 2021

ericstj commented on Mar 30, 2021

tmds commented on Apr 6, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions