On ppc64le, VMR at 8 Preview 6 doesn't build using itself

[omajid]

Describe the Bug

We are trying to confirm that source-build can build using itself.

This works on x64. We can build .NET 8 Preview 6 offline (after using the bits downloaded by prep.sh), then use what we just-built to build .NET 8 Preview 6 itself completely offline.

This doesn't work on ppc64le. We can build .NET 8 Preview 6 using a cross-compiled ppc64le SDK and the bits downloaded by prep.sh. We can do this offline too. But then using that just-built SDK to build .NET 8 Preview 6 again fails on ppc64le. The error happens when building runtime.

Steps to Reproduce

Other Information

    'ARCADE_SHARED_FX_SDK' for 'Microsoft.DotNet.SharedFramework.Sdk/8.0.0-beta.23371.1' at 'dotnet/artifacts/source-built-sdks/Microsoft.DotNet.SharedFramework.Sdk/'
    'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/8.0.0-beta.23371.1' at 'dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/'
    'ARCADE_PACKAGING' for 'Microsoft.DotNet.Build.Tasks.Packaging/8.0.0-beta.23371.1' at dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Build.Tasks.Packaging/'
    [SourceBuiltSdkResolver] Overriding Microsoft.DotNet.SharedFramework.Sdk with 'ARCADE_SHARED_FX_SDK'
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error : Failure has occurred while loading a type. [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Shared.ExceptionHandling.NotExpectedIoOrXmlException(Exception e) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement.LoadDocument(String fullPath, Boolean preserveFormatting, Boolean loadAsReadOnly) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement..ctor(String path, ProjectRootElementCacheBase projectRootElementCache, Boolean preserveFormatting) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement.CreateProjectFromPath(String projectFile, ProjectRootElementCacheBase projectRootElementCache, Boolean preserveFormatting) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement.<>c.<OpenProjectOrSolution>b__209_0(String path, ProjectRootElementCacheBase cache) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Evaluation.ProjectRootElementCache.GetOrLoad(String projectFile, OpenProjectRootElement loadProjectRootElement, Boolean isExplicitlyLoaded, Nullable`1 preserveFormatting) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Evaluation.ProjectRootElementCache.Get(String projectFile, OpenProjectRootElement loadProjectRootElement, Boolean isExplicitlyLoaded, Nullable`1 preserveFormatting) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement.OpenProjectOrSolution(String fullPath, IDictionary`2 globalProperties, String toolsVersion, ProjectRootElementCacheBase projectRootElementCache, Boolean isExplicitlyLoaded) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Evaluation.Evaluator`4[[Microsoft.Build.Execution.ProjectPropertyInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectMetadataInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemDefinitionInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExpandAndLoadImportsFromUnescapedImportExpression(String directoryOfImportingFile, ProjectImportElement importElement, String unescapedExpression, Boolean throwOnFileNotExistsError, List`1& imports) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Evaluation.Evaluator`4[[Microsoft.Build.Execution.ProjectPropertyInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectMetadataInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemDefinitionInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExpandAndLoadImportsFromUnescapedImportExpressionConditioned(String directoryOfImportingFile, ProjectImportElement importElement, List`1& projects, SdkResult& sdkResult) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Evaluation.Evaluator`4[[Microsoft.Build.Execution.ProjectPropertyInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectMetadataInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Execution.ProjectItemDefinitionInstance, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExpandAndLoadImports(String directoryOfImportingFile, ProjectImportElement importElement, SdkResult& sdkResult) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
...
...
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Execution.ProjectInstance.FromFile(String file, ProjectOptions options) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.<>c__DisplayClass38_0.<LoadProjects>b__0(String path, Dictionary`2 properties, ProjectCollection collection) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Graph.GraphBuilder.ParseProject(ConfigurationMetadata configurationMetadata) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Graph.GraphBuilder.<>c__DisplayClass35_0.<SubmitProjectForParsing>b__0() [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at System.Lazy`1[[Microsoft.Build.Graph.ParsedProject, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ViaFactory(LazyThreadSafetyMode mode) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at System.Lazy`1[[Microsoft.Build.Graph.ParsedProject, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at System.Lazy`1[[Microsoft.Build.Graph.ParsedProject, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].CreateValue() [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at System.Lazy`1[[Microsoft.Build.Graph.ParsedProject, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].get_Value() [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
  new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Graph.ParallelWorkSet`2[[Microsoft.Build.BackEnd.ConfigurationMetadata, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Microsoft.Build.Graph.ParsedProject, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecuteWorkItem() [dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
    Nothing to do. None of the projects specified contain packages to restore.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

cc @janani66 @Swapnali911

[Swapnali911]

Able to reproduce on x86 as well with mono

/root/jenkins-scripts/ci/check_sb/new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error : Failure has occurred while loading a type. [/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
    /root/jenkins-scripts/ci/check_sb/new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Shared.ExceptionHandling.NotExpectedIoOrXmlException(Exception e) [/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
    /root/jenkins-scripts/ci/check_sb/new_tarball/sdk/8.0.100-preview.6.23330.1/NuGet.RestoreEx.targets(19,5): error :    at Microsoft.Build.Construction.ProjectRootElement.LoadDocument(String fullPath, Boolean preserveFormatting, Boolean loadAsReadOnly) [/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/Build.proj]
...
...
 ##vso[task.logissue type=error;sourcepath=/root/jenkins-scripts/ci/check_sb/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets;linenumber=195;columnnumber=5;code=MSB3073;]The command "./build.sh  --arch x64 --configuration Release --allconfigurations --verbosity minimal --nodereuse false --warnAsError false --outputrid rhel.8-x64 /p:PackageOS=rhel.8 /p:ToolsOS=rhel.8 /p:AdditionalRuntimeIdentifierParent=linux /p:ArcadeBuildFromSource=true /p:OfficialBuildId=20230629.7 /p:ContinuousIntegrationBuild=true --usemonoruntime /p:ArcadeInnerBuildFromSource=true /p:DotNetBuildFromSource=true /p:RepoRoot=/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/ /p:ArtifactsDir=/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/ /bl:/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/sourcebuild.binlog /p:ContinuousIntegrationBuild=true /p:SourceBuildOutputDir=/root/jenkins-scripts/ci/check_sb/dotnet/src/runtime/artifacts/source-build/ /p:SourceBuiltBlobFeedDir= /p:DotNetBuildOffline=true /p:DotNetPackageVersionPropsPath=/root/jenkins-scripts/ci/check_sb/dotnet/artifacts/obj/x64/Release/PackageVersions.runtime.props /p:FullAssemblySigningSupported=false" exited with code 1.

  Build FAILED.

[uweigand]

Hi @MichaelSimons it seems this failure happens in or near the IsXmlException routine that already caused problems during our initial 6.0 build due to .NET Framework 4.7 dependencies described here: #2618

The message "Failure has occurred while loading a type." also sounds suspiciously like the above problem.

Not sure if this is just a coincidence or if that problem is coming back somehow?

[tmds]

As was said earlier, the issue can be reproduce on x64.
I can reproduce it with an SDK built from https://github.com/dotnet/dotnet release/8.0.1xx branch (with the --use-mono-runtime argument).

With this SDK, the issue can be reproduced by restoring an invalid project file:

1. Create a new console project:

dotnet new console -o console

2. Edit the console/console.csproj file to make it invalid, for example add a / on the first line.

Run dotnet restore console.

3. The restore fails while trying to load some type during NotExpectedIoOrXmlException:

MSBUILD : error : This is an unhandled exception in MSBuild -- PLEASE UPVOTE AN EXISTING ISSUE OR FILE A NEW ONE AT https://aka.ms/msbuild/unhandled [/tmp/repro/console/console.csproj]
MSBUILD : error :     System.TypeLoadException: Failure has occurred while loading a type. [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Shared.ExceptionHandling.NotExpectedIoOrXmlException(Exception e) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Construction.ProjectRootElement.LoadDocument(String fullPath, Boolean preserveFormatting, Boolean loadAsReadOnly) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Construction.ProjectRootElement..ctor(String path, ProjectRootElementCacheBase projectRootElementCache, Boolean preserveFormatting) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Construction.ProjectRootElement.CreateProjectFromPath(String projectFile, ProjectRootElementCacheBase projectRootElementCache, Boolean preserveFormatting) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Construction.ProjectRootElement.<>c.<OpenProjectOrSolution>b__209_0(String path, ProjectRootElementCacheBase cache) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Evaluation.ProjectRootElementCache.GetOrLoad(String projectFile, OpenProjectRootElement loadProjectRootElement, Boolean isExplicitlyLoaded, Nullable`1 preserveFormatting) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Evaluation.ProjectRootElementCache.Get(String projectFile, OpenProjectRootElement loadProjectRootElement, Boolean isExplicitlyLoaded, Nullable`1 preserveFormatting) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Construction.ProjectRootElement.OpenProjectOrSolution(String fullPath, IDictionary`2 globalProperties, String toolsVersion, ProjectRootElementCacheBase projectRootElementCache, Boolean isExplicitlyLoaded) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.Execution.ProjectInstance..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, BuildParameters buildParameters, ILoggingService loggingService, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, Nullable`1 projectLoadSettings) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.BackEnd.BuildRequestConfiguration.<>c__DisplayClass61_0.<LoadProjectIntoConfiguration>b__0() [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.BackEnd.BuildRequestConfiguration.InitializeProject(BuildParameters buildParameters, Func`1 loadProjectFromFile) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.BackEnd.BuildRequestConfiguration.LoadProjectIntoConfiguration(IBuildComponentHost componentHost, BuildRequestDataFlags buildRequestDataFlags, Int32 submissionId, Int32 nodeId) [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildProject() [/tmp/repro/console/console.csproj]
MSBUILD : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport() [/tmp/repro/console/console.csproj]

@akoeplinger @lambdageek can you help debug this further?

[uweigand]

Hi @tmds we've had some more progress in tracking this down. Enabling Mono logging shows the following failure:

Mono: The following assembly referenced from /root/sb_issue/new_tarball/sdk/8.0.100-preview.6.23330.1/Microsoft.Build.dll could not be loaded:
     Assembly:   System.Security.Permissions    (assemblyref_index=34)
     Version:    7.0.0.0
     Public Key: cc7b13ffcd2ddd51

This is consistent with the error message, as the NotExpectedIoOrXmlException routine references a type that is defined in the System.Security.Permissions assembly. This is the same symptom we saw back in the .NET 6 days - but in those days msbuild was trying to refer to a .NET Framework 4.7 version of System.Security.Permissions.

What is still unclear is why it is looking for the 7.0.0 version here. My understanding is that in source-build, this version string should be overridden to refer to the version built as part of the source-build runtime (i.e. a 8.0.0-preview something version). We are still trying to figure out why there's still a 7.0.0 reference remaining here.

[tmds]

System.Security.Permissions.

When I disassemble this dll (from the sdk folder), it looks like a reference assembly (throw null; ...), while the corresponding dll from a coreclr build does not.

[uweigand]

When I disassemble this dll (from the sdk folder), it looks like a reference assembly (throw null; ...), while the corresponding dll from a coreclr build does not.

Interesting ... is the corresponding dll from a coreclr build also version 7.0.0 or something else?