Failures during symbol publishing step

Multiple reports of failures during the symbol publishing job that look like:

.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : Failed processing D:\a\1\a\PDBArtifacts\VS.Web.CG.Design\net461\dotnet-aspnet-codegenerator-design.pdb Blob:3c9f66097f09e7f701c0b53b26791a00561f31b587e665a7a0b9b38becab547300 index dotnet-aspnet-codegenerator-design.pdb/aca39894871b497d90298a438424d3821/dotnet-aspnet-codegenerator-design.pdb
.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : 1 exceptions occurred during processing. First 1 exception(s) stored. (Object reference not set to an instance of an object.)
.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : Object reference not set to an instance of an object.
.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : AttemptCount 0
.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : LastBackoff 
.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.19556.10\tools\SdkTasks\PublishToSymbolServers.proj(86,5): error : 1 exceptions occurred during processing. First 1 exception(s) stored. (Object reference not set to an instance of an object.)
at Microsoft.SymbolUploader.PublishOperation.PublishFilesAsync(SymbolHttpClient httpClient, CancellationToken cancellationToken, IEnumerable`1 fileInfos) in /_/src/Microsoft.SymbolUploader/PublishOperation.cs:line 521
at Microsoft.SymbolUploader.PublishOperation.PerformOperationAsync(IEnumerable`1 fileInfos) in /_/src/Microsoft.SymbolUploader/PublishOperation.cs:line 478
at Microsoft.SymbolUploader.PublishOperation.PerformOperationAsync(IEnumerable`1 fileInfos) in /_/src/Microsoft.SymbolUploader/PublishOperation.cs:line 481
at Microsoft.SymbolUploader.PublishOperation.PublishFiles(IEnumerable`1 fileInfos) in /_/src/Microsoft.SymbolUploader/PublishOperation.cs:line 292
at Microsoft.SymbolUploader.Build.Task.PublishSymbolsHelper.Publish(TaskLoggingHelper log, String symbolServerPath, String personalAccessToken, IEnumerable`1 inputPackages, IEnumerable`1 inputFiles, HashSet`1 packageExcludeFiles, String requestName, Int32 expirationInDays, DateTime expirationDate, Boolean convertPortablePdbsToWindowsPdbs, HashSet`1 pdbConversionTreatAsWarning, Boolean dryRun, Boolean timer, Boolean verboseLogging) in /_/src/Microsoft.SymbolUploader.Build.Task/PublishSymbols.cs:line 335
Build failed.

Example builds:
https://dnceng.visualstudio.com/internal/_build/results?buildId=419023
https://dnceng.visualstudio.com/internal/_build/results?buildId=417308

@lukas-lansky started looking into this so assigning to him.

@JohnTortugo in case he has any input here. Does the symbol uploader tool require some hardening?

[dagood]

Does the symbol uploader tool require some hardening?

+ @mikem8361

[JohnTortugo]

I'm also looking into this.

[mikem8361]

Because of the way exceptions are aggregated where exactly the original NullReferenceException was thrown was lost. It looks like it is a result of a temporary network problem on the symweb endpoint.

The default msbuild logging level used here is too low for the original exception stack frame to be displayed. It is logged at MessageImportance.Normal (-verbosity normal).

The NullReferenceException definitely needs to be fixed but I'm sure it is either network time out or error path. I can make some simple fixes to log all inner exceptions in the aggregated one to diagnose this in the future.

[JohnTortugo]

Thanks a lot for the input @mikem8361. I re-started the failing jobs. I'll post an update here once they finish.

[JohnTortugo]

Looks like it was indeed a transient problem, both retries passed:

https://dnceng.visualstudio.com/internal/_build/results?buildId=417308&view=logs&j=008ffa3d-fe44-5900-d778-99915720ce87&t=9c22da64-94ab-5d40-4d29-b0d3225bfd52&l=49
https://dnceng.visualstudio.com/internal/_build/results?buildId=419023&view=logs&j=008ffa3d-fe44-5900-d778-99915720ce87&t=9c22da64-94ab-5d40-4d29-b0d3225bfd52&l=54

I'll create an issue to track the improvements in symb uploader and close the current issue.

[mikem8361]

Can you create the new issue in the symstore repo, assign it to me and add the "symbol-uploader" label

[JohnTortugo]

Issue is here: dotnet/symstore#148 . I don't have permission to add labels.