Repository navigation
Failures to restore from Azure artifacts feeds due to throttling #4190
Description
We have started seeing some throttling errors when attempting to restore NuGet packages from Azure Artifacts that manifest like this:
/root/coresetup/.dotnet/sdk/3.0.100/NuGet.targets(123,5): error : Failed to download package 'transport.runtime.linux-musl-x64.Microsoft.NETCore.Jit.3.1.0-preview1.19504.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/9c2ea29a-00e0-4bae-b470-161fdab1f360/nuget/v3/flat2/transport.runtime.linux-musl-x64.microsoft.netcore.jit/3.1.0-preview1.19504.3/transport.runtime.linux-musl-x64.microsoft.netcore.jit.3.1.0-preview1.19504.3.nupkg'.
/root/coresetup/.dotnet/sdk/3.0.100/NuGet.targets(123,5): error : Response status code does not indicate success: 429 (Request was blocked due to exceeding usage of resource 'Concurrency' in namespace 'IPAddress'. For more information on why your request was blocked, see the topic "Rate limits" on the Microsoft Web site (https://go.microsoft.com/fwlink/?LinkId=823950). (DevOps Activity ID: 5B41D91F-6ED5-41D1-814B-0328F8821422)).
##[error]/root/coresetup/.dotnet/sdk/3.0.100/NuGet.targets(123,5): error : Failed to download package 'transport.runtime.linux-musl-x64.Microsoft.NETCore.Jit.3.1.0-preview1.19504.3' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/9c2ea29a-00e0-4bae-b470-161fdab1f360/nuget/v3/flat2/transport.runtime.linux-musl-x64.microsoft.netcore.jit/3.1.0-preview1.19504.3/transport.runtime.linux-musl-x64.microsoft.netcore.jit.3.1.0-preview1.19504.3.nupkg'.
This does not appear to be causing widespread failures so far, but as we increase our reliance on these feeds, we're starting to get more reports.
Example builds where this has been seen:
https://dev.azure.com/dnceng/public/_build/results?buildId=398862
https://dnceng.visualstudio.com/internal/_build/results?buildId=384184
All unauthenticated NuGet requests are getting thrown into the same throttling bucket by AzDO, and due to the multi-feed lookup for each package that NuGet does, depending on how many feeds you have in your NuGet config, and how many packages you need to restore, the problem gets worse.
The short term suggestions from the AzDO team are:
- Reduce the number of feeds used (Their docs mention we should only be using a single feed for restores)
- Use private feeds instead of public, that way NuGet won't make unauthenticated requests
- Change NuGet.config to limit the maxHttpRequestsPerSource:
<config>
<add key='maxHttpRequestsPerSource' value='2' />
</config>
We are still in talks with the AzDO team as these workarounds will end up requiring a lot of changes to our infrastructure for this.
Activity
JohnTortugo commentedon Oct 24, 2019
Happened again here: https://dev.azure.com/dnceng/public/_build/results?buildId=399891&view=logs&j=09811346-8274-5b72-2d96-1dd38f87c84b
jashook commentedon Dec 6, 2019
Runtime-coreclr outerloop hit this multiple times yesterday.
https://dev.azure.com/dnceng/public/_build/results?buildId=447821
https://dev.azure.com/dnceng/public/_build/results?buildId=447950
https://dev.azure.com/dnceng/public/_build/results?buildId=448056
https://dev.azure.com/dnceng/public/_build/results?buildId=448358
https://dev.azure.com/dnceng/public/_build/results?buildId=448386
https://dev.azure.com/dnceng/public/_build/results?buildId=448795
JohnTortugo commentedon Dec 19, 2019
@riarenas - Is this still an thing?
safern commentedon Apr 8, 2020
@riarenas thanks for looking into this, do we have an ETA?
would it make sense to bring back the dotnet blob feed back as a restore source in the meantime?
safern commentedon Apr 8, 2020
Thanks @riarenas
nattress commentedon Apr 29, 2020
I'm seeing this today in the runtime CI:
https://dev.azure.com/dnceng/public/_build/results?buildId=623702&view=logs&jobId=a21a5f2c-b5cf-50e3-b899-59e91a62c520&j=240def99-ab66-58b8-7d7e-e9b53327ef3c&t=0d65e4e8-9786-5617-5520-c80e454608a0
markwilkie commentedon May 5, 2020
Ok - AzDO is saying they've fixed it (for real) now.