Some tests timing out on helix

[radical]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618&view=results
Build error leg or test failing: Aspire.Hosting.Elasticsearch.Tests.WorkItemExecution
Pull request: #5243

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Aborting test run: test run timeout of [0-9]+ milliseconds exceeded",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=804618
Error message validated: [Aborting test run: test run timeout of [0-9]+ milliseconds exceeded]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/11/2024 7:28:44 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[radical]

cc Eric Erhardt (@eerhardt) Sébastien Ros (@sebastienros)

One of the tests timing out is Aspire.Hosting.Elasticsearch.Tests - log.

Why is this timing out after 10minutes on helix when all the tests combined didn't take that long on the build machine?

• One reason could be that the helix agents are under-powered compared to the build machines. And these tests start up a new app per xunit test in ElasticsearchFunctionalTests.

[radical]

I chose a broader error message to match against, so we can get any tests that are timing out on helix now.

[mitchdenny]

I am still hitting this today on this PR: #5223

[radical]

I am still hitting this today on this PR: #5223

I'll bump the timeouts. We are hitting new ones as we moved new tests to helix last week.

[radical]

Aspire.Hosting.Elasticsearch.Tests - I think this is timing out because each of the functional tests starts a new app, and the elasticsearch containers take a few minutes to start up, causing the time taken for all the tests to be more than 10 minutes.

[mitchdenny]

This just happened for mongo in the playground tests. I was briefly able to repro a timeout locally as well, but when I stopped the test and retried it worked fine (multiple times). When it was timing out I noticed that mongo express didn't have an external port allocated. Make me think that Docker failed to forward the port into the container. This would explain it waiting forever to be able to connect to the container via the endpoint.

[joperezr]

Ankit Jain (@radical) Is there any additional logging that we can add to help us narrow down what is going on here? It would be good if we can make this issue somehow actionable.