[ci-scan] Build break: Windows Helix work items fail on unexpanded $HELIX_PYTHONPATH in gen-debug-dump-docs post-command

[github-actions]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1524472
Build error leg or test failing: Build / android-arm64 Release NativeAOT / Send to Helix — Windows Helix work items (e.g. System.Security.Cryptography.Tests, System.Net.Sockets.Tests)
Pull request: (rolling main build, not a PR build)

Error Details

gen-debug-dump-docs.py invokes the shell
variable $HELIX_PYTHONPATH using Unix syntax. On Windows work items cmd.exe does not
expand $HELIX_PYTHONPATH, so the command is not found and the work item exits with code 1
after the test run itself has already completed and reported results. Every Windows Helix
work item on the affected legs of runtime-extra-platforms fails this way, and the failure
recurs across consecutive builds (source build 1524472 and follow-up build 1526281). -->

D:\h\w\...\e>$HELIX_PYTHONPATH $HELIX_CORRELATION_PAYLOAD/gen-debug-dump-docs.py -buildid 1524472 -workitem $HELIX_WORKITEM_FRIENDLYNAME -jobid $HELIX_CORRELATION_ID -outdir $HELIX_WORKITEM_UPLOAD_ROOT -templatedir $HELIX_CORRELATION_PAYLOAD -dumpdir $HELIX_DUMP_FOLDER -productver 11.0.0
'$HELIX_PYTHONPATH' is not recognized as an internal or external command,
operable program or batch file.

D:\h\w\...\e>EXIT /b 1

Error Message

{
  "ErrorMessage": "'$HELIX_PYTHONPATH' is not recognized as an internal or external command",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

---

Filed by ci-failure-scan (detection only). ci-failure-fix walks open [ci-scan] KBEs and either opens a small fix PR or comments here to loop in owners — it never disables the test.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by CI Outer-Loop Failure Scanner · 360.6 AIC · ⌖ 15.3 AIC · ⊞ 18.6K · ◷

Report

Build	Repository	Test	Pull Request
1626941	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135302
1627097	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135372
1627077	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135370
1627054	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135363
1627048	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135357
1627043	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135367
1627027	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#132727
1626997	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135250
1626990	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135364
1626966	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#134359
1626932	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1626856	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135257
1626853	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135362
1626830	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135360
1626838	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135326
1626835	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135361
1626825	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135209
1626821	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135359
1626776	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135356
1626743	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135355
1626740	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#134883
1626704	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135351
1626658	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135295
1626661	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135318
1626639	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135209
1626618	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135344
1626493	dotnet/runtime	System.Net.Sockets.Tests.SendReceiveMisc.SocketSendWouldBlock_ReturnsBytesSent
1626607	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135326
1626526	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135342
1626478	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#134359
1626454	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#134418
1626440	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#132788
1626430	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#133943
1626401	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias	#135338
1626347	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135337
1626305	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135336
1626278	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135032
1626189	dotnet/runtime	System.Xml.Tests.AttributeReadingPerformanceTests_XmlNodeReader.AttributeDuplicatesCheck_LongUris_DistinctLocalNames_AboveThreshold
1626226	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135335
1626222	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135321
1626192	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135334
1625894	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1625742	dotnet/runtime	System.Net.NameResolution.Functional.Tests.WorkItemExecution
1625588	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#125436
1624965	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	#125436
1624450	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http2.ServerDisconnectDuringSetup_PropagatesMeaningfulException
1624231	dotnet/runtime	System.Security.Cryptography.Pkcs.Tests.WorkItemExecution
1624228	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange
1623598	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#125436
1623410	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#125436
1622866	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#134388
1622443	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1622039	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1621670	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.Open_AuthenticationFailure_FirstCiphertextByteModified(name: "X25519-HKDF256-ChaCha-Base")
1621013	dotnet/runtime	System.Net.Http.Functional.Tests.Http1ResponseStreamZeroByteReadTest.ZeroByteRead_BlocksUntilDataIsAvailable	#135010
1620717	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1620429	dotnet/runtime	System.Net.Http.Functional.Tests.NtAuthTests.Http2_SessionAuthChallenge_SubsequentRequestsUseHttp11
1620034	dotnet/runtime	Microsoft.Bcl.Cryptography.Tests.WorkItemExecution
1619846	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#135029
1619076	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1618934	dotnet/runtime	System.Net.Http.Functional.Tests.NtAuthTests.Http2_SessionAuthChallenge_SubsequentRequestsUseHttp11
1618254	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1617850	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate	#134549
1617342	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1615702	dotnet/runtime	System.Security.Cryptography.Tests.ChaCha20Poly1305Tests.EncryptTamperAADDecrypt
1615485	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate	#134819
1614940	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1614331	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias	#134779
1614142	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip_EmptyAadAndInfo
1613852	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias	#134745
1613681	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias	#134729
1613653	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip_EmptyAadAndInfo
1613174	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1613145	dotnet/runtime	System.Net.Sockets.Tests.ArgumentValidation.Socket_ConnectAsync_IPv6AddressAsStringHost_Supported
1612511	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1612276	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#133197
1611740	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1610794	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias	#134560
1610154	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1610145	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange	#133222
1609955	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1609928	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange
1609316	dotnet/runtime	System.Security.Cryptography.Tests.HpkeKeyTests.ImportDecapsulationKey_KnownAnswerAndInputOwnership
1609227	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpRequestErrorTest_Http20.ConnectionError	#130463
1608288	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip_EmptyAadAndInfo
1608312	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpRequestErrorTest_Http20.ConnectionError	#130463
1608142	dotnet/runtime	System.Security.Cryptography.Tests.HpkeKeyTests.ImportEncapsulationKey_KnownAnswerAndInputOwnership
1608148	dotnet/runtime	System.Security.Cryptography.Tests.HpkeImplementationTests.SingleShot_Roundtrip
1606626	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1606540	dotnet/runtime	System.Net.Http.Functional.Tests.NtAuthTests.Http2_SessionAuthChallenge_SubsequentRequestsUseHttp11
1605355	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.Android_GetCertificateFromKeyStoreViaAlias
1602914	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate	#134060
1604459	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1604252	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange	#133804
1604027	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange
1603402	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.BuildChainForExpiredSelfSignedCertificate
1603249	dotnet/runtime	System.Security.Cryptography.Tests.ChaCha20Poly1305Tests.EncryptTamperAADDecrypt
1602810	dotnet/runtime	System.Security.Cryptography.Tests.ChaCha20Poly1305Tests.EncryptTamperAADDecrypt
1602441	dotnet/runtime	System.Security.Cryptography.Tests.ChaCha20Poly1305Tests.EncryptTamperAADDecrypt
1602019	dotnet/runtime	System.Security.Cryptography.Tests.ChaCha20Poly1305Tests.EncryptTamperAADDecrypt
Displaying 100 of 161 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
41	59	161

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[matouskozak]

I see the same failure on maccatalyst x64 jobs

+ /usr/local/Cellar/python@3.14/3.14.6/Frameworks/Python.framework/Versions/3.14/bin/python3.14 /tmp/helix/working/B75209F8/p/gen-debug-dump-docs.py -buildid 1529233 -workitem IcuAppLocal.Tests -jobid b63e5a2a-ba0c-4583-91ba-ad08e7f64b7a -outdir /tmp/helix/working/B75209F8/w/AD2F0948/uploads -templatedir /tmp/helix/working/B75209F8/p -dumpdir /cores -productver 11.0.0
/usr/local/Cellar/python@3.14/3.14.6/Frameworks/Python.framework/Versions/3.14/Resources/Python.app/Contents/MacOS/Python: can't open file '/tmp/helix/working/B75209F8/p/gen-debug-dump-docs.py': [Errno 2] No such file or directory
+ exit 143

https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-b63e5a2aba0c458391/IcuAppLocal.Tests/1/console.cb7b36f7.log?helixlogtype=result