Skip to content

Job Monitor: non-PR builds trigger 500s in Helix when querying jobs聽#17371

Description

@premun

Possibly something about the source property that we send fails there

info: 馃攣 Checking for failed Helix jobs to resubmit the failed work items...
##[error]Helix Job Monitor terminated with an unhandled exception.
      Microsoft.DotNet.Helix.Client.RestApiException`1[Microsoft.DotNet.Helix.Client.Models.ApiError]: The response contained an invalid status code 500 Internal Server Error
      
      Body: {"Message":"An error occured.","ActivityId":"134f4a2a7afac75f9f23fcf1cb3c6109"}
         at Microsoft.DotNet.Helix.JobMonitor.HelixService.RetryAsync[T](Func`1 action, CancellationToken cancellationToken) in /_/src/arcade/src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs:line 431
         at Microsoft.DotNet.Helix.JobMonitor.HelixService.GetJobsForBuildAsync(String source, String buildId, CancellationToken cancellationToken) in /_/src/arcade/src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs:line 56
         at Microsoft.DotNet.Helix.JobMonitor.JobMonitorRunner.ExecuteRetryPassAsync(CancellationToken cancellationToken) in /_/src/arcade/src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.cs:line 133
         at Microsoft.DotNet.Helix.JobMonitor.JobMonitorRunner.RunAsync(CancellationToken cancellationToken) in /_/src/arcade/src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorRunner.cs:line 89
         at Microsoft.DotNet.Helix.JobMonitor.Program.Main(String[] args) in /_/src/arcade/src/Microsoft.DotNet.Helix/JobMonitor/Program.cs:line 86

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1559220&view=logs&j=078edc60-2a90-5618-e72d-426a045b11f0&t=0ca3371c-1de8-59d8-daef-35ea5811e911&s=6884a131-87da-5381-61f3-d7acc3b91d76

Activity

  1. self-assigned this
    on Aug 20, 2026
  2. premun commented on Aug 20, 2026

    @premun
    MemberAuthor

    The problem is that we're trying to get all jobs belonging to a PR branch and then filter by properties.
    We do this by calling /api/jobs?Source=[SOURCE]&count=100000 which for the rolling build was ci/public/dotnet/runtime/refs/heads/main which returns thousands of jobs and the API fails with 500.

    Instead, we should use the /api/jobs?properties[BuildId]=1559220 API but that also returns 500 because the SQL query is not formatted properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions