Repository navigation
Auto Start CI delayed running 2 - 3 hours instead of every 5 minutes #4443
Description
Situation
The workflow node/actions/workflows/auto-start-ci.yml is set up with:
on:
schedule:
# Runs every five minutes (fastest the scheduler can run). Five minutes is
# optimistic, it can take longer to run.
# To understand why `schedule` is used instead of other events, refer to
# ./doc/contributing/commit-queue.md
- cron: '*/5 * * * *'to run every 5 minutes. On Aug 27, 2026 it is running instead every 2 - 3 hours.
3 hours ago 2026-08-27T09:18:41Z
6 hours ago 2026-08-27T06:08:17Z
8 hours ago 2026-08-27T03:56:58Z
10 hours ago 2026-08-27T02:02:58Z
12 hours ago 2026-08-27T00:20:11Z
13 hours ago 2026-08-26T23:08:20Z
14 hours ago 2026-08-26T22:01:17Z
15 hours ago 2026-08-26T20:57:46Z
16 hours ago 2026-08-26T20:05:20Z
17 hours ago 2026-08-26T19:24:51Z
18 hours ago 2026-08-26T18:43:42Z
18 hours ago 2026-08-26T18:00:43Z
19 hours ago 2026-08-26T16:58:54Z
20 hours ago 2026-08-26T16:39:22Z
20 hours ago 2026-08-26T16:19:49Z
21 hours ago 2026-08-26T15:49:10Z
21 hours ago 2026-08-26T14:59:57Z
22 hours ago 2026-08-26T14:38:41Z
22 hours ago 2026-08-26T14:20:22Z
22 hours ago 2026-08-26T14:00:04Z
There was a GitHub Incident for Actions & Pull Requests that was supposed to be resolved on Aug 27, 2026 00:6 UTC.
This is preventing timely response to request-ci labels.
For example, request-ci was applied to nodejs/node#65492 at 2026-08-27T11:03Z and 2 hours later, the PR is still waiting for the request to be fulfilled.
The schedule event can be delayed during periods of high loads of GitHub Actions workflow runs. High load times include the start of every hour. If the load is sufficiently high enough, some queued jobs may be dropped. To decrease the chance of delay, schedule your workflow to run at a different time of the hour.
A delay of 2 - 3 hours over an extended period of time is outside of what could be expected.
Activity
MikeMcC399 commentedon Aug 27, 2026
This should possibly be raised to GitHub Support.
There is no
workflow_dispatchevent trigger in the workflow, so no means to start it manually.MikeMcC399 commentedon Aug 27, 2026
MikeMcC399 commentedon Aug 28, 2026
It's still having a long pause between runs: 2 - 3 hours gap:
targos commentedon Aug 28, 2026
I opened nodejs/node#65612
If it doesn't help, I don't think we can do much more with GitHub actions. Another approach could be to have the schedule in Jenkins and trigger workflow runs using the API.
tools: offset GitHub crons by 3 minutes
MikeMcC399 commentedon Aug 28, 2026
I guess it's worth a try to skew the start, however since it was working reasonably before, and it has stopped doing what is expected of it, it may need a support ticket to GitHub opened.
GitHub is generally coming under increased load due to the proliferation of AI-based submissions.
https://www.theregister.com/devops/2026/08/21/we-let-you-down-github-pledges-to-scale-up-before-developers-give-up/5291031
tools: offset GitHub crons by 3 minutes
MikeMcC399 commentedon Aug 28, 2026
https://github.com/nodejs/node/blob/main/.github/workflows/auto-start-ci.yml is now showing a skewed start of 3 minutes:
This hasn't produced any change and the workflow is still showing a last run of almost 3 hours ago.
I would suggest opening a ticket on https://support.github.com/ (https://support.github.com/contact-next)
panva commentedon Aug 28, 2026
And so far, today looks worse than yesterday.
sxa commentedon Aug 28, 2026
I seem to recall mention of an overuse of our monthly github actions allocation from the 20th August and I'm wondering if that may have caused us to be restricted. Ping @nodejs/tsc since I believe this may have been discussed by you somewhere...
MikeMcC399 commentedon Aug 28, 2026
It does look like a deliberate throttling.
panva commentedon Aug 28, 2026
That's for private repo action use.
MikeMcC399 commentedon Aug 28, 2026
I would suggest anyway to reduce the requested cron frequency to once every 15 minutes, instead of every 5 minutes. When the Jenkins runs are taking several hours, there is little advantage to checking every 5 minutes, even if GitHub Actions were running it that frequently.
aduh95 commentedon Aug 28, 2026
https://github.com/nodejs/corepack/actions/workflows/sync.yml (weekly cron) was scheduled to run at 00:05 UTC this morning, it ended up running at 3:20 UTC. I doubt that changing the requested frequency would have much effect
MikeMcC399 commentedon Aug 28, 2026
I didn't mean change the frequency to solve the issue and I still think this needs escalation to GitHub Support.
sxa commentedon Aug 28, 2026
Looks like it's not just us: https://github.com/orgs/community/discussions/205984
In the interests of reducing the backlog and using up some "quiet time" for the machines over the weekend I'm manually starting a few of the tests. They'll get throttled to avoid overwhelming the system due to reducing node-test-commit to only being able to run 10 in parallel. Surprisingly there aren't that many with the
request-cilabel (14 at the moment) I'll also note that I've seen some PRs where the result has not been propagated back once the job is complete which could be indicative of a greater problem somewhere. Adding in the PR links so the originators get this issue link to know that these have been manually triggered. I did see that some of these had the label added 2-3 weeks ago and weren't triggered. May have been some other issue at that time. I've left the "Commits" column to be able to track issues such as status not being reported back if they occur.panva commentedon Aug 28, 2026
Then they didn't have approvals for the latest commit. That they don't run or get scheduled is expected.
aduh95 commentedon Aug 28, 2026
One of the upside of using a cron is that is not possible: if GHA is down is disabled (or down) at some point, once it's up again it simply picks up those old jobs.
panva commentedon Aug 29, 2026
Friday night mode engaged.
MikeMcC399 commentedon Aug 29, 2026
I'll close this issue, since the frequency of the Auto Start CI cron job is now around 10 - 15 minutes.
I suspect that GitHub was deliberately throttling these jobs and has now removed the throttle.
tools: offset GitHub crons by 3 minutes
tools: offset GitHub crons by 3 minutes
tools: offset GitHub crons by 3 minutes
tools: offset GitHub crons by 3 minutes
tools: offset GitHub crons by 3 minutes