Repository navigation
Review all tests #1136
Description
... we should perhaps consider doing another cycle of #959 since the current process for @nodejs/releasers (who i'm tagging for awareness/input since there are a few of these removal PRs from Mike) is typically "compare with the last version and make sure it's no worse" which I'd suggest is not ideal although generally works.
Originally posted by @sxa in #1123 (review)
Activity
MikeMcC399 commentedon Sep 7, 2026
Looking at the results of running the
citgm-smokeron the Node.js 24.x LTS branch, it is failing a significant number of modules, including ones likeundici&node-gypIf a package test fails with citgm in the baseline of the released Node.js version, then it's likely that the prospective new Node.js release is also going to fail, taking away the possibility of registering a regression.
It seems like this is a significant issue and I don't get the feeling that there is sufficient capacity in the citgm team to deal with it at the moment. The few PRs I have submitted are only scratching at the surface, and even they have hardly seen much response overall.
Ideally citgm should:
This means going through all the failures and making sure they no longer surface, possibly also needing coordination with each affected package maintainer. I can well imagine that this would be a full-time job for one person, and a significant effort for a team if spread out over multiple people. It could involve fixing tests, or skipping / removing them, if they aren't fixable.
MikeMcC399 commentedon Sep 11, 2026
MikeMcC399 commentedon Oct 2, 2026
MikeMcC399 commentedon Oct 2, 2026
MikeMcC399 commentedon Oct 4, 2026
MikeMcC399 commentedon Oct 8, 2026
I've submitted multiple issues for failures that I have found, so I don't think this issue needs to track individual problems anymore.
The @nodejs/citgm team does however need to give some thought about how the repo can be maintained effectively in future.
To be easy to use for checking regressions in planned Node.js releases the number of regularly failing tests from citgm in Jenkins should be reduced to a minimum, ideally zero. The latest https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/lastCompletedBuild/testReport/ is showing
108 failures.(we've reduced that to 42 in the meantime).There are rules for adding modules to test, but there is no governance description about what should happen if there is a failure later on.
What should happen initially when tests fail? Who should be notified and take action?
If tests continue to fail, should there be a time limit set so that a module is temporarily skipped - for instance after one month?
If there is still no resolution after a longer period of time, when should a module be removed from citgm - after three months perhaps?
What about when a module becomes deprecated, or the repo it is based on is archived? Should there be a governance rule recorded somewhere in this repo describing what should happen?