Repository navigation
Silent CI failures due to enforced action restrictions with no prior notice #574
Description
Continuation of #550 (closed; unable to reopen)
This issue focuses specifically on verified actions silently failing in GitHub workflows following the recently enforced policies.
Problem
Several GitHub workflows are now failing silently as a result of the new action restrictions. For example: https://github.com/apache/iceberg-python/actions/runs/23416549033
These failures are marked with "Status: Startup failure" and the following error:
The action pypa/cibuildwheel@v3.4.0 is not allowed in apache/iceberg-python because all actions
must be from a repository owned by your enterprise, created by GitHub, or match one of the
allowed patterns.
While it's great to see efforts to improve ASF's security posture, it would be really helpful to have some advance notice before changes like this are enforced, so projects aren't caught off guard.
Impact
- CI silently drops checks due to enforcement — see #550 (comment) — with no indication that workflows have failed.
- No notifications are sent when GitHub workflows stop working. The failed nightly run linked above was only discovered by manually checking the Actions tab. This project has CI job notifications configured, yet no alerts were received.
Solution
EDIT: Added a new github action (apache/infrastructure-actions/allowlist-check@main) to address this issue, #587
Activity
kevinjqliu commentedon Mar 23, 2026
In #550, @dfoulks1 mentioned there was an email sent. I couldn't find any related emails
clambertus commentedon Mar 23, 2026
https://infra.apache.org/blog/trivy_security_incident.html
kevinjqliu commentedon Mar 23, 2026
@clambertus thats the trivy incident. I see github actions like
pypa/cibuildwheelandastral-sh/setup-uvsuddenly failing.Is it due to this? I didnt understand what "verified creator" means
kevinjqliu commentedon Mar 23, 2026
another example, apache/iceberg#15742 (comment)
potiuk commentedon Mar 23, 2026
Known issue with GitHub Actions - lack of jobs causes "green" PR by definition. But adding "context" in protected_branch definition (required status check for the PR to be mergable) solves the issue.
potiuk commentedon Mar 24, 2026
I thought a bit - I think we should add some incentive for people who use pinning properly and prevent those sillent errors to happen. That woudl be a carrot instead of stick:
Then it would be enitrely up to the PMC to react to those.
With Claude /AI - it can be entirely Clauded in a rather short time (hours) and maintenance of it will be near 0 (you can always ask Claude to fix any errors you experience in minutes). I would do it - but I am not sure if such contributions are really useful - the one for actions review I did in 2 hours and reviewed/closed > 10 prs - but it might take much longer for @raboof @dfoulks1 and others who review the actions to accept it a nd start using it
This is what we learn from current AI driven development - that new tools like that can be created very quickly by those who started using them - and quickly, but it still takes stakeholders a lot of time to accept that such tooling can now be created in almost no time.
My friend @kaxil wrote about it in this post - https://kaxil.substack.com/p/i-havent-written-a-line-of-code-in - and yes- what I see is exactly that - people need to get used to the fact that you can create such tools 10x faster now.
So I do not want to go ahead of those decisions even more - But - at some point in time I am happy to contribute it here if there is an appetite for it and willingness to accept such generated code.
kevinjqliu commentedon Mar 25, 2026
Thanks for the hint here @potiuk . I took a look at this solution. It involves very carefully specifying the exact name of each CI that should be "green" (which honestly should be all of them!). There are a lot of caveats and gotchas according to Github and ASF.
I found another potential solution/workaround. I added a runtime check in CI, comparing all the github actions against the
approved_patterns.ymlfile. Heres the code: apache/iceberg@f6c8e58This gives the failure signal we're looking for in the CI (and PR) (https://github.com/apache/iceberg/actions/runs/23525855725/job/68479029725?pr=15749). The "Checks" tab still shows a small black status for the silent failure, but this check will loudly tell everyone what the problem is.
Might be a good idea to turn this into an ASF github action that all projects can pull. WDYT?
EDIT: heres a first draft of what the github action would look like, #587
potiuk commentedon Mar 25, 2026
Fantastic idea !!! looks really good :)
kevinjqliu commentedon Mar 25, 2026
thanks @potiuk could you give it a review when you get a chance?
raboof commentedon Mar 26, 2026
Indeed. While it's indeed well-known, do you know if this is actually tracked somewhere?
Right, but many projects will not do this (either because of ignorance or because of a legitimate desire to have some jobs that are ran but not block the merge. Also that doesn't help for workflows that are triggered some other way, such as scheduled ones.
I like #587, but that's only for projects that are already aware. I wonder if we should also have a way to 'proactively' notify projects. One way might be to include this check in https://github.com/apache/infrastructure-gha-workflow-scanner - though we'd want to make sure it doesn't get too noisy. Also perhaps we could create some kind of report somewhere?
potiuk commentedon Mar 28, 2026
I tracked it for about 2 years in a personal ticket, but It seems to be an architectural issue and will not be fixed and I closed it.
I am not sure if anything else can be done on GH side to fix it. So this issue is basically tracking it on our side and maybe we will find some solutions to that.
Correct - that might not be a viable path for many. Unfortunately - I do not have any other idea how to deal with it other than halping to develop tools (or contribute to them) that might be ok for infra to use long term. As volunteers we migh help to create solutions - but regularly elmaintaining them when we have vacations, sudden spikes of other work is quite infeasible. I've learned that proposing things without earlier buy-in from infra and then trying to push it to them is a bad idea (I anticipated htat in my comment above as well. But if there is an infra buy-in and agreement on the kind of notification schema that we should do, I am happy to help to add such features.
That's basically what I proposed in #574 (comment) - (if it can be integrated in the scanner, then that would be great).
But I believe the action proposed by @kevinjqliu is way better solution.
The https://github.com/apache/infrastructure-gha-workflow-scanner might be used to check for PRESENCE of this action - not for action hashes. This would be way, way, way better - because you would just notify about lack of this action - and then once added, the action would automatically notify the maintainers about any issues with their action hashes. I think it's way more noisy, way less invasive and also it opens up the nice capability that we could check for MORE things - once we make everyone in ASF using GH actions to add this action.
I think this is a brilliant way of solving the noise problem. Eventially the https://github.com/apache/infrastructure-gha-workflow-scanner could ONLY check for presence of the action and be even very annoying about it - making it essentially a requirement for all ASF projects. And then the action called from every repo could do all kinds of check we want - for example zizmor check for security etc. This is I think fantastic way for infra (if they would want to be active in this area) force whatever checks they do and they will run not out-of-bound - but whenever PRs or "main" builds happen in the ASF repos - making them actually owners to solve their failing builds, rather than responding to email notifications.
Of course that also bears certain responsibility on whoever is the owner of such action - to follow up and resolve any problems it might cause - answer questions of the PMCs and so on, but this is a very elegant problem to solve INFRA "basic checks" issue that @killerbees mentioned in slack today. Such action might very cleary define what the INFRA enforces on every repo and make it part of INFRA very concrete, automatically enforcable set of expectations they have.
dave2wave commentedon Apr 5, 2026
For the technical solution #606 and #482 discuss approaches for handling this problem by periodic checking of your workflows and actions against the currently allowed actions.
kevinjqliu commentedon Apr 16, 2026
This is resolved by adding the
apache/infrastructure-actions/allowlist-checkaction project's CI.For example, in the apache/iceberg repo: https://github.com/apache/iceberg/blob/dde712ec9ed6c9d28183ee4615d50f97b246af5d/.github/workflows/asf-allowlist-check.yml
I tihnk the final step here is to add this to asf infra documentation somewhere as best practice recommendation
kevinjqliu commentedon Apr 16, 2026
perhaps in https://infra.apache.org/github-actions-policy.html or https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+Security
dave2wave commentedon Apr 16, 2026
I think that fits here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=321719166#GitHubActionsSecurity-Actionallowlisting
Publish test results inline; drop the CI merge gate