Repository navigation
runner VMs: setup-python RUNPATH /opt/hostedtoolcache vs _work/_tool broke python3 without LD_LIBRARY_PATH (Strix exit 127) #2506
Description
Symptom
Required strix jobs that land on cwlab-s1-02 (group cwl mcp remediation) fail before scanning:
python3: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory
Strix run failed for model 'orchestrator/free' after 1s (exit code 127).
No Strix vulnerability report artifact was produced ... failing closed.
pythonLocation / LD_LIBRARY_PATH point at /home/runner/actions-runner/_work/_tool/Python/3.13.15/x64, so the toolcache entry that actions/setup-python reuses on this runner is incomplete (the shared library is missing from lib/).
Evidence: ContextualWisdomLab/fast-mlsirm job 109231001166 (PR #2062, head 25bb92c7, 2026-09-29 02:53–03:00 UTC). The same PR's previous Strix attempt, which ran on a GitHub-hosted runner, reached the scan and reported 0 vulnerabilities.
Suggested fix
On cwlab-s1-02, remove _work/_tool/Python/3.13.15 (plus its .complete marker) so setup-python re-downloads a complete build, or check that lib/libpython3.13.so.1.0 exists. Other toolcache hosts may need the same check.
🤖 Generated with Claude Code
Activity
seonghobae commentedon Sep 29, 2026
Correction and resolution
The original diagnosis was wrong.
libpython3.13.so.1.0was present in the toolcache. The setup-python Linux builds hardcodeRUNPATH /opt/hostedtoolcache/Python/<ver>/x64/lib(non-relocatable, per setup-python's advanced-usage docs). These self-hosted VMs keep their tool cache at/home/runner/actions-runner/_work/_tool, so Python loaded only whileLD_LIBRARY_PATHwas set. The Strix gate's launcher runspython3without it and hit exit 127. Hosted runners were unaffected because the RUNPATH matches there.Affected: 3.13.15 on cwlab-s1-01, -02, -05, -06 (the rest would break on their first 3.13 install).
Fixed on 2026-09-29 ~03:55 UTC:
ln -s /home/runner/actions-runner/_work/_tool /opt/hostedtoolcache. Verified withenv -u LD_LIBRARY_PATH …/python3 -c …for every cached version. No runner restart was needed.prepare.py,prepare-s1-runners.pyunder/data/contextualwisdomlab/actions-runners, backups*.bak-20260929) now create the same link, so re-provisioned VMs keep it.Not checked: the containerized GPU runner (cwlab-s1-gpu-01).
🤖 Generated with Claude Code
[-]runner cwlab-s1-02: toolcache Python 3.13.15 missing libpython3.13.so.1.0 (Strix exits 127)[/-][+]runner VMs: setup-python RUNPATH /opt/hostedtoolcache vs _work/_tool broke python3 without LD_LIBRARY_PATH (Strix exit 127)[/+]