Repository navigation
fix(ci): bypass unresolvable Azure apt mirror when installing fish and zsh - #3570
Merged
Daniel Meppiel (danielmeppiel) merged 3 commits intoNov 11, 2026
Merged
Conversation
…d zsh GitHub-hosted Ubuntu runners route apt through azure.archive.ubuntu.com. Since ~14:35 UTC on 2026-11-11 that host does not resolve on the runners and the mirror-list fallback still fails, so "apt-get install -y fish zsh" exits 100 after ~7 minutes and every Lifecycle Smoke shard (and the Merge Gate) fails. Add scripts/ci-apt-install.sh, which points apt at the public Ubuntu archive when the Azure mirror does not resolve or the install fails, and use it for the three fish/zsh install steps. Refs #3569 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel)
requested a review
from Sergio Sisternes (sergio-sisternes-epam)
as a code owner
November 11, 2026 16:18
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the CI apt-install resilience changes.
No open issues 🎉
What changed in this PR
This PR addresses #3569 by making CI installation of fish and zsh resilient to Azure Ubuntu mirror failures, without changing product behavior.
Changes:
- Adds a shared apt installer that switches to an architecture-appropriate public Ubuntu archive when Azure DNS resolution fails or the initial installation fails.
- Retries failed installation once, with bounded apt retry and timeout settings.
- Uses the helper in PR lifecycle smoke, merge-queue integration, and release integration workflows.
| File | Description |
|---|---|
scripts/ci-apt-install.sh |
Adds mirror detection, public archive fallback, and installation retry logic. |
.github/workflows/ci.yml |
Uses the helper for lifecycle smoke shell dependencies. |
.github/workflows/ci-integration.yml |
Uses the helper for merge-queue integration shell dependencies. |
.github/workflows/build-release.yml |
Uses the helper for release integration shell dependencies. |
Review effort: Balanced
…solve The first attempt pointed apt at archive.ubuntu.com, but the runner log showed that host does not resolve either: the 2026-11-11 Canonical outage takes down ubuntu.com DNS, so every *.ubuntu.com mirror fails. On amd64, fall back to mirrors.edge.kernel.org; apt still verifies the archive with the Ubuntu keyring from ubuntu.sources. Other architectures keep the default behaviour. Refs #3569 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The release Integration Tests matrix installs fish and zsh on ubuntu-24.04-arm too, which uses ports.ubuntu.com and is hit by the same ubuntu.com DNS outage. Fall back to https://mirrors.mit.edu/ubuntu-ports/ there; amd64 keeps mirrors.edge.kernel.org. Refs #3569 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel)
deleted the
danielmeppiel-ci-apt-mirror-fallback
branch
November 11, 2026 18:01
This was referenced Nov 11, 2026
Merged
Daniel Meppiel (danielmeppiel)
added a commit
that referenced
this pull request
Nov 12, 2026
…pt (#3580) * feat(preflight): add preflight-gate local hook package Adds packages/preflight-gate, a repo-local APM package with a diff-aware preflight runner (mirrors the ci.yml PR lanes verbatim, fingerprint-bound against drift) and a preToolUse hook that denies git push / gh pr create|ready / PR-creating tools unless a fresh, green, commit-bound receipt matches the pushed commit. The hook is the only agent-facing surface: deny reasons are self-sufficient. Wired into the root apm.yml (dependency + executables.allow grant) and deployed to .github/hooks/ by apm install. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * perf(preflight): run independent checks concurrently; drop draft guidance - Run checks inside a tier on a bounded pool (--jobs, default min(4, cpus)); xdist/E2E suites are marked exclusive in the policy and run alone. Tiers stay sequential with stop-after-failing-tier; logs stay per check and the summary prints in policy order. The receipt records parallel, jobs and tier_wall_s. - Drop the "open the PR as draft" deny guidance; CI-only lanes are a neutral README note. - Engage only for publish actions: a preToolUse matcher limits the hook to shell and create-PR tools, and the launcher starts Python only for push / gh pr / gh api pulls / create_pull_request payloads (pure shell match). Git alias expansion is dropped (documented). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * perf(preflight): overlap lightweight checks across tiers Replace per-tier waves with a two-phase schedule over all selected tiers. Phase A runs every non-exclusive check concurrently on the bounded pool (so arch.contracts overlaps pylint); phase B runs the exclusive xdist/E2E checks one at a time in tier order. A phase A failure skips phase B and a failing exclusive check skips the rest, unless --keep-going. Tiers stay the selection unit: a tier counts as ran only if all its checks ran. The receipt records phase_wall_s and skipped_checks instead of tier_wall_s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(preflight): close chained-push and subdirectory gaps - D1: a gated push/PR is only allowed when every preceding segment in the same tool call is read-only (cd/pushd/popd/true/echo/printf and git status/log/diff/show/fetch/rev-parse/remote -v/branch --show-current/ config --get); otherwise deny and name the segment. - D2: the hook command resolves the repo root when the repo-relative launcher path does not exist, so sessions in subdirectories are gated; user-scope absolute paths keep working. - D3: command checks run in a CI-like env (allowlisted vars, isolated HOME/XDG, USER=runner, short TMPDIR, GIT_EXEC_PATH kept, real uv cache/config); env.lock-sync fails when the uv env drifts from uv.lock; policy VOLATILE_OUTPUTS (CONFORMANCE.*, .github/mcp.json) are restored and recorded as restored_outputs instead of failing tree mutation. - D4/D9: per-cause Fix lines (tags/other-branch/chained/unparseable do not say "run preflight"); failed checks list their actual log files. - D5: gate update_pull_request only when it clears draft / marks ready; document git -c alias, $(echo git) push and git send-pack as limits. - D6: runner exits 1 when the receipt is not gate-eligible. - D8: `gh pr create --help/-h` is exempt. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(preflight): self-provision lock-matched env env.lock-sync now runs before phase A and repairs drift in place (on by default; --no-provision disables): `uv sync --frozen --extra dev`, falling back to `uv export` + `uv pip sync` through the configured index and an editable re-install of the project, then re-checks. uv.lock is never modified. APM_PREFLIGHT_LOCK_TOLERATE (machine-local, comma list) names packages a mirror cannot serve at the locked version: provisioning installs the highest same-major.minor version (or keeps the installed one), the drift is printed as [!] and recorded as lock_drift, and the receipt stays gate-eligible. Any other drift still fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(preflight): self-sufficient fresh clones and fail-fast - env.lock-sync is selected on every run (T0-only included) and provisions a missing/drifted env with ci.yml's PYTHON_VERSION; child checks always get UV_NO_SYNC=1, UV_FROZEN=1 and the project env's bin/ first on PATH, so no host tool (e.g. a newer ruff) is ever used and callers need not export anything. - Lock tolerance is also read from machine-local git config apm.preflight.lockTolerate (APM_PREFLIGHT_LOCK_TOLERATE wins). - Fail-fast: the first failing check (or a failed env.lock-sync) cancels running Phase A checks by process group and skips the rest; receipt records cancelled_checks. --keep-going runs everything. - Deny polish: chained calls spell out commit / preflight / push alone; failed receipts say fix, commit, re-run and list up to 6 failed ids with the logs dir; other-branch pushes say push only HEAD. - spec.conformance runs with APM_NO_DIRECT_FALLBACK=1 so apm skips its public release lookup: CI tests the merge commit (latest version) and never sees the update banner that breaks JSON-parsing spec tests on a branch cut before the latest release. The isolated HOME also marks the daily update check as done. - README: hooks load only from the session's starting directory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(preflight): refresh lifecycle smoke CI fingerprint main #3570 replaced the lifecycle apt install step with scripts/ci-apt-install.sh; preflight does not mirror that step, so only the fingerprint changes. Deployed copy and apm.lock.yaml hash updated without unrelated lock churn. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(preflight): verify local-package snapshot hashes; correct lock content_hash CI failed test_local_package_snapshots_match_lockfile on #3580: the fingerprint refresh updated the deployed file hash but not the package content_hash in apm.lock.yaml, and preflight did not select that test for packages/ edits (apm audit --ci does not check it). Add apm.lock-snapshots to the self-check lane with a regression-trap test, and correct the hash. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Since ~14:35 UTC on 2026-11-11, every
Lifecycle Smoke Shard 1/2 (Linux)job failsat
sudo apt-get install -y fish zsh(exit 100) withTemporary failure resolving 'azure.archive.ubuntu.com', so thegatecheck failson every PR (runs 43039820516, 43045315009, 43050371139, 43055276136). The same
install runs in the release Integration Tests legs and in
ci-integration.yml.Root cause is upstream: status.canonical.com reports a Major Outage for
ubuntu.com(and launchpad, keyserver, ...) today, and Google DNS answersName servers did not respond [185.125.190.65, 185.125.190.66] ... At delegation ubuntu.comfor uncached names. Every
*.ubuntu.commirror in the runner's apt list(
azure.archive.ubuntu.com,archive.ubuntu.com,security.ubuntu.com) failsto resolve; a first iteration of this PR that switched to
archive.ubuntu.comfailed the same way (job 130649415952). Steps that only need preinstalled
packages (
upx-ucl) still pass;fish/zshare not preinstalled.This PR adds
scripts/ci-apt-install.shand uses it for the threefish zshinstall steps (
ci.yml,ci-integration.yml,build-release.yml). Behaviour:archive.ubuntu.comon amd64,ports.ubuntu.comon arm64)resolves: plain
apt-get update && apt-get install(unchanged); on failure,retry once with the fallback mirror.
doomed retries. Fallbacks:
http://mirrors.edge.kernel.org/ubuntu/(amd64),https://mirrors.mit.edu/ubuntu-ports/(arm64; both synced today).InReleasewith the Ubuntu archive keyring (Signed-Byinubuntu.sources), so the mirror is only an untrusted transport.ubuntu.sourceskeep the default behaviour.Issue and approved scope
Issue: Fixes #3569
Human scope-approval comment: scope approved by Daniel Meppiel (@danielmeppiel)'s explicit
session mandate ("loop until the CI pipelines in main are fixed, and that the
release pipeline is passing on a dry run"), recorded in #3569. Review contact:
Daniel Meppiel (@danielmeppiel). Touches
.github/workflows/**(CODEOWNERS: Lead Maintainer review).Does this PR complete the issue, or what remains? Completes it.
Type of change
Testing
Tested locally
All existing tests pass
Added tests for new functionality (if applicable)
shellcheck scripts/ci-apt-install.sh-> clean;bash -n-> ok; file is printable ASCII.Workflow contract tests:
.venv/bin/pytest tests/unit/test_diagnostics.py tests/unit/test_windows_compat_gate_workflow.py tests/unit/test_windows_signing_contract.py tests/unit/test_platform_contract_workflow.py tests/unit/test_windows_unit_diagnostics.py tests/unit/test_windows_native_symlink_workflow.py tests/unit/test_ado_live_workflow_contract.py tests/unit/test_security_workflow_contracts.py -q -n 4-> 184 passed.Live validation (amd64): on this PR, both
Lifecycle Smoke Shard 1/2 (Linux)jobs installed fish/zsh through the fallback during the outage and passed (run 43059590234; the install step took ~35s instead of failing after ~7 min). The arm64 path is exercised only by the release Integration Tests leg (not run on PRs).Spec conformance (OpenAPM v0.1)
docs/src/content/docs/specs/openapm-v0.1.mdupdated(new/changed
<a id="req-XXX"></a>anchor + prose + Appendix Crow).
docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.ymlupdated.
@pytest.mark.req("req-XXX")test undertests/spec_conformance/added or extended.CONFORMANCE.{md,json}regenerated viauv run --extra dev python -m tests.spec_conformance.gen_statementand committed.