Skip to content

fix(ci): bypass unresolvable Azure apt mirror when installing fish and zsh - #3570

Merged
Daniel Meppiel (danielmeppiel) merged 3 commits into
mainfrom
danielmeppiel-ci-apt-mirror-fallback
Nov 11, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 3 commits into
mainfrom
danielmeppiel-ci-apt-mirror-fallback

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Since ~14:35 UTC on 2026-11-11, every Lifecycle Smoke Shard 1/2 (Linux) job fails
at sudo apt-get install -y fish zsh (exit 100) with
Temporary failure resolving 'azure.archive.ubuntu.com', so the gate check fails
on every PR (runs 43039820516, 43045315009, 43050371139, 43055276136). The same
install runs in the release Integration Tests legs and in ci-integration.yml.

Root cause is upstream: status.canonical.com reports a Major Outage for
ubuntu.com (and launchpad, keyserver, ...) today, and Google DNS answers
Name servers did not respond [185.125.190.65, 185.125.190.66] ... At delegation ubuntu.com
for uncached names. Every *.ubuntu.com mirror in the runner's apt list
(azure.archive.ubuntu.com, archive.ubuntu.com, security.ubuntu.com) fails
to resolve; a first iteration of this PR that switched to archive.ubuntu.com
failed the same way (job 130649415952). Steps that only need preinstalled
packages (upx-ucl) still pass; fish/zsh are not preinstalled.

This PR adds scripts/ci-apt-install.sh and uses it for the three fish zsh
install steps (ci.yml, ci-integration.yml, build-release.yml). Behaviour:

  • If the archive host (archive.ubuntu.com on amd64, ports.ubuntu.com on arm64)
    resolves: plain apt-get update && apt-get install (unchanged); on failure,
    retry once with the fallback mirror.
  • If it does not resolve: go straight to the fallback mirror, skipping minutes of
    doomed retries. Fallbacks: http://mirrors.edge.kernel.org/ubuntu/ (amd64),
    https://mirrors.mit.edu/ubuntu-ports/ (arm64; both synced today).
  • apt still verifies InRelease with the Ubuntu archive keyring (Signed-By in
    ubuntu.sources), so the mirror is only an untrusted transport.
  • Runners without a deb822 ubuntu.sources keep the default behaviour.

Issue and approved scope

Issue: Fixes #3569

Human scope-approval comment: scope approved by Daniel Meppiel (@danielmeppiel)'s explicit
session mandate ("loop until the CI pipelines in main are fixed, and that the
release pipeline is passing on a dry run"), recorded in #3569. Review contact:
Daniel Meppiel (@danielmeppiel). Touches .github/workflows/** (CODEOWNERS: Lead Maintainer review).

Does this PR complete the issue, or what remains? Completes it.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally

  • All existing tests pass

  • Added tests for new functionality (if applicable)

  • shellcheck scripts/ci-apt-install.sh -> clean; bash -n -> ok; file is printable ASCII.

  • Workflow contract tests: .venv/bin/pytest tests/unit/test_diagnostics.py tests/unit/test_windows_compat_gate_workflow.py tests/unit/test_windows_signing_contract.py tests/unit/test_platform_contract_workflow.py tests/unit/test_windows_unit_diagnostics.py tests/unit/test_windows_native_symlink_workflow.py tests/unit/test_ado_live_workflow_contract.py tests/unit/test_security_workflow_contracts.py -q -n 4 -> 184 passed.

  • Live validation (amd64): on this PR, both Lifecycle Smoke Shard 1/2 (Linux) jobs installed fish/zsh through the fallback during the outage and passed (run 43059590234; the install step took ~35s instead of failing after ~7 min). The arm64 path is exercised only by the release Integration Tests leg (not run on PRs).

Spec conformance (OpenAPM v0.1)

  • Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
    (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
    row).
  • Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
    updated.
  • Test edit: a @pytest.mark.req("req-XXX") test under
    tests/spec_conformance/ added or extended.
  • CONFORMANCE.{md,json} regenerated via
    uv run --extra dev python -m tests.spec_conformance.gen_statement
    and committed.
  • N/A -- this PR does not change OpenAPM-observable behaviour.

…d zsh

GitHub-hosted Ubuntu runners route apt through azure.archive.ubuntu.com.
Since ~14:35 UTC on 2026-11-11 that host does not resolve on the runners
and the mirror-list fallback still fails, so "apt-get install -y fish zsh"
exits 100 after ~7 minutes and every Lifecycle Smoke shard (and the
Merge Gate) fails. Add scripts/ci-apt-install.sh, which points apt at the
public Ubuntu archive when the Azure mirror does not resolve or the
install fails, and use it for the three fish/zsh install steps.

Refs #3569

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the CI apt-install resilience changes.

No open issues 🎉

What changed in this PR

This PR addresses #3569 by making CI installation of fish and zsh resilient to Azure Ubuntu mirror failures, without changing product behavior.

Changes:

  • Adds a shared apt installer that switches to an architecture-appropriate public Ubuntu archive when Azure DNS resolution fails or the initial installation fails.
  • Retries failed installation once, with bounded apt retry and timeout settings.
  • Uses the helper in PR lifecycle smoke, merge-queue integration, and release integration workflows.
File Description
scripts/​ci-apt-install.sh Adds mirror detection, public archive fallback, and installation retry logic.
.github/​workflows/​ci.yml Uses the helper for lifecycle smoke shell dependencies.
.github/​workflows/​ci-integration.yml Uses the helper for merge-queue integration shell dependencies.
.github/​workflows/​build-release.yml Uses the helper for release integration shell dependencies.

Review effort: Balanced

…solve

The first attempt pointed apt at archive.ubuntu.com, but the runner log
showed that host does not resolve either: the 2026-11-11 Canonical outage
takes down ubuntu.com DNS, so every *.ubuntu.com mirror fails. On amd64,
fall back to mirrors.edge.kernel.org; apt still verifies the archive with
the Ubuntu keyring from ubuntu.sources. Other architectures keep the
default behaviour.

Refs #3569

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The release Integration Tests matrix installs fish and zsh on
ubuntu-24.04-arm too, which uses ports.ubuntu.com and is hit by the same
ubuntu.com DNS outage. Fall back to https://mirrors.mit.edu/ubuntu-ports/
there; amd64 keeps mirrors.edge.kernel.org.

Refs #3569

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 816c940 into main Nov 11, 2026
25 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-ci-apt-mirror-fallback branch November 11, 2026 18:01
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
…pt (#3580)

* feat(preflight): add preflight-gate local hook package

Adds packages/preflight-gate, a repo-local APM package with a
diff-aware preflight runner (mirrors the ci.yml PR lanes verbatim,
fingerprint-bound against drift) and a preToolUse hook that denies
git push / gh pr create|ready / PR-creating tools unless a fresh,
green, commit-bound receipt matches the pushed commit. The hook is
the only agent-facing surface: deny reasons are self-sufficient.

Wired into the root apm.yml (dependency + executables.allow grant)
and deployed to .github/hooks/ by apm install.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* perf(preflight): run independent checks concurrently; drop draft guidance

- Run checks inside a tier on a bounded pool (--jobs, default
  min(4, cpus)); xdist/E2E suites are marked exclusive in the policy
  and run alone. Tiers stay sequential with stop-after-failing-tier;
  logs stay per check and the summary prints in policy order. The
  receipt records parallel, jobs and tier_wall_s.
- Drop the "open the PR as draft" deny guidance; CI-only lanes are a
  neutral README note.
- Engage only for publish actions: a preToolUse matcher limits the
  hook to shell and create-PR tools, and the launcher starts Python
  only for push / gh pr / gh api pulls / create_pull_request payloads
  (pure shell match). Git alias expansion is dropped (documented).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* perf(preflight): overlap lightweight checks across tiers

Replace per-tier waves with a two-phase schedule over all selected
tiers. Phase A runs every non-exclusive check concurrently on the
bounded pool (so arch.contracts overlaps pylint); phase B runs the
exclusive xdist/E2E checks one at a time in tier order. A phase A
failure skips phase B and a failing exclusive check skips the rest,
unless --keep-going. Tiers stay the selection unit: a tier counts as
ran only if all its checks ran. The receipt records phase_wall_s and
skipped_checks instead of tier_wall_s.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(preflight): close chained-push and subdirectory gaps

- D1: a gated push/PR is only allowed when every preceding segment in the
  same tool call is read-only (cd/pushd/popd/true/echo/printf and git
  status/log/diff/show/fetch/rev-parse/remote -v/branch --show-current/
  config --get); otherwise deny and name the segment.
- D2: the hook command resolves the repo root when the repo-relative
  launcher path does not exist, so sessions in subdirectories are gated;
  user-scope absolute paths keep working.
- D3: command checks run in a CI-like env (allowlisted vars, isolated
  HOME/XDG, USER=runner, short TMPDIR, GIT_EXEC_PATH kept, real uv
  cache/config); env.lock-sync fails
  when the uv env drifts from uv.lock; policy VOLATILE_OUTPUTS
  (CONFORMANCE.*, .github/mcp.json) are restored and recorded as
  restored_outputs instead of failing tree mutation.
- D4/D9: per-cause Fix lines (tags/other-branch/chained/unparseable do not
  say "run preflight"); failed checks list their actual log files.
- D5: gate update_pull_request only when it clears draft / marks ready;
  document git -c alias, $(echo git) push and git send-pack as limits.
- D6: runner exits 1 when the receipt is not gate-eligible.
- D8: `gh pr create --help/-h` is exempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* feat(preflight): self-provision lock-matched env

env.lock-sync now runs before phase A and repairs drift in place (on by
default; --no-provision disables): `uv sync --frozen --extra dev`,
falling back to `uv export` + `uv pip sync` through the configured index
and an editable re-install of the project, then re-checks. uv.lock is
never modified.

APM_PREFLIGHT_LOCK_TOLERATE (machine-local, comma list) names packages a
mirror cannot serve at the locked version: provisioning installs the
highest same-major.minor version (or keeps the installed one), the drift
is printed as [!] and recorded as lock_drift, and the receipt stays
gate-eligible. Any other drift still fails.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(preflight): self-sufficient fresh clones and fail-fast

- env.lock-sync is selected on every run (T0-only included) and
  provisions a missing/drifted env with ci.yml's PYTHON_VERSION; child
  checks always get UV_NO_SYNC=1, UV_FROZEN=1 and the project env's bin/
  first on PATH, so no host tool (e.g. a newer ruff) is ever used and
  callers need not export anything.
- Lock tolerance is also read from machine-local git config
  apm.preflight.lockTolerate (APM_PREFLIGHT_LOCK_TOLERATE wins).
- Fail-fast: the first failing check (or a failed env.lock-sync)
  cancels running Phase A checks by process group and skips the rest;
  receipt records cancelled_checks. --keep-going runs everything.
- Deny polish: chained calls spell out commit / preflight / push alone;
  failed receipts say fix, commit, re-run and list up to 6 failed ids
  with the logs dir; other-branch pushes say push only HEAD.
- spec.conformance runs with APM_NO_DIRECT_FALLBACK=1 so apm skips its
  public release lookup: CI tests the merge commit (latest version) and
  never sees the update banner that breaks JSON-parsing spec tests on a
  branch cut before the latest release. The isolated HOME also marks the
  daily update check as done.
- README: hooks load only from the session's starting directory.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(preflight): refresh lifecycle smoke CI fingerprint

main #3570 replaced the lifecycle apt install step with scripts/ci-apt-install.sh; preflight does not mirror that step, so only the fingerprint changes. Deployed copy and apm.lock.yaml hash updated without unrelated lock churn.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(preflight): verify local-package snapshot hashes; correct lock content_hash

CI failed test_local_package_snapshots_match_lockfile on #3580: the fingerprint refresh updated the deployed file hash but not the package content_hash in apm.lock.yaml, and preflight did not select that test for packages/ edits (apm audit --ci does not check it). Add apm.lock-snapshots to the self-check lane with a regression-trap test, and correct the hash.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Lifecycle Smoke fails on every PR, azure.archive.ubuntu.com does not resolve on runners

2 participants