Reduce CI per-job wall time

[doraemonmj]

Platform

All / Unknown

Runtime Variant

All / Unknown

Summary

Several CI jobs are close to or above 10 minutes. The goal is to reduce the
wall time of each existing job without splitting jobs or changing the current
pre-commit gate.

The initial focus is:

• st-sim-a2a3
• st-sim-a5
• st-onboard-a2a3
• st-onboard-a5

The solution must support both GitHub-hosted and self-hosted execution. The
self-hosted sim runner also starts from a fresh server for each run, while
Onboard runners have different performance tiers.

Git Commit ID

04791a7

CANN Version

N/A — CI-level optimization across simulation and hardware jobs.

Driver Version

N/A — CI-level optimization across simulation and hardware jobs.

Host Platform

Other (please specify): GitHub-hosted Ubuntu/macOS and self-hosted Linux
runners.

Reproduction

Run the pull request CI on the same commit and record each job's wall time and
the following step groups:

# Group Actions step durations into:
# setup / install / runtime or kernel build / cache / test / DFX

Compare p50 and p90 separately by job, OS, execution path, and anonymous
runner performance tier. Do not combine GitHub-hosted, self-hosted, macOS,
Linux, or different self-hosted tiers into one average.

Expected Performance

• Reduce the p50 and p90 wall time of the target jobs without splitting them.
• Keep the existing pre-commit dependency and test semantics.
• Avoid regressions on slower self-hosted runner tiers.
• Keep representative per-PR coverage and run lower-frequency coverage in
  Daily CI.

Actual Performance

Recent successful PR CI runs show the following p50 values:

Job / runner	Job	Install	Scene	DFX
a2a3sim / Ubuntu	10.9m	2.3m	6.9m	1.2m
a2a3sim / macOS	11.2m	1.8m	7.0m	1.4m
a5sim / Ubuntu	10.5m	2.3m	6.5m	59s
a5sim / macOS	10.9m	2.6m	6.5m	1.0m

Onboard results vary significantly by runner tier: the observed p50 ranges
from 7.4m to 22.7m for a2a3 and from 3.4m to 8.6m for a5.

Profiling Data (Optional)

Scene execution is about 60% of sim ST wall time, while package installation
usually costs another 1.8-2.6 minutes. Package installation currently invokes
the CMake ALL target and can build runtimes that the job does not use.

Additional Context

Planned work:

• Add coarse step timing for setup, installation, build, cache, test, and DFX.
• Build only the runtime platform required by each CI job and remove repeated
  installation or runtime build work.
• Evaluate remote cache by its total restore/save cost and control job-local
  parallelism by runner tier.
• Re-measure p50 and p90 before changing the per-PR test set.
• If jobs remain slow, retain representative PR smoke coverage and move
  lower-frequency or redundant cases to Daily CI.

Initial test migration candidates:

• Move redundant Collective P4 variants on a2a3sim, a5sim, and a2a3 Onboard
  to Daily CI. Keep all P2 cases, one representative Allreduce Ring P4, and
  the invalid-rank check in PR CI.
• Move the large a5 Onboard Case1 variants for basic Paged Attention, Batch
  Paged Attention, and Unroll Manual Scope when an equivalent small case is
  already in PR CI.
• Move Benchmark BGEMM, SPMD Paged Attention, 4D Unroll, and similar large
  cases only after adding or promoting an equivalent small PR smoke case.
• Keep Pod, UT, DFX, SDMA, fault recovery, timeout, cross-device dependency,
  and state-machine tests in PR CI because they provide independent or unique
  coverage.

Acceptance criteria:

• No job splitting and no change to the pre-commit gate.
• Before/after p50 and p90 are reported for every affected runner path and
  tier.
• Cache changes include hit rate, transfer time, and storage size.
• Every case removed from PR CI records the saved time, retained smoke
  coverage, and its Daily CI destination.

[doraemonmj]

Post-change CI timing snapshot

This is a current-state snapshot after the related CI optimization changes were merged.

Method

• Sample: 19 complete, successful PR CI runs from 2026-08-21 through 2026-08-24 UTC.
• To avoid overweighting PRs with repeated pushes, only the latest successful run from each PR branch was used.
• The unmerged CI: run DeepSeek device smokes per PR #1949 run was excluded because it changes the Per-PR test selection.
• Timings are GitHub Actions job wall times (started_at to completed_at).
• p50 is the median: half of the observed runs completed within this time. p90 is the 90th percentile and represents tail latency.
• Results below are grouped by job name, without separating individual self-hosted runners.

Results

Job	n	p50	p90	Observed range
detect-changes	19	0m09s	0m10s	0m07s-0m12s
pre-commit	19	2m35s	3m50s	0m22s-5m13s
packaging / Ubuntu	19	4m25s	4m37s	3m08s-4m41s
packaging / macOS	19	6m24s	7m41s	5m08s-10m15s
profiling-flags-smoke	19	6m38s	7m10s	5m23s-7m20s
ut / Ubuntu	18	5m27s	5m39s	4m25s-5m40s
ut / macOS	18	5m31s	7m17s	4m38s-8m47s
ut-a2a3	18	1m18s	3m28s	1m10s-6m26s
ut-a5	18	3m00s	6m12s	1m09s-12m29s
st-sim-a2a3 / Ubuntu	19	5m28s	5m57s	4m27s-6m09s
st-sim-a2a3 / macOS	19	5m04s	7m11s	4m22s-7m30s
st-sim-a5 / Ubuntu	19	5m26s	5m35s	4m21s-5m46s
st-sim-a5 / macOS	19	4m53s	6m15s	4m06s-6m42s
st-onboard-a2a3	19	7m10s	9m47s	6m23s-15m09s
st-onboard-a5	19	5m14s	9m53s	4m36s-13m16s
st-network1-onboard-a2a3	19	2m51s	4m00s	2m40s-5m29s

The complete PR workflow, measured from workflow start until all jobs completed, was:

Metric	p50	p90	Observed range
Complete PR CI wall time	12m26s	20m09s	5m18s-50m30s

Simulator comparison with the issue baseline

Job	Initial p50	Current p50	Change
a2a3sim / Ubuntu	10.9m	5m28s	-49.8%
a2a3sim / macOS	11.2m	5m04s	-54.8%
a5sim / Ubuntu	10.5m	5m26s	-48.3%
a5sim / macOS	10.9m	4m53s	-55.2%

The simulator p50 has therefore fallen by approximately 48%-55%. The remaining simulator tail is most visible on macOS, where the p90 values are 7m11s for a2a3sim and 6m15s for a5sim.

Onboard interpretation

The Onboard wall times above include time spent waiting inside task-submit after environment preparation when the requested NPU devices are occupied. The same applies to hardware UT jobs that acquire devices through task-submit. These numbers are useful as developer-visible CI latency, but they are not queue-free execution measurements and must not be used by themselves to infer NPU or runner performance.

For a queue-free before/after comparison, each hardware run should record and subtract every device acquisition wait:

queue_time  = task-submit START_TIME - SUBMIT_TIME
active_time = job wall time - sum(queue_time)

The queue-free Onboard p50/p90 should then be reported separately by runner tier, as required by the acceptance criteria. The stable work before the first NPU acquisition currently has these p50 values: approximately 2m13s for a2a3 Onboard and 3m22s-4m22s for a5 Onboard, depending on the host tier.

[doraemonmj]

GitHub CI 每周性能

最后更新：2026-09-03 11:47 Asia/Shanghai · 统计周期：2026-W35（北京时间周一至周日）

飞书完整历史与周报

Job / OS / runner tier	n	p50	p90	p50 vs 前4周
st-onboard-a2a3 / linux / unclassified-024d035e	30	8m59s	9m52s	—
st-onboard-a2a3 / linux / unclassified-19a820eb	53	7m18s	16m27s	-51.5%
st-onboard-a2a3 / linux / unclassified-1edf9790	27	8m49s	9m49s	—
st-onboard-a2a3 / linux / unclassified-a9dee0af	25	8m49s	10m57s	—
st-onboard-a2a3 / linux / unclassified-e009bd72	46	7m17s	16m17s	—
st-onboard-a2a3 / linux / unclassified-e31ae59a	23	8m51s	11m04s	—
st-onboard-a5 / linux / unclassified-543453d1	79	10m14s	15m50s	+47.2%
st-onboard-a5 / linux / unclassified-ae073d55	131	4m56s	5m34s	+18.4%
st-sim-a2a3 / macos / github-standard	215	6m07s	7m36s	-43.4%
st-sim-a2a3 / ubuntu / github-standard	219	5m41s	6m05s	-45.5%
st-sim-a5 / macos / github-standard	208	5m57s	7m25s	-38.3%
st-sim-a5 / ubuntu / github-standard	214	5m22s	5m42s	-43.7%

p50/p90 仅使用成功 job；失败、取消和样本量记录在飞书周报中。 Set up environment 当前按 install+build 合并统计。