ci: share one Linux native build across CI workflows

[sunchao]

Which issue does this PR close?

Part of #5830. Follow-up to #3249, which shared native artifacts within individual workflows.

Rationale for this change

The umbrella CI workflow still compiles the same default Linux native library separately for Linux, each selected Spark version, and each selected Iceberg version. A source-change PR normally builds three copies; a full merge-group or manual run builds nine. This change builds one copy and shares it with those consumers, reducing duplicate compilation and runner usage. Actual time savings depend on cache state and will need measurement in CI.

What changes are included in this PR?

• Add one reusable Linux native producer using the existing Cargo ci profile, JDK 17, compiler flags, and main-only cache save policy.
• Make Linux JVM/TPC tests, Spark SQL, and Iceberg callers wait for that producer and pass its artifact through a required input. Keep Spark/JDK-specific compiled JVM artifacts separate.
• Run independent Linux lint, compile-only checks, Celeborn compatibility tests, and Rust debug tests in a separate workflow alongside the native producer.
• Derive build_linux_native in compute-changes.py as the union of its nine selected consumer outputs, after path/event/label decisions. The workflow reads one output, including on empty-input manual dispatch. Tests exercise Python selection directly and configuration guards verify workflow output wiring.
• Remove duplicate native compilation and uploads from consumers. Keep Iceberg shard preparation as a small separate job. The Spark JVM producer sets up Java only; it retains the test container to preserve workspace and Maven-cache paths for reused Spark compilation.
• Add configuration guards and regression coverage for producer selection, caller dependencies, output exports, artifact mapping, and accidental duplicate producers. Include both new caller jobs in required_checks.needs, so producer failure cannot be hidden by skipped consumers.
• Limit GITHUB_TOKEN to contents: read in the two new reusable workflows and their callers, addressing the CodeQL missing-permissions alerts without changing deployment permissions.

Both Linux callers preserve the existing path and event selection. Independent checks have no dependency on the shared native build, so they still report results if native compilation fails. They appear under PR Checks (Linux); the JVM/TPC consumers remain under PR Build (Linux). Regression checks prevent restoring the dependency or moving independent jobs back into the gated workflow. Rust formatting runs before native compilation and before the independent build/test jobs. The moved Scala syntactic lint preserves upstream's retried dependency download and offline check. Artifact transfers retain upstream's retry wrappers, including the added download in the Spark JVM producer, and the local-action checkout guard remains intact. macOS, Rust debug tests, and feature-specific workflows retain their separate builds. Artifact retention remains one day.

How are these changes tested?

Passed locally on the rebased branch:

• actionlint -color -shellcheck=
• python3 dev/ci/check-ci-config.py
• python3 dev/ci/test-ci-config.py (28 tests, including mutated producer/consumer gates, missing or remapped output exports, unregistered consumers, and rejection of both direct and retried native downloads by independent checks)
• python3 dev/ci/test-native-build-selection.py (19 tests, including 67,584 path/event combinations, direct/CLI merge-group and cache-refresh push selection, Spark 3.5 label opt-in, nonconsumer labels, and empty-input manual dispatch)
• python3 dev/ci/check-suites.py
• python3 dev/ci/check-benchmark-runner.py (49 suites)
• python3 dev/ci/test-iceberg-shards.py (15 tests)
• node --test dev/ci/pr-type-label.test.mjs (4 tests)
• Compared all six moved independent Linux jobs against the new base, preserving their behavior including the Scala lint retry. Verified merge-queue triggers, concurrency, base/head path comparison, download retries, checkout guards, and Required Checks. The aggregator retains both new jobs. Non-overlapping upstream files, including queue rules and the DataFusion 55.1 update, are unchanged.
• Prettier check on the changed Markdown and git diff --check

Apache RAT was not rerun locally: the earlier local attempt was blocked by the read-only Maven cache and unavailable Maven Central. Hosted preflight passed on the previous head. No full native or Spark build was run locally.

Hosted CI on the previous head 232560cafa passed. The branch is now rebased onto main at 71ccee975b, including the merged queue policy from #5843; fresh hosted CI must confirm the rebased head.

The current policy selects Linux, Spark 4.1, and Iceberg 1.11 by default on pull requests; merge groups select all nine native consumers when their paths match. Main pushes select the Linux group to refresh caches, plus documentation deployment where applicable. Manual runs select every route. The native producer is derived from the selected consumers in every case.

Opt-in external Spark 3.4/JDK 11 and failed-job reruns remain unverified. Failed-job reruns can reuse the successful producer's artifact while it is retained; after expiry, rerun the full workflow.

[rich7420]

LGTM

[andygrove]

LGTM but I would like to merge #5842 first so we can unblock setting up the merge queue to reduce CI resource usage. I reviewed this PR assuming that it will go in after #5842.

Once #5842 is in, two things will need to happen on the rebase here:

1. ci.yml merges cleanly between the two PRs, but #5842 adds a required_checks aggregator whose needs: must list every ci.yml job, and a preflight guard in check-ci-config.py that enforces it. The merged file fails that guard until pr_build_linux_checks and build_linux_native are added to required_checks.needs. The guard message names them, so the rebase will tell you. build_linux_native in particular has to be there and cannot be exempted: if the producer fails, GitHub marks all nine consumers skipped, and the aggregator treats skipped as pass, so without the producer in needs a broken native build would produce a green Required Checks.

2. #5842 hardens Lint Scala (syntactic) against Maven Central connection resets by splitting it into a retried cs launch scalafix:0.14.6 -- --version warm-up followed by the real check under cs launch --mode offline. Since this PR moves that job into pr_build_linux_checks.yml, that split needs to come along. The iceberg_spark_test_reusable.yml change in #5842 (routing the shard-inventory upload through upload-artifact-retry) merges cleanly onto your prepare-shards version.

After both land, check-ci-config.py will carry two hand-rolled YAML readers (block_mapping here and the line regexes in #5842), both justified by PyYAML not being on the preflight runner. I will open a follow-up issue to install PyYAML in preflight and collapse them rather than leave both in place.

I verified the two-way compatibility locally: your checker passes against a ci.yml that includes the required_checks job, and the #5842 checker passes against your ci.yml once the two ids are added.

[andygrove]

#5850 moved routing policy out of ci.yml expressions and into compute-changes.py, and this nine-way OR is the same kind of policy. Would it work to compute a build_linux_native output there as the union of the consumer outputs, so this becomes needs.changes.outputs.build_linux_native == 'true'? Then test-native-build-selection.py would not need to regex-parse the YAML if: block to prove the two sides agree, and the README routing table gets a single row like the others.

[andygrove]

Now that this job only compiles JVM test classes, does it still need the amd64/rust container and the Rust toolchain from setup-builder? If the skip-native-build Maven path genuinely needs them, a short comment here would save the next reader the same question.

[andygrove]

One more heads-up from the merge-queue side, following on from my review above.

#5843 (the PR that actually enables the queue, stacked on #5842) changes POLICY in dev/ci/compute-changes.py so that a push to main selects only docs. Every test job moves to the merge_group event instead, because the queue has already tested the exact tree that lands.

That will break test_main_and_manual_runs_include_legacy_consumers in dev/ci/test-native-build-selection.py here, which asserts event="push" selects every native consumer. Whichever of #5841 and #5843 lands second needs to update that test (the workflow_dispatch half stays true). If this PR lands first, #5843 will pick it up on its rebase along with adding pr_build_linux_checks and build_linux_native to required_checks.needs. If #5843 lands first, the push assertion here should become merge_group.

Also FYI, run-spark-3.4-tests and run-spark-3.5-tests now exist as labels, so the label-gated selection tests here have real labels behind them.

[andygrove]

Three things, all about the rebase rather than the design. The producer and consumer wiring itself looks right to me, and I checked that build_linux_native is the exact union of the nine consumer outputs on this head.

#5852 landed on main after this branch was last pushed, adding a Bootstrap Maven step calling ./.github/actions/maven-bootstrap to five jobs in pr_build_linux.yml. Three of those five are the jobs this PR moves into pr_build_linux_checks.yml, namely lint-java, build-spark-4-1 and celeborn-reflection-compatibility. A rebase cannot carry that step into a file this branch creates from scratch, and pr_build_linux.yml conflicts, so the natural resolution is to take the deletion and lose the retry for those three. The two TPC jobs stay behind and keep theirs. Nothing in check-ci-config.py verifies that a job calling a bare ./mvnw has the bootstrap step ahead of it, so this would be silent until the next Maven Central blip takes out a queue-gating job. Could you add it back to those three after the rebase, and update the ROUTING_CASES comment that says the composite is called only from pr_build_linux.yml? Given how easily it drops out in a file move, is it worth a guard for it?

NATIVE_CONSUMERS is a flat list of output keys, one per consumer job, and #5871 has already broken that assumption on main. It added spark_4_1_hive as a second output feeding the same spark_4_1 job, which is now gated on needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'. On a labeled run with run-spark-4.1-hive-tests, spark_4_1 is false and spark_4_1_hive is true, so the union computes build_linux_native as false, the producer is skipped, and spark_4_1 is then skipped through its needs even though its if is true. The label would quietly do nothing and Required Checks would still go green. compute-changes.py is the one file that auto-merges cleanly here, so the union will miss the new key silently. native_selection_failures will fail preflight on the if: string mismatch, but the tempting fix there is to relax the comparison, which leaves the hole. Would it make sense to let a consumer map to a set of output keys rather than one?

Last one is coverage. The hosted run only selected pr_build_linux, spark_4_1 and iceberg_1_11, so six of the nine rewired consumers have not run against the shared artifact. spark_3_4 is the one I would most want to see, because its build job in spark_sql_test_reusable.yml is the only consumer whose toolchain setup actually changed, swapping ./.github/actions/setup-builder for a bare actions/setup-java@v4 at JDK 11, and it now consumes a libcomet.so linked against JDK 17. The linux-test Spark 3.4 lane already proves the library loads under JDK 11, but the new setup path has no coverage. These are queue-tier jobs, so a break blocks every merge rather than one PR. Could you apply run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests and run-iceberg-tests once on the rebased head?