Add opt-in CDC chunked uploads to GrpcStore

[erneestoc]

Problem

CDC chunking (#2497) covers the client↔CAS edge, but NativeLink's own
transfers don't use it: GrpcStore's split_blob/splice_blob are proxy
forwarders only, so workers upload huge output artifacts whole on every
build, even when almost all of their content is already in the CAS from a
previous version.

Mechanism

Opt-in GrpcSpec.experimental_chunked_uploads { min_blob_size_bytes (8MiB default), avg_chunk_size_bytes (512KiB default), max_chunk_count }. Blobs
at/above the threshold are FastCDC-split with the same parameters as the
server's on-demand splitter (shared chunk digests), each ~8MiB window does
FindMissingBlobs → BatchUpdateBlobs for the missing chunks only, then
one SpliceBlob assembles the blob remotely. Single pass, memory bounded
to the window. Blobs that could exceed max_chunk_count take the plain
path up front (the chunked path cannot fall back mid-stream). The digest
function is resolved once at update() entry and threaded explicitly, so
chunk hashes, the digest_function field, and the blob digest cannot
diverge.

Measured (256MiB artifact, real gRPC over counted TCP, byte-verified)

round	plain: wire / wall	chunked: wire / wall
v1 cold	269.2MB / 125ms	268.7MB / 912ms
v2, 5% clustered churn	269.2MB / 112ms	28.3MB (10.5%) / 737ms
v3, 64KiB insertion	269.3MB / 139ms	1.07MB (0.4%) / 706ms
v3 re-upload	269.3MB / 129ms	67KB (0.03%) / 201ms

Honest scope notes:

• The win requires clustered churn (what linkers/bundlers produce);
  uniformly scattered small edits touch every chunk and save nothing —
  falsified explicitly during benching.
• The hash fan-out is capped at min(8, available_parallelism) and does
  the same total CPU work as serial hashing (narrower burst, not more
  work). The hash stage follows the instance digest function: measured
  serial 473ms (SHA256) vs 117ms (BLAKE3) per 256MiB, cold chunked upload
  872ms vs 478ms — BLAKE3 deployments roughly halve the cold cost.
• Cold uploads pay a CPU pass: chunk digests are hashed in parallel on
  the blocking pool (profiled: hashing was ~4x the FastCDC boundary pass;
  parallelizing cut the pass from ~590ms to ~190ms per 256MiB). Still a
  real cost on fast LANs, irrelevant when bandwidth-bound — hence opt-in
  with a size floor. Pipelining window hashing with the previous window's
  network flush is a named follow-up, as is a worker-side chunk-map cache
  for re-upload short-circuiting.
• No GetCapabilities probe yet: the operator must only enable this
  against a chunking-capable backend (first large upload fails loudly
  otherwise). Probe is a named follow-up.
• Precedence: for blobs above this threshold, chunked wins over
  experimental_remote_cache_compression (chunks transfer uncompressed);
  documented on both config fields.

Tests

7 chunked-upload tests (routing below threshold and above chunk-count cap,
chunk+splice byte-identity, missing-chunks-only transfer, splice-NotFound
semantics, duplicate missing_blob_digests tolerance, config validation).
Suites: cargo check --tests --workspace clean; bazel store+config 34/34
with clippy pedantic + nightly rustfmt aspects. Metrics group
chunked_uploads.

Related: #2591 / #2592 / #2593 (upload-path series); the download side
(worker chunk cache) is a design-stage follow-up.

Production-failure follow-up: a cold-build test surfaced eviction amplification on a size-capped CAS (chunks + assembled blob store each large output twice, and splice promotes chunks in the LRU, evicting unrelated still-referenced blobs). Root cause and server-side fix (post-splice chunk demotion) are in #2627, which predates this PR (any chunking client since #2497 triggers it); this PR's config docs now carry the ~2x-headroom sizing warning.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC

---

This change is 

Review update (2026-07-22)

The CDC upload path was hardened after adversarial review:

• Retryable per-entry BatchUpdateBlobs failures are retried using the configured retrier before failing the upload.
• Batch responses now require exact requested-digest coverage; duplicate, unrequested, omitted, invalid, or missing-status entries are rejected as malformed.
• The splice path rechecks missing chunks immediately before SpliceBlob and fails clearly if chunks remain unavailable.
• Chunk-count gating uses ceiling division, and CasServer/GrpcStore average-chunk-size configuration is validated for consistency.
• Added retry, omitted-response, malformed-response, and configuration regressions.

This feature remains opt-in and applies to NativeLink worker/StoreDriver uploads; ordinary external ByteStream uploads are unchanged. There is still no capability/seed negotiation, so operators must enable it only against a backend supporting the required Split/Splice behavior.

Focused Cargo and Bazel chunked-upload suites, formatting checks, and git diff --check pass.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 26, 2026 2:55pm
nativelink-aidm	Ready	Preview, Comment	Jul 26, 2026 2:55pm

[MarcusSorealheis]

@erneestoc All I did here was catch it up to (merge in) main, add documentation, and add some validation logic to inform users/agents at startup that try to enable chunked uploads without the required backend capabilities that this feature requires.

[MarcusSorealheis]

There are a couple fast follows so upon your approval you should merge.

[MarcusSorealheis]

LGTM - pending author's re-review.

[MarcusSorealheis]

and @amankrx

[erneestoc]

@MarcusSorealheis We hit a correctness failure while testing this PR CDC uploads on a filesystem-backed, size-capped CAS: after ~17.5 minutes, actions failed with Lost inputs no longer available remotely for the same 78-byte bundletool_experimental.build_data.txt digest, and Bazel’s automatic retry failed immediately on that digest too.

The likely cause is CDC’s temporary storage/recency behavior: a large output exists as retained chunk blobs plus the assembled blob. Before #2627, SpliceBlob promotes the chunks in LRU order; when the assembled blob is committed under CAS pressure, eviction can choose an unrelated, still-action-cache-referenced input. That leaves AC pointing at bytes no longer in CAS.

#2627 addresses that mechanism by demoting the CDC chunks before the splice commit’s eviction pass. It retains chunks normally, but makes them preferred eviction candidates under pressure—trading an incremental CDC hit for build correctness. We think #2595 should be reviewed/merged with #2627 (or an equivalent fix), not independently.

We will validate the pair with a cold-cache correctness run first, then a controlled incremental large-artifact rebuild. Note that #2627 still requires sufficient CAS headroom; it prevents the chunk-recency eviction pathology but cannot make a genuinely undersized CAS safe.