**TLDR: This PR enables support for content-defined chunking (FastCDC) for large uploads/downloads to remote cache, saving ~40% on storage and upload bandwidth, and making builds faster by deduplicating similar artifacts across builds.**

RELNOTES[NEW]: Added `--experimental_remote_cache_chunking` flag to read and write large blobs to/from the remote cache in chunks. Requires server support.

## Motivation

Actions like `GoLink` and `CppLink` produce very large output files that are often similar between builds. A small source change can cause a cache miss, wasting storage, bandwidth, and time on nearly-identical artifacts.

Content-Defined Chunking (CDC) addresses this by splitting files at content-determined cut points. Because cut points are derived from the file content itself, small changes, even ones that shift bytes around, tend to affect only a few chunks. This makes action outputs effectively incremental: even though the action must re-run, the upload, download, and storage costs shrink dramatically.

## Results

Benchmarked across the last 50 commits of the BuildBuddy repo (server and client on the same host):

| Scenario | Upload | Download | RPCs | Disk Cache | Avg Build Time |
|---|---|---|---|---|---|
| chunking + disk cache | 52.0 GB | 0 B | 626K | 146.6 GB | 55s |
| chunking, no disk cache | 49.2 GB | 343.2 GB | 4.1M | — | 54s |
| no chunking + disk cache | 85.6 GB | 0 B | 273K | 246.5 GB | 100s |
| no chunking, no disk cache | 89.7 GB | 343.8 GB | 2.5M | — | 97s |

Key takeaways:
- **~40% less data uploaded** (52 GB vs 90 GB)
- **~40% smaller disk cache** (147 GB vs 247 GB)
- Download size is mostly unchanged (~0.2% increase) because we don't yet store downloaded chunks in the output base. Using a disk cache is recommended for full benefit; output-base chunk reuse is planned.
- RPC count increases as expected since requests become smaller and more granular.
- **faster builds** (depends on conditions, like cache async, compression, & network speed)

Additional benefits: better load balancing across distributed clusters (fewer long-running RPCs) and more granular retries on unstable networks.

## Try It Out

Anyone can try chunking today using BuildBuddy:

1. Sign up for a free account at [buildbuddy.io](https://buildbuddy.io)
2. Get an API key with write access
3. Use the Bazel fork from bazelbuild/bazel#28903
4. Build!

```
USE_BAZEL_VERSION="tyler-french/9.1.0-cdc" bazel build //... \
  --experimental_remote_cache_chunking \
  --remote_header=x-buildbuddy-cdc-enabled=true \
  --remote_cache=grpcs://remote.buildbuddy.io
```

## How It Works

**Write path:**
1. Check if blob exceeds the chunking threshold.
2. Run FastCDC to compute chunk boundaries.
3. Call `FindMissingBlobs` to identify which chunks the server already has.
4. Upload only the missing chunks.
5. Call `SpliceBlob` to register the blob-to-chunks mapping on the server.

**Read path:**
1. Check if blob exceeds the chunking threshold.
2. Call `SplitBlob` to get the chunk list for this blob.
3. Download and reassemble the chunks.

If `--disk_cache` is enabled, previously downloaded chunks are served locally.

Closes #28437.

PiperOrigin-RevId: 885108353

**TLDR: This PR enables support for content-defined chunking (FastCDC) for large uploads/downloads to remote cache, saving ~40% on storage and upload bandwidth, and making builds faster by deduplicating similar artifacts across builds.**

RELNOTES[NEW]: Added `--experimental_remote_cache_chunking` flag to read and write large blobs to/from the remote cache in chunks. Requires server support.

## Motivation

Actions like `GoLink` and `CppLink` produce very large output files that are often similar between builds. A small source change can cause a cache miss, wasting storage, bandwidth, and time on nearly-identical artifacts.

Content-Defined Chunking (CDC) addresses this by splitting files at content-determined cut points. Because cut points are derived from the file content itself, small changes, even ones that shift bytes around, tend to affect only a few chunks. This makes action outputs effectively incremental: even though the action must re-run, the upload, download, and storage costs shrink dramatically.

## Results

Benchmarked across the last 50 commits of the BuildBuddy repo (server and client on the same host):

| Scenario | Upload | Download | RPCs | Disk Cache | Avg Build Time |
|---|---|---|---|---|---|
| chunking + disk cache | 52.0 GB | 0 B | 626K | 146.6 GB | 55s |
| chunking, no disk cache | 49.2 GB | 343.2 GB | 4.1M | — | 54s |
| no chunking + disk cache | 85.6 GB | 0 B | 273K | 246.5 GB | 100s |
| no chunking, no disk cache | 89.7 GB | 343.8 GB | 2.5M | — | 97s |

Key takeaways:
- **~40% less data uploaded** (52 GB vs 90 GB)
- **~40% smaller disk cache** (147 GB vs 247 GB)
- Download size is mostly unchanged (~0.2% increase) because we don't yet store downloaded chunks in the output base. Using a disk cache is recommended for full benefit; output-base chunk reuse is planned.
- RPC count increases as expected since requests become smaller and more granular.
- **faster builds** (depends on conditions, like cache async, compression, & network speed)

Additional benefits: better load balancing across distributed clusters (fewer long-running RPCs) and more granular retries on unstable networks.

## Try It Out

Anyone can try chunking today using BuildBuddy:

1. Sign up for a free account at [buildbuddy.io](https://buildbuddy.io)
2. Get an API key with write access
3. Use the Bazel fork from #28903
4. Build!

```
USE_BAZEL_VERSION="tyler-french/9.1.0-cdc" bazel build //... \
  --experimental_remote_cache_chunking \
  --remote_header=x-buildbuddy-cdc-enabled=true \
  --remote_cache=grpcs://remote.buildbuddy.io
```

## How It Works

**Write path:**
1. Check if blob exceeds the chunking threshold.
2. Run FastCDC to compute chunk boundaries.
3. Call `FindMissingBlobs` to identify which chunks the server already has.
4. Upload only the missing chunks.
5. Call `SpliceBlob` to register the blob-to-chunks mapping on the server.

**Read path:**
1. Check if blob exceeds the chunking threshold.
2. Call `SplitBlob` to get the chunk list for this blob.
3. Download and reassemble the chunks.

If `--disk_cache` is enabled, previously downloaded chunks are served locally.

Closes #28437.

PiperOrigin-RevId: 885108353
Change-Id: I395e2af68016cc2a9dc32ab546f680788d5c5c55

**TLDR: This PR enables support for content-defined chunking (FastCDC) for large uploads/downloads to remote cache, saving ~40% on storage and upload bandwidth, and making builds faster by deduplicating similar artifacts across builds.**

RELNOTES[NEW]: Added `--experimental_remote_cache_chunking` flag to read and write large blobs to/from the remote cache in chunks. Requires server support.

Actions like `GoLink` and `CppLink` produce very large output files that are often similar between builds. A small source change can cause a cache miss, wasting storage, bandwidth, and time on nearly-identical artifacts.

Content-Defined Chunking (CDC) addresses this by splitting files at content-determined cut points. Because cut points are derived from the file content itself, small changes, even ones that shift bytes around, tend to affect only a few chunks. This makes action outputs effectively incremental: even though the action must re-run, the upload, download, and storage costs shrink dramatically.

Benchmarked across the last 50 commits of the BuildBuddy repo (server and client on the same host):

| Scenario | Upload | Download | RPCs | Disk Cache | Avg Build Time |
|---|---|---|---|---|---|
| chunking + disk cache | 52.0 GB | 0 B | 626K | 146.6 GB | 55s |
| chunking, no disk cache | 49.2 GB | 343.2 GB | 4.1M | — | 54s |
| no chunking + disk cache | 85.6 GB | 0 B | 273K | 246.5 GB | 100s |
| no chunking, no disk cache | 89.7 GB | 343.8 GB | 2.5M | — | 97s |

Key takeaways:
- **~40% less data uploaded** (52 GB vs 90 GB)
- **~40% smaller disk cache** (147 GB vs 247 GB)
- Download size is mostly unchanged (~0.2% increase) because we don't yet store downloaded chunks in the output base. Using a disk cache is recommended for full benefit; output-base chunk reuse is planned.
- RPC count increases as expected since requests become smaller and more granular.
- **faster builds** (depends on conditions, like cache async, compression, & network speed)

Additional benefits: better load balancing across distributed clusters (fewer long-running RPCs) and more granular retries on unstable networks.

Anyone can try chunking today using BuildBuddy:

1. Sign up for a free account at [buildbuddy.io](https://buildbuddy.io)
2. Get an API key with write access
3. Use the Bazel fork from bazelbuild#28903
4. Build!

```
USE_BAZEL_VERSION="tyler-french/9.1.0-cdc" bazel build //... \
  --experimental_remote_cache_chunking \
  --remote_header=x-buildbuddy-cdc-enabled=true \
  --remote_cache=grpcs://remote.buildbuddy.io
```

**Write path:**
1. Check if blob exceeds the chunking threshold.
2. Run FastCDC to compute chunk boundaries.
3. Call `FindMissingBlobs` to identify which chunks the server already has.
4. Upload only the missing chunks.
5. Call `SpliceBlob` to register the blob-to-chunks mapping on the server.

**Read path:**
1. Check if blob exceeds the chunking threshold.
2. Call `SplitBlob` to get the chunk list for this blob.
3. Download and reassemble the chunks.

If `--disk_cache` is enabled, previously downloaded chunks are served locally.

Closes bazelbuild#28437.

PiperOrigin-RevId: 885108353
Change-Id: I395e2af68016cc2a9dc32ab546f680788d5c5c55