Skip to content

fix(stream): report a reusable layer-shard cache at its real size in the disk pre-flight - #1709

Merged
MakazhanAlpamys merged 3 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1703-fix
Oct 13, 2026
Merged

MakazhanAlpamys merged 3 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1703-fix

Conversation

@drakeo338

@drakeo338 drakeo338 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1703.

What does this PR do?

When the layer-shard cache was reusable, the disk pre-flight still printed the estimate for writing a fresh cache as the "Layer-shard cache" size, so it and "Projected total on disk" described a cache that was not there. shard_cache_disk_bytes now adds up the cache's own files (index, extras shard, large shards, each layer file across stripe folders), and the pre-flight reports that. Stray files in the folder are not counted, and an unmeasurable file keeps the estimate.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Tests

Checklist

  • ruff check src/soup_cli/ scripts/ tests/ passes
  • pytest tests/ -v passes
  • Updated relevant docs (README.md and the matching page under docs/) if needed
  • Added a changelog fragment, or this change has no user-visible impact

Not ticked: the full suite was not run. On head a3ab7f2, tests/test_disk_tier_stripe_setup.py gives 42 passed and ruff check src/ scripts/ tests/ is clean; the 62 related test files fail only the same 109 tests that fail on main (missing optional deps). No docs describe this output.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drakeo338, thank you: the source change is right, and I checked that it is right in the way the issue cares about most. shard_cache_disk_bytes (src/soup_cli/utils/layer_shard.py:1763) names the files by what the cache check requires (index, extras shard, each large_* shard from index.large_keys, each layer file through layer_paths), so a cache striped over several folders is counted on every drive, a stray or stale file is not in the sum (your tests plant .soup.x.tmp, notes.bin and a large_embed.safetensors in each folder), only file metadata is read, and OSError returns None, so the pre-flight keeps its estimate.

To check "write admission is untouched" by running it rather than reading it, I drove the real _setup_streaming_transformers on your fixture in 16 states (cache miss, reusable cache, materialization, and materialization with a cache already on disk; single and striped; 1 byte free and 10 GB free) on main (66cc8b9ad) and on your head. Every refusal message and every write figure is identical; the only difference is the reported size of a reusable cache (316,032 B estimate vs 316,539 B real, and 316,667 B striped). Your new tests fail 4 of 23 on main and pass 23 of 23 here.

What needs to change is what pins it. I mutated the change in 15 ways; 13 fail a test. Two survive:

  1. The vocabulary shards are not pinned. Deleting paths.extend(large_shard_path(shard_dir, key) for key in index.large_keys) (layer_shard.py:1771) passes all 23 tests, because the tiny checkpoint ties its embeddings and large_keys is (). The issue attributes its 4.23 vs 6.07 GB gap to exactly those unquantized vocabulary weights, so this is the part a real run depends on.
  2. The materialization state is only tested at the renderer. test_a_materialization_is_refused_only_without_room calls _render_stream_disk_preflight directly. Making _disk_preflight pass materialize_bytes=0 (stream_setup.py:732) passes all 23. The reusable state is also only tried with 1 byte free.

Also not exercised through the setup: the issue's "a cache file that disappears after validation does not fail the pre-flight". Your helper-level test deletes the file and your other test patches the helper to return None; nothing deletes the file between inspect_shard_cache and the sizing. These tests close all three; with them applied I get 23 -> 30 passing, and each of the two surviving mutations, and dropping the OSError handler, now fails its own test. Add to TestAReusableCacheIsReportedAtItsRealSize (after test_a_deleted_layer_file_is_not_measured):

    def test_a_cache_file_lost_after_the_check_does_not_fail_the_preflight(
        self, tmp_path, monkeypatch
    ):
        """The file goes between `inspect_shard_cache` and the sizing; the estimate stays."""
        seen = _spy_preflight(monkeypatch)
        captured, _said, _drive = _drive_setup(tmp_path, monkeypatch, striped=True)
        estimate = seen[-1]["shard_bytes"]  # the first run writes the cache: a plain estimate
        assert _cache_files_bytes(tmp_path) != estimate  # else the last assert proves nothing
        primary = _primary_folder(tmp_path)
        victim = layer_shard.layer_paths(primary, captured["index"])[1]  # on the second drive
        assert os.path.exists(victim) and os.path.dirname(victim) != primary
        real = layer_shard.inspect_shard_cache
        lost = []

        def inspect_then_lose_a_file(*args, **kwargs):
            result = real(*args, **kwargs)
            if not lost and result[0] is not None:
                os.remove(victim)
                lost.append(victim)
            return result

        monkeypatch.setattr(layer_shard, "inspect_shard_cache", inspect_then_lose_a_file)
        _drive_setup(tmp_path, monkeypatch, striped=True)  # must not raise
        assert lost == [victim]
        assert seen[-1]["shard_write_bytes"] == 0
        assert seen[-1]["shard_bytes"] == estimate

    def test_an_untied_striped_cache_counts_its_vocabulary_shards(self, tmp_path):
        """The tied fixture above has no `large_*` file; an untied checkpoint writes two."""
        from tests.test_disk_tier_stripe_runtime import _tiny_llama

        weights = _tiny_llama(tmp_path, tie=False)
        stripe = tmp_path / "stripe"
        stripe.mkdir()
        cache = str(tmp_path / "cache" / "tiny")
        index = layer_shard.shard_checkpoint(
            weights, cache, dtype="float32", arch="llama", stripe_roots=(str(stripe),)
        )
        assert index.large_keys and index.layer_roots == (0, 1, 0)
        folders = layer_shard.stripe_dirs(cache, index.stripe_roots)
        files = [
            os.path.join(folder, name)
            for folder in folders
            for name in os.listdir(folder)
            if name != layer_shard.STRIPE_MARKER_NAME
        ]
        assert any(os.path.basename(path).startswith("large_") for path in files)
        assert layer_shard.shard_cache_disk_bytes(cache, index) == sum(
            os.path.getsize(path) for path in files
        )

In TestRefusalIsUnchanged, parametrize the reusable test and add the setup-level materialization test:

    @pytest.mark.parametrize("free", [1, 10_000_000_000])
    def test_a_reusable_cache_needs_no_space(self, tmp_path, monkeypatch, free):
        _drive_setup(tmp_path, monkeypatch, striped=False)
        self._free(monkeypatch, free)
        _captured, said, _drive = _drive_setup(tmp_path, monkeypatch, striped=False)
        assert "(reusable)" in said

    @pytest.mark.parametrize("copy_bytes, refused", [(2_000_000_000, True), (100_000_000, False)])
    @pytest.mark.parametrize("cache_exists", [False, True])
    def test_a_materialization_through_the_setup_is_refused_only_without_room(
        self, tmp_path, monkeypatch, copy_bytes, refused, cache_exists
    ):
        """The copy is charged through the setup's own call, with or without a cache on disk."""
        import dataclasses

        import soup_cli.utils.spectrum_scan as spectrum_scan

        if cache_exists:
            _drive_setup(tmp_path, monkeypatch, striped=False)

        def needs_a_copy(model, *, before_materialize=None):
            plan = spectrum_scan.plan_model_weights(model)
            plan = dataclasses.replace(
                plan, materialize_bytes=copy_bytes, materialized_copy_bytes=copy_bytes
            )
            before_materialize(plan)
            return plan.weights_dir

        monkeypatch.setattr(spectrum_scan, "resolve_model_weights", needs_a_copy)
        self._free(monkeypatch, 1_000_000_000)  # the tiny cache fits; the copy decides
        if refused:
            with pytest.raises(ValueError, match="materialized weight copy"):
                _drive_setup(tmp_path, monkeypatch, striped=False)
        else:
            _drive_setup(tmp_path, monkeypatch, striped=False)

Two smaller things:

  • The fragment should be changelog.d/0.75.1/1709.fixed.md (named after the pull request), ending (#1703 by @drakeo338 in #1709). It currently ends (#1703) with no credit.
  • docs/performance-and-quantization.md:803 describes this pre-flight ("the per-layer shard cache"). One sentence there would do: a cache Soup will reuse is reported at the size of its files, and one it has to write is reported at the estimate.

Ran on 14eaa91f: tests/test_disk_tier_stripe_*.py (all nine) plus test_issue374_stream_disk_preflight.py and the changelog-fragment test: 193 passed, 6 skipped; ruff check clean on the three Python files. The 51 test files that name layer_shard or stream_setup, plus the fragment test, on your head merged with current main (a517bd189, a clean merge): 1710 passed, 95 skipped. Once those are in, I have nothing else open on this one.

drakeo338 added a commit to drakeo338/Soup that referenced this pull request Oct 9, 2026
…nd a lost cache file (MakazhanAlpamys#1703)

Review of MakazhanAlpamys#1709 found two mutations that survived: dropping the large_* shards from
`shard_cache_disk_bytes` and passing `materialize_bytes=0` from the setup both left
every test green. The new tests fail on each of them.

- an untied striped cache is counted with its two large_* vocabulary shards
- a materialization copy is charged through `_setup_streaming_transformers`, with and
  without a cache on disk, and a reusable cache is tried with and without room
- a cache file removed between `inspect_shard_cache` and the sizing keeps the estimate
  and does not fail the pre-flight

The changelog fragment is renamed to the pull request number and credited, and the
performance guide says what the pre-flight reports for a reusable cache.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Pushed d84bcdd: Added the three requested tests (vocabulary shards of an untied striped cache, materialization charged through the setup, cache file lost after the check), parametrized the reusable-cache test, renamed the changelog fragment to 1709, and added the docs sentence.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drakeo338, thank you: I reviewed the delta from 14eaa91 to d84bcdd and everything I asked for is in.

  • The three tests (lost cache file after the check, untied striped cache with large_* shards, materialization charged through the setup, plus the 1 B / 10 GB parametrization) match what I proposed and assert the specific numbers and the materialized weight copy refusal.
  • I mutated the fix four ways on d84bcdd: dropping the large_keys line fails the untied-cache test, materialize_bytes=0 in the pre-flight fails two materialization cases, dropping OSError fails the lost-file test, and removing shard_estimate = cached_bytes fails the two real-size tests.
  • tests/test_disk_tier_stripe_setup.py plus tests/test_issue487_changelog_fragments.py: 51 passed; ruff check is clean on the three changed Python files.
  • The fragment is changelog.d/0.75.1/1709.fixed.md (the newest released heading in CHANGELOG.md is 0.75.1) and ends (#1703 by @drakeo338 in #1709); the assembler accepts it. The docs sentence is in the right place.
    I have nothing else open on this one.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head d84bcdd, the commit the delta review covered. See the COMMENT review above for the checks that were run.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Thank you, @drakeo338. Heads-up: main has moved and this PR no longer merges cleanly. A merge of d84bcdd1 with current main conflicts in tests/test_disk_tier_stripe_setup.py, and main is 84 commits ahead of this branch. The source files merge on their own, but #1726 changed the same part of _disk_preflight (the stripe write shares and the effective shard size), so the sizing logic from this PR has to be re-checked on top of it.

Please rebase on current main and re-run tests/test_disk_tier_stripe_setup.py. When the new head is up we will review it again; our earlier review covers d84bcdd1 only, not the merged result.

@drakeo338

Copy link
Copy Markdown
Contributor Author

Merged current main in (687f864) because I can't force-push. New head: 75da8dc. With #1726, a reusable cache has write_total 0, so its shares are 0 and shard_bytes is the measured size, counted once. The write path is the same as on main. 75da8dc also stops qwen4_exp's fallback from labelling a measured cache "approximately", and adds a test for that. tests/test_disk_tier_stripe_setup.py: 42 passed. With the fix reverted, three of those tests fail.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of 75da8dc against the head we reviewed before (d84bcdd). The source hunks are the same apart from the resolved area in _disk_preflight: shard_estimate = cached_bytes still runs before estimate_stripe_write_shares, write_total stays 0 for a reusable cache, so every share is 0 and max(shard_estimate, sum(shares)) equals the measured bytes, counted once.
A file lost after the check makes shard_cache_disk_bytes return None, so the estimate is kept and the write share stays 0 ("reusable"), which is what main does today.
approximate and cached_bytes is None is correct: the qwen4_exp fallback says approximate even for a total of 0, and a measured size is exact. test_a_measured_cache_is_not_called_approximate fails without the mask.
Tests: 42 in test_disk_tier_stripe_setup.py, none dropped or weakened (only the spy moved into _drive_setup's fourth return value). Reverting the sizing line fails 3 of them, dropping the OSError handler fails 2.
Merging into current main (5d7f2f7) is clean, and the merged tree passes the same files plus the sharder tests (94 passed). The fragment changelog.d/0.75.1/1709.fixed.md (0.75.1 is the only directory on main) and the docs line are in place.
Nit, not blocking: the docs line says a cache Soup writes "is reported at the estimate"; since #1726 it is max(estimate, bytes derived from the source headers).
CI on this head was still running when I looked (one check in progress).

@drakeo338

Copy link
Copy Markdown
Contributor Author

Rebased on current main (837794f) as one commit, a3ab7f2, replacing the merge commit. tests/test_disk_tier_stripe_setup.py: 42 passed; with main's stream_setup.py restored, 3 fail. ruff clean.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head a3ab7f2, the rebased head; the earlier approval was on d84bcdd. I reviewed 75da8dc and re-checked this push: it only moves the branch onto main 837794f, and the net patch against main is identical (same patch-id). Re-run on that code: tests/test_disk_tier_stripe_setup.py 42 passed; with the other stripe files, the pre-flight test and the fragment test 253 passed, 6 skipped, and 265 passed, 6 skipped on the head merged with main 837794f. ruff is clean on the three changed Python files.

Ten mutations of the change (the sizing line, the large_keys line, the OSError handler, the approximate mask, materialize_bytes=0, the layer files, the extras shard, index.json, a whole-folder sum, a write planned for a reusable cache) each fail at least one test; none survives. The fragment is changelog.d/0.75.1/1709.fixed.md with the credit.

Nit, not blocking: the docs line says a cache Soup writes "is reported at the estimate"; with two or more drives it is max(estimate, bytes derived from the source headers) since #1726.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta check of a3ab7f2 (rebased on main 837794f) against the previously reviewed head 75da8dc: stream_setup.py, the test file, the docs line and the changelog fragment are byte-identical, and the layer_shard.py hunk is identical apart from a line-number shift caused by main.
On a3ab7f2, ruff is clean on the three changed .py files, and test_disk_tier_stripe_setup.py + test_issue374_stream_disk_preflight.py + test_issue487_changelog_fragments.py give 72 passed.
Replacing shard_estimate = cached_bytes with pass makes 3 tests in TestAReusableCacheIsReportedAtItsRealSize fail, so the guard is mutation-checked.
The required test (ubuntu-latest, 3.12) check was still running on this head when checked.

@drakeo338

Copy link
Copy Markdown
Contributor Author

Docs nit fixed in 1c4fa0b: a cache Soup writes is now described as the estimate, or with two or more drives the larger of the estimate and the header-derived bytes. Docs only.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head 1c4fa0b. The only change since the head I approved (a3ab7f2) is one docs commit, docs/performance-and-quantization.md +2/-1: a cache Soup has to write is reported at the estimate, or with two or more drives at the larger of the estimate and the bytes derived from the source headers. That matches how the pre-flight behaves since #1726 and closes the non-blocking nit from the review. No code or test file changed, so the earlier test and mutation results stand.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Heads-up for the merge, @drakeo338: the Release v0.76.0 commit landed on main, so new changelog fragments now go under changelog.d/0.76.0/, and a fragment still under changelog.d/0.75.1/ fails test_repository_fragments_match_the_newest_release once this branch contains main. Please rebase on current main (dd62aef or newer) and move it: git mv changelog.d/0.75.1/1709.fixed.md changelog.d/0.76.0/1709.fixed.md. Nothing else changes. Tell us here when you have pushed and we will re-check the new head.

…the disk pre-flight (MakazhanAlpamys#1703)

The disk pre-flight charged a reusable layer-shard cache at the
estimated shard size. It now measures the files the cache check
requires (index, extras shard, large shards, layer files) and counts
that size once, falling back to the estimate when a file cannot be
read. A measured cache is no longer labelled "approximately", even
when the stripe-share estimate for the model is approximate.

Tests pin the vocabulary shards, the materialization charge, a lost
cache file and the label.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Rebased on main cfc27ed and moved the fragment to changelog.d/0.76.0/1709.fixed.md in 17984b7, with nothing else changed (stripe-setup and changelog-fragment tests: 63 passed, ruff clean).

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head 17984b7. I reviewed the earlier heads of this PR in full; the last one I approved was 1c4fa0b.

This push only rebases onto main (cfc27ed) and moves the fragment to changelog.d/0.76.0/1709.fixed.md: the net patch against main is identical to the approved head (same patch-id, ignoring the changelog and contributors files), the fragment text is unchanged, the fragment test passes (21 passed) and the trial merge with current main is clean.

@MakazhanAlpamys MakazhanAlpamys added the ci:full Run the full CI matrix on this PR (a maintainer adds it at approval) label Oct 13, 2026
@MakazhanAlpamys
MakazhanAlpamys merged commit 1aecae0 into MakazhanAlpamys:main Oct 13, 2026
23 checks passed
@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Merged as 1aecae0d. Thank you, @drakeo338: you made the pre-flight report the cache that is on disk - shard_cache_disk_bytes adds up the files the cache is made of (the index, the extras shard, the large shards and each layer file, wherever it is striped) instead of keeping the estimate for a write that is not going to happen. Counting those files rather than the folder keeps a stray file out of the sum, a file that cannot be measured falls back to the estimate, and a cache that has to be written is reported as before. After the first review you added the three tests that were missing (the vocabulary shards of an untied checkpoint, the materialization charge through the setup, and a cache file lost after the check), and you made a measured size stop reading "approximately". The PR says plainly that the full suite was not run. It ships in the next release.

MakazhanAlpamys added a commit that referenced this pull request Oct 13, 2026
Display names from `gh api users/<login>`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run the full CI matrix on this PR (a maintainer adds it at approval)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Layer streaming disk pre-flight reports a reusable shard cache at the coarse estimate (4.23 GB), not its real size (6.07 GB)

2 participants