Repository navigation
fix(stream): report a reusable layer-shard cache at its real size in the disk pre-flight - #1709
Conversation
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you: the source change is right, and I checked that it is right in the way the issue cares about most. shard_cache_disk_bytes (src/soup_cli/utils/layer_shard.py:1763) names the files by what the cache check requires (index, extras shard, each large_* shard from index.large_keys, each layer file through layer_paths), so a cache striped over several folders is counted on every drive, a stray or stale file is not in the sum (your tests plant .soup.x.tmp, notes.bin and a large_embed.safetensors in each folder), only file metadata is read, and OSError returns None, so the pre-flight keeps its estimate.
To check "write admission is untouched" by running it rather than reading it, I drove the real _setup_streaming_transformers on your fixture in 16 states (cache miss, reusable cache, materialization, and materialization with a cache already on disk; single and striped; 1 byte free and 10 GB free) on main (66cc8b9ad) and on your head. Every refusal message and every write figure is identical; the only difference is the reported size of a reusable cache (316,032 B estimate vs 316,539 B real, and 316,667 B striped). Your new tests fail 4 of 23 on main and pass 23 of 23 here.
What needs to change is what pins it. I mutated the change in 15 ways; 13 fail a test. Two survive:
- The vocabulary shards are not pinned. Deleting
paths.extend(large_shard_path(shard_dir, key) for key in index.large_keys)(layer_shard.py:1771) passes all 23 tests, because the tiny checkpoint ties its embeddings andlarge_keysis(). The issue attributes its 4.23 vs 6.07 GB gap to exactly those unquantized vocabulary weights, so this is the part a real run depends on. - The materialization state is only tested at the renderer.
test_a_materialization_is_refused_only_without_roomcalls_render_stream_disk_preflightdirectly. Making_disk_preflightpassmaterialize_bytes=0(stream_setup.py:732) passes all 23. The reusable state is also only tried with 1 byte free.
Also not exercised through the setup: the issue's "a cache file that disappears after validation does not fail the pre-flight". Your helper-level test deletes the file and your other test patches the helper to return None; nothing deletes the file between inspect_shard_cache and the sizing. These tests close all three; with them applied I get 23 -> 30 passing, and each of the two surviving mutations, and dropping the OSError handler, now fails its own test. Add to TestAReusableCacheIsReportedAtItsRealSize (after test_a_deleted_layer_file_is_not_measured):
def test_a_cache_file_lost_after_the_check_does_not_fail_the_preflight(
self, tmp_path, monkeypatch
):
"""The file goes between `inspect_shard_cache` and the sizing; the estimate stays."""
seen = _spy_preflight(monkeypatch)
captured, _said, _drive = _drive_setup(tmp_path, monkeypatch, striped=True)
estimate = seen[-1]["shard_bytes"] # the first run writes the cache: a plain estimate
assert _cache_files_bytes(tmp_path) != estimate # else the last assert proves nothing
primary = _primary_folder(tmp_path)
victim = layer_shard.layer_paths(primary, captured["index"])[1] # on the second drive
assert os.path.exists(victim) and os.path.dirname(victim) != primary
real = layer_shard.inspect_shard_cache
lost = []
def inspect_then_lose_a_file(*args, **kwargs):
result = real(*args, **kwargs)
if not lost and result[0] is not None:
os.remove(victim)
lost.append(victim)
return result
monkeypatch.setattr(layer_shard, "inspect_shard_cache", inspect_then_lose_a_file)
_drive_setup(tmp_path, monkeypatch, striped=True) # must not raise
assert lost == [victim]
assert seen[-1]["shard_write_bytes"] == 0
assert seen[-1]["shard_bytes"] == estimate
def test_an_untied_striped_cache_counts_its_vocabulary_shards(self, tmp_path):
"""The tied fixture above has no `large_*` file; an untied checkpoint writes two."""
from tests.test_disk_tier_stripe_runtime import _tiny_llama
weights = _tiny_llama(tmp_path, tie=False)
stripe = tmp_path / "stripe"
stripe.mkdir()
cache = str(tmp_path / "cache" / "tiny")
index = layer_shard.shard_checkpoint(
weights, cache, dtype="float32", arch="llama", stripe_roots=(str(stripe),)
)
assert index.large_keys and index.layer_roots == (0, 1, 0)
folders = layer_shard.stripe_dirs(cache, index.stripe_roots)
files = [
os.path.join(folder, name)
for folder in folders
for name in os.listdir(folder)
if name != layer_shard.STRIPE_MARKER_NAME
]
assert any(os.path.basename(path).startswith("large_") for path in files)
assert layer_shard.shard_cache_disk_bytes(cache, index) == sum(
os.path.getsize(path) for path in files
)In TestRefusalIsUnchanged, parametrize the reusable test and add the setup-level materialization test:
@pytest.mark.parametrize("free", [1, 10_000_000_000])
def test_a_reusable_cache_needs_no_space(self, tmp_path, monkeypatch, free):
_drive_setup(tmp_path, monkeypatch, striped=False)
self._free(monkeypatch, free)
_captured, said, _drive = _drive_setup(tmp_path, monkeypatch, striped=False)
assert "(reusable)" in said
@pytest.mark.parametrize("copy_bytes, refused", [(2_000_000_000, True), (100_000_000, False)])
@pytest.mark.parametrize("cache_exists", [False, True])
def test_a_materialization_through_the_setup_is_refused_only_without_room(
self, tmp_path, monkeypatch, copy_bytes, refused, cache_exists
):
"""The copy is charged through the setup's own call, with or without a cache on disk."""
import dataclasses
import soup_cli.utils.spectrum_scan as spectrum_scan
if cache_exists:
_drive_setup(tmp_path, monkeypatch, striped=False)
def needs_a_copy(model, *, before_materialize=None):
plan = spectrum_scan.plan_model_weights(model)
plan = dataclasses.replace(
plan, materialize_bytes=copy_bytes, materialized_copy_bytes=copy_bytes
)
before_materialize(plan)
return plan.weights_dir
monkeypatch.setattr(spectrum_scan, "resolve_model_weights", needs_a_copy)
self._free(monkeypatch, 1_000_000_000) # the tiny cache fits; the copy decides
if refused:
with pytest.raises(ValueError, match="materialized weight copy"):
_drive_setup(tmp_path, monkeypatch, striped=False)
else:
_drive_setup(tmp_path, monkeypatch, striped=False)Two smaller things:
- The fragment should be
changelog.d/0.75.1/1709.fixed.md(named after the pull request), ending(#1703 by @drakeo338 in #1709).It currently ends(#1703)with no credit. docs/performance-and-quantization.md:803describes this pre-flight ("the per-layer shard cache"). One sentence there would do: a cache Soup will reuse is reported at the size of its files, and one it has to write is reported at the estimate.
Ran on 14eaa91f: tests/test_disk_tier_stripe_*.py (all nine) plus test_issue374_stream_disk_preflight.py and the changelog-fragment test: 193 passed, 6 skipped; ruff check clean on the three Python files. The 51 test files that name layer_shard or stream_setup, plus the fragment test, on your head merged with current main (a517bd189, a clean merge): 1710 passed, 95 skipped. Once those are in, I have nothing else open on this one.
…nd a lost cache file (MakazhanAlpamys#1703) Review of MakazhanAlpamys#1709 found two mutations that survived: dropping the large_* shards from `shard_cache_disk_bytes` and passing `materialize_bytes=0` from the setup both left every test green. The new tests fail on each of them. - an untied striped cache is counted with its two large_* vocabulary shards - a materialization copy is charged through `_setup_streaming_transformers`, with and without a cache on disk, and a reusable cache is tried with and without room - a cache file removed between `inspect_shard_cache` and the sizing keeps the estimate and does not fail the pre-flight The changelog fragment is renamed to the pull request number and credited, and the performance guide says what the pre-flight reports for a reusable cache.
|
Pushed d84bcdd: Added the three requested tests (vocabulary shards of an untied striped cache, materialization charged through the setup, cache file lost after the check), parametrized the reusable-cache test, renamed the changelog fragment to |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you: I reviewed the delta from 14eaa91 to d84bcdd and everything I asked for is in.
- The three tests (lost cache file after the check, untied striped cache with
large_*shards, materialization charged through the setup, plus the 1 B / 10 GB parametrization) match what I proposed and assert the specific numbers and thematerialized weight copyrefusal. - I mutated the fix four ways on d84bcdd: dropping the
large_keysline fails the untied-cache test,materialize_bytes=0in the pre-flight fails two materialization cases, droppingOSErrorfails the lost-file test, and removingshard_estimate = cached_bytesfails the two real-size tests. tests/test_disk_tier_stripe_setup.pyplustests/test_issue487_changelog_fragments.py: 51 passed;ruff checkis clean on the three changed Python files.- The fragment is
changelog.d/0.75.1/1709.fixed.md(the newest released heading in CHANGELOG.md is 0.75.1) and ends(#1703 by @drakeo338 in #1709); the assembler accepts it. The docs sentence is in the right place.
I have nothing else open on this one.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Approved on head d84bcdd, the commit the delta review covered. See the COMMENT review above for the checks that were run.
|
Thank you, @drakeo338. Heads-up: main has moved and this PR no longer merges cleanly. A merge of Please rebase on current main and re-run |
|
Merged current main in (687f864) because I can't force-push. New head: 75da8dc. With #1726, a reusable cache has |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Delta review of 75da8dc against the head we reviewed before (d84bcdd). The source hunks are the same apart from the resolved area in _disk_preflight: shard_estimate = cached_bytes still runs before estimate_stripe_write_shares, write_total stays 0 for a reusable cache, so every share is 0 and max(shard_estimate, sum(shares)) equals the measured bytes, counted once.
A file lost after the check makes shard_cache_disk_bytes return None, so the estimate is kept and the write share stays 0 ("reusable"), which is what main does today.
approximate and cached_bytes is None is correct: the qwen4_exp fallback says approximate even for a total of 0, and a measured size is exact. test_a_measured_cache_is_not_called_approximate fails without the mask.
Tests: 42 in test_disk_tier_stripe_setup.py, none dropped or weakened (only the spy moved into _drive_setup's fourth return value). Reverting the sizing line fails 3 of them, dropping the OSError handler fails 2.
Merging into current main (5d7f2f7) is clean, and the merged tree passes the same files plus the sharder tests (94 passed). The fragment changelog.d/0.75.1/1709.fixed.md (0.75.1 is the only directory on main) and the docs line are in place.
Nit, not blocking: the docs line says a cache Soup writes "is reported at the estimate"; since #1726 it is max(estimate, bytes derived from the source headers).
CI on this head was still running when I looked (one check in progress).
75da8dc to
a3ab7f2
Compare
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Approved on head a3ab7f2, the rebased head; the earlier approval was on d84bcdd. I reviewed 75da8dc and re-checked this push: it only moves the branch onto main 837794f, and the net patch against main is identical (same patch-id). Re-run on that code: tests/test_disk_tier_stripe_setup.py 42 passed; with the other stripe files, the pre-flight test and the fragment test 253 passed, 6 skipped, and 265 passed, 6 skipped on the head merged with main 837794f. ruff is clean on the three changed Python files.
Ten mutations of the change (the sizing line, the large_keys line, the OSError handler, the approximate mask, materialize_bytes=0, the layer files, the extras shard, index.json, a whole-folder sum, a write planned for a reusable cache) each fail at least one test; none survives. The fragment is changelog.d/0.75.1/1709.fixed.md with the credit.
Nit, not blocking: the docs line says a cache Soup writes "is reported at the estimate"; with two or more drives it is max(estimate, bytes derived from the source headers) since #1726.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Delta check of a3ab7f2 (rebased on main 837794f) against the previously reviewed head 75da8dc: stream_setup.py, the test file, the docs line and the changelog fragment are byte-identical, and the layer_shard.py hunk is identical apart from a line-number shift caused by main.
On a3ab7f2, ruff is clean on the three changed .py files, and test_disk_tier_stripe_setup.py + test_issue374_stream_disk_preflight.py + test_issue487_changelog_fragments.py give 72 passed.
Replacing shard_estimate = cached_bytes with pass makes 3 tests in TestAReusableCacheIsReportedAtItsRealSize fail, so the guard is mutation-checked.
The required test (ubuntu-latest, 3.12) check was still running on this head when checked.
|
Docs nit fixed in 1c4fa0b: a cache Soup writes is now described as the estimate, or with two or more drives the larger of the estimate and the header-derived bytes. Docs only. |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Approved on head 1c4fa0b. The only change since the head I approved (a3ab7f2) is one docs commit, docs/performance-and-quantization.md +2/-1: a cache Soup has to write is reported at the estimate, or with two or more drives at the larger of the estimate and the bytes derived from the source headers. That matches how the pre-flight behaves since #1726 and closes the non-blocking nit from the review. No code or test file changed, so the earlier test and mutation results stand.
|
Heads-up for the merge, @drakeo338: the Release v0.76.0 commit landed on main, so new changelog fragments now go under |
…the disk pre-flight (MakazhanAlpamys#1703) The disk pre-flight charged a reusable layer-shard cache at the estimated shard size. It now measures the files the cache check requires (index, extras shard, large shards, layer files) and counts that size once, falling back to the estimate when a file cannot be read. A measured cache is no longer labelled "approximately", even when the stripe-share estimate for the model is approximate. Tests pin the vocabulary shards, the materialization charge, a lost cache file and the label.
1c4fa0b to
17984b7
Compare
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Approved on head 17984b7. I reviewed the earlier heads of this PR in full; the last one I approved was 1c4fa0b.
This push only rebases onto main (cfc27ed) and moves the fragment to changelog.d/0.76.0/1709.fixed.md: the net patch against main is identical to the approved head (same patch-id, ignoring the changelog and contributors files), the fragment text is unchanged, the fragment test passes (21 passed) and the trial merge with current main is clean.
|
Merged as |
Fixes #1703.
What does this PR do?
When the layer-shard cache was reusable, the disk pre-flight still printed the estimate for writing a fresh cache as the "Layer-shard cache" size, so it and "Projected total on disk" described a cache that was not there.
shard_cache_disk_bytesnow adds up the cache's own files (index, extras shard, large shards, each layer file across stripe folders), and the pre-flight reports that. Stray files in the folder are not counted, and an unmeasurable file keeps the estimate.Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passespytest tests/ -vpassesREADME.mdand the matching page underdocs/) if neededNot ticked: the full suite was not run. On head a3ab7f2,
tests/test_disk_tier_stripe_setup.pygives 42 passed andruff check src/ scripts/ tests/is clean; the 62 related test files fail only the same 109 tests that fail on main (missing optional deps). No docs describe this output.