fix: copy block_ids before storing in prefix index#391
Conversation
|
Solid fix. The mutable reference bug is real — when CoW does One minor thing: the type annotations for Test looks good too. Merging, thanks! |
|
@andafterall v0.3.0rc1 is out with your prefix index fix included. https://github.com/jundot/omlx/releases/tag/v0.3.0rc1 — if you get a chance, please give it a test and let me know if anything looks off. thanks! |
|
@jundot
Everything looks good on my side. Thanks for the merge and the quick follow-up! Really like the direction of this project — with agents taking off, optimizing prefix cache for long-context agent workloads on Apple Silicon is exactly what's needed. |
Problem
In
_update_prefix_index(), theblock_idslist is stored directly into_prefix_index.But this is the same object as
block_table.block_ids. So if this list is changed later (for example, CoW replaces a block ID), the prefix index entry also changes.This may cause wrong block hit when another request looks up the same prefix.
Fix
Use
tuple(block_ids[:i+1])to store an immutable copy.tuplecannot be changed after creation[:i+1]only keeps blocks needed for this prefix lengthTest
test_prefix_index_immutable_after_storepytest tests/test_prefix_cache.py -v— all 73 passedpytest -m "not slow"— 2971 passed, no new failures