Skip to content

fix: copy block_ids before storing in prefix index#391

Merged
jundot merged 1 commit intojundot:mainfrom
andafterall:fix/prefix-index-mutable-reference
Mar 29, 2026
Merged

fix: copy block_ids before storing in prefix index#391
jundot merged 1 commit intojundot:mainfrom
andafterall:fix/prefix-index-mutable-reference

Conversation

@andafterall
Copy link
Copy Markdown
Contributor

Problem

In _update_prefix_index(), the block_ids list is stored directly into _prefix_index.

But this is the same object as block_table.block_ids. So if this list is changed later (for example, CoW replaces a block ID), the prefix index entry also changes.

This may cause wrong block hit when another request looks up the same prefix.

Fix

Use tuple(block_ids[:i+1]) to store an immutable copy.

  • tuple cannot be changed after creation
  • [:i+1] only keeps blocks needed for this prefix length

Test

  • Added test_prefix_index_immutable_after_store
  • pytest tests/test_prefix_cache.py -v — all 73 passed
  • pytest -m "not slow" — 2971 passed, no new failures

@jundot
Copy link
Copy Markdown
Owner

jundot commented Mar 29, 2026

Solid fix. The mutable reference bug is real — when CoW does table.block_ids[i] = new_block.block_id in get_blocks_for_generation(), the stored prefix index entry gets silently corrupted. Using tuple() is the right call since these entries should never be modified after storage, and the [:i+1] slice is a nice touch that avoids storing unnecessary trailing block ids.

One minor thing: the type annotations for _prefix_index (line 101) and _find_best_prefix_match return type (line 1983) still say List[int] but now the actual stored value is a tuple. I'll fix that in a follow-up commit after merging.

Test looks good too. Merging, thanks!

@jundot jundot merged commit be9cb7b into jundot:main Mar 29, 2026
@jundot
Copy link
Copy Markdown
Owner

jundot commented Mar 29, 2026

@andafterall v0.3.0rc1 is out with your prefix index fix included. https://github.com/jundot/omlx/releases/tag/v0.3.0rc1 — if you get a chance, please give it a test and let me know if anything looks off. thanks!

@andafterall
Copy link
Copy Markdown
Contributor Author

@jundot
Tested on v0.3.0rc1:

  • pytest -m "not slow" — 3229 passed, 0 failed
  • prefix index fix is in, and I see you already updated the type annotations, nice

Everything looks good on my side. Thanks for the merge and the quick follow-up! Really like the direction of this project — with agents taking off, optimizing prefix cache for long-context agent workloads on Apple Silicon is exactly what's needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants