Repository navigation
Conversation
The image decode cache is process-global and nothing clears it between tests. test_mlx_embeddings_compat.py decodes a 1x1 PNG through fetch_images() and leaves it cached. ProcessMemoryEnforcer drops that cache under pressure and re-reads usage when the drop freed anything. The single-model hard-pressure test scripts usage as a fixed sequence of reads, so one leftover entry shifts it by a read: the post-unload value arrives before the eviction loop, the loop never runs, and abort_all_requests is never awaited. Run alone the test passes; after the embeddings test in the same process it fails. The enforcer is right to re-read after freeing memory. Clear the cache around every test, as conftest already does for the decode-activity registry.
Contributor
Author
|
CI failures match the current upstream
This PR introduces no additional test failures. |
# Conflicts: # tests/conftest.py
Contributor
|
Closing this PR because it has merge conflicts and no new commits for over 30 days. This isn't a judgment on the change. If you'd like to continue, rebase on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestSingleModelMemoryPressure::test_single_idle_model_unloads_at_hard_pressureintests/test_process_memory_enforcer.pypasses on its own, but fails if an image-decoding test ran earlier in the same process and none of the enforcer file's earlier tests ran in between:Running the whole enforcer file after the embeddings file passes, and so does CI's default order, so CI doesn't hit this today. It shows up when you select that class or test directly.
The image decode cache in
omlx/utils/image.pyis process-global and nothing clears it between tests. The embeddings test leaves a decoded 1x1 PNG in it. Under pressure,ProcessMemoryEnforcer._check_and_enforcedrops that cache and, if the drop freed anything, reads usage again. The test scriptsmx.get_active_memoryas a fixed three-read sequence, so the extra read eats the loop-check value and the eviction loop never runs. The enforcer is doing the right thing; the test state leaked.The fix is an autouse fixture in
tests/conftest.pythat clears the cache before and after each test, next to the existing decode-activity reset. Test-only change. With it, the command above gives 11 passed.I merged main to resolve the conflict with #3933's
_hermetic_metal_release_accountingfixture; both fixtures are kept as they are.