[ci] don't run sana layerwise casting tests in CI.#12551
Merged
Conversation
dg845
approved these changes
Oct 28, 2025
Collaborator
dg845
left a comment
dg845
left a comment
There was a problem hiding this comment.
Thanks, LGTM! I've observed that some other test_layerwise_casting_inference tests seem to fail in the CI as well. For example, on the CI job when PRX was merged, both Qwen Image (https://github.com/huggingface/diffusers/actions/runs/18701189489/job/53330184168):
FAILED tests/pipelines/qwenimage/test_qwenimage_edit.py::QwenImageEditPipelineFastTests::test_layerwise_casting_inference - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 14.74 GiB of which 14.03 GiB is free. Process 19357 has 724.00 MiB memory in use. Of the allocated memory 465.60 MiB is allocated by PyTorch, and 132.40 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED tests/pipelines/qwenimage/test_qwenimage_edit_plus.py::QwenImageEditPlusPipelineFastTests::test_layerwise_casting_inference - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 14.74 GiB of which 14.05 GiB is free. Process 19357 has 708.00 MiB memory in use. Of the allocated memory 484.53 MiB is allocated by PyTorch, and 97.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED tests/pipelines/sana/test_sana.py::SanaPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_controlnet.py::SanaControlNetPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_sprint.py::SanaSprintPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
FAILED tests/pipelines/sana/test_sana_sprint_img2img.py::SanaSprintImg2ImgPipelineFastTests::test_layerwise_casting_inference - RuntimeError: GET was unable to find an engine to execute this computation
test_layerwise_casting_inference tests failed.
Member
Author
|
@dg845 please check now. Regarding the Qwen failures, I am not too sure but OOMs in our CI can be also triggered by a prior test failure in the mix. I printed the So, I suspect these OOMs are caused by previous test failures. But LMK if you have an another perspective. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Tests pass locally perfectly fine on different GPUs (RT 4090, A100, H100).
Example failures: