Skip to content

Nightly MacOS wheels are no longer build with OpenMP support #123225

Closed

Description

@malfet

🐛 Describe the bug

Consider:

 python -c "import torch;from pathlib import Path;print(torch.__version__, torch.backends.openmp.is_available(), '\n'.join(str(x) for x in (Path(torch.utils.cmake_prefix_path).parent.parent / 'include').glob('omp.h')))"
2.3.0 True /Users/nshulga/miniforge3/envs/py311/lib/python3.11/site-packages/torch/include/omp.h

But:

% python -c "import torch;from pathlib import Path;print(torch.__version__, torch.backends.openmp.is_available(), '\n'.join(str(x) for x in (Path(torch.utils.cmake_prefix_path).parent.parent / 'include').glob('*.h')))"
2.4.0.dev20240401 False

Versions

2.3.0, nightly

cc @seemethere @osalpekar @atalman @albanD

Activity

  1. added
    module: binariesAnything related to official binaries that we release to users
    triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
    on Apr 2, 2024
  2. desertfire commented on Apr 3, 2024

    @desertfire
    Contributor

    xref #122705

  3. malfet commented on Apr 3, 2024

    @malfet
    CollaboratorAuthor

    Regression happen between Mar 20st and Mar 21st:

    % python -c "import torch;from pathlib import Path;print(torch.__version__, torch.backends.openmp.is_available(), '\n'.join(str(x) for x in (Path(torch.utils.cmake_prefix_path).parent.parent / 'include').glob('omp.h')))"
    2.4.0.dev20240321 False 
    (py311) nshulga@nshulga-mbp ~ % pip install torch==2.4.0.dev20240320 --pre --index-url https://download.pytorch.org/whl/nightly/cpu
    Looking in indexes: https://download.pytorch.org/whl/nightly/cpu
    Collecting torch==2.4.0.dev20240320
      Using cached https://download.pytorch.org/whl/nightly/cpu/torch-2.4.0.dev20240320-cp311-none-macosx_11_0_arm64.whl (61.3 MB)
    Requirement already satisfied: filelock in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (3.9.0)
    Requirement already satisfied: typing-extensions>=4.8.0 in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (4.8.0)
    Requirement already satisfied: sympy in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (1.11.1)
    Requirement already satisfied: networkx in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (3.0rc1)
    Requirement already satisfied: jinja2 in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (3.1.2)
    Requirement already satisfied: fsspec in ./miniforge3/envs/py311/lib/python3.11/site-packages (from torch==2.4.0.dev20240320) (2023.4.0)
    Requirement already satisfied: MarkupSafe>=2.0 in ./miniforge3/envs/py311/lib/python3.11/site-packages (from jinja2->torch==2.4.0.dev20240320) (2.1.3)
    Requirement already satisfied: mpmath>=0.19 in ./miniforge3/envs/py311/lib/python3.11/site-packages (from sympy->torch==2.4.0.dev20240320) (1.2.1)
    Installing collected packages: torch
      Attempting uninstall: torch
        Found existing installation: torch 2.4.0.dev20240321
        Uninstalling torch-2.4.0.dev20240321:
          Successfully uninstalled torch-2.4.0.dev20240321
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    torchaudio 2.2.0.dev20231005 requires torch==2.2.0.dev20231005, but you have torch 2.4.0.dev20240320 which is incompatible.
    Successfully installed torch-2.4.0.dev20240320
    (py311) nshulga@nshulga-mbp ~ % python -c "import torch;from pathlib import Path;print(torch.__version__, torch.backends.openmp.is_available(), '\n'.join(str(x) for x in (Path(torch.utils.cmake_prefix_path).parent.parent / 'include').glob('omp.h')))"
    2.4.0.dev20240320 True /Users/nshulga/miniforge3/envs/py311/lib/python3.11/site-packages/torch/include/omp.h
    
  4. malfet commented on Apr 3, 2024

    @malfet
    CollaboratorAuthor

    Ok, this regression is caused by pytorch/builder#1747 which moved to newer numpy and as result no longer installs llvm-openmp

    Likely fixed by pytorch/builder@47a8e4b running the builds now in https://github.com/pytorch/pytorch/actions/runs/8519101815/job/23371854491

  5. added a commit that references this issue on Apr 3, 2024
  6. added this to the 2.3.0 milestone on Apr 4, 2024
  7. atalman commented on Apr 4, 2024

    @atalman
    Collaborator

    closing since its fixed

  8. reopened this on Apr 4, 2024
  9. malfet commented on Apr 4, 2024

    @malfet
    CollaboratorAuthor
  10. moved this from Review Required to Validation Required in Pytorch conda/test-infra refactoringon Apr 10, 2024
  11. atalman commented on Apr 11, 2024

    @atalman
    Collaborator

    Validated with 2.3

    python -c "import torch;from pathlib import Path;print(torch.__version__, torch.backends.openmp.is_available(), '\n'.join(str(x) for x in (Path(torch.utils.cmake_prefix_path).parent.parent / 'include').glob('omp.h')))"
    2.3.0 True /Users/atalman/miniconda3/envs/py311/lib/python3.11/site-packages/torch/include/omp.h
    
  12. albanD commented on Apr 11, 2024

    @albanD
    Collaborator

    Any hint on Nikita's question above on why this was missed by testing?

  13. huydhn commented on Apr 11, 2024

    @huydhn
    Contributor

    Oh, I think I figure out the reason. We don't run that test script on nightly (or release branch) at all because it has been turned off by https://github.com/pytorch/pytorch/blob/main/.github/workflows/generated-macos-arm64-binary-wheel-nightly.yml#L29. I lack the context here, maybe it was intentional and replaced by something else?

    I can see that the script that Nikita mentions are called from either:

    1. Build wheel/conda scripts on builder, but setting SKIP_ALL_TESTS makes it a no-op.
    2. or from the ancient PyTorch binary_macos_test.sh script, that is not used anywhere atm AFAICT.

    As the next step, I think we should either clean up that file because it's not used, or update workflows on PT to turn it back on again.

    cc @atalman

  14. albanD commented on Apr 15, 2024

    @albanD
    Collaborator

    From the discussion here, it feels like we really want to re-enable it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: binariesAnything related to official binaries that we release to usersmodule: macosMac OS related issuesmodule: regressionIt used to work, and now it doesn'ttriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions