py-modin: add new package by adamjstewart · Pull Request #33724 · spack/spack

adamjstewart · 2022-11-05T18:52:41Z

Successfully builds on macOS 12.6.1 (arm64) with Python 3.9.13 and Apple Clang 14.0.0.

spackbot-app · 2022-11-05T18:52:48Z

Hi @adamjstewart! I noticed that the following package(s) don't yet have maintainers:

py-modin

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers = ["adamjstewart"]

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame py-modin

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

adamjstewart · 2022-11-05T22:52:43Z

I was unable to get npm and a couple of other packages to install, so I can't vouch for py-modin engine=ray, but py-modin engine=dask seems to work for me.

adamjstewart · 2022-11-05T22:53:06Z

@spackbot help

spackbot-app · 2022-11-05T22:53:07Z

You can interact with me in many ways!

@spackbot hello: say hello and get a friendly response back!
@spackbot help or @spackbot commands: see this message
@spackbot run pipeline or @spackbot re-run pipeline: to request a new run of the GitLab CI pipeline
@spackbot rebuild everything: to run a pipeline rebuilding all specs from source.
@spackbot fix style if you have write and would like me to run spack style --fix for you.
@spackbot maintainers or @spackbot request review: to look for and assign reviewers for the pull request.

I'll also help to label your pull request and assign reviewers!
If you need help or see there might be an issue with me, open an issue here

adamjstewart · 2022-11-05T22:53:19Z

@spackbot maintainers

spackbot-app · 2022-11-05T22:53:28Z

Hi @adamjstewart! I noticed that the following package(s) don't yet have maintainers:

npm
py-modin

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers = ["adamjstewart"]

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame npm

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

adamjstewart · 2022-11-05T23:00:07Z

var/spack/repos/builtin/packages/npm/package.py

    version("3.10.5", sha256="ff019769e186152098841c1fa6325e5a79f7903a45f13bd0046a4dc8e63f845f")

    depends_on("node-js", type=("build", "run"))
+    depends_on("libvips")


Without this I see:

../src/common.cc:25:10: fatal error: 'vips/vips8' file not found #include <vips/vips8> ^~~~~~~~~~~~

With this, I get a bit further in the build, but it still fails:

FATAL ERROR: wasm code commit Allocation failed - process out of memory ... bash: line 1: 83387 Abort trap: 6 node ../bin/npm-cli.js run build:static echo 1>&2

Not sure what's wrong, but this version is also super old. Would love to update npm, but newer versions don't have a configure script.

adamjstewart · 2022-11-05T23:03:01Z

var/spack/repos/builtin/packages/py-dask/package.py

+    depends_on("[email protected]:2021.8.0", type=("build", "run"), when="@0.18.0: +distributed")
+    depends_on("[email protected]:2021.8.0", type=("build", "run"), when="@2.0.0: +distributed")
+    depends_on("[email protected]:2021.8.0", type=("build", "run"), when="@2020.12.0: +distributed")
+    depends_on("[email protected]", type=("build", "run"), when="@2021.6.2 +distributed")


Dask and distributed have a weird circular dependency between them. Starting with dask 2021.?.?, the devs now pin to a specific version of distributed. If you try to use newer distributed with older dask, you get an error like below:

Traceback (most recent call last): File "/Users/ajstewart/urbanair/data/purpleair/speed.py", line 11, in <module> x = pd.read_csv(fpath) File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log return obj(*args, **kwargs) File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/_compat/pandas_api/latest/io.py", line 156, in read_csv return _read(**kwargs) File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/_compat/pandas_api/common/io.py", line 35, in _read Engine.subscribe(_update_engine) File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/config/pubsub.py", line 217, in subscribe callback(cls) File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/pandas/__init__.py", line 159, in _update_engine from modin.core.execution.dask.common import initialize_dask File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/core/execution/dask/common/__init__.py", line 16, in <module> from .engine_wrapper import DaskWrapper File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-modin-0.16.2-qsxgepq2k5b5vrqnhryljjxuedz6k2dm/lib/python3.9/site-packages/modin/core/execution/dask/common/engine_wrapper.py", line 16, in <module> from distributed.client import default_client File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-distributed-2022.2.1-z7tfusun4h66nzcxybmk2jpmmn4xumt4/lib/python3.9/site-packages/distributed/__init__.py", line 2, in <module> from . import widgets # isort:skip; load distributed widgets second File "/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-distributed-2022.2.1-z7tfusun4h66nzcxybmk2jpmmn4xumt4/lib/python3.9/site-packages/distributed/widgets/__init__.py", line 3, in <module> from dask.widgets import FILTERS, TEMPLATE_PATHS ModuleNotFoundError: No module named 'dask.widgets'

This dask.widgets import was added in distributed 2021.8.1, so 2021.8.0 is prob the newest version that works for older dask. Honestly, we could think about pinning older versions of dask to specific versions of distributed since that's what newer versions of dask do.

adamjstewart · 2022-11-06T18:33:25Z

@spackbot fix style

spackbot-app · 2022-11-06T18:33:27Z

Let me see if I can fix that for you!

spackbot-app · 2022-11-06T18:35:13Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  var/spack/repos/builtin/packages/npm/package.py
  var/spack/repos/builtin/packages/py-dask/package.py
  var/spack/repos/builtin/packages/py-modin/package.py
  var/spack/repos/builtin/packages/py-ray/package.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 569 source files
  mypy checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/py-dask/package.py
All done! ✨ 🍰 ✨
1 file reformatted, 3 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

adamjstewart · 2022-11-06T19:14:17Z

@spackbot run pipeline

spackbot-app · 2022-11-06T19:14:24Z

I've started that pipeline for you!

Co-authored-by: adamjstewart <[email protected]>

py-modin: add new package

54135cf

spackbot-app bot added dependencies new-package new-version python labels Nov 5, 2022

adamjstewart marked this pull request as draft November 5, 2022 18:57

py-modin: specify engine

3d45320

spackbot-app bot added new-variant update-package labels Nov 5, 2022

adamjstewart marked this pull request as ready for review November 5, 2022 22:52

spackbot-app bot requested review from pradyunsg and skosukhin November 5, 2022 22:53

adamjstewart commented Nov 5, 2022

View reviewed changes

[@spackbot] updating style on behalf of adamjstewart

4e71877

alalazo approved these changes Nov 7, 2022

View reviewed changes

alalazo merged commit 9dcd4fa into spack:develop Nov 7, 2022

adamjstewart deleted the packages/py-modin branch November 7, 2022 14:51

dylex mentioned this pull request Nov 18, 2022

Installation issue: libvips-8.10.5 #34003

Closed

charmoniumQ pushed a commit to charmoniumQ/spack that referenced this pull request Nov 19, 2022

py-modin: add new package (spack#33724)

821d508

Co-authored-by: adamjstewart <[email protected]>

amd-toolchain-support pushed a commit to amd-toolchain-support/spack that referenced this pull request Feb 16, 2023

py-modin: add new package (spack#33724)

83ceb05

Co-authored-by: adamjstewart <[email protected]>

Conversation

adamjstewart commented Nov 5, 2022

Uh oh!

spackbot-app bot commented Nov 5, 2022

Uh oh!

adamjstewart commented Nov 5, 2022

Uh oh!

adamjstewart commented Nov 5, 2022

Uh oh!

spackbot-app bot commented Nov 5, 2022

Uh oh!

adamjstewart commented Nov 5, 2022

Uh oh!

spackbot-app bot commented Nov 5, 2022

Uh oh!

adamjstewart Nov 5, 2022

Choose a reason for hiding this comment

Uh oh!

adamjstewart Nov 5, 2022

Choose a reason for hiding this comment

Uh oh!

adamjstewart commented Nov 6, 2022

Uh oh!

spackbot-app bot commented Nov 6, 2022

Uh oh!

spackbot-app bot commented Nov 6, 2022

Uh oh!

adamjstewart commented Nov 6, 2022

Uh oh!

spackbot-app bot commented Nov 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants