Skip to content

[SPARK-45179][PYTHON] Increase Numpy minimum version to 1.21 #42944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zhengruifeng wants to merge 1 commit into from
Closed

[SPARK-45179][PYTHON] Increase Numpy minimum version to 1.21 #42944

zhengruifeng wants to merge 1 commit into from

Conversation

@zhengruifeng
Copy link
Contributor

@zhengruifeng zhengruifeng commented Sep 15, 2023 •
edited
Loading

What changes were proposed in this pull request?

Increase Numpy minimum version to 1.21

Why are the changes needed?

  • according to the release history, Numpy 1.15 was released about 5 years ago, while the last maintenance release in 1.21 was released 1 year ago;
  • with 1.21 as the minimum version, we can discard all version checking in PySpark;
  • pandas==1.4.4 just depends on numpy>=1.21.0;

Does this PR introduce any user-facing change?

No

How was this patch tested?

search with ag

(spark_dev_310) ➜  spark git:(master) ag --py 'numpy\.__version' python
(spark_dev_310) ➜  spark git:(master)
(spark_dev_310) ➜  spark git:(master) ag --py 'np\.__version' python
python/pyspark/ml/image.py
231:        if LooseVersion(np.__version__) >= LooseVersion("1.9"):

python/pyspark/pandas/typedef/typehints.py
152:    if sys.version_info >= (3, 8) and LooseVersion(np.__version__) >= LooseVersion("1.21"):

python/pyspark/pandas/tests/test_typedef.py
365:            if sys.version_info >= (3, 8) and LooseVersion(np.__version__) >= LooseVersion("1.21"):

python/pyspark/pandas/tests/computation/test_apply_func.py
257:        if sys.version_info >= (3, 8) and LooseVersion(np.__version__) >= LooseVersion("1.21"):

Was this patch authored or co-authored using generative AI tooling?

no

@zhengruifeng
Copy link
Contributor Author

zhengruifeng commented Sep 15, 2023

thanks, merged to master

@zhengruifeng zhengruifeng deleted the bump_min_np_ver branch September 15, 2023 12:31
@itholic itholic mentioned this pull request Jun 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@yaooqinn yaooqinn yaooqinn approved these changes

Assignees

No one assigned

Labels

ML PANDAS API ON SPARK PYTHON

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhengruifeng @HyukjinKwon @yaooqinn