Skip to content

[SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 - #41767

Closed
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-44222
Closed

[SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0#41767
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-44222

Conversation

@dongjoon-hyun

@dongjoon-hyun dongjoon-hyun commented Jun 28, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to upgrade grpc from 1.48.1 to 1.56.0 for Apache Spark 3.5.0.

Why are the changes needed?

spark/pom.xml

Line 283 in 3cd4860

<io.grpc.version>1.47.0</io.grpc.version>

spark/dev/requirements.txt

Line 53 in 3cd4860

grpcio==1.48.1

spark/python/docs/source/getting_started/install.rst

Line 162 in 3cd4860

`grpc` 1.48.1 Required for Spark Connect

Does this PR introduce any user-facing change?

This is a dependency change, but connect module itself is still experimental.

How was this patch tested?

Pass the CIs

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-44222][BUILD] Upgrade grpc to 1.56.0 [SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 Jun 28, 2023
@dongjoon-hyun

dongjoon-hyun commented Jun 28, 2023

Copy link
Copy Markdown
Member Author

Could you review this please, @grundprinzip and @HyukjinKwon ?

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 [SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 Jun 28, 2023
@dongjoon-hyun

dongjoon-hyun commented Jun 28, 2023

Copy link
Copy Markdown
Member Author

Thank you, @yaooqinn !

@dongjoon-hyun

dongjoon-hyun commented Jun 28, 2023

Copy link
Copy Markdown
Member Author

All tests passed. Merged to master for Apache Spark 3.5.0.

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-44222 branch June 28, 2023 07:02
@LuciferYang

LuciferYang commented Jun 28, 2023

Copy link
Copy Markdown
Contributor

late LGTM ~

@dongjoon-hyun

dongjoon-hyun commented Jun 28, 2023

Copy link
Copy Markdown
Member Author

Thank you, @LuciferYang !

HyukjinKwon
HyukjinKwon reviewed Jun 30, 2023
View reviewed changes

@HyukjinKwon HyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM2

@HyukjinKwon

HyukjinKwon commented Jul 14, 2023

Copy link
Copy Markdown
Member

Hmmm .. seems like this actually brings some problem to Spark Connect dev. e.g.,:

The conflict is caused by:
    The user requested protobuf==3.19.5
    grpcio-status 1.56.0 depends on protobuf>=4.21.6

But we specify protobuf:

https://github.com/apache/spark/blob/master/dev/requirements.txt#L55

that is only used for dev.

We can't easily upgrade protobuf otherwise it will change the generated codes, and we should take a closer look for compatibility at https://github.com/apache/spark/tree/master/python/pyspark/sql/connect/proto because it requires a major version bumpup.

@HyukjinKwon

HyukjinKwon commented Jul 14, 2023

Copy link
Copy Markdown
Member

Seems like:

python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.56.0' 'protobuf==3.19.5'

this combination only works with Python 3.8 ... for some reasons.

@HyukjinKwon

HyukjinKwon commented Jul 14, 2023

Copy link
Copy Markdown
Member

Let me revert this out for now (since the branchcut is very soon) and take another look for the upgrade together if you don't mind. Sorry for a bit rushing on this.

@HyukjinKwon HyukjinKwon mentioned this pull request Jul 14, 2023
Closed
@HyukjinKwon

HyukjinKwon commented Jul 14, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Actually we can work around by setting the range .. it seems ... I opened a PR to bring this back.

HyukjinKwon added a commit that referenced this pull request Jul 14, 2023
@HyukjinKwon
[SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 with lower/upperb…
de59caa 
…ound

### What changes were proposed in this pull request?

This PR revert the revert of #41767 with setting grpc lowerbounds.

### Why are the changes needed?

See #41767

### Does this PR introduce _any_ user-facing change?

See #41767

### How was this patch tested?

Manually tested with Conda environment, with `pip install -r dev/requirements.txt` in Python 3.9, Python 3.10 and Python 3.11.

Closes #41997 from HyukjinKwon/SPARK-44222.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
asl3 pushed a commit to asl3/spark that referenced this pull request Jul 17, 2023
@HyukjinKwon @asl3
[SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 with lower/upperb…
7eb3f0b 
…ound

### What changes were proposed in this pull request?

This PR revert the revert of apache#41767 with setting grpc lowerbounds.

### Why are the changes needed?

See apache#41767

### Does this PR introduce _any_ user-facing change?

See apache#41767

### How was this patch tested?

Manually tested with Conda environment, with `pip install -r dev/requirements.txt` in Python 3.9, Python 3.10 and Python 3.11.

Closes apache#41997 from HyukjinKwon/SPARK-44222.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments
@yaooqinn yaooqinn yaooqinn approved these changes

Assignees

No one assigned

Labels

BUILD CONNECT INFRA PYTHON

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dongjoon-hyun @LuciferYang @HyukjinKwon @yaooqinn