Skip to content

[SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions #34732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
AngersZhuuuu wants to merge 8 commits into from
Closed

[SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions #34732

AngersZhuuuu wants to merge 8 commits into from

Conversation

@AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented Nov 29, 2021

What changes were proposed in this pull request?

In this pr, when create SparkSession, we pass initialSessionOptions to SparkSession, to keep same code path with scala code.

Why are the changes needed?

Keep same code path with scala code.

Does this PR introduce any user-facing change?

No

How was this patch tested?

existed UT

@AngersZhuuuu
Copy link
Contributor Author

AngersZhuuuu commented Nov 29, 2021

ping @HyukjinKwon @dongjoon-hyun

@AngersZhuuuu AngersZhuuuu changed the title [SPARK-37291][PYSPARK] PySpark create SparkSession should pass initialSessionOptions [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions Nov 29, 2021
HyukjinKwon
HyukjinKwon reviewed Nov 29, 2021
View reviewed changes
HyukjinKwon
HyukjinKwon reviewed Nov 29, 2021
View reviewed changes
@HyukjinKwon
Copy link
Member

HyukjinKwon commented Nov 29, 2021

Thanks for the followup!

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Test build #145694 has finished for PR 34732 at commit 7be6862.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Test build #145697 has finished for PR 34732 at commit f6df6a8.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50166/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Test build #145696 has finished for PR 34732 at commit d374536.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50167/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50164/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50164/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50169/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50170/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50169/

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50170/

HyukjinKwon
HyukjinKwon reviewed Nov 29, 2021
View reviewed changes
@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Test build #145699 has finished for PR 34732 at commit b33d254.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 29, 2021

Test build #145701 has finished for PR 34732 at commit 4598e8b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50212/

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50212/

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Nov 30, 2021

Merged to master!

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Test build #145742 has finished for PR 34732 at commit 21ab18f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon mentioned this pull request Dec 15, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

CORE PYTHON SQL WEB UI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AngersZhuuuu @HyukjinKwon @SparkQA @dongjoon-hyun