[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

pan3793 · 2024-03-04T11:54:27Z

What changes were proposed in this pull request?

This PR aims to do a pre-test before the Apache Hive 2.3.10 official release, it includes

it should unblock Spark to upgrade to Guava 32+
it should unblock Spark to upgrade to Thrift 0.16+
it should allow Spark to drop Jackson 1.x while Hive UDF still works well
it should allow Spark to drop javax.servlet(HIVE-28104. (2.3) Move HTTP related methods from Utils to HttpUtils in shims hive#5114)
it should allow Spark to get rid of the low version of jodd which is affected by CVEs (HIVE-25054: (2.3) Drop vulnerable jodd-core dependency hive#5151)
it should allow Spark to drop commons-lang:2.6

Why are the changes needed?

Integration test of Hive 2.3.10 RC1 https://lists.apache.org/thread/3ton9oxlkkb1v7tbqqw1rhjpz9y8ymv5

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No

pan3793 · 2024-03-04T17:51:08Z

cc @sunchao, so far, the Hive branch-2.3 looks good on the Spark side.

sunchao · 2024-03-04T17:55:10Z

Thanks, that's good to know. I was informed that I should consider the following JIRAs too. Plan to check soon.

dongjoon-hyun · 2024-03-04T18:44:27Z

Thank you for making progress, @pan3793 and @sunchao .

LuciferYang · 2024-03-05T05:13:04Z

happy to see this hive upgrade, thanks to @pan3793 and @sunchao

LuciferYang · 2024-03-05T05:18:56Z

pom.xml

@pan3793 If we upgrade the version of Guava, and it is also packaged into the newwork-shuffle and core module through the shaded+relocation method, then we need to package com.google.guava:failureaccess at the same time.

We actually cannot avoid exposing vanilla Guava to classpath, except to Spark internal usage, 3-rd party libs still require Guava to work, e.g. Curator, Hive. Then, should we continue to do shading after upgrading?

pan3793 · 2024-03-21T16:10:01Z

@sunchao I checked the JIRA tickets you listed, all are related to LICENSE, I created those PRs except for the last one (obviously it has conflicts with previous patches), should be easy to review, please take a look when you have time.

pan3793 · 2024-04-21T09:27:05Z

@dongjoon-hyun @sunchao @LuciferYang the integration test with Hive 2.3.10 RC0 passed, do you have other concerns or extra cases that need to be verified?

pan3793 · 2024-04-21T09:28:20Z

also cc @wangyum @HyukjinKwon @yaooqinn

LuciferYang · 2024-04-22T02:41:53Z

dev/deps/spark-deps-hadoop-3-hive-2.3

Is this a cascading dependency brought by the new version of Guava?

LuciferYang · 2024-04-22T02:47:35Z

project/SparkBuild.scala

Will similar configurations still be needed after this pr?

This PR is mainly of PoC, so I just make minimal changes to verify it works. I think we can unify the Guava version in all places after upgrading Hive 2.3.10, in an independent PR.

Let me unblock this like

[SPARK-47025][BUILD][TESTS] Upgrade Guava dependency in docker-integration-tests test module #45088

sql/hive/src/test/java/org/apache/spark/sql/hive/test/Complex.java

dongjoon-hyun

Thank you for making a progress on this issue. Since this is Draft PR, I believe it's okay to use the existing JIRA ID.

BTW, as mentioned by Yang Jie. There are several irrelevant updates in this PR. I guess we can minimize them before the official Hive release.

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

dongjoon-hyun · 2024-04-22T03:58:52Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala

Ditto. Do you know why DEFAULT_ARTIFACT_REPOSITORY change isn't enough here?

I remember it did not work on my first try, let me try again

I searched DEFAULT_ARTIFACT_REPOSITORY, it looks like affects multiple places, I addressed the ADD JAR in another approach ec9848f

dongjoon-hyun · 2024-04-22T17:37:18Z

For Docker Integration test dependency, I made the following PR with @pan3793 's authorship.

[SPARK-47940][BUILD][TESTS] Upgrade guava dependency to 33.1.0-jre in Docker IT #46167

dongjoon-hyun · 2024-04-22T17:39:06Z

dev/test-dependencies.sh

Just a question. Why do we need this change?

The recent Guava version has the suffix -jre or -android, the change is required to match the new version

…` in Docker IT ### What changes were proposed in this pull request? This PR aims to upgrade `guava` dependency to `33.1.0-jre` in Docker Integration tests. ### Why are the changes needed? This is a preparation of the following PR. - #45372 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46167 from dongjoon-hyun/SPARK-47940. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2024-05-06T18:26:39Z

Could you update the PR description with RC1 link, @pan3793 ?

https://lists.apache.org/thread/3ton9oxlkkb1v7tbqqw1rhjpz9y8ymv5

pan3793 · 2024-05-06T18:34:29Z

@dongjoon-hyun thanks for checking, updated

### What changes were proposed in this pull request? This PR aims to bump Spark's built-in Hive from 2.3.9 to Hive 2.3.10, with two additional changes: - due to API breaking changes of Thrift, `libthrift` is upgraded from `0.12` to `0.16`. - remove version management of `commons-lang:2.6`, it comes from Hive transitive deps, Hive 2.3.10 drops it in apache/hive#4892 This is the first part of apache#45372 ### Why are the changes needed? Bump Hive to the latest version of 2.3, prepare for upgrading Guava, and dropping vulnerable dependencies like Jackson 1.x / Jodd ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. (wait for sunchao to complete the 2.3.10 release to make jars visible on Maven Central) ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45372 Closes apache#46468 from pan3793/SPARK-47018. Lead-authored-by: Cheng Pan <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 2d609bf)

github-actions bot added SQL BUILD CORE CONNECT labels Mar 4, 2024

LuciferYang reviewed Mar 5, 2024

View reviewed changes

pan3793 force-pushed the hive-2310 branch from 8591dbd to 9880a55 Compare March 29, 2024 05:48

dongjoon-hyun marked this pull request as draft March 29, 2024 20:38

pan3793 force-pushed the hive-2310 branch from b575573 to 7194f8d Compare April 21, 2024 04:34

github-actions bot added the DSTREAM label Apr 21, 2024

pan3793 changed the title ~~[DO-NOT-MERGE] Test Hive pre-2.3.10~~ [DO-NOT-MERGE] Test Apache Hive 2.3.10 RC0 Apr 21, 2024

pan3793 force-pushed the hive-2310 branch from a4902c6 to 2f220f6 Compare April 21, 2024 06:04

LuciferYang reviewed Apr 22, 2024

View reviewed changes

sql/hive/src/test/java/org/apache/spark/sql/hive/test/Complex.java Outdated Show resolved Hide resolved

dongjoon-hyun changed the title ~~[DO-NOT-MERGE] Test Apache Hive 2.3.10 RC0~~ [SPARK-47018][BUILD] Upgrade built-in Hive to 2.3.10 Apr 22, 2024

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 22, 2024

View reviewed changes

pan3793 force-pushed the hive-2310 branch from 04e3613 to f0eb2d9 Compare April 23, 2024 01:55

pan3793 added 14 commits May 6, 2024 14:03

github-repo

d457f85

Hive 2.3.101

647b0a4

jackson

f504753

Bump Hive 2.3.102 and exclude javax.servlet

2ecdfd8

Hive 2.3.10-3

59323ce

remove jodd

c8ca897

Apache Hive 2.3.10 RC0

eb7cc46

guava

e191b99

Remove codehaus.jackson.version

05c511e

Remove commons-lang definition

d7619bb

aggdesigner-algorithm

32d481f

guava

613eb85

ADD JAR

71e8f93

Apache Hive 2.3.10 RC1

a31b060

pan3793 force-pushed the hive-2310 branch from f0eb2d9 to a31b060 Compare May 6, 2024 06:07

fix

0a14994

pan3793 mentioned this pull request May 8, 2024

[SPARK-47018][BUILD][SQL] Bump built-in Hive to 2.3.10 #46468

Closed

pan3793 changed the title ~~[SPARK-47018][BUILD] Upgrade built-in Hive to 2.3.10~~ [DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 May 10, 2024

dongjoon-hyun closed this in 2d609bf May 10, 2024

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

[DRAFT][BUILD] Test upgrading built-in Hive to 2.3.10 #45372

Uh oh!

Conversation

pan3793 commented Mar 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Mar 4, 2024

Uh oh!

sunchao commented Mar 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Mar 4, 2024

Uh oh!

LuciferYang commented Mar 5, 2024

Uh oh!

LuciferYang Mar 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Mar 21, 2024

Uh oh!

pan3793 commented Apr 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Apr 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 6, 2024

Uh oh!

pan3793 commented May 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pan3793 commented Mar 4, 2024 •

edited

Loading

sunchao commented Mar 4, 2024 •

edited

Loading

LuciferYang Mar 5, 2024 •

edited

Loading

pan3793 commented Apr 21, 2024 •

edited

Loading