[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x` #49159

panbingkun · 2024-12-12T01:52:56Z

What changes were proposed in this pull request?

The pr aims to migrate kubernetes-client from 6.x to 7.x, includes:

upgrade kubernetes-client from 6.13.2 to 7.0.1.
switch default HttpClient implementation from OkHttp to Vert.x.

Why are the changes needed?

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

panbingkun · 2024-12-12T11:51:05Z

dev/deps/spark-deps-hadoop-3-hive-2.3

 objenesis/3.3//objenesis-3.3.jar
-okhttp/3.12.12//okhttp-3.12.12.jar
-okio/1.17.6//okio-1.17.6.jar
+okhttp/3.14.2//okhttp-3.14.2.jar


After removing the mandatory specification of okio dependency, the version of esdk-obs-java-3.20.4.2 dependency was restored as follows:
https://repo1.maven.org/maven2/com/huaweicloud/esdk-obs-java/3.20.4.2/esdk-obs-java-3.20.4.2.pom

panbingkun · 2024-12-12T12:04:55Z

...s/kubernetes/core/src/main/java/org/apache/spark/deploy/k8s/SparkVertxHttpClientFactory.java

+    try {
+      System.setProperty("vertx.disableDnsResolver", "true");
+      vertx = Vertx.vertx((new VertxOptions())
+        .setUseDaemonThread(true)


Because some threads (vertx-blocked-thread-checker,
vert.x-eventloop-thread-0, vert.x-internal-blocking-0, vert.x-acceptor-thread-0) in vert.x are daemon=false, When using val pb = new ProcessBuilder().command(fullCommand: _*) in testing for submitting a Spark application, the process did not end when the application had successfully ended, causing the test to hang indefinitely.

Therefore, the following improvements were made based on VertxHttpClientFactory, setting useDaemonThread as true.

https://github.com/fabric8io/kubernetes-client/blob/main/httpclient-vertx/src/main/java/io/fabric8/kubernetes/client/vertx/VertxHttpClientFactory.java#L45-L49

Non-daemon threads block app exit is a common issue for Spark on K8s, and this does not happen on YARN, which surprises a lot of users when migrating their workloads from YARN to K8s, I think #46889 might help

panbingkun · 2024-12-14T03:44:10Z

After more than a week of investigation and research, when we upgraded Kubernetes client from 6.x to 7.0 and switched default HttpClient implementation from OkHttp to Vert.x, the issue of Kubernetes Integration test failure has been resolved.
The root cause is that some thread attributes daemon of Vert.x are set to false, so that after using ProcessBuilder to start the spark application on K8S (based on spark-submit), the main thread has already ended, but the above threads will not end, resulting in the overall process based on spark-submit not ending either.

panbingkun · 2024-12-14T04:52:30Z

...s/kubernetes/core/src/main/java/org/apache/spark/deploy/k8s/SparkVertxHttpClientFactory.java

+    this.vertx = createVertxInstance();
+  }
+
+  public static SparkVertxHttpClientFactory instance() {


why singleton?
fabric8io/kubernetes-client#6709 (comment)

panbingkun · 2024-12-14T04:58:40Z

cc @dongjoon-hyun @pan3793 @Yikun @LuciferYang

dongjoon-hyun

Please restore okio version to 1.17.6 explicitly.

It's because the downgrade is not supposed to be happen.

- okio/1.17.6//okio-1.17.6.jar
+ okio/1.17.2//okio-1.17.2.jar

panbingkun · 2024-12-16T02:46:43Z

Please restore okio version to 1.17.6 explicitly.

It's because the downgrade is not supposed to be happen.
- okio/1.17.6//okio-1.17.6.jar
+ okio/1.17.2//okio-1.17.2.jar

Okay.

Because after switch default HttpClient implementation from OkHttp to Vert.x, kubernetes-client no longer needs to rely on okio. Currently, only hadoop-cloud-storage depends on it (okio), as follows:

[INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.1:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:3.4.1:compile
[INFO] |  +- org.apache.hadoop:hadoop-aliyun:jar:3.4.1:compile
[INFO] |  |  +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile
[INFO] |  |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  |  +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile
[INFO] |  |  |  |  +- org.ini4j:ini4j:jar:0.5.4:compile
[INFO] |  |  |  |  +- io.opentracing:opentracing-api:jar:0.33.0:compile
[INFO] |  |  |  |  \- io.opentracing:opentracing-util:jar:0.33.0:compile
[INFO] |  |  |  |     \- io.opentracing:opentracing-noop:jar:0.33.0:compile
[INFO] |  |  |  +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile
[INFO] |  |  |  \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile
[INFO] |  |  \- org.codehaus.jettison:jettison:jar:1.5.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.1:compile
[INFO] |  |  \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile
[INFO] |  \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.1:compile
[INFO] |     \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile
[INFO] |        +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile
[INFO] |        +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile
[INFO] |        \- com.squareup.okio:okio:jar:1.17.6:compile

So I will migrate the dependency declaration for com.squareup.okio:okio from pom.xml to hadoop-cloud/pom.xml to reduce misunderstandings, WDYT? @dongjoon-hyun

dongjoon-hyun · 2024-12-16T02:48:56Z

FYI, okio 1.7.16 was CVE-related update at that time, @panbingkun .

[SPARK-45590][BUILD] Upgrade okio to 1.17.6 from 1.15.0 #47758

cc @roczei , @yaooqinn from #47758

dongjoon-hyun · 2024-12-16T03:19:13Z

+1 if that's the only one.

panbingkun · 2024-12-16T03:19:33Z

pom.xml

    <!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
    <leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
-    <kubernetes-client.version>6.13.4</kubernetes-client.version>
-    <okio.version>1.17.6</okio.version>


Move to hadoop-cloud/pom.xml

panbingkun · 2024-12-16T03:19:47Z

pom.xml

        <artifactId>javax.servlet-api</artifactId>
        <version>${javaxservlet.version}</version>
      </dependency>
-      <dependency>


Move to hadoop-cloud/pom.xml

panbingkun · 2024-12-16T03:26:36Z

+1 if that's the only one.

Yea, I use following command to analy it. Currently, that's the only one.

./build/mvn dependency:tree -Phadoop-3 -Phadoop-cloud -Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pkubernetes -Pvolcano -Pyarn -Pdocker-integration-tests -Pconnect -Phive -Pyarn -Pjvm-profiler > dependency/20241216.txt

pan3793 · 2024-12-20T02:20:30Z

LICENSE-binary


 Eclipse Public License (EPL) 2.0
 --------------------------------
+io.vertx:vertx-core https://github.com/eclipse-vertx/vert.x


for dual-licensed 3rd deps, we only need to pick one. for this case, it's under EPL-2.0 and AL 2.0, simply pick the AL 2.0 and no additional LICENSE copy is required.

Because these are dual-licensed 3rd deps, should we choose EPL-2.0 or AL 2.0?
Is there a clear standard for this? Is it still possible to freely express oneself?
cc @dongjoon-hyun

the downstream projects are free to pick one or all licenses they want, as an ASF project, obviously, we prefer to choose AL 2.0

the downstream projects are free to pick one or all licenses they want, as an ASF project, obviously, we prefer to choose AL 2.0

If so, +1

+1 for AL 2.0, too.

Updated, thanks all!

dongjoon-hyun · 2024-12-20T19:57:30Z

BTW, sorry for the late response, @pan3793 .

dongjoon-hyun

+1, LGTM. Thank you, @panbingkun, @LuciferYang , and @pan3793 .

Merged to master for Apache Spark 4.0.0. Happy New Year!

dongjoon-hyun · 2025-01-03T16:25:30Z

I collected this as a subtask for the umbrella JIRA, SPARK-49524 (Improve K8s support).

### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 7.0.1 in line with Apache Spark 4. - apache/spark#49159 Please note that `spark-kubernetes-operator` still uses `okhttp3` because `KubernetesMetricsInterceptor` depends on `okhttp3` `Interceptor` class. This is the main difference from Apache Spark 4 which migrated from `okhttp3` to `vertex`. ### Why are the changes needed? To bring the latest K8s client feature by matching with the latest Apache Spark 4 dependency, - https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 - https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #150 from dongjoon-hyun/SPARK-50723. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129)

…t in spark (#66) * ODP-3237: Fix compile * [SPARK-44537][BUILD] Upgrade kubernetes-client to 6.8.0 The pr aims to upgrade kubernetes-client from 6.7.2 to 6.8.0. - The newest version brings some bug fixed & improvment, eg: Fix fabric8io/kubernetes-client#5221: Empty kube config file causes NPE Fix fabric8io/kubernetes-client#5281: Ensure the KubernetesCrudDispatcher's backing map is accessed w/lock Fix fabric8io/kubernetes-client#5298: Prevent requests needing authentication from causing a 403 response Fix fabric8io/kubernetes-client#5233: Generalized SchemaSwap to allow for cycle expansion Fix fabric8io/kubernetes-client#5262: all built-in collections will omit empty in their serialized form. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/ No. Pass GA. Closes apache#42142 from panbingkun/SPARK-44537. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 6b6216c) * [SPARK-44821][BUILD][K8S] Upgrade `kubernetes-client` to 6.8.1 ### What changes were proposed in this pull request? This PR aims to upgrade kubernetes-client to 6.8.1 ### Why are the changes needed? To bring two additional bug fixes. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.8.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes apache#42505 from dongjoon-hyun/SPARK-44821. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit e25a654) * [SPARK-45465][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.0 for K8s 1.28 This PR aims to upgrade `kubernetes-client` to 6.9.0 for K8s 1.28. `kubernetes-client` 6.9.0 includes new features and bug fixes on top of K8s 1.23 model. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.0 - fabric8io/kubernetes-client#5401 No. Pass the CIs. No. Closes apache#43284 from dongjoon-hyun/SPARK-45465. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 89acebb) * [SPARK-45728][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.1 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.9.1. ### Why are the changes needed? This will bring us the following bug fixes including the CRD generator fix. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.1 - Fix fabric8io/kubernetes-client#5501: (crd-generator) Fix fallback value of Default annotation in presence of multiple accessors ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43589 from dongjoon-hyun/SPARK-45728. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 311602a) * [SPARK-46662][K8S][BUILD] Upgrade `kubernetes-client` to 6.10.0 Upgrade `kubernetes-client` from 6.9.1 to 6.10.0 [Release notes 6.10.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.10.0) [Release notes 6.9.2](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.2) [Updated okio to version 1.17.6 to avoid CVE-2023-3635](fabric8io/kubernetes-client#5587) [Upgrade Kubernetes Model to Kubernetes v1.29.0](fabric8io/kubernetes-client#5686) No. Pass GA No. Closes apache#44672 from bjornjorgensen/kubclient6.10. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dcfd37c) * [SPARK-47550][K8S][BUILD] Update `kubernetes-client` to 6.11.0 ### What changes were proposed in this pull request? Update `kubernetes-client` from 6.10.0 to 6.11.0 ### Why are the changes needed? [Release notes for 6.11.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.11.0) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45707 from bjornjorgensen/kub-client6.11.0. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 7b9b3cb) * [SPARK-47860][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.0 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.12.0. ### Why are the changes needed? To bring the latest bug fixes. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.12.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46066 from dongjoon-hyun/SPARK-47860. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit becbca6) * [SPARK-47915][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.1 ### What changes were proposed in this pull request? Upgrade `kubernetes-client` from 6.12.0 to 6.12.1 ### Why are the changes needed? [Release notes](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.12.1) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46137 from bjornjorgensen/kub-client6.12.1. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 8aa8ad6) * [SPARK-48514][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.0 ### What changes were proposed in this pull request? Upgrade kubernetes-client from 6.12.1 to 6.13.0 ### Why are the changes needed? Upgrade Fabric8 Kubernetes Model to Kubernetes v1.30.0 [Release log 6.13.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.0) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46854 from bjornjorgensen/kubclient6.13.0. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 6475ddf) * [SPARK-48801][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.1 ### What changes were proposed in this pull request? The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`. ### Why are the changes needed? - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1 - The newest version fixed some bug, eg: Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47206 from panbingkun/SPARK-48801. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 5b00786) * [SPARK-49196][BUILD] Upgrade `kubernetes-client` to 6.13.2 ### What changes were proposed in this pull request? The pr aims to upgrade `kubernetes-client` from `6.13.1` to `6.13.2`. ### Why are the changes needed? - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.2 - The newest version fixed some bug, eg: Fix fabric8io/kubernetes-client#6066: Added support for missing v1.APIVersions in KubernetesClient Fix fabric8io/kubernetes-client#6110: VolumeSource (and other file mode fields) in Octal are correctly interpreted Fix fabric8io/kubernetes-client#6137: ConfigBuilder.withAutoConfigure is not working ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47703 from panbingkun/SPARK-49196. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit d09e1c5) * [SPARK-45590][BUILD] Upgrade okio to 1.17.6 from 1.15.0 This PR aims to upgrade `okio` from 1.15.0 to 1.17.6. Okio 1.15.0 is vulnerable due to CVE-2023-3635, details: https://nvd.nist.gov/vuln/detail/CVE-2023-3635 Previous attempts to fix this security issue: Update okio to version 1.17.6 apache#5587: fabric8io/kubernetes-client#5587 Followup to Update okio to version 1.17.6 apache#5935: fabric8io/kubernetes-client#5935 Unfortunately it is still using 1.15.0: https://github.com/apache/spark/blob/v4.0.0-preview1/dev/deps/spark-deps-hadoop-3-hive-2.3#L227 https://github.com/apache/spark/blob/v3.5.2/dev/deps/spark-deps-hadoop-3-hive-2.3#L210 No. Pass the CIs. No. Closes apache#47758 from roczei/SPARK-45590. Authored-by: Gabor Roczei <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit c8cf394) * [SPARK-49255][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.3 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.13.3. ### Why are the changes needed? Unfortunately, there is an important bug fix after we upgraded to 6.13.2. This PR aims to bring the fix. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.3 - fabric8io/kubernetes-client#6249 - fabric8io/kubernetes-client@1a66228 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47782 from dongjoon-hyun/SPARK-49255. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 7a670d7) * [SPARK-49800][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.4 ### What changes were proposed in this pull request? Upgrade `kubernetes-client` from 6.13.3 to 6.13.4 ### Why are the changes needed? New version that have 5 fixes [Release log 6.13.4](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.4) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48268 from bjornjorgensen/k8sclient6.13.4. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 54e62a1) * [SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129) --------- Co-authored-by: panbingkun <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Co-authored-by: Bjørn Jørgensen <[email protected]> Co-authored-by: panbingkun <[email protected]> Co-authored-by: Gabor Roczei <[email protected]> Co-authored-by: panbingkun <[email protected]>

…` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129) # Conflicts: # hadoop-cloud/pom.xml # resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala

… from `6.x` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129) # Conflicts: # hadoop-cloud/pom.xml # resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala

… from `6.x` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129) (cherry picked from commit b58f9d9)

attilapiros · 2025-08-27T20:52:49Z

Please restore okio version to 1.17.6 explicitly.
It's because the downgrade is not supposed to be happen.
- okio/1.17.6//okio-1.17.6.jar
+ okio/1.17.2//okio-1.17.2.jar

Okay.

Because after switch default HttpClient implementation from OkHttp to Vert.x, kubernetes-client no longer needs to rely on okio. Currently, only hadoop-cloud-storage depends on it (okio), as follows:

[INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.1:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:3.4.1:compile
[INFO] |  +- org.apache.hadoop:hadoop-aliyun:jar:3.4.1:compile
[INFO] |  |  +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile
[INFO] |  |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  |  +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile
[INFO] |  |  |  |  +- org.ini4j:ini4j:jar:0.5.4:compile
[INFO] |  |  |  |  +- io.opentracing:opentracing-api:jar:0.33.0:compile
[INFO] |  |  |  |  \- io.opentracing:opentracing-util:jar:0.33.0:compile
[INFO] |  |  |  |     \- io.opentracing:opentracing-noop:jar:0.33.0:compile
[INFO] |  |  |  +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile
[INFO] |  |  |  \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile
[INFO] |  |  \- org.codehaus.jettison:jettison:jar:1.5.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.1:compile
[INFO] |  |  \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile
[INFO] |  \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.1:compile
[INFO] |     \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile
[INFO] |        +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile
[INFO] |        +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile
[INFO] |        \- com.squareup.okio:okio:jar:1.17.6:compile

So I will migrate the dependency declaration for com.squareup.okio:okio from pom.xml to hadoop-cloud/pom.xml to reduce misunderstandings, WDYT? @dongjoon-hyun

@panbingkun
Do I get this right: okio and okhttp are only needed because of huaweicloud which is not part of upstream hadoop?

attilapiros · 2025-08-27T21:36:05Z

@panbingkun I doublechecked this and now I see I was wrong: thehuaweicloud is a standard dependency of hadoop-cloud-storage

… from `6.x` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit dccb129) (cherry picked from commit b58f9d9)

### What changes were proposed in this pull request? This PR aims to add a new profile, `huaweicloud-provided`. ### Why are the changes needed? Since Apache Spark 4.0.0, Apache Spark module is moving from `OkHttp` to `Vert.x` implementation via the following. - #49159 - [SPARK-37687 Cleanup direct usage of OkHttpClient](https://issues.apache.org/jira/browse/SPARK-37687) Like Apache Hadoop community, we are moving away further from `okhttp` transitive dependencies from `hadoop-huaweicloud` dependency. - [HADOOP-18503](https://issues.apache.org/jira/browse/HADOOP-18503) Upgrade Huawei OBS client to 3.22.3.1 - [HADOOP-18890](https://issues.apache.org/jira/browse/HADOOP-18890) Remove okhttp usage This PR will allow users to exclude and add their `huaweicloud` and its transitive dependencies. Technically, the scope of following dependencies are changed to `provided`. As a result, those are removed from Spark distribution. ``` -esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar -hadoop-huaweicloud/3.4.2//hadoop-huaweicloud-3.4.2.jar -java-xmlbuilder/1.2//java-xmlbuilder-1.2.jar -okhttp/3.12.12//okhttp-3.12.12.jar -okio/1.17.6//okio-1.17.6.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is a new profile which is disabled by default. ### How was this patch tested? Manually check like the following. ``` $ mvn dependency:tree -Phadoop-cloud | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile $ mvn dependency:tree -Phadoop-cloud -Phuaweicloud-provided | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:provided ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52346 from dongjoon-hyun/SPARK-53590. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…c8.kubernetes.client.http.Interceptor` ### What changes were proposed in this pull request? Like Apache Spark main repository, this PR aims to make `Apache Spark K8s Operator` be independent from `OkHttp3` library for long-term maintainability. - apache/spark#49159 - [SPARK-50493 Migrate kubernetes-client from `6.x` to `7.x`](https://issues.apache.org/jira/browse/SPARK-50493) - [SPARK-37687 Cleanup direct usage of OkHttpClient](https://issues.apache.org/jira/browse/SPARK-37687) - apache/spark#52346 Technically, this goal is achieved by the following in this PR. - SPARK-53647 Use `io.fabric8.kubernetes.client.http.Interceptor` instead of `okhttp3.Interceptor` - SPARK-53648 Use `VertxHttpClientFactory` instead of `OkHttpClientFactory` ### Why are the changes needed? Currently, `Apache Spark K8s Operator` has a hard compilation dependency on `OkHttp3` library like the following. https://github.com/apache/spark-kubernetes-operator/blob/a04c2bb9aeee5856681f796129e2f698a38e6ac1/spark-operator/src/main/java/org/apache/spark/k8s/operator/client/KubernetesClientFactory.java#L38-L41 From `Fabric8` v7.0.0, we should avoid `OkHttp3` because `Fabric8` community moved away from it like the following. - `VertxHttpClientFactory` is the default HTTP client factory now. - fabric8io/kubernetes-client#6470 - `io.fabric8.kubernetes.client.http.Interceptor` is the `fabric8`'s interceptor layer which we should use to be independent from the underlying HTTP factories. We should depend on this instead of exposing `okhttp3.Interceptor`. ### Does this PR introduce _any_ user-facing change? Yes, but Apache Spark K8s versions are still 0.x releases. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #327 from dongjoon-hyun/TODO_METRICS. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to add a new profile, `huaweicloud-provided`. ### Why are the changes needed? Since Apache Spark 4.0.0, Apache Spark module is moving from `OkHttp` to `Vert.x` implementation via the following. - apache#49159 - [SPARK-37687 Cleanup direct usage of OkHttpClient](https://issues.apache.org/jira/browse/SPARK-37687) Like Apache Hadoop community, we are moving away further from `okhttp` transitive dependencies from `hadoop-huaweicloud` dependency. - [HADOOP-18503](https://issues.apache.org/jira/browse/HADOOP-18503) Upgrade Huawei OBS client to 3.22.3.1 - [HADOOP-18890](https://issues.apache.org/jira/browse/HADOOP-18890) Remove okhttp usage This PR will allow users to exclude and add their `huaweicloud` and its transitive dependencies. Technically, the scope of following dependencies are changed to `provided`. As a result, those are removed from Spark distribution. ``` -esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar -hadoop-huaweicloud/3.4.2//hadoop-huaweicloud-3.4.2.jar -java-xmlbuilder/1.2//java-xmlbuilder-1.2.jar -okhttp/3.12.12//okhttp-3.12.12.jar -okio/1.17.6//okio-1.17.6.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is a new profile which is disabled by default. ### How was this patch tested? Manually check like the following. ``` $ mvn dependency:tree -Phadoop-cloud | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile $ mvn dependency:tree -Phadoop-cloud -Phuaweicloud-provided | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:provided ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52346 from dongjoon-hyun/SPARK-53590. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-50493][BUILD] Migrate kubernetes-client to 7.0.0

130b299

github-actions bot added KUBERNETES BUILD labels Dec 12, 2024

panbingkun added 3 commits December 12, 2024 10:04

fix

988a1e1

fix

a7ff81c

setUseDaemonThread(true) for vert.x thread

e515bd9

panbingkun commented Dec 12, 2024

View reviewed changes

panbingkun added 5 commits December 12, 2024 20:14

Merge branch 'master' into k8s_client_7_vert_x

2ceefbf

fix

832431e

Merge branch 'master' into k8s_client_7_vert_x

81e7169

Merge branch 'master' into k8s_client_7_vert_x

fb3ae33

fix

8eb27f8

improve

d440796

panbingkun commented Dec 14, 2024

View reviewed changes

panbingkun marked this pull request as ready for review December 14, 2024 04:57

panbingkun mentioned this pull request Dec 14, 2024

[SPARK-50493][BUILD] Upgrade kubernetes-client to 7.0.0 #49066

Closed

panbingkun added 2 commits December 14, 2024 15:40

fix

484f8ee

Merge branch 'master' into k8s_client_7_vert_x

abe769c

dongjoon-hyun requested changes Dec 16, 2024

View reviewed changes

Merge branch 'master' into k8s_client_7_vert_x

edf5e43

okio 1.17.6

ae45d24

panbingkun commented Dec 16, 2024

View reviewed changes

update

6c66751

pan3793 reviewed Dec 20, 2024

View reviewed changes

panbingkun added 2 commits December 21, 2024 08:15

Merge branch 'master' into k8s_client_7_vert_x

68c5735

update LICENSE

b7d1b5c

panbingkun requested a review from dongjoon-hyun December 26, 2024 00:36

dongjoon-hyun approved these changes Jan 3, 2025

View reviewed changes

dongjoon-hyun closed this in dccb129 Jan 3, 2025

dongjoon-hyun mentioned this pull request Jan 3, 2025

[SPARK-50723] Upgrade kubernetes-client to 7.0.1 apache/spark-kubernetes-operator#150

Closed

dongjoon-hyun mentioned this pull request Jan 17, 2025

[SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client #48446

Closed

dongjoon-hyun mentioned this pull request Sep 15, 2025

[SPARK-53590][BUILD] Add huaweicloud-provided profile #52346

Closed

dongjoon-hyun mentioned this pull request Sep 19, 2025

[SPARK-53647][SPARK-53648] Use VertxHttpClientFactory and io.fabric8.kubernetes.client.http.Interceptor apache/spark-kubernetes-operator#327

Closed

[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from 6.x to 7.x #49159

[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from 6.x to 7.x #49159

Uh oh!

Conversation

panbingkun commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Dec 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Dec 14, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Dec 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Dec 20, 2024

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 3, 2025

Uh oh!

attilapiros commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

attilapiros commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x` #49159

[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x` #49159

panbingkun commented Dec 12, 2024 •

edited

Loading

panbingkun Dec 12, 2024 •

edited

Loading

pan3793 Dec 16, 2024 •

edited

Loading

panbingkun commented Dec 16, 2024 •

edited

Loading

dongjoon-hyun commented Dec 16, 2024 •

edited

Loading

panbingkun Dec 20, 2024 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

attilapiros commented Aug 27, 2025 •

edited

Loading