Repository navigation
[bugfix](k8s)roll back jackson version - #18046
Conversation
|
run buildall |
|
Thank you. Thanks for the test code, I'll revisit the changes here later. |
|
run buildall |
|
TeamCity pipeline, clickbench performance test result: |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
when Upgrade the version of jackson,k8s client will failed java.lang.NoClassDefFoundError: org/yaml/snakeyaml/LoaderOptions at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:191) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2] at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:509) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2] at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:413) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2] at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:386) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2] at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677) ~[jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645) ~[jackson-databind-2.14.2.jar:2.14.2] at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfigFromString(KubeConfigUtils.java:47) ~[kubernetes-client-5.12.2.jar:?] ...
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0. 1.12.0 brings the client side of REST catalog labels (`Labels`, `SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750, apache#18045, apache#18046), which Doris needs before it can expose catalog-provided labels for iceberg tables. This change only upgrades the dependency and adapts Doris to it. - Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version` 1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built against (the pom requires the two to move together). - Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4. Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so it wins Maven mediation; leaving it at 2.9.3 would package an older caffeine into the iceberg plugin than iceberg-core is built against. - Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply the Doris changes (public class/builder methods, schemasById comment, equality-field-id sets). The vendored class shadows the SDK class in the same package, and 1.12.0 changed it (an extra `specsById` constructor argument and delete/data file partition-match validation), so keeping the 1.11.0 copy would break the SDK's own callers at runtime. Since Doris passes `specsById` to the builder, the cached scan planning path now also rejects delete files whose partition spec or tuple does not match the data file, the same as upstream iceberg. - Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`. Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)` overload, which read the partition specs from the manifest file's own metadata (only possible for Avro manifests). This matches the delete manifest path next to it (`readDeleteManifest(..., table.specs())`) and `IcebergRemoveOrphanFilesAction`. - Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type` (`apache`) builds an `Apache5HttpClient` from `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK 2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3 integrity checksums that break S3-compatible services), so with the default every S3FileIO, Glue and STS client iceberg builds failed with `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every catalog flavor, now sets `http-client.type=urlconnection` unless the catalog sets it. S3FileIO keeps the setting when it is serialized, so the BE iceberg metadata scanner, which rebuilds the FileIO from it, uses the same client; `url-connection-client` is added to that scanner, which only bundled `apache-client` and `netty-nio-client`. An explicit `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of failing on the first S3 or Glue call. `SchemaAwareDataTableScan` (the other vendored class) needs no change: `DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext` are identical between 1.11.0 and 1.12.0. ### Release note Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO, Glue, STS) now use the URLConnection HTTP client by default, and the `http-client.type=apache` catalog property is no longer accepted. ### Check List (For Author) - Test: Unit Test - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection (builds a real S3 client builder with the default) and #buildBaseKeepsAnExplicitHttpClientType, and IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType; IcebergCatalogOptionsSnapshotTest snapshots now include the default. - `build.sh --fe` passes. Ran every iceberg FE unit test with run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg, be-java-extensions/iceberg-metadata-scanner): 1630 tests, 0 failures, 5 skipped. - Iceberg regression suites not run. - Behavior changed: Yes. - Iceberg's AWS clients use the URLConnection HTTP client by default, so `http-client.apache.*` properties no longer apply, and `http-client.type=apache` is rejected at CREATE/ALTER CATALOG. - Iceberg scans whose delete files have a partition spec or partition tuple that does not match the data file now fail with a ValidationException (iceberg 1.12.0 behavior) instead of applying the deletes. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0. 1.12.0 brings the client side of REST catalog labels (`Labels`, `SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750, apache#18045, apache#18046), which Doris needs before it can expose catalog-provided labels for iceberg tables. This change only upgrades the dependency and adapts Doris to it. - Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version` 1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built against (the pom requires the two to move together). - Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4. Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so it wins Maven mediation; leaving it at 2.9.3 would package an older caffeine into the iceberg plugin than iceberg-core is built against. - Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply the Doris changes (public class/builder methods, schemasById comment, equality-field-id sets). The vendored class shadows the SDK class in the same package, and 1.12.0 changed it (an extra `specsById` constructor argument and delete/data file partition-match validation), so keeping the 1.11.0 copy would break the SDK's own callers at runtime. Since Doris passes `specsById` to the builder, the cached scan planning path now also rejects delete files whose partition spec or tuple does not match the data file, the same as upstream iceberg. - Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`. Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)` overload, which read the partition specs from the manifest file's own metadata (only possible for Avro manifests). This matches the delete manifest path next to it (`readDeleteManifest(..., table.specs())`) and `IcebergRemoveOrphanFilesAction`. - Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type` (`apache`) builds an `Apache5HttpClient` from `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK 2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3 integrity checksums that break S3-compatible services), so with the default every S3FileIO, Glue and STS client iceberg builds failed with `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every catalog flavor, now sets `http-client.type=urlconnection` unless the catalog sets it. S3FileIO keeps the setting when it is serialized, so the BE iceberg metadata scanner, which rebuilds the FileIO from it, uses the same client; `url-connection-client` is added to that scanner, which only bundled `apache-client` and `netty-nio-client`. An explicit `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of failing on the first S3 or Glue call. - Report the mixed-version case clearly. FE sends iceberg system-table splits to the BE as Java-serialized iceberg objects, and iceberg 1.12.0 changed the serialized form of classes every split carries (`BaseFile`, `PartitionData`, their anonymous inner classes; 24 serializable classes changed `serialVersionUID`). Unlike 1.11.0, where only `Schema` changed and is shimmed, this cannot be bridged by remapping UIDs, so while FE and BE run different iceberg versions (a rolling upgrade) iceberg system tables cannot be read. `IcebergSerializationCompat` now turns the bare `InvalidClassException` / `ClassNotFoundException` into an error naming the cause and the BE's iceberg version, and the shim comments no longer claim a shared wire across versions. `SchemaAwareDataTableScan` (the other vendored class) needs no change: `DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext` are identical between 1.11.0 and 1.12.0. ### Release note Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO, Glue, STS) now use the URLConnection HTTP client by default, and the `http-client.type=apache` catalog property is no longer accepted. During a rolling upgrade from a release on Iceberg 1.11, Iceberg system tables (`tbl$snapshots`, `tbl$files`, ...) are unavailable until FE and BE are both upgraded. ### Check List (For Author) - Test: Unit Test - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection (builds a real S3 client builder with the default) and #buildBaseKeepsAnExplicitHttpClientType, and IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType; IcebergCatalogOptionsSnapshotTest snapshots now include the default. - Added IcebergSerializationCompatTest#reportsATaskFromAnotherIcebergVersion (a DataFile carrying Iceberg 1.11's BaseFile UID) and #reportsAClassThisIcebergDoesNotHave. - Verified the mixed-version break outside the build: a DataFile written with iceberg 1.11.0 cannot be read with 1.12.0 and vice versa, even after rewriting every changed class UID in the stream. - `build.sh --fe` passes. Ran every iceberg FE unit test with run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg, be-java-extensions/iceberg-metadata-scanner): 1650 tests, 0 failures, 5 skipped. - Iceberg regression suites not run. - Behavior changed: Yes. - Iceberg's AWS clients use the URLConnection HTTP client by default, so `http-client.apache.*` properties no longer apply, and `http-client.type=apache` is rejected at CREATE/ALTER CATALOG. - Iceberg scans whose delete files have a partition spec or partition tuple that does not match the data file now fail with a ValidationException (iceberg 1.12.0 behavior) instead of applying the deletes. - While FE and BE run different iceberg versions (rolling upgrade from iceberg 1.11), iceberg system tables fail with an explicit "different Iceberg version" error until both sides are upgraded. - Does this need documentation: No
Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
when Upgrade the version of jackson,k8s client will failed
java.lang.NoClassDefFoundError: org/yaml/snakeyaml/LoaderOptions
at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:191) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:509) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:413) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:386) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15) ~[jackson-dataformat-yaml-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677) ~[jackson-databind-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645) ~[jackson-databind-2.14.2.jar:2.14.2]
at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfigFromString(KubeConfigUtils.java:47) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.Config.loadFromKubeconfig(Config.java:589) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.Config.tryKubeConfig(Config.java:555) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.Config.autoConfigure(Config.java:277) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.Config.(Config.java:243) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.Config.(Config.java:237) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.ConfigBuilder.(ConfigBuilder.java:11) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.ConfigBuilder.(ConfigBuilder.java:8) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:40) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.BaseKubernetesClient.(BaseKubernetesClient.java:151) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:34) ~[kubernetes-client-5.12.2.jar:?]
at org.apache.doris.deploy.impl.K8sDeployManager.client(K8sDeployManager.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.deploy.impl.K8sDeployManager.statefulSet(K8sDeployManager.java:265) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.deploy.impl.K8sDeployManager.getGroupHostInfosByStatefulSet(K8sDeployManager.java:177) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.deploy.impl.K8sDeployManager.getGroupHostInfos(K8sDeployManager.java:167) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.deploy.DeployManager.getHelperNodes(DeployManager.java:294) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.getHelperNodeFromDeployManager(Env.java:1242) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.getHelperNodes(Env.java:1188) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.initialize(Env.java:820) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.PaloFe.start(PaloFe.java:146) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.PaloFe.main(PaloFe.java:73) ~[doris-fe.jar:1.2-SNAPSHOT]
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.LoaderOptions
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_342]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_342]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
... 28 more
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...