Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47046

Apache Spark 4.0.0 Dependency Audit and Cleanup

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Attachments

      Issue Links

      1.
      Upgrade Derby to 10.15.1.3+ Sub-task Resolved Yang Jie Actions
      2.
      Upgrade Derby to 10.16.1.1 Sub-task Resolved Yang Jie Actions
      3.
      Add `derby-provided` profile Sub-task Resolved Dongjoon Hyun Actions
      4.
      Add `hive-jackson-provided` profile Sub-task Resolved Dongjoon Hyun Actions
      5.
      Provide `CodeHaus Jackson` dependencies via a new optional directory Sub-task Resolved Dongjoon Hyun Actions
      6.
      Remove Guava from shared classes from IsolatedClientLoader Sub-task Resolved Cheng Pan Actions
      7.
      Upgrade `Guava` dependency in `docker-integration-tests` test module Sub-task Resolved Dongjoon Hyun Actions
      8.
      Upgrade Guava used by the connect module to 33.1.0-jre Sub-task Resolved Pan Bingkun Actions
      9.
      Upgrade ZooKeeper to 3.9.1 Sub-task Resolved Bjørn Jørgensen Actions
      10.
      Upgrade `ZooKeeper` to 3.9.2 Sub-task Resolved Dongjoon Hyun Actions
      11.
      Upgrade ZooKeeper to 3.9.3 Sub-task Resolved Pan Bingkun Actions
      12.
      Upgrade dropwizard metrics 4.2.21 Sub-task Resolved Yang Jie Actions
      13.
      Upgrade dropwizard metrics 4.2.25 Sub-task Resolved Pan Bingkun Actions
      14.
      Upgrade dropwizard metrics to 4.2.26 Sub-task Resolved Wei Guo Actions
      15.
      Upgrade dropwizard metrics to 4.2.27 Sub-task Resolved Yang Jie Actions
      16.
      Remove the special Zookeeper version in the `streaming-kafka-0-10` and `sql-kafka-0-10` modules Sub-task Resolved Yang Jie Actions
      17.
      Migrating Junit4 to Junit5 Sub-task Resolved Yang Jie Actions
      18.
      Upgrade jetty to 9.4.53.v20231009 Sub-task Resolved Yang Jie Actions
      19.
      Migrate jetty 9 to jetty 10 Sub-task Resolved HiuFung Kwok Actions
      20.
      Upgrade Jetty to 11 Sub-task Resolved HiuFung Kwok Actions
      21.
      Upgrade jetty to 11.0.21 Sub-task Resolved Yang Jie Actions
      22.
      Upgrade jetty to 11.0.23 Sub-task Resolved Yang Jie Actions
      23.
      Upgrade Jetty to 11.0.24 Sub-task Resolved Dongjoon Hyun Actions
      24.
      Remove shaded `protobuf-java` 2.6.1 dependency from `kinesis-asl-assembly` Sub-task Resolved Dongjoon Hyun Actions
      25.
      Fix `kvstore` module to have explicit `commons-lang3` test dependency Sub-task Resolved Dongjoon Hyun Actions
      26.
      Use `org.seleniumhq.selenium.htmlunit3-driver` instead of `net.sourceforge.htmlunit` Sub-task Resolved Dongjoon Hyun Actions
      27.
      Upgrade `aircompressor` to 0.26 Sub-task Resolved Dongjoon Hyun Actions
      28.
      Update `aircompressor` to 0.27 Sub-task Resolved Bjørn Jørgensen Actions
      29.
      Upgrade `aircompressor` to 2.0.2 Sub-task Resolved Pan Bingkun Actions
      30.
      Ban non-shaded Hadoop dependencies Sub-task Resolved Dongjoon Hyun Actions
      31.
      Add a checkstyle rule to ban `commons-lang` in Java code Sub-task Resolved Dongjoon Hyun Actions
      32.
      Add scalastyle and checkstyle rules to ban `FileBackedOutputStream` Sub-task Resolved Dongjoon Hyun Actions
      33.
      When there are corrupted files in the local maven repo, retry to skip this cache Sub-task Resolved Pan Bingkun Actions
      34.
      Add `scalastyle` and `checkstyle` rules to ban `AtomicDoubleArray|CompoundOrdering` Sub-task Resolved Dongjoon Hyun Actions
      35.
      Upgrade several Maven plugins to the latest versions Sub-task Resolved Dongjoon Hyun Actions
      36.
      Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir Sub-task Resolved Dongjoon Hyun Actions
      37.
      Migrate mockito 4 to mockito5 Sub-task Resolved Pan Bingkun Actions
      38.
      Add `getUriBuilder` to `o.a.s.u.Utils` and use it Sub-task Resolved Dongjoon Hyun Actions
      39.
      Upgrade `mockito` to 5.10.0 Sub-task Resolved Dongjoon Hyun Actions
      40.
      Upgrade Netty to 4.1.106.Final Sub-task Resolved Pan Bingkun Actions
      41.
      Add netty-tcnative-boringssl-static dependency Sub-task Resolved Hasnain Lakhani Actions
      42.
      Upgrade netty to 4.1.107.Final and netty-tcnative to 2.0.62.Final Sub-task Resolved Dongjoon Hyun Actions
      43.
      Upgrade `netty` to 4.1.108.Final and `netty-tcnative` to 2.0.65.Final Sub-task Resolved Pan Bingkun Actions
      44.
      Upgrade netty to `4.1.109.Final` Sub-task Resolved Pan Bingkun Actions
      45.
      Upgrade Netty to 4.1.100.Final Sub-task Resolved Dongjoon Hyun Actions
      46.
      Upgrade `netty-tcnative` to 2.0.66.Final Sub-task Resolved Pan Bingkun Actions
      47.
      Upgrade netty to `4.1.114.Final` Sub-task Resolved Pan Bingkun Actions
      48.
      Upgrade `netty-tcnative` to 2.0.69.Final Sub-task Resolved Pan Bingkun Actions
      49.
      Upgrade Netty to 4.1.115.Final Sub-task Resolved Yang Jie Actions
      50.
      Upgrade Netty to 4.1.117.Final Sub-task Resolved Dongjoon Hyun Actions
      51.
      Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final Sub-task Resolved Dongjoon Hyun Actions
      52.
      Upgrade `commons-text` to `1.13.0` Sub-task Resolved Pan Bingkun Actions
      53.
      Upgrade scalatest related dependencies to the 3.2.18 series Sub-task Resolved Yang Jie Actions
      54.
      Fix `HistoryServerSuite` to get `getNumJobs` in `eventually` Sub-task Resolved Dongjoon Hyun Actions
      55.
      Set `derby.connection.requireAuthentication` to false explicitly in CLIs Sub-task Resolved Dongjoon Hyun Actions
      56.
      Upgrade `commons-compress` to 1.26.1` Sub-task Resolved Dongjoon Hyun Actions
      57.
      Upgrade `commons-compress` to 1.24.0 Sub-task Resolved Hyukjin Kwon Actions
      58.
      Upgrade `commons-compress` to 1.25.0 Sub-task Resolved Yang Jie Actions
      59.
      Upgrade `commons-compress` to 1.26.0 Sub-task Resolved Dongjoon Hyun Actions
      60.
      Upgrade `commons-compress` to 1.27.0 Sub-task Resolved Pan Bingkun Actions
      61.
      Revert S3A endpoint fixup logic of SPARK-35878 Sub-task Resolved Steve Loughran Actions
      62.
      Remove javax-servlet-api exclusion rule for SBT Sub-task Resolved HiuFung Kwok Actions
      63.
      Update `SKIP_SPARK_RELEASE_VERSIONS` in Maven CIs Sub-task Resolved Dongjoon Hyun Actions
      64.
      Fix `MavenUtilsSuite` to use `MavenUtils.resolveMavenCoordinates` properly Sub-task Resolved Dongjoon Hyun Actions
      65.
      Use `spark.jars.ivy` instead `spark.driver.extraJavaOptions` in `DepsTestsSuite` Sub-task Resolved Dongjoon Hyun Actions
      66.
      Upgrade Ivy to 2.5.2 Sub-task Resolved Dongjoon Hyun Actions
      67.
      Upgrade ivy to 2.5.3 Sub-task Resolved Yang Jie Actions
      68.
      Improve `ArtifactSuite` to use unique `MavenCoordinate`s Sub-task Resolved Dongjoon Hyun Actions
      69.
      Migrate jersey 2 to jersey 3 Sub-task Resolved Unassigned Actions
      70.
      Fix `kafka-0-10-sql` to use `ResetSystemProperties` if `KafkaTestUtils` is used Sub-task Resolved Dongjoon Hyun Actions
      71.
      Update K8s `Dockerfile` to include `hive-jackson` directory if exists Sub-task Resolved Dongjoon Hyun Actions
      72.
      Fix `make-distribution.sh` to check `jackson-core-asl-1.9.13.jar` existence first Sub-task Resolved Dongjoon Hyun Actions
      73.
      Remove redundant scope identifier for `jakarta.servlet-api` and `javax.servlet-api` Sub-task Resolved HiuFung Kwok Actions
      74.
      Remove ZOOKEEPER-1844 comment from KafkaTestUtils Sub-task Resolved Dongjoon Hyun Actions
      75.
      Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro-*` Sub-task Resolved Dongjoon Hyun Actions
      76.
      Upgrade slf4j to 2.0.12 Sub-task Resolved Yang Jie Actions
      77.
      Upgrade `slf4j` to 2.0.14 Sub-task Resolved Pan Bingkun Actions
      78.
      Bump ap-loader 3.0(v8) to support for async-profiler 3.0 Sub-task Resolved Nicholas Jiang Actions
      79.
      Upgrade jetty to 11.0.20 Sub-task Resolved Yang Jie Actions
      80.
      Upgrade `grpcio*` and `grpc-java` to 1.62 Sub-task Resolved Yang Jie Actions
      81.
      Upgrade rocksdbjni to 8.8.1 Sub-task Resolved Pan Bingkun Actions
      82.
      Upgrade rocksdbjni to 8.11.3 Sub-task Resolved Yang Jie Actions
      83.
      Upgrade rocksdbjni to 8.11.4 Sub-task Resolved Neil Ramaswamy Actions
      84.
      Upgrade `rocksdbjni` to 9.2.1 Sub-task Resolved Pan Bingkun Actions
      85.
      Upgrade rocksdbjni to 9.5.2 Sub-task Resolved Pan Bingkun Actions
      86.
      Upgrade rocksdbjni to 9.7.3 Sub-task Resolved Pan Bingkun Actions
      87.
      Upgrade rocksdbjni to 9.8.4 Sub-task Resolved Pan Bingkun Actions
      88.
      Upgrade RoaringBitmap to 1.0.5 Sub-task Resolved Pan Bingkun Actions
      89.
      Upgrade `gcs-connector` to 2.2.16 Sub-task Resolved Dongjoon Hyun Actions
      90.
      Upgrade `gcs-connector` to 2.2.17 Sub-task Resolved Dongjoon Hyun Actions
      91.
      Upgrade `gcs-connector` to 2.2.18 Sub-task Resolved Dongjoon Hyun Actions
      92.
      Upgrade `gcs-connector` to 2.2.20 Sub-task Resolved Dongjoon Hyun Actions
      93.
      Upgrade `gcs-connector` to 2.2.21 and add a note for 3.0.0 Sub-task Resolved Dongjoon Hyun Actions
      94.
      Upgrade `gcs-connector` to `2.2.25` Sub-task Resolved Dongjoon Hyun Actions
      95.
      Upgrade `gcs-connector` to `2.2.26` Sub-task Resolved Dongjoon Hyun Actions
      96.
      Update code style' plugins to latest version Sub-task Resolved Pan Bingkun Actions
      97.
      Upgrade jackson to 2.16.0 Sub-task Resolved Pan Bingkun Actions
      98.
      Upgrade jackson to 2.17.0 Sub-task Resolved Pan Bingkun Actions
      99.
      Upgrade jackson to 2.17.1 Sub-task Resolved Pan Bingkun Actions
      100.
      Upgrade jackson to 2.17.2 Sub-task Resolved Wei Guo Actions
      101.
      Upgrade jackson to 2.18.0 Sub-task Resolved Pan Bingkun Actions
      102.
      Upgrade jackson to 2.18.1 Sub-task Resolved Pan Bingkun Actions
      103.
      Upgrade jackson to 2.18.2 Sub-task Resolved Pan Bingkun Actions
      104.
      Upgrade jmock-junit5 to 2.13.1 Sub-task Resolved Yang Jie Actions
      105.
      Remove `commons-logging` dependency Sub-task Resolved Dongjoon Hyun Actions
      106.
      Remove unused `commons-beanutils` dependency Sub-task Resolved Dongjoon Hyun Actions
      107.
      Upgrade `sbt-assembly` to `2.2.0` and `sbt-protoc` to `1.0.7` Sub-task Resolved Pan Bingkun Actions
      108.
      Upgrade zstd-jni to 1.5.5-11 Sub-task Resolved Dongjoon Hyun Actions
      109.
      Upgrade `zstd-jni` to 1.5.6-1 Sub-task Resolved Dongjoon Hyun Actions
      110.
      Upgrade `zstd-jni` to 1.5.6-2 Sub-task Resolved Pan Bingkun Actions
      111.
      Upgrade `zstd-jni` to 1.5.6-3 Sub-task Resolved Pan Bingkun Actions
      112.
      Upgrade `zstd-jni` to 1.5.6-5 Sub-task Resolved Pan Bingkun Actions
      113.
      Upgrade `zstd-jni` to 1.5.6-7 Sub-task Resolved Pan Bingkun Actions
      114.
      Upgrade `zstd-jni` to 1.5.6-8 Sub-task Resolved Dongjoon Hyun Actions
      115.
      Upgrade `zstd-jni` to 1.5.6-9 Sub-task Resolved Dongjoon Hyun Actions
      116.
      Exclude `org.junit.jupiter` and `org.junit.platform` from `jmock-junit5` Sub-task Resolved Yang Jie Actions
      117.
      Upgrade `tink` to 1.12.0 Sub-task Resolved Pan Bingkun Actions
      118.
      Upgrade tink to 1.13.0 Sub-task Resolved Yang Jie Actions
      119.
      Upgrade tink to 1.14.1 Sub-task Resolved Wei Guo Actions
      120.
      Upgrade tink to 1.15.0 Sub-task Resolved Pan Bingkun Actions
      121.
      Upgrade tink to 1.16.0 Sub-task Resolved Pan Bingkun Actions
      122.
      Upgrade snappy to 1.1.10.4 Sub-task Resolved Bjørn Jørgensen Actions
      123.
      Upgrade snappy to 1.1.10.5 Sub-task Resolved Pan Bingkun Actions
      124.
      Upgrade snappy to 1.1.10.6 Sub-task Resolved Pan Bingkun Actions
      125.
      Upgrade snappy to 1.1.10.7 Sub-task Resolved Yang Jie Actions
      126.
      Remove redundant json4s-jackson definition in sql/api POM Sub-task Resolved Cheng Pan Actions
      127.
      Upgrade `commons-io` to 2.14.0 Sub-task Resolved Pan Bingkun Actions
      128.
      Upgrade commons-io to 2.15.0 Sub-task Resolved Yang Jie Actions
      129.
      Upgrade commons-io to 2.15.1 Sub-task Resolved Pan Bingkun Actions
      130.
      Upgrade commons-io to 2.16.0 Sub-task Resolved Pan Bingkun Actions
      131.
      Upgrade `commons-io` to 2.16.1 Sub-task Resolved Dongjoon Hyun Actions
      132.
      Upgrade `commons-io` to 2.17.0 Sub-task Resolved Pan Bingkun Actions
      133.
      Upgrade `commons-io` to 2.18.0 Sub-task Resolved Pan Bingkun Actions
      134.
      Upgrade `commons-codec` to 1.16.1 Sub-task Resolved Pan Bingkun Actions
      135.
      Upgrade `commons-codec` to 1.17.0 Sub-task Resolved Pan Bingkun Actions
      136.
      Upgrade `commons-codec` to 1.17.1 Sub-task Resolved Pan Bingkun Actions
      137.
      Upgrade `commons-codec` to 1.17.2 Sub-task Resolved Pan Bingkun Actions
      138.
      Upgrade `commons-lang3` to 3.15.0 Sub-task Resolved Pan Bingkun Actions
      139.
      Upgrade `commons-lang3` to 3.16.0 Sub-task Resolved Pan Bingkun Actions
      140.
      Upgrade `commons-lang3` to 3.17.0 Sub-task Resolved Pan Bingkun Actions
      141.
      Upgrade commons-text to 1.12.0 Sub-task Resolved Yang Jie Actions
      142.
      Bump json4s 4.0.7 Sub-task Resolved Cheng Pan Actions
      143.
      Upgrade `postgresql` to 42.7.3 Sub-task Resolved Pan Bingkun Actions
      144.
      Upgrade `slf4j` to 2.0.13 Sub-task Resolved Dongjoon Hyun Actions
      145.
      Upgrade `guava` dependency to `33.1.0-jre` in Docker IT Sub-task Resolved Cheng Pan Actions
      146.
      Upgrade built-in Hive to 2.3.10 Sub-task Resolved Cheng Pan Actions
      147.
      Upgrade RoaringBitmap to 1.0.6 Sub-task Resolved Pan Bingkun Actions
      148.
      Upgrade ASM to 9.7 Sub-task Resolved Pan Bingkun Actions
      149.
      Upgrade ASM to 9.7.1 Sub-task Resolved Pan Bingkun Actions
      150.
      Upgrade maven plugin to latest version Sub-task Resolved Pan Bingkun Actions
      151.
      Add `commons-lang:commons-lang:2.6` back to support legacy Hive UDF jars Sub-task Resolved Dongjoon Hyun Actions
      152.
      Upgrade scalacheck to 1.18.0 Sub-task Resolved Wei Guo Actions
      153.
      Upgrade curator to 5.7.0 Sub-task Resolved Wei Guo Actions
      154.
      Upgrade `curator` to 5.7.1 Sub-task Resolved Pan Bingkun Actions
      155.
      Upgrade RoaringBitmap to 1.1.0 Sub-task Resolved Wei Guo Actions
      156.
      Upgrade `scalatest` related dependencies to the 3.2.19 series Sub-task Resolved Wei Guo Actions
      157.
      Upgrade dropwizard metrics to 4.2.28 Sub-task Resolved Pan Bingkun Actions
      158.
      Upgrade mysql-connector-j to 9.0.0 Sub-task Resolved Wei Guo Actions
      159.
      Upgrade mariadb-java-client to 3.4.0 Sub-task Resolved Unassigned Actions
      160.
      Upgrade docker-java to 3.4.0 Sub-task Resolved Wei Guo Actions
      161.
      Upgrade slf4j to 2.0.16 Sub-task Resolved Yang Jie Actions
      162.
      Upgrade `xz` to `1.10` Sub-task Resolved Dongjoon Hyun Actions
      163.
      Upgrade joda-time to 2.13.0 Sub-task Resolved Pan Bingkun Actions
      164.
      Upgrade `protobuf-java` to `3.25.3` Sub-task Resolved Pan Bingkun Actions
      165.
      Upgrade `protobuf-java` to `3.25.4` Sub-task Resolved pengqli Actions
      166.
      Upgrade `protobuf-java` to `3.25.5` Sub-task Resolved Dongjoon Hyun Actions
      167.
      Simplify dependency management in YARN module Sub-task Resolved Cheng Pan Actions
      168.
      Upgrade `sbt-assembly` to `2.3.0` Sub-task Resolved Pan Bingkun Actions
      169.
      Remove unused Hadoop dependency management Sub-task Resolved Cheng Pan Actions
      170.
      Upgrade datasketches-java to 6.1.1 Sub-task Resolved Yang Jie Actions
      171.
      Upgrade `grpcio*` to 1.67.0 and `grpc-java` to 1.67.1 Sub-task Resolved Dongjoon Hyun Actions
      172.
      Upgrade ICU4J to `76.1` Sub-task Resolved Pan Bingkun Actions
      173.
      Upgrade mima-core & sbt-mima-plugin to 1.1.4 Sub-task Resolved Pan Bingkun Actions
      174.
      Upgrade commons-compress to 1.27.1 Sub-task Resolved Pan Bingkun Actions
      175.
      Upgrade log4j2 to 2.21.0 Sub-task Resolved Yang Jie Actions
      176.
      Upgrade log4j2 to 2.22.0 Sub-task Resolved Yang Jie Actions
      177.
      Upgrade log4j2 to 2.22.1 Sub-task Resolved Yang Jie Actions
      178.
      Upgrade log4j2 to 2.23.0 Sub-task Closed Unassigned Actions
      179.
      Upgrade log4j2 to 2.24.1 Sub-task Resolved Pan Bingkun Actions
      180.
      Upgrade log4j2 to 2.24.2 Sub-task Resolved Pan Bingkun Actions
      181.
      Upgrade log4j2 to 2.24.3 Sub-task Resolved Pan Bingkun Actions
      182.
      Upgrade PostgreSQL docker image to 17.2 Sub-task Resolved Dongjoon Hyun Actions
      183.
      Upgrade checkstyle to 10.20.2 Sub-task Resolved Yang Jie Actions
      184.
      Upgrade `rexml` to 3.3.9 Sub-task Resolved Dongjoon Hyun Actions
      185.
      Upgrade `scala-parallel-collections` to 1.2.0 Sub-task Resolved Yang Jie Actions
      186.
      Upgrade dropwizard metrics to 4.2.30 Sub-task Resolved Pan Bingkun Actions
      187.
      Upgrade Guava to 33.4.0-jre Sub-task Resolved Yang Jie Actions
      188.
      Remove `hive-llap-common` compile dependency Sub-task Resolved Dongjoon Hyun Actions
      189.
      Upgrade `wildfly-openssl` to 2.2.5.Final Sub-task Resolved Dongjoon Hyun Actions
      190.
      Upgrade `snowflake-jdbc` to 3.22.0 Sub-task Resolved Dongjoon Hyun Actions
      191.
      Prevent `HiveClient.runSqlHive` invocation in non-testing environment Sub-task Resolved Dongjoon Hyun Actions
      192.
      Fix `hive-llap-common` dependency to use `hive.llap.scope` in root `pom.xml` Sub-task Resolved Dongjoon Hyun Actions
      193.
      Upgrade `bouncycastle` to 1.80 Sub-task Resolved Dongjoon Hyun Actions
      194.
      Remove unused jopt dependency Sub-task Resolved Chris Nauroth Actions
      195.
      Remove unused `javax.activation:activation` dependency Sub-task Resolved Wei Guo Actions
      196.
      Upgrade `JPMML` to 1.7.1 Sub-task Resolved Wei Guo Actions

      Activity

        This comment will be Viewable by All Users Viewable by All Users
        Cancel

        People

          dongjoon Dongjoon Hyun
          dongjoon Dongjoon Hyun
          Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Slack

              Issue deployment