Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49524

Improve K8s support

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • Build, Kubernetes
    • None

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-54016 Improve K8s support in Spark 4.1.0

        • Major - Major loss of function.
        • Resolved

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        relates to

        Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-45869 Revisit and Improve Spark Standalone Cluster

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        New Feature - A new feature of the product, which has yet to be developed. SPARK-45923 Spark Kubernetes Operator

        • Major - Major loss of function.
        • Closed
        1.
        		 Drop K8s v1.25 and lower version support Sub-task Resolved Dongjoon Hyun
        2.
        		 Drop K8s v1.26 Support Sub-task Resolved Dongjoon Hyun
        3.
        		 Drop K8s v1.27 Support Sub-task Resolved Dongjoon Hyun
        4.
        		 Drop K8s v1.28 Support Sub-task Resolved Dongjoon Hyun
        5.
        		 Drop K8s v1.29 Support Sub-task Resolved Dongjoon Hyun
        6.
        		 Update K8s IT CI to use K8s 1.32 Sub-task Resolved Dongjoon Hyun
        7.
        		 Update K8s IT CI to use K8s 1.33 Sub-task Resolved Dongjoon Hyun
        8.
        		 Add a symbolic link file `spark-examples.jar` in K8s Docker images Sub-task Resolved Dongjoon Hyun
        9.
        		 Support K8s volume `mount.subPathExpr` and `hostPath` volume `type` Sub-task Resolved Enrico Minack
        10.
        		 Support user-defined annotations for OnDemand PVCs Sub-task Resolved Prathit Malik
        11.
        		 Support AWS_ENDPOINT_URL env variable Sub-task Resolved Dongjoon Hyun
        12.
        		 Support APP_ID and EXECUTOR_ID placeholder in labels Sub-task Resolved Xi Chen
        13.
        		 Add `SPARK_EXECUTOR_ATTRIBUTE_(APP|EXECUTOR)_ID` if `CUSTOM_EXECUTOR_LOG_URL` is defined Sub-task Resolved Dongjoon Hyun
        14.
        		 Enable `spark.ui.prometheus.enabled` by default Sub-task Resolved Dongjoon Hyun
        15.
        		 Enable `spark.kubernetes.executor.checkAllContainers` by default Sub-task Resolved Dongjoon Hyun
        16.
        		 Skip deleting pod from k8s if the pod does not exists Sub-task Resolved leesf
        17.
        		 Use JDK for Spark 3.5+ Docker image Sub-task Resolved Dongjoon Hyun
        18.
        		 Use Java 21 instead of 21-jre in K8s Dockerfile Sub-task Resolved Dongjoon Hyun
        19.
        		 Publish Java 21 Docker image for preview1 Sub-task Resolved Dongjoon Hyun
        20.
        		 Make `spark-docker` repository's Dockerfiles up-to-date Sub-task Resolved Dongjoon Hyun
        21.
        		 Add `OpenContainers` Annotations in docker images Sub-task Resolved Dongjoon Hyun
        22.
        		 Fix to use the exit code of executor container always Sub-task Resolved Oleksiy Dyagilev
        23.
        		 Fix `RELEASE` file to have the correct information in Docker images Sub-task Resolved Dongjoon Hyun
        24.
        		 Fix `spark.ui.custom.executor.log.url` docs by adding K8s Sub-task Resolved Dongjoon Hyun
        25.
        		 Fix `ENV` key value format in K8s Dockerfiles Sub-task Resolved Dongjoon Hyun
        26.
        		 Fix `docker-image-tool.sh` to be up-to-date Sub-task Resolved Dongjoon Hyun
        27.
        		 Fix `spark.kubernetes.allocation.batch.delay` to prevent small values less than 100 Sub-task Resolved Dongjoon Hyun
        28.
        		 Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles Sub-task Resolved Dongjoon Hyun
        29.
        		 Remove `connector/docker` in favor of `Apache Spark Operator` Sub-task Resolved Dongjoon Hyun
        30.
        		 Upgrade `kubernetes-client` to 6.8.1 Sub-task Resolved Dongjoon Hyun
        31.
        		 Upgrade kubernetes-client to 6.9.0 for K8s 1.28 Sub-task Resolved Dongjoon Hyun
        32.
        		 Upgrade `kubernetes-client` to 6.9.1 Sub-task Resolved Dongjoon Hyun
        33.
        		 Upgrade kubernetes-client to 6.10.0 for K8s v1.29.0 Sub-task Resolved Bjørn Jørgensen
        34.
        		 Upgrade kubernetes-client to 6.11.0 Sub-task Resolved Bjørn Jørgensen
        35.
        		 Upgrade `kubernetes-client` to 6.12.0 Sub-task Resolved Dongjoon Hyun
        36.
        		 Upgrade kubernetes-client to 6.12.1 Sub-task Resolved Bjørn Jørgensen
        37.
        		 Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0 Sub-task Resolved Bjørn Jørgensen
        38.
        		 Upgrade `kubernetes-client` to 6.13.1 Sub-task Resolved Pan Bingkun
        39.
        		 Upgrade `kubernetes-client` to 6.13.2 Sub-task Resolved Pan Bingkun
        40.
        		 Upgrade `kubernetes-client` to 6.13.3 Sub-task Resolved Dongjoon Hyun
        41.
        		 Update `kubernetes-client` to 6.13.4 Sub-task Resolved Bjørn Jørgensen
        42.
        		 Migrate kubernetes-client from `6.x` to `7.x` Sub-task Resolved Pan Bingkun
        43.
        		 Update `YuniKorn` docs with v1.4 Sub-task Resolved Dongjoon Hyun
        44.
        		 Update `YuniKorn` docs with v1.5.1 Sub-task Resolved Dongjoon Hyun
        45.
        		 Update `YuniKorn` docs with v1.5 Sub-task Resolved Dongjoon Hyun
        46.
        		 Update `YuniKorn` docs with v1.5.2 Sub-task Resolved Dongjoon Hyun
        47.
        		 Update `YuniKorn` docs with 1.6.0 Sub-task Resolved Dongjoon Hyun
        48.
        		 Update `YuniKorn` docs with `1.6.1` Sub-task Resolved Dongjoon Hyun
        49.
        		 Update `YuniKorn` docs with `1.6.2` Sub-task Resolved Dongjoon Hyun
        50.
        		 Update `YuniKorn` docs with `1.6.3` Sub-task Resolved Dongjoon Hyun
        51.
        		 Upgrade `Volcano` to 1.8.0 Sub-task Resolved Dongjoon Hyun
        52.
        		 Upgrade `Volcano` to 1.8.1 Sub-task Resolved Dongjoon Hyun
        53.
        		 Upgrade `Volcano` to 1.8.2 Sub-task Resolved Dongjoon Hyun
        54.
        		 Upgrade `Volcano` to 1.9.0 Sub-task Resolved Dongjoon Hyun
        55.
        		 Upgrade Volcano to 1.10.0 Sub-task Resolved Pan Bingkun
        56.
        		 Upgrade Volcano to 1.11.0 Sub-task Resolved Dongjoon Hyun
        57.
        		 `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5` Sub-task Resolved Dongjoon Hyun
        58.
        		 Use `built-in` storage classes in PVTestsSuite Sub-task Resolved Dongjoon Hyun
        59.
        		 Create and use a K8s test tag for `PersistentVolume` Sub-task Resolved Dongjoon Hyun
        60.
        		 Use the latest minikube in K8s IT Sub-task Resolved Dongjoon Hyun
        61.
        		 Add `VolumeSuite` to K8s IT Sub-task Resolved Dongjoon Hyun
        62.
        		 Update docs to mention delegation token support on K8s mode Sub-task Resolved Cheng Pan
        63.
        		 Update `publish-java17.yaml` and `publish-java21.yaml` to use `preview2` by default Sub-task Resolved Dongjoon Hyun
        64.
        		 Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug Sub-task Resolved Dongjoon Hyun
        65.
        		 Fix `getReusablePVCs` to use `podCreationTimeout` instead of `podAllocationDelay` Sub-task Resolved Dongjoon Hyun
        66.
        		 Make ExecutorPodsAllocatorSuite independent from default allocation batch size Sub-task Resolved Dongjoon Hyun
        67.
        		 Remove `clock` parameter from `DriverServiceFeatureStep` Sub-task Resolved Dongjoon Hyun
        68.
        		 Detect and fail on invalid volume sizes (< 1KiB) in K8s Sub-task Resolved Dongjoon Hyun
        69.
        		 Add `spark.kubernetes.legacy.useReadWriteOnceAccessMode` for old K8s clusters Sub-task Resolved Dongjoon Hyun
        70.
        		 Improve SBT K8s IT to skip R image build step if not needed Sub-task Resolved Dongjoon Hyun
        71.
        		 Fix MountVolumesFeatureStep to use ReadWriteOncePod instead of ReadWriteOnce Sub-task Resolved Dongjoon Hyun
        72.
        		 Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped Sub-task Resolved Dongjoon Hyun
        73.
        		 Document shuffle data recovery from the remounted K8s PVCs Sub-task Resolved Dongjoon Hyun
        74.
        		 Validate checksum of remounted PVC's shuffle data before recovery Sub-task Resolved Dongjoon Hyun
        75.
        		 K8s default service token file should not be materialized into token Sub-task Resolved Dongjoon Hyun
        76.
        		 Add ShuffleChecksumUtils.compareChecksums by reusing ShuffleChecksumTestHelp.compareChecksums Sub-task Resolved Dongjoon Hyun
        77.
        		 Promote SparkKubernetesClientFactory to DeveloperApi Sub-task Resolved Dongjoon Hyun
        78.
        		 VolcanoFeatureStep should not create `DefaultVolcanoClient` if not needed Sub-task Resolved Dongjoon Hyun
        79.
        		 ExecutorFailureTracker sometimes miss failed executors on k8s Sub-task Resolved Attila Zsolt Piros
        80.
        		 Simplify symbolic link creation of `spark-examples.jar` Sub-task Resolved Dongjoon Hyun
        81.
        		 Cleanup direct usage of OkHttpClient Sub-task Resolved Pan Bingkun
        82.
        		 Update `setup-minikube` to v0.0.19 Sub-task Resolved Dongjoon Hyun
        83.
        		 Increase `spark.kubernetes.allocation.batch.size` to 10 Sub-task Resolved Dongjoon Hyun
        84.
        		 Fix `spark.kubernetes.configMap.maxSize` default value in docs Sub-task Resolved Dongjoon Hyun
        85.
        		 Install `libssl-dev` in K8s Dockerfile Sub-task Resolved Dongjoon Hyun
        86.
        		 Fix `KerberosConfDriverFeatureStep` to warn in case of failures Sub-task Resolved Dongjoon Hyun

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: