Skip to content

Commit d4c47c6

Browse files
committed
[SPARK-59918][K8S] Support deployment pods allocator in ExecutorResizePlugin
### What changes were proposed in this pull request? This PR aims to support `ExecutorResizePlugin` with the `deployment` pods allocator for Apache Spark 4.4.0 in addition to the default `direct` pods allocator. ### Why are the changes needed? [SPARK-56670](#55615) restricted `ExecutorResizePlugin` to the `direct` pods allocator because the higher-level controller's pod template is the source of truth for the other allocators with different mechanism. However, after SPARK-59914, Spark becomes capable of support `Deployment` because we never updates its pod template, and the `ReplicaSet` controller neither reverts in-place pod resizes nor rolls out a new revision for them. - #59175 ### Does this PR introduce _any_ user-facing change? Yes. In Apache Spark 4.2.0, `ExecutorResizePlugin` logs a warning and does not start with the `deployment` pods allocator. After this PR, it works with the `deployment` pods allocator. ### How was this patch tested? Pass the CIs with the updated test case. I also verified manually on K8s v1.37.0 that the executor pods of the `Deployment` are resized in place up to `spark.kubernetes.executor.resizeMaxMemory` without restarts, while the `Deployment` keeps its pod template and a single `ReplicaSet`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #59180 from dongjoon-hyun/dongjoon/executor-resize-plugin-deployment-0e6321. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 913175e) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 7eba355 commit d4c47c6

3 files changed

Lines changed: 21 additions & 4 deletions

File tree

‎docs/running-on-kubernetes.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -579,7 +579,8 @@ change to the application code. They apply to executor pods of every resource pr
579579

580580
The resize plugins are registered via `spark.plugins` and run in the driver, while the recovery mode is
581581
built into the executor pod allocator and needs no plugin. All three features require the default
582-
`direct` pods allocator (`spark.kubernetes.allocation.pods.allocator`).
582+
`direct` pods allocator (`spark.kubernetes.allocation.pods.allocator`), except that the memory resize
583+
plugin also supports the `deployment` pods allocator.
583584

584585
### Executor Memory Resize
585586

‎resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorResizePlugin.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,8 @@ class ExecutorResizeDriverPlugin extends DriverPlugin with Logging {
5656

5757
override def init(sc: SparkContext, ctx: PluginContext): JMap[String, String] = {
5858
val allocator = sc.conf.get(KUBERNETES_ALLOCATION_PODS_ALLOCATOR)
59-
if (allocator != "direct") {
60-
logWarning(log"ExecutorResizePlugin requires the 'direct' pods allocator; " +
59+
if (allocator != "direct" && allocator != "deployment") {
60+
logWarning(log"ExecutorResizePlugin requires the 'direct' or 'deployment' pods allocator; " +
6161
log"${MDC(CONFIG, KUBERNETES_ALLOCATION_PODS_ALLOCATOR.key)} is " +
6262
log"${MDC(CONFIG2, allocator)}. Plugin will not start.")
6363
return Map.empty[String, String].asJava

‎resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorResizePluginSuite.scala‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -381,7 +381,7 @@ class ExecutorResizePluginSuite
381381
assert(countSkipLogs(2) === 1)
382382
}
383383

384-
Seq("statefulset", "deployment").foreach { allocator =>
384+
Seq("statefulset").foreach { allocator =>
385385
test(s"init returns early when pods allocator is '$allocator'") {
386386
val plugin = new ExecutorResizeDriverPlugin()
387387
val sparkConf = new SparkConf().set(KUBERNETES_ALLOCATION_PODS_ALLOCATOR, allocator)
@@ -395,6 +395,22 @@ class ExecutorResizePluginSuite
395395
}
396396
}
397397

398+
test("SPARK-59918: init schedules the resize task when pods allocator is 'deployment'") {
399+
val plugin = new ExecutorResizeDriverPlugin()
400+
val sparkConf = new SparkConf().set(KUBERNETES_ALLOCATION_PODS_ALLOCATOR, "deployment")
401+
val sc = mock(classOf[SparkContext])
402+
when(sc.conf).thenReturn(sparkConf)
403+
val pluginCtx = mock(classOf[PluginContext])
404+
try {
405+
assert(plugin.init(sc, pluginCtx).isEmpty)
406+
val field = plugin.getClass.getDeclaredField("periodicService")
407+
field.setAccessible(true)
408+
assert(field.get(plugin).asInstanceOf[ScheduledThreadPoolExecutor].getQueue.size === 1)
409+
} finally {
410+
plugin.shutdown()
411+
}
412+
}
413+
398414
Seq(
399415
(EXECUTOR_RESIZE_THRESHOLD, Seq("0", "1", "1.5"), "The threshold should be in (0, 1)"),
400416
(EXECUTOR_RESIZE_FACTOR, Seq("-0.1", "0", "1.5"), "The factor should be in (0, 1]")

0 commit comments

Comments
 (0)