Repository navigation
Commit d4c47c6
committed
### What changes were proposed in this pull request?
This PR aims to support `ExecutorResizePlugin` with the `deployment` pods allocator for Apache Spark 4.4.0 in addition to the default `direct` pods allocator.
### Why are the changes needed?
[SPARK-56670](#55615) restricted `ExecutorResizePlugin` to the `direct` pods allocator because the higher-level controller's pod template is the source of truth for the other allocators with different mechanism.
However, after SPARK-59914, Spark becomes capable of support `Deployment` because we never updates its pod template, and the `ReplicaSet` controller neither reverts in-place pod resizes nor rolls out a new revision for them.
- #59175
### Does this PR introduce _any_ user-facing change?
Yes. In Apache Spark 4.2.0, `ExecutorResizePlugin` logs a warning and does not start with the `deployment` pods allocator. After this PR, it works with the `deployment` pods allocator.
### How was this patch tested?
Pass the CIs with the updated test case.
I also verified manually on K8s v1.37.0 that the executor pods of the `Deployment` are resized in place up to `spark.kubernetes.executor.resizeMaxMemory` without restarts, while the `Deployment` keeps its pod template and a single `ReplicaSet`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59180 from dongjoon-hyun/dongjoon/executor-resize-plugin-deployment-0e6321.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 913175e)
Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 7eba355 commit d4c47c6
3 files changed
Lines changed: 21 additions & 4 deletions
File tree
- docs
- resource-managers/kubernetes/core/src
- main/scala/org/apache/spark/scheduler/cluster/k8s
- test/scala/org/apache/spark/scheduler/cluster/k8s
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
579 | 579 | | |
580 | 580 | | |
581 | 581 | | |
582 | | - | |
| 582 | + | |
| 583 | + | |
583 | 584 | | |
584 | 585 | | |
585 | 586 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
60 | | - | |
| 59 | + | |
| 60 | + | |
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
381 | 381 | | |
382 | 382 | | |
383 | 383 | | |
384 | | - | |
| 384 | + | |
385 | 385 | | |
386 | 386 | | |
387 | 387 | | |
| |||
395 | 395 | | |
396 | 396 | | |
397 | 397 | | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
398 | 414 | | |
399 | 415 | | |
400 | 416 | | |
| |||
0 commit comments