Repository navigation
Commit 40beedd
committed
fix: size the shared metadata cache by the stage's running tasks
The executor's core count that reaches native code is an estimate on a
standalone executor without spark.executor.cores, so sizing the cache by
it could give a two-core executor the limit of the host's processors.
Count each task's plan in while it holds its stage's cache and out when
it drops it, and size the cache by the tasks running. A cache with no
task running shrinks to one task's limit, so a retained cache holds no
more than a task's own cache would. The handle ignores the limit a
task's CacheManager sets.1 parent d14249f commit 40beedd
5 files changed
Lines changed: 223 additions & 133 deletions
File tree
- docs/source/contributor-guide
- native/core/src
- execution
- parquet
- spark/src/main/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
511 | 511 | | |
512 | 512 | | |
513 | 513 | | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
518 | 518 | | |
519 | 519 | | |
520 | 520 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
995 | 995 | | |
996 | 996 | | |
997 | 997 | | |
998 | | - | |
999 | | - | |
1000 | | - | |
1001 | | - | |
1002 | | - | |
1003 | | - | |
1004 | | - | |
1005 | | - | |
1006 | | - | |
1007 | 998 | | |
1008 | 999 | | |
1009 | 1000 | | |
| |||
1022 | 1013 | | |
1023 | 1014 | | |
1024 | 1015 | | |
1025 | | - | |
1026 | | - | |
1027 | | - | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
1028 | 1020 | | |
1029 | 1021 | | |
1030 | 1022 | | |
| |||
4333 | 4325 | | |
4334 | 4326 | | |
4335 | 4327 | | |
4336 | | - | |
4337 | | - | |
4338 | | - | |
4339 | | - | |
4340 | | - | |
4341 | | - | |
4342 | | - | |
4343 | | - | |
4344 | | - | |
4345 | | - | |
4346 | | - | |
4347 | | - | |
4348 | | - | |
4349 | | - | |
4350 | | - | |
4351 | | - | |
4352 | | - | |
4353 | | - | |
4354 | | - | |
4355 | | - | |
4356 | | - | |
4357 | | - | |
4358 | | - | |
4359 | | - | |
4360 | | - | |
4361 | | - | |
4362 | 4328 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2166 | 2166 | | |
2167 | 2167 | | |
2168 | 2168 | | |
2169 | | - | |
2170 | | - | |
2171 | | - | |
| 2169 | + | |
2172 | 2170 | | |
2173 | 2171 | | |
2174 | 2172 | | |
| |||
2197 | 2195 | | |
2198 | 2196 | | |
2199 | 2197 | | |
2200 | | - | |
2201 | | - | |
2202 | | - | |
2203 | | - | |
2204 | | - | |
2205 | | - | |
2206 | | - | |
2207 | | - | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
2208 | 2203 | | |
2209 | 2204 | | |
2210 | 2205 | | |
| |||
0 commit comments