Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.1.0
-
None
Description
When I set spark.shuffle.manager = sort, Executor full of log, It confuse me a lot:
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (277 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (278 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (279 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (280 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (281 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (282 spills so far)
14/11/13 17:35:18 INFO collection.ExternalSorter: Thread 111 spilling in-memory batch of 0 MB to disk (283 spills so far)