Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.1.1
-
None
Description
Currently, when number of reduces is above 2000, HighlyCompressedMapStatus is used to store size of blocks. in HighlyCompressedMapStatus, only average size is stored for non empty blocks. Which is not good for memory control when we shuffle blocks. It makes sense to store the accurate size of block when it's above threshold.