WIP: Store compressed data on the Worker
#3968
Conversation
jakirkham
force-pushed
the
store_compressed_data_worker
branch
from
73a736d to
f3fa583
Compare
|
Friendly nudge 😉 Would be great to get others take on this 😄 |
|
In principle the approach seems ok to me. I think that as you say there are some active questions to resolve:
|
|
Thanks Matt! 😄
With this my current thinking is we have another weight and make this configurable (not yet done here). Though this is just my naive thought. Would welcome thoughts from others here 🙂
This is a good question. It would be helpful to identify some workloads where we expect this matters. Happy to think about this some, but would also be interested in knowing if anyone else has workloads they'd like to try here (@fjetter? 😉). One point worth raising is this is the same compression step we have today. The only difference is it stays around in memory before eventually being moved to disk. So the overall workflow itself hasn't changed. We have merely split one step into two. |
jakirkham
force-pushed
the
store_compressed_data_worker
branch
3 times, most recently
from
3184c93 to
047323e
Compare
Adds a `Buffer` for transitioning in-memory data to in-memory compressed data.
jakirkham
force-pushed
the
store_compressed_data_worker
branch
from
047323e to
121f5e9
Compare
|
Have now pushed in some logic to handle configuring when compression occurs. This probably requires some more playing on realistic workloads to determine an appropriate default configuration. Though it is overridable in any event. |
3 participants
Another approach to issue ( #3656 ). Found it a bit easier to articulate this as a PR as opposed to a comment. Hope that is ok.
As
serialize_bytelistalready performs compression, this approach simply leverages that feature and stores the data in adictin memory. Nests anotherBufferinself.datafor transitioning in-memory data to in-memory compressed data and then to on disk compressed data (this last step is the same as before).If we decide we like this approach, something that I could use some advice on would be determining an appropriate transition/exposing that in a sensible way to the user. Though we need not worry about that if another approach may be more appropriate.
cc @prasunanand @TomAugspurger @martindurant @mrocklin @madsbk @quasiben