forked from TRI-ML/prismatic-vlms
-
Notifications
You must be signed in to change notification settings - Fork 609
Closed
Description
Hi, guys. Thanks for your great open-source work. When I use your rlds dataset built on tf.data.Dataset, I found there is memory leak when train=True. Specifically, when it's true and shuffle is called without cache, the memory gradually increases as the iteration goes.
| dataset = dataset.shuffle(shuffle_buffer_size) |
However, I don't understand why it happens. In my understanding, shuffle preloads a fixed size of data before iteration and then replace the used data with new data on the fly during iteration. So the memory should keep constant after preloading.
Could you provide some solution to the memory leak? If the leak is hard to solve, could you provide an approximate memory it needs to run the whole iteration during training?
swstbecrpgmail
Metadata
Metadata
Assignees
Labels
No labels