-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
When repeating a computation on a dask.DataFrame the memory builds up and the computation slows considerably.
for _ in range(10): %time mag_std = ddf.mag.resample('15Min').std().compute()
CPU times: user 3min 21s, sys: 10.6 s, total: 3min 32s
Wall time: 1min
CPU times: user 3min 17s, sys: 11.1 s, total: 3min 28s
Wall time: 59.9 s
CPU times: user 3min 14s, sys: 10.9 s, total: 3min 25s
Wall time: 58.8 s
CPU times: user 3min 6s, sys: 9.98 s, total: 3min 16s
Wall time: 1min 8s
CPU times: user 2min 57s, sys: 14.3 s, total: 3min 11s
Wall time: 2min 2s
CPU times: user 2min 51s, sys: 13.7 s, total: 3min 5s
Wall time: 3min 41s
CPU times: user 2min 44s, sys: 17.5 s, total: 3min 1s
Wall time: 5min 20s
CPU times: user 2min 44s, sys: 11.4 s, total: 2min 55s
Wall time: 5min 37s
CPU times: user 2min 42s, sys: 11.3 s, total: 2min 53s
Wall time: 7min 6s
CPU times: user 2min 50s, sys: 14.8 s, total: 3min 5s
Wall time: 12min 3s