Skip to content

Memory is not being cleared #1160

@simonkamronn

Description

@simonkamronn

When repeating a computation on a dask.DataFrame the memory builds up and the computation slows considerably.

for _ in range(10): %time mag_std = ddf.mag.resample('15Min').std().compute()

CPU times: user 3min 21s, sys: 10.6 s, total: 3min 32s
Wall time: 1min
CPU times: user 3min 17s, sys: 11.1 s, total: 3min 28s
Wall time: 59.9 s
CPU times: user 3min 14s, sys: 10.9 s, total: 3min 25s
Wall time: 58.8 s
CPU times: user 3min 6s, sys: 9.98 s, total: 3min 16s
Wall time: 1min 8s
CPU times: user 2min 57s, sys: 14.3 s, total: 3min 11s
Wall time: 2min 2s
CPU times: user 2min 51s, sys: 13.7 s, total: 3min 5s
Wall time: 3min 41s
CPU times: user 2min 44s, sys: 17.5 s, total: 3min 1s
Wall time: 5min 20s
CPU times: user 2min 44s, sys: 11.4 s, total: 2min 55s
Wall time: 5min 37s
CPU times: user 2min 42s, sys: 11.3 s, total: 2min 53s
Wall time: 7min 6s
CPU times: user 2min 50s, sys: 14.8 s, total: 3min 5s
Wall time: 12min 3s

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions