fread: memory not being freed #3292
Description
I have found that reading data into a data.table will cause a memory leak. This seems to happen whether I use fread or read_fst(from the fst package). I suspect that this could even be an R or OS issue, however, I have not found any cases of this on Stack Overflow...
Steps to reproduce:
- Download the sample file here:
download.file("https://mdodemodiag770.blob.core.windows.net/testcontainer/test.csv", "test.csv")
- Create an R script as the following:
tst <- data.table::fread("tst.csv");
rm(tst);
gc()
- Execute the R script and check the results on your OS. I am using Ubuntu 18.04.1. I see the following results of my R process when I run the top command (2.6 gbs in Virtual memory);
PID USER PR VIRT RES SHR S %CUP %MEM TIME + COMMAND
15827 patrick+ 20 0 2.625g 1.269g 0.013g S 0.0 1.2 0:12.93 rsession
--If you want to run in it with valgrind, you can use the following:
4. Run the following valgrind command (change the source script to the location of your R script from step 2):
R -d "valgrind --show-leak-kinds=all" -e "source('/home/patrick.howerter/valgrind_test.R')"
Here is the end of the valgrind output.
5.
==533==
==533== HEAP SUMMARY:
==533== in use at exit: 134,671,721 bytes in 19,047 blocks
==533== total heap usage: 42,608 allocs, 23,561 frees, 1,113,301,337 bytes allocated
==533==
==533== LEAK SUMMARY:
==533== definitely lost: 0 bytes in 0 blocks
==533== indirectly lost: 0 bytes in 0 blocks
==533== possibly lost: 4,320 bytes in 15 blocks
==533== still reachable: 134,667,401
bytes in 19,032 blocks
==533== suppressed: 0 bytes in 0 blocks
==533== Rerun with --leak-check=full to see details of leaked memory
==533==
==533== For counts of detected and suppressed errors, rerun with: -v
==533== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] bit_1.1-14 compiler_3.5.2 tools_3.5.2 yaml_2.2.0 bit64_0.9-7 data.table_1.12.0
``