Change Snapshot option from file cache to TTree auto-flush parameter#595
Conversation
|
Starting build on |
| } | ||
| trees[slot]->Fill(); | ||
| if (files[slot]->GetBytesWritten() >= cachesize) files[slot]->Write(); | ||
| if (autoflush > 0 && trees[slot]->GetEntries() % autoflush == 0) |
There was a problem hiding this comment.
I wanted to have used the condition below, but somehow that blocks progress and the snapshot gets stuck when autoflush < 0.
if ((autoflush > 0 && trees[slot]->GetEntries() % autoflush == 0) ||
(autoflush < 0 && trees[slot]->GetZipBytes() >= -autoflush))
files[slot]->Write();There was a problem hiding this comment.
From where that code stands, it should 'never' act when autoflush is negative anyway (because it needs to act only after at least one cluster has been closed).
There was a problem hiding this comment.
That is true. However, never acting when autoflush is negative means that buffers are never sent to be merged until the end, in which case the user may blow up the memory on his system and lose his whole analysis...
There was a problem hiding this comment.
in which case the user may blow up the memory
The memory is still limited by the TTree itself (i.e. the size of the cluster * the compression) and adding the test where you have would not, by definition, change that limit ....
There was a problem hiding this comment.
The memory of the TTree is limited. However, the memory of the TMemFile where things end up is not (i.e. TBufferMergerFile).
There was a problem hiding this comment.
For the record, the disconnect was me mis-reading the current code which instead of using tree->GetAutoFlush() was using the parameter passed by the user. Using tree->GetAutoFlush is essential as the value changes as the TTree move from the first events (where GetAutoFlush is negative and expressed the desired size in compressed data size and during which the real cluster size is being determined) to the normal regime (when GetAutoFlush returns the size of the cluster in number of event).
There was a problem hiding this comment.
Ok, after the discussion I understand the problem and the solution now. However, after I changed the variable for the function call to tree->GetAutoFlush(), buffers are written every 4KB, which is very slow.
There was a problem hiding this comment.
If by 'buffer are written' you mean the 'Write' is triggered very often ... then I am very confused (because it should only triggered when the cluster is full to 30Mb of compressed data) ... something must still be wrong with the test itself ...
There was a problem hiding this comment.
what are the value of GetAutoFlush and GetEntries when the if-statement triggers?
There was a problem hiding this comment.
My bad. I used the wrong branch of the tests by mistake. Those still use megabytes as the unit, which made the write happen way too often.
|
Build failed on mac1012/native. Failing tests: |
53b33e3 to
aa40254
Compare
|
Starting build on |
|
Build failed on ubuntu14/native. Failing tests: |
|
Build failed on slc6/gcc62. Failing tests: |
aa40254 to
8dce471
Compare
|
Starting build on |
8dce471 to
fab2053
Compare
|
Starting build on |
fab2053 to
2353b78
Compare
|
Starting build on |
|
Build failed on mac1012/native. Failing tests: |
Please take a look. Although the tests pass, this is not supposed to be merged yet, as I have yet to understand why it's so slow when we pass a positive auto-flush value to snapshot the tree.