Dear all,

I found out an extreme slow down affecting for instance the hadd command that was introduced by the following commit
06b9878#diff-765b2bfb5aed033baddbf7036ddda5a103b3b82fa5456b462b74b31cd79dc51b
which modifies the TFileMerger class.

The slow down affect ROOT versions >= 6.22.08

The problem is the call of the

else
  dirtodelete.Add(result);

When hadding files, those two lines are called for each object found in a TDirectory.
So for instance if you store histograms in a TDirectory the directory is going to be added to dirtodelete as many times as you have histograms in that directory. And this slows down condiderably the hadd command as previously there was not such thing done. The deletion is called several times over the same object as well which is also inneficient.

With some inputs containing several TDirectories, hadding with ROOT version 6.22.06 takes around 1 hour on lxplus
whereas with ROOT version >= 6.22.08 it would take more than a day...

I can provide inputs files to illustrate the problem that are stored here
/eos/user/b/bouquet/VHbbcc_results/testHadd_3periods/
(3 files of size ~5GB each they contain ~640 TDirectory each one containing around 6000 histograms)

Or available thanks to the following link
https://cernbox.cern.ch/index.php/s/FBryG2p5Lshp6Lx
(in practive you can download just one file and replicate it 2 times to have 3 files to hadd and do tests)

I can tell the hadd is much slower as the names of Tdirectory processed are printed out and the difference in time to process only just one TDirectory is large depending on the ROOT version used it takes few seconds (v6.22.06) versus 10-15 minutes (>= v6.22.08).

This bug needs to be absolutely fixed as it would affect many analyses and it needs backported to all the root versions starting from 6.22.08.

I made a test commenting the two lines with the following gitlab repository https://gitlab.cern.ch/robouque/haddtests

else
  dirtodelete.Add(result);

the speed is restored even using ROOT 6.22.08. So the problem really comes from adding the directory many times to that list.

Many thanks in advance,
Best,
Romain