Hi,

I found bugs in the two ensemble methods (AlgoEnsembleBestN, AlgoEnsembleBestByFold) of AlgoEnsemble in Auto3dseg.

Concerning AlgoEnsembleBestByFold, your version of the class sets the value of best_score=-1.0 (the Dice Coefficient) when it starts searching for the best model for each fold. However, the code does not update the value of best_score with that of the best model found up to that iteration for that fold. Consequently, since the Dice coefficient value cannot be negative, each new analysed model becomes the best_model. The problem is that in this way the class AlgoEnsembleBestByFold does not return the best model for each fold, but always the last analysed model according to the order in which it runs the algorithm_templates (which is independent of model performance).

As regards AlgoEnsembleBestN, in collect_algos() function, ranks is a list that contains models sorted from the worst to the best based on Dice Coefficient: the positions of ranks are the so called 'ranks' but the values are the positions linked to the self.algos models list. As consequence the models to be removed are the N-n_best first ones (N=total number of models, n_best=the number of best models you want to ensemble, that are at the end of the ranks list). When you use (i,r) in enumerate(ranks), i is rank position in ranks while r is the correspondent ranks value (the model position in self.algos). As consequence, you have to select models to be removed based on i (the first N-n_best rank positions) but you have to remove models from self.algos, so you have to return r. On the contrary, your implementation returns i and select i based on r>=n_best, as consequence you are not doing the ensemble of the true n_best models.

I have corrected the bugs for both classes.
I'm sending you a pull request with the correct code implemented.