Hi @heyufan1995 , I have a question that confuses me quite a bit. What's the benefits expectation of using multi-node to run the ensemble, as the ensemble execution is basically for-loops as below:

for file in files:
    for algo in algos:
        pred = algo[some_key].predict(...)

Hi @heyufan1995 , I have a question that confuses me quite a bit. What's the benefits expectation of using multi-node to run the ensemble, as the ensemble execution is basically for-loops as below:

for file in files:
    for algo in algos:
        pred = algo[some_key].predict(...)

@mingxin-zheng The files are partitioned to all nodes and gpus. So if you have 16 files, 2 nodes 16 GPUs, then each GPU will sequentially run 5 fold models on 1 file. I also think the for loop should be changed to

for algo in algos:
    for file in files:
        pred = algo[some_key].predict(...)

Since loading model weights can be slow.

Hi @heyufan1995 , thanks for the explanation. Your proposal also makes sense. One possible concern I had was that the memory needs to hold (n_algos x n_files) before the ensemble takes place, which was the reason that I went one file by another. We may need to be careful and do something when there are lots of files to infer.

By the way, infer_instance.predict supports multiple files in one batch in the predict_files argument.

Hi @heyufan1995 , thanks for the explanation. Your proposal also makes sense. One possible concern I had was that the memory needs to hold (n_algos x n_files) before the ensemble takes place, which was the reason that I went one file by another. We may need to be careful and do something when there are lots of files to infer.

By the way, infer_instance.predict supports multiple files in one batch in the predict_files argument.

Thanks, we can keep the current for loop. It's safer.

/black

Thanks @heyufan1995 for the PR. The design looks good to me as we minimize the number of breaking changes. There are some format issues and test failures to fix. CC @wyli for viz.

/integration-test

/build