np.split looks to split the array into N equal spaced arrays wherein N is 100 here. As a result, it appears that for any combination of input where the number of evaluations returned is not a multiple of 100, this function will throw an error.

To recreate:

what works

import openml
df = openml.evaluations.list_evaluations_setups(function='predictive_accuracy', flow=[8353], task=[6], output_format='dataframe', parameters_in_separate_columns=True)

what fails

df = openml.evaluations.list_evaluations_setups(function='predictive_accuracy', flow=[5891], task=[37], output_format='dataframe', parameters_in_separate_columns=True)

comparison

Since list_evaluations_setup calls list_evaluations internally.

df = openml.evaluations.list_evaluations(function='predictive_accuracy', flow=[8353], task=[6], output_format='dataframe')
print(len(df)) #1000

df = openml.evaluations.list_evaluations(function='predictive_accuracy', flow=[5891], task=[37], output_format='dataframe')
print(len(df)) #2429