@betatim @glouppe

DISCLAIMER: I varied the random state till gp_search performs better than random_search :P

🍒 picking :)

To avoid the option of tweaking random_state I would suggest running the whole thing several times and plotting the mean score after each iteration and the error on the mean.

I thought about that as well.

But finding the mean score at every iteration, kind of defeats the purpose right? Typically you have the budget to only run n iterations and not n * x, so it does depends on the random state. If we average this randomness out, we are more likely to get a better score in every iteration.

Maybe, I'll give it a try.

Doing n repeats isn't something you'd do in a real case. For the example it is a way to allow people to judge if there is a statistically significant advantage for GP and how big that advantage is.

If we repeat the example 100 times with 100 different seeds and ~50 times GP wins and ~50 random wins the conclusion would be that there isn't much difference. If however in 90 cases GP wins, you'd conclude that GP is really much smarter.

Indeed, I agree with you!

Wondering if average score after each iteration is the best way to visualise this or if it would be better to show the distribution of the final (after 100 iterations) score for each method.

I averaged the best scores across 5 random states. Hope the new graphs are convincing. Please merge if happy.

I have changed the example.

Thanks! However I think we should not mix the messages and illustrate only one concept at a time. What do you think of only showing how to use gp_minimize for hyperparameter search, without comparison with RandomizedSearchCV?

Removed the comparison with dummy_search and updated just to show how to use in combination with sklearn estimator.

@glouppe Addressed!

Merging!