-
Notifications
You must be signed in to change notification settings - Fork 94
Custom combinations of sampling schemes #411
Description
At the moment, the workbench by default samples levers and uncertainties, resulting in policies and scenarios. Next, these are combined in a full factorial manner so that each policy is evaluated over all scenarios. A single sampling scheme is used for levers, and a separate one for the uncertainties.
However, sometimes users want to create more complicated experimental designs. For example, one might want to sample several uncertainties in a full factorial manner, while sampling the remainder using a Latin hypercube. At the moment, this is not well supported by the workbench directly.
One possible direction is to draw a sharper distinction between BaseEvaluator.perform_experiments and BaseEvaluator.evaluate_experiments. At the moment, the former handles experiment generation, while the latter implements the details of running experiments depending on the kind of evaluator (e.g., ipyparallel, mpi, multiprocessing).
It might be an idea to turn evaluate_experiments into something that just does that: given some collection of experiments, it just runs them and that's it. If we then expand the sampling side with some helper functions for combining samples, it becomes possible to generate any complicated experimental design that you like.
builds on #47