I want to propose a middle ground. We could define a storage protocol and Mesa ships with one default implementation for new users (e.g. SQLite). More experienced users can swamp out for more advanced set ups.

Yes that is indeed the direction I am thinking of.

This touches on one of my open questions

Should we just return the data as stored by the data_recorder, or should we allow for a user-specified callable that takes the model as input and returns a pickleable object containing the result of the scenario the user wants to extract?

I am inclined to go with the callable route. But this does bring up a new question. Where to handle the replications. Inside the worker function or at the executor level? The less data is sent back from the worker, the better. If the plan is to use descriptive statistics over the replications, it makes more sense to handle replications within the worker. If the plan is to store the individual replications, it makes more sense to handle it at the executor level. My original proposal assumed the former, but below I give a quick sketch of the latter.

def run_scenario(model_cls, scenario, n_replications, n_steps, user_callable):
    results = {}
    for replication in scenario.spawn_replications(n_replications):
        model = model_cls(replication)
        model.run_for(n_steps)
        results[replication.replication_id] = model.data_recorders.get_data()

    processed_results = user_callable(results)
    return processed_results

I think it would be better to keep run_scenario focused on executing a single scenario and return the result for that run. In that case, replications would be handled at the executor level instead of inside the worker. Conceptually it keeps the responsibility of the worker simple by running one scenario and returning its result. Then the executor or experiment logic can decide whether to run multiple replications and how to aggregate or store them.Handling replications outside the worker also seems more flexible, since some experiments might want to store every replication separately instead of aggregating them immediately.

I have really enjoyed this thread. Tell me if I'm wrong, but this feels like the same decision as the data store.

The cleaner implementation is the default aimed at easy onboarding (executor level), with something like a mesa_hpc module that overrides for inside-the-worker behavior. Users learn one pattern and it applies consistently across both execution and storage. Does that module belong in core or external? (I can see either way.)

My opinion would be to keep something like mesa_hpc outside of core initially. The default execution pattern in core should stay simple, and more specialized execution patterns can evolve separately without adding complexity to Mesa itself. If the patterns stabilize and see broad use, it can always be reconsidered for addition to core later.

If we can realize what I am proposing, there is no need for a mesa.hpc. Let me try to explain. Below is a simple example (using my scenario development branch, #3493). Here, I use a ProcessPoolExecutor to execute scenarios in parallel over 4 cores. If I want to go to an HPC, the only change required is to do from mpi4py.futures import MPIPoolExecutor and use this instead of the ProcessPoolExecutor.

from itertools import cycle
from concurrent.futures import ProcessPoolExecutor
from scipy.stats import qmc

from mesa.experimental.scenarios.scenario import Scenario

from mesa import Model

class MyModel(Model):

    def __init__(self, scenario=Scenario):
        super().__init__(scenario=scenario)

    def step(self):
        pass

samples = qmc.LatinHypercube(4, rng=42).random(100,)
scenarios = Scenario.from_numpy(samples, ["a", "b", "c", "d"], rng=42)


def run_func(cls, scenario, n_steps):
    model = cls(scenario=scenario)
    model.run_for(n_steps)
    return scenario.scenario_id, scenario.replication_id, -1


if __name__ == '__main__':
    n_steps = 100
    with ProcessPoolExecutor(max_workers=4) as executor:
        results = executor.map(run_func, cycle([MyModel,]), scenarios, cycle([n_steps,]))

What is also clear from this example is that there are still various unresolved issues.

1. I need to use cycle to pass extra arguments, such as MyModel and n_steps. I want a more elegant solution for this. This will probably involve using a so-called initializer function that is passed to the executor: ProcessPoolExecutor(max_workers=4, initializer=init_fun, init_args=(MyModel, n_steps)).
2. I don't process results in this example. Results is a generator object that yields the result from each experiment. In this example, that will be a tuple with the scenario_id, the replication_id, and -1. In principle, this should be sufficient but the correct return will at least partly depend on the backend storage protocal that we have to decide on.

Thanks for the feedback

1. As long as we have not resolved how to handle run_while/stopping conditions inside mesa, it's difficult to design for it in something like RunSpec.
2. I deliberately stayed away from passing a callable. Lambda's and partials for example are not allways pickleable and so might break a ProcessPoolExecutor or an mpi4py.futures.MPIMPIPoolExecutor.
3. I like the idea of adding hooks to the base RunSpec class.

There is a performance concern regarding the dict[str, pd.DataFrame] return type in the sketch.

It is indeed a concern. But mainly if you are returning full agent data over time, which, in my experience, is rare to do when running a parameter sweep. That kind of data is more often used for diagnostic work on individual runs.

Python dicts, OR the future storage backend protocol should allow workers to write chunks directly to the backend (e.g., appending to a partitioned Parquet dataset or a SQLite database) rather than sending all heavy data back to the main process executor.

I have not thought about that. The problem is that when running in parallel (potentially even across multiple physical machines and thus relying on networking), multiple processes can simultaneously try to access the back end. So it becomes trickier to design. If the results are sent to the back end from the main process, it's a lot simpler.

As long as we have not resolved how to handle run_while/stopping conditions inside mesa, it's difficult to design for it in something like RunSpec.

This can be addressed by treating the execution logic as an overridable hook on RunSpec. The base class can provide a simple default implementation (like run_for) so that the common case works out of the box, while subclasses override the execution step when different run semantics are needed.

class RunSpec:

    def __init__(self, model_class: type[Model], steps: int = 100):
        self.model_class = model_class
        self.steps = steps

    def execute(self, model: Model) -> None:
        """Default execution logic."""
        model.run_for(self.steps)

    def extract_data(self, model: Model) -> Any:
        """Hook for extracting results from the model."""
        return model.recorder.get_data()

    def __call__(self, scenario: Scenario):
        model = self.model_class(scenario=scenario)
        self.execute(model)
        data = self.extract_data(model)
        return scenario.scenario_id, scenario.replication_id, data

Alternative run patterns can then be expressed by subclassing

class UntilRunSpec(RunSpec):

    def __init__(self, model_class, end_time):
        super().__init__(model_class)
        self.end_time = end_time

    def execute(self, model: Model):
        model.run_until(self.end_time)

I have not thought about that. The problem is that when running in parallel (potentially even across multiple physical machines and thus relying on networking), multiple processes can simultaneously try to access the back end. So it becomes trickier to design. If the results are sent to the back end from the main process, it's a lot simpler.

For local runs or simple setups, funneling data back to the main process to handle the writing is definitely the safest and simplest default. However, when scaling up to multiple physical machines (like with mpi4py), funneling heavy data back to the main process becomes a bottleneck. It forces the master node to hold the combined results of all workers in RAM.

future storage backend protocol should allow workers to write chunks directly to the backend (e.g., appending to a partitioned Parquet dataset or a SQLite database)

To elaborate it, Instead of workers fighting to write to the same file, each worker writes its own file to a shared network directory (e.g., results/run_1.parquet, results/run_2.parquet). The worker then just returns a lightweight status tuple back to the master node, like (scenario_id, "SUCCESS"). SQLIte database for main process

Because they are writing to separate files, there are zero database locks and zero network collisions. At the end of the experiment, Pandas or Dask can load that entire directory.

This is actually why I am inclined to hook idea! FOR eg. if an HPC user needs to scale, they can just subclass RunSpec, override the hook to write a partitioned Parquet file and good to go.

It forces the master node to hold the combined results of all workers in RAM.

Just to be nitpicky, but the main process can still flush to disk from time to time. So there is no hard need to keep everything in RAM but otherwise, I agree.

To elaborate it, Instead of workers fighting to write to the same file, each worker writes its own file to a shared network directory

This is indeed the simpler design for an HPC-style situation. The drawback from a user's point of view is that there will now be a difference between running locally and on an HPC, and the custom RunSpec might need to be tailored to the hardware you are running on.

I agree that a subclassable RunSpec is still the way to go. But I am wondering what the default implementation should be designed for.

But I am wondering what the default implementation should be designed for.

I think the default RunSpec should be designed for the simple case where a worker executes a single Scenario and returns its result to the main process. In that setup the default implementation stays very small: construct the model from the Scenario, run it and return the result together with the scenario_id and replication_id. More specialized workflows could then be implemented by subclassing RunSpec and overriding parts of the execution.

As long as RunSpec.__call__, or some equivalent returns the scenario_id, replication_id, and the results, you are fine.

Thanks, there are two other things around RunSpec that i wanted to settle.

1. For extract, a concrete default based on the recorder (e.g. get_table_dataframe(name) per dataset) is the most practical option. It gives a usable result, and anything more specialized can be handled by overriding.

2. format_output is not necessary and could be dropped. Letting subclasses change the return shape breaks the (scenario_id, replication_id, result) , which should stay fixed and be owned by __call__.