1. However, historically and conceptually, next-event time progression and incremental time progression are distinct paradigms.

Curious what the reasoning for that was. Was it natural evolution or technology limitations, or where the fundamental arguments to split it that way?

True, next event time progression is more fundamental in that incremental time progression can be expressed in it, but still.

I agree that a nice endpoint for incremental time progression could be useful for incremental learning (@tpike3 you had a nice term for this type of learning, can you remember it?)

2. I am not sure Mesa should worry about experimentation with models.

Yes and no. Experimentation is key, but that doesn't mean we have to implement it ourselves.

NetLogo has to, since they're in a very closed ecosystem, but we're in Python, so we can leverage the ecosystem. But simple guides and example how to connect (and if needed connectors) could replace current batch_run stuff.

3. I am not sure about the UI side of things in general.

While as a power user I largely agree, it's by far the most frequent request by teachers to have good visualisation (in my experience).

4. Not sure about the AI/ML stuff.

I'm also not. Especially LLMs I think are extreme overkill. But it's an area in very active development, and I think the current explicit rule-based paradigm has severe limitations.

5. Yes to the module idea.

Awesome. Now to find the maintenance time...

Curious what the reasoning for that was. Was it natural evolution or technology limitations, or where the fundamental arguments to split it that way?

I suppose it's more of a domain issue. Queuing stuff is more naturally expressed in the next event, while, e.g., crowd simulations are easier to express in incremental time.

Yes and no. Experimentation is key, but that doesn't mean we have to implement it ourselves.

I think we agree on this. What matters is that it is easy to run experiments with Mesa models. This means that it is easy to set parameters, run a model, and extract data from it. The first two are in a good position, the last is not because of the trouble with the data collector.

While as a power user I largely agree, it's by far the most frequent request by teachers to have good visualisation (in my experience).

My concern here is also about the long-term viability of Solara as the basis, as well as the need, in my view, to switch to a reactive design for the UI. I want to be able to vary the run speed of the animation without changing the run speed of the model. I also want to be able to replay (part of) a run.

I suppose it's more of a domain issue.

So that would mean Mesa would have a great opportunity to connect domains by supporting both / the full spectrum, right?

Yes we do agree.

My concern here is also about the long-term viability of Solara as the basis

Yeah this indeed might be a problem.

reactive design for the UI

Mesa-interactive was certainly moving in the right direction.

One complication: So many stuff is build around model.step() progressing time. While that served Mesa 3 well, it means DataCollector, batch_run and SolaraViz rely on it, with it often being deeply embedded in core logic. That make it difficult to overhaul. My initial idea would be removing DataCollector and batch_run for Mesa 4. Then we can do a clean sheet design for both, considering what's already present in the Python ecosystem, the latest Python features (like dataclasses) and our new core logic.

This highlights my point above: if you can "connect" in a reactive manner, we have a much weaker coupling, thus avoiding these problems in the future.

Hi everyone!

I have been researching the alternatives to replace the current batch_run and data_collector and make them obsolete. Following the suggestion from @EwoutH and @quaquel in #3103, I suggest a new architecture. It directly addresses the need for a clean API to set values, run models, and extract output.

Problem

1. Tools like Vensim (DLLs) , NetLogo (NetLogoLink) , and Simio all provide fine-grained control: load, set_value, run_for, and get_data.
   Mesa currently lacks this. batch_run relies on its own internal loop, and users often write wrapper functions just to run a model for steps. We lack a standardized way to say model.run_for(10) or model.reset().

2. Simio and others use a Scenario object to centralize parameters (inputs, seeds, replications).
   Mesa models often rely on untyped **kwargs in __init__. (Thanks to @quaquel, it is introduced in Add explicit support for Scenarios #3103 )

3. The current DataCollector tries to do too much. It requires manual "pulling" (self.datacollector.collect(self)), which is error-prone. It also tightly couples the model to the output format, making optimization (which might not need a dataframe) difficult.

Conceptual Design

I propose an architecture that separates Execution, Configuration, and Data.

The Architecture Diagram

Key Components

1. The Engine (RunControl)
   Standardizes execution logic. It enables the desired API:

model.run_for(100)
model.reset()

2. The Configuration (Scenario)
A standardized object to hold initialization parameters. It absorbs **kwargs, creating a clean interface(see #3103)

3. The Data System (CollectorListener)
Moves from "Pull" to "Push". The Model emits signals, and the Listener reacts. This solves the "DataCollector doing too much" problem by allowing us to swap listeners (e.g., a Polars listener for speed, a Pandas listener for compatibility, or no listener for pure speed).

Above architecture also addresses @tpike3's concern

I have been implementing this architecture lately. If these design align with future goals for mesa 4.0, I'd love to open a pathfinding PR to explore more ideas and invite discussions.

Sounds great. Some good ideas in here.

I think Jan and I have the scenario and execution covered. Jan has done a lot with signals, I think the two main concerns here are A) simple, preferably elegant, API for the user and B) performance. I would zoom in there.

Elegant data storage objects/structure could also be an area of interest. Since we often have sparse data sets, track values for a whole AgentSet, agents get added and removed, etc.

Quick update: I’ve implemented the new listener architecture using an Abstract Base Class (ABC) pattern with suggestions and reviews from @quaquel (ref #3145) . This allows users to swap storage backends (In-Memory, SQL, Parquet, JSON) without touching their model code.

1. Architecture Overview

The design splits the collection logic into three layers:

1. Registry (DataRegistry): Defines what to collect.
2. Orchestrator (BaseCollectorListener): Defines when to collect. Handles config, intervals, and subscribes to model.steps.
3. Backend (Subclasses): Defines where to store. Implements the _store_dataset_snapshot contract.

Design Decisions:

• Observable-Driven: Collection is triggered reactively via model.steps, removing the need for manual collect() calls (though manual is still supported).
• Block-Based Storage: Data accumulates in blocks, and is only converted to DataFrames upon retrieval.

Class Diagram

classDiagram
    class BaseCollectorListener {
        <<Abstract>>
        #_on_step_change()
        #_store_dataset_snapshot()*
    }
    class CollectorListener {
        +get_table_dataframe()
    }
    class SQLListener {
        +query()
    }
    class ParquetListener {
        +_flush_buffer()
    }

    BaseCollectorListener <|-- CollectorListener
    BaseCollectorListener <|-- SQLListener
    BaseCollectorListener <|-- ParquetListener

2. Usage Examples

A. In-Memory (Default replacement for DataCollector)
Zero-config, stores data in memory blocks.

listener = CollectorListener(model)
for _ in range(100):
    model.step()

# Converted to DataFrame on demand
wealth = listener.get_table_dataframe("wealth")

B. SQL Backend (Streaming)
Useful for large sims where memory is a constraint.

listener = SQLListener(model, db_path="simulation.db")
for _ in range(100):
    model.step()

# Query directly without loading into Python
df = listener.query("SELECT avg(wealth) FROM wealth WHERE step > 50")

C. Parquet Backend (Batching)
Writes compressed columnar files.

listener = ParquetListener(model, output_dir="results/")
for _ in range(100):
    model.step()
# Creates 'results/wealth.parquet', etc.

D. Scheduling
Intervals are configured per-dataset, independent of the backend.

config = {
    "wealth": {"interval": 1},      # Every step
    "network": {"interval": 10},    # Every 10 steps
    "heavy_metric": {"interval": 100, "start": 500} # Start late, collect rarely
}
listener = CollectorListener(model, config=config)

3. Current Implementation

I have the core architecture and 4 working backends:

1. CollectorListener: In-memory block storage.
2. SQLListener: SQLite streaming with auto-schema inference.
3. ParquetListener: Batched Parquet writer.
4. JSONListener: Simple JSON export.

4. Roadmap & Performance

I'm currently targeting two retrieval optimizations:

1. Using np.vstack on raw storage blocks to merge arrays before DataFrame creation (avoids the concatenation overhead).
2. Updating the ABC to accept start_step/end_step/max_steps/retention_size(or window_size)

I would appreciate any feedback or code reviews on:-

1. Does BaseCollectorListener expose the right methods for custom backends?
2. Is the DatasetConfig flexible enough for complex scheduling?
3. Should we lean harder into Polars for the list-of-dicts optimization, or keep Pandas as the first-class citizen?
4. Are above ideas worth following up
5. Anything else...

Thanks!

This looks great! I have been pursuing some other things (mesa-rust, and some pathfinding for mesa-signals), but I hope to spend some time this week again on DataSet and DataRegistry.

Some quick thoughts

Observable-Driven: Collection is triggered reactively via model.steps, removing the need for manual collect() calls (though manual is still supported).

I suggest tying it to model.time. This is the path forward anyway and will also make this work in purely discrete-event models. As you might have seen in #3212, it is trivial to make model.time observable.

Intervals are configured per-dataset, independent of the backend.

I think this is really cool and powerful. I'll have to take a closer look, but the fact that you allow for interval and also specify when to start is great. Models can have a warm-up period, and you might only want to collect data after it. With this design, we have that covered in a very natural way. One minor consideration is whether to do this config via a dict or via a lightweight dataclass. The advantage of the dataclass is that it's easy to use in an IDE and that you can set defaults, so you only have to specify what you want to change.

Should we lean harder into Polars for the list-of-dicts optimization, or keep Pandas as the first-class citizen?

I am torn on this. Pandas is a staple in most academic teaching, so it's what at least students will know. Do we use pandas anywhere outside of data collection at the moment? Probably inside solara UI stuff.

One minor consideration is whether to do this config via a dict or via a lightweight dataclass. The advantage of the dataclass is that it's easy to use in an IDE and that you can set defaults, so you only have to specify what you want to change.

You are spot on. I actually implemented a DatasetConfig dataclass in the code (see baseCollectorListener.py), but I used a plain dict in the example for brevity. The dictionary input in init is just syntactic sugar to allow users to easily override specific fields without importing the class (e.g., config={"wealth": {"interval": 10}}). Internally, the listener immediately converts these dicts into strict DatasetConfig objects.

self.configs: dict[str, DatasetConfig] = {}

I will expose this in API soon to end-users

I am torn on this. Pandas is a staple in most academic teaching, so it's what at least students will know. Do we use pandas anywhere outside of data collection at the moment? Probably inside solara UI stuff.

Pandas is the lingua franca for students and researchers. Let's keep Pandas as the first-class citizen for the return type (get_table_dataframe returns a pd.DataFrame). However, we can use Polars internally (if installed) as an optimization engine to speed up the conversions.

I suggest tying it to model.time. This is the path forward anyway and will also make this work in purely discrete-event models. As you might have seen in #3212, it is trivial to make model.time observable

Agreed 100%. If model.time is observable, switching the listener to subscribe to time instead of steps is the right move for full DES support

I have some first results:

The Rust code is not feature-complete yet, and there is still room for further improvement. But it now contains everything: Grid, Cell, and CellCollection. For example, select on CellCollection is missing. I also just added CellCollection, but it uses an internal Python dictionary rather than a Rust HashMap. There are a bunch of similar things where I was using Python datatypes instead of Rust datatypes while developing and testing.

For those curious, the Rust code can be found here: https://github.com/quaquel/mesa-rust. The modified examples can be found here: https://github.com/quaquel/mesa/tree/mesa-rust.

I checked https://github.com/quaquel/mesa-rust but it’s currently returning a 404.

The repo was set to private by default. It should now be visible.

I would like to help on this if u are willing.

I think we need three things:

• Classes you can catch
• Relevant metadata included (Mesa version, Agent which caused the error if in Agent, place on grid if by grid, etc.)
• Informative messages that clearly explain

The first is for it being machine-processable, the second two for it being human-interpretable.

This allows users to wrap their simulation logic in a try/except block that catches only Mesa-related failures

This would be a very nice capability, with a clear use case.

Here are the references to how these libraries structure their exceptions:

1. Pandas
2. Scikit-Learn

In both cases, they avoid scattering exception classes throughout the codebase. They stick to standard Python exceptions (ValueError, TypeError) for argument validation, but use these centralized custom exceptions for library-specific state errors.

Classes you can catch

As long as you subclass (in)directly from BaseException (starting from Exception is recommended), this will work.

Relevant metadata included (Mesa version, Agent which caused the error if in Agent, place on grid if by grid, etc.)

It will be difficult to do with default Python exceptions, which only provide stack trace information (so, where in the code the error originates). I also fail to see why you would need this if you already have the stack trace.

Informative messages that clearly explain

Not sure what you mean here. There are at least two parts to this: having well-defined exception types, and including a clear text message when raising the exception.

You are right that for debugging, the stack trace is usually sufficient.

However, metadata on the exception object becomes valuable for programmatic handling and structured logging, especially in large scale batch runs.

I agree we should avoid over engineering. We can start with a simple class hierarchy (MesaError) and only add metadata fields to specific subclasses when we have a concrete use case for them.

I agree that stack traces are usually sufficient for debugging, and that we should avoid adding complexity without a clear use case. The main value I see in a Mesa-specific exception is not better debugging, but explicit programmatic handling: allowing users to catch Mesa-related failures without swallowing unrelated Python errors.

To keep this minimal, I think it would be reasonable to start with:

1. a single MesaError base class (subclassing Exception) defined in one central place,
2. continued use of built-in exceptions (ValueError, TypeError, etc.) for argument validation.

Hi @EwoutH,
That sounds like a great idea for Mesa 4. Keyword-only arguments can significantly improve readability and safety, especially for methods with boolean flags or multiple optional parameters.

To get started, here are a few candidate methods in agent.py that seem like perfect fits for this change. What do you think about starting with these?

1. AgentSet.select: Force keywords for the filter options.
2. AgentSet.sort: Force keywords for the sorting options.
3. AgentSet.get: Make the error handling and default values explicit.

Would you like me to proceed with applying these changes to agent.py first?
Thanks for any help.

We’re not in the implementation phase yet. First we need consensus on directions, goals and priorities.

Agreed, I'd suggest we prioritize methods where:
There are multiple optional configurations.
There are boolean flags (to avoid True/False confusion).

Hello, @EwoutH
As a researcher who uses Mesa mainly to develop and refine models over extended periods of time (often revisiting code months later), I firmly believe that Mesa 4 should move toward a more explicit and evolution-friendly API design.
In particular, it seems to be consistent with research workflows to use keyword-only arguments for basic Model and Agent methods. In practice, models tend to accumulate optional parameters over time and positional arguments become brittle; small API changes can subtly alter semantics or require intricate refactoring across multiple projects. With keyword-only arguments serving as a mechanism to support additive evolution.

Is it anticipated that Mesa 4.x will treat core APIs (such as Model/Agent interfaces and run-control entry points) as comparatively stable once finalized?

Keyword-only APIs trade a small amount of verbosity for long-term stability, readability, and safer evolution. This is especially valuable in academic contexts, where models are reused, shared, and extended by others who may not be deeply familiar with the original implementation.

In practice, models tend to accumulate optional parameters over time

I recognize that.

Is it anticipated that Mesa 4.x will treat core APIs as comparatively stable

We follow SemVer. That mean with major version we may break API (from 3 to 4), but with minor version (from 4.0 to 4.1) we never break API. See also our deprecation policy.

Thanks @ShreyasN707, I broadly agree, with one additional detail. Next to lifecycle signals, I am still wondering whether time changes should not be reactive as well. If time changes are reactive, the UI can listen to this and update its rendering. Likewise, a data collector that listens to this can now know when to collect. One can then even specify the frequency. For example, only collect on every full-time unit increment.

Thanks, that helps clarify it — and I agree the time side is important here.
Making time changes reactive feels like a really good fit for things like the UI and data collection, without turning the whole model into an event-driven system. If there’s a simple time_changed (or similar) signal whenever the model clock advances, the scheduler can stay in control, but anything observing the model gets a clean, reliable hook.

That would make collectors and visualizations much simpler too. Instead of being tied to step() or a specific scheduler, they could just listen to time and decide when to run — for example, only on full time units, every N ticks, or when some threshold is crossed.

Combined with lifecycle signals for agents coming and going, this gives a nice separation: structure changes and time progression are observable, but the simulation logic itself stays straightforward and explicit.

I've been thinking about the signal overhead concern you raised, especially for lifecycle events like bulk agent creation. What if we added opt-in context managers to control when signals fire?

The Overhead Problem

If we add lifecycle signals (register_agent, deregister_agent), initializing 1000 agents means 1000 individual notifications. Same issue during batch removals in NumpyAgentDataSet — each tombstone triggers a signal storm listeners don't really need individually.

Two Context Managers

1. suppress_signals() — silence during bulk operations

with model.suppress_signals():
    for i in range(1000):
        model.add_agent(Agent(i))  # No signals emitted
# Signals resume after exit

2. batch_signals() — deduplicate and emit on exit

with model.batch_signals():
    agent.x = 1  # Queued
    agent.x = 2  # Overwrites (same observable)
    agent.y = 5  # Queued
# Exit: emits x=2, y=5 (not intermediate x=1)

See #3227

I agree that suppression breaks reactivity by design - that's its explicit purpose. The key is that it's opt-in and scoped. Consider the real-world scenario that motivated this:

# Without suppression - 1000 signals during initialization
for i in range(1000):
    model.add_agent(Agent(i))  # Counter increments 1000 times

# With suppression - observers choose their semantic level
with model.suppress_signals():
    for i in range(1000):
        model.add_agent(Agent(i))
model.notify("agents_initialized", count=1000)  # Counter increments once

The Counter observer in your example should be out of sync during the block that's the contract.

The question is: does the observer care about 1000 individual AGENT_ADDED events, or does it care about the final state? If it needs every event, don't use suppression. If it needs the aggregate, emit a summary signal after.

On Batching Ambiguity
You're absolutely right that batching events vs. state is different, and I should have been clearer. The proposal distinguishes between these:

State updates (observables on agents) - batching deduplicates:

with model.batch_signals():
    agent.x = 1  # Queued
    agent.x = 2  # Overwrites
# Exit: emits x=2 only

Lifecycle events (register_agent) - batching doesn't deduplicate, it aggregates:

with model.batch_signals():
    model.add_agent(Agent(1))
    model.add_agent(Agent(2))
# Exit: emits [Agent(1), Agent(2)] as single "agents_registered" event

If we fire 1,000 individual signals at the end, you're right - that's just shifted overhead. The implementation should aggregate lifecycle events into a batch notification:

# Instead of 1000 × AGENT_ADDED
model.notify("AGENT_ADDED", agent=agent_1)
model.notify("AGENT_ADDED", agent=agent_2)
# ...

# Emit once with collection
model.notify("AGENTS_ADDED", agents=[agent_1, agent_2, ...])

On Complexity and Debugging
The concern about consistency bugs is valid. To mitigate this:

• Fail-fast validation - Warn if users try to nest incompatible contexts (e.g., suppress inside batch)
• Optional strict mode - A model.signals.strict = True flag could raise if any suppression/batching is attempted, helping users debug "why isn't my observer firing?"

The trade-off is: some complexity in signal machinery vs. forcing every user to manually implement batching/suppression in their own code.

1. The same argument, i.e., the lack of microbenchmarks, can be made for all of Mesa. Why single out signals?
2. I agree that performance matters. But, the general advise is to first profile the code. Typically, 20% of the code is responsible for 80% of the run time. So, why would we need microbenchmarks when we haven no profiling results?
3. We have benchmark models. They give us a more thorough insight into the overal runtime costs. Why are they not sufficient? Clearly since core mesa currently does not yet use signals, we don't see the performance. But once Making Model reactive #3212 is merged, we start the integration and, as evidenced by the benchmarks there, we start seeing the impact of signals

Thank you for the feedback, @quaquel. Those are fair points.

You're right that:

1. The lack of isolated benchmarks applies broadly to Mesa, not just signals
2. Profiling should come before targeted micro-optimization
3. Once Making Model reactive #3212 is merged, the existing benchmark models will naturally capture signal overhead

I think I may have framed this incorrectly. My concern wasn't about micro-optimizing signals in isolation, but rather about having visibility into signal overhead as reactive patterns become more central to Mesa.

To be clear, I would not mind having a wider suite of microbenchmarks that covers agentset, mesa.discrete_space, and mesa.experimental. continuous_space. However, this is not essential right now, and I would not run these as part of the default CI. It's just useful at times (e.g., while doing stuff on mesa-rust).

Once the new data collection PRs are merged (So that is #3212, #3145, and #3156), and used in the benchmark models, we can get a better sense of where we stand.

Also, I am not sure that the current benchmarking models cover all relevant parts of the code base. For example, I still want to add SugarScape to the benchmark suite because it is the only model that uses property layers. Also, we lack a good model with dynamic adding and removing of agents in continuous space, missing a critical component for benchmarking the performance of continous space.

That makes sense. A few follow-up thoughts:

On the benchmark model coverage gap you mentioned:

1. SugarScape for PropertyLayers - I can help add this to the benchmark suite once the new data collection PRs are merged.

Also, we lack a good model with dynamic adding and removing of agents in continuous space, missing a critical component for benchmarking the performance of continous space.

2. Dynamic agent model for ContinuousSpace - Do you have a specific model in mind, or should we design one? Something like a predator-prey with births/deaths in continuous space?

I’m in doubt about this. It’s adding another step in our, sometimes already long process.

I’m also not sure how this interacts with our experimental space, where most features land initially. In practice, we see most features change relative little from their initial design to stabilization (which is good!). However, if you require a MEP before landing in experimental, you take away velocity. If you require it before stabilization, it just becomes fancy documentation because all discussions already have been had.

In my perception, the de facto process is already quite close to what @jackiekazil outlines. Most large-scale changes start with a discussion. Next, one or more pathfinding PRs are opened. Once a PR has been accepted, there is often a tracking issue for working out the remaining implications and possible refinements. Also, most big changes land in experimental first, giving ample time to discover the real-world implications of a change. It is extremely rare for a major change to Mesa to land within a few days of opening a PR. Thus, there is ample time for everyone to provide input both at the original inception, at the PR, and when deciding to stabilize.

What is, however, lacking is indeed a single place where it is all written up. Look at how many discussions there have been on data collection. Ideas are often spread across several discussions, PRs, and issues. This can make it difficult, at times, to reconstruct how and why we ended up with a design. At the same time, I am afraid that introducing a new structure, such as a MEP, will not fundamentally change any of this. I fear that a MEP would just replace the original first discussion. The other option is to actively close MEPs that won't be merged. But this would force us to rewrite and summarize older discussions whenever we are refining an idea. This seems a lot of work with, in my perception, limited benefit.

I don't think we have to write and rewrite -- we can commit to documenting the finalize path and having some sort of collective description, which can reference links -- but tracking at the moment is incredibly painful unless you track an item from the very beginning.

Maybe there is an initial propose and then highlighted bullets at the end of the proposal that describe the final result (whether it deviated or aligned, and if it deviated, how so).

This is the problem I want to solve for: "What is, however, lacking is indeed a single place where it is all written up."

I don't know of a clean solution to this. Generally, the final write-up is in the big PR. We do actively ensure that this correctly reflects whatever is merged. All of these can still be found, but are not actively linked from anywhere. I am, however, not sure that git or, more broadly, GitHub is the right place to store this kind of information. One direction might be to include it in readthedocs. Basically, take the detailed big PR message and turn it into an MD file, and have a curated list somewhere on the website.