I am less sure about the tiered API. Flat is better than nested.

I think this may refer specifically to nested conditionals. While a chain of method calls should be considered to be something along the line of the paradigm of the UNIX pipes, that are found in Bash pipes, Golang interfaces, and DataFrame composition of operations.

DataCollector(model, collectors={"wealth": collect(model.agents, "wealth"),
                                                          "gini": collect("wealth", func=calculate_gini)}
	})

Has a problem in that "wealth"'s definition depends on the order of evaluation of the dictionary, where "wealth" has to be evaluated first before "gini". A Python dictionary is unordered, and just happens to preserve key-value insertion order.

Has a problem in that "wealth"'s definition depends on the order of evaluation of the dictionary, where "wealth" has to be evaluated first before "gini". A Python dictionary is unordered, and just happens to preserve key-value insertion order.

Exactly, but that can be left to the internal logic of the DataCollector class to sort out. For example, you built up an internal defaultdict(list) with the object from which to retrieve attributes as keys and a list of attributes to retrieve. You first execute this. Next, you could have a second internal dict that maps each object+attribute pair back to the associated Collector (there are other ways as well but that would require some smartness in the Collector). This would not require any specification by the user because DataCollector is smart enough to figure this out itself. I hope this is clear, otherwise I am happy to code up a quick example of what I am trying to say.

Even if it can work under the hood, even if the behavior can be documented, I still find it problematic to deviate from the known behavior of Python dict, with the dict elements becoming ordered, and the next elements that may depend on the previous element being defined first. The dict elements should have all been defined simultaneously.

I am not sure what you are trying to say. I was trying to describe the internal logic of the DataCollector class and how it is pretty easy to reduce the number of times specific data is retrieved.

It seems that you are saying that the behavior of a class (DataCollector in this case) should be determined by the arguments used to instantiate it. This to me, however, is nonsensical. Is Agent an int just because one of the arguments (i.e., unique_id) is an int?

So we still would have to work around this problem internally which complicates the code.

I haven't tried a quick implementation, but it can be done in just five lines of code, so it is not that complicated from a code base point of view. The understandability of the API to the user is a more significant concern.

This way, you don't mix any logic into your data collector. I think it is actually bad practice to calculate things in the data collector. It should basically an observer.

I agree with this in principle. However, from practical experience, I know it can be convenient to support it (for example, to reduce the amount of data that needs to be stored, not needing to touch the source code itself if you want to quickly track something on the fly). It also would be a backward incompatible change.

I think users shouldn't be imposed on whether they should or should not process the raw data they have collected. With a small change in the API

model.datacollector.collect({"wealth": lambda model: model.agents.get("wealth")})
    .process({"gini": lambda data: calculate_gini(data["wealth"])})
    .process({"wealth**gini": lambda data: data["wealth"] ** data["gini"]})
    .store(["gini", "wealth**gini"])  # optional, if the user needs to store only a subset of the data collected

This is no longer a tiered data collection, but rather a series of processing over the first collection, and so is easier to explain.

An API along these lines also occurred to me. What I like is that it is declarative and clear. The question for me is whether the user should specify what to collect and process when initializing the DataCollector, in which case this is executed whenever datacollector.collect() is called. This is how the current DatacCollector is designed. Or, do the full specification of what to collect and process whenever you call datacollector.collect() as in the example given by @rht.

Or do we want to be even more radical and fully rerig datacollection on top of the observer design pattern?

Thanks for this useful summary. I broadly agree. Some additional thoughts below

On 5. Yes, it is important that the datacollection mechanism is open for extension so users can easily replace the default behavior with their own custom (database) solution. The question is whether to do this at the level of each individual collect call, at the level of a complete run, or both.
On 7. I personally think this is indeed the way to go. It will be quite a bit more performant. For example, when testing this on a few models over Christmas, it was about 20%-30% faster than the current data collection. This speed up is mainly due to a shift from "pull" based data collection to "push" based data collection. So, e.g., instead of querying each agent for each collect call, you have a Measure that reflects the underlying data to which agents push their state updates. Another reason why I am in favor of this is that it enables a much cleaner separation of concerns (as evidenced by the discussion #1994 on tight vs. loose coupling between cells and the DiscreteSpace to which they belong. But, as indicated by @Corvince in #1947, adding pub/sub requires a very careful design and will add to the code base that needs to be maintained.
On 8. Can you elaborate on this. I am not sure I fully understand what you mean.
On 9. Not sure about this. If Collectors and Measures take a callable, the basic mechanism is in place for this. Should MESA maintain a list of default operations?

On 8. Can you elaborate on this. I am not sure I fully understand what you mean.

Users sometimes may want to get all the data collected of a particular group. With the current data collector, it would be get_model_vars_dataframe and get_agent_vars_dataframe. This needs to be generalized to groups. I have updated the summary with this info

On 9. Not sure about this. If Collectors and Measures take a callable, the basic mechanism is in place for this. Should MESA maintain a list of default operations?

I think @EwoutH wants an easy (minimal ceremony) way to apply statistics on the measures. The "object" in the current data collector is too indirect to be manipulated. I think your idea (and in @Corvince's wish list) of making it a concrete object solves this issue, because it allows one to do np.mean(model.wealth) (or np.mean(model.wealth.value) depending on the implementation detail), etc.

Optional comment: This seems to stem from the fact that Python's dict-key and Python object-attribute are not fully equivalent, unlike in JavaScript where I can specify an object as a dictionary

const quiescents = {wealth: ..., x: ..., y: ...}

And I'd be able to access wealth via quiescents.wealth, as an object attribute of quiescents.

Should MESA maintain a list of default operations?

Only as a last resort. But the idea of measure as a concrete object/attribute of a model, seems to render this unnecessary.

If Collectors and Measures take a callable, the basic mechanism is in place for this.

You actually already wrote this down before I did. And I somehow missed it. So point 9 already has a proposed solution.

I should have provided a problem that I want to solve, which is problem 8 in #1944 (comment). I think measure-containing-multiple-attributes/functions may solve problem 8, because it could be considered a preemptive organization method, instead of downstream, for the user to group-by attributes/functions for 1 group in 1 measure. The problem with the user doing group-by at downstream, is that while they may be aware which measures are from the same group, the Mesa framework isn't, because it is not specified explicitly.

Moreover, once one allows for multi-indexing, virtually anything can be gathered into a dataframe.

How does multi-indexing work? An example to see it in action would help.

Between the 2 choices, I prefer whichever has an overall simpler structure, both user facing (easy to conceptualize) and the underlying code.

The problem with the user doing group-by at downstream, is that while they may be aware which measures are from the same group, the Mesa framework isn't, because it is not specified explicitly.

This is not a problem. I suppose the framework does have an awareness of each measure's group. If I implement a function

combine_into_one_dataframe(measure1, measure2, ...)

in #2024's implementation, I can always check each measure's group attribute to see if there a group mismatch.

How does multi-indexing work?

I meant pandas multi indexing

This is not a problem. I suppose the framework does have an awareness of each measure's group. If I implement a function

That is basically in line with how I have approached it in my branch. Each Collector has a to_dataframe method. If the user wants to combine multiple dataframes into a bigger dataframe, for example, because each dataframe contains data about the same group, the user can easily do this via pd.concat. The benefit of this to me was that it keeps the MESA side of things relatively simple and easy to explain, while opening up more sophisticated use cases for the user to implement themselves on top of this by leveraging e.g., pandas.

I'd say the ReactJS-state API of the self.gini_coeff Measure above would require the __get__ method that would retrieve the latest value as of the most recent self.set_gini_coeff. The difference between Solara's API and ReactJS's API is that the former combines var, set_var into 1 variable.

I thought that using the model.measure1 to retrieve the measure1 object, while model.measure1.value to retrieve the value has a parallel with Solara's use_reactive, which is Solara's alternative API to ReactJS's useState:

Ok, this is slightly different from what I was trying to achieve and, indeed, easier. The consequence of going this route is that at the level of Collectors, you need to differentiate between collectors that collect attributes and collectors that collect measures. The former can do obj.attr, while the latter needs to do obj.attr.value. This can be handled either within the collector class (by checking the return of each obj.attr, or by having separate Collector subclasses.

I am not familiar with either Solara's use_reactive or ReactJS's useState. I'll try to find time to read up on this.

At the moment, I am trying to develop a conceptual design and working implementation for data collection that does not rely on pub/sub. Ideally, this design is such that the public API can stay in place even if the implementation is switched to pub/sub. Once we all agree on the conceptual design, I'll try to demonstrate how the implementation becomes much easier if you use pub/sub (see @EwoutS comment in #1947).

@Corvince what is your take on the ReactJS angle on the Measure class?

I asked ChatGPT about the similarity/difference between ReactJS state management and pub/sub pattern.
Similarity:

While React's state management and the pub/sub pattern serve different purposes and operate within different scopes, they share underlying principles related to event-driven data flow and decoupling of components or actors. Understanding these similarities can help developers design more effective and efficient React applications, particularly when integrating with or implementing event-driven architectures.

Difference:

ReactJS state management is specifically tailored for handling UI state within the React framework, facilitating a component-based and unidirectional data flow that is closely tied to the React component lifecycle for rendering dynamic interfaces. In contrast, the Publish/Subscribe (pub/sub) pattern is a general-purpose, loosely coupled messaging paradigm designed for decoupled communication across different parts of an application or between different applications, supporting a dynamic and multi-directional flow of information. While React state is integral to component reactivity and structure within a UI context, pub/sub offers broader application-wide communication capabilities, enabling flexible and scalable architectures without being tied to any specific UI or framework.

I think the ChatGPT answer regarding the differences is pretty good. But first on useState and reactive. Reacts useState and solaras reactive are fundamentally different strategies to solve the same problem. The latter is not the former combined in a single object. Most importantly useState can only be used inside a component, or, in solara terms with a rendering context (that @solara.component provides). It returns a plain Python object with no magic attached. What triggers state updates are the side effects of the set_* function, which "inform" the rendering context of the state update.

On the other hand solaras reactive can be used anywhere and is not bound to any component, providing global state (there is an equivalent use_reactive for local state). But again, something needs to track these changes. So if you access a reactive variable inside a @solara.component, the component actually subscribes to the reactive object, which publishes its changes.

So in summary, yes, solaras reactive and pub/sub are strongly related. UseState, not so much. You can read more about reactive in the Vue docs

Regarding the API, after seeing the implementation of @quaquel I think the solution provided by @rht in his PR probably suitable. Especially it indeed could be turned into being reactive (or pub/sub) with the same API. Then again, maybe Measure and MeasureDescriptor can indeed be merged and we get the best of both worlds?

what is the drawback of combining Measure and MeasureDescriptor?

Then again, maybe Measure and MeasureDescriptor can indeed be merged and we get the best of both worlds?

To the best of my knowledge, it is technically impossible to combine Measure and MeasureDescriptor. Descriptors are defined on classes, not on instances. So, in my example with 2 model instances, you have two different Measure instances (one per Model instance) but only 1 ModelDescriptor instance (on the Model class). This is also one reason for my code example's rather complicated __setattr__ implementation (the other has to do with how the Descriptor protocol interacts with __new__). Of course, the descriptor and __setattr__ complicatedness would be hidden from the user if we decide to go down this route. They would become part of the MESA framework, and users would only need to work with the Measure class.

This is a fair summary of the state of the conversation.

At present, I guess we have identified four classes: DataCollector/CollectorContainer, Collector, Measure, and Group. Each of these might have several subclasses (e.g., AgentSetCollector). Let my try to summarize:

Group is a collection of objects, typically agents, and can be static or dynamic. Dynamic groups are groups were membership changes over the course of the simulation. This can be because agents are added or removed from the model, or because membership is based on the agent being in some state.

Measure reflects a state variable of the model at a given time instant. Ideally, it would be possible to access the value of a measure through attribute lookup

Collector collects measures and attributes and stores them. This allows one to capture the change of them over the course of the simulation.

DataCollector/CollectorContainer, in my thinking, is more of a convenience class. It contains all collectors, and a single collect call on it invokes collect on all collectors.

Quick update: I discussed this conceptual design with the scientific programmer working on several large MESA-based ABMs. Based on our discussion, this design seems to cover virtually all of their use cases. She particularly liked the ideas of dynamic/conditional Groups/AgentSets, the separation between Measure (part of the state of the Model at a given time instant) and Collector (tracking state over time), and the idea of making Collectors easily extendible on the storage side of things.

An interesting thought we had: It might be useful to let an "aggerate" collector collect another "group" collector. So for example, a group collector collects wealth, and then the aggerate collector aggerates to a single value.

An interesting thought we had: It might be useful to let an "aggerate" collector collect another "group" collector. So for example, a group collector collects wealth, and then the aggerate collector aggerates to a single value.

That's point no. 6 in the summary: #1944 (comment)

Quick update: I discussed this conceptual design with the scientific programmer working on several large MESA-based ABMs. Based on our discussion, this design seems to cover virtually all of their use cases. She particularly liked the ideas of dynamic/conditional Groups/AgentSets, the separation between Measure (part of the state of the Model at a given time instant) and Collector (tracking state over time), and the idea of making Collectors easily extendible on the storage side of things.

It's great to have user validation of the idea, in that it covers complex use cases. Just needs to make sure the DynamicAgentSet is performant. Does she have an additional wish list regarding with the data collection?

What are the other contenders for the DynamicAgentSet backend implementation?

The class would need to have a base set of agents (or default to model._agents if no base set is provided). Next select, do, get, shuffle, and sort would all take this base set, run it through the condition and then do their normal thing. So, effectively, they all would first call select on the base set of agents. So it's not that complicated.

You could expand on this, just as we discussed with Measure, by having some kind of updated flag that is reset each time tick, while having a force_update keyword to override the flag if necessary. This can be a performance saver if if the DynamicAgentSet/ConditionalAgentSet is used several times without any intervening state changes to any of the agents within the set.

I don't see other viable backend implementation other than message passing, where the model holds a global message inbox object (this is apparently how hash.ai does it). But this sounds similar to the flow library, which @quaquel described the shortcomings of

I don't know enough about hash.ai to comment, but in principle pub/sub or the Observer design pattern is a a software design idea, and I would not conflate it with what I called a flow library which is a discrete simulation concept.

I don't see other viable backend implementation other than message passing,

I wrote this in a way that could be easily misunderstood. I did not mean message passing in the way that hash.ai to be the only viable way instead of pub/sub. I wanted to say that it is one major contender to pub/sub as the backend implemententation. And so, would need to compare contrast between the 2 implementations. I have not decided on either.

hash.ai's agent messages are specifically for communication among agents that are not neighbors. They are not used for communicating between agents and non-agent objects, as such their design shouldn't necessarily be taken into consideration as a backend for dynamic AgentSet. The only other alternative is that for a condition-based membership, that it is always computed from scratch. This has a simple naive implementation, but not performant.
As such, I am on board with pub/sub as the backend for dynamic AgentSet, which should be implemented and merged first before the new data collection construct.

I have done some more work on pub/sub and datacollection in my datacollection branch. I hope to find time over the weekend to do an actual performance comparison between 2 implementations for ConditionalAgentSet.

1. pub/sub based, this is already there and working nicely. Initial performance assessments on the Epstein example suggests that it adds little overhead (0.3 seconds on a 4 second run for all data collection). And the entire pub/sub structure adds less than 0.1 second to the run if no datacollection is used.
2. A more traditional/naive implementation where the ConditionalAgentSet has a base set and for do, shuffle, get, select, and sort, you first apply the condition to the base set. This can be quite straightforwardly implemented because applying the condition is like an annotation to each of these methods, so this can be handled in the constructor (i.e., __new__. I am curious to see what the performance overhead on the full data collection will be of this implementation.

Note that I prefer ConditionalAgentSet over DynamicAgentSet as a name.

I committed an alternative implementation along the lines of point 2. Initial testing doesn't show a massive performance difference between pub/sub and the naive implementation, at least for the Epstein model.

If you operate over the "columns", you are aggregating the information across all objects, while if you operate over "rows", the function is applied to the collected data for each object separately.

Yes, I like this. You could even do both, in some order.

I feel we're close! What's needed to get this done?

mainly time from my side I guess, but with my talk last week done, I hope to get back to this and devs.

It is indeed becoming simpler, but we are not there yet.

1. It is still not trivial to refer to the current agents of a given type. So your Wolf or Sheep example API does currently not work. Likewise, we don't yet have an easy solution for dynamically changing agent sets.
2. In my vision, a given collector c can take as an object another collector. This makes it possible to avoid double collection of data:

# partial and incomplete sketch of collect func. signature
def collect(name:str, target:Any, attributes:strList[str], func:callable):
    ... # logic goes here

wealth = collect("wealth", model.agents, "wealth")
gini = collect("gini", wealth_c, method="calculate_gini")
avg_energy = collect("average energy", model.agents, "energy", np.mean) # no need to use agg in agentset here. 

3 I don't fully follow your agg idea

@EwoutH I like this. I think I like this a lot!

Let me think a bit more about this, but I think this is very close!

Not sure I share the concerns of @quaquel . But let me think about this a bit more

These remarks are not so much concerns, but more puzzle pieces that are still missing. I think the main one is point 1. How can we make it easy to say that you want to collect data for a given Agent class or a subset of agents that is dynamically changing over time?

In terms of API, I broadly agree. I am just not sure about the dynamic agent filtering. The api as shown will evaluate at instantiation, rather than at runtime.

Can you elaborate on the schema stuff because I haven't seen this before and I have no idea what is going on.

It's basically a way to efficiently store data with highly repeating data. Instead of storing a "energy" or ""Steps" key every step, you save all dictionary keys once and then you apply them when reconstructing it into a dataframe or whatever.

This way you can store lists with only values, instead of dicts with keys + values.

The api as shown will evaluate at instantiation, rather than at runtime.

That's a good catch!

I really like this API! It’s simple to use but powerful enough to cover many different scenarios. Nice work!

Basically we go through 3 steps:

• Consensus on the problem
• Consensus on the conceptual solution (this often includes user API and high level design)
• Consensus on the implementation

But finding a good solution that has a clean API, is extendable, has acceptable implementation complexity and is useful for a broad set of users, often does require extensive discussion and iteration.

Excellent question by the way!

Thanks!

Thank you for highlighting!

@tpike3 @EwoutH - has Mesa has a package that advanced Mesa functionality that was fully independent of Mesa yet? (We have had packages that were officially associated or were domain specific applications (eg economics model), but I don't think the Mesa community has had a package that was fully independent of Mesa and specifically focused on advancing/modifying functionality.
This might be the first one.

Thanks for the write-up, appreciated, and excited we are moving forward on this again

Thank you for moving forward and putting together the summary.

Open issue 2 -

• The first seems to be more scabable.
• The second seems to be more aligned. with our property layers.

What do you see as the trade offs for each of the conceptual directions?

Having a reactive dataset is something I can implement quite quickly to see how it performs. Its efficiency will likely be model-specific. For example, in a boltzmann style model, any given agent interaction results in two attributes being updated, resulting in two signals that have to be handled. With 100 agents, each doing an exchange, you have 200 signals. Versus a simple for loop that would require only 100 agent attributes to be accessed. In constrast, something like sugarscape might have a lot less attribute updates that you want to gather. The main agent attribute gathered here is the trading partners, and if trade-partners don't change every tick, signals are sparser and likely to be faster.

I have a working proof of concept for the second option. It was not that difficult to implement, but as discussed here, it has various consequences and limitations. Most importantly, all attributes in a given DataSet must be of the same type, and it only really makes sense to do this for numbers (float, int). I want to test pandas/polars for performance, in which case we might have a bit more flexibility with multiple data types in a single dataset.

Regardless, both an observable-based agent data set, or a numpy-based agent data set is bound to be better than the current situation. Both will allways be in sync with the agents, and thus can be used for calculating other stuff. For examle, we can store agent.wealth in a dataset, use this dataset to calculate model.gini, and store both via the collector. Avoiding at least one, and possibly two loops over all agents. I am therefore inclined to just implement both and explain their use cases, leaving it to the user to profile to see which is most appropriate for their use case.