Fix AgentSet inplace shuffle (and thus RandomActivation), add tests by EwoutH · Pull Request #2007 · mesa/mesa

EwoutH · 2024-01-26T12:26:36Z

Adds a test that checks if the RandomActivation doesn't trigger agents in a sequential order. This test now fails, because as noted in #2006, it currently does do it in sequential order.

In theory this could give false positives (a test passing when it shouldn't, but that chance is around ~0.1^18).

Adds a test that checks if the RandomActivation doesn't trigger agents in a sequential order. In theory this could give false positives (a test passing when it shouldn't, but that chance is around ~0.1^18).

github-actions · 2024-01-26T12:31:33Z

Performance benchmarks:

Model	Size	Init time [95% CI]	Run time [95% CI]
Schelling	small	🔵 -1.3% [-1.5%, -1.1%]	🔵 -0.5% [-0.6%, -0.3%]
Schelling	large	🔵 -1.4% [-1.5%, -1.3%]	🔵 -0.2% [-1.1%, +0.7%]
WolfSheep	small	🔵 -0.2% [-0.5%, +0.2%]	🔵 -0.2% [-0.3%, -0.1%]
WolfSheep	large	🔵 +0.7% [-0.0%, +1.3%]	🔵 -0.4% [-1.4%, +0.4%]
BoidFlockers	small	🔵 -0.2% [-0.6%, +0.2%]	🔵 -0.4% [-1.0%, +0.3%]
BoidFlockers	large	🔵 +0.1% [-0.3%, +0.5%]	🔵 -0.2% [-1.0%, +0.7%]

quaquel · 2024-01-26T12:51:45Z

If you change do_each to

    def do_each(self, method, shuffle=False):
        if shuffle:
            self._agents.shuffle(inplace=True)
        self._agents.do(method)

it works correctly. I am going to check RandomActivationByType as well.

Do I have write access to this?

quaquel · 2024-01-26T12:56:07Z

I also think this bug fix will actually produce a speedup because we avoid a needless copy operation of the AgentSet.

RandomActivationByType should not have this bug in the first place because it does not rely on do_each

fixes mesa#2006

EwoutH · 2024-01-26T13:18:59Z

Thanks! For clarity, we should document the difference between agents, _agents and agents_ somewhere

EwoutH · 2024-01-26T13:43:33Z

@quaquel could you add unittests for shuffle? And maybe we should check coverage of the AgentSet again at some point (not this PR).

quaquel · 2024-01-26T14:03:51Z

do you mean for AgentSet.shuffle?

for more information, see https://pre-commit.ci

quaquel · 2024-01-26T14:14:44Z

Can you trigger the benchmarks? I am curious to see what this fix has done with those given that it removes an unnecessary copy operation.

rht · 2024-01-26T14:23:26Z

/rerun-benchmarks

…into test-randomactivation

for more information, see https://pre-commit.ci

EwoutH · 2024-01-26T14:44:42Z

Looks good! Benchmarks depend on #2002.

Can’t approve this PR, but looks good to me. Recommend squash merge.

tests/test_agent.py

fix for the second issue in mesa#2006. If super is not present, we create the data structure but forget to add the agent to it. This is just a backward compatibility fix.

…into test-randomactivation

No the chance on a false negative is one in 12! instead of 4! (40 million instead of 24)

rht · 2024-01-26T17:00:04Z

mesa/agent.py

        except AttributeError:
            # model super has not been called
            self.model.agents_ = defaultdict(dict)
+            self.model.agents_[type(self)][self] = None


This fix is out of the scope of this PR, but it's fine to add it here.

Yeah preferably this would have been a seperate PR.

…2007) * tests: Add test to check if RandomActivation is not sequential Adds a test that checks if the RandomActivation doesn't trigger agents in a sequential order. In theory this could give false positives (a test passing when it shouldn't, but that chance is around ~0.1^18). * fix for RandomActivation bug fixes #2006 * add agentset.shuffle unittest * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ruff fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add agent to model.agent correctly even if super was not called fix for the second issue in #2006. If super is not present, we create the data structure but forget to add the agent to it. This is just a backward compatibility fix. * test: Shuffle more agents to prevent false negatives No the chance on a false negative is one in 12! instead of 4! (40 million instead of 24) --------- Co-authored-by: Jan Kwakkel <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

quaquel · 2024-01-26T19:13:53Z

I was torn. It was raised in the same issue, and it was just 1 line.

yannickoswald · 2025-11-04T12:11:13Z

Hi, was any of these bugs present at 1.1.1? Could you clarify? Or are you sure the randomactivation scheduler used to work in 1.1.1 correctly? I cannot produce certain modelling results that I published with 1.1.1 in any of the subsequent versions. So far I tested 2.3.2 and 3.1.3. Thanks a lot that would be super helpful

quaquel · 2025-11-04T12:14:48Z

AgentSet was added near the end of MESA 2, so it most definately was not there for MESA 1.1.1.

yannickoswald · 2025-11-04T12:23:28Z

Ok great. Thanks for the quick reply. I used the time module still here like this in 1.1.1 mesa.time.RandomActivation(self). I am just trying to trace back what changed in (random) agent activation exactly compared to later versions. Even in 3+ versions I get strange results that I do not get using 1.1.1

quaquel · 2025-11-04T12:25:45Z

Are you sure the issue is just agent activation?

yannickoswald · 2025-11-04T12:29:26Z

Not 100% of course, but given that with 1.1.1 the results look smooth, as expected given model specifications and are reproducible entirely using this version, but created abrupt strange behavior from time step 1 onwards in later versions I am also definitely not sure it is not agent activation in mesa.

yannickoswald · 2025-11-04T12:48:22Z

Of course I consider the likelihood for mesa to have a bug there quite low, as compared to my own model or how i used scheduler and time. Could you pinpoint me to the agent scheduling /sequencing in the newest release? I cannot find which module it is part of. Just so i understand what is happening in my 1.1.1 model.

quaquel · 2025-11-04T12:52:55Z

Agent scheduling was removed in MESA 3. All possible schedulers that existed can be implemented on top of AgentSet.

Random activation in the simplest case is just model.agents.shuffle_do("step")

EwoutH · 2025-11-04T13:01:48Z

Could also be in changes in seed / rng?

If you want to get to the root of the issue, open a new issue with a Minimal, Reproducible Example

EwoutH added the testing Release notes label label Jan 26, 2024

EwoutH mentioned this pull request Jan 26, 2024

RandomActivation scheduler not working after upgrade 2.1 -> 2.2.3 #2006

Closed

tests: Add test to check if RandomActivation is not sequential

a7a2c87

Adds a test that checks if the RandomActivation doesn't trigger agents in a sequential order. In theory this could give false positives (a test passing when it shouldn't, but that chance is around ~0.1^18).

EwoutH force-pushed the test-randomactivation branch from 612caa2 to a7a2c87 Compare January 26, 2024 12:30

quaquel mentioned this pull request Jan 26, 2024

Reimplement schedulers to use AgentSet #1926

Merged

5 tasks

fix for RandomActivation bug

90f0408

fixes mesa#2006

quaquel force-pushed the test-randomactivation branch from b7603d9 to 90f0408 Compare January 26, 2024 13:05

EwoutH added the bug Release notes label label Jan 26, 2024

quaquel and others added 2 commits January 26, 2024 15:10

add agentset.shuffle unittest

1b62560

[pre-commit.ci] auto fixes from pre-commit.com hooks

6c2c5ad

for more information, see https://pre-commit.ci

quaquel and others added 3 commits January 26, 2024 15:26

ruff fix

3abd882

Merge branch 'test-randomactivation' of https://github.com/EwoutH/mesa …

ef31723

…into test-randomactivation

[pre-commit.ci] auto fixes from pre-commit.com hooks

8c0d9c1

for more information, see https://pre-commit.ci

EwoutH commented Jan 26, 2024

View reviewed changes

tests/test_agent.py Outdated Show resolved Hide resolved

EwoutH requested a review from Corvince January 26, 2024 14:46

EwoutH changed the title ~~tests: Add test to check if RandomActivation is not sequential~~ Fix AgentSet inplace shuffle (and thus RandomActivation), add tests Jan 26, 2024

quaquel and others added 3 commits January 26, 2024 16:52

add agent to model.agent correctly even if super was not called

430d6f7

fix for the second issue in mesa#2006. If super is not present, we create the data structure but forget to add the agent to it. This is just a backward compatibility fix.

Merge branch 'test-randomactivation' of https://github.com/EwoutH/mesa …

a8c2b3a

…into test-randomactivation

test: Shuffle more agents to prevent false negatives

ed371ab

No the chance on a false negative is one in 12! instead of 4! (40 million instead of 24)

EwoutH requested a review from rht January 26, 2024 16:39

rht reviewed Jan 26, 2024

View reviewed changes

rht approved these changes Jan 26, 2024

View reviewed changes

rht merged commit 29f5dad into mesa:main Jan 26, 2024

This was referenced Jan 26, 2024

replace weakref dict with normal dict #1993

Closed

Speedup of Agentset.shuffle #2010

Merged

Uh oh!

Conversation

EwoutH commented Jan 26, 2024

Uh oh!

github-actions bot commented Jan 26, 2024

Uh oh!

quaquel commented Jan 26, 2024

Uh oh!

quaquel commented Jan 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EwoutH commented Jan 26, 2024

Uh oh!

EwoutH commented Jan 26, 2024

Uh oh!

quaquel commented Jan 26, 2024

Uh oh!

quaquel commented Jan 26, 2024

Uh oh!

rht commented Jan 26, 2024

Uh oh!

EwoutH commented Jan 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rht Jan 26, 2024

Choose a reason for hiding this comment

Uh oh!

EwoutH Jan 26, 2024

Choose a reason for hiding this comment

Uh oh!

quaquel commented Jan 26, 2024

Uh oh!

yannickoswald commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quaquel commented Nov 4, 2025

Uh oh!

yannickoswald commented Nov 4, 2025

Uh oh!

quaquel commented Nov 4, 2025

Uh oh!

yannickoswald commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yannickoswald commented Nov 4, 2025

Uh oh!

quaquel commented Nov 4, 2025

Uh oh!

EwoutH commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

quaquel commented Jan 26, 2024 •

edited

Loading

EwoutH commented Jan 26, 2024 •

edited

Loading

yannickoswald commented Nov 4, 2025 •

edited

Loading

yannickoswald commented Nov 4, 2025 •

edited

Loading