Skip to content

[PERF] Improve move_to_empty performance#1116

Merged
jackiekazil merged 1 commit intomesa:mainfrom
rht:move_to_empty
Jan 31, 2022
Merged

[PERF] Improve move_to_empty performance#1116
jackiekazil merged 1 commit intomesa:mainfrom
rht:move_to_empty

Conversation

@rht
Copy link
Copy Markdown
Contributor

@rht rht commented Dec 23, 2021

This fixes #1052. This is based on JuliaDynamics/Agents.jl#541 (cc @Libbum).
Note:

I benchmarked this on the Schelling model.

benchmark
The black line is a horizontal line at y = 1.
Haven't had the time to repeat the run multiple times to get a plot with standard deviation, but at least here you can see how much the speedup increases as we increase the grid size.

densities = np.linspace(0.01, 0.9, 20)
for L in [5, 10, 15, 20]:
    speedups = []
    for density in densities:
        model_params = {
            "height": L,
            "width": L,
            # Agent density, from 0.8 to 1.0
            "density": density,
            # Fraction minority, from 0.2 to 1.0
            "minority_pc": 0.2,
            # Homophily, from 3 to 8
            "homophily": 3,
        }
        elapseds = []
        for cutoff in [0.0, 0.998]:
            model = Schelling(move_to_empty_cutoff=cutoff, **model_params)
            tic = time.time()
            for i in range(100):
                model.step()
            elapsed = time.time() - tic
            elapseds.append(elapsed)
        speedup = elapseds[0] / elapseds[1]
        speedups.append(speedup)
    plt.plot(densities, speedups, label=f"L={L}")
plt.axhline(1, color="black", label="1")
plt.legend()
plt.xlabel("Agent density")
plt.ylabel("Speedup (current main branch / this PR)")
plt.savefig("benchmark.png")

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Dec 24, 2021

The next question is to whether remove the grid's self.empties attribute altogether. Currently, the remaining use of this attribute is that to check whether the grid is full in the function position_agent. Agents.jl doesn't do this. And I can see that instead of checking that the grid is full every time, we can do during the initializatio to check that the ratio between the number of agents and the grid area must not exceed 1. Thoughts?

This decision needs to be resolved before I can fix the tests.

@rht rht force-pushed the move_to_empty branch 2 times, most recently from 5ddbe7d to 1c11a9a Compare December 24, 2021 13:58
@tpike3
Copy link
Copy Markdown
Member

tpike3 commented Dec 27, 2021

Sorry, it took me awhile to really think through this. I don't think we should get rid of self.empties as my sense is it will break a lot of models.

On question in comments you say Agents.jl uses 0.998 as the cutoff score but say the number is different here. Not being familiar with how Julia stores numbers are you saying that although the numbers are the same the way they are stored is actually comparing a different value?

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Dec 28, 2021

I don't think we should get rid of self.empties as my sense is it will break a lot of models.

I have checked self.empties usage. It is only used in move_to_empty and position_agent (when x and y are 'random'). It shouldn't be a public attribute that Mesa users should use either.

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Dec 28, 2021

On question in comments you say Agents.jl uses 0.998 as the cutoff score but say the number is different here. Not being familiar with how Julia stores numbers are you saying that although the numbers are the same the way they are stored is actually comparing a different value?

My code comment is certainly misleading. I currently hardcode the cutoff to be 0.998 (same as Agents.jl) because I haven't done the search for the optimum number for Mesa. I will probably have to use the exact same code @Libbum use (translated to Python from Julia) to get the number.

Just wanting to emphasize that the speedup is much, much higher when the grid size is 200 x 200 (i.e. the problem raised in #1052), it's just that it is taking too long on my laptop to measure. I'd consider 200x200 to be a common laptop-scale problem.

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 17, 2022

Codecov Report

Merging #1116 (c24b612) into main (246c69d) will decrease coverage by 0.36%.
The diff coverage is 73.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1116      +/-   ##
==========================================
- Coverage   89.46%   89.09%   -0.37%     
==========================================
  Files          19       19              
  Lines        1253     1266      +13     
  Branches      256      259       +3     
==========================================
+ Hits         1121     1128       +7     
- Misses         97      101       +4     
- Partials       35       37       +2     
Impacted Files Coverage Δ
mesa/space.py 94.92% <73.33%> (-0.98%) ⬇️
mesa/batchrunner.py 91.55% <0.00%> (-0.65%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 246c69d...c24b612. Read the comment docs.

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Jan 17, 2022

@tpike3 I have made sure that the logical diff of this PR is as small as possible, and so this PR is ready for review. This means that self.empties is still present. But this PR still improves the speed of move_to_empty.
See
benchmark

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Jan 17, 2022

I had to modify the test because the move_to_empty requires the total number of agents in order to compute the threshold. This requires access to the model object. Some of the the mock agents don't have model object attached to them.

I checked Agents.jl's implementation, and it operates on the model directly (https://github.com/JuliaDynamics/Agents.jl/pull/541/files#diff-007fa315900dda19e79dff8b7fe546cd50232c4faf79c5551ca37fa9fc9d1f3bR85), and so a grid is never independent of a model.

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Jan 17, 2022

If we choose to always require a grid object to be tied to a model, this PR will become simpler.
The only drawback is that the grid & agents are no longer standalone, and so we sacrifice modularity.

I'm not aware of projects that use Mesa's grid & its move_to_empty without using Mesa's model and scheduler.
Thoughts?

@tpike3
Copy link
Copy Markdown
Member

tpike3 commented Jan 17, 2022

Boy, I had to really chew on this one.

I agree I have never seen a model that uses Mesa's grid & its move_to_empty without using Mesa's model and scheduler. So I am confident with backwards compatibility we are fine there.

I am hesitant to reduce the modularity by always having a grid, the seminal zero intelligence traders does not need one and I have done SIR models just on a network. In general, I think being able to use a network abstraction without a physical location is a useful option. So, I am in favor of keeping it modular.

So @jackiekazil thoughts? This LGTM if you agree and want to merge.

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Jan 18, 2022

There are 2 directions of dependency here:

  • a grid that depends on a model.
  • a model that depends on a grid. While I read that Agents.jl has the random_empty requiring a model (with a discrete space) as an argument, they can totally have a model without physical location, too.

@jackiekazil jackiekazil added this to the Page milestone Jan 30, 2022
@jackiekazil
Copy link
Copy Markdown
Member

I did a review of this... yeah. I think we can merge and move the remaining to future discussions and PRs.

@jackiekazil jackiekazil merged commit 193e383 into mesa:main Jan 31, 2022
@rht rht deleted the move_to_empty branch January 31, 2022 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SingleGrid position_agent very slow for large grids

3 participants