Skip to content

Refactor PropertyLayer to implement NumPy interface#3074

Merged
quaquel merged 12 commits intomesa:mainfrom
codebreaker32:feat/refactor_property_layer
Jan 7, 2026
Merged

Refactor PropertyLayer to implement NumPy interface#3074
quaquel merged 12 commits intomesa:mainfrom
codebreaker32:feat/refactor_property_layer

Conversation

@codebreaker32
Copy link
Copy Markdown
Collaborator

@codebreaker32 codebreaker32 commented Jan 5, 2026

Summary

Refactors PropertyLayer to serve as a direct wrapper around the underlying NumPy array, enabling standard NumPy syntax (slicing, broadcasting and methods)

Motive

The current PropertyLayer class acts as an overly complex wrapper that duplicates native NumPy functionality with custom methods like set_cells and modify_cells. This design:

  • Violates Pythonic principles: It hides standard operations behind a slower, restrictive API ("explicit is better than implicit").
  • Hinders Performance: Methods like modify_cells rely on np.vectorize for lambdas, which is significantly slower than native ufuncs and broadcasting.
  • Blocks Future Architecture: As discussed regarding the model.world vision, layers should ideally be flexible views on data. The current design tightly couples data to legacy access patterns, acting as an obstacle to the "Single Source of Truth" concept.

Implementation

  1. Direct exposure of NumPy data
  • Replaced the internal attribute _mesa_data with a public attribute data
  • data is now a public NumPy array, enabling slicing, masking, and aggregation without wrappers
  • Eliminates unnecessary np.copyto overhead.
  1. More efficient in-place updates
  • set_cells and modify_cells now operate directly on self.data.
  • Conditional updates use boolean masks instead of rebuilding arrays.
  1. Improved modify_cells semantics

@github-actions

This comment was marked as outdated.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 5, 2026

Since you are now raising a deprecation warning, all tests are failing. So, the tests need to be updated in line with the new API.

Performance gains look modest but are still very welcome.

@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from b82907e to 34d213a Compare January 5, 2026 14:44
@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from b740272 to 182bbb7 Compare January 5, 2026 14:52
@codebreaker32
Copy link
Copy Markdown
Collaborator Author

I believe it is fit to review now @quaquel

@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from d6a5b64 to 6842547 Compare January 5, 2026 17:48
@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from e623d41 to f6675c0 Compare January 5, 2026 17:57
@codebreaker32
Copy link
Copy Markdown
Collaborator Author

Hi @quaquel
I was working on it for past days so I made some unrelated changes unintentionally. Sorry for that
Now I've reverted them to clean up the diff

The PR is now back to just the essentials:

  1. Added the new dunder methods to support the NumPy array interface.
  2. Added deprecation warnings to the old methods (set_cells, etc.) without removing them.

Tests have been updated to handle the warnings.

@quaquel quaquel added performance Release notes label enhancement Release notes label deprecation When a new deprecation is introduced labels Jan 5, 2026
@codebreaker32 codebreaker32 requested a review from quaquel January 6, 2026 04:26
@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from dd00441 to f0da9a4 Compare January 6, 2026 09:12
@codebreaker32
Copy link
Copy Markdown
Collaborator Author

codebreaker32 commented Jan 6, 2026

Hi @quaquel I have now removed all the dunder(magic) methods and accordingly updated the PR description

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 6, 2026

How do you want to proceed? Just merge this for now or do you want to take a second stab at the dunder methods as part of this PR. I am fine either way.

@codebreaker32
Copy link
Copy Markdown
Collaborator Author

I think we should merge this first so that it would be easier for anyone(or us) working on #3072

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 6, 2026

@EwoutH, what do you think? PropertyLayer.data or PropertyLayer.values as the name for getting the numpy array behind the property layer.

I am fine either way, and we should not overthink it.

@codebreaker32
Copy link
Copy Markdown
Collaborator Author

codebreaker32 commented Jan 6, 2026

I did a quick search to check the conventions.

While Pandas uses .values, they are slowly moving away from it in favor of .to_numpy(). On the other hand, Xarray and NumPy Masked Arrays (which are arguably closer to our grid structure) both use .data to expose the underlying array.

I think sticking with .data makes more sense given those precedents.

Checkout : https://stackoverflow.com/questions/13187778/convert-pandas-dataframe-to-numpy-array#:~:text=Use%20df.,values%20%2C%20here's%20why.&text=It's%20time%20to%20deprecate%20your,Index%20and%20Series%20objects%20only.

@EwoutH
Copy link
Copy Markdown
Member

EwoutH commented Jan 7, 2026

I think .values is slightly more intuitive. But both would work.

I while I really appreciate this effort, I would really like to put of making an (breaking) API change until we have figured out how PropertyLayers are going to fit in the bigger picture.

If some changes could be made without changing the API, like by making the implementations behind the current methods more elegant or NumPy native, that could already happen.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

I while I really appreciate this effort, I would really like to put of making an (breaking) API change until we have figured out how PropertyLayers are going to fit in the bigger picture.

Everything here is backward compatible, so there are no API-breaking changes; just a cleaner and faster alternative API.

@EwoutH
Copy link
Copy Markdown
Member

EwoutH commented Jan 7, 2026

We're deprecating stuff, that's basically an announced API break. Just not enforced for now.

Also I have my doubts if we should already add another API. While direct nparray access is probably useful in some cases, we might also expose it through the spaces those propertylayers are attached to.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

Also I have my doubts if we should already add another API. While direct nparray access is probably useful in some cases, we might also expose it through the spaces those propertylayers are attached to.

This is exactly what this PR does: space.property_layer.data, while already working, with th is PR works much more cleanly and becomes the preferred method. That is: this PR makes doing space.property_layer.data faster by eliminating intermediate steps, and highlights to users that this faster approach is the preferred method going forward.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

@codebreaker32, I had a longer in person conversation with @EwoutH. Given the ongoing conversations about spaces in Mesa 4, we decided not to deprecate anything yet. So, if you can remove those warnings, we think this should be good to go.

@codebreaker32
Copy link
Copy Markdown
Collaborator Author

Thanks for the update, @quaquel.

I will go ahead and remove the deprecation warnings.

One question: Can we still optimize the existing functions (like set_cells, modify_cells)? I could rewrite them to use the direct .data operations internally, so they are faster and cleaner, even if we aren't officially deprecating them yet.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

yes, feel free to rewrite them but then also profile the difference

@EwoutH
Copy link
Copy Markdown
Member

EwoutH commented Jan 7, 2026

I could rewrite them to use the direct .data operations internally, so they are faster and cleaner

Yes, this is fine and appreciated.

This also could be separate PR(s), that might make testing and reviewing easier.

@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from edfa173 to 04f91cd Compare January 7, 2026 15:37
@codebreaker32 codebreaker32 force-pushed the feat/refactor_property_layer branch from b492da1 to 80be02f Compare January 7, 2026 15:47
@codebreaker32
Copy link
Copy Markdown
Collaborator Author

codebreaker32 commented Jan 7, 2026

Summary of Changes

This update simplifies the PropertyLayer implementation, improves performance, and makes NumPy integration more explicit and predictable.

1. Direct exposure of NumPy data

  • Removed the @property data getter/setter indirection.
  • data is now a public NumPy array, enabling slicing, masking, and aggregation without wrappers
  • Eliminates unnecessary np.copyto overhead.

2. More efficient in-place updates

  • set_cells and modify_cells now operate directly on self.data.
  • Conditional updates use boolean masks instead of rebuilding arrays.

3. Improved modify_cells semantics

  • Operations now update only the selected cells, not the whole grid.

@codebreaker32
Copy link
Copy Markdown
Collaborator Author

Hi @quaquel @EwoutH,

I have updated the PR. I initially tried a simple rename from .data to .values, but it caused cascading failures. The reason is that the existing data attribute was actually a @property with specific setter logic (calling set_cells) that legacy code relies on.

I will change it in subsequent PRs

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

Can you update the original PR starting post to reflect the final version? I hope to review this later today, and if I don't see anything strange, merge it.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Jan 7, 2026

I will change it in subsequent PRs

I am fine with leaving it as .data for now. @EwoutH and I discussed removing the PropertyLayer class in Mesa 4 in favour of directly using numpy arrays. Additionally, changing the name now would be a deprecation and a hassle we can avoid.

@quaquel quaquel added trigger-benchmarks Special label that triggers the benchmarking CI and removed deprecation When a new deprecation is introduced labels Jan 7, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 7, 2026

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
BoltzmannWealth small 🔵 +2.0% [+1.0%, +3.0%] 🔵 -1.4% [-1.5%, -1.2%]
BoltzmannWealth large 🔵 -0.3% [-0.6%, -0.1%] 🔵 -1.4% [-3.7%, +0.6%]
Schelling small 🔵 +0.1% [-1.0%, +1.2%] 🔵 -0.4% [-1.0%, +0.2%]
Schelling large 🔵 +0.3% [-0.3%, +0.9%] 🔵 +0.8% [-0.0%, +1.9%]
WolfSheep small 🔵 -0.7% [-1.0%, -0.4%] 🔵 -0.9% [-1.2%, -0.7%]
WolfSheep large 🔵 -1.4% [-4.2%, +0.7%] 🔵 -1.6% [-3.3%, -0.2%]
BoidFlockers small 🔴 +4.4% [+3.8%, +5.0%] 🔵 +2.7% [+2.4%, +3.0%]
BoidFlockers large 🔴 +4.0% [+3.3%, +4.6%] 🔵 +2.2% [+1.9%, +2.5%]

@quaquel quaquel merged commit 7cebeb2 into mesa:main Jan 7, 2026
14 of 15 checks passed
@EwoutH
Copy link
Copy Markdown
Member

EwoutH commented Jan 9, 2026

@codebreaker32 Thanks for the PR. Could you check if the PR description is now fully up to date with the final implementation in this merged PR?

@codebreaker32
Copy link
Copy Markdown
Collaborator Author

Yes, the PR description is now up to date

@codebreaker32 codebreaker32 deleted the feat/refactor_property_layer branch January 26, 2026 06:58
@codebreaker32 codebreaker32 changed the title Refactor PropertyLayer to implement NumPy interface and deprecate wrappers Refactor PropertyLayer to implement NumPy interface Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Release notes label performance Release notes label trigger-benchmarks Special label that triggers the benchmarking CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants