Skip the subset check in as_index() (in some cases)#1515
Merged
lkstrp merged 4 commits intoPyPSA:masterfrom Jan 6, 2026
Merged
Skip the subset check in as_index() (in some cases)#1515lkstrp merged 4 commits intoPyPSA:masterfrom
as_index() (in some cases)#1515lkstrp merged 4 commits intoPyPSA:masterfrom
Conversation
lkstrp
approved these changes
Jan 6, 2026
Member
lkstrp
left a comment
There was a problem hiding this comment.
Thanks for digging into profiling @chriselion ! There are probably tons of other things to improve. The entire as_index helper isn't implemented very well and is called too often. I'll ref #1487 to clean this up further
We are happy to receive PRs if you find any other things!
lkstrp
reviewed
Jan 6, 2026
Contributor
Author
|
@lkstrp Thanks, I'll have a look at that other issue later. I'm still learning my way around the code (and don't know pandas very well) but hopefully I can find something to improve. Are there any larger examples that would be worth examining? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (if applicable): n/a
Changes proposed in this Pull Request
I was doing some basic profiling, and it looked like a decent portion of the setup time was being spent in this part of the code:
PyPSA/pypsa/common.py
Lines 275 to 280 in 59a7d5d
In the case the input
valuesisNone, since we setvalues_ = n_attr, the subset check is redundant and we can skip that block.(more timing information below)
Checklist
docs. (n/a it's a one-line change with a comment)docs/release-notes.mdof the upcoming release is included.I used this to measure the total time:
Running before and after the change 5 times each, the best times were:
before:
elapsed: 7.80225883400999after:
elapsed: 7.512157290999312So it's not a huge difference overall, but since most of that time is spent in the solver, I'd say it's still worthwhile.
To profile where the time was going, I used the technique described here: https://www.youtube.com/watch?v=ey_P64E34g0
In order to exclude the solver time from the profile, I used this to save a
pstatsfile:and converted to an svg with
That produced a graph that looked like this:

So roughly half of the non-solver time was spent in
as_index(), in particular theallblock (although take the actual times with a grain of salt, because the profiler introduces some overhead)