Skip to content

Skip the subset check in as_index() (in some cases)#1515

Merged
lkstrp merged 4 commits intoPyPSA:masterfrom
chriselion:celion/speedup-as_index
Jan 6, 2026
Merged

Skip the subset check in as_index() (in some cases)#1515
lkstrp merged 4 commits intoPyPSA:masterfrom
chriselion:celion/speedup-as_index

Conversation

@chriselion
Copy link
Copy Markdown
Contributor

@chriselion chriselion commented Jan 6, 2026

Closes # (if applicable): n/a

Changes proposed in this Pull Request

I was doing some basic profiling, and it looked like a decent portion of the setup time was being spent in this part of the code:

PyPSA/pypsa/common.py

Lines 275 to 280 in 59a7d5d

if force_subset and not all(val in n_attr for val in values_):
msg = (
f"Values must be a subset of the network attribute "
f"'{network_attribute}'. Pass force_subset=False to disable this check."
)
raise ValueError(msg)

In the case the input values is None, since we set values_ = n_attr, the subset check is redundant and we can skip that block.

(more timing information below)

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in docs. (n/a it's a one-line change with a comment)
  • Unit tests for new features were added (if applicable). (n/a)
  • A note for the release notes docs/release-notes.md of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

I used this to measure the total time:

from pypsa.networks import Network
import time

PATH = 'examples/networks/model-energy/model-energy.nc'

def main():

    start_time = time.monotonic()
    n = Network(PATH)
    n.optimize(solver_name="highs", log_to_console=False)
    
    end_time = time.monotonic()
    print(f"elapsed: {end_time - start_time}")

if __name__ == "__main__":
    main()

Running before and after the change 5 times each, the best times were:
before: elapsed: 7.80225883400999
after: elapsed: 7.512157290999312

So it's not a huge difference overall, but since most of that time is spent in the solver, I'd say it's still worthwhile.


To profile where the time was going, I used the technique described here: https://www.youtube.com/watch?v=ey_P64E34g0
In order to exclude the solver time from the profile, I used this to save a pstats file:

pr = cProfile.Profile()
pr.enable()
def stop_profile(n, snapshots):
    pr.disable()
n.optimize(solver_name="highs", log_to_console=False, extra_functionality=stop_profile)
pr.dump_stats("model-energy.optimize.pstats")

and converted to an svg with

gprof2dot model-energy.optimize.pstats -n 5 | dot -Tsvg -o model-energy.optimize.svg

That produced a graph that looked like this:
image
So roughly half of the non-solver time was spent in as_index(), in particular the all block (although take the actual times with a grain of salt, because the profiler introduces some overhead)

Copy link
Copy Markdown
Member

@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into profiling @chriselion ! There are probably tons of other things to improve. The entire as_index helper isn't implemented very well and is called too often. I'll ref #1487 to clean this up further

We are happy to receive PRs if you find any other things!

@lkstrp lkstrp enabled auto-merge (squash) January 6, 2026 09:06
@chriselion
Copy link
Copy Markdown
Contributor Author

@lkstrp Thanks, I'll have a look at that other issue later. I'm still learning my way around the code (and don't know pandas very well) but hopefully I can find something to improve.

Are there any larger examples that would be worth examining? examples/networks/model-energy looked like it was the slowest.

@lkstrp lkstrp merged commit 788d7e3 into PyPSA:master Jan 6, 2026
26 of 27 checks passed
@chriselion chriselion deleted the celion/speedup-as_index branch January 6, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants