Skip to content

Networks statistics (i)plot refactor and new sanitize functions#1401

Merged
lkstrp merged 41 commits intomasterfrom
networks-iplot
Jan 15, 2026
Merged

Networks statistics (i)plot refactor and new sanitize functions#1401
lkstrp merged 41 commits intomasterfrom
networks-iplot

Conversation

@FabianHofmann
Copy link
Copy Markdown
Contributor

Enhances interactive and non-interactive plotting functionality for network collections and stochastic networks, with automatic carrier management.

Key Features:

  • New add_missing_carriers() method: Automatically discovers and adds carriers used in components with optional color palette assignment (defaults to matplotlib's tab10)
  • Error bars for stochastic networks: Interactive bar plots now aggregate scenarios and display standard deviation as error bars
  • Improved collection plotting: Bar charts properly handle multi-level indices with automatic grouping/faceting based on network structure
  • Multi-period support: Better handling of investment periods in bar charts with automatic stacking

How to check

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in docs.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes docs/release-notes.md of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

FabianHofmann and others added 7 commits October 7, 2025 13:43
- Extract carrier names from MultiIndex in stochastic networks
- Automatically replicate new carriers across all scenarios
- Preserve MultiIndex structure when concatenating carriers
- Add comprehensive unit tests for stochastic network support
- Update stochastic_network() example to set carriers and call add_missing_carriers()
Detect stochastic scenarios and adjust plotting behavior to ensure seaborn
automatically aggregates scenarios with error bars. Collection indices are
used for faceting instead of color when scenarios are present. This maintains
error bar functionality while supporting NetworkCollection plotting.
Stochastic networks now display error bars in plotly interactive bar plots.
Scenarios are automatically aggregated (mean) and standard deviations are
shown as error bars. Detection ensures only true stochastic networks are
aggregated, not collections with indices named 'scenario'.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@FabianHofmann
Copy link
Copy Markdown
Contributor Author

FabianHofmann commented Oct 15, 2025

Test script:

# %%
import pypsa
import matplotlib.pyplot as plt
import pandas as pd

# Set to True to run only iplot calls, False to run only plot calls
RUN_INTERACTIVE = True

# Directory to save figures
OUTPUT_DIR = "dev-scripts/plots"
import os
os.makedirs(OUTPUT_DIR, exist_ok=True)

# # Network collection of standard networks
n1 = pypsa.examples.ac_dc_meshed()
n2 = pypsa.examples.ac_dc_meshed()
n2.loads_t.p_set *= 2
n1.optimize()
n2.optimize()

# single indexed
index = pd.Index(["normal", "double"], name="a")
nc = pypsa.NetworkCollection([n1, n2], index=index)

if not RUN_INTERACTIVE:
    fig, ax, g = nc.statistics.energy_balance.plot.bar()
    plt.suptitle("Energy Balance - Network Collection (Single Indexed)", fontsize=14, y=1.02)
    plt.savefig(os.path.join(OUTPUT_DIR, "network_collection_single_indexed.png"), dpi=300, bbox_inches='tight')
if RUN_INTERACTIVE:
    fig = nc.statistics.energy_balance.iplot.bar(title="Energy Balance - Network Collection (Single Indexed)")
    fig.write_html(os.path.join(OUTPUT_DIR, "network_collection_single_indexed.html"))

# multi-indexed
index = pd.MultiIndex.from_product([["normal", "double"], ["normal", "double"]], names=["a", "b"])
nc = pypsa.NetworkCollection([n1, n2, n1, n2], index=index)

if not RUN_INTERACTIVE:
    fig, ax, g = nc.statistics.energy_balance.plot.bar()
    plt.suptitle("Energy Balance - Network Collection (Multi-Indexed)", fontsize=14, y=1.02)
    plt.savefig(os.path.join(OUTPUT_DIR, "network_collection_multi_indexed.png"), dpi=300, bbox_inches='tight')
if RUN_INTERACTIVE:
    fig = nc.statistics.energy_balance.iplot.bar(title="Energy Balance - Network Collection (Multi-Indexed)")
    fig.write_html(os.path.join(OUTPUT_DIR, "network_collection_multi_indexed.html"))

# Multi invest networks
nm = pypsa.examples.ac_dc_meshed()
nm.investment_periods = [1, 2]
nm.optimize()

if not RUN_INTERACTIVE:
    fig, ax, g = nm.statistics.energy_balance.plot.bar()
    plt.suptitle("Energy Balance - Multi-Period Investment Network", fontsize=14, y=1.02)
    plt.savefig(os.path.join(OUTPUT_DIR, "multi_invest_network.png"), dpi=300, bbox_inches='tight')
if RUN_INTERACTIVE:
    fig = nm.statistics.energy_balance.iplot.bar(title="Energy Balance - Multi-Period Investment Network")
    fig.write_html(os.path.join(OUTPUT_DIR, "multi_invest_network.html"))


# Stochastic Network

ns = pypsa.examples.stochastic_network()
ns.optimize()

if not RUN_INTERACTIVE:
    fig, ax, g = ns.statistics.energy_balance.plot.bar()
    plt.suptitle("Energy Balance - Stochastic Network", fontsize=14, y=1.02)
    plt.savefig(os.path.join(OUTPUT_DIR, "stochastic_network.png"), dpi=300, bbox_inches='tight')
if RUN_INTERACTIVE:
    fig = ns.statistics.energy_balance.iplot.bar(title="Energy Balance - Stochastic Network")
    fig.write_html(os.path.join(OUTPUT_DIR, "stochastic_network.html"))


# Network collection of multi stochastic networks
nc = pypsa.NetworkCollection([ns, ns], index=pd.Index(["normal", "double"], name="a"))

if not RUN_INTERACTIVE:
    fig, ax, g = nc.statistics.energy_balance.plot.bar()
    plt.suptitle("Energy Balance - Network Collection of Stochastic Networks", fontsize=14, y=1.02)
    plt.savefig(os.path.join(OUTPUT_DIR, "network_collection_stochastic.png"), dpi=300, bbox_inches='tight')
if RUN_INTERACTIVE:
    fig = nc.statistics.energy_balance.iplot.bar(title="Energy Balance - Network Collection of Stochastic Networks")
    fig.write_html(os.path.join(OUTPUT_DIR, "network_collection_stochastic.html"))
multi_invest_network network_collection_multi_indexed network_collection_single_indexed network_collection_stochastic stochastic_network

FabianHofmann and others added 2 commits October 15, 2025 17:23
- Fix 15/20 failing tests by accounting for AC carrier from buses
- Add NetworkCollection.has_scenarios property for consistent API
- Simplify stochastic carrier wrapping logic in add_missing_carriers()
- Remove duplicated scenario checking code, use native has_scenarios
- Add release notes for new features
- Clean up .gitignore duplicates and add development files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Use self.n_save.components for safer component iteration
- Change logger level from info to debug for no missing carriers
- Move generate_colors to common.py as standalone function
- Update tests to use common.generate_colors
- Simplify _get_investment_periods to use n.periods directly
- Use n._index_names directly (available on both Networks and Collections)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@fneum
Copy link
Copy Markdown
Member

fneum commented Oct 15, 2025

Strong support for automatic carrier management!

FabianHofmann and others added 18 commits October 15, 2025 23:17
- Add Network.unique_carriers property to collect all carriers from components
- Create Carriers.assign_colors() method for flexible color assignment
- Refactor add_missing_carriers() to use new assign_colors() method
- Simplify API: replace assign_colors/color_palette params with single palette param
- Update all tests to match new API (palette instead of assign_colors/color_palette)

Addresses review comments #1 and #3:
- Issue #1: Create separate assign_colors() method instead of parameters
- Issue #3: Add c.unique_carriers property similar to c.ports

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Addresses review comment #10 (lkstrp at plotter.py:759) about reducing boilerplate
in statistics plotting code. This change consolidates ~110 lines of duplicated logic
between StatisticPlotter and StatisticInteractivePlotter.

Changes:
- Created _StatisticPlotterBase base class with shared _prepare_chart_data() method
- Both StatisticPlotter and StatisticInteractivePlotter now inherit from this base
- Moved common logic for parameter validation, schema application, and data
  preparation into the base class method
- Fixed _get_investment_periods() to handle NetworkCollection gracefully by
  catching NotImplementedError when accessing the periods attribute

Benefits:
- Reduces code duplication and maintenance burden
- Ensures consistency between static and interactive plotters
- Makes future changes easier to implement in one place
- Improves code readability

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Add unique_carriers as a property on the Components base class, following
the established pattern for component-related properties like ports.

Changes:
- Added unique_carriers property to pypsa.components.components.Components
- API: `n.c.<component>.unique_carriers` (e.g., `n.c.generators.unique_carriers`)
- Updated carriers.py to iterate over all components and collect unique carriers
- Updated release notes to document the new API location

Benefits:
- Follows consistent API pattern similar to ports property
- Each component can report its own unique carriers
- More modular and intuitive API design

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Refactor get_carrier_colors and get_carrier_labels to reduce code duplication
by extracting common logic into a helper method _get_carrier_attribute.

Changes:
- Added _get_carrier_attribute() helper method to extract carrier attributes
  with optional nice name mapping
- Simplified get_carrier_colors() to use the helper method
- Simplified get_carrier_labels() with clearer logic
- Added comprehensive docstrings to all methods
- Added explanatory comments for stochastic network handling

Benefits:
- Reduces code duplication (~15 lines)
- Makes the code more maintainable and easier to extend
- Provides a pattern that could be generalized for other component attributes
- Clearer separation of concerns

The helper method could potentially be extended in the future to work with
other component types beyond carriers, making it a foundation for more
general component attribute access patterns.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Add a `unique` property to the Carriers class that handles extracting unique
carrier names from both regular and stochastic networks. This eliminates
repeated conditional logic throughout the codebase.

Changes:
- Added `unique` property to pypsa.components._types.carriers.Carriers
- Returns unique carrier names, handling MultiIndex for stochastic networks
- Updated assign_colors() to use self.unique (2 usages simplified)
- Updated add_missing_carriers() to use self.unique (1 usage simplified)

Benefits:
- Eliminates 3 instances of if/else blocks checking for scenarios
- Makes code more readable and maintainable
- Provides a single source of truth for accessing unique carrier names
- Reduces coupling to internal index structure

Example usage:
>>> n.c.carriers.unique
Index(['AC', 'gas', 'solar', 'wind'], dtype='object', name='name')

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add timestep-to-snapshot mapping for multi-period networks in area/line plots
- Fix artificial trace generation for stacked area plots with proper array dimensions
- Extend filtered columns to include scenario and period for proper grouping
- Add auto-faceting for multi-period, stochastic, and collection networks
- Add missing `import pypsa` statements to doctest examples
- Mark illustrative examples with `# doctest: +SKIP` where appropriate
- Add `# doctest: +NORMALIZE_WHITESPACE` to handle pandas DataFrame formatting
- Update expected output in add_missing_carriers examples to include 'AC' carrier

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Add tests for previously untested functionality to improve code coverage:

- Test assign_colors method with various scenarios:
  - Basic color assignment
  - Specific carrier selection
  - Different color palettes
  - Overwrite behavior
  - Stochastic networks

- Test unique_carriers property:
  - Empty components
  - Components with and without carrier attribute
  - Filtering empty strings and NaN values

- Test has_scenarios property for NetworkCollection:
  - Collections without scenarios
  - Collections with mixed networks
  - Collections with only stochastic networks

- Test error handling for conflicting arguments in add_missing_carriers

All 32 tests pass successfully.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Add tests to achieve 100% coverage on pypsa/components/_types/carriers.py:

- Test assign_colors with single string carrier name (covers line 127)
- Test add_missing_carriers on stochastic network without palette
  (covers branch 295->300 with palette=None case)

All 33 carrier tests pass. Carriers module now has 100% line and branch coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@lkstrp lkstrp added this to the v1.1 milestone Nov 25, 2025
@bobbyxng
Copy link
Copy Markdown
Contributor

This is amazing, thanks @FabianHofmann and @lkstrp, I tested this PR in my notebooks, it fixes the errors I also experienced when using n.statistics() on stochastic networks. What's still missing to merge this into master? :)

Happy holidays!

@lkstrp
Copy link
Copy Markdown
Member

lkstrp commented Dec 17, 2025

This is amazing, thanks @FabianHofmann and @lkstrp, I tested this PR in my notebooks, it fixes the errors I also experienced when using n.statistics() on stochastic networks. What's still missing to merge this into master? :)

Happy holidays!

Great! Just some clean up and tests. I haven't been rushing this since it goes in v1.1 and we are still firing v1.0 patches on the master.

lkstrp and others added 2 commits January 8, 2026 13:37
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@lkstrp
Copy link
Copy Markdown
Member

lkstrp commented Jan 8, 2026

@FabianHofmann I fixed some bugs which I found via the pypsa-app and changed a couple of things. Specially the API:

- n.sanitize: Run the following methods to fix consistency issues.
- n.components.buses.add_missing_buses: Add buses referenced by components but not yet defined.
- n.components.carriers.add_missing_carriers: Add carriers used by components but not yet defined.
- n.components.carriers.assign_colors: Assign colors to carriers using matplotlib palettes.
- c.unique_carriers: Get all unique carrier values for a component.

Also add_missing_carriers does not also assign colors anymore, you need to call two functions. I hope you agree with the changes

@fneum Could you have a look, specially on the newly introduced API?

@lkstrp lkstrp requested a review from fneum January 8, 2026 14:30
@FabianHofmann
Copy link
Copy Markdown
Contributor Author

@lkstrp wonderful, looked through your changes and like the cleanup.

Copy link
Copy Markdown
Member

@fneum fneum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Only optional stuff, mainly the idea to rename assign_colors to add_missing_colors to align with the other sanitize functions.

@lkstrp lkstrp changed the title Networks statitics (i)plot refactor Networks statistics (i)plot refactor and new sanitize functions Jan 15, 2026
@lkstrp lkstrp merged commit f44ced7 into master Jan 15, 2026
4 of 5 checks passed
@lkstrp lkstrp deleted the networks-iplot branch January 15, 2026 17:14
@github-project-automation github-project-automation bot moved this from Todo to Done in PyPSA - Roadmap Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants