Skip to content

feat: pickling of SolutionArray objects (fixes #543)#1964

Merged
speth merged 7 commits intoCantera:mainfrom
mruijzendaal:feature/pickle-solutionarray
Sep 7, 2025
Merged

feat: pickling of SolutionArray objects (fixes #543)#1964
speth merged 7 commits intoCantera:mainfrom
mruijzendaal:feature/pickle-solutionarray

Conversation

@mruijzendaal
Copy link
Copy Markdown
Contributor

@mruijzendaal mruijzendaal commented Sep 5, 2025

Changes proposed in this pull request

  • Implement __reduce__ and __copy__ methods for SolutionArray, such that it is picklable and copyable.

If applicable, fill in the issue number this pull request is fixing

Fixes #543, continuation of #1426, see cantera/enhancements/#162

If applicable, provide an example illustrating new features this pull request is introducing

In short, this pull request enables pickling and copying of SolutionArray objects with YAML as a proxy:

import cantera as ct
import pickle
import copy

sol = ct.Solution('gri30.yaml')  # load GRI-Mech 3.0 mechanism
solarr = ct.SolutionArray(sol, 100)  # create a solution array to store results

with open('sol_saved.pkl', 'wb') as f:
    pickle.dump(solarr, f)

with open('sol_saved.pkl', 'rb') as f:
    solarr_loaded = pickle.load(f)

copy.copy(solarr_loaded)

Checklist

  • The pull request includes a clear description of this code change
  • Commit messages have short titles and reference relevant issues
  • Build passes (scons build & scons test) and unit tests address code coverage
  • Style & formatting of contributed code follows contributing guidelines
  • The pull request is ready for review

(Edit: format proposed changes)

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 5, 2025

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.05%. Comparing base (4b6712f) to head (0386a7d).
⚠️ Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
interfaces/cython/cantera/composite.py 95.83% 1 Missing ⚠️
src/base/SolutionArray.cpp 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1964      +/-   ##
==========================================
+ Coverage   74.99%   75.05%   +0.06%     
==========================================
  Files         450      450              
  Lines       56237    56259      +22     
  Branches     9299     9302       +3     
==========================================
+ Hits        42175    42226      +51     
+ Misses      10930    10900      -30     
- Partials     3132     3133       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member

@ischoegl ischoegl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mruijzendaal ... thank you for the PR! I have a couple of minor comments.

Regarding temporary files, I confirmed that the following works (at least on macOS), which should simplify things somewhat.

temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
    yaml_data = fid.read()

PS: In addition, feel free to add yourself to the AUTHORS.md file.

@ischoegl ischoegl added the Python label Sep 6, 2025
@mruijzendaal
Copy link
Copy Markdown
Contributor Author

mruijzendaal commented Sep 7, 2025

Thanks for the review and feedback!

Regarding temporary files, I confirmed that the following works (at least on macOS), which should simplify things somewhat.

temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
    yaml_data = fid.read()

That's weird, copying your code verbatim doesn't work on my system (MacOS 15.6.1, Python 3.11.10, cantera 3.1.0)

import tempfile
import cantera as ct

sol = ct.Solution('gri30.yaml') 
solarr = ct.SolutionArray(sol, 10)

temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
    yaml_data = fid.read()

and gives the following error

CanteraError: 
*******************************************************************************
InputFileError thrown by AnyMap::fromYamlFile:
Error on line 0 of /var/folders/q9/b2jlk94n25n5w5bjvp02rsc00000gn/T/tmp1tcb07ez.yaml:
bad conversion
|  Line |
*******************************************************************************

However, with an os.unlink added back in, it works:

import tempfile
import cantera as ct
+ import os

sol = ct.Solution('gri30.yaml') 
solarr = ct.SolutionArray(sol, 10)

temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
+ os.unlink(temp_file.name)
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
    yaml_data = fid.read()

I'll make the necessary changes implementing your requests.

@ischoegl
Copy link
Copy Markdown
Member

ischoegl commented Sep 7, 2025

@mruijzendaal ... looks like there was an issue on my side (apparently I scrambled file names while testing; I had run into a similar error). I pushed a fix to the C++ code base for this PR that enables writing to empty files. I can now confirm that the following works:

import tempfile
import cantera as ct

sol = ct.Solution('gri30.yaml')
solarr = ct.SolutionArray(sol, 10)

with tempfile.NamedTemporaryFile(suffix='.yaml') as temp_file:
    solarr.save(temp_file.name, name='solarr', overwrite=True)
    with open(temp_file.name, 'r') as fid:
        yaml_data = fid.read()

You can verify that temp_file is gone outside the context manager.

@ischoegl ischoegl force-pushed the feature/pickle-solutionarray branch 6 times, most recently from 7257914 to 052f58d Compare September 7, 2025 19:27
@ischoegl
Copy link
Copy Markdown
Member

ischoegl commented Sep 7, 2025

@mruijzendaal … Turns out that temporary file access is more restrictive on Windows, so additional tweaks were necessary.

@bryanwweber / @speth … since I pushed to this PR also, would either of you provide a final review so this can be merged?

Copy link
Copy Markdown
Member

@bryanwweber bryanwweber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all seems fine, although I guess a function that supports a buffer would be more generally useful. Not needed here though!

@ischoegl ischoegl dismissed their stale review September 7, 2025 21:15

Pushed to this PR also - other approvals needed.

@ischoegl ischoegl force-pushed the feature/pickle-solutionarray branch from 052f58d to 0386a7d Compare September 7, 2025 21:23
@ischoegl ischoegl requested a review from bryanwweber September 7, 2025 22:00
@ischoegl
Copy link
Copy Markdown
Member

ischoegl commented Sep 7, 2025

@bryanwweber … agreed that writing to a buffer would be useful, but that won’t change the structure of the pickled object (the dictionary key names fed to the pickle are the only things we need to settle on for good). Thus, I believe that using the temporary file output is an adequate solution.

Copy link
Copy Markdown
Member

@speth speth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine to me.

@speth speth merged commit aa8a385 into Cantera:main Sep 7, 2025
50 of 51 checks passed
Copy link
Copy Markdown
Member

@bryanwweber bryanwweber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess Ray already merged this, but I thought I should mention these two very small things.

Thus, I believe that using the temporary file output is an adequate solution.

Yep, agreed

return meta

def _to_picklable(self):
temp_file = _NamedTemporaryFile(suffix=".yaml", delete=False).name
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep a reference to the original temp file object so that it doesn't get garbage collected. I guess with delete=False there's not a concern about deleting the file, but I still think it makes sense

Copy link
Copy Markdown
Member

@ischoegl ischoegl Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine as delete=False as you said. PS: reverting to the context manager (i.e., previous version) resolves this.


@classmethod
def _from_pickle(cls, state):
phase = state.get("phase") # Recreate Solution object from pickled state
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I didn't catch this the first time. What happens if phase is None? I can foresee that happening for incorrectly formed input. It may be better to check for this case and raise a useful error?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fwiw, None causes a pretty nasty crash - it needs to be caught elsewhere.

ischoegl added a commit to ischoegl/cantera that referenced this pull request Sep 8, 2025
ischoegl added a commit to ischoegl/cantera that referenced this pull request Sep 8, 2025
ischoegl added a commit to ischoegl/cantera that referenced this pull request Sep 8, 2025
ischoegl added a commit to ischoegl/cantera that referenced this pull request Sep 9, 2025
@mruijzendaal
Copy link
Copy Markdown
Contributor Author

Thanks for working with me on this one, and thanks for all the work maintaining Cantera. It is very useful for me and my colleagues at Maastricht University in The Netherlands. I hope to be able to contribute more in the coming months.

speth pushed a commit that referenced this pull request Sep 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SolutionArray save / load

4 participants