feat: pickling of SolutionArray objects (fixes #543)#1964
feat: pickling of SolutionArray objects (fixes #543)#1964speth merged 7 commits intoCantera:mainfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1964 +/- ##
==========================================
+ Coverage 74.99% 75.05% +0.06%
==========================================
Files 450 450
Lines 56237 56259 +22
Branches 9299 9302 +3
==========================================
+ Hits 42175 42226 +51
+ Misses 10930 10900 -30
- Partials 3132 3133 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Hi @mruijzendaal ... thank you for the PR! I have a couple of minor comments.
Regarding temporary files, I confirmed that the following works (at least on macOS), which should simplify things somewhat.
temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
yaml_data = fid.read()
PS: In addition, feel free to add yourself to the AUTHORS.md file.
|
Thanks for the review and feedback!
That's weird, copying your code verbatim doesn't work on my system (MacOS 15.6.1, Python 3.11.10, cantera 3.1.0) import tempfile
import cantera as ct
sol = ct.Solution('gri30.yaml')
solarr = ct.SolutionArray(sol, 10)
temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
yaml_data = fid.read()and gives the following error However, with an import tempfile
import cantera as ct
+ import os
sol = ct.Solution('gri30.yaml')
solarr = ct.SolutionArray(sol, 10)
temp_file = tempfile.NamedTemporaryFile(suffix='.yaml')
+ os.unlink(temp_file.name)
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
yaml_data = fid.read()I'll make the necessary changes implementing your requests. |
|
@mruijzendaal ... looks like there was an issue on my side (apparently I scrambled file names while testing; I had run into a similar error). I pushed a fix to the C++ code base for this PR that enables writing to empty files. I can now confirm that the following works: import tempfile
import cantera as ct
sol = ct.Solution('gri30.yaml')
solarr = ct.SolutionArray(sol, 10)
with tempfile.NamedTemporaryFile(suffix='.yaml') as temp_file:
solarr.save(temp_file.name, name='solarr', overwrite=True)
with open(temp_file.name, 'r') as fid:
yaml_data = fid.read()You can verify that |
7257914 to
052f58d
Compare
|
@mruijzendaal … Turns out that temporary file access is more restrictive on Windows, so additional tweaks were necessary. @bryanwweber / @speth … since I pushed to this PR also, would either of you provide a final review so this can be merged? |
bryanwweber
left a comment
There was a problem hiding this comment.
This all seems fine, although I guess a function that supports a buffer would be more generally useful. Not needed here though!
Pushed to this PR also - other approvals needed.
052f58d to
0386a7d
Compare
|
@bryanwweber … agreed that writing to a buffer would be useful, but that won’t change the structure of the pickled object (the dictionary key names fed to the pickle are the only things we need to settle on for good). Thus, I believe that using the temporary file output is an adequate solution. |
bryanwweber
left a comment
There was a problem hiding this comment.
I guess Ray already merged this, but I thought I should mention these two very small things.
Thus, I believe that using the temporary file output is an adequate solution.
Yep, agreed
| return meta | ||
|
|
||
| def _to_picklable(self): | ||
| temp_file = _NamedTemporaryFile(suffix=".yaml", delete=False).name |
There was a problem hiding this comment.
I think we should keep a reference to the original temp file object so that it doesn't get garbage collected. I guess with delete=False there's not a concern about deleting the file, but I still think it makes sense
There was a problem hiding this comment.
I think it is fine as delete=False as you said. PS: reverting to the context manager (i.e., previous version) resolves this.
|
|
||
| @classmethod | ||
| def _from_pickle(cls, state): | ||
| phase = state.get("phase") # Recreate Solution object from pickled state |
There was a problem hiding this comment.
Sorry I didn't catch this the first time. What happens if phase is None? I can foresee that happening for incorrectly formed input. It may be better to check for this case and raise a useful error?
There was a problem hiding this comment.
Fwiw, None causes a pretty nasty crash - it needs to be caught elsewhere.
|
Thanks for working with me on this one, and thanks for all the work maintaining Cantera. It is very useful for me and my colleagues at Maastricht University in The Netherlands. I hope to be able to contribute more in the coming months. |
Changes proposed in this pull request
__reduce__and__copy__methods forSolutionArray, such that it is picklable and copyable.If applicable, fill in the issue number this pull request is fixing
Fixes #543, continuation of #1426, see cantera/enhancements/#162
If applicable, provide an example illustrating new features this pull request is introducing
In short, this pull request enables pickling and copying of SolutionArray objects with YAML as a proxy:
Checklist
scons build&scons test) and unit tests address code coverage(Edit: format proposed changes)