Repository navigation
Add Mesh.save_npy / Mesh.load_npy for fast large-mesh I/O - #1222
Merged
Merged
Conversation
Each array is stored in its own .npy file under a directory created by save_npy. This bypasses the meshio round-trip in Mesh.save() and avoids bundling everything into a single archive as save_npz does, which matters for meshes with millions of nodes (see kinnala#1109). Round-trips p, t, named boundaries and named subdomains. allow_pickle is explicitly False everywhere. The default MeshTri/Quad/Tet/Hex/Line and their with_boundaries/with_subdomains variants are all covered by tests. Benchmark on MeshTet().refined(5) (~33k nodes, Windows, numpy 2.4.6): save_npy : 1.9 ms size=3.2 MB save_npz : 3.2 ms size=3.2 MB save vtk : 126.9 ms size=7.6 MB (meshio) Closes kinnala#1109.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Mesh.save()routes throughmeshio, andMesh.save_npz()bundles everything into one.npzarchive. Both are inconvenient for the large-mesh case: the meshio conversion is slow and produces oversized files, and a single archive forces the user to load everything even when only some arrays are needed.Proposal
Add
Mesh.save_npy(path)andMesh.load_npy(path)on the baseMeshclass.save_npywrites one.npyfile per array into a directory it creates underpath:doflocs.npy— node coordinatest.npy— element topologyb_<name>.npy— each named boundarys_<name>.npy— each named subdomainThis mirrors the approach sketched in #1109 (which uses
np.savewithallow_pickle=False) and avoids both the meshio round-trip and the.npzbundling.Implementation
save_npz/load_npzon the base class, so everyMesh*subclass inherits it unchanged.allow_pickle=Falseis passed explicitly on everynp.save/np.loadcall.b_/s_naming convention already used bysave_npz, so the two formats are conceptually aligned.load_npyrebuilds the mesh throughcls(...), so subclass identity andelemcome fromcls— no extra metadata file.Tests
Added to
tests/test_mesh.py:test_save_load_npy_cycle— parametrized overMeshTri(),MeshQuad(),MeshTet(),MeshHex(),MeshLine(), verifyingpandt.test_save_load_npy_cycle_with_tags— aMeshTri().refined(2)with two boundaries and one subdomain, verifyingboundariesandsubdomainsarrays survive the round-trip.test_load_npy_missing— confirms a missing directory raisesFileNotFoundError.All 7 new parametrized cases pass. Existing
test_mesh.py(176 tests) unaffected.Local benchmark
Windows, Python 3.11.7, numpy 2.4.6,
MeshTet().refined(5)(30481 nodes / 163840 elements):save_npyis ~46× faster and ~2.4× smaller thanMesh.save()at this scale; the gap widens on the >1M-node meshes that motivated #1109. Loading is slightly slower for.npybecause each array is a separate file, but the write path is where the issue reported pain.Notes for review
save_npy/load_npy) and the multi-file.npydesign come directly from Add save_npy/load_npy #1109. If you'd prefer the alternative of a single.npzwithnp.savez_compressedunder the same names, that's a small change — happy to iterate.save_npz/load_npzare left as-is for backwards compatibility.