Skip to content

Support headerless format in getFrameParameters and decompress #217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
swehner wants to merge 4 commits into from
Closed

Support headerless format in getFrameParameters and decompress #217

swehner wants to merge 4 commits into from

Conversation

@swehner
Copy link
Contributor

@swehner swehner commented Mar 16, 2024 •
edited
Loading

Adding support for magic-less format in a few functions:

  • Add optional paramater format to get_frame_paramaters and switch to using getFrameHeader_advanced internally
  • Use getFrameHeader_advanced in decompress instead of getFrameContentSize

Testing done:

$ export ZSTD_SLOW_TESTS=1  

$ for i in cext rust cffi; do echo "=== PYTHON_ZSTANDARD_IMPORT_POLICY=$i python setup.py test"; PYTHON_ZSTANDARD_IMPORT_POLICY=$i python setup.py test; done 2>&1 | grep "\(==\|OK\|Ran\|failed\|ERROR\)"
=== PYTHON_ZSTANDARD_IMPORT_POLICY=cext python setup.py test
Ran 292 tests in 13.942s
OK
=== PYTHON_ZSTANDARD_IMPORT_POLICY=rust python setup.py test
Ran 292 tests in 15.512s
OK
=== PYTHON_ZSTANDARD_IMPORT_POLICY=cffi python setup.py test
Ran 292 tests in 20.358s
OK (skipped=26)

swehner
swehner commented Mar 16, 2024
View reviewed changes
tests/test_compressor_fuzzing.py

for i, frame in enumerate(result):
self.assertEqual(dctx.decompress(frame), original[i])
self.assertEqual(dctx.decompress(frame.tobytes()), original[i])
Copy link
Contributor Author

@swehner swehner Mar 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the rust backend the test was erroring out becase frame wasn't a buffer but a zstandard.backend_rust.BufferSegment - not sure what the best way to fix it is...
This does work, so I added the conversion

@swehner swehner changed the title Add format to get frame params Support headerless format in getFrameParameters and decompress Mar 16, 2024
swehner
swehner commented Mar 16, 2024
View reviewed changes
.cargo/config
Comment on lines +4 to +5
[target.aarch64-apple-darwin]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]
Copy link
Contributor Author

@swehner swehner Mar 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For compiling on M1

@swehner swehner mentioned this pull request Mar 16, 2024
Closed
indygreg
indygreg approved these changes Aug 17, 2025
View reviewed changes
Copy link
Owner

@indygreg indygreg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the feature addition!

I'm going to cherry pick this change locally, fix the bit rot, then push it for you. The PR will be marked as closed but it will be merged in effect. This will be in a release this weekend unless I run into trouble with it.

@indygreg indygreg closed this in 7312fae Aug 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@indygreg indygreg indygreg approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@swehner @indygreg