Adapt to msgpack 1.2.0 - #3167
Merged
Merged
Conversation
phoebusm
force-pushed
the
adapt_to_msgpack_1.2.0
branch
from
June 11, 2026 16:28
d833ac2 to
ad918ea
Compare
phoebusm
marked this pull request as ready for review
June 11, 2026 17:13
Contributor
ArcticDB Code Review SummaryReviewed the full diff ( No blocking issues found. PR Title & Description
|
IvoDD
approved these changes
Jun 12, 2026
IvoDD
left a comment
IvoDD
left a comment
Collaborator
There was a problem hiding this comment.
Can you only add no-release-notes label as this is not user facing.
| # DEFAULT_RECURSE_LIMIT broke when msgpack 1.2.0 raised it to 1024, letting writes through at depths the | ||
| # reader could not reconstruct. | ||
| # https://man312219.monday.com/boards/7852509418/pulses/12254825163 | ||
| DEFAULT_RECURSE_LIMIT = min(511, _DEFAULT_RECURSE_LIMIT) |
Collaborator
There was a problem hiding this comment.
I find this explanation confusing. The recursion limit has always been 1000
As far as I understand the problem is that our metastruct reader uses 3 stackframe recursion so we reach the limit at around 330.
Should we just set the threshold for nesting to something like (DEFAULT_RECURSION_LIMIT - 10) // 3? The 10 is for the stack frames which call the decoding code.
After we fix our metastruct reading to not be recursive we will be able to return it to the (DEFAULT_RECURSION_LIMIT - 10) // 2
Collaborator
There was a problem hiding this comment.
Anyway I'm fine with the current fix to unblock the CI and it essentially preserves the old limit.
Collaborator
Author
There was a problem hiding this comment.
I have updated the comment to better explain the write and read limits
phoebusm
force-pushed
the
adapt_to_msgpack_1.2.0
branch
from
June 12, 2026 14:00
cfaba39 to
0b6a573
Compare
IvoDD
approved these changes
Jun 12, 2026
IvoDD
added a commit
that referenced
this pull request
Jun 17, 2026
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> Fix `test_deep_nesting_metastruct_size_over_limit` and `test_deep_nesting_metastruct_size_under_limit` Culprit: `msgpack 1.2.0` has just released. It includes [Raise DEFAULT_RECURSE_LIMIT from 511 to 1024 by Copilot · Pull Request #676 · msgpack/msgpack-python](msgpack/msgpack-python#676) which changes `DEFAULT_RECURSE_LIMIT` from 511 to 1024 `python 3.9` is unaffected as the new msgpack version has dropped the support #### What does this implement or fix? Hard-pin the arcticdb limit to `511` as msgpack <1.2.0 #### Any other comments? msgpack <1.2.0 has the recurse limit set at 511. arcticdb imports the limit from msgpack to calculate the metastruct layer limit msgpack 1.2.0 has raised limit from 511 to 1024. Since reconstructing the metastruct from is done by recursion, increasing the no. of layers in the metastruct will make arcticdb hit the python recursion limit. #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> Co-authored-by: Phoebus Mak <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Fix
test_deep_nesting_metastruct_size_over_limitandtest_deep_nesting_metastruct_size_under_limitCulprit:
msgpack 1.2.0has just released. It includes Raise DEFAULT_RECURSE_LIMIT from 511 to 1024 by Copilot · Pull Request #676 · msgpack/msgpack-pythonwhich changes
DEFAULT_RECURSE_LIMITfrom 511 to 1024python 3.9is unaffected as the new msgpack version has dropped the supportWhat does this implement or fix?
Hard-pin the arcticdb limit to
511as msgpack <1.2.0Any other comments?
msgpack <1.2.0 has the recurse limit set at 511. arcticdb imports the limit from msgpack to calculate the metastruct layer limit
msgpack 1.2.0 has raised limit from 511 to 1024. Since reconstructing the metastruct from is done by recursion, increasing the no. of layers in the metastruct will make arcticdb hit the python recursion limit.
Checklist
Checklist for code changes...