Skip to content

Conversation

@etrepum
Copy link
Member

@etrepum etrepum commented Apr 4, 2023

Implement security hardening measures based on a source code audit of simplejson 3.18.4 by X41 D-Sec GmbH and sponsored by the OSTIF.

  • Fix invalid handling of unicode escape sequences in the pure Python
    implementation of the decoder (SJ-PT-23-01)
  • Fix missing reference count decrease if PyOS_string_to_double raises
    an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
  • Backport the integer string length limitation from Python 3.11 to
    limit quadratic number parsing (SJ-PT-23-03)
  • Fix inconsistencies with error messages between the C and Python
    implementations (SJ-PT-23-100)
  • Remove unused unichr import from encoder (SJ-PT-23-101)
  • Remove unused namedtuple_as_object and tuple_as_array arguments from
    simplejson.load (SJ-PT-23-102)
  • Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
  • Change default of allow_nan from True to False and add allow_nan
    to decoder (SJ-PT-23-107)

Several suggested improvements were not implemented in this release and will be considered in the future:

  • SJ-PT-23-104: Type Hints Not Used - Implementing type hints with annotations is not possible for as long as Python 2 is supported. Using stub files or revisiting this in the future when Python 2 support is removed will be considered in a later release.
  • SJ-PT-23-105: Deprecated Python Versions Supported - Without a way to get usage metrics, it's hard to say how many people are still using recent versions of simplejson and Python 2, so I would prefer to maintain support for a while longer.
  • SJ-PT-23-108: Support of Duplicate Key Names - I haven't had a lot of requests to provide this feature, and anyone looking to do this can implement it with object_pairs_hook. The default behavior of "last key wins" is consistent with JavaScript's JSON implementation.
  • SJ-PT-23-106: Unsigned Git Commits - PR merges were already verified since I update them with the GitHub UX but I will also start signing my commits with an SSH key. I have also enabled a tag protection rule for *. I did not enable a branch protection rule to require all commits to be signed since that would prevent accepting third party contributions without first rebasing myself.

The full public report is available here: https://www.x41-dsec.de/static/reports/X41-OSTIF-simplejson-CodeRview-2023-04-18.pdf
See also:

etrepum added 7 commits April 4, 2023 09:59
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
@etrepum etrepum enabled auto-merge April 6, 2023 16:53
@etrepum etrepum merged commit 1a4995d into master Apr 6, 2023
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 7, 2023
Version 3.19.1 released 2023-04-06

* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
kraj pushed a commit to YoeDistro/meta-openembedded that referenced this pull request Apr 10, 2023
Changelog:
============
* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)

Signed-off-by: Wang Mingyu <[email protected]>
Signed-off-by: Khem Raj <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants