CVE-2023-50980 + CVE-2023-50981 defence-in-depth hardening - #22
Merged
Conversation
The DER decoder for F(2^m) curve parameters previously accepted invalid reduction polynomial exponents, and also allowed m values up to 2^32. That meant malformed input could reach PolynomialMod2::Trinomial or Pentanomial, where the runtime checks are deliberately relaxed for ECIES<EC2N> compatibility. Separately, m was unbounded and went straight into PolynomialMod2's bit-vector allocation. Move the validation to the DER boundary by requiring: - trinomial: 0 < t1 < m - pentanomial: 0 < t1 < t2 < t3 < m - field degree m <= MAX_GF2N_FIELD_DEGREE, currently 4096 The cap still covers B-571 with plenty of headroom. Adds hardening tests for invalid ordering, invalid bounds, oversized m, SECT233R1's reduction polynomial, and the m=4096 boundary.
The Rabin private-key DER decoder previously checked m_p and m_q for primality with CRYPTOPP_ASSERT, which is compiled out in release builds. A non-prime m_p or m_q could then reach CalculateInverse, where ModularSquareRoot's Jacobi-search loop can spin indefinitely on a non-prime modulus. Promote the checks to runtime BERDecodeError throws. The DER itself parsed cleanly, so BERDecodeError is the right exception, not InvalidArgument. CalculateInverse keeps its defensive CRYPTOPP_ASSERT as a double-check. Cost: one IsPrime call per private-key load (~500ms on a 2048-bit modulus). This is on a key-load path, not a hot one. Adds tests for non-prime m_p, non-prime m_q, and a valid round-trip through rabi1024.dat.
ModularSquareRoot uses Tonelli-Shanks, which assumes a prime modulus. Its non-residue search and outer Tonelli-Shanks loop were both unbounded. On a non-prime p, either loop could spin indefinitely, while the CRYPTOPP_ASSERT(IsPrime(p)) guard is compiled out in release builds. Cap both loops at MAX_MODULAR_SQRT_ITERATIONS = 10000 and throw InvalidArgument when the cap is exceeded. For a prime p, this leaves plenty of headroom for the expected O(log log p) non-residue search and O((log p)^2) Tonelli-Shanks step. The inner do-while in the Tonelli-Shanks step is already self-bounded by m == r returning early, so it does not need its own cap. Adds tests for p = 9 hitting the cap and p = 17 covering the normal Tonelli-Shanks path.
Coralesoft
force-pushed
the
feature/cve-hardening
branch
from
May 2, 2026 10:29
3e6b530 to
86fd055
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #21. The published PoCs for these CVEs were already mitigated in 2025.11.0, but reviewing the same code paths turned up a few places where malformed input could still get further than it should before being rejected.
I do not believe these are exploitable based on the current code paths, but they are still worth tightening up.
This PR includes three commits:
BERDecodeGF2NP (CVE-2023-50980)
Validates the trinomial / pentanomial ordering at decode time instead of assuming it is already sane:
0 < t1 < m0 < t1 < t2 < t3 < mAlso caps the field degree at
MAX_GF2N_FIELD_DEGREE = 4096.That is already well beyond realistic
F(2^m)use in this code, but if a legitimate use case needs more, the cap can be revisited.InvertibleRabinFunction::BERDecode (CVE-2023-50981)
The
IsPrimechecks onm_pandm_qwere debug-only asserts, so they did not protect release builds.This promotes those checks to runtime
BERDecodeErrorvalidation.ModularSquareRoot iteration cap (CVE-2023-50981)
Bounds the non-residue search and the outer Tonelli-Shanks loop at
10000iterations.If the cap is reached,
InvalidArgumentis thrown. For valid prime moduli, this should not get close to the cap in normal use.Test plan
cryptest.exe v 21andcryptest.exe v 24cleanCloses #21