Skip to content

Commit 6cd8320

Browse files
committed
test: robustness on Python 3.14 / 3.14t CI (UnicodeDecodeError + timing flake)
Two version-specific CI failures, both in tests (no production change): - test_path_traversal_security.py: on Python 3.14 (Windows) url2pathname raises UnicodeDecodeError while decoding the overlong-UTF-8 ``%c0%af`` payload, where <3.14 replaces the bytes and returns a literal not-found name (LookupError). Either way the payload is rejected -- never a leak -- so accept both outcomes: the sentinel property test swallows any find() exception, and the double-encoded test asserts (LookupError, UnicodeDecodeError) for the overlong case while still pinning the valid ``%252f`` case to LookupError. - test_quadratic_dos.py: the SyllableTokenizer linearity ratio test flaked on the free-threaded 3.14t runner -- a 4000-rep base tokenize measured ~0.01s, below timer resolution, so a noisy fast baseline made a linear run look super-linear. Use a larger base (8000/32000) so the measurement is well above timer noise and take the median of 3 runs; an 8x threshold still cleanly separates linear (~4x) from the pre-fix O(n^2) (~16x). Verified stable across repeated runs.
1 parent e965330 commit 6cd8320

2 files changed

Lines changed: 34 additions & 13 deletions

File tree

‎nltk/test/unit/test_path_traversal_security.py‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,10 @@ def test_no_payload_reads_a_sentinel_outside_the_data_root():
149149
for p in payloads:
150150
try:
151151
ptr = D.find(p)
152-
except (ValueError, LookupError):
152+
except Exception:
153+
# Any exception (ValueError guard, LookupError not-found, or a
154+
# UnicodeDecodeError from decoding overlong %c0%af on Python 3.14+)
155+
# means the payload did not resolve to a readable file -- no leak.
153156
continue
154157
try:
155158
content = ptr.open().read()
@@ -175,13 +178,16 @@ def test_double_encoded_stays_literal_in_root():
175178
saved = list(D.path)
176179
D.path.insert(0, dataroot)
177180
try:
178-
for p in (
179-
"..%252fsecret.txt",
180-
"..%c0%afsecret.txt",
181-
"..%252f..%252fsecret.txt",
182-
):
181+
# Valid percent-encoding: url2pathname single-decodes -> literal in-root
182+
# name that does not exist -> LookupError (never a separator/traversal).
183+
for p in ("..%252fsecret.txt", "..%252f..%252fsecret.txt"):
183184
with pytest.raises(LookupError):
184-
D.find(p) # literal in-root name that does not exist -> not found
185+
D.find(p)
186+
# Overlong-UTF-8 ``%c0%af`` never decodes to "/". On Python <3.14 unquote
187+
# replaces the bytes (-> literal in-root -> LookupError); on 3.14+ unquote
188+
# raises UnicodeDecodeError. Either way it is rejected, never a leak.
189+
with pytest.raises((LookupError, UnicodeDecodeError)):
190+
D.find("..%c0%afsecret.txt")
185191
finally:
186192
D.path[:] = saved
187193

‎nltk/test/unit/test_quadratic_dos.py‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -231,15 +231,30 @@ def test_low_vowel_run_is_linear_and_unbounded(self):
231231
def test_direction1_vowelless_syllables_are_linear(self, monkeypatch):
232232
# Pre-patch: `validate_syllables` rebuilt the whole list
233233
# (`valid_syllables[:-1] + [...]`) for every vowelless syllable, so a
234-
# token like 'aebcd'*n cost O(n^2) (8000 rep = 0.77s, quadratic curve).
235-
# In-place merge makes it linear. Guard is lifted so the loop actually
236-
# runs on the large input. A 4x-input ratio test tolerates the constant.
234+
# token like 'aebcd'*n cost O(n^2). In-place merge makes it linear.
235+
#
236+
# This is a wall-clock ratio test, so it must be robust to CI timing
237+
# noise: (1) a base size large enough that a single tokenize is well
238+
# above timer resolution (a 4000-rep base measured ~0.01s -- noise
239+
# dominated -- and made the ratio explode on a free-threaded runner); and
240+
# (2) the MEDIAN of a few runs, robust to both a fast and a slow outlier.
241+
# Linear -> t4 ~ 4*t1; the pre-fix O(n^2) -> t4 ~ 16*t1, so an 8x
242+
# threshold separates them with wide margin.
243+
import statistics
244+
237245
from nltk.tokenize import SyllableTokenizer
238246

239247
monkeypatch.setattr(SyllableTokenizer, "MAX_TOKEN_LEN", 10**9)
240-
t1 = _elapsed(lambda: SyllableTokenizer().tokenize("aebcd" * 4000))
241-
t4 = _elapsed(lambda: SyllableTokenizer().tokenize("aebcd" * 16000))
242-
assert t4 < 10 * t1 + 0.5
248+
249+
def _median(n):
250+
return statistics.median(
251+
_elapsed(lambda: SyllableTokenizer().tokenize("aebcd" * n))
252+
for _ in range(3)
253+
)
254+
255+
t1 = _median(8000)
256+
t4 = _median(32000) # 4x the input
257+
assert t4 < 8 * t1 + 0.5
243258

244259
def test_direction2_giant_multivowel_token_is_bounded(self):
245260
# A multi-vowel token passes the early return and reaches the O(n)

0 commit comments

Comments
 (0)