Repository navigation
fix(tests): strip ANSI/wrap before asserting the #808 guard stayed silent - #1377
Conversation
…guard stayed silent (MakazhanAlpamys#1352) The negative assertions in test_default_equivalent_spellings_load_silently and test_load_config_file_default_equivalent_spelling_loads_silently substring-matched capsys's raw output. Rich's Console wraps at the render width, so at a narrow terminal (COLUMNS=50) the 'read by nothing' phrase can be split across the wrap and the assertion passes even when the warning was printed, letting a staged-field regression through silently. Wrap the captured output in the file's existing _plain() helper (ANSI- stripped, whitespace-collapsed) before the substring check, matching the other assertions in the same file that already use it. Add test_issue1352_narrow_terminal_guard.py, which simulates the regression (every staged field always reported as staged) and confirms the MakazhanAlpamys#808 guard's warn-case tests fail under it at COLUMNS=50, proving the fix restores discrimination regardless of terminal width.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you for picking this up, and for coming back after #1256.
The two-line fix is right, and complete. Routing both checks through the file's own _plain(...) is exactly what #1352 needed. It is also the correct helper: tests/conftest.py's strip_ansi would not have fixed this, because it keeps the newline inside a wrapped phrase, and _plain collapses whitespace. I checked it against the regression the issue describes (src/soup_cli/config/staged_fields.py:115, if val != default_val: changed to if True:, so default spellings warn), on the 8 cases of the two tests:
| tree | COLUMNS=50 | 80 | 120 | FORCE_COLOR=1 TERM=xterm-256color |
|---|---|---|---|---|
main + regression |
4 failed / 4 passed | 8 failed | 8 failed | 8 failed |
| this PR + regression | 8 failed | 8 failed | 8 failed | 8 failed |
| this PR, unmutated | 8 passed | 8 passed | 8 passed | 8 passed |
So the four warn cases that passed with the warning printed at 50 columns now fail, and nothing breaks unmutated. ruff check is clean on both files, and no changelog fragment is needed for a test-only change.
What needs to change is the new file, tests/test_issue1352_narrow_terminal_guard.py. Two problems, both executed:
-
It rewrites a tracked source file while the suite runs (
:41-56). For ~3-4 s,src/soup_cli/config/staged_fields.pyholds the regression, and it is restored in afinally:- While the test runs, another process that imports
soup_clifrom the same checkout gets the broken loader. I loadedsplit_thinking: 'false'from a second process during that window and it printedWarning: data.split_thinking is accepted but read by nothing; v0.77 will refuse it; the same command on the clean tree prints nothing. A parallel run (pytest-xdist, a second pytest, orsoupitself in that checkout) would see this. - A hard kill of the outer pytest (what a cancelled CI job or an out-of-memory kill does) skips the
finally. I killed it mid-run and the file stayed mutated in the working tree. read_text()/write_text()withoutnewline=""turn an LF checkout into CRLF on Windows. On an LF copy of the file, the test's own restore left 141 CRLF line endings where there were 0, so on such a checkout the restore itself leaves a modified file.
- While the test runs, another process that imports
-
It does not pin the two lines it was written for. The nested run selects
-k warn, which is 8 tests, and "returncode != 0" is met by any of them failing:- Revert only
:302: the new test passes (1 passed). Revert only:312: also passes (1 passed). Only reverting both makes it fail. - #1379, which is open and edits the same file, adds
-k warntests that fail under this regression at any width. On a merge of the two PRs, reverting both of your lines still leaves the new test green (1 passed). - It also relies on
COLUMNS=50happening to split the phrase for today's message. At 80 columns the unfixed file already fails under the regression, so there it cannot tell fixed from unfixed.
- Revert only
A static check of the file pins the same thing without touching any source file. Could you replace the content of tests/test_issue1352_narrow_terminal_guard.py with this (and keep the two-line change as it is)?
"""The #808 silence checks must not read raw captured output (#1352).
Rich wraps the loader's warning at the render width, so
``"read by nothing" not in capsys.readouterr().out`` passes while the warning
is printed whenever the line break lands inside the phrase (all four ``warn``
cases at ``COLUMNS=50``). Those checks go through ``_plain(...)``; this scan
fails if either one goes back to raw output, without editing any source file.
"""
from __future__ import annotations
import re
from pathlib import Path
_GUARDED = Path(__file__).with_name("test_issue808_staged_config_fields.py")
_RAW_NEGATIVE = re.compile(r"\bnot\s+in\s+capsys\.readouterr\(\)\.(?:out|err)\b")
_PLAIN_CHECK = '"read by nothing" not in _plain(capsys.readouterr().out)'
def test_808_negative_checks_do_not_read_raw_output() -> None:
lines = _GUARDED.read_text(encoding="utf-8").splitlines()
offenders = [
f"{_GUARDED.name}:{number}: {line.strip()}"
for number, line in enumerate(lines, start=1)
if _RAW_NEGATIVE.search(line)
]
assert not offenders, (
"a negative substring check on raw captured output passes when Rich wraps "
"the phrase; route it through _plain(...):\n" + "\n".join(offenders)
)
def test_808_negative_checks_are_still_there() -> None:
# The scan above is vacuous if the checks are deleted rather than reverted.
assert _GUARDED.read_text(encoding="utf-8").count(_PLAIN_CHECK) >= 2What I ran with it on this branch: 2 passed in 0.27 s. With both lines reverted: 2 failed. With only :302 reverted: 2 failed. With only :312 reverted: 2 failed. With both checks deleted: 1 failed. ruff check and ruff format --check are clean. On a merge with #1379 it flags that PR's new raw not in capsys.readouterr().out line, which is the same pattern coming back, and that is what it is for.
Once the file is replaced, I will re-run the same mutations on the new head.
The subprocess-based regression test mutated staged_fields.py on disk while the suite ran, risking cross-process leakage, a stuck mutation on a hard kill, and CRLF on Windows, and its -k warn selection did not pin either guarded line on its own. Replace it with a static scan of test_issue808_staged_config_fields.py that fails if either check goes back to raw captured output, per review.
|
Replaced the test file with your static scan in fc16e30 and kept the two-line fix as it is; the scan fails with either line reverted on its own and passes unmutated. |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you for the quick turnaround, and for a commit message that records why the old test went.
I re-ran everything on fc16e30, merged with current main:
tests/test_issue1352_narrow_terminal_guard.pymatches the proposed scan exactly, and the two-line_plain(...)fix is unchanged.- No source file is touched any more. The scan only reads the test file. I fingerprinted every file under
src/(content and modification time) before and after running both test files, and nothing changed. - Both files: 48 passed in 1.46 s (the previous version took 5.19 s, most of it spent in a second pytest). The whole #808 file at
COLUMNS=50: 46 passed.ruff checkandruff format --checkare clean. - The scan now pins each line on its own:
change to tests/test_issue808_staged_config_fields.py |
scan |
|---|---|
| none | 2 passed |
| both checks back to raw output | 2 failed |
only :302 back to raw output |
2 failed |
only :312 back to raw output |
2 failed |
| both checks deleted | 1 failed |
- The fix itself still holds on the merged tree: with default spellings forced to warn (
staged_fields.py:115changed toif True:), the 8 cases fail at 50, 80 and 120 columns and underFORCE_COLOR.
One ordering note, in case it matters: #1379 adds a new raw "read by nothing" not in capsys.readouterr().out check at line 409 of the same file, and its review already asks for _plain(...) there. If it were merged first without that change, the scan here would flag that line on the merged tree, and routing it through _plain(...) is the whole fix. I checked that combination: 1 failed before the change, 77 passed after.
This is ready from my side and now waits only on the required checks.
|
Merged as |
…dline (#808) (#1379) Nine training-intelligence and convergence knobs were validated and accepted, but nothing read them during training: forgetting_eval_steps, forgetting_benchmark, forgetting_stop, checkpoint_eval_steps, checkpoint_eval_metric, checkpoint_eval_tasks, checkpoint_keep_top, convergence_window and convergence_rel_tol. `soup train` mentioned them only in an undated "set but not enforced" note. They now join config/staged_fields.py STAGED_FIELDS (29 entries) with their schema defaults, under the existing STAGED_FIELD_REJECTION_VERSION ("0.77"). A non-default value prints `training.<knob> is accepted but read by nothing; v0.77 will refuse it` at load, and is refused under the error severity. Default spellings stay silent. The knobs leave commands/train.py _UNWIRED_TRAINING_TUNABLES, so each is reported once, with the date; the note keeps the four enable flags and forgetting_threshold. The convergence_window/convergence_rel_tol descriptions now say "Staged, not enforced during training". docs/peft-and-efficiency.md states the deadline, and #748's KNOWN_UNCONSUMED reasons point at #808 instead of the closed #799. Two tests that load a non-default knob now pin the "warn" severity, so the v0.77 switch stays a one-line change. Users notice one dated warning where they used to get an undated note. Deleting the key keeps a config loading after v0.77. Verified on the head merged with main: - The #808, #748, #761, #755, #799 and #487 files, plus training-intelligence and auto-tuning: 339 passed. - 18 config, loader, recipe, template and CLI files: 2003 passed, 1 skipped. One failure is environmental and identical on main. - 203 shipped configs load under the error severity, with no finding. - 10 of 10 mutations fail a test, including the two that survived review round 1. - The simulated v0.77 switch leaves only the #761 failure that main already has. - With #1377 on top: 78 passed. Refs #808 (group B). The issue stays open for forgetting_threshold and the enable flags.
What does this PR do?
Fixes #1352. The two
#808guard assertions substring-matched capsys's raw output for"read by nothing". Rich'sConsolewraps that warning at the render width, so at a narrow terminal (COLUMNS=50) the phrase can split across the wrap and the assertion passes even though the warning was printed, letting a staged-field regression through silently. This wraps both checks in the file's existing_plain()helper (ANSI-stripped, whitespace-collapsed) before the substring test, matching the other assertions in the same file, and adds a test that simulates the regression and confirms the guard now fails atCOLUMNS=50as it should.Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passes — not run (ruff not installed in the environment used to verify this)pytest tests/ -vpasses — not run in full;pytest tests/test_issue808_staged_config_fields.py tests/test_issue1352_narrow_terminal_guard.py -vwas run and passed (47 passed)README.mdand the matching page underdocs/) if needed — not needed, no user-facing behavior change