Skip to content

Comments

Attribute panics to the mdtests that cause them#15241

Merged
dcreager merged 3 commits intomainfrom
dcreager/mdtest-panic
Jan 3, 2025
Merged

Attribute panics to the mdtests that cause them#15241
dcreager merged 3 commits intomainfrom
dcreager/mdtest-panic

Conversation

@dcreager
Copy link
Member

@dcreager dcreager commented Jan 3, 2025 •
edited
Loading

This updates the mdtest harness to catch any panics that occur during type checking, and to display the panic message as an mdtest failure. (We don't know which specific line causes the failure, so we attribute panics to the first line of the test case.)

Sample output (for an artificial panic):

---- mdtest__unary_integers stdout ----

integers.md - Unary Operations - Unary Addition

  crates/red_knot_python_semantic/resources/mdtest/unary/integers.md:6 panicked at crates/red_knot_python_semantic/src/types/infer.rs:3264:66:
  crates/red_knot_python_semantic/resources/mdtest/unary/integers.md:6 bonk

To rerun this specific test, set the environment variable: MDTEST_TEST_FILTER="integers.md - Unary Operations - Unary Addition"
MDTEST_TEST_FILTER="integers.md - Unary Operations - Unary Addition" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__unary_integers

integers.md - Unary Operations - Unary Subtraction

  crates/red_knot_python_semantic/resources/mdtest/unary/integers.md:14 panicked at crates/red_knot_python_semantic/src/types/infer.rs:3265:66:
  crates/red_knot_python_semantic/resources/mdtest/unary/integers.md:14 foobar

To rerun this specific test, set the environment variable: MDTEST_TEST_FILTER="integers.md - Unary Operations - Unary Subtraction"
MDTEST_TEST_FILTER="integers.md - Unary Operations - Unary Subtraction" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__unary_integers

Closes #13899

dcreager
dcreager commented Jan 3, 2025
View reviewed changes
crates/ruff_db/src/panic.rs
/// [`catch_unwind`](std::panic::catch_unwind) wrapper that sets a custom [`set_hook`](std::panic::set_hook)
/// to extract the backtrace. The original panic-hook gets restored before returning.
pub(crate) fn catch_unwind<F, R>(f: F) -> Result<R, PanicError>
pub fn catch_unwind<F, R>(f: F) -> Result<R, PanicError>
Copy link
Member Author

@dcreager dcreager Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the chatter in this crate is just to publicize this existing helper function.

crates/red_knot_test/src/lib.rs Outdated
Comment on lines 147 to 151
info.info
.to_string()
.split('\n')
.map(String::from)
.collect(),
Copy link
Member Author

@dcreager dcreager Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not include the full backtrace of the panic, since that seemed to be more verbose than we need at this point. If we want it, we just change info.info to info here.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

@AlexWaygood AlexWaygood added ty Multi-file analysis & type inference testing Related to testing Ruff itself labels Jan 3, 2025
AlexWaygood
AlexWaygood approved these changes Jan 3, 2025
View reviewed changes
Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this looks great! :D

MichaReiser
MichaReiser approved these changes Jan 3, 2025
View reviewed changes
Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat

@dcreager dcreager merged commit 75015b0 into main Jan 3, 2025
21 checks passed
@dcreager dcreager deleted the dcreager/mdtest-panic branch January 3, 2025 18:45
dcreager added a commit that referenced this pull request Jan 3, 2025
@dcreager
Merge branch 'main' into dcreager/class-match
0374a77 
* main:
  [`ruff`] Avoid reporting when `ndigits` is possibly negative (`RUF057`) (#15234)
  Attribute panics to the mdtests that cause them (#15241)
  Show errors for attempted fixes only when passed `--verbose` (#15237)
  [`RUF`] Add rule to detect empty literal in deque call (`RUF025`) (#15104)
  TD003: remove issue code length restriction (#15175)
  Preserve multiline implicit concatenated strings in docstring positions (#15126)
  [`pyflakes`] Ignore errors in `@no_type_check` string annotations (`F722`, `F821`) (#15215)
  style(AIR302): rename removed_airflow_plugin_extension as check_airflow_plugin_extension (#15233)
  [`pylint`] Re-implement `unreachable` (`PLW0101`) (#10891)
  refactor(AIR303): move duplicate qualified_name.to_string() to Diagnostic argument (#15220)
  Misc. clean up to rounding rules (#15231)
  Avoid syntax error when removing int over multiple lines (#15230)
  Migrate renovate config (#15228)
  Remove `Type::tuple` in favor of `TupleType::from_elements` (#15218)
@AlexWaygood
Copy link
Member

AlexWaygood commented Jan 5, 2025

Hmm, this PR unfortunately now seems to be suppressing some of the internal error messages from red_knot_test, for example this one here:

ruff/crates/red_knot_test/src/lib.rs

Lines 32 to 37 in b264489

let suite = match test_parser::parse(short_title, &source) {
Ok(suite) => suite,
Err(err) => {
panic!("Error parsing `{path}`: {err:?}")
}
};

On a PR branch I'm working on, I got this (not very informative!) test failure:

image

If I revert this commit on that PR branch, I get this instead, which is much more helpful:

image

@AlexWaygood AlexWaygood mentioned this pull request Jan 7, 2025
Closed
@AlexWaygood
Copy link
Member

AlexWaygood commented Jan 7, 2025

I opened #15317 so we don't forget about this 👍

@dcreager
Copy link
Member Author

dcreager commented Jan 7, 2025

Thanks Alex! I'll take a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaReiser MichaReiser MichaReiser approved these changes

@AlexWaygood AlexWaygood AlexWaygood approved these changes

@carljm carljm Awaiting requested review from carljm carljm is a code owner

@sharkdp sharkdp Awaiting requested review from sharkdp sharkdp is a code owner

Assignees

No one assigned

Labels

testing Related to testing Ruff itself ty Multi-file analysis & type inference

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[red-knot] mdtest doesn't tell you the path of the Markdown file if there's a panic

3 participants

@dcreager @AlexWaygood @MichaReiser