Skip to content

Comments

[ty] Set INSTA_FORCE_PASS and INSTA_OUTPUT environment variables from mdtest.py#21029

Merged
AlexWaygood merged 1 commit intomainfrom
alex/mdtest-snapshot-integration2
Oct 22, 2025
Merged

[ty] Set INSTA_FORCE_PASS and INSTA_OUTPUT environment variables from mdtest.py#21029
AlexWaygood merged 1 commit intomainfrom
alex/mdtest-snapshot-integration2

Conversation

@AlexWaygood
Copy link
Member

@AlexWaygood AlexWaygood commented Oct 22, 2025

Summary

Setting these environment variables makes it much less painful to iterate on an mdtest that has snapshots enabled.

Currently if any snapshots change, you can't see which mdtest assertions are failing until you accept the snapshots (which requires terminating mdtest.py). So you accept the snapshots, then start mdtest.py running again, then you find some mdtest assertions are failing, so then you update them... and now the snapshots are failing again.

By setting these environment variables, .snap.new files will still be created for any snapshots that change, but mdtest.py will continue as normal anyway so that you can see which assertions are failing.

Reviewing the snapshot changes still has to be done in a separate process to the mdtest.py script. I experimented with having mdtest.py auto-accept all snapshot changes, but invoking cargo insta accept every time an mdtest failed made the script much slower.

Test Plan

I manually tested it by making this change:

diff --git a/crates/ty_python_semantic/src/types.rs b/crates/ty_python_semantic/src/types.rs
index 465df24583..37b2f890a4 100644
--- a/crates/ty_python_semantic/src/types.rs
+++ b/crates/ty_python_semantic/src/types.rs
@@ -9180,7 +9180,7 @@ impl<'db> IterationError<'db> {
             #[expect(clippy::wrong_self_convention)]
             fn is_not(self, because: impl std::fmt::Display) -> LintDiagnosticGuard<'a, 'a> {
                 let mut diag = self.builder.into_diagnostic(format_args!(
-                    "Object of type `{iterable_type}` is not {maybe_async}iterable",
+                    "Object of type `{iterable_type}` is nott {maybe_async}iterable",
                     iterable_type = self.iterable_type.display(self.db),
                     maybe_async = if self.mode.is_async() { "async-" } else { "" }
                 ));

and then incrementally updated failing assertions that were presented to me by mdtest, observing that snapshot failures no longer blocked the script from continuing.

@AlexWaygood AlexWaygood added internal An internal refactor or improvement ty Multi-file analysis & type inference labels Oct 22, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2025 •
edited
Loading

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2025 •
edited
Loading

mypy_primer results

No ecosystem changes detected ✅
No memory usage changes detected ✅

@AlexWaygood AlexWaygood force-pushed the alex/mdtest-snapshot-integration2 branch from 1c3c87b to e375bb9 Compare October 22, 2025 14:06
sharkdp
sharkdp approved these changes Oct 22, 2025
View reviewed changes
Copy link
Contributor

@sharkdp sharkdp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool — I didn't know these exist. Thank you!

@AlexWaygood AlexWaygood merged commit 20510e1 into main Oct 22, 2025
41 checks passed
@AlexWaygood AlexWaygood deleted the alex/mdtest-snapshot-integration2 branch October 22, 2025 14:32
dcreager added a commit that referenced this pull request Oct 22, 2025
@dcreager
Merge branch 'main' into dcreager/real-constraint-sets
b4e8963 
* main: (65 commits)
  [ty] Some more simplifications when rendering constraint sets (#21009)
  [ty] Make `attributes.md` mdtests faster (#21030)
  [ty] Set `INSTA_FORCE_PASS` and `INSTA_OUTPUT` environment variables from mdtest.py (#21029)
  [ty] Fall back to `Divergent` for deeply nested specializations (#20988)
  [`ruff`] Autogenerate TypeParam nodes (#21028)
  [ty] Add assertions to ensure that we never call `KnownClass::Tuple.to_instance()` or similar (#21027)
  [`ruff`] Auto generate ast Pattern nodes (#21024)
  [`flake8-simplify`] Skip `SIM911` when unknown arguments are present (#20697)
  Render a diagnostic for syntax errors introduced in formatter tests (#21021)
  [ty] Support goto-definition on vendored typeshed stubs (#21020)
  [ty] Implement go-to for binary and unary operators (#21001)
  [ty] Avoid ever-growing default types (#20991)
  [syntax-errors] Name is parameter and global (#20426)
  [ty] Disable panicking mdtest (#21016)
  [ty] Fix completions at end of file (#20993)
  [ty] Fix out-of-order semantic token for function with regular argument after kwargs (#21013)
  [ty] Fix auto import for files with `from __future__` import (#20987)
  [`fastapi`] Handle ellipsis defaults in FAST002 autofix (`FAST002`) (#20810)
  [`ruff`] Skip autofix for keyword and `__debug__` path params (#20960)
  [`flake8-bugbear`] Skip `B905` and `B912` if <2 iterables and no starred arguments (#20998)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sharkdp sharkdp sharkdp approved these changes

@carljm carljm Awaiting requested review from carljm carljm is a code owner

@dcreager dcreager Awaiting requested review from dcreager dcreager is a code owner

Assignees

No one assigned

Labels

internal An internal refactor or improvement ty Multi-file analysis & type inference

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AlexWaygood @sharkdp