Skip to content

Comments

[red-knot] document test framework#13695

Merged
carljm merged 9 commits intomainfrom
cjm/test-framework-docs
Oct 10, 2024
Merged

[red-knot] document test framework#13695
carljm merged 9 commits intomainfrom
cjm/test-framework-docs

Conversation

@carljm
Copy link
Contributor

@carljm carljm commented Oct 9, 2024

This adds documentation for the new test framework.

I also added documentation for the planned design of features we haven't built yet (clearly marked as such), so that this doc can become the sole source of truth for the test framework design (we don't need to refer back to the original internal design document.)

Also fixes a few issues in the test framework implementation that were discovered in writing up the docs.

@carljm carljm added the ty Multi-file analysis & type inference label Oct 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Copy link
Contributor

@T-256 T-256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is circular imports concept in Python, do you plan support it into mdtests?

@carljm
Copy link
Contributor Author

carljm commented Oct 10, 2024

There is circular imports concept in Python, do you plan support it into mdtests?

I'm not sure what extra support mdtests would need for circular imports? It allows you to create whatever Python files you like with whatever contents you like, so you can create and test any circular import scenario you would want.

Co-authored-by: T-256 <[email protected]>
Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!! This is extremely valuable.

Despite having previously reviewed your proposal for this framework in Notion, and the actual implementation, it actually made me realise there were several aspects I wasn't aware of before reading this :-)

Some comments below, none huge:

carljm and others added 2 commits October 10, 2024 08:27
Co-authored-by: Alex Waygood <[email protected]>
Co-authored-by: Dhruv Manilawala <[email protected]>
@carljm
Copy link
Contributor Author

carljm commented Oct 10, 2024

Ok, pushed an updated that I think addresses all comments.

I also realized in working on the change to require either rule or message-contains, that when a diagnostic doesn't match any assertion we should include its actual column in the "unexpected error" output, otherwise it's hard to debug a failure to match on the column, so I made this change as well. And in the course of making that change, I realized that some tests were not using dedent when they should, resulting in silly column numbers, so I fixed that as well.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@carljm
Copy link
Contributor Author

carljm commented Oct 10, 2024

Thanks for the thorough review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants