Skip to content

CLI regression tests#245

Merged
micprog merged 17 commits intomasterfrom
fischeti/cli-regression
Jan 12, 2026
Merged

CLI regression tests#245
micprog merged 17 commits intomasterfrom
fischeti/cli-regression

Conversation

@fischeti
Copy link
Copy Markdown
Contributor

@fischeti fischeti commented Jan 9, 2026

Adds a comprehensive test suite cli_regression that tests output of certain CLI commands against the old version. The way it works is that it installs a second "golden" bender and diffs the two outputs. The idea of those tests is not to freeze the script output of bender, but rather to give a feedback to the user whether there has been an unwanted regression.

It's also added in the CI as a separate workflow that only triggers with PR. Running it on push to master would not make any sense anyway since it would just compare master against master. Unfortunately, GitHub does not have any allow-failure feature that would just warn users (there is an open feature request since 5 years), meaning this CI workflow might indicate failure which can be ignored (after double-checking the diff output).

Note on windows tests

There is an inconsistency in the script output in windows CI builds. The reason is that during checkout of a repository, git can transform newline characters to platform specific newline characters. For some reason, the git checkout done by the CI and cargo install --git checks out differently, but specifying the newline character in .gitattributes solves this issue.

TODOs:

  • Check test list for consistency

@fischeti fischeti marked this pull request as ready for review January 10, 2026 14:44
Copy link
Copy Markdown
Member

@micprog micprog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 It might make sense to go through the list of formats being tested and make sure it follows a consistent pattern, but I think this is already great! Bender's testing infrastructure is a very limited at the moment, so this will help improve consistency across versions!

@fischeti
Copy link
Copy Markdown
Contributor Author

The current test list was mostly derived from things that changed in #240 and that I wanted to check for regression. But since this is now a more general test, I agree that it would make sense to go over the list again and extend it!

@fischeti fischeti force-pushed the fischeti/cli-regression branch from 32cd259 to 4a00c4c Compare January 12, 2026 09:53
@micprog micprog merged commit 16c4d1f into master Jan 12, 2026
9 checks passed
@fischeti fischeti deleted the fischeti/cli-regression branch January 12, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants