Repository navigation
Add a test runner #2
Description
We want to make something that can be invoked from the command line to automatically run all the tests in the package. An important component of this is separating out configuration from the actual test code. Right now, the way tests are run and output is formatted is controlled entirely by the test file itself via functions like solo_test and useCompactVmConfig. These should be controlled by the test runner instead.
Activity
Add some basic interfaces that will be used for the test runner.
a14n commentedon Feb 5, 2015
Do you plan to remove
solo_test?During development phase it's really straightforward to add
solo_and run a test. It allows to stay in the editor to run a test that needs some attention. Without that it could be boring to find/copy/paste the name to use in an external test runner.zoechi commentedon Feb 5, 2015
This is not only about enable/disable a test.
Especially the xxxConfig should be easily switchable depending for example whether you run the tests from an IDE (which may have some jUnit output support), or from a browser, or from console, ....
I experimented with a test runner GUI where I planned to add a GUI to run subsets of the tests on some criterias (failed >0 times in the previous x runs, some meta info added to tests/groups, test run duration - only run tests finished < 0,5sec in the last run, ...)
I was able to address this with features from the unittest library it shows that there might be situations where enabling/disabling by changing the source is not the most appropriate way.
There might be configuration necessary depending on the browser used for the test (content_shell, Firefox, IE) like
--allow-file-access-from-fileswhen the client code loads ressources not served frompub serveBut this is mostly configuration for https://pub.dartlang.org/packages/test_runner then.
nex3 commentedon Feb 5, 2015
@a14n I'm hoping that an editor will be able to detect the test's name and invoke only that test without any modification of the test file. That's how it typically works in other languages.
Add an Invoker class that manages a running test.
Add Declarer and Group classes.
Add an Engine class for running groups of test suites.
Add a ConsoleReporter for printing human-friendly test progress reports.
Add a VmListener and IsolateTest class for running tests in other iso…
Add a Loader class for loading tests from files.
Add a test runner executable.
Replace the existing unittest APIs with the new runner infrastructure.