Dr. Gleb Bahmutov, PhD
uTest -> Applause @bahmutov
https://github.com/bahmutov/talks
In the wild testing for mobile / web apps
- real people
- real devices
- real locations
Important to have base quality level before handing off an app to the testers.
If collecting code coverage is hard - it will be skipped
Axiom 2 - tests add weight
Test classification
- small tests - extremely useful
- medium tests - less helpful
- large tests - not helpful
function add(a, b) {
return a + b;
}
function sub(a, b) {
return a - b;
}
test('addition', function () {
equal(add(2, 3), 5, '2 + 3 = 5');
equal(add(-2, 2), 0, '-2 + 2 = 0');
});
Code vs small tests
JavaScript and related istanbul
Java and related Emma
Everything else: see c2.com
gt math.js
gt tests/**/s*.js
gt -t "add*" math.js
gt -m "math" lots-of-tests.js
Detect features not covered by tests
coverage
Added subtraction test
Don't forget branches
-
85% for the code
-
95% for the tests
- Use Jenkins to check the coverage level
- Keep tests DRY
Jenkins Cobertura plugin
- avoid coverage by accident
- covering all paths
- IE7 + print media CSS + small screen in portrait
Developers are stateless
Test skipped but code is still covered
// individual coverage info file
DA:1,1 //-- line number 1, covered
DA:2,1
DA:5,1
DA:6,0 //-- line number 6, not covered
DA:9,1
...
If tests run in multiple environments, there should be a way to combine results to show unified results.
coverage-ie OR coverage-chrome = coverage-all
Show technical debt
function add(a, b) { return a + b }
// cyclomatic = 1
// halstead volume = 2
function abs(a) {
return a >= 0 ? a : -a
}
// cyclomatic = 2
// halstead volume = 3
jsc, grunt-complexity, complexity-report, plato
risk-map = coverage + complexity
Speed up test execution
- Run each unit test separately
- Keep track of source files covered by each unit test
add sub
addition test 100% 0%
subtraction test 100% 100%
If running subtraction test, do not run addition test
At least, run subtraction test first
Every test updates local untested database
$ gt mathTests.js
// mathTests.js covers math.js AND utils.js
$ gt utilsTest.js
// utilsTest.js covers utils.js
// edit utils.js
$ untested
// runs utilsTest.js AND mathTests.js
works great with pre-git hooks
- Keeping tests up to date
- Code coverage collection
- Speed, flexibility
- Multi-environment setup
- Fundamental limit
- Code coverage vs input coverage
foo/
bar/
10k source files
cat/
2k source files
1 changed file
zoo/
10k source files
100% code coverage means nothing and is very hard to get
var foo = function () {
foo = null;
return true;
}
test('foo', funciton () {
ok(foo(), 'foo returns true');
// hmm, what if we call foo again?
});
Checking email format
var emailRegex =
/^[A-Z0-9.-]+@[A-Z0-9.-]+\.[A-Z]$/
ok(emailRegex.test('[email protected]'));
Official RFC 2822 standard email regex is over 400 characters long!
test('valid emails', function () {
ok(isEmail('[email protected]'));
ok(isEmail('[email protected]'));
...
});
test('invalid emails', function () {
ok(!isEmail('bb.com'));
ok(!isEmail('b@[email protected]'));
...
});
Default options using code
Default options using object
N * 0.8 ⋘ N/2 * 0.8 + N/2 * 0.8
Split a large file and cover at same ratio.
Medium tests
Tests check how pieces come together, some pieces are mocks.
Covering all code paths seems pointless.
Large tests
Tests try to verify that the user features are working.
Covering all source code is impossible.
note: top level project can still set a test coverage limit for each of its parts.
Cross language / system barriers
Unused and 3rd party code
worse in reality
Unit testing can be exhaustive
Large testing is limited
- Set up code coverage to be as easy as pie
- Use code coverage with small tests
- combine with complexity
- speed up tests
- remember limitations
- Do not shoot for 100% code coverage
[slides-now-footer]: "@bahmutov code coverage and testing" [slides-now-theme]: "bw" [slides-now-timer]: "40"