Software Testing
Introduction
Beat Fluri
software evolution & architecture lab
Organization
Course Info
VU Software Testing
AP (ECTS): 3
Language: English
For MSc & PPO’01 after 5th semester
[Link]
Exercises
Teams of two
Grade
Wirten or oral exam depending on the number of course participants
Dr. Beat Fluri © 2011 2
Course
No course: 17.03., 24.03., 28.04., 02.06.
Preparation for course
Reading
Solving the exercises
During a course
Discussion of topics (summary of reading)
Presentation of a possible solution for the exercises
Information for upcoming exercise
Information on the website
Dr. Beat Fluri © 2011 3
Resources
John F. Smart’s blog
[Link]
[Link]
Dr. Beat Fluri © 2011 4
Content (tentative)
Test-Driven Development (TDD)
JUnit and Eclipse
Unit Testing
Testing smallest units (in isolation if possible)
Continuous Integration
One environment to build and test the application
Multi-stage builds for rapid feedback
Integration Testing
Testing interaction between components (compounds of units)
Dr. Beat Fluri © 2011 5
Content (tentative)
Web Testing with Selenium
End-to-end (acceptance) testing of web applications
Software Quality Assurance with Continuous Integration
Tools for Continuous Integration environments
System Testing (guest speaker)
Regression Testing
How to choose system tests for regression
Load and Performance Testing
How does your system behave under heavy load?
Dr. Beat Fluri © 2011 6
What is software testing?
Design and implementation of a special
kind of software system; one that
exercises another software system with
the intent to finding bugs.
Rober V. Binder, TOOSMPT
Dr. Beat Fluri © 2011 7
Test design
Identify, model, and analyze the responsibilities of the system
under test
On different level of abstraction: methods, classes, packages, subsystems, system
Design test cases based on this external perspective
Functional testing
Add test cases based on code analysis, suspicions, and heuristics
Structural testing
Develop expected results for each test case or choose an approach
to evaluate the pass or no pass status of each test case
Dr. Beat Fluri © 2011 8
Test execution
Implementation under test is minimally operational
Bottom-up to reduce test driver and test stub overhead
Test suite execution: result is evaluated as pass or no pass.
Coverage tool to instrument the implementation under test.
Define goal: 95% statements, 80% branches, 70% conditions, etc.:
If necessary, develop additional tests to exercise uncovered code.
Stop: coverage goal is met and all tests pass.
Dr. Beat Fluri © 2011 9
Why is software testing so hard?
Input/state space
public Line(int x0, int y0, int x1, int y1)
Input per int: 232 different values: 232 x 232 x 232 x 232 = 2128
1000 lines per second: 1028 years
Execution sequences
for (int i = 0; i < n; i++) {
if ([Link](i) == [Link](i)) {
x[i] += 100;
2n + 1 paths with n iterations
} else {
x[i] /= 2;
}
}
Dr. Beat Fluri © 2011 10
Why is software testing so hard?
Coincidental correctness
public class Account {
protected long lastTxDate;
public long daysSinceLastTx() {
long today = [Link]().getTimeInMillis();
return 1 + (today - lastTxDate) / (24 * 3600 * 1000);
}
}
public class DepositAccount extends Account {
public long daysSinceLastTx() {
long today = [Link]().getTimeInMillis();
return (today - lastTxDate) / (24 * 3600 * 1000);
}
}
Dr. Beat Fluri © 2011 11
Why is software testing so hard?
Undecidability problem (halting problem)
There isn’t any program that can decide whether the execution of another program
will halt or not
For simple program it is decidable but not in general
Dr. Beat Fluri © 2011 12
Why software testing?
Find bugs early
Airbag does not work: find out before or after releasing the car?
Part of quality assurance
Would you trust the airbag software if it wasn’t tested?
Supports changing a software system
Verify component integrity after a change
Dr. Beat Fluri © 2011 13
Why is testing neglected?
Expensive (software testing is hard)
Test case design and selection
Building test environment
Integration environment
Few methodologies for test automation
Not part of development process
Hard to include for specific projects (customer won’t pay extra effort)
Not part of the software engineering culture
Missing testing environment
Dr. Beat Fluri © 2011 14
Short glossary
Test case
Set of inputs, execution conditions, and expected result (pass and no pass criterion)
Test suite
Set of test cases, usually related to a testing goal
Test driver
Class or utility program that applies test cases
Stub
Partial, temporary implementation of a component
Test harness
Substitution for parts of the deployment environment, i.e., a system of test drivers
and other tools to support test execution
Dr. Beat Fluri © 2011 15
Short glossary
Oracle
A mechanism to evaluate the actual result of a test case as pass or no pass
Equivalence class
Set of input values: if one value is processed correctly/incorrectly, all other values will
also be processed correctly/incorrectly
Coverage
Percentage of elements required by a test strategy that have been exercised by a
given test suite
Dr. Beat Fluri © 2011 16
Short glossary
Fault model
Identifies relationships and components that are most likely to have faults
Test model
Testable representation of the relationships among elements of an implementation
based on a fault model
Test strategy
Algorithm or heuristic to create test cases from an implementation or test model
Test design
Produces a test suite using a test strategy: identification of interesting test inputs,
placing these test inputs into a sequence, and defining the expected results
Dr. Beat Fluri © 2011 17
Short glossary
Fault-directed testing
Testing that seeks to reveal implementation faults
Conformance-directed testing
Testing that seeks to establish conformance to requirements or specifications
Dr. Beat Fluri © 2011 18
V-Model (historically)
Specification Implementation
Acceptance Test
User
Delivery
needs
System Test
System Spec System
Integration Test
Subsystem
Subsystem
Design/Spec
Unit Test
Unit
Unit
Design/Spec
Dr. Beat Fluri © 2011 19
V-Model (historically)
Specification Implementation
User
needs Validation Delivery
System Spec System
Subsystem
Design/Spec Verification Subsystem
Unit
Unit
Design/Spec
Dr. Beat Fluri © 2011 20
Test-Driven Development (TDD)
Software development process with very short cycles
Write a test Test
case succeeds
Check if the Test Write app Test(s)
test case fails fails code fail
Test(s)
Run all tests Refactor
succeed
Repeat
Dr. Beat Fluri © 2011 21
Test-Driven Development (TDD)
Three rules of TDD of Robert Martin1
1. You may not write production code until you have written a failing unit test.
2. You may not write more of a unit test than is sufficient to fail, ant not compiling is
failing.
3. You may not write more production code than is sufficient to pass the currently
failing test.
1Robert Martin. Clean Code, Prentice Hall, 2009.
Dr. Beat Fluri © 2011 22
Exercise 0
Line segment Triangle (three line segments)
(x1,y1)
(x0,y0)
scalene isosceles equilateral
Valid triangle (s-rule)
No side may have a length of zero and each side must be
shorter than the sum of all sides divided by 2:
s > a, s > b, s > c with s = (a + b + c) / 2
Dr. Beat Fluri © 2011 23
Exercise 0
Install Eclipse 3.6.x
Download assignment_0.jar
LineSegment and Triangle classes
Download assignment_0-[Link]
Javadoc for LineSegment and Triangle classes
Write test cases for LineSegment and Triangle
Dr. Beat Fluri © 2011 24
