Software Testing – Automation and

JUnit
Fadi Wedyan, Bashar Al shboul
Department of Software Engineering
Hashemite University

Software Testing - Automation and JUnit 1

 What is Test Automation?
The use of software to control the execution of tests, the comparison of
actual outcomes to predicted outcomes, the setting up of test preconditions,
and other test control and test reporting functions

Reduces cost
Reduces human error
Reduces variance in test quality from different individuals
Significantly reduces the cost of regression testing

Software Testing - Automation and JUnit 2

 Software Testability (3.1)

The degree to which a system or component facilitates the establishment of

test criteria and the performance of tests to determine whether those
criteria have been met

Plainly speaking – how hard it is to find faults in the software

Testability is dominated by two practical problems
– How to provide the test values to the software
– How to observe the results of test execution

Software Testing - Automation and JUnit 3

 Observability and Controllability
Observability
How easy it is to observe the behavior of a program in terms of its outputs,
effects on the environment and other hardware and software components

– Software that affects hardware devices, databases, or remote files have low
observability
Controllability
How easy it is to provide a program with the needed inputs, in terms of
values, operations, and behaviors
– Easy to control software with inputs from keyboards
– Inputs from hardware sensors or distributed software is harder
Data abstraction reduces controllability and observability
Software Testing - Automation and JUnit 4
Terminology
 Program - Sort
◼ Collection of functions (C) or classes (Java)
 Specification
◼ Input: p - array of n integers, n>0
◼ Output: q - array of n integers such that
 q[0]  q[1]  …  q[n]
 Elements in q are a permutation of elements in p, which are
unchanged
◼ Description of requirements of a program

Software Testing - Automation and JUnit 5

More terminology
 Test input
◼ A set of values given as input to a program (along with
the expected output)
◼ Includes environment variables
◼ {2, 3, 6, 5, 4}
 Test case
◼ Test input with expected output
 Test set (Suite)
◼ A set of test cases
Software Testing - Automation and JUnit 6
Test Oracle
 Function that determines whether the results of
executing a program under test are as per the
program’s specifications
 Often a person, who monitors output.
◼ Not a reliable method.
 Automatic oracles check output using another
program.
◼ Requires some kind of executable specification.

Software Testing - Automation and JUnit 7

Test Scaffolding
 Code developed to facilitate testing.
 Allows us to test incomplete systems.
◼ Test harnesses: substituting for parts of the deployment
environment
◼ Test drivers: test components.
 substituting for a main or calling program
◼ Stubs: test a system when some functionality it uses are
not yet implemented.
 Often a short, dummy program --- a method with an empty
body.

Software Testing - Automation and JUnit 8

JUnit
 A framework for writing and running tests
 Written by Erich Gamma (Design Patterns) and Kent
Beck (eXtreme Programming)
 JUnit uses Java's reflection capabilities (Java programs
can examine their own code) and (as of version 4)
annotations
 JUnit allows us to:
◼ define and execute tests and test suites
◼ write code and use the tests to support refactoring
◼ integrate code and always be ready to release a working
version
Software Testing - Automation and JUnit 9
JUnit
 JUnit can be used to test …
◼ an entire object
◼ part of an object – a method or some interacting methods
◼ interaction between several objects
 It is primarily for unit testing, has limited support for integration
testing
◼ Not suitable for system testing
 Each test is embedded into one test method
 A test class contains one or more test methods

Software Testing - Automation and JUnit 10

Annotations in J2SE 5
 J2SE 5 introduces the Metadata feature (data about
data)
 Annotations allow you to add decorations to your
code (remember javadoc tags: @author )
 Annotations are used for code documentation,
compiler processing (@Deprecated ), code
generation, runtime processing

 ( http://java.sun.com/docs/books/tutorial/java/javaOO/annotations.html )

Software Testing - Automation and JUnit 3.11

Example: Old way vs. new way
 int max(int a, int b) {  @Test
if (a > b) void testMax() {
return a; assertEquals(7, max(3, 7));
else assertEquals(3, max(3, -7));
return b;
} }
 void testMax() {
int x = max(3, 7);
if (x != 7)
System.out.println("max(3, 7) gives " + x);
x = max(3, -7);
if (x != 3)
System.out.println("max(3, -7) gives " + x);
}
 public static void main(String[] args) {
new MyClass().testMax();
}

12 Software Testing - Automation and JUnit

Test Class
 Test classes include:
◼ A test runner to run the tests (main())
◼ A collection of test methods
◼ Methods
 Set up the state before each test
 Update the state after each test
 Before all tests
 After all tests

Software Testing - Automation and JUnit 13

Availability
 JUnit is available on several IDEs (e.g., BlueJ,
JBuilder, and Eclipse)
 Used versions
◼ JUnit 3
◼ JUnit 4
◼ Junit 5
 You will learn Junit 4 and 5!

Software Testing - Automation and JUnit 14

Resources for JUnit
 JUnit website
◼ http://www.junit.org/
 JUnit eclipse tool
◼ https://wiki.eclipse.org/Eclipse_Articles,_Tutorials,_Dem
os,_Books,_and_More?id=408#Java_Unit_testing_with_J
Unit_4.x_in_Eclipse
 Google for it!

Software Testing - Automation and JUnit 15

JUnit's Terminology
 A test runner: a program that runs tests and reports results.
◼ Many implementations: standalone GUI, command line, integrated into
IDE
 A test suite: a collection of test cases.
 A test case: tests the response of a single method to a particular
set of inputs.
 A unit test: a test of the smallest element of code you can
sensibly test
◼ usually a single class.

Software Testing - Automation and JUnit 16

More Terminology
 A test fixture is the environment in which a test is run.
◼ A new fixture is set up before each test case is executed and torn down
afterwards. (i.e., test harness)
◼ Objects and variables that are used by more than one test
◼ Initializations (prefix values)
◼ Reset values (postfix values) Example: if you are testing a database
client, the fixture might place the database server in a standard initial
state, ready for the client to connect.
 An integration test is a test of how well classes work together.
◼ JUnit provides some limited support for integration tests.
 Proper unit testing would involve mock objects
◼ Mock object: fake versions of the other classes with which the class under
test interacts.
◼ JUnit does not help with this. It is worth knowing about, but not always
necessary.

Software Testing - Automation and JUnit 17

Writing a JUnit 4 test class
 Start by importing the JUnit 4 classes
import org.junit.*;
import static org.junit.Assert.*; // note static import
 Declare your test class in the usual way
public class MyProgramTest { }
 Declare an instance of the class being tested
 You can declare other variables, but don’t give them
initial values here
public class MyProgramTest {
MyProgram program;
int someVariable;
}

Software Testing - Automation and JUnit 18

Writing a JUnit 4 test class (2)
 You can declare one method to be executed just once, when the
class is first loaded
◼ Use expensive setup, such as connecting to a database
@BeforeClass
public static void setUpClass() throws Exception {
// one-time initialization code
}
 You can declare one method to be executed just once, to do
cleanup after all the tests have been completed
@AfterClass
public static void tearDownClass() throws Exception {
// one-time cleanup code
}

Software Testing - Automation and JUnit 19

Writing a JUnit 4 test class (3)
 Define a method (or several methods) to be executed before
each test
 Initialize your variables in this method, so that each test starts
with a fresh set of values
@Before
public void init() {}
 Example
@Before
public void setUp() {
program = new MyProgram();
array = new int[] { 1, 2, 3, 4, 5 };
}
Software Testing - Automation and JUnit 20
Writing a JUnit 4 test class (4)
 You can define one or more methods to be executed after each
test;
 Typically, such methods release resources, such as files
◼ Java usually does this for free, but les, network connections etc. might not
get tidied up automatically.
 Not used frequently
@After
public void tearDown() {
}

Software Testing - Automation and JUnit 21

JUnit Lifecycle
1. Create a different test class instance for each test method
2. Repeat for each test class instance:
1. call setup
2. call the test method

Software Testing - Automation and JUnit 22

JUnit Object Initialization
 For the Class Under Test, initialize in a @Before method to catch
failures.
 For other classes, initialize in the declaration:
◼ in addition, one can mark fields final.
◼ unless it is complex initialization that could fail, in which case use @Before,
to catch failures.
 For global state (esp. slow initialization, like a database), use
@BeforeClass, but be careful of dependencies between tests.
 Initialization of an object used in a single test should of course be
done in the test method itself.

Software Testing - Automation and JUnit 23

@Before and @After methods
 You can have as many @Before and @After methods as you want
◼ Be warned: You don’t know in what order they will execute
 You can inherit @Before and @After methods from a superclass;
execution is as follows:
◼ Execute the @Before methods in the superclass
◼ Execute the @Before methods in this class
◼ Execute a @Test method in this class
◼ Execute the @After methods in this class
◼ Execute the @After methods in the superclass

Software Testing - Automation and JUnit 24

The Test Method
 A test method is annotated with @Test, takes no
parameters, and returns no result
Printed if
 All the assertXXX methods can be used assert fails
@Test
public void sum() { Test Input
assertEquals(“Sum is incorrect”, 15, program.sum(array));
assertTrue(program.min(array) > 0);
}
Actual
output
Expected
output
Software Testing - Automation and JUnit 25
The Test Method
 @Test public void testSum(),
 @Test public void testMultiply(), etc.
◼ These methods contain tests for the sum method and the
multiply method.

Software Testing - Automation and JUnit 26

Asserts
 An assert method is a JUnit method that
performs a test, and throws an AssertionError if
the test fails
◼ JUnit catches these Errors and shows you the result
 Within a test,
◼ Call the method being tested and get the actual result.
◼ assert a property that should hold of the test result.
◼ Each assert is a challenge on the test result.

Software Testing - Automation and JUnit 27

Assert
 static void assertTrue(boolean test)
static void assertTrue(String message, boolean test)
Throws an AssertionError if the test fails.
The optional message is included in the Error.
 static void assertFalse(boolean test)
static void assertFalse(String message, boolean test)
Throws an AssertionError if the test succeeds.

Software Testing - Automation and JUnit 28

A simple example
 Suppose you have a class Arithmetic with methods int multiply(int x, int y), and boolean
isPositive(int x)

import org.junit.*;
import static org.junit.Assert.*;

public class ArithmeticTest {

@Test
public void testMultiply() {
assertEquals(4, Arithmetic.multiply(2, 2));
assertEquals(-15, Arithmetic.multiply(3, -5));
}

@Test
public void testIsPositive() {
assertTrue(Arithmetic.isPositive(5));
assertFalse(Arithmetic.isPositive(-5));
assertFalse(Arithmetic.isPositive(0));
}
}

Software Testing - Automation and JUnit 29

Example: Counter Class
 The constructor will create a counter and set it to
zero
 The increment method will add one to the counter
and return the new value
 The decrement method will subtract one from the
counter and return the new value
 The getCount method returns the current count

Software Testing - Automation and JUnit 30

JUnit tests for Counter
public class CounterTest {
Counter counter1; // declare a Counter here
@Before
void setUp() {
counter1 = new Counter(); // initialize the Counter here
}

@Test  Note that each test

public void testIncrement() { begins with a brand
assertTrue(counter1.increment() == 1);
assertTrue(counter1.increment() == 2);
new counter
}  This means you don’t
have to worry about
@Test the order in which the
public void testDecrement() { tests are run
assertTrue(counter1.decrement() == -1);
}
}
Software Testing - Automation and JUnit 31
How many asserts a test can have?
 As many as you want!
 Its better to have one assert per test
◼ JUnit only report the first failed assert in the test
(masking)
◼ small tests allow us to more easily identify failures …
 However, sometimes you need a long test to reach
a certain state of the object (and verify the states
on the path)

Software Testing - Automation and JUnit 32

More Asserts
 assertEquals(expected, actual)
◼ This method is heavily overloaded: expected and actual must be both objects or both of the
same primitive type. For objects, uses your equals method, if you have defined it properly,
as public boolean equals(Object o), otherwise it uses ==
◼ This method works great for Strings and a few other Java classes
◼ For objects of classes that you create, you have to define equals
 To define equals for your own objects, define exactly this method:
public boolean equals(Object obj) { ... }
◼ The argument must be of type Object, which isn’t what you want, so you must
cast it to the correct type (say, Person):
public boolean equals(Object something) {
Person p = (Person)something;
return this.name == p.name; // test whatever you like here
}
◼ Refer to your Java course material!

Software Testing - Automation and JUnit 33

More Asserts
 assertSame(Object expected, Object actual)
assertSame(String message, Object expected, Object actual)
◼ Asserts that two objects refer to the same object (using ==)

 assertNotSame(Object expected, Object actual)

assertNotSame(String message, Object expected, Object actual)
◼ Asserts that two objects DO NOT refer to the same object (using ==)

Software Testing - Automation and JUnit 34

More Asserts
 assertNull(Object object)
assertNull(String message, Object object)
◼ Asserts that the object is null
 assertNotNull(Object object)
assertNotNull(String message, Object object)
◼ Asserts that the object is null

 fail()
◼ Causes the test to fail and throw an AssertionError. Useful as a result of a
complex test, when the other assert methods are not quite what you want

Software Testing - Automation and JUnit 35

Demo- JUnit 4 with Eclipse

Software Testing - Automation and JUnit 36

Step 1
Add Junit 4 to
the project
build path

Write click on
source-new-
JUnit test case

Or
File-new- JUnit
test case

Software Testing - Automation and JUnit 37

Step 2
Choose Version

Create a package
for the test cases
(recommended)
The test class

Choose stubs you

want to create

The tested class

Software Testing - Automation and JUnit 38

Step 3

Choose
which
methods to
test

Software Testing - Automation and JUnit 39

The Created Template
package testjunit4;
import static org.junit.Assert.*;
import org.junit.Test;

public class TestTriangleJUnit4 {

@Test
public void testIsEquilateral() {
fail("Not yet implemented");
}
@Test
public void testIsScalene() {
fail("Not yet implemented");
}
@Test
public void testIsIsoceles() {
fail("Not yet implemented");
}
}

Software Testing - Automation and JUnit 40

Build your test case
package testjunit4;
import static org.junit.Assert.*;
import org.junit.Test;
import triangle.Triangle;
public class TestTriangleJUnit4 {
Triangle t;
@Test public void testIsEquilateral() {
t = new Triangle(4, 4, 4);
assertTrue(t.isEquilateral());
}
@Test public void testIsScalene() {
t = new Triangle(2, 4, 4);
assertTrue(t.isScalene());
}
@Test public void testIsIsoceles() {
t = new Triangle(3, 4, 5);
assertTrue(t.isIsoceles());
}
}
Software Testing - Automation and JUnit 41
Set the object state
 objects that have significant internal state (e.g. collections with
some additional structure) are harder to test because it may
take many method calls to get an object into a state you want
to test.
◼ Write long tests that call some methods many times.
◼ Add additional methods in the interface to allow observation of state (or
make private variables public?)
◼ Add additional methods in the interface that allow the internal state to be
set to a particular value

Software Testing - Automation and JUnit 42

Test Suites
 When you have test cases in many test classes, you
can combine and run them together as one test suite
 Sometimes you don’t have the source code of the test
cases, but you want to run the bytecode
 For the triangle class, you might have a test class for
the valid test cases and another test class for the
invalid test cases (i.e., testing invalid inputs)

Software Testing - Automation and JUnit 43

Creating and running a test suite (JUnit 4)
 Create a java class.
 Attach @RunWith(Suite.class) Annotation with
class.
 Add reference to Junit test classes using
@Suite.SuiteClasses annotation

Software Testing - Automation and JUnit 44

Creating and running a test suite (JUnit 4)
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
TestSuite1.class,
TestSuite2.class
})
public class TestAll
{ }

Software Testing - Automation and JUnit 45

Special features of @Test
 You can limit how long a method is allowed to take
 This is good protection against infinite loops
 The time limit is specified in milliseconds
 The test fails if the method takes too long
@Test (timeout=10)
public void greatBig() {
assertTrue(program.ackerman(5, 5) > 10e12);
}

46 Software Testing - Automation and JUnit

Special features of @Test
 Some method calls should throw an exception
◼ You can specify that a particular exception is expected
◼ The test will pass if the expected exception is thrown, and fail
otherwise
1. @Test (expected=IllegalArgumentException.class)
public void factorialThrowAnException() {
program.factorial(-5);
}
2. @Test public void factorialThrowAnException() {
try {
program.factorial(-5);
} catch (IllegalArgumentException e) {
return;
}
fail(“IllegalArgumentException expected”);
}

Software Testing - Automation and JUnit 47

 Tests with Parameters: JUnit Theories
Unit tests can have actual parameters
– So far, we’ve only seen parameterless test methods
Contract model: Assume, Act, Assert
– Assumptions (preconditions) limit values appropriately
– Action performs activity under scrutiny
– Assertions (postconditions) check result

@Theory public void removeThenAddDoesNotChangeSet (

Set<String> someSet, String str) { // Parameters!
assumeTrue (someSet != null) // Assume
assumeTrue (someSet.contains (str)) ; // Assume
Set<String> copy = new HashSet<String>(someSet); // Act
copy.remove (str);
copy.add (str);
assertTrue (someSet.equals (copy)); // Assert
}

Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 48

 Question: Where Do The Data Values Come
Answer:
From?
– All combinations of values from @DataPoints annotations where assume clause is true
– Four (of nine) combinations in this particular case
– Note: @DataPoints format is an array

@DataPoints
public static String[] animals = {"ant", "bat", "cat"};
Nine combinations of
animalSets[i].contains (animals[j])
@DataPoints is false for five combinations
public static Set[] animalSets = {
new HashSet (Arrays.asList ("ant", "bat")),
new HashSet (Arrays.asList (“bat", “cat", “dog“, “elk”)),
new HashSet (Arrays.asList (“Snap”, “Crackle”, “Pop"))
};
Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 49
 JUnit Theories Need BoilerPlate
import org.junit.*;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
import static org.junit.Assume.*;

import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;

import java.util.*;

@RunWith (Theories.class)
public class SetTheoryTest
{
… // See Earlier Slides
}

Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 50

Example- Test this class
public class Calculator {
int a, b;
public Calculator(int a, int b){
this.a=a;
this.b=b;
}
double div(){
return a/b;
}
}

Software Testing - Automation and JUnit 51

Example- Test case
@Test(expected = ArithmeticException.class)
public void test1() {
Calculator calculator = new Calculator(1,0);
System.out.println("method should through an exception!");
calculator.div();
}

Software Testing - Automation and JUnit 52

Assertion of real numbers
 Consider this test case
@Test
public void test2() {
calculator = new Calculator(1,1);
assertEquals(1, calculator.div());
}
Failure, why?!

 Fix
assertEquals(1, calculator.div(),0.005);

Software Testing - Automation and JUnit 53

Ignoring a test
 The @Ignore annotation says to not run a test
@Ignore("I don’t want the boss to know this doesn’t work")
@Test
public void add() {
assertEquals(4, program.sum(2, 2));
}
 You shouldn’t use @Ignore without a very good reason!

54 Software Testing - Automation and JUnit

Run this test!
package testjunit4;
import org.junit.Test; //Check that an object isn't null
import static org.junit.Assert.*; assertNotNull(str1);
public class TestAssertions {
@Test //Check that an object is null
public void testAssertions() { assertNull(str3);
//test data
String str1 = new String ("abc"); /*Check if two object references point to the same object */
String str2 = new String ("abc"); assertSame(str4,str5);
String str3 = null;
String str4 = "abc"; /*Check if two object references not point to the same object */
String str5 = "abc"; assertNotSame(str1,str2);

String[] expectedArray={"one", "two", "three"}; //Check whether two arrays are equal to each other.
String[] resultArray = {"one", "two", "three"}; assertArrayEquals(expectedArray, resultArray);
}
//Check that two objects are equal }
assertEquals(str1, str2);

//Check that a condition is false

assertFalse(str1.length() > str2.length());

Software Testing - Automation and JUnit 55

main() for command line execution
Package testjunit4;
import org.junit.runner.JUnitCore;
public class Tester {
public static void main(String[] args) {
if (args.length !=1){
System.out.println("Usage TestAllV2 parameter, " +
"parameter can be either \n" +
"1 - run testsuite1 \n"+
"2 - run testsuite2 \n"+
"3 - run testsuite3");
System.exit(0);
}

Software Testing - Automation and JUnit 56

main() for command line execution
[Contd.]
int option= Integer.parseInt(args[0]);
switch (option){
case 1: JUnitCore.main( new String [ ]
{"ValidTestCases"});
break;
case 2: JUnitCore.main( new String [ ]
{"InvalidTestCases"});
break;
case 3: JUnitCore.main( new String [ ]
{"ValidTestCases", "InvalidTestCases"});
}
}
}
Software Testing - Automation and JUnit 57
Running JUnit 4 Test Class from command
line
 Run using the JUnit 4 Runner
java -classpath .;c:\jars\junit4.jar;c:\TestingWS\Triangle\bin
org.junit.runner.JUnitCore testjunit4.TestAssertions

 Or Run as a Java class

java -classpath .;c:\jars\junit4.jar;c:\TestingWS\Triangle\bin
testjunit4.Tester 1

Software Testing - Automation and JUnit 58

 JUnit 5 changes: min() Example
JUnit 5 uses assertions, not annotations, for exceptions
@Test public void testForNullList()
{
assertThrows(NullPointerException.class, () -> Min.min(null));
}

Other JUnit 5 differences

– Java lambda expressions play a role
– @Before, @After change to @BeforeEach, @AfterEach
– imports, some assertions change
– Test runners change (no simple replacement for AllTests.java)
– @Theory construct moved to 3rd party extensions
• google “property-based testing”
See MinTestJUnit5.java on the book website
Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 59
 Summary
The only way to make testing efficient as well as effective
is to automate as much as possible
Test frameworks provide very simple ways to automate
our tests
It is no “silver bullet” however … it does not solve the
hard problem of testing :
What test values to use ?
• This is test design … the purpose of test criteria

Introduction to Software Testing, Edition 2 (Ch 3) © Ammann & Offutt 60

Resources
 Unit Test Infected: Programmers Love Writing
Tests- an introduction to JUnit.
 Using JUnit With Eclipse IDE- an O'Reilly article
 Software Testing Course slides, Stuart Anderson,
University of Edinburgh
 Introduction to Software Testing: Chapter 3, Test
Automation

Software Testing - Automation and JUnit 61