8-5-2019
Specialisten in vooruitgang
Exploratory testing in
theory and practice
Jan Jaap Cannegieter
Principal consultant Squerist
Software testen Business Process Transformation Security testen
Exploratory testing in theory and practice
Missie: verder brengen van mens, klant en expertise
Wendbaar-
Specialisme Mens op 1
heid
1
8-5-2019
Exploratory testing in theory and practice
Test diensten
(Agile) testen Test automation BI testen Security
Test advies Test uitvoering Test management Test opleidingen
Test partnership
Specialisten in vooruitgang
Why testing?
Software testen Business Process Transformation Security testen 4
2
8-5-2019
Exploratory testing in theory and practice
Een short movie
Exploratory testing in theory and practice
The most expensive piece of software. Untested
declare
vertical_veloc_sensor: float;
horizontal_veloc_sensor: float;
vertical_veloc_bias: integer;
horizontal_veloc_bias: integer;
...
begin
declare
pragma suppress(numeric_error, horizontal_veloc_bias);
begin
sensor_get(vertical_veloc_sensor);
sensor_get(horizontal_veloc_sensor);
vertical_veloc_bias := integer(vertical_veloc_sensor);
horizontal_veloc_bias := integer(horizontal_veloc_sensor);
...
exception
when numeric_error => calculate_vertical_veloc();
when others => use_irs1();
end;
end irs2;
€ 6.522.600.000
3
8-5-2019
Exploratory testing in theory and practice
TU Delft geeft gezakte
studenten een diploma
Laatste update: 8 mei 2019 07:28
I want to
prevent this Studenten van de TU Delft die
onvoldoendes voor vakken hebben
gehaald hebben toch een diploma
gekregen. Dit komt door een
softwarefout in de systemen
De fout is ontstaan door een softwarefout.
Volgens de bank komt dit door een fout bij een
leverancier. De klanten halen massaal hun
spaargeld weg bij de bank. Dit is niet nodig,
aldus een woordvoerder van de bank. “alle
klanten worden schadeloosgesteld” aldus de
woordvoerder. , NIBC probeert al jaren een
positie op de consumentenmarkt probeert te
krijgen. Dit helpt daar niet bij.
Exploratory testing in theory and practice
Exploratory testing
10:30
4
8-5-2019
Exploratory testing in theory and practice
Main streams in testing Testing
Scripted testing Exploratory testing
Test management
Strategy &
planning
Design &
TP 3
Execution Closure
Specification
TP 4
TP 2
Test point
Exploratory testing in theory and practice
Exploratory testing is…
…a style of software testing…
…that emphasizes the personal freedom and responsibility of the individual tester…
…to continually optimize the value of her work…
…by treating test-related learning, test design, test execution, and test result
interpretation as mutually supportive activities that run in parallel throughout the
project.
Source: http://kaner.com/?p=46
and ‘Explore it!’ by Elisabeth Hendrickson
10
10
5
8-5-2019
Exploratory testing in theory and practice
Essence of exploratory testing
TP 3
TP 4
TP 2
Test point / test idea 1 Based on RST 3.0 by James Bach
and Michael Bolton
11
11
Exploratory testing in theory and practice
Or populair said
12
12
6
8-5-2019
Exploratory testing in theory and practice
Simultanuous working
Execution Learning
Strategy
(thinking)
13
13
Exploratory testing in theory and practice
Optional discussion
What are the possible benefits of exploratory testing compared to scripted
testing?
What are the possible backdraws of exploratory testing compared to
scripted testing?
How can we structure exploratory testing to make sure we don’t do ad hoc
testing without losing the benefits?
14
14
7
8-5-2019
Exploratory testing in theory and practice
Possible structuring elements
Overall mission of testing
List of product risks
Insight in the structure of the product
Determine test units
Track coverage
Test charters with testpoints
Test design techniques
Checklists
ISO 25010 / 9126
Defect administration
15
15
Exploratory testing in theory and practice
Example test charter
16
16
8
8-5-2019
Exploratory testing in theory and practice
Case PHPTravels
Go to www.phptravels.net
This is a test system!
Explore the hotel parts
Make up a test design with
Test goals
Test points
14:00
17
Exploratory testing in theory and practice
15:30
18
9
8-5-2019
Exploratory testing in theory and practice
Session based testing
13:30
19
Exploratory testing in theory and practice
The need for more stucture
Exploratory testing without good guidance is like wandering around a city
looking for cool tourist attractions. It helps to have a guide and to understand
something about your destination. - James Whittaker
20
20
10
8-5-2019
Exploratory testing in theory and practice
Session based testing
Session-based testing is a software test method that aims to combine
accountability and exploratory testing to provide rapid defect discovery, creative on-
the-fly test design, management control and metrics reporting
- Wikipedia -
From
To
Based on ‘Exploratory software testing’ by James Whittaker 21
21
Exploratory testing in theory and practice
Elements session based testing
Sessions (timebox)
Mission
Test charters
Test points
Session notes
Session debrief
Metrics
22
22
11
8-5-2019
Exploratory testing in theory and practice
Session
Timebox
60 min. – 90 min. – 120 min.
Short enough to report in detail
Short enough to plan it easily
Long enough to test in depth
Long enough to debrief effectively
Don’t make a detail planning of the session
Eventually pair testing
Tester – tester
Tester – product owner
Tester – user
Tester – developer
23
23
Exploratory testing in theory and practice
Test mission
Representation of the goal of the test session
Risk based
Defined scope
My mission is to test <<the risk you want to cover>>
for <<the scope/depth/coverage>>
24
24
12
8-5-2019
Exploratory testing in theory and practice
Test charter
Also named a session sheet
Information of the test item
Mission
Test points
Information about the test execusion (optional)
Defects (or references to the defects in the tool)
New test charters or new test points
See example
25
25
Exploratory testing in theory and practice
Session notes / session ideas
Things you want to know
Things you don’t want to miss
Things you want to test
≈ logical test cases
A good test point starts with a verb
26
26
13
8-5-2019
Exploratory testing in theory and practice
Session notes
What did I test?
What didn’t I test?
Why did I test it?
How did I test it?
With which coverage and depth did I test it?
Which test data did I use?
On which environment did I test?
What have I found out?
What defects did I find?
What more should I test?
27
27
Exploratory testing in theory and practice
Possible follow up items
Defects
Questions
Possible new charters
Possible new points
Tests that should be automated
Tests that should be documented
Instructions for the next time this charter will be executed
28
28
14
8-5-2019
Exploratory testing in theory and practice
Session debrief
Mission
Risks Coverage
Techniques Environment
Test session
Status Defects Obstacles
29
29
Exploratory testing in theory and practice
Important skill of an exploratory tester
Focus - defocus
Based on RST 3.0 by James Bach and Michael Bolton
30
30
15
8-5-2019
Exploratory testing in theory and practice
It is finding the balance between
Discipline - freedom
31
31
Exploratory testing in theory and practice
Choosing the right way
to test
16:00
32
16
8-5-2019
Exploratory testing in theory and practice
Situational testing bingo
Scripted Exploratory
Category Situation testing testing
System Calculations
User interface oriented system
Backend oriented system
Mobile app
Test goals Checking a set of req. / regulation
Value based testing
Usability
Business rules testing
Performance
Automated checking
Security
Organization Planning and preperation oriented
Young, modern startup
Hierarchical, traditional organisation
Self management, resp. low in org. 33
33
Exploratory testing in theory and practice
Scripted Exploratory
Category Situation testing testing
Documentation A lot and detailed documentation
Little documentation
Constantly changing doc./req.
System Waterval
Agile
Budget A lot of budget
Little budget
Time Early involved
Late involved
A lot of time available
Little time available
Test skills Analytical, accurate testers
Critical thinking (question everything)
Flexible
Professional testers
Non-professional testers 34
34
17
8-5-2019
Exploratory testing in theory and practice
Testing is exploration and experimentation
Who doesn’t like that?
In case you are interested in a career in QA or testing, please contact me.
All the best with your studies
35
35
18
