Introduction to Software Testing
Chapter 7.2
Engineering Criteria for Technologies
Paul Ammann & Jeff Offutt
http://www.cs.gmu.edu/~offutt/softwaretest/
The Technologies
Chapters 1-5 emphasize criteria on four models of
software
Emphasis in each chapter was first on the criteria, then
on how to construct the models from different software
artifacts
This chapter discusses how to apply the criteria to
specific technologies
– Most of the ideas in this chapter were developed after the year
2000
– Thus they are still evolving
2
Chapter 7 Outline
1. Object-Oriented Software
2. Applications and
Web Applications and Web
Web Services
Services
3. Graphical User Interfaces
4. Real-Time and Embedded Software
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 3
Section 7.2 Outline
Most of these ideas were developed after 2000
Few are widely used
Most adapt graph-based testing from Chapter 2
1. Overview
2. Static Hyper Text Web Sites
3. Dynamic Web Applications
1. Client-side testing
2. Server-side testing
4. Web Services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 4
Issues in Testing Web Software
A web application is a program that is deployed on the web
– Usually uses HTML as the user interface
– Web-deployment means they are available worldwide
– They accept requests through HTTP and return responses
– HTTP is stateless – each request/response pair is independent
Web applications are usually very competitive
A web service is a web-deployed program that accepts XML
messages wrapped in SOAP
– Usually no UI with humans
– Service must be published so other services and applications can discover
them
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 5
Web Software
Composed of independent, loosely coupled software
components
– All communication is through messages
– Web application messages always go through clients
– The only shared memory is through the session object – which
is very restricted
– The definition of state is quite different
Inherently concurrent and often distributed
Most components are relatively small
Uses numerous new technologies, often mixed together
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 6
Deploying Software
Bundled : Pre-installed on computer
Shrink-wrap : Bought and installed by end-users
Contract : Purchaser pays developer to develop and
install, usually for a fixed price
Embedded : Installed on a hardware device, usually with
no direct communication with user
Web : Executed across the Internet through
HTTP
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 7
General Problem
Web applications are heterogeneous, dynamic and must
satisfy very high quality attributes
Use of the Web is hindered by low quality Web sites and
applications
Web applications need to be built better and tested
more
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 8
Problem Parameters
HTTP is a stateless protocol
– Each request is independent of previous request
Servers have little information about where a request
comes from
Web site software is extremely loosely coupled
– Coupled through the Internet – separated by space
– Coupled to diverse hardware devices
– Written in diverse software languages
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 9
Separation of Concerns in Web Apps
Presentation layer
Data content layer
Data representation layer
Data storage layer
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 10
Differences in Testing Web Software
Traditional graphs do not apply
– Control flow graph
– Call graph
State behavior is hard to model and describe
All inputs go through the HTML UI – low controllability
Hard to get access to server-side state (memory, files, database) –
low observability
Not clear what logic predicates can be effectively used
No model for mutation operators on web software
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 11
New Essential Problems of Web Apps
1. Web site applications feature distributed integration and are
extremely loosely coupled
• Internet and diverse hardware / software
2. HTML forms are created dynamically by web applications
• UI created on demand and can vary by user and time
3. Users can change the flow of control arbitrarily
• back button, forward button, URL rewriting, refresh
4. Dynamic integration of new software components
• new components can be added during execution
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 12
Problem 1: Loosely Coupled
server
server server
client
server
Traditional software Web-based software
Connected by calls and message passing Connected with HTTP and XML
High and moderate coupling Loose, extremely loose, distributed coupling
How can we ensure the reliability of this type of software?
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 13
Extremely Loose Coupling
Tight Coupling : Dependencies among the methods are encoded
in their logic
– Changes in A may require changing logic in B
Loose Coupling : Dependencies among the methods are
encoded in the structure and data flows
– Changes in A may require changing data uses in B
Extremely Loose Coupling (ELC) : Dependencies are encoded
only in the data contents
– Changes in A only affects the contents of B’s data
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 14
Problem 2: Dynamic Flow of Control
WebPics WebPics
How you’ns doin’ Jeff Offutt! Huan ying guang ling, Wang Shuang!
Search Search
Recommended Movies Recommended Movies
X XX XXX A B C D
Examine queue (Warning: Queue empty) Examine queue
View account View account
Frequent customer bonus
How can we ensure the reliability of this type of system?
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 15
Dynamic Execution of Web Apps
Parts of the program are generated dynamically
Dynamic web pages are created when users make
requests
Different users will see different programs !
The potential control, ala the traditional control flow
graph, cannot be known ahead of time
The potential flow of control cannot be known statically
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 16
Problem 3: User Control Flow
Users can make unexpected changes to the flow of control
– Back buttons, refreshing, caching, forward button, URL hacking
State is stored in the server and in the HTML in the client’s
browser
Operational transitions : Transitions NOT based on an HTML link
or program statement: back, forward, URL rewriting, refresh
These transitions can cause unanticipated changes to the state of
the web application
How can we ensure the reliability of this type of software?
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 17
Problem 4: Dynamic Integration
Software modules can dynamically integrate with others if they
use the same data structures
EJBs can be inserted into web applications, which can
immediately start using them
Web services find and bind to other web services dynamically
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 18
Section 7.2 Outline
1. Overview
Text Web
2. Static Hyper Text WebSites
Sites
3. Dynamic Web Applications
1. Client-side testing
2. Server-side testing
4. Testing Web Services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 19
Testing Static Hyper Text Web Sites
This is not program testing, but checking that all the HTML
connections are valid
The main issue to test for is dead links
We should also evaluate
– Load testing
– Performance evaluation
– Access control issues
The usual model is that of a graph
– Nodes are web pages
– Edges are HTML links
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 20
Section 7.2 Outline
1. Overview
2. Static Hyper Text Web Sites
3. Dynamic Web Applications
Applications
1. Client-side testing
2. Server-side testing
4. Testing Web Services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 21
Testing Dynamic Hyper Text Web Sites
The user interface is on the client
Some software is on the client (scripts such as
Javascript)
Most software is on the server
Client-side testing does not access source or state on the
server
Server-side testing can use the source or the server state
Introduction to Software Testing (Ch 7.2) 22
Section 7.2 Outline
1. Overview
2. Testing Static Hyper Text Web Sites
3. Testing Dynamic Web Applications
1. Client-side testing
2. Server-side testing
4. Testing Web Services
Introduction to Software Testing (Ch 7.2) 23
Client-Side (Black-Box) Testing
The UI and the software are on separate computers
The inputs to web software are defined by the HTML form
elements
– Text boxes, buttons, dropdown lists, links, etc
Techniques for generating values
– Supplied by the tester
– Generated randomly
– User session data – data collected from previous users of the software
Choosing values
– Bypass testing – values that violate constraints on the inputs, as defined by
client-side information
The problem of finding all screens in a web application is
undecidable
Introduction to Software Testing (Ch 7.2) 24
Test Value Selection
Challenge
– How to automatically provide effective test values ?
Semantic Domain Problem (SDP)
– Values within the application domain are needed
– Enumeration of all possible test values is inefficient
Possible solutions
– Random values (ineffective)
– Automatically generated values (very hard!)
– User data (incomplete)
– Study application and construct a set of values (feasible)
– Tester-supplied inputs (feasible but expensive)
Introduction to Software Testing (Ch 7.2) 25
Web Application Input Validation
Bad Data
Check data • Corrupts data base
Check data • Crashes server
• Security violations
Sensitive
Data
Client
Server
Malicious
Data
Can “bypass”
data checking
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 26
Bypass Testing
“bypass” client-side constraint enforcement
Bypass testing constructs tests to intentionally
violate constraints :
– Eases test automation
– Validates input validation
– Checks robustness
– Evaluates security
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 27
User Name: Age:
Version to purchase:
Small Medium Large
$150 $250 $500
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 28
Invalid data, please correct …
User Name: Alan<Turing Age: 500
Username should be Age should be
plain text only. between 18 and 150.
Version to purchase:
Small Medium Large
$150 $250 $500
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 29
Abbreviated HTML
<FORM >
<INPUT Type=“text” Name=“username” Size=20>
<INPUT Type=“text” Name=“age” Size=3 Maxlength=3>
<P> Version to purchase:
…
<INPUT Type=“radio” Name=“version” Value=“150” Checked>
<INPUT Type=“radio” Name=“version” Value=“250”>
<INPUT Type=“radio” Name=“version” Value=“500”>
<INPUT Type="submit" onClick="return checkInfo(this.form)">
<INPUT Type=“hidden” isLoggedIn=“no”>
</FORM>
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 30
Bypass Behavior
Extremely loose coupling …
combined with the stateless protocol …
allows users to easily bypass client-side checking :
Users can save and modify the HTML
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 31
Saved & Modified HTML
<FORM >
<INPUT Type=“text” Name=“username” Size=20>
<INPUT Type=“text” Name=“age” Size=3 Maxlength=3>
<P> Version to purchase: Allows an input with arbitrary age,
no checking, cost=$25 …
…
‘<‘ can crash an XML parser
<INPUT Type=“radio” Name=“version” Value=“150”>
Text fields can have SQL statements
<INPUT Type=“radio” Name=“version” Value=“250”>
25
<INPUT Type=“radio” Name=“version” Value=“500” Checked>
<INPUT Type=“submit” onClick=“return checkInfo (this.form)”>
yes >
<INPUT Type=“hidden” isLoggedIn= “no”
</FORM>
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 32
SQL Injection
User Name: turing ’ OR ‘1’=‘1 Password: enigma ’ OR ‘1’=‘1
Original SQL:
SELECT username FROM adminuser WHERE
username=‘turing’ AND password =‘enigma’
“injected” SQL:
SELECT username FROM adminuser WHERE username=‘turing’
OR ‘1’ = ‘1’ AND password =‘enigma’ OR ‘1’ = ‘1’
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 33
Bypass Testing
This example illustrates how users can “bypass” client-
side constraint enforcement
Bypass testing constructs tests to intentionally violate
constraints
– Eases test automation
– Checks robustness
– Evaluates security
Preliminary results
– Rules for constructing tests
– Successfully found errors in numerous Web apps
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 34
Applying Bypass Testing
Validating input data on the client is like asking
your opponent to hold your shield in a sword fight
Analyze HTML to extract each form element
Model constraints imposed by HTML and JavaScript
Rules for data generation :
– From client-side constraints
– Typical security violations
– Common input mistakes
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 35
Types of Client Input Validation
Client side input validation is performed by HTML form
controls, their attributes, and client side scripts that access
DOM
Validation types are categorized as HTML and scripting
– HTML supports syntactic validation
– Client scripting can perform both syntactic and semantic validation
HTML Constraints Scripting Constraints
• Length (max input characters) • Data Type (e.g. integer check)
• Value (preset values) • Data Format (e.g. ZIP code format)
• Transfer Mode (GET or POST) • Data Value (e.g. age value range)
• Field Element (preset fields) • Inter-Value (e.g. credit # + exp. date)
• Target URL (links with values) • Invalid Characters (e.g. <,../,&)
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 36
Example Client-Side Constraint Rules
Violate size restrictions on strings
Introduce values not included in static choices
– Radio boxes
– Select (drop-down) lists
Violate hard-coded values
Use values that JavaScripts flag as errors
Change “transfer mode” (get, post, …)
Change destination URLs
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 37
Example Server-Side Constraint Rules
Data type conversion
Data format validation
Inter-field constraint validation
Inter-request data fields (cookies, hidden)
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 38
Example Security Violation Rules
Potential Illegal Character Symbol
Empty String
Commas ,
Single and double quotes ’ or ”
Tag symbols Tag symbols < and >
Directory paths .. ../
Strings starting with forward slash /
Strings starting with a period .
Ampersands &
Control character NIL, newline
Characters with high bit set 254 and 255
Script symbols <javascript> or <vbscript>
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 39
Test Value Selection
Challenge:
– How to automatically provide effective test values?
Semantic Domain Problem (SDP)
– Values within the application domain are needed
– Enumeration of all possible test values is inefficient
Possible Solutions
– Random Values (ineffective – lots of junk)
– Automatically generated values (very hard)
– Taking values from session log files (feasible but incomplete)
– Tester input (feasible)
Our tool used an input domain created by parsing the
interface and tester input
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 40
Real-World Examples
atutor.ca nytimes.com bankofamerica.com
Atalker Us-markets ATM locator, Site search
demo.joomla.or mutex.gmu.edu comcast.com
Poll, Users Login form Service availability
phpMyAdmin yahoo.com ecost.com
Main page, Notepad, Composer, Detail submit,
Set Theme, Search reminder, Shopping cart control
SQL Query, Weather Search
DB Stats google.com
barnesandnoble.com Froogle, Language tools
brainbench.com
Pure black-box testing
Submit Request
Cart manager,
pageflakes.com
Book search/results
Info, New user
means
amazon.com
Registration
no
myspace.com source
Events & Music
(or
Item permission)
dispatch, needed !
wellsfargolife.com
Handle buy Quote search
Search
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 41
Output Checking
(V) Valid Responses : invalid inputs are adequately processed by
the server
(V1) Server acknowledges the invalid request and provides an explicit
message regarding the violation
(V2) Server produces a generic error message
(V3) Server apparently ignores the invalid request and produces an
appropriate response
(V4) Server apparently ignores the request completely
(F) Faults & Failures : invalid inputs that cause abnormal server
behavior (typically caught by web server when application fails to
handle the error)
(E) Exposure : invalid input is not recognized by the server and
abnormal software behavior is exposed to the users
These do not capture whether the valid responses corrupted data
on the server
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 42
Results
Introduction to Software Testing
© Ammann and Offutt 43
Section 7.2 Outline
1. Overview
2. Testing Static Hyper Text Web Sites
3. Testing Dynamic Web Applications
1. Client-side testing
2. Server-side testing
4. Testing Web Services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 44
Server-Side (White-Box) Testing
If we have access to the source on the server, we can try to model
the web software
Many testing criteria on non-web software rely on a static
control flow graph
– Edge testing, data flow, logic coverage, …
– Also slicing, change impact analysis, …
The standard control flow graph cannot be computed for web
applications !
But all the pieces of the web pages and the programs are
contained in the software presentation layer …
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 45
Atomic Sections :
HTML with static structure and content variables
PrintWriter out = response.getWriter();
P1 = out.println ("<HTML>")
out.println ("<HEAD><TITLE>" + title
title + "</TITLE></HEAD>)"
out.println ("<BODY>")
if (isUser) {
Atomic P2 = out.println (“ <CENTER> Welcome!</CENTER>");
sections for (int i=0; i<myVector.size(); i++)
if (myVector.elementAt(i).size > 10)
P3 = out.println (“<p><b>" + myVector.elementAt(i) + "</b></p>");
else
P4 = out.println (“<p>" + myVector.elementAt(i) + "</p>");
Empty } else
atomic P5 = { }
Content
section P6 = out.println (“</BODY></HTML>"); variables
out.close ();
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 46
Atomic Sections
A section of HTML with the property that if any part of
the section is sent to a client, the entire section is
– May include JavaScript
– All or nothing property
An HTML file is an atomic section
Content variable : A program variable that provides
data to an atomic section
Atomic sections may be empty
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 47
Modeling Web Applications
Restricted to the presentation layer only
Two levels of abstraction
1. Component Interaction Model (CIM)
– Models individual components
– Combines atomic sections
– Intra-component
2. Application Transition Graph (ATG)
– Each node is one CIM
– Edges are transitions among CIMs
– Inter-component
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 48
Component Expressions
Atomic sections are combined to model dynamically
generated web pages
Four ways to combine:
1. Sequence : p1 p2
2. Selection : (p1 | p2)
3. Iteration : p1*
4. Aggregation : p1 {p2}
– p2 is included inside of p1
The previous example produces:
p p1 (p2 (p3 | p4)* | p5) p6
Composite sections can be produced automatically
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 49
Modeling Component Transitions
Five types of transitions
1. Simple Link Transition : An HTML link (<A> tag)
2. Form Link Transition : Form submission link
3. Component Expression Transition : Execution of a software
component causes a component expression to be sent to the client
4. Operational Transition : A transition out of the software’s control
• Back button, Forward button, Refresh button, User edits the URL,
Browser reloads from cache
5. Redirect Transition : Server side transition, invisible to user
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 50
Component Interaction Model : gradeServlet
ID = request.getParameter ("Id");
passWord = request.getParameter ("Password");
retry = request.getParameter ("Retry");
PrintWriter out = response.getWriter();
P1 = out.println (“<HTML> <HEAD><TITLE>" + title + "</TITLE></HEAD><BODY>)"
if ((Validate (ID, passWord)) {
P2 = out.println (“ <B> Grade Report </B>");
for (int I=0; I < numberOfCourse; I++)
P3 = out.println(“<p><b>" + courseName (I) + "</b>“ + courseGrade (I) + “</p>”);
} else if (retry < 3) {
P4 = retry++;
out.println ("Wrong ID or wrong password");
out.println ("<FORM Method=\“get\" Action=\"gradeServlet\">);
out.println ("<INPUT Type=\“text\" Name=\"Id\" Size=10>");
out.println ("<INPUT Type=\“password\" Name=\"Password\" Width=20>");
out.println ("<INPUT Type=\“hidden\" Name=\"Retry\" Value=" + (retry) + ">");
out.println ("<INPUT Type=\“submit\" Name=\“Submit\" Value=\“submit\">");
out.println ("<A Href=\"sendMail\">Send mail to the professor</A>");
} else if (retry >= 3) {
P5 = out.println (“<p>Wrong ID or password, retry limit reached. Good bye.") }
P6 = to Software
out.println(“</BODY></HTML>");
Introduction Testing (Ch 7.2)
© Ammann and Offutt 51
CIM for gradeServlet
S = login.html
A = {p1, p2, p3, p4, p5, p6 }
CE = gradeServlet = p1 • ((p2 • p3* ) | p4 | p5) • p6
T = {login.html gradeServlet [get, (Id, Password, Retry)],
gradeServlet.p4 sendMail [get, ()],
gradeServlet.p4 gradeServlet [get, (Retry)] }
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 52
Application Transition Graph
Γ : Finite set of web components
Θ : Set of transitions among web software
components
– Includes type of HTTP request and data
Σ : Set of variables that define the web application
state
α : Set of start pages
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 53
ATG for gradeServlet
Γ = { login.html, gradeServlet, sendMail, syllabus.html }
Θ = {login.html syllabus.html [get, ()],
login.html gradeServlet [get, (Id, Password, Retry)],
gradeServlet.p4 sendMail [get, ()],
gradeServlet.p4 gradeServlet [get, (Retry)] }
Σ = { Id, Password, Retry }
α = { login.html }
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 54
ATG for gradeServlet
get () get ()
login.html syllabus.html
get
(Id, Password, Retry) get (Id, Password, Retry)
gradeServlet p1
p2
p4 p5 get ()
sendMail
p3
p6
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 55
Atomic Section Modeling
Atomic sections provide a fundamental mechanism to
model Web applications presentation layer
Can handle :
– Distributed integration
– Dynamically created HTML pages
– Operational transitions
Requires deep analysis of software source
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 56
Some Current Open Questions
How to define data flow?
– DU-pairs cannot be determined statically – uses cannot
always be found
Automatically generating ATG
Issues not handled:
– Session data
– Multiple users
– Concurrency
– Input data
– Output validation
– Dynamic integration
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 57
Section 7.2 Outline
1. Overview
2. Static Hyper Text Web Sites
3. Dynamic Web Applications
1. Client-side testing
2. Server-side testing
4. Testing Web
Web Services
Services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 58
Web Services
A Web Service is a program that offers services over the
Internet to other software programs
– Internet-based
– Uses SOAP and XML
– Peer-to-peer communication
Web service components can integrate dynamically, by
finding other services during execution
Web services transmit data that are formatted in XML
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 59
Web Service Architecture
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 60
Web Service Technologies
Find UDDI WSDL
Applications
SOAP / XML Registr Points to URL Specification
y
SOAP / XML Specification Specification
Bind Publish
Wrapped Wrapped Legacy
Components Services
System
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 61
Difficulties of Testing Web Services
Web services are always distributed
Most “peer-to-peer” communication is between services
published by different organizations
– Trust is a major issue holding back the adoption of web services !
Design and implementation are almost never available
Structured messages are transmitted
Most testing research so far has focused on messages
– Syntax-based test criteria have been proposed for Web services
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 62
Conclusions
The Web provides a new way to deploy software
• Web applications:
– offer many advantages
– use many new technologies
– introduce fascinating new problems
• Web software engineering is just starting
• Two very useful techniques:
– Atomic sections : A fundamental model
– Bypass testing : Easy to automate – no source
needed
This is a very active research area
Introduction to Software Testing (Ch 7.2)
© Ammann and Offutt 63