Software Testing-Challenges and Strategies 8/12/2004
Software Testing – Challenges
& Strategies
ht
r ig
p y
Mr. Arunkumar Khannur
Co
Managing Director- QSIT
Quality Solutions for Information Technology Pvt Ltd
A-706 & 707, 7th Floor, Mittal Towers,
I T
[Link], Bangalore-560 001
QS
Email: khannur@[Link]
URL: [Link]
Ph: +91-80-25585386/51134334
The Challenges we cover today…
z Challenge # 1 – What to Test…
z Challenge # 2 – How Much to Test…
z Challenge # 3 – When to Start Testing…
ht
ir g
z Challenge # 4 – Integration Testing
y
z Challenge # 5 –– Only Test Team follows
process
p
Co
z Challenge # 6 – Intelligent Testing
z
I T
Challenge # 7 – Test Execution Management
QS
2
khannur@[Link] 1
Software Testing-Challenges and Strategies 8/12/2004
Challenge #1: What to Test…
z Deciding on What to test is becoming more and
more more strategic.
z WHY??!!
ht
r ig
y
Likability
p
Co
Non-
Functionality
Functionality
Performance Test
T
Usability
I
Feature Tests
Scalability Test
QS
Scenario Test Platform Test
Inter-operability Test
Reliability Test
Challenge #1: What to Test…(Contd.)
z Functional Requirements – more or less clear –
what the system does and why part
z But
ht
ir g
– Performance Requirements – what are the target #
z Transactions/second
z Download times
p y
Co
– Scalability Requirements
– Inter-operability – with which version of platforms,
I T
products, utilities and so on….
QS
4
khannur@[Link] 2
Software Testing-Challenges and Strategies 8/12/2004
Challenge #1: What to Test…(Contd.)
z Implicit Requirements that come up later in the
cycle
– I assume we use JVM [Link]…..
ht
ig
– I assume we are using double encryption algorithms
r
z There are certain requirements for which coding
happens after testing!
p y
Co
– Support Red Hat Linux [Link].x
Support Solaris [Link]
T
–
I
z No need to code anything for these – just test, if we get any
QS
defects, we will fix
Challenge #1: What to Test…(Contd.)
z In summary
– All requirements are not clear
–
ht
For all requirements we do not have inputs
ir g
– For different requirements we should talk to different
stakeholders
–
p y
For those requirements there is no coding, we have
Co
no clue what is happening till we get a defect!
– ……..
I T
QS
6
khannur@[Link] 3
Software Testing-Challenges and Strategies 8/12/2004
Strategy #1a What to Test…
z Be in the “game”
– Be stakeholder of requirements
–
ht
Do some Benchmarks and provide “data” for
ig
performance, scalability requirements
r
– Be prepared to do R&D
z Be “proactive” where required
p y
Co
– Make effective use of “scope” section in your test
plan
–
I T
Communicate details of the plan & strategy in
QS
nutshells rather than full fledged document mails
– Be involved in the project all the time (if not whole of
test team, may be the lead/manager)
Strategy #1b What to Test…
z Have suggestions and solutions apart from
questions and issues – after all Testing is one of
t
the “stakeholder”
h
ir g
z Take initiative and get things done! – rather than
being in the receiving end – move out of the
p y
mode - I test whatever is “stated” – be a relay
Co
partner
I T
QS
8
khannur@[Link] 4
Software Testing-Challenges and Strategies 8/12/2004
Strategy #1 What to Test…Summary
z Visualize Requirements using models
z Models are to understand the context
z Express the “perceived” behavior
ht
ig
z Any for of the model can be used
z
y
Be consistent within project
r
p
Co
z Picture Vs Words
z Mental Models can also be a tool
z
I T
Key is to “gain” product knowledge
QS
9
Challenge #2: How much to Test…
z Deciding on How much to test is becoming
more and more more tricky.
z WHY??!!
ht
ir g
– Increased Abstraction, Reusability
Computer, Content, Communication, Connectivity
y
–
p
– Global Computing, Heterogeneous Resources
Co
– 24/7 Uptime Requirements
I T
QS
10
khannur@[Link] 5
Software Testing-Challenges and Strategies 8/12/2004
Strategy #2 How much to Test…
z Analyze Specified Requirements to understand
– Test Factors
– Testability
ht
ig
– Product Success Criteria
yr
p
T Co
I
QS
11
ht
ir g
p y
T Co
I
QS
12
khannur@[Link] 6
Software Testing-Challenges and Strategies 8/12/2004
Challenge #3: When to Start Testing…
z Who will tell us that its ideal time to start?
z Or Should we decide, if yes who will “certify”
you can start now?
ht
ig
z When will we meet the “Entry Criteria” 100%?
z
yr
What is the guarantee that we needn’t stop if we
p
start now?
Co
z Who will ensure smooth sailing?
I T
z How do we know that this “work around” is
QS
acceptable for now?
z By when we fix all defects, so that we can start
all over again?
13
Strategy #3 When to Start Testing…
z Have a broad Entry Criteria for “formal” start
– Make sure that we don’t end up repeating things…
t
z Start as early as possible in case resource usage is
h
moderately flexible (triple constraints theory)
z
ir g
For entry criteria start tracking the same while we
y
are nearing to “code complete” milestone
p
Co
z Have the ownership of entry criteria – make sure
that we get what we want, rather then wait and see
I T
what we “got”!
QS
14
khannur@[Link] 7
Software Testing-Challenges and Strategies 8/12/2004
Challenge # 4 – Integration Testing
z We all know definitions!
– When it comes to practice especially in an advanced
t
technology environment – this is difficult to strategize
h
ig
z If I do integration testing should I do
r
“component” or “Module” testing?
z
p y
How do I scope between IT and ST?
Co
z Does the architecture permit a top-down or
bottom-up integration?
I T
QS
z Who will own the “integration” so that we can
test it?
15
Strategy #4 – Integration Testing
z Decide upfront whether IT is required, feasible or NOT –
most of the time we do this as a routine in the process
t
than looking for benefit – all products may not need IT!!!
h
z Have a high level Strategy for Unit, Component,
ir g
Integration and System Testing scope
y
z Train your testing team in the related technology – since
p
integration involves a lot of white-box stuff
Co
z Make the project to decide – who, when and how will
“integrate” the product – so that we can test the same
I T
(ideally integration would be a stepwise approach and
QS
each step can be tested separately)
16
khannur@[Link] 8
Software Testing-Challenges and Strategies 8/12/2004
Challenge # 5 – Only Test Team follows process
z We ask for Requirements specification & review
them
z We ask for Design & review them
ht
ig
z We seek review comments closure
z
yr
We write Test Plans and get them reviewed
z
p
We write TC’s and get them reviewed
Co
z We observe “entry criteria”
z
I T
We observe “exit criteria”
QS
17
Challenge # 5 – Only Test Team follows process
(contd.)
z Does this sound crazy?
z Are we the only ones bothered about processes
(spare that poor SQA!!)
ht
ir g
z Do only we need documents – how will they
code – how will they make changes??
z
p y
People are sidelining us saying “you in testing
Co
know only to crib”……
T
z Should we also stop following process?
I
QS
18
khannur@[Link] 9
Software Testing-Challenges and Strategies 8/12/2004
Strategy # 5 – Only Test Team follows process
z This is more of a “mindset” – we should do
things the way we want to, because we believe
t
in them
h
ig
z More of team building
z Awareness building
yr
p
z Bringing the feel that all part of the Project team
Co
– differentiation of we or they should be stopped
T
z Make sure that we go by setting expectations
I
QS
with our peers, other functions and so on…
z Needs some “cultural” orientation…..
19
Challenge # 6 – Intelligent Testing
z Should we be repeating all TC’s in all Cycles – release
after release? What is the basis for TC selection for
t
“re-testing”?
h
z How do we decide on regression test strategy – what is
ir g
the basis?
y
z How effective our regression testing would be?
p
z How much time we may need to regress defects?
Co
z Does it have dependency on the development team?
T
z Who can tell me what to test and what not while
I
regressing defects?
QS
z How do we equip ourselves on deciding TC’s for
regression?
20
khannur@[Link] 10
Software Testing-Challenges and Strategies 8/12/2004
Strategy # 6 – Intelligent Testing
z Have Regression test strategy for each
feature/functionality – this would also depend on the
t
relationship across features – if possible develop a
h
‘n’by’n’ relationship matrix for all features
ig
z Mark Regression TC’s for every feature – irrespective of
r
defects we found in the earlier rounds
z
p y
Make sure that remaining TC’s selection for regression
Co
test happen just before we start the same – depends on
the “defect fix” details – look for possible areas that
I T
might have got impacted – get a developer or designer
QS
support while doing this
z Take intermittent developer builds of the product and
run through some TC’s – decide on the priority for these
for formal cycles based on result
21
Challenge # 7 – Test Execution Management
z We get a build and start the cycle, within 2 days
we get another!! – set up takes half day off all
t
my test engineers!!!
h
ir g
z TC’s passed one build fails in another!
z When we report a defect, we get a reply “try with
p y
this new build” – no details on the fix!
Co
z The more the #of defects, the more the time we
T
need to complete testing
I
QS
z Testing is the last activity – hence all previous
delays get forgotten – onus of the release slip is
getting thrust upon test teams!!!
22
khannur@[Link] 11
Software Testing-Challenges and Strategies 8/12/2004
Challenge # 7 – Test Execution Management (contd.)
z We have no clue when all the defects would get
fixed, so that we can regress!
z
ht
For non-functional testing we can’t change
ig
builds frequently also we need more time to
r
complete scaling up the product and data
z
p y
We find a lot of non-reproducible defects – we
Co
spend a lot of time in this attempt (reproduction)
T
z We find a lot of defects which gets fixed only in
I
the “next” release
QS
z Some time we get orders “roll back” to previous
build and continue testing!!
23
Strategy # 7 – Test Execution Management
z Have a definite plan for build “transition” – make sure
that all your resources doesn’t move from one build to
t
another same day – provide cooking time for each build
h
to be rolled out to entire test set-up
ir g
z Make sure that development provides the exact tasks or
check-in’s that have gone into every new build – with
p y
details which help us in identification of change
Co
z Follow a statistical defect prediction model (preferably
internal) to predict defects – plan the regression cycle
I T
based on the prediction
QS
z Make sure that when the “baton” is passed to you the
relevant delay is accounted – also demark the “bug
fixing” cycles between your test cycles
24
khannur@[Link] 12
Software Testing-Challenges and Strategies 8/12/2004
Strategy # 7 – Test Execution Management (contd.)
z Analyze the defects from the past – educate the
testing team to recognize valid defects and
t
ignore others
h
ig
z Have a criteria for build reproduction – beyond
r
which the defect to be kept under observation –
p y
during slack time we try to reproduce them
Co
z Have a set of minimum details to be assessed
while reporting a defect – this could be even
I T
feature specific – get this list done by Dev team
QS
z Make sure the Fix Next Release is decided
based on an agreeable criteria to you as well –
stakeholder concept!!
25
Other New Challenges
z Application Integrations
z New Technologies
z Open source models
ht
ir g
z Re-use Models
y
z “Knowledge-able” users – a blessing in
disguise??
p
T Co
I
QS
26
khannur@[Link] 13
Software Testing-Challenges and Strategies 8/12/2004
ht
r ig
p y
T Co
I
QS
Summary
27
New Tester’s Sources…..
z Requirements z User Stories
z
+
Functional Specifications z Customer Scenarios
t
z Designs z Defect Database
h
ir g
z Product Descriptions z Benchmark Reports
z Use Cases z Research Reports
p y z Beta Feedbacks
Co
z Customer Support DB
z Developer Communities
I T z International Forums
QS
28
khannur@[Link] 14
Software Testing-Challenges and Strategies 8/12/2004
New Tester’s Career Path/Options
t
Test Engineer
h
r ig
Automation Expert Platform Specialist
Use Case Specialist
p y Environment
Co
Specialist
Test Lead
Test Architect
I T Performance Analyst
Human Factors
Specialist
QS Test Manager
Domain
Specialist
29
New Tester’s Skill sets
z Equivalent to Developers
z Need to know the Programming Skills
z Automation Skills
ht
ir g
z Understand the “User”
y
z Intuitive
p
Co
z Creative
z Know/Appreciate “Systems Thinking”
z ……….
I T
QS
30
khannur@[Link] 15
Software Testing-Challenges and Strategies 8/12/2004
Your Questions Please……..
ht
r ig
p y
T Co
I
QS
31
ht
ir g
p y
T Co
I
QS
THANK YOU
32
khannur@[Link] 16
