Page 1
Early fault detection with
model-based testing
ACM SIGPLAN Erlang Workshop 2008
Jonas Boberg
Victoria, British Columbia, Canada, September 27, 2008
Erlang Training and Consulting Ltd
Page 2
Copyright 2008
Agenda
Model-based testing
Research approach
Results
Challenges and recommendations
Conclusions
Page 3
Model-based testing
The whats and whys
Model-based testing
Research approach
Results
Challenges and
recommendations
Conclusions
Page 4
Copyright 2008
“Traditional” test case design
Specification
Test case
Page 5
Copyright 2008
Trend: increasing complexity
Increasingly complex requirements on interaction between
systems
Difficult to achieve high coverage with hand-crafted test cases
Page 6
Copyright 2008
Model-based testing (MBT)
Specification
Test case
Model
+ “Informal requirements”
Page 7
Copyright 2008
Steps of model-based testing
Build an abstract model of system
Validate the model
Generate and execute test cases
Assigning pass/fail verdict
Analyzing the execution result.
Page 8
Copyright 2008
Steps of model-based testing - automation
Can be automated?
Build an abstract model of system
Validate the model
Generate and execute test cases
Assigning pass/fail verdict
Analyzing the execution result.
Page 9
Copyright 2008
Model-based testing – with QuickCheck
Specification
-module(imap_eqc).
-behaviour(eqc_statem).
command(S) ->
…
precondition(State, {call, _, select, [CPid,_]}) ->
min_state(client(CPid,State), ?AUTH)
postcondition(State, {call, _, select, [CPid,_]}, Result) ->
is_status(Result, ?OK_RESP);
eqc:quickcheck( … )
eqc:sample( … )
Test case
Page 10
Copyright 2008
Why is MBT feasible?
Only small part of system-code typically concerned with
functionality
Functionality: 10-30%
Glue
Concurrency
Fault-tolerance
Distribution
…
Page 11
Copyright 2008
Why is MBT feasible?
You don’t model the full system
Abstractions are applied when modeling
Page 12
Copyright 2008
Model-based testing advantages
•Fault prevention
•Reduced cost of updating test cases
Page 13
Copyright 2008
Model-based testing in industry
AGEDIS Case Studies (France Telecom, Intrasoft and IBM)
Dalal et al.
Identifies organizational obstacles to practice introduction
Model-basted testing must be well integrated in the test process
Page 14
Copyright 2008
Model-based testing in industry
Pretschner et al
“Model-based test cases does not detect more faults than hand-crafted test cases”
“Tests were executed after the system was completely implemented”
Artho et al.
component level testing of NASAs K9 planetary rover
“the testing was conducted after the implementation of the system was finished”
Page 15
Research approach
How we applied model-based system testing to a
large scale Erlang system
Model-based testing
Research approach
Results
Challenges and
recommendations
Conclusions
Page 16
Copyright 2008
Project: Messaging Gateway
Study executed as a process improvement initiative
Goal: reduce fault-slip-through to acceptance testing
System test
Customer
acceptance test
Unit test Integration test
Page 17
Copyright 2008
Costs of faults-slip-through
-> Higher cost of tracking and fixing faults
-> Reduced confidence in system
-> Build and deploy additional release candidates
Page 18
Copyright 2008
Boehm curve
* Boehm, Software Engineering Economics,Prentice Hall PTR, 1981
Page 19
Copyright 2008
Central question
How can model-based testing be applied at
the system-level, to enable early fault-detection
and increased confidence in the system?
Page 20
Copyright 2008
Messaging gateway system
IMAP Front-end
Wireless Village front-end
POP3 Backend IMAP Backend
Protocol
Backend
Shared components
E-mail gateway (EMGW)
Instant messaging gateway (IMGW)
Protocol
Backend
Protocol
Backend
Page 21
Copyright 2008
Test approach
Client
Client
Client
E-mail
gateway
POP3
IMAP
Page 22
Copyright 2008
Test approach
QuickCheck
Abstract State
Machine model
IMAP
Client
E-mail
gateway
POP3
IMAP
Page 23
Copyright 2008
Hypothesis: Model-based will result in lower
fault-slip through from system testing to customer
acceptance testing
Page 24
Copyright 2008
Study timeline
Three studied releases:
Release X
Release X+0.5
Release X+1
Page 25
Copyright 2008
Model-based testing in iterative development
For each iteration:
Select requirements
Model requirements
Validate and execute
Page 26
Copyright 2008
Fault-slip-through measurement
System test
Customer
acceptance test
Unit test Integration test
Fault found here
Should have
been found here
Page 27
Results
Model-based testing
Research approach
Results
Challenges and
recommendations
Conclusions
Page 28
Copyright 2008
Results – Release X
EMGW
IMGW
Page 29
Copyright 2008
Results – Release X+1
EMGW
IMGW
7
12
Page 30
Copyright 2008
Phase Input Quality
Phase Input Quality (PIQ) =
Should have been found on earlier level
Total found on level
Page 31
Challenges and recommendations
(As usual,) there is no silver bullet
Model-based testing
Research approach
Results
Challenges and
recommendations
Conclusions
Page 32
Copyright 2008
Challenges
Finding suitable abstractions is difficult
We cannot execute partial tests
Page 33
Copyright 2008
Recommendations
Iterative model development crucial
Test techniques are complementary
Page 34
Copyright 2008
Challenges and recommendations
A substantial initial investment is required to integrate the
model-based testing into the test process
Visibility of results crucial to success
both internal and external
Page 35
Conclusions
Model-based testing
Research approach
Results
Challenges and
recommendations
Conclusions
Page 36
Copyright 2008
Conclusions
Significantly more faults found by model-based testing
Results supports our hypothesis - model-based testing decreased
the fault-slip-through
Page 37
Copyright 2008
Questions?
Page 38
Copyright 2008
Agenda
Model-based testing
Research approach
Results
Challenges and recommendations
Conclusions