View
220
Download
0
Category
Tags:
Preview:
Citation preview
bEgInSlIdE
2
Software Life Cycle
Sommerville , 1992: Development efforts are typically distributed as follows:
Specifications / Design 30% - 40%
Implementation 15% - 30%
Testing 25% - 50%
bEgInSlIdE
3
Remarks by Bill Gates17th Annual ACM Conference on Object-Oriented Programming,
Seattle, Washington, November 8, 2002
“… When you look at a big commercial software company like
Microsoft, there's actually as much testing that goes in as
development. We have as many testers as we have developers.
Testers basically test all the time, and developers basically are
involved in the testing process about half the time…
… We've probably changed the industry we're in. We're not in
the software industry; we're in the testing industry, and writing
the software is the thing that keeps us busy doing all that
testing.”
bEgInSlIdE
4
Remarks by Bill Gates (cont’d)
“…The test cases are unbelievably expensive; in fact,
there's more lines of code in the test harness than
there is in the program itself. Often that's a ratio of
about three to one.”
“… Well, one of the interesting questions is, when you
change a program, … what portion of these test cases
do you need to run?“
bEgInSlIdE
5
The V-model of development
Requirementsspecification
Systemspecification
Systemdesign
Detaileddesign
Module andunit codeand tess
Sub-systemintegrationtest plan
Systemintegrationtest plan
Acceptancetest plan
ServiceAcceptance
testSystem
integration testSub-system
integration test
Implementation and unit testing
bEgInSlIdE
7
Specification-based testing Reflects true
intention of testing
Does not propagate errors from previous versions
Regression testing Does not need a
specification
Easy to implement
Finds subtle errors
Testing methodologies
bEgInSlIdE
8
How shall we check the I/O relation ? Manually (specification-based)
Table of expected results (specification-based)
Compare results to previous version (Regression testing)
Input OutputSystem under test
Black Box Testing(Behavioral testing)
bEgInSlIdE
9
Testing Input-Output relationships only
Pros
• This is what the product is about.
• Implementation independent.
Cons
• For complicated products it is hard to identify erroneous output.
• It is hard to estimate whether the product is error-free.
• Practically: Choosing input with high probability of error detection is very difficult (e.g. division of two numbers).
Black Box Testing(Behavioral testing)
Input Output
bEgInSlIdE
10
White Box Testing(Operational Testing)
Testing how input becomes output (including algorithms) Pros
• Easier to detect errors.
• Enables to find better tests (direct the tests)
• The only way to check coverage.
Cons
• Implementation weaknesses are not necessarily those of the product.
• Code is not always available
bEgInSlIdE
11
Dynamic testing (Run your program) Predefined tests
• Good for Regression Testing (comparing an old version against a new one)
• Testing the product under extreme conditions
Random tests
“real life” tests
Static testing (Inspect your code) Code analyzers (e.g., tools like lint and purify)
Inspection (code review)
Proofs (by tools, or by mathematical arguments)
Static and Dynamic Testing
bEgInSlIdE
12
Dynamic Testing
In combination with black-box testing
In white-box testing:
Preprocessor controlled code
• The only way for digging into the heart of the code
• Code usually outputs the status of some objects.
• Requires modification whenever the code is modified + compilation.
Specification-based monitoring
Other methods …
bEgInSlIdE
13
Special Testing Methods.
Stress Testing
A product that will work under heavy load (e.g, on-line banking system) should be tested under increasing load - much heavier than expected.
bEgInSlIdE
14
Code analysis Unreachable code
Objects declared and never used
Parameters type/number mismatch
Variable used before initialization
Variable is assigned to twice, without using the first value
Function results not used
Possible array bound violations
Static Testing
bEgInSlIdE
15
Code inspection
Self - The default choice.
• Subtle errors and micro-flaws may be overlooked.
• Wrong conceptions propagate to review…
Code review by others - Very efficient !
Static Testing
bEgInSlIdE
16
One more quote…
Dijkstra:
“Testing can only prove the existence of bugs, not their absence…”
bEgInSlIdE
17
In general – it is undecidable, i.e. can’t be done.
In most cases it is possible, but with manual assistance – the same way we would prove a math theorem.
In some cases properties of software can be proved automatically.
Chances for errors increase with length of text• Write short code (e.g, divide into more functions).
• Provide short proofs for correctness (even if they are informal).
… So why not try to prove correctness?
bEgInSlIdE
18
Estimate how clean is your software
Error Implantation (For measuring the effectiveness of testing)
Introduce errors.
See how many of them are detected.
This gives us an “educated guess” about testing quality.
bEgInSlIdE
19
Estimate how much of the software’s behavior is covered
Coverage is a mean to estimate how rigorous is the testing effort
We can use coverage information in order to guide the process of test generation (some times even automatically)
bEgInSlIdE
20
Statement CoverageExample 1
int a, b, sum;
int list1[10] = {00, 11, 22, 33, 44, 55, 66, 77, 88, 99};
int list2[10] = {99, 88, 77, 66, 55, 44, 33, 22, 11, 00};
cin >> a >> b;
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
cout << sum << "\n";
bEgInSlIdE
21
Statement CoverageExample 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
Statement coverage may be achieved by __ test case(s):
bEgInSlIdE
22
Statement CoverageExample 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
But statement coverage may not cater for all conditions
such as when a and b are beyond the array size.
bEgInSlIdE
23
Branch CoverageSame Example 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
Branch coverage may be achieved by __ test cases:
bEgInSlIdE
24
branch coverageExample
switch (x){
case 1: x = 1; break;
case 2: switch (y){
case 1: x = 3; break;
case 2: x = 2; break;
otherwise: x = 1; }
otherwise: 4;}
branch coverage may be achieved by __ test cases
bEgInSlIdE
26
if (a >= 0 && a <= 9)
sum = list1[a];
else sum = 0;
if (b >= 0 && b <= 9)
sum = sum + list2[b];
else sum = 0;
Path coverage may be achieved by __ test cases:
Path CoverageSame Example 2
bEgInSlIdE
27
Path coverage
subsumes Branch coverage
subsumes Statement coverage
Subsumption Relationships
But can wealways demandpath coverage?
bEgInSlIdE
28
Branch CoverageExample 3
if (a >= 0 && a <= 9)
sum = list1[a];
else sum = 0;
if (b >= 0 && b <= 9)
sum = sum + list2[b];
else sum = 0;
...
if (z >= 0 && z <= 9)
sum = sum + list26[b];
else sum = 0;
Branch coverage may be achieved by __ test cases
bEgInSlIdE
29
Path CoverageSame Example 3
if (a >= 0 && a <= 9)
sum = list1[a];
else sum = 0;
if (b >= 0 && b <= 9)
sum = sum + list2[b];
else sum = 0;
...
if (z >= 0 && z <= 9)
sum = sum + list26[b];
else sum = 0;
Path coverage may be achieved by __ test cases
bEgInSlIdE
30
Statement CoverageExample 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Statement coverage may be achieved by __ test cases:
bEgInSlIdE
31
Branch CoverageSame Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Branch coverage may be achieved by __ test cases:
bEgInSlIdE
32
Loop coverage
Skip the loop entirely
Only 1 pass through the loop
2 passes through the loop
n–1, n and n+1 passes through the loop, where n is the maximum number of allowable passes
We only wentthrough theloop once.
bEgInSlIdE
33
Loop CoverageSame Example 4sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Loop coverage may be achieved by __ test cases:
bEgInSlIdE
34
Path CoverageSame Example 4
sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Path coverage may be achieved by __ test cases
bEgInSlIdE
35
Path coverage
subsumes Branch coverage
subsumes Statement coverage
Subsumption Relationships
But can wealways demandpath coverage?
Path coverage
subsumes Loop coverage
bEgInSlIdE
36
Path CoverageExample 5sum = 0;
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1; };
while (b >= 0 && b <= 9) {
sum = list2[a];
b = b + 1; };
...
while (z >= 0 && z <= 9) {
sum = list26[z];
z = z + 1; };
Path coverage may be achieved by __ test cases
bEgInSlIdE
37
Path Coverage is not EverythingExample 1
z = x + y;
Path coverage may be achieved by 1 test case
x = 8, y = 0
Cannot detect z = x - y;
x = 2, y = 2
Cannot detect z = x * y;
x = 8, y = 9
Cannot detect
z = 8 + y;
Cannot detect z = x + 9;
We need 2 test cases:
x = 8, y = 9
x = 29, y = 18.
bEgInSlIdE
38
Path Coverage is not EverythingExample 2
int a[10];
if (b > 0)
a[b] = 1;
else …
Same path with b = 5 And b = 12 behave
differently
bEgInSlIdE
39
Condition coverageExample
if (b1 ||(b2 && b3))
a = 1;
else …
Sub-expression coverage may be achieved by __ test cases
Every sub-expression in a predicate should be evaluated to TRUE and FALSE
bEgInSlIdE
40
Multiple condition coverageExample
if (b1 ||(b2 && b3))
a = 1;
else …
Multiple condition coverage may be achieved by __ test cases
Every Boolean combination of sub-expressions in a predicate should be evaluated to TRUE and FALSE
bEgInSlIdE
41
Condition/Branch coverage (MC / DC)
if (!b1 || b2)
a = 1;
else …
MC/DC coverage may be achieved by __ test cases
Union of Condition coverage and Branch coverage
This is an industry standard
bEgInSlIdE
42
Specification-based testing
We will see an example of a system for specification-based testing of real-time applications.
The testing system is called “Logic Assurance”
It monitors the specification, and can also intervene in the execution of the program.
bEgInSlIdE
43
Monitoring
Logic Assurance
SystemUnderDevelopment
Environment Report State
control
Optional
Report stateEvent Reporting
=
control
Overview
bEgInSlIdE
45
A typical specification language: The LA Language (LAL)
Derived from Temporal Logic and C, LAL enables specification of:
Event order.
Relative frequency ("fairness").
Timing demands.
Logical and mathematical operators.
More...
bEgInSlIdE
46
Examples (1/4):
1. Using event names, time directives and messages:
OPEN_DOOR follows CLOSE_DOOR after not more than 10 sec ?: message("Open door is late");
bEgInSlIdE
47
...Examples (2/4)
2. Logical operators and functions:
when time>20: time([CLOSE_DOOR])> time([OPEN_DOOR])+10?: message("CLOSE_DOOR is early");
bEgInSlIdE
48
...Examples (3/4)
3. User-defined functions are used to enhance the language and enable external control:
if JOB_DONE(10) then HEAT(3,5) < 30?: REDUCE_HEAT(5);
bEgInSlIdE
49
...Examples (4/4)
4. Locators are used to scan the log and retrieve event index:
[2nd SEND s.t. Time>=10, packet=5] > 0 ?: REDUCE_HEAT(5);
bEgInSlIdE
50
2. Report events:
From inside the SUD by using the LA Report Facility.
From outside the SUD by using black-box techniques (e.g. OS events)
From the environment (Sensors, etc.)
bEgInSlIdE
51
The LA Report Facility:
The following directive should be inserted where the event occurs in the program:
LA_report_event (
int identifier,
float time-stamp,
user additional data...
)
bEgInSlIdE
52
LA will:
1. Keep an event log.
2. Analyze the rules in real time* (during execution) using the LA Analyzer.
* Off-line analysis is also possible
bEgInSlIdE
53
3. When a rule is violated, do one of the following:
Report the exact place and time the rule was violated.
Invoke a user-defined function.
bEgInSlIdE
54
Demo
A basic communication protocol ("Stop & Wait"):
2 2 23
time-out time-out
Rules:1. Resend packet after time-out…2. Do not resend packet that was acknowledged……
send
1
Ack 1
bEgInSlIdE
55
# event time message index arguments
-- ------ ------- ---------- ------- ------------
1: send 10 A 1 7, 10, 21..
2: Tout 15 A 1 8, 9 ,10..
3: send 16 B 2 20,30,21.. :
Rule 1, event 3: failed to resend packet
‘A’ sent at 10, Ack expected at 13..
Tout for A activated at 15, must send A again..
‘B’ sent at 16, Ack expected at 19..
.
Incoming events: messages:
user screen:
bEgInSlIdE
56
Advantages of real time monitoring:
Tests can be planned, maintained, expanded and applied throughout the development process.
Problems can be detected sooner.
The product is ‘tied’ to the specification.
bEgInSlIdE
57
A generic tool and methodology.
By receiving input from different sources, it enables testing of:
Multiprocessor systems
Dynamic environment systems.
Advantages of real time monitoring:
Recommended