Testing
● Testing is the process of running a program with the intent of finding bugs
● Requires running the program on sample data● Might require writing extra code to emulate
missing parts of the system● Requires the programmer to be "egoless"● Often performed by programmers other than the
developers
Types of Testing
● Static: Done without actually executing program– Code inspections– Walkthroughs
● Dynamic: Done by executing program or its parts– Module or unit testing– Integration testing– System testing
Module or Unit Testing
● Tests individual classes or small sets of classes● Generally done by the programmer responsible
for the classes● Bottom-up process● May require the use of a throw-away test driver
program
Test Driver for FarmerStateInfoint main() {
Side farmer = EAST; Side wolf = EAST; Side goat = WEST; Side cabbage = WEST;
FarmerState s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();
cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test wrong" << endl; else cout << "Safety test OK" << endl;
farmer = WEST; s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();
cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test OK" << endl; else cout << "Safety test wrong" << endl;}
Test Driver Output
67% farmer ||F ||WG|| C||
Testing safety:Safety test OKF|| ||WG|| C||
Testing safety:Safety test OK68%
Notes on the Output
● What gets tested:– The constructor– The display() method– Safety of the cabbage, both when it's vulnerable and
when it's not– The safety of the goat when it's not vulnerable
● What does not get tested:– The safety of the goat when it is vulnerable
● Note: the isSafe() method had to be made temporarily public
Testing State-Changing Methodsint main() {
FarmerState s = new FarmerStateInfo(WEST, WEST, WEST, WEST); s->display();
State newst = FarmerStateInfo::self(s); cout << "Action farmer-takes-self " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;
newst = FarmerStateInfo::wolf(s); cout << "Action farmer-takes-wolf " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; . . .
newst = FarmerStateInfo::cabbage(s); cout << "Action farmer-takes-cabbage " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;}
State-Changing Test Output
69% farmerF|| W|| G|| C||
Action farmer-takes-self not allowed
Action farmer-takes-wolf not allowed
Action farmer-takes-goat ||FW|| ||GC||
Action farmer-takes-cabbage not allowed
70%
Testing FarmerProblemInfo and State Equality
int main() {
Problem p = new FarmerProblemInfo(); State start = p->getStartState(); State final = p->getFinalState();
cout << "Testing state display:" << endl;
start->display(); final->display();
cout << "Testing state equality:" << endl;
if ( start->equals(final) ) cout << "Equality check wrong" << endl; else cout << "Equality check OK" << endl;
if ( start->equals(start) ) cout << "Equality check OK" << endl; else cout << "Equality check wrong" << endl;
}
Equality Test Output71% farmerTesting state display:F|| W|| G|| C||
||F ||W ||G ||C
Testing state equality:Equality check OKEquality check OK72%
Note that the test driver could be used for otherproblems just by changing the constructor call.
Testing expand()
int main() {
Problem p = new FarmerProblemInfo(); State start = p->getStartState(); start->display();
Item testItem = new ItemInfo(start, NULL, NULL, 0, 0); ItemArray children = testItem->expand(p); cout << "Num children: " << testItem->getNumChildren() << endl;
for (Integer i = 0; i < testItem->getNumChildren(); i++) { State nextState = children[i]->getState(); nextState->display(); }
}
Integration Testing
● Putting together the results of module testing● Top-down approach● Example: testing the SolverInfo class tests
the integration of ProblemInfo and QueueInfo
● Often includes a primitive graphical user interface (GUI) so that:– system has look and feel of final product– separate drivers not needed
● Might require stubs or dummy routines for code to be added later
System Testing● Attempts to find problems in a complete program● Done when
– program is first completed, and– as the program evolves (maintenance testing)
● In industry, often done by an independent group of programmers
● Requires creation of a test suite that:– is used over and over as program evolves– attempts to identify all combinations of inputs and
actions that could cause program to fail
Testing and Proof of Correctness
● It is not possible to test a nontrivial system with all possible inputs and outputs
● So testing cannot formally prove that a system is correct, only give some level of confidence in it
● Formal proofs of dynamic program correctness are not possible either, because executing programs are subject to physical influences
● Formal proofs of abstract algorithm correctness are possible though difficult, and require machine assistance
Debugging
● The process of fixing problems (bugs) that testing discovers
● Debugging has two parts:– finding the cause of a bug (can be 95% of debugging)– fixing the bug
● A good programmer becomes adept at using a run-time debugger
Types of Program Bugs
● Syntactic bugs not caught by the compiler
Example:for (... ; ... ; ...);{
...
}● Design bugs: easy to find, can be difficult to fix● Logic bugs: can be difficult to find, easy to fix
Types of Bugs (cont'd)
● Interface bugs: – When assumptions made by a method caller are
different than those of the callee. – Can be difficult to find, easy to fix
● Memory bugs:– Freeing or overwriting storage that is later used– Especially difficult to find– Example:
IntArray A; // = new Integer[4]; ...A[0] = 10;A[1] = 20: ...
Fixing Bugs● Fixing might entail correcting a single line of
code● Fixing might entail redesign, recoding, and
further testing:– For example, incorrect circularity check in expand()
● Fixing a bug might cause other bugs to appear● It has been observed that code in which a bug has
been found is more likely to still contain bugs
Defensive Programming
● The easiest way to do debugging is to not have bugs
● Failing that, one should strive for locality:
Ensure that bugs that are identified are local problems found where and when they occur
● Locality is the goal of defensive programming● Defensive programming begins with defensive
design
Defensive Design
● First goal: Simplicity of Design
Example: Take the time and trouble to make a repeated piece of code a method, even if small
● Also: Simplicity of Class Interfaces
Example: refactoring the abstract StateInfo and ProblemInfo classes
● Another goal: Program for Errors– Anticipate exceptions even when they are not
expected– Example: add(Item) and remove() should check
for fullness and emptiness even though currently all callers check first
Program Robustness
void FrontQueueInfo::add(Item item) { front++; items[front] = item; numItemsAdded++;}
Consider this add method for FrontQueueInfo:
If add is called and the queue is full, a probable segmentation fault will occur.
As written, this code is brittle.
A robust program will not crash and burn whensomething unexpected happens.
Relying On the Caller
FrontQueue q = new FrontQueue(n); . . .if ( !q->full() ) {
q->add(item);}else {
<deal with full queue>}
However, as the program evolves, it is possible thata call will be added that is not this protective.
One approach: try to ensure that every call protectsitself:
Defensive Coding Using assert
add itself could have some kind of defense againstan incorrect call:
void FrontQueueInfo::add(Item item) {assert( !full() );front++;items[front] = item;numItemsAdded++;
}
If the queue is full, this will result in a helpfulmessage and a program halt.
Although this makes it easier to debug, the programis still brittle.
Defensive Coding Through Exception Handling
An exception is an unexpected error condition:
1)Array index out of bounds
2)Divide by zero
3)Heap memory exhausted
4)Keyboard interrupt signal
5)Abort signal
6)Conditions specific to application
C++ Approaches to Exception Handling
1 Try to guarantee that an exception condition does not exist through an assertion, and generate an unrecoverable error if the assertion is false.
2 Deal with an exception condition through the use of a handler function that is called when a system-defined exception signal is raised.
3 Deal with general exception conditions by unwinding the stack with catches and throws.
Defensive Coding Example: Safe Arrays
expected output: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
actual output:
0 1 2 3 4 5 6 7 0 1 0 1 2 3 4 5 6 7 8 9
problem: accessing array elements that are out of bounds
typedef int Integer;typedef Integer * IntegerArray;
main () { IntegerArray a = new Integer[5]; IntegerArray b = new Integer[5]; for (Integer i = 0; i < 10; i++) a[i] = i; for (Integer i = 0; i < 10; i++) b[i] = i; for (Integer i = 0; i < 10; i++) cout << a[i] << " "; cout << endl; for (Integer i = 0; i < 10; i++) cout << b[i] << " "; cout << endl;}
A Safe Array VectorInfo Class#include <assert.h>
typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;
class VectorInfo {private: IntegerArray p; Integer size;public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i);};
Notes On VectorInfo
● One class attribute is a dynamic array● One class attribute is the maximum array size● One constructor will implement a default array
size● The element method will check for index out
of bounds● assert.h is included
Exception Handling Using assert
VectorInfo::VectorInfo() { size = 10; p = new Integer[size];}
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}
Integer& VectorInfo::element(Integer i) { assert (i >= 0 && i < size); // line 40 return (p[i]);}
Main Programmain () { Vector a = new VectorInfo(5); Vector b = new VectorInfo(5); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl;}
Main Program Output
VectorInfo2.cc:40: failed assertion `i >= 0 && i < size'Abort
Although this code aborts, it is preferable becauseit gives the programmer useful information.
From the point of view of the user, it is still brittle.
Exception Handling with signal.h● The signal.h file provides a standard mechanism for
handling system-defined exceptions. Some:#define SIGHUP 1 /* hangup */
#define SIGINT 2 /* interrupt */
#define SIGILL 4 /* illegal instruction */
#define SIGFPE 8 /* floating point exception */
#define SIGBUS 10 /* bus error */
#define SIGSEGV 11 /* segmentation violation */
A programmer can arrange to have a handler function called when an exception arises using:
signal(signal, handler);
Signal Handler Function● Returns void and takes an integer (signal number) as
argument:– void handler(Integer signal);
● If part of a class, must be static ● Installed by associating it with a signal:
– void signal(Integer signal, void (*handler) (Integer))
● Automatically invoked when an exception of the associate signal type occurs
● Can be intentionally invoked by explicitly raising the associated signal:– void raise(Integer signal);
Signal Example: Keyboard InterruptMain Program
#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype
main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;
signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''
while (i < j) // interrupt with ctl-C during this loop ++i;
cout << " HIT " << j/1000000 << " MILLION" << endl; }
Keyboard Interrupt Handler
void cntrl_c_handler(int sig){ char c;
cout << "KEYBOARD INTERRUPT"; cout << "\ntype y to continue: "; cin >> c; if (c != 'y') exit(0); signal(SIGINT, cntrl_c_handler); // reset "trap" // and return}
Signal Example Output
6% testCOUNT TO J MILLION, Enter j: 100
[control-C from keyboard]
KEYBOARD INTERRUPTtype y to continue: y
[control-C from keyboard]
KEYBOARD INTERRUPTtype y to continue: yHIT 100 MILLION7%
Raising An Exception Under Program Control
#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype
main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;
signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''
while (i < j) // interrupt with ctl-C during this loop ++i;
if (i % 1000000 == 0) // generate interrupt after raise(SIGINT); // each million
cout << " HIT " << j/1000000 << " MILLION" << endl; }
Signal Example Output
9% testCOUNT TO J MILLION, Enter j: 4KEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: n530%
Note that the program does not continue to completion.
User-Defined Signals
. . .#define SIGUSR1 16 /* user defined signal 1 */#define SIGUSR2 17 /* user defined signal 2 */ . . .
Two signal codes are designed to be raised by programmers in application-specific circumstances:
User-Defined Signal Example
Recall the safe array constructor:
If the heap is exhausted (p==0), this code will abort.
Suppose the programmer can do some garbagecollection to reclaim parts of the heap.
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}
User-Defined Signal Example (cont'd)#define SIGHEAP SIGUSR1
#include <assert.h>
typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;
class VectorInfo {private: IntegerArray p; Integer size;public:
VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer
sig); };
User-Defined Signal Example (cont'd)
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; if (p == 0) { // invoke handler to do
raise(SIGHEAP); // garbage collection p = new int[size]; // try again to create
array }}
main () {signal(SIGHEAP, VectorInfo::vectorHeapHandler);vect a(5);vect b(5); . . .
}
void VectorInfo::vectorHeapHandler(int sig) { // possible action to reclaim heap storage}
Limitations of Signal Handling for C++
It would be nice to handle the other VectorInfo exceptions in this way, for example:
#define SIGHEAP SIGUSR1#define SIGSIZE SIGUSR2
class VectorInfo {private: IntegerArray p; Integer size;public:
VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer
sig);static void vectorSizeHandler(Integer
sig); };
Limitations of Signal Handling for C++ (cont'd)
VectorInfo::VectorInfo(int n) { if ( n < 1 ) {
raise(SIGSIZE); // bad vector size signal}
size = n; p = new Integer[size]; if (p == 0) {
raise(SIGHEAP); p = new int[size];
}}
Limitations (cont'd)
main () {signal(SIGHEAP,
VectorInfo::vectorHeapHandler);signal(SIGSIZE,
VectorInfo::vectorSizeHandler);vect a(5);vect b(5); . . .
}
void VectorInfo::vectorSizeHandler(int sig) { cout << "Size error. Going with default." size = 10; p = new int[size];}Unfortunately, this will not work since vectorSizeHandler must be static. It therefore has no access to size and p.
Exception Handling with catch/throw
● A more sophisticated and general form of exception handling
● Not intended to handle asynchronous exceptions defined in signal.h
● Works by causing dynamic, non-local exit from runtime stack frames ("unwinding" the stack)
Normal Runtime Stack Handling
main() {. . .foo();. . .
}
void foo() {. . .bar();. . .
}
void bar() {. . .wowie();. . .
}
void wowie() {. . .zowie();. . .
}
void zowie() {. . .. . .
}
call
return
call
return
call
return
call
return
Normal Runtime Stack Handling (cont'd)
In zowie, the runtime stack looks like:
zowiewowie
barfoo
main
Each normal return causes the stack to be popped:
zowiewowie
barfoo
main
wowiebarfoo
main
barfoo
mainfoo
main main
Non-Local Exits Using catch/throw
● Non-local exits from stack frames are caused by throwing an exception to a handler (a catch)
● The throw and the catch can be widely separated on the stack
● All intervening stack frames are "unwound" (removed from stack along with all automatic values)
● Control does not pass to the code after a call● Control ends up at the catch, which should take
appropriate action
Non-Local Exits Using catch/throw (cont'd)
zowiewowie
barfoo
main main
main () { try {
foo(); } catch(int n) { . . . }}
. . .
void zowie () { int i; . . . throw i; . . .}
● The try block establishes the context for any throws done within its dynamic extent
● The try block is immediately followed by handlers which catch a thrown exception
● There can be more than one handler which differ according to parameter type signature
code stack
Using catch/throw in VectorInfo
VectorInfo::VectorInfo(Integer n){ if ( n < 1 ) throw (n); size = n; p = new Integer[size]; if ( p == 0 ) throw ("Free Store Exhausted");}
Integer& VectorInfo::element(Integer i){ if (i < 0 || i >= size) throw ("Vector Index Out Of Bounds"); return (p[i]);}
VectorInfo Main Program
void process(Integer m) { try { Vector a = new VectorInfo(m); Vector b = new VectorInfo(m); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl; } ...
main (Integer argc, StringArray argv) { Integer n = atoi(argv[1]); process(n);}
VectorInfo Main Program (cont'd)
. . . catch (Integer n) { cerr << "Size error. Going with default." << endl; process(10); } catch (ConstString message) { cerr << message << endl; exit(0); }} 7% test 10
0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9
8% test 0Size error. Going with default.0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9
9% test 5Vector Index Out Of Bounds
Output:
catch/throw in the Queue Command Interpreter
void CommandInterpreterInfo::execute() { cin >> cmd; try { while ( !cin.eof() ) { count++; process(); cin >> cmd; } } catch (ConstString message) { cout << message << endl; } cout << count << " commands processed." << endl; free();}
Command Interpreter (cont'd)
void CommandInterpreterInfo::process() { if (cmd == '#') { processComment(); count--; } else if (count == 1) { processCreate(); } else { processOther(); }}
Command Interpreter (cont'd)
void CommandInterpreterInfo::processCreate() { if (cmd != 'c') throw ("First command must be a create..."); cin >> qtype; checkEOF(); if ( (qtype == 'f') || (qtype == 'r') || (qtype == 'p') ) { if (qtype == 'p') { cin >> pqtype; checkEOF(); if ( !(pqtype == '>' ) && !(pqtype == '<' ) ) { throw ("Priority queue type must be > or <."); } } cin >> qsize; checkEOF(); switch ( qtype ) { case 'f': q = new FrontQueueInfo(qsize); ... case 'r': q = new RearQueueInfo(qsize+1); ... case 'p': switch ( pqtype ) { case '>': q = new MaxPriorityQueueInfo(qsize); ... case '<': q = new MinPriorityQueueInfo(qsize); ... } } } else throw ("Queue type must be f, r, or p.");}
Command Interpreter (cont'd)
void CommandInterpreterInfo::checkEOF() { if ( cin.eof() ) { throw ("Unexpected end of file"); }}
execute
process
processCreate
checkEOF
RuntimeStack:
try/catch here
throw from here
throw from here
Philosophy of Error Recovery Using Exception Handling
● Exception handling is about error recovery and secondarily about transfer of control
● Undisciplined transfer of control leads to chaos (like the days before structured programming)
● In most cases programming that raises exceptions should print a diagnostic message and gracefully terminate
● Heroic attempts at repair are legitimate in real-time processing and fault-tolerant computing