Upload
evelyn
View
54
Download
0
Tags:
Embed Size (px)
DESCRIPTION
(Chapter 8). Debugging and Testing. Testing. Testing is the process of running a program with the intent of finding bugs Requires running the program on sample data Might require writing extra code to emulate missing parts of the system Requires the programmer to be "egoless" - PowerPoint PPT Presentation
Citation preview
Debugging and Testing
(Chapter 8)
Testing
● Testing is the process of running a program with the intent of finding bugs
● Requires running the program on sample data● Might require writing extra code to emulate
missing parts of the system● Requires the programmer to be "egoless"● Often performed by programmers other than the
developers
Types of Testing
● Static: Done without actually executing program– Code inspections– Walkthroughs
● Dynamic: Done by executing program or its parts– Module or unit testing– Integration testing– System testing
Module or Unit Testing
● Tests individual classes or small sets of classes● Generally done by the programmer responsible
for the classes● Bottom-up process● May require the use of a throw-away test driver
program
Test Driver for FarmerStateInfoint main() {
Side farmer = EAST; Side wolf = EAST; Side goat = WEST; Side cabbage = WEST;
FarmerState s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();
cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test wrong" << endl; else cout << "Safety test OK" << endl;
farmer = WEST; s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();
cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test OK" << endl; else cout << "Safety test wrong" << endl;}
Test Driver Output
67% farmer ||F ||WG|| C||
Testing safety:Safety test OKF|| ||WG|| C||
Testing safety:Safety test OK68%
Notes on the Output
● What gets tested:– The constructor– The display() method– Safety of the cabbage, both when it's vulnerable and
when it's not– The safety of the goat when it's not vulnerable
● What does not get tested:– The safety of the goat when it is vulnerable
● Note: the isSafe() method had to be made temporarily public
Testing State-Changing Methodsint main() {
FarmerState s = new FarmerStateInfo(WEST, WEST, WEST, WEST); s->display();
State newst = FarmerStateInfo::self(s); cout << "Action farmer-takes-self " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;
newst = FarmerStateInfo::wolf(s); cout << "Action farmer-takes-wolf " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; . . .
newst = FarmerStateInfo::cabbage(s); cout << "Action farmer-takes-cabbage " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;}
State-Changing Test Output
69% farmerF|| W|| G|| C||
Action farmer-takes-self not allowed
Action farmer-takes-wolf not allowed
Action farmer-takes-goat ||FW|| ||GC||
Action farmer-takes-cabbage not allowed
70%
Testing FarmerProblemInfo and State Equality
int main() {
Problem p = new FarmerProblemInfo(); State start = p->getStartState(); State final = p->getFinalState();
cout << "Testing state display:" << endl;
start->display(); final->display();
cout << "Testing state equality:" << endl;
if ( start->equals(final) ) cout << "Equality check wrong" << endl; else cout << "Equality check OK" << endl;
if ( start->equals(start) ) cout << "Equality check OK" << endl; else cout << "Equality check wrong" << endl;
}
Equality Test Output71% farmerTesting state display:F|| W|| G|| C||
||F ||W ||G ||C
Testing state equality:Equality check OKEquality check OK72%
Note that the test driver could be used for otherproblems just by changing the constructor call.
Testing expand()
int main() {
Problem p = new FarmerProblemInfo(); State start = p->getStartState(); start->display();
Item testItem = new ItemInfo(start, NULL, NULL, 0, 0); ItemArray children = testItem->expand(p); cout << "Num children: " << testItem->getNumChildren() << endl;
for (Integer i = 0; i < testItem->getNumChildren(); i++) { State nextState = children[i]->getState(); nextState->display(); }
}
expand() Test Output
73% farmerF|| W|| G|| C||
Num children: 1 ||FW|| ||GC||
74%
Integration Testing
● Putting together the results of module testing● Top-down approach● Example: testing the SolverInfo class tests
the integration of ProblemInfo and QueueInfo
● Often includes a primitive graphical user interface (GUI) so that:– system has look and feel of final product– separate drivers not needed
● Might require stubs or dummy routines for code to be added later
System Testing● Attempts to find problems in a complete program● Done when
– program is first completed, and– as the program evolves (maintenance testing)
● In industry, often done by an independent group of programmers
● Requires creation of a test suite that:– is used over and over as program evolves– attempts to identify all combinations of inputs and
actions that could cause program to fail
Testing and Proof of Correctness
● It is not possible to test a nontrivial system with all possible inputs and outputs
● So testing cannot formally prove that a system is correct, only give some level of confidence in it
● Formal proofs of dynamic program correctness are not possible either, because executing programs are subject to physical influences
● Formal proofs of abstract algorithm correctness are possible though difficult, and require machine assistance
Debugging
● The process of fixing problems (bugs) that testing discovers
● Debugging has two parts:– finding the cause of a bug (can be 95% of debugging)– fixing the bug
● A good programmer becomes adept at using a run-time debugger
Types of Program Bugs
● Syntactic bugs not caught by the compiler
Example:for (... ; ... ; ...);{
...
}● Design bugs: easy to find, can be difficult to fix● Logic bugs: can be difficult to find, easy to fix
Types of Bugs (cont'd)
● Interface bugs: – When assumptions made by a method caller are
different than those of the callee. – Can be difficult to find, easy to fix
● Memory bugs:– Freeing or overwriting storage that is later used– Especially difficult to find– Example:
IntArray A; // = new Integer[4]; ...A[0] = 10;A[1] = 20: ...
Fixing Bugs● Fixing might entail correcting a single line of
code● Fixing might entail redesign, recoding, and
further testing:– For example, incorrect circularity check in expand()
● Fixing a bug might cause other bugs to appear● It has been observed that code in which a bug has
been found is more likely to still contain bugs
Defensive Programming
● The easiest way to do debugging is to not have bugs
● Failing that, one should strive for locality:
Ensure that bugs that are identified are local problems found where and when they occur
● Locality is the goal of defensive programming● Defensive programming begins with defensive
design
Defensive Design
● First goal: Simplicity of Design
Example: Take the time and trouble to make a repeated piece of code a method, even if small
● Also: Simplicity of Class Interfaces
Example: refactoring the abstract StateInfo and ProblemInfo classes
● Another goal: Program for Errors– Anticipate exceptions even when they are not
expected– Example: add(Item) and remove() should check
for fullness and emptiness even though currently all callers check first
Program Robustness
void FrontQueueInfo::add(Item item) { front++; items[front] = item; numItemsAdded++;}
Consider this add method for FrontQueueInfo:
If add is called and the queue is full, a probable segmentation fault will occur.
As written, this code is brittle.
A robust program will not crash and burn whensomething unexpected happens.
Relying On the Caller
FrontQueue q = new FrontQueue(n); . . .if ( !q->full() ) {
q->add(item);}else {
<deal with full queue>}
However, as the program evolves, it is possible thata call will be added that is not this protective.
One approach: try to ensure that every call protectsitself:
Defensive Coding Using assert
add itself could have some kind of defense againstan incorrect call:
void FrontQueueInfo::add(Item item) {assert( !full() );front++;items[front] = item;numItemsAdded++;
}
If the queue is full, this will result in a helpfulmessage and a program halt.
Although this makes it easier to debug, the programis still brittle.
Defensive Coding Through Exception Handling
An exception is an unexpected error condition:
1)Array index out of bounds
2)Divide by zero
3)Heap memory exhausted
4)Keyboard interrupt signal
5)Abort signal
6)Conditions specific to application
C++ Approaches to Exception Handling
1 Try to guarantee that an exception condition does not exist through an assertion, and generate an unrecoverable error if the assertion is false.
2 Deal with an exception condition through the use of a handler function that is called when a system-defined exception signal is raised.
3 Deal with general exception conditions by unwinding the stack with catches and throws.
Defensive Coding Example: Safe Arrays
expected output: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
actual output:
0 1 2 3 4 5 6 7 0 1 0 1 2 3 4 5 6 7 8 9
problem: accessing array elements that are out of bounds
typedef int Integer;typedef Integer * IntegerArray;
main () { IntegerArray a = new Integer[5]; IntegerArray b = new Integer[5]; for (Integer i = 0; i < 10; i++) a[i] = i; for (Integer i = 0; i < 10; i++) b[i] = i; for (Integer i = 0; i < 10; i++) cout << a[i] << " "; cout << endl; for (Integer i = 0; i < 10; i++) cout << b[i] << " "; cout << endl;}
A Safe Array VectorInfo Class#include <assert.h>
typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;
class VectorInfo {private: IntegerArray p; Integer size;public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i);};
Notes On VectorInfo
● One class attribute is a dynamic array● One class attribute is the maximum array size● One constructor will implement a default array
size● The element method will check for index out
of bounds● assert.h is included
Exception Handling Using assert
VectorInfo::VectorInfo() { size = 10; p = new Integer[size];}
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}
Integer& VectorInfo::element(Integer i) { assert (i >= 0 && i < size); // line 40 return (p[i]);}
Main Programmain () { Vector a = new VectorInfo(5); Vector b = new VectorInfo(5); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl;}
Main Program Output
VectorInfo2.cc:40: failed assertion `i >= 0 && i < size'Abort
Although this code aborts, it is preferable becauseit gives the programmer useful information.
From the point of view of the user, it is still brittle.
Exception Handling with signal.h● The signal.h file provides a standard mechanism for
handling system-defined exceptions. Some:#define SIGHUP 1 /* hangup */
#define SIGINT 2 /* interrupt */
#define SIGILL 4 /* illegal instruction */
#define SIGFPE 8 /* floating point exception */
#define SIGBUS 10 /* bus error */
#define SIGSEGV 11 /* segmentation violation */
A programmer can arrange to have a handler function called when an exception arises using:
signal(signal, handler);
Signal Handler Function● Returns void and takes an integer (signal number) as
argument:– void handler(Integer signal);
● If part of a class, must be static ● Installed by associating it with a signal:
– void signal(Integer signal, void (*handler) (Integer))
● Automatically invoked when an exception of the associate signal type occurs
● Can be intentionally invoked by explicitly raising the associated signal:– void raise(Integer signal);
Signal Example: Keyboard InterruptMain Program
#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype
main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;
signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''
while (i < j) // interrupt with ctl-C during this loop ++i;
cout << " HIT " << j/1000000 << " MILLION" << endl; }
Keyboard Interrupt Handler
void cntrl_c_handler(int sig){ char c;
cout << "KEYBOARD INTERRUPT"; cout << "\ntype y to continue: "; cin >> c; if (c != 'y') exit(0); signal(SIGINT, cntrl_c_handler); // reset "trap" // and return}
Signal Example Output
6% testCOUNT TO J MILLION, Enter j: 100
[control-C from keyboard]
KEYBOARD INTERRUPTtype y to continue: y
[control-C from keyboard]
KEYBOARD INTERRUPTtype y to continue: yHIT 100 MILLION7%
Raising An Exception Under Program Control
#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype
main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;
signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''
while (i < j) // interrupt with ctl-C during this loop ++i;
if (i % 1000000 == 0) // generate interrupt after raise(SIGINT); // each million
cout << " HIT " << j/1000000 << " MILLION" << endl; }
Signal Example Output
9% testCOUNT TO J MILLION, Enter j: 4KEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: n530%
Note that the program does not continue to completion.
User-Defined Signals
. . .#define SIGUSR1 16 /* user defined signal 1 */#define SIGUSR2 17 /* user defined signal 2 */ . . .
Two signal codes are designed to be raised by programmers in application-specific circumstances:
User-Defined Signal Example
Recall the safe array constructor:
If the heap is exhausted (p==0), this code will abort.
Suppose the programmer can do some garbagecollection to reclaim parts of the heap.
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}
User-Defined Signal Example (cont'd)#define SIGHEAP SIGUSR1
#include <assert.h>
typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;
class VectorInfo {private: IntegerArray p; Integer size;public:
VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer
sig); };
User-Defined Signal Example (cont'd)
VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; if (p == 0) { // invoke handler to do
raise(SIGHEAP); // garbage collection p = new int[size]; // try again to create
array }}
main () {signal(SIGHEAP, VectorInfo::vectorHeapHandler);vect a(5);vect b(5); . . .
}
void VectorInfo::vectorHeapHandler(int sig) { // possible action to reclaim heap storage}
Limitations of Signal Handling for C++
It would be nice to handle the other VectorInfo exceptions in this way, for example:
#define SIGHEAP SIGUSR1#define SIGSIZE SIGUSR2
class VectorInfo {private: IntegerArray p; Integer size;public:
VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer
sig);static void vectorSizeHandler(Integer
sig); };
Limitations of Signal Handling for C++ (cont'd)
VectorInfo::VectorInfo(int n) { if ( n < 1 ) {
raise(SIGSIZE); // bad vector size signal}
size = n; p = new Integer[size]; if (p == 0) {
raise(SIGHEAP); p = new int[size];
}}
Limitations (cont'd)
main () {signal(SIGHEAP,
VectorInfo::vectorHeapHandler);signal(SIGSIZE,
VectorInfo::vectorSizeHandler);vect a(5);vect b(5); . . .
}
void VectorInfo::vectorSizeHandler(int sig) { cout << "Size error. Going with default." size = 10; p = new int[size];}Unfortunately, this will not work since vectorSizeHandler must be static. It therefore has no access to size and p.
Exception Handling with catch/throw
● A more sophisticated and general form of exception handling
● Not intended to handle asynchronous exceptions defined in signal.h
● Works by causing dynamic, non-local exit from runtime stack frames ("unwinding" the stack)
Normal Runtime Stack Handling
main() {. . .foo();. . .
}
void foo() {. . .bar();. . .
}
void bar() {. . .wowie();. . .
}
void wowie() {. . .zowie();. . .
}
void zowie() {. . .. . .
}
call
return
call
return
call
return
call
return
Normal Runtime Stack Handling (cont'd)
In zowie, the runtime stack looks like:
zowiewowie
barfoo
main
Each normal return causes the stack to be popped:
zowiewowie
barfoo
main
wowiebarfoo
main
barfoo
mainfoo
main main
Non-Local Exits Using catch/throw
● Non-local exits from stack frames are caused by throwing an exception to a handler (a catch)
● The throw and the catch can be widely separated on the stack
● All intervening stack frames are "unwound" (removed from stack along with all automatic values)
● Control does not pass to the code after a call● Control ends up at the catch, which should take
appropriate action
Non-Local Exits Using catch/throw (cont'd)
zowiewowie
barfoo
main main
main () { try {
foo(); } catch(int n) { . . . }}
. . .
void zowie () { int i; . . . throw i; . . .}
● The try block establishes the context for any throws done within its dynamic extent
● The try block is immediately followed by handlers which catch a thrown exception
● There can be more than one handler which differ according to parameter type signature
code stack
Using catch/throw in VectorInfo
VectorInfo::VectorInfo(Integer n){ if ( n < 1 ) throw (n); size = n; p = new Integer[size]; if ( p == 0 ) throw ("Free Store Exhausted");}
Integer& VectorInfo::element(Integer i){ if (i < 0 || i >= size) throw ("Vector Index Out Of Bounds"); return (p[i]);}
VectorInfo Main Program
void process(Integer m) { try { Vector a = new VectorInfo(m); Vector b = new VectorInfo(m); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl; } ...
main (Integer argc, StringArray argv) { Integer n = atoi(argv[1]); process(n);}
VectorInfo Main Program (cont'd)
. . . catch (Integer n) { cerr << "Size error. Going with default." << endl; process(10); } catch (ConstString message) { cerr << message << endl; exit(0); }} 7% test 10
0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9
8% test 0Size error. Going with default.0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9
9% test 5Vector Index Out Of Bounds
Output:
catch/throw in the Queue Command Interpreter
void CommandInterpreterInfo::execute() { cin >> cmd; try { while ( !cin.eof() ) { count++; process(); cin >> cmd; } } catch (ConstString message) { cout << message << endl; } cout << count << " commands processed." << endl; free();}
Command Interpreter (cont'd)
void CommandInterpreterInfo::process() { if (cmd == '#') { processComment(); count--; } else if (count == 1) { processCreate(); } else { processOther(); }}
Command Interpreter (cont'd)
void CommandInterpreterInfo::processCreate() { if (cmd != 'c') throw ("First command must be a create..."); cin >> qtype; checkEOF(); if ( (qtype == 'f') || (qtype == 'r') || (qtype == 'p') ) { if (qtype == 'p') { cin >> pqtype; checkEOF(); if ( !(pqtype == '>' ) && !(pqtype == '<' ) ) { throw ("Priority queue type must be > or <."); } } cin >> qsize; checkEOF(); switch ( qtype ) { case 'f': q = new FrontQueueInfo(qsize); ... case 'r': q = new RearQueueInfo(qsize+1); ... case 'p': switch ( pqtype ) { case '>': q = new MaxPriorityQueueInfo(qsize); ... case '<': q = new MinPriorityQueueInfo(qsize); ... } } } else throw ("Queue type must be f, r, or p.");}
Command Interpreter (cont'd)
void CommandInterpreterInfo::checkEOF() { if ( cin.eof() ) { throw ("Unexpected end of file"); }}
execute
process
processCreate
checkEOF
RuntimeStack:
try/catch here
throw from here
throw from here
Philosophy of Error Recovery Using Exception Handling
● Exception handling is about error recovery and secondarily about transfer of control
● Undisciplined transfer of control leads to chaos (like the days before structured programming)
● In most cases programming that raises exceptions should print a diagnostic message and gracefully terminate
● Heroic attempts at repair are legitimate in real-time processing and fault-tolerant computing