Upload
blake-jackson
View
215
Download
1
Embed Size (px)
Citation preview
Software Testing
Sudipto GhoshCS 406 Fall 99(Also
used later)November 2, 1999
11/02/99 CS 406 Testing 2
Learning Objectives
• Testing• What is testing?• Why test?
• How testing increase our confidence in program correctness?
• When to test?• What to test?• How to test?
• Different types of testing
• How to assess test adequacy?
11/02/99 CS 406 Testing 3
Testing
• The act of checking if a part or product performs as expected.
• Why do we test?• Check if there are any
errors?• Increase confidence in
the “correctness”
11/02/99 CS 406 Testing 4
Testing
• When should we test?• Separate phase for
testing? OR• At the end of every
phase? OR• During each phase,
simultaneously with all development and maintenance activities
11/02/99 CS 406 Testing 5
What can be tested?
• All products generated during lifecycle
Phase Product How to test?Specification Req. doc Scenario construction,
simulations
Design Design doc Simulations
Implementation Modules Run against test cases
Integration System Run against test cases
Maintenance System Regression testing
11/02/99 CS 406 Testing 6
Our focus
• Focus: Testing programs• Programs may be sub-
systems or complete systems
• Programs are written in a programming language
• Explore techniques and tools for testing programs
11/02/99 CS 406 Testing 7
Verification, Validation, Testing
• Verification: • Process of determining whether a phase has
been correctly carried out• Are we building the product right? (Boehm)
• Validation:• Intensive evaluation process that takes place
just before the product is delivered to the client• Are we building the right product? (Boehm)
• V & VV & V denotes testing• (Verification: mathematically prove program correctness)
11/02/99 CS 406 Testing 8
Problems with formal verification
• Adequate mathematical training• Economic viability of proofs• Proving hard for non-trivial products• Correctness of the “prover” itself• Finding input, output specifications, loop
invariants• Wrong specifications
11/02/99 CS 406 Testing 9
Terminology
• Program - Sort• Collection of functions (C) or classes (Java)
• Specification• Input: p - array of n integers, n>0• Output: q - array of n integers such that
• q[0] q[1] … q[n]• Elements in q are a permutation of elements in p,
which are unchanged
• Description of requirements of a program
11/02/99 CS 406 Testing 10
Tests
• Test input (Test case)• A set of values given as input to a program• Includes environment variables• {2, 3, 6, 5, 4}
• Test set• A set of test cases• { {0}, {9, 8, 7, 6, 5}, {1, 3, 4, 5}, {2, 1, 2, 3} }
11/02/99 CS 406 Testing 11
Oracle
• Function that determines whether or not the results of executing a program under test is as per the program’s specifications
SoftwareUnder Test
Oracle
TestCases
output
output
Failure?Success?Compare
Problems• Correctness of Oracle• Correctness of specs• Generation of Oracle• Need more formal specs
11/02/99 CS 406 Testing 12
Correctness
• Program Correctness• A program P is considered with respect to a
specification S, if and only if:• For each valid input, the output of P is in
accordance with the specification S
• What if the specifications are themselves incorrect?
11/02/99 CS 406 Testing 13
Errors, defects, faults
• Often used interchangeably.• Error:
• Mistake made by programmer.• Human action that results in the software
containing a fault/defect.
• Defect / Fault:• Manifestation of the error in a program.• Condition that causes the system to fail.• Synonymous with bug
11/02/99 CS 406 Testing 14
Failure
• Incorrect program behavior due to a fault in the program.
• Failure can be determined only with respect to a set of requirement specs.
• For failure to occur, the testing of the program should force the erroneous portion of the program to be executed.
11/02/99 CS 406 Testing 15
Errors and failure
Inputs
Outputs
Error revealinginputs cause failures
Erroneous outputsindicate failures
Program
11/02/99 CS 406 Testing 16
Debugging
• Suppose that a failure is detected during the testing of P.
• The process of finding and removing the cause of this failure is known as debugging.
• Testing usually leads to debugging.• Testing and debugging usually happen in
a cycle.
11/02/99 CS 406 Testing 17
Test-debug cycle
Test
Debug
Done
Failure?
Testingcomplete?
Yes
Yes
No
No
11/02/99 CS 406 Testing 18
Software quality
• Extent to which the product satisfies its specifications.
• Get the software to function correctly.• SQA group is in charge of ensuring
correctness.• Testing• Develop various standards to which the
software must conform• Establish monitoring procedures for assuring
compliance with those standards
11/02/99 CS 406 Testing 19
Nonexecution-Based Testing
• Person creating a product should not be the only one responsible for reviewing it.
• A document is is checked by a team of software professionals with a range of skills.
• Increases chances of finding a fault.• Types of reviews
• Walkthroughs• Inspections
11/02/99 CS 406 Testing 20
Walkthroughs
TEAM
Client representative Rep of team from next phase
Rep from SQA group
Rep from Specs team Manager from Specs team
• Reviewer prepares two lists:• Items that the reviewer does not understand• Items that the reviewer believes are incorrect
11/02/99 CS 406 Testing 21
Managing walkthroughs
• Distribute material for walkthrough in advance.
• Includes senior technical staff.• Chaired by the SQA representative.• Task is to record fault for later correction
• Not much time to fix it during walkthrough• Other individuals are trained to fix it better• Cost of 1 team vs cost of 1 person• Not all faults need to be fixed as not all “faults”
flagged are incorrect
11/02/99 CS 406 Testing 22
Managing walkthroughs (contd.)
• Two ways of doing walkthroughs• Participant driven
• Present lists of unclear items and incorrect items• Rep from specs team responds to each query
• Document driven• Person responsible for document walks the
participants through the document• Reviewers interrupt with prepared comments or
comments triggered by the presentation
• Interactive process• Not to be used for the evaluation of
participants
11/02/99 CS 406 Testing 23
Inspections
• Proposed by Fagan for testing • designs• code
• An inspection goes beyond a walkthrough• Five formal stages• Stage 1 - Overview
• Overview document (specs/design/code/ plan) to be prepared by person responsible for producing the product.
• Document is distributed to participants.
11/02/99 CS 406 Testing 24
Inspections (contd.)
• Stage 2 - Preparation• Understand the document in detail.• List of fault types found in inspections ranked
by frequency used for concentrating efforts.
• Stage 3 - Inspection• Walk through the document and ensure that
• Each item is covered• Every branch is taken at least once• Find faults and document them (don’t correct)• Leader (moderator) produces a written report
11/02/99 CS 406 Testing 25
Inspections (contd.)
• Stage 4 - Rework• Resolve all faults and problems
• Stage 5 - Follow-up• Moderator must ensure that every issue has
been resolved in some way
• Team• Moderator-manager, leader of inspection team• Designer - team responsible for current phase• Implementer - team responsible for next phase• Tester - preferably from SQA team
11/02/99 CS 406 Testing 26
What to look for in inspections
• Is each item in a specs doc adequately and correctly addressed?
• Do actual and formal parameters match?• Error handling mechanisms identified?• Design compatible with hardware
resources?• What about with software resources?
11/02/99 CS 406 Testing 27
What to record in inspections
• Record fault statistics• Categorize by severity, fault type• Compare # faults with average # faults in
same stage of development• Find disproportionate # in some modules,
then begin checking other modules• Too many faults => redesign the module• Information on fault types will help in code
inspection in the same module
11/02/99 CS 406 Testing 28
Pros and cons of inspection
• High number of faults found even before testing (design and code inspections)
• Higher programmer productivity, less time on module testing
• Fewer faults found in product that was inspected before
• If faults are detected early in the process there is a huge savings
• What if walkthroughs are used for performance appraisal?
11/02/99 CS 406 Testing 29
Comparison of inspection and walkthroughs• Inspection
• Checklist used to find faults
• Five step process• Formalized
procedure in each step
• Longer time
• Walkthrough• No checklist used
• Two step process• No formalized
procedure in the steps
• Shorter time
11/02/99 CS 406 Testing 30
Strengths and weaknesses of reviews• Effective way of detecting faults• Faults detected early in development• Effectiveness reduced if software process
is inadequate• Large scale software hard to review if it
consists of smaller largely independent units (OO)
• Need access to complete, updated documents from previous phase
11/02/99 CS 406 Testing 31
Testing and code inspection
• Code inspection is generally considered complimentary to testing
• Neither is more important than the other• One is not likely to replace testing by code
inspection or by formal verification
11/02/99 CS 406 Testing 32
Program testing
• Testing is a demonstration that faults are not present.
• Program testing can be an effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. - Dijkstra
11/02/99 CS 406 Testing 33
Execution-based testing
• Execution-based testing is a process of inferring certain behavioral properties of a product, based, in part, on the results of executing the product in a known environment with selected inputs.
• Depends on environment and inputs• How well do we know the environment?• How much control do we have over test
inputs?• Real time systems
11/02/99 CS 406 Testing 34
Utility
• Extent to which a user’s needs are met when a correct product is used under conditions permitted by its specs.
• How easy is the product to use?• Does it perform useful functions?• Is it cost-effective?
11/02/99 CS 406 Testing 35
Reliability
• Probability of failure-free operation of a product for a given time duration
• How often does the product fail?• mean time between failures
• How bad are the effects?• How long does it take to repair it?
• mean time to repair
• Failure behavior controlled by• Number of faults
• Operational profile of execution
11/02/99 CS 406 Testing 36
Robustness
• How well does the product behave with• range of operating conditions• possibility of unacceptable results with valid
input?• possibility of unacceptable results with invalid
input?
• Should not crash even when not used under permissible conditions
11/02/99 CS 406 Testing 37
Performance
• To what extent does the product meet its requirements with regard to • response time• space requirements
• What if there are too many clients/ processes, etc...
11/02/99 CS 406 Testing 38
Levels of testing
• Unit testing• Integration testing• System testing• Acceptance testing
11/02/99 CS 406 Testing 39
Test plan
• Test unit specification• Features to be tested• Approach for testing• Test deliverables• Schedule• Personnel allocation
11/02/99 CS 406 Testing 40
References
• Textbook• Scach - Classical and Object-Oriented Software
Engineering
• Other books• Beizer - Software Testing Techniques
• Beizer - Black-Box Testing: Techniques for Functional Testing of Software and Systems
• Marick - The Craft of Software Testing
• Myers - The Art of Software Testing