Software Engineering – Software Testing Strategies

Software Engineering – Software Testing StrategiesSoftware Engineering – Software Testing Strategies

STRATEGIC APPROACHSTRATEGIC APPROACH

Testing is a set of activities that can be planned in advance and conducted Testing is a set of activities that can be planned in advance and conducted systematically.systematically.

Many strategies exist, all follow these generic characteristics:Many strategies exist, all follow these generic characteristics:

To perform testing, a software team should conduct effective formal technical To perform testing, a software team should conduct effective formal technical reviews. This eliminates errors before testing commences.reviews. This eliminates errors before testing commences.

Testing begins at the component level and works outward toward the integration of Testing begins at the component level and works outward toward the integration of the entire computer-based system.the entire computer-based system.

Different testing techniques are appropriate at different points of time.Different testing techniques are appropriate at different points of time.

Testing is conducted by the developer of the software and (for large projects) an Testing is conducted by the developer of the software and (for large projects) an independent test group.independent test group.

Testing and debugging are different activities, but debugging must be accommodated Testing and debugging are different activities, but debugging must be accommodated in any testing strategy. in any testing strategy.


VERIFICATION VS. VALIDATIONVERIFICATION VS. VALIDATION

Verification refers to the set of activities the ensures that software correctly implements Verification refers to the set of activities the ensures that software correctly implements a specific function.a specific function.

Validation refers to a different set of activities that ensures that the software that has Validation refers to a different set of activities that ensures that the software that has been built is traceable to customer requirements.been built is traceable to customer requirements.

Verification: Are we building the product right?Verification: Are we building the product right?

Validation: Are we building the right product?Validation: Are we building the right product?


ORGANIZING TESTINGORGANIZING TESTING

““Optimism is the occupational hazard of programming; testing is the treatment.”Optimism is the occupational hazard of programming; testing is the treatment.”

Kent BeckKent Beck

““The first mistake that people make is thinking that the testing team is responsible for The first mistake that people make is thinking that the testing team is responsible for assuring quality”assuring quality”



Software testing may be viewed as a spiral process:Software testing may be viewed as a spiral process:



Software Testing StepsSoftware Testing Steps


WHEN IS TESTING COMPLETE?WHEN IS TESTING COMPLETE?

How do you know when you are done testing?How do you know when you are done testing?

There is no definitive answer to this question.There is no definitive answer to this question.

Many possible answers:Many possible answers:

You are never done testing.You are never done testing. You are done testing when you run out of money or time.You are done testing when you run out of money or time. Depends upon the certainty of failure you desire.Depends upon the certainty of failure you desire.


STRATEGIC ISSUESSTRATEGIC ISSUES

The following issues must be addressed if a successful software testing strategy is to be The following issues must be addressed if a successful software testing strategy is to be implemented:implemented:

Specify product requirements in a quantifiable manner long before testing Specify product requirements in a quantifiable manner long before testing commences.commences.

State testing objectives explicitly.State testing objectives explicitly.– test coverage, mean time to failure, cost to find and fix defects, remaining defect density or frequency test coverage, mean time to failure, cost to find and fix defects, remaining defect density or frequency

of occurrenceof occurrence

Understand the users of the software and develop a profile for each user Understand the users of the software and develop a profile for each user category.category.

Build robust software that is designed to test itself.Build robust software that is designed to test itself.

Use effective formal technical reviews prior to testing.Use effective formal technical reviews prior to testing.

Conduct formal technical reviews to assess the test strategy and test cases Conduct formal technical reviews to assess the test strategy and test cases themselves.themselves.

Develop a continuous improvement approach for the testing process.Develop a continuous improvement approach for the testing process.


TEST STRATEGIESTEST STRATEGIES

Unit Testing – focuses verification effort on the smallest unit of software design. (The Unit Testing – focuses verification effort on the smallest unit of software design. (The component or module)component or module)

Considerations:Considerations:

The module interface is tested to ensure that information properly flows into and The module interface is tested to ensure that information properly flows into and out of the program unit under test.out of the program unit under test.

Local data structures are examined to ensure that data stored temporarily Local data structures are examined to ensure that data stored temporarily maintains its integrity during all steps in an algorithm’s execution. maintains its integrity during all steps in an algorithm’s execution.

All independent paths through the control structure are exercised to ensure that All independent paths through the control structure are exercised to ensure that all statements in a module have been executed at least once.all statements in a module have been executed at least once.

Boundary conditions are tested to ensure that the module operates properly.Boundary conditions are tested to ensure that the module operates properly.

Error handling paths are tested.Error handling paths are tested.



Start testing data flow across the module interface.Start testing data flow across the module interface.

Exercise all local data structures, test impact on global structures.Exercise all local data structures, test impact on global structures.

Common errors include:Common errors include:

misunderstood or incorrect arithmetic precedencemisunderstood or incorrect arithmetic precedence mixed mode operationsmixed mode operations incorrect initializationincorrect initialization precision inaccuracyprecision inaccuracy incorrect symbolic representation of an expressionincorrect symbolic representation of an expression



UNIT TESTING PROCEDURESUNIT TESTING PROCEDURES

Unit testing is normally considered as an adjunct to the coding step.Unit testing is normally considered as an adjunct to the coding step.

Design of tests can be done before coding begins (Agile programming)Design of tests can be done before coding begins (Agile programming)







Since each component is not a stand alone program, a bit of work must be done to Since each component is not a stand alone program, a bit of work must be done to perform the test.perform the test.

Create a driver, usually a main program, that accepts test case data and passes it ot Create a driver, usually a main program, that accepts test case data and passes it ot the component to be tested. It also may print relevant results.the component to be tested. It also may print relevant results.

Stubs replace modules that may not be coded and are subordinate to the component to Stubs replace modules that may not be coded and are subordinate to the component to be tested. A stub may do minimum data manipulation , provides verification of be tested. A stub may do minimum data manipulation , provides verification of entry, and returns control to the module undergoing testing.entry, and returns control to the module undergoing testing.




Unit testing is simpler when modules are highly cohesive.Unit testing is simpler when modules are highly cohesive.



INTERGRATION TESTINGINTERGRATION TESTING

Why is unit testing not enough?Why is unit testing not enough?

Data can be lost across an interface.Data can be lost across an interface.

One module can have an inadvertent effect on another.One module can have an inadvertent effect on another.

Global data structures can present problemsGlobal data structures can present problems

Individually acceptable imprecision may become magnified.Individually acceptable imprecision may become magnified.

Resource issues.Resource issues.




Integration testing is a systematic technique for constructing the software architecture Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors associated with while at the same time conducting tests to uncover errors associated with interfacing.interfacing.

The objective is to take unit tested components and build a program structure that has The objective is to take unit tested components and build a program structure that has been dictated by design.been dictated by design.




There is a tendency to attempt nonincremental integration – construct the whole There is a tendency to attempt nonincremental integration – construct the whole program via a “big-bang” approach. program via a “big-bang” approach.

Thus to combine all components at once and test it as an entire program.Thus to combine all components at once and test it as an entire program.




Top Down Integration is an incremental approach to construction of the software Top Down Integration is an incremental approach to construction of the software architecture. Modules are integrated by moving downward through the control architecture. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module.hierarchy, beginning with the main control module.

Modules subordinate to the main control module are incorporated into the structure in Modules subordinate to the main control module are incorporated into the structure in either a depth first or breath first manner. either a depth first or breath first manner.


TEST STRATEGIES - INTERGRATION TESTINGTEST STRATEGIES - INTERGRATION TESTING

The integration process is preformed in five steps:The integration process is preformed in five steps:

1.1. The main control module is used as a test driver, and stubs are substituted for all The main control module is used as a test driver, and stubs are substituted for all components directly subordinate to the main control module.components directly subordinate to the main control module.

2.2. Depending on the integration approach selected (i.e. depth or breath first), Depending on the integration approach selected (i.e. depth or breath first), subordinate stubs are replaced one at a time with actual components.subordinate stubs are replaced one at a time with actual components.

3.3. Tests are conducted as each component is integrated.Tests are conducted as each component is integrated.

4.4. On completion of each set of tests, another stub is replaced with the real On completion of each set of tests, another stub is replaced with the real component.component.

5.5. Regression testing may be conducted to ensure that new errors have not been Regression testing may be conducted to ensure that new errors have not been introduced.introduced.



Verifies major control or decision points earlyVerifies major control or decision points early

Problems exist however:Problems exist however:

When processing at low levels in the hierarchy is required to test upper levels stubs When processing at low levels in the hierarchy is required to test upper levels stubs replace low-levels in the hierarchy, therefore no significant data can flow upward replace low-levels in the hierarchy, therefore no significant data can flow upward in the program structure.in the program structure.

Three options:Three options:

1.1. delay many tests until stubs are replaced with actual modelsdelay many tests until stubs are replaced with actual models

2.2. develop stubs that perform limited functions that simulate the actual moduledevelop stubs that perform limited functions that simulate the actual module

3.3. integrate the software from the bottom of the hierarchy upwardintegrate the software from the bottom of the hierarchy upward

1.1. The first method causes us to lose control over specific tests and specific The first method causes us to lose control over specific tests and specific modules. Thus leading to difficulty determining the cause of errors and violates modules. Thus leading to difficulty determining the cause of errors and violates the nature of the top down approach.the nature of the top down approach.

2.2. The second method is workable, but leads to significant overhead.The second method is workable, but leads to significant overhead.

3.3. The third approach is called bottom-up testing, to be covered shortly.The third approach is called bottom-up testing, to be covered shortly.



Bottom Up IntegrationBottom Up Integration

Begins construction and testing with atomic modules. Since all subordinates are always Begins construction and testing with atomic modules. Since all subordinates are always available, there is no need for stubs.available, there is no need for stubs.

Bottom up integration follows the following steps:Bottom up integration follows the following steps:

1.1. Low-level components are combined into clusters that perform a specific Low-level components are combined into clusters that perform a specific software subfunction.software subfunction.

2.2. A driver (control program for testing) is written to coordinate test cases input A driver (control program for testing) is written to coordinate test cases input and output.and output.

3.3. The cluster is tested.The cluster is tested.

4.4. Drivers are removed and clusters are combined moving upward in the program Drivers are removed and clusters are combined moving upward in the program structure.structure.

One major disadvantage of bottom up integration is that the program as an entity does One major disadvantage of bottom up integration is that the program as an entity does not exist until the last module is added.not exist until the last module is added.



Bottom Up IntegrationBottom Up Integration



Regression TestingRegression Testing

Each time a new module is added, the software changes. Each time a new module is added, the software changes.

These changes may cause problems with functions that previously worked.These changes may cause problems with functions that previously worked.

Therefore, we must re-execute some subset of tests that were already conducted to Therefore, we must re-execute some subset of tests that were already conducted to ensure that changes have not propagated unintended side effects. ensure that changes have not propagated unintended side effects.

Any time you test, if successful you will find errors. When those errors are corrected, Any time you test, if successful you will find errors. When those errors are corrected, software changes. Therefore, regression testing must be performed in order to software changes. Therefore, regression testing must be performed in order to ensure that those changes did not propagate unintended side effects.ensure that those changes did not propagate unintended side effects.

Regression tests may be conducted manually – I do not recommend this.Regression tests may be conducted manually – I do not recommend this.



Integration Test DocumentationIntegration Test Documentation

The overall plan for integration of the software and a description of specific tests are The overall plan for integration of the software and a description of specific tests are documented in a Test Specification.documented in a Test Specification.

The document contains:The document contains:

a test plana test plan a test procedurea test procedure is a work product of the software processis a work product of the software process becomes a part of the software configurationbecomes a part of the software configuration

A test plan describes the overall strategy of integration.A test plan describes the overall strategy of integration.

Testing is divided into phases and builds that address specific functional and behavioral Testing is divided into phases and builds that address specific functional and behavioral characteristics of the software.characteristics of the software.


TEST STRATEGIES FOR OBJECT-ORIENTED SOFTWARETEST STRATEGIES FOR OBJECT-ORIENTED SOFTWARE

Unit Testing in the OO ContextUnit Testing in the OO Context

With OO the concept of the unit changes.With OO the concept of the unit changes.

Because of the dependence of sub classes, unit testing is a bit more complicated.Because of the dependence of sub classes, unit testing is a bit more complicated.

In the OO context, unit testing is basically Class testing.In the OO context, unit testing is basically Class testing.



Integration Testing in the OO ContextIntegration Testing in the OO Context

Unlike non OO programming, OO programming does not have an hierarchical control Unlike non OO programming, OO programming does not have an hierarchical control structure.structure.

Integrating one object at a time is also difficult because of the direct and indirect Integrating one object at a time is also difficult because of the direct and indirect interactions of the components that make up the class.interactions of the components that make up the class.

There are two basic strategies:There are two basic strategies:

Thread-basedThread-based Use-basedUse-based



Thread-based testing integrates the set of classes required to respond to one input or Thread-based testing integrates the set of classes required to respond to one input or event for the system.event for the system.

Each thread is integrated and tested individually.Each thread is integrated and tested individually.

Regression testing is applied to ensure no side effects occur.Regression testing is applied to ensure no side effects occur.

Use-based testing begins the construction of system by testing those classes that use Use-based testing begins the construction of system by testing those classes that use very few server classes. very few server classes.

After independent classes are tested, the next layer of classes, called dependent After independent classes are tested, the next layer of classes, called dependent classes are tested.classes are tested.

This continues until the entire system is constructed.This continues until the entire system is constructed.



Drivers can be used to test operations at the lowest level and for testing groups of Drivers can be used to test operations at the lowest level and for testing groups of classes.classes.

A driver may be used to replace the user interface.A driver may be used to replace the user interface.

Stubs may be used where collaboration between classes is required but one or more of Stubs may be used where collaboration between classes is required but one or more of the collaborating classes is not fully implemented.the collaborating classes is not fully implemented.



Cluster testing is one step in integration testing of OO software.Cluster testing is one step in integration testing of OO software.

Here, a cluster of collaborating classes is exercised by designing test cases that Here, a cluster of collaborating classes is exercised by designing test cases that attempt to uncover errors in the collaborations.attempt to uncover errors in the collaborations.



Validation TestingValidation Testing

Validation testing begins at the culmination of integrations testing.Validation testing begins at the culmination of integrations testing.

Testing focuses on the user-visible actions and user-recognizable output from the Testing focuses on the user-visible actions and user-recognizable output from the system.system.

Validation can be defined as:Validation can be defined as:

Validation succeeds when software functions in a manner that can be reasonably Validation succeeds when software functions in a manner that can be reasonably expected by the customer.expected by the customer.

Software validation is achieved through a series of tests that demonstrate conformity Software validation is achieved through a series of tests that demonstrate conformity with software requirements.with software requirements.



Most software product builders use a process called alpha and beta testing to uncover Most software product builders use a process called alpha and beta testing to uncover errors that only the end-user seems able to find.errors that only the end-user seems able to find.

The alpha test is conducted at the developer’s site by end-users.The alpha test is conducted at the developer’s site by end-users.

The beta test is conducted at the end-user sites. Beta tests usually do not have the The beta test is conducted at the end-user sites. Beta tests usually do not have the developer present.developer present.



System TestingSystem Testing

Recovery Testing – most computer-based systems must recover from faults and resume Recovery Testing – most computer-based systems must recover from faults and resume processing within a prespecified time. processing within a prespecified time.

Recovery testing is a system test that forces the software to fail in a variety of ways Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly preformed.and verifies that recovery is properly preformed.



Security Testing – Security Testing –

Any computer system that manages sensitive information or causes actions that can Any computer system that manages sensitive information or causes actions that can improperly harm or benefit individuals is a target for improper or illegal improperly harm or benefit individuals is a target for improper or illegal penetration.penetration.

Lot’s of security issues.Lot’s of security issues.

Security testing verifies that protection mechanisms built into the system will, in fact, Security testing verifies that protection mechanisms built into the system will, in fact, protect it from improper penetration.protect it from improper penetration.

During security testing, the tester plays the role(s) of individual who desires to During security testing, the tester plays the role(s) of individual who desires to penetrate the system.penetrate the system.

Anything goes!Anything goes!

What is the biggest source of security issues?What is the biggest source of security issues?



Usually it’s not the software, but people issues.Usually it’s not the software, but people issues.

People often give out secure information. No system can truly protect against this People often give out secure information. No system can truly protect against this 100%.100%.



Stress TestingStress Testing

So far all testing assumed normal program functions and performance.So far all testing assumed normal program functions and performance.

Stress tests are designed to confront programs with abnormal quantity, frequency, or Stress tests are designed to confront programs with abnormal quantity, frequency, or volume.volume.

Multiple interrupts per secondMultiple interrupts per second Increased data rateIncreased data rate Test max memoryTest max memory Excessive disk huntingExcessive disk hunting



Performance TestingPerformance Testing

Designed to test the run-time performance of software within the context of an Designed to test the run-time performance of software within the context of an integrated system.integrated system.

Should be performed at all steps of the testing process.Should be performed at all steps of the testing process.

Not until system elements are fully integrated that performance of a system can really Not until system elements are fully integrated that performance of a system can really be ascertained.be ascertained.

Sometime coupled with stress testing.Sometime coupled with stress testing.

Often requires hardware and software instrumentation to measure resource utilization.Often requires hardware and software instrumentation to measure resource utilization.

Documents

Software Engineering – Software Testing Strategies