PowerPoint presentation (785KB)

04/07/23

Quality Assurance Institute’s (QAI)Effective Methods of Software Testing

A PerspectiveSeattle Area Software Quality Assurance Group

John Napier, CSQE, CSTEBEST Consulting

Purpose Value of Professional Associations Value of Software Certification programs QAI’s “Effective Methods of Software Testing”

Disclaimers

The opinions expressed in this presentation are my own, and not those of:

BEST Consulting The Boeing Company Quality Assurance Institute SASQAG

What its like in the trenches... No common language “In my last job…” No individual professional context No organizational professional context Can’t see the forest...

What’s a person to do? Establish a personal professional context Get to know “the forest” Help create a common language Help bring best practices into your organization

04/07/23

The Quality Assurance Institute

What’s QAI?

The Quality Assurance Institute An international association of 1000

organizations banded together to share best practices, benchmark data and experiences

Software quality focus Certification Programs

Certified Quality Analyst Certified Software Quality Engineer

QAI Certified Software Test Engineer (CSTE)

Mandatory Skills Test Planning Use of Test Tools Test Performance Defect Management

Optional Skills Risk Analysis Measurement Test Case Design Establishing and Maintaining

a Test Environment

ASQC* Certified Quality Software Engineer (CQSE)

Experience Professional Association Software Engineering Body of Knowledge

Exam Many other ASQC Certification Programs

*American Society for Quality Control

04/07/23

Effective Methods of Software Testing

Carolyn Fairbanks

Q-Star Consultants

10-14 March 1997

Seattle

04/07/23

Effective Methods of Software TestingThere are really only two issues

in software testing:

Requirements Organizational Culture

QAI Test Materials Organization Test Strategy Test Plan

Supporting Materials Metrics Test Toolbox - Techniques Test Conditions - Techniques Test Case Design - Techniques

Test Strategy Definition 1. Type of Project 2. Type of Software 3. When Testing will Occur 4. Critical Success Factors 5. Who will Conduct the Testing 6. Tradeoffs

1. Types of Project Traditional Client/Server Prototyping Object Oriented Purchased / Contracted

2. Types of SoftwareBatchProcess ControlGraphic User InterfaceObject OrientedProcedure ControlExpert SystemMulti-MediaAdvanced Math Models

Diagnostic SoftwareSensor and SignalSimulationDB ManagementData PresentationDecision AnalysisPattern & ImageComputer OperationsSoftware Development Tools

3. When will Testing Occur? Requirements Phase Activities Design Phase Activities Build Phase Activities Test Phase Activities Installation Phase Activities Maintenance Phase Activities

4. Critical Success FactorsCorrectnessAuthorizationData IntegrityAudit TrailContinuity of ProcessingService LevelsAccess Control

MethodologyReliabilityEase of UseMaintainablePortableCouplingPerformanceEase of Operations

Testing Concerns Matrix

Critical SuccessFactors Life Cycle Stages

Correctness

Requirements

Complete, Clear, Correct,

Testable, Feasible

Design

Etc

7. Tradeoffs (What can be given up?) Schedule Cost / Resources Scope Technology Quality / Critical Success Factors

04/07/23

Some QAI Philosophy

Test Life Cycle, Test Workbench

Testing Relationship Definitions -Testing Lifecycle Concept

Operational orBusiness Need

DefineRequirements

DesignSystem

BuildSystem

UnitTest

IntegrationTest

SystemTest

AcceptanceTest

Testing Relationship Definitions -Continuous Life Cycle Testing

Operational orBusiness Need

DefineRequirements

DesignSystem

BuildSystem

UnitTest

IntegrationTest

SystemTest

AcceptanceTest

Verify

Verify

Verify

Verify Validate

Validate

Validate

Validate

Testing Relationship Definitions -Test Support Activities

Operational orBusiness Need

DefineRequirements

DesignSystem

BuildSystem

UnitTest

IntegrationTest

SystemTest

AcceptanceTest

Verify

Verify

Verify

Verify Validate

Validate

Validate

ValidateTEST OBJECTIVES

TEST PLANNING

TEST EXECUTION(STATIC)

TEST EXECUTION(DYNAMIC) &TEST REPORTING

TestQualityControl

QAI Definitions - “Test Workbench” Concept

TestProcesses

Test Toolbox

TestMeasures

ValidatedTestResults

Problems

ProductsforTest

The QAI Test Process

1. Set Test Objectives2. Develop Test Plan3. Execute Tests4. Summarize and Report Results

Step 1 - Set Test Objectives Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control

Set Test ObjectivesTask 1 - Organize Test Team

Define Test Responsibilities / Tasks Determine People to Perform Tasks

Worksheets 4,5

Set Test Objectives 2. Perform Risk Assessment

Dimensions of Risk Criticality / Mission Impact System Size, Scale and Complexity System Environment and Stability Reliability and Integrity Technology Integration

Impact of Failure

Likelihoodof

Failure

Simple or Complex Scoring Approach

Set Test Objectives 3. Set Test Objectives

Using: System Objectives Test Strategy Risk Score Analysis Team Assignments

Set Objectives Brainstorming Relate Objectives to Function / Output Relate Objectives to Events / System Input

Prioritize Each- Low Medium High

Worksheet 6

Set Test Objectives Quality Control

Objectives QC Checklist Objectives Incident Report (Change Request)

…lots of checklists!

Develop Test Plan 1. Define Business Processes 2. Define Structural Attributes 3. Define Tests 4. Create Test / Function Matrix 5. Define Admin Requirements 6. Formulate System Test Plan 7. QC System Test Plan 8. Develop Unit Test Plan 9. QC Unit Test Plan

Develop Test Plan1. Define Business Functions

Describe what the system should do …stands alone as an activity …is controllable …can be a transition …can be described by a process

Business Function Matrix

Function Description Evaluation Criteria

Worksheet 8

Develop Test Plan2. Define Structural Attributes

Reliability Efficiency Integrity Usability Maintainability

Testability Flexibility Portability Reusability Interoperability

Worksheet 9

Structural Attribute Matrix

Attribute Priority Evaluation CriteriaDescription

Develop Test Plan3. Define Tests

Using Test Objectives Worksheet Business Functions Worksheet Structural Attributes Worksheet

Define Tests Associate test factors, techniques and test types Associate evaluation criteria with techniques and types of

tests Determine type of test which satisfies each objective

Type of Test Techniques Functional

Requirements Regression Error Handling Manual Support Intersystems Control Parallel

Structural Stress Execution Recovery Operations Compliance (to process) Security

Test Data Sources Test Data / Test Desks Production Data Simulation / Prototyping Review / Confirmation Instrumentation / Tracing Test Script

Test Definition Each Test is defined in terms of:

Controls Inputs Outputs Procedures

...At a high level

Develop Test Plan4. Create Function / Test Matrix

TestsFunction

Test 1 Test 2 Ect.

Function #1

Function #2

25 17,30

5,7 45

Develop Test Plan5. Define Admin Test Requirements

Schedule Budget Resources Testing Materials Test Training Tests to be Conducted Defect Estimation

Worksheets 13,14

Develop Test Plan6. Formalize Test Plan

Section I - General Information Section 2 - Plan Section 3 - Specifications and Evaluation Section 4 - Test Description

See Test Plan Example

Develop Test Plan7. Test Plan Quality Control

Quality Control Checklist Test Plan Incident Report (Change Request)

…even more checklists!

Develop Test Plan8. Develop Unit Test Plan

Using: System Test Plan Program Initiation Document Program Design / Specification Documents

Develop Test Plan Administration Subfunction / Test Matrix Interface / Test Matrix

Worksheets 20,21,22

Develop Test Plan9. Unit Test Plan Quality Control

Checklists

Step 3 - Execute Tests Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control

QAI Tool Definition What’s a Test Tool?

…any vehicle which assists in performing a test.

Examples- Decision Tables Automated capture / playback Checklists Program Analyzer

Execute TestsStep 1 - Select Test Tools

Using: System Test Plan Unit Test Plan Tool Selection Methodology

Execute TestsStep 2 - Develop Test Cases

Worksheet 23

Expected ResultTest Condition Test Procedures

Execute TestsStep 3 - Execute Tests

Test Activities Log Regression Testing

Full Partial

Worksheet 24

Step 4 - Summarize and Report Test Results

Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings and Recommendations Task 4 - Finalize Test Report Task 5 - Test Report Quality Control

Test Defect Report Test Defect List Test Defect Log

Worksheets 25,26,27,28

04/07/23

Supporting Material Metrics Toolbox Techniques Test Condition Analysis Test Design Analysis

Test Metrics

Method -

Goal - Question - Metric

QAI Recommend Test Metrics Defect Reports Requirements Implemented Backlog of Defects (Not Corrected) Status of -

Project, Tests, Schedule, Budget Selected Test Metrics

QAI Test Metrics Selection

Types General Subject

1 Involvement in Testing4 Extent of Testing3 Resources Consumed in Testing20 Effectiveness of Testing2 Assessment of Testing

The QAI Test Toolbox Data Modeling Boundary Value Analysis Capture/Playback Cause-Effect Graphing Change Control Trackers Checklists Checkpoint Review Code Comparison Compiler-based Analysis Complexity-based Analysis

Compliance Checkers Control Flow Analysis Correctness Proofs Coverage Based Analysis Data Dictionary Decision Tables Defect Trackers Desk Checking Equivalence Partitioning Error Guessing

The QAI Test Toolbox File Comparison Flowcharts Help Desk Trackers Inspections Instrumentation Integrated Test Facility Mapping Model Animation Model Balancing Network Analyzers

Performance / Timing Analyzers Population Analysis Risk Matrix Scoring Snapshots Symbolic Execution System Logs Test Case Generator Test Data Generator Test Scripts

The QAI Test Toolbox Test Stubs and Harnesses Test Suite Managers Tracing Walkthroughs

The QAI Test Toolbox

Test Tools

Boundary Analysis

Types of Functional andStructural Tests

Population Analysis

Checklists

X

X

X

X X

Test Condition Analysis Sources -

1. Specification Decomposition 2. Population Analysis 3. Business Process Analysis 4. Structural Analysis 5. Test Transactions

Test Condition - Expected Result - Test Process

Identify Test Conditions 1. System Specifications or Production Environment 2. Population Analysis

Files and Tables Screens Field/Data Element

Worksheet 4

Test Condition - Expected Result - Test Process

Identify Test Conditions

Field Record File Relationships Error Use (Outputs) Search

Match / Merge Stress Control Attributes States Procedures Control Flow

3. Test Transaction Types

Test Condition - Expected Result - Test Process

Identify Test Conditions 4. Business Cases

…business scenarios… 5. Structural Analysis

…decision points...

Test Condition - Expected Result - Test Process

Design Test Cases Design Test Cases Develop Scripts / Sets

Expand Test Conditions

Design Test Cases Expand Conditions into Test Cases using -

Boundary Value Analysis Equivalence Partitioning Decision Tables Code Coverage

Statement Coverage Condition Coverage

Expand Test Conditions

Design Test Cases

Test Scripts

Actor/Scene Dialog

Detailed Test Script Descriptions

Script Reuse

…Sam scowlsat the report...

“Where did thiscome from?”

Expand Test Conditions

QAI Summary - Ten Key Test Concepts Use strategic test questions to set direction Develop Test Objectives Use a cross-functional “test team” Prepare a test plan, then execute it Utilize both verification and validation methods Collect and analyze defects Prepare test management reports

QAI Summary - Ten Key Test Concepts Make test efforts reusable Feed the continuous improvement process with

information from testing Continuously improve the process

04/07/23

Incorporating the Process

Models within Models

…Make them your own

Questions?