Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation

© Copyright IBM Corporation 2013

Applying Continuous Verification and Validationto achieve the right Quality in Systems delivery

TechnicalRisk

Business Risk

Kerim Cakmak, PhD

IBM Rational Tiger Team CEE

Moshe S. Cohen

QM Market Manager and Evangelist

© 2012 IBM Corporation

Software and Systems Engineering | Rational

The V is dead! Long live the new V with continuous Verification and Validation across the full V!

Abstract:

While we know how to reduce technical risks in Systems delivery through Verification activities, our business risk is on a steep rise. This is largely due to the fact that "Validation" is performed at the end of the development lifecycle, rather than throughout, and it is done against customers requirements rather than end-users needs. In this session we will discuss some new best practices around Verification and Validation that were observed through recent successful and even more unsuccessful projects, and show you how to continuously verify and validate your designs throughout the whole lifecycle and not only at the tail end of it. Not only will this improve the quality of your products and systems, but it will also allow you to converge earlier on the right requirements, compressing the time to end users feedback, and improve your time to market predictability. While it doesn't change the V model, it is a new V.

2



Bottom Line is…

While delivering Quality is already a major challenge, the definition of Quality is changing…

– Low tolerance for bad quality (real and perceived)

– Expected short cycles with continuous improvements

To deliver Quality, we need a holistic approach to Quality– Continuously verify and validate our designs, throughout the whole lifecycle.

Business results:

– Deliver the right system

• Compressing time to customer’s feedback

– Better time to market predictability

• Earlier convergence on the right Requirements

– Lower cost of Quality

• Defect* Avoidance, Lower defect density, Higher defect removal efficiency

3

* Defect, as defined in ISO 9000 (2008): The non-fulfillment of intended usage requirements. The term Defect will be used here in the general sense, regardless whether it is a product defect, design defect, manufacturing defect, an error, a fault or a failure.



Agenda

4



5

In the News… Business Implications of a Failed missile testCould result in major financial consequences, unless Raytheon succeeds in demonstrating that it is not its fault.



Business Implications of RecallsNot only recalls are very expensive, but they also drive corporate behaviors.

Sony recalls: “Sony-related recalls are following one another, and that may ruin the company’s brand image,” said Keita Wakabayashi, an analyst at Mito Securities Co. with a “neutral plus” rating on the stock. Bloomberg, Oct 2011.

Toyota Prius gas pedal recall, 2010: ~$3B (CNNMoney.com)

– Repair cost (quality related expenses): $1.12B

– Legal cost (class-action lawsuit): $1.1B (WSJ, Dec 27, 2012)

– Image cost (sales losses): $770M-$880M

– “Toyota said Thursday that a software problem is causing problems with the brakes”…

CNNMoney, Chris Isidore, senior writer, Feb 4, 2010.

GM presses suppliers for future recall costs (Automotive News, Aug 5, 2013)

– GM has adopted a new purchasing contract that would allow it to recover the cost of safety recalls from its suppliers.

– The new GM contract has open-ended implications, stating that the supplier's components "will not, at any time (including after expiration or termination of this contract), pose an unreasonable risk to consumer or vehicle safety."

6

http://money.cnn.com/2010/02/04/autos/Toyota_Prius_complaints/index.htm?postversion=2010020407



So… What is Quality? Exceeding end users needs, even when they keep changing.

No single definition, most definition are very detailed and verbose… but the common theme is most definitions is:

Yesterday: Meeting the Spec.

Today: Meeting or exceeding customers and end-users needs and expectations

– “End users needs” ≠ “Initial Requirements”

7

“The test of innovation lies not in its novelty, its scientific content or its cleverness.

It lies in success in the marketplace.”Peter Drucker.

(Business Management Philosopher)



Agenda

8



Continuous Verification & Validation ProcessEnd users needs, at least initially, are imprecise and incomplete – This is Normal!

Starts with early elicitation of end-users needs Helping the customer/end user to flash out their needs (Requirements Elicitation)

– Imagine you want to build a new home, do you know the exact size of the house?

ContinuouslyVerify…

ContinuouslyValidate…

Continuously validate with the customer, as the design is getting more concrete, that you are building the right system.

• As an architect and/or the builder validating assumptions they have made as your house is being built.

Continuously verify, throughout Development, that you are building the system right.

• As the builder checking with the architect.• As the city inspections, making sure the house is

being built in compliance with regulations.

Delivery

Initial Elicitationof end-users needs

Notice that this results in Initial understanding of the needs, still partial and incomplete

Continuously improving understanding of customer/end-users needs



What all the stakeholders need

What the customer needs

What the customer thinks they need

Different perspectives

What the customer says they want

Given thisHow do we get to this?



Verification vs. Validation ExamplesWhile Verification is “Internal”, Validation is “External”

Verification: The evaluation of whether or not a product, service or system complies with a regulation requirement, specification, or imposed condition. It is often an internal process. (IEEE)

• Code should never run thru a divide by zero as it may cause a crash or unexpected results.

• Did you implement the design pattern properly?

• Does this block diagram represent the best architecture for the system?

Validation: The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers. (IEEE)

• On a cycling computer… The customer asked for certain data fields to be displayed… but wouldn't the end user want to customize them?

• Is this {…} the right sequence of voice commands for making a phone call while driving your car?

• What's the right behavior of the wifi enabled home dialysis machine when the connection is intermittent?

11



Verification vs. ValidationWhile Verification is “internal”, Validation is “External”

Verification: Building the system right

– Testing against requirements

• “Internal” requirements

– Utilizes Test, Simulations, Analysis, Inspections and Demonstrations

– Assuring robust design and implementation

– Compliance with industry regulations

– We “know” how to do that… often done well!

Validation: Building the right system

– Validating against your customers needs

• As expected, imprecise and incomplete

• Often negotiated with several teams

• Often negotiated with people who may not be the end users

– Do we know how to do that? Do we do it?

12

SystemTesting

RequirementsAnalysis

Systems AnalysisSystems Design

Detailed Design&

Implementation

IntegrationTesting

CustomersNeeds

Note that the V does not represent a development process but rather a flow of information as we decompose and recompose a system.



Verification vs. ValidationWhile Verification detects defects, Validation is about avoiding them.

13

Left side of the V

Right side of the V

Verification • Early defect removal

• Expensive defect removal

Validation • Defect Avoidance

• Early detection of potentially business or mission failures

• Early Acceptance tests

• Final Acceptance test

SystemTesting



Detailed Design&

Implementation

IntegrationTesting

CustomersNeeds



Verification vs. ValidationWhile Verification reduces Technical risk, Validation reduces Business or Mission risk

Verification: Reduces Technical risks

– It is a Testing process…

– We “know” how to do that… often done well!

Validation: Reduces Business risks

– Focuses on closing the loop with the end-user as often as possible

• Can’t just blame the Requirements being wrong…

• “Meeting the spec” is not a guarantee for business or mission success…

– It is NOT a Testing process… but rather an on-going Discovery process

• Important to “developers”

• Critical to End Users – Helping users figuring out their own Requirements!

– Do we know how to do that? Do we do it?

14

TechnicalRisk

Business Risk



Continuous Verification and Validation (V&V)Compressing time to customers feedback!

CustomersRequirements

Late Validation (Re-Evaluate)

SystemTesting


IntegrationTesting

Detailed Design&

Implementation


SystemTesting


IntegrationTesting

Detailed Design&

Implementation


Very Common:

SystemTesting


IntegrationTesting

Detailed Design&

Implementation


Long/CostlyProcess

Example:

AA and United approached Boeing for a new flight entertainment system

Job has been sub’ed to Sony Trans Com., who designed & delivered a state-of-the-art system.

End result:

– Premium paying passengers had trouble operating the system

– Flight attendants had trouble supporting the passengers

End-End result: Flight attendants shutting down the system before take off, claiming malfunction.



Continuous Verification and Validation (V&V)Compressing time to customers feedback!

16

CustomersRequirements

Late Validation (Re-Evaluate)

SystemTesting


IntegrationTesting

Detailed Design&

Implementation


SystemTesting


IntegrationTesting

Detailed Design&

Implementation


Validation

CustomersNeeds

Verification

Validation

Verification

Validation

Verification

Validation

Verification

Validation

Verification

Continuous V&V per Feature:

Very Common:

SystemTesting


IntegrationTesting

Detailed Design&

Implementation


Long/CostlyProcess

Use

Scenarios?Operational Scenarios? Prototypes? HIL/SIL? Final?

Compressing time to customers feedback!



Agenda

19



Collaboration through ReviewsTesting for early feedback from Internal and External stakeholders

Review process where peers and other stakeholders collaborate in order to identify potential errors and deviations from the spec.

Objectives include:

– Detect wrong understanding, bad assumptions and defects close to when introduced (Functional review)

– Identify potential improvements and/or risks (Functional review)

– Verify compliance with regulations and conformance to standards (Quality review)

Audience:

– External Reviews: With customers and/or end-users,often focused on Validation

– Internal Reviews: With internal stakeholders (peers, management), often focused on Verification

Formal inspections (with defined roles & process) to ad-hoc team reviews and walkthroughs.

FDA Example: On Going Reviews / Late Validation

20



Reviews are collaboration pointsInclude the voice of the customer for on-going validation

Applies to the full development life cycle and to all artifacts

Reviews are short, and per feature (best practice: focus on end user feature)

Examples:

21

Review Objective Internal/External

Requirements elicitation Identifying Use Scenarios

External Review What are the main functions/activities?

Requirements Review Defining Operational Scenarios

External Review How does the system operate?

Architecture Design Review

Is this the “best” architecture?

Internal Review Design documents

Initial Prototype Review Early feedback based on feature execution

External review Low Fidelity prototype, early feedback.

Advanced Prototype Review

Production code in HIL/SIL environment

External Review Hi Fidelity prototype, still early feedback.

Test Plan Review Agreement across all stakeholders

Internal and/or External Review

Sunny & Rainy days scenarios

Test Execution Review Discuss Quality status and next actions

Internal and/or External Review

Focused on high level operational scenarios



Reviews ImpactReviews are ~2x more efficient in removing defects than Testing(Not including Defect Avoidance)

22

Reviews

Testing

Source: Capers Jones, 2012



23

Reviews ImpactReviews are ~2x more efficient in removing defects than Testing(Not including Defect Avoidance)

Reviews

Testing



End Users Operational ScenariosUse them as top level test cases to continuously drive all V&V activities

Operational Scenarios

– Quasi formal but yet intuitive

– Can be extended with sketches and Interaction Diagrams

Operational Scenarios are a Testable versionof the requirements!

– Can be converted into System level test cases

System level test cases:

– Should be Validated (reviewed and approved) with the customer and/or end-user

– To be used as a Golden Reference model throughout design and implementation for Verification purposes.

Although diagrams shown here are SysML diagrams, concept applies to any system.

24



25

Validate your understanding of the needs by reviewing the expected scenarios with the stakeholders and getting them approved.

A Modeling environment that support Use Case analysis

A Quality Management process that support the conversion of Operational scenarios into test cases

– Test cases for either Manual Testing or Automated Testing

– May require internal review and approval

Traceability: Needs Requirements Operational Scenarios Test Cases

Virtual

Prototype

RealSystemLevel Test

Cases

OperationalScenarios

To leverage System Level Test Cases, you need…Use Operational scenarios as Tests throughout the whole development Lifecycle



Example User Scenario

To have sailedand

survived

Overall

goal

Sub-goals

Ready to sail

Sailed

Returnedhome

Boatloaded

Boatunloaded

Boatrigged

sequential

Boat lifted

Boat on car

Mast rigged

Center-plate rigged

Rudder riggedparallel

Maneuvered

periodic

Ashore

Gybed

Tacked

Cruisedalternate

alternateRecover

from Capsize

Sailednormally

exception

Righted

Coast guard

contacted



Requirements from Scenarios

Goal hierarchy

Stakeholder requirements

Two people shall be able to lift the boat onto the

roof of the average saloon car.

The sailor shall be able to perform a gybe.

The sailor shall be able to contact the coastguard

when the boat is capsized.

traceability

To have sailedand

survived

Overallgoal

Sub-goals

Ready to sail

Sailed

Returnedhome

Boatloaded

Boatunloaded

Boatrigged

sequential

Boat lifted

Boat on car

Mast rigged

Center-plate rigged

Rudder riggedparallel

Maneuvered

periodic

Ashore

Gybed

Tacked

Cruisedalternate

alternateRecover

from Capsize

Sailednormally

exception

Righted

Coast guard

contacted



Virtual ModelsEnabling early Verification and Validation… before building anything real.

Virtual models that can be used for early analysis and trade studies (functionality, behavior, architecture, structure, performance, reliability, safety, etc.)

– Models abstract mechanics, electronics and/or software entities, operating independently or integrated

– Different domains:

• Mechanical/Control models (such as Mathworks, NI)

• Graphical languages (such as SysML)

• Textual language (such as SystemC for HW design)

Can be used for

– The system being designed AND its TestBench (Plant models)

Use System level test cases to drive analysis

– Execution, Simulation or Prototyping

Various configurations driving on going system analysis:• Eg. 1: Virtual sensors driving virtual HW+SW models

• Eg. 2: Mechanical prototype driving virtual HW+SW models

• Eg 3.: Mechanical prototype driving virtual HW+production SW

28



Models are Executable

– You can ask Questions and get consistent answers!

Models can be Analyzed

– Tradeoff studies

– Performance

– Behaviors

– Etc.

Leverage correct-by-construction Synthesis capabilities to guide the design and implementation process

– Eg. 3D printers, C/C++ for SW, SystemC for HW

– Enabling Co-Execution for V&V purposes

29

Source: Capers Jones, 2012

Model Based:Low Defect Potential,

Best Removal Efficiency

Modeling has low potential of introducing defectsModels are best in removing defects



Effective and Efficient TestingSingle source of truth for all Quality related artifacts and processes

Central Quality Management Hub, acting as a single source of truth, enabling:

– Collaborative Test Planning

• Development and approval for Verification and Validation test plans– As well as specific plans as needed, such as Safety.

• Traceability back to Requirements (why we need which test?)

– Managing test execution (manual and automated) according to Test Plan

• Who is running which test, when, why, pre-conditions, test lab/equipment reservation etc.

– Integrated Defect Management

• What went wrong, how to reproduce, defect resolution, reporting, etc.

– Traceability across the full lifecycle

• User Needs Requirements Test cases Test Results Defects Resolutions

Other key capabilities

– Collaboration around Rainy days scenarios

– Test assets ReUsability

– Reconciliation process between Requirements and Test Plans/Test Cases

– Actionable Analytics to drive process improvements

30



Overall Impact on Testing EffectivenessCollaborative Test Planning, Traceability, Test Execution and Analytics

Impact of Quality Management on Process Efficiency

15%

30%

60%

75%

85%

32%

58%

76%

85% 87%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5

CMMI Levels W/O QM W QM

20% 40% 40% 40% 10%QM Impact:

W/O QM: Percentage of defects detected during Requirements review, design reviews, unit testing and Functional testing – Current practice

W QM: Percentage of defects detected thru Requirements review, design reviews, unit testing and Functional testing – Improved practice

Source: Analysis of over 850 GBS projects and Industry data

Detect 3, release 7

Detect 3, release 2

31



Agenda

32



Key Take-Aways

Verification in Inbound; Validation is Outbound. Both needed.

Verification is a Testing process. Validation is a Discovery process, helping the customer refine their understanding of their own needs.

Compressing time to feedback allows for change… and therefore reduces surprises.

Reduce business risk with Early and Frequent feedback from all stakeholders

– Use Scenarios and Operational Scenarios

– Validation Test Plan

– Low Fidelity and Hi Fidelity prototypes

Operational scenarios = Testable Requirements = Golden reference

– Use them for your Testing throughout the whole development process and not only at the end!

Question:

Unit Testing and Integration Testing are both very important.

Which is more important?

Which one would you do first?

34



Continuous V&V – Reducing Technical and Business Risks

35

Quality can be delivered by compressing the time to customer feedback with

continuous Verification and Validation.

Both Technical and Business risks can be lowered

Business Implications:

– Lower cost of Quality

• Defect Avoidance, Lower defect density, higher defect removal efficiency

– Deliver the right system

• Compressing time to customer feedback

– Better time to market predictability

• Earlier convergence on the right Requirements

TechnicalRisk

Business Risk



3636



© Copyright IBM Corporation 2013. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or

otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its

suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM

software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change

at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, and other IBM products and services are trademarks of the International Business

Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be

trademarks or service marks of others.

www.ibm.com/software/rational

37

http://www.ibm.com/software/rational

Technology

Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation