Cost of Quality How to Save Money

Sibgatullina Taisiya

Solution Consultant, CEE

Cost of Quality

How to Save money

2 HP Application Lifecycle Management & HP Quality Center 11.00

Part 1: Quality assurance

3 HP Application Lifecycle Management & HP Quality Center 11.00 3

Cost Of Quality


Application Lifecycle model


ALM

5

HP

(41.1%)

IDC Worldwide Automated Software Quality Revenue by Vendor in „09

Market Leading Solutions

IBM (Rational) 11.1%

Microsoft 5.6%

Compuware 5.3%

Borland 2.9%

Empirix 1.9%

Others (Combined)

32.1%

Source: Melinda Ballou IDC, “Worldwide Automated Software Quality 2009-2013 Forecast Update and 2008 Vendor Shares: Evolving Quality in a Challenging Market” (Doc # 219695E) October 2009

“As the market revenue leader, with a 41.1% share, HP is in a position to disproportionately influence the overall ASQ market.”

©2009 HP Confidential 7

New Gartner report just published in January 2011

HP ALM #1 AGAIN

Designed for the End-to-End Application Lifecycle

HP ALM 11.00 is a Unified Platform

Requirements Management


Requirements Definition

Quality Management

Functional Performance Security

Development Management

IDE, SCM, Build

Integrations

Defect Mgmt

Secure Develop-

ment

Foundation

Traceability Reporting Process Standardization

(i.e. Waterfall, Agile)

Extensibility (Open APIs)

Inte

gra

tions

into

Str

ate

gy a

nd P

lannin

g

(PPM

, G

ov)

In

tegra

tions in

to R

un

(Deplo

yment, A

PM, ITS

M)

Unified for the Core App Lifecycle, Integrated with the Complete App Lifecycle

Application Lifecycle Management

Project Planning & Tracking Enterprise Release Management

ALM Platform

Process Standardization (i.e. Waterfall, Agile)

A single platform for managing the application lifecycle

HP Application Lifecycle Management

Dashboard

Single Data Repository Reporting Open APIs (REST)

HP Application Lifecycle Management

Release Management


Defect Management

Project Planning & Tracking

Business Models

Test Lab Test Plan

HP Unified Functional Testing

Performance Optimization Application Security Testing

Business Application Validation

HP QuickTest Professional

HP Service Test

HP Performance Center

HP LoadRunner

HP QAInspect HP Business Process

Testing

Services

Service Test Management

Test Data Management

Asset Sharing & Reuse


Project Planning & Tracking Track & measure project milestones and KPIs

E2E Testing

Readiness Assessment

Testing Progress

92% Req Reviewed

Req Approved

0 Defects High

Defects Med

Test Exec

Tests Passed

15

98%

92%

10 Defects High

Defects Med

Test Auth

Test Exec

Tests Passed

25

98%

89%

45 Defects High

Defects Med

Test Exec

Tests Passed

95

78%

90%

93%

Test Authoring

Integration E2E Testing

User Acceptance

Testing

Requirements Authoring

Development Test

Execution

Functional, Performance and Security

95%

85% Req Implemented

Tests Reviewed

75% Development

Progress

Requirements Progress

11

Integrated solution for defining, managing and tracking requirements


– Business analyst can collaborate on application requirements without fear of overriding other‟s work

– Assess overall risk to the business through real-time visibility of test coverage and associated defects against business needs

– Assess the impact of change before changing requirements

– Easy adoption by Business Analysts

– Define and track multiple requirement types

– Establish templates to ensure higher quality requirements

– Manage requirements dependencies

– Bi-directional traceability from requirements to requirements, tests and defects

– Manage change requests through formal approval process

– Familiar rich test editor interface

– Version Control and Baselining

– Leverage existing assets in MS Word

12

Track changes made to entities in your project over time

Baselining

– Compare releases and cycles

– A snapshot of the project state at a

particular point in time

– Enable rollback of assets to key points in the

application lifecycle

– Compare baselines to assess changes

– Ability to use baselines in test execution

– Create baselines – including traceability

between assets

– Compare baselines both at high level and

drill down to specific changes including

traceability

– Report and print baselines for sign-off

– Control who can create and modify

baselines

13

Track and measure the progress and quality of releases and cycles

Release & Cycle Management

– Manage projects and assets against release

and cycles

– Track release progress

– Gain visibility into release readiness and

release quality

– Compare releases and cycles for continuous

process improvement

– Manage requirements & tests according to

application releases and dev/QA cycles

– Compare planned vs. actual progress for

each application release and cycle

– Assess release quality across multiple cycles

14

Track and measure project milestones and KPIs

Project Planning & Tracking

– Visibility into application projects progress

and status

– Visibility into key project milestones

– Identify delays or poor quality before it

impacts the schedule

– Proactively adjust plans when a project is

not on track

– Define and track project milestones

– Define and measure key performance

indicators across the application lifecycle

– Automatically updates project status against

milestones

– Drill down on KPIs for more details

15

Define, manage and track all test scripts

Test Plan

– Create test cases to adequately test the requirements

– Manage all test types (Functional, Performance, Security) in one place

– Ensure requirements coverage through traceability to requirements

– Manage change associated with requirements changes

– Design, build and manage manual and

automated test cases

– Link test cases to requirements

– Leverage test cases from MS Word and MS

Excel

– Version Control and Baselining

16

The next generation of manual testing

HP Sprinter

– Accelerate manual testing

– Improved testing accuracy

– Better communication of defects to development (accelerated defect remediation)

– Unobtrusive, easy to use user interface

– Supports exploratory testing

– Reduce time and resources by testing multiple environments at once

– Toolkit – Screen capture and annotations, measure objects position etc.

– Record user actions to support exploratory testing

– Automatic data injection

– Mirror testing

– Log defects with associated information directly into HP Quality Center/ALM

17

Track defects across the entire application lifecycle

Defect Management

– Manage, track and enforce defects across the application lifecycle

– Create defects from manually or directly from the execution of manual and automated tests

– Clearly communicate bugs to developers with context from requirements and tests

– Integrated into developers IDE

– Enforce defect lifecycle process through custom fields and workflow

– Log defects directly from test runs with test details (test step, result, screenshots, attachments) to ensure the developers can reproduce the issue

– Traceability to requirements, tests and other defects

– Alert key stakeholders through automated emails

18

Clear communication and visibility into your application projects

Reports, Graphs & Dashboard

– Make informed business decisions

– Clearly communicate to all stakeholders

– Understand the real-time status of an application

– Full trending analysis and insight into application projects

– Understand release and cycle progress

– Fully customizable project reports (export to MS Word)

– Integrated Dashboard

– Out of the box or customizable analysis reports and graphs

– Report to MS Excel

– Embed graphs and charts into web portals

– Requirements traceability matrix

©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

Development tools integration

• Traceability between requirements, code changes and defects

• Adjustable processes

• Side-tools integrations

Working with defects in various tools

Quality Center

Release & Requirements

Test Plan

Test Lab

Defects

Developer

STRATEGY/

DEMAND

Strategic

demand

• New

applications

• New services

• Application

integrations Operational

demand

Other tools

JIRA

BugZilla

StarTeam

CCE Tracker

VersionOne

Development tools integrations

• Tasktop & CollabNet

• Eclipse & Eclipse-based clients

• Visual Studio

• CollabNet

• AccuRev

• Perforce

• CVS

• ClearCase

• CollabNet

• Hudson

• Electric Cloud

• AnthillPro

TFS

HP Application Lifecycle Intelligence

SVN Jenkins

TFS

CVS Electric Cloud

SVN Hudson

Waterfall

RUP

Scrum

Hybrid

Single Platform

Test Build Req

HP ALM

TFS

The Decision Support System for ALM…

SVN Jenkins

TFS

CVS Electric Cloud

SVN Hudson

Waterfall

RUP

Scrum

Hybrid

Single Platform

Test Build Req

HP ALM

Key ALI Features

Code changes from Subversion

Code changes from TFS

New Module

Drill down

End-to-end app delivery Report

Key ALI Features

HP ALM 11 and TFS integration

QC TFS

TFS adapter for QC Synchronizer 1.4

Defects

Bugs & Reqs

Work Items

Reqs


Part 2: Performance validation

Solutions for management and automation of performance optimization

HP Performance Center

User/privilege management

Infrastructure management

Central repository

Global access & collaboration

Requirements & Defects

Management Framework: Performance Center

Resource pooling & scheduling

HP Performance Center Family

Performance Testing: LoadRunner / Performance Center

VuGen Controller Load Generator Analysis Monitors

Diagnostics

J2EE .NET SOA SAP Oracle

The market leading solution for performance validation

HP LoadRunner

Enables a consistent, repeatable process to: • Emulate production

workload to test a broad range of applications

• Identify performance

bottlenecks, diagnose the root cause of issues

• Optimize application

performance before deployment

Reduce risk of deploying systems that do not meet performance requirements

Reduce cost by predicting scalability and capacity and eliminating over-deployment

Reduce time by shortening test cycle time. Accelerate delivery of high quality applications

– Sequential process takes time

– Scripting expertise needed (especially for complex apps)

– Protocol level scripts break with minor app changes

Current scripting process

The challenge: Scripting process

Record a script

Correlations Parameterize

data Add

transactions Add logic

– Many technologies, frameworks and toolkits

– Very customizable – almost no “vanilla” usage

– Dynamic, asynchronous behavior

– Client-side processing and logic

– Most testing solutions do not support all the customizations

New Web 2.0 technologies

The challenge: Scripting for web 2.0 apps

Services

Web Server

Web Services

Web Server

Web Server

Web Server

App Server

Data Server

Networks Data

Server

Cloud

App Server

EDW

Client

LoadRunner VuGen

Record an application business process into an automated script Add end-user transactions, parameterize input data, add verification checkpoints Apply user behaviors such as network connectivity or pacing

Application

Scripts

Business process

VuGen Recorder

Browser

LoadRunner Controller

Design and execute load scenarios that mimic production conditions

Emulate hundreds or thousands of concurrent users

Gather end-to-end transaction response times

Collect application, network and system data across all architectural tiers and components

Diagnostics

Monitors

Load

balancer

Web

server

App servers

Database Network Controller Load

generators

VUsers

US

Europe

The revolutionary way of scripting modern apps

INTRODUCING: LoadRunner TruClient

Scripting engine embedded within

browser

Transactions, and logic can be added while scripting

Data can be parameterized

while scripting

Automatic suggestion of automated steps

No correlations required!

Interactive, dynamic scripting!

Supports all Ajax

applications

Diagnose performance bottlenecks

Key benefits

• Monitor system performance during tests

• Diagnose performance bottlenecks as they occur

• Identify application code level problems

• Resolve issues faster

Key capabilities

• Sitescope monitoring for system performance metrics

• Diagnostics to drill down to the method level

• Data available during run time as well as for post-test analysis

Monitoring and Diagnostics

LoadRunner Performance Center 36

6 November 2011

Customize and extend the industry-standard platform

Key benefits

• Build niche/custom protocols

• Export analysis data for custom reports

• Include custom monitoring and diagnostics

Key capabilities

• Custom protocol development via a SDK

• Analysis API to export data for custom reports

• Extensible monitoring and diagnostics framework

Extensible Platform

HP Partners

--> •HP supports 46% of protocols customers need.

•New Demand = 30% of current usage

•25% of customers requested AJAX, RDP and VoIP

•Smaller demand for multiple protocols sums up to big opportunity

LoadRunner Performance Center 37 6 November 2011

Performance Requirements & Defects, Coverage reports

Performance Center 11.0

Performance Requirements

Requirement status

Requirement coverage

New PC Lifecycle User to access performance requirements and defects

Next-gen platform: based on the ALM platform


Performance Center

ALM Platform

Performance Testing

Performance Lab Mgmt

(scheduling)

Performance Requirements &

Defects

Performance Test

Management

Admin Console Users & license Projects CoE

Collaboration Traceability Reporting Version Control

Authentication Authorization Workflow APIs, Integrations

Topology


Drag & drop from list to build your

topology diagram

Define machine names and properties

Right click to set up monitors

– Timeslots

• Redone in Outlook fashion

• More information available regarding booked timeslots

• Allows for maintenance windows

Miscellaneous


Shows when you can start a test based on your

requirements

Timeslots


Gives information on used timeslots

– Timeslots

• Redone in Outlook fashion

• More information available regarding booked timeslots

• Allows for maintenance windows

Miscellaneous


Shows when you can start a test based on your

requirements

Timeslots


Gives information on used timeslots

Operational security

management

OPERATIONS

Service desk

Production monitoring

Connect to

pro

ductio

n

Go/ No Go

Performance Center – fully integrated into the application lifecycle

45 6 November 2011

A repeatable quality management process mitigates risk

45 6 November 2011

Inte

gra

te w

ith d

em

and

Functional requirements

Business requirements

Security requirements

Performance requirements

Other non-functional

requirements

REQUIREMENTS MANAGEMENT

Align with management and stakeholders

Collaborate with design and development teams

Assess and

Analyze risk

Establish testing

priorities

Create test plans

RISK-BASED TEST PLANNING

TEST MANAGEMENT AND EXECUTION

Execute security scans

Identify and customize security

policies

DEFECT MANAGEMENT

Execute functional tests

Create manual test cases

Automate regression test

cases

Execute tests, diagnose and

resolve problems

Create performance scripts and scenarios

Connect to

pro

ductio

n

STRATEGY/ DEMAND

Strategic demand

• New applications

• New services • Application

integrations

Operational demand

• Defects • Enhancements • Change requests

Enterprise Architecture and

Policies

• SOA • Security


management

OPERATIONS

Service desk



management

OPERATIONS

Service desk


STRATEGY/ DEMAND

Strategic demand

• New applications

• New services • Application

integrations

Operational demand

• Defects • Enhancements • Change requests

Enterprise Architecture and

Policies

• SOA • Security

Go/ No Go

Execute tests, diagnose and

resolve problems

Create performance scripts and scenarios

Part 3: HP UFT

Part 3.1 Functional Testing and Service Test

A Modern testing team has new needs

People

• Expand skills to test beyond the GUI

• Extend testing to more members of the team

• Increased collaboration with developers and business analysts

Processes

• Automate as early as possible

• Accommodate faster change

Technology

• Complex integrated systems

• Test GUIs, APIs, web services and vertical integration

• Test RIA, Web 2.0

Late stage testing is too expensive…

Iteration 1 Iteration 2 Iteration 3 Test

When it was injected way back here? Why wait to find a bug here?

Remember, the age old QA principle…

Time between injection

and identification

Cost to

Fix

Test & automate as early as possible

C1 C1

C2

C1

C2

C3

G2

C1

C2

G1 G1

C3

C1

C2

G1

G3

G2

C3

C1

C2

G1

… …

BP1

BP2

G3

G2

C3

C1

C2

G1

Regression test at the end of each iteration or as part of continuous build

Automate tests when available components

become stable

C GUI-less component

G GUI element

BP business process

stable

component

Modern applications have many layers Process Layer

A new transaction

Exchange

Screen 5 (Java)

Screen 6 (.NET)

SAP

Screen 3 (.NET)

Screen 4 (Ajax)

Web 2.0 Billing

Screen 1 (Ajax)

Screen 2 (Java)

End to end business process

Graphical user interface

layer

Headless Layer

Interact with modern GUI interfaces, Web 2.0

Continue the scenario to the headless layer; test the integration between the layers

Report on the quality of the entire end to end process

Component 1 (.Net API)

Component 2 (Java API)

Component 3 (web service)

Component 4 (JMS service)


Component 6 (REST service)

HP Software solutions Process Layer

A new transaction

Exchange

Screen 5 (Java)

Screen 6 (.NET)

SAP

Screen 3 (.NET)

Screen 4 (Ajax)

Web 2.0 Billing

Screen 1 (Ajax)

Screen 2 (Java)

End to end business process

Graphical user interface

layer

Headless Layer

Component 1 (.Net API)

Component 2 (Java API)


Component 4 (JMS service)


Component 6 (REST service)

HP Unified Functional Testing

HP Unified Functional Testing includes the ability to test the GUI layer + the Headless layer + the end to end process.

Unified Functional Testing

HP Service Test HP Functional Testing

(HP QuickTest Pro + all add-ins)

Call activities from one product to the

other

Transfer complex data structures

Integration

Unified report

What‟s new with HP Unified Functional Testing 11.0?

Report for a Unified Functional Testing scenario

Shows activities from both HP Functional Testing &

HP Service Test

Multiple views of the flow

Full traceability for easier debugging

Testing Web 2.0 GUIs

• Ajax toolkits including Dojo, GWT, YUI

• RIA such as Flex, Silverlight v2.0, v3.0, (v4.0 soon), WPF extensibility

• New support for Firefox

OOB Web technology support

• Wizard-like tool helps users extend support to more web 2.0 controls

• Partners and customers can share code created to customize support for add‟l toolkits

Community

Easy, wizard-like interface

Extends HP Functional Testing fast; uses Intellisense

Automated deployment

Free for anyone? Yes, FREE!

Opening web extensibility to the online community: The Extensibility Accelerator

HP Service Test 11.0 - new Visual Test Designer

11/6/2011

Modern, intuitive test design interface

Reusable modules improve maintainability

Visual test designer extends testing to non-developers

Flexible and extensible service processing

BPT Positioning

What is BPT?

Business Process Testing is a framework for QA organizations to build and reuse structured testing components in both manual and automated testing.

Target users

Non-technical QA personnel

Subject Matter Experts

BPT is ideal to Build frameworks to maximize reuse of testing components

Mitigate risks by aligning business requirements with end-to-end test scenarios

Part 3.2 BPT

BPT 11.0 Simpler. Faster. Better.

What‟s new in BPT 11

Significantly improved performance

Built an all new simple and intuitive user interface

Provided the ability to leverage QC 11.0 Capabilities

Enhanced Traceability via Component Criteria

Rapid Manual Testing with Sprinter Integration

BPT 11 Exponential Performance Boost

More Components = Better Performance

Number of components

Perf

orm

anc

e

Test Script view – New look & Feel

Cleaner UI showing all BPT Resources and Dependencies

Clean and Simple Iteration View

•Tree structure instead of a flat list

• Improved Iterations dialog (multi-

iterations view & Single Iterations view)

• new ‘Comments’ column for each

Component instance in the test

• Search option inside the Iterations

dialog

One-step Component Creation Process

Simplified the component creation process from three steps to just one

step

Enhanced Drag and Drop Feature

Proving the ability to drag and drop multiple test components into the test

script area

Summary of Key Enhancements and Features

• All new BPT UI built using QC web toolkit for seamless look and feel

• Ability to create a manual testing component in just one step

• Ability to drag and drop multiple components into test script pane

• Enhanced test iteration view to show different combinations

• Simple and easy to understand all new test scrip view

• Cleaner look and feel of QC resource dependencies in BPT

• Application area is a standard test resource in BPT

• BPT integration with QC Sprinter

• Enhanced traceability to map requirements to component level

• Major performance improvements with a redesigned execution

engine

Traceability & Metrics

Single Platform

Simplified Web Access

Technology Agnostic

Powerful Automation

Quality at the Center

Modular

Integrated

Heterogeneous

Best-in-class

HP Lifecycle Solutions & Operating Principles

COMPLETE LIFECYCLE

COVERAGE

PLAN

Project & Portfolio

Management

DELIVER RUN RETIRE

HP ALM

Quality Center

Perform Center

App. Security Center

Business Availability

Deployment Automation

Service Management

Records Management

Archiving

CORE LIFECYCLE EXCELLENCE

HP APPLICATION LIFECYCLE MANAGEMENT

Project Planning & Tracking Enterprise Release Management

ALM Foundation

Process standardization Reporting Extensibility


Development Management

Quality Management

Definition Versioning &

Change Control Link to

IDE, SCM Defect Mgmt.

Secure Code

Function Perform Secure

SOA Systinet

Why HP?

Integrated management & automation: instead of loosely coupled point tools, HP provides a cohesive platform for lifecycle management and automation.

Technology & methodology agnostic: broadest support for heterogeneous environments (.NET, Java, SAP, Oracle, etc.) – covers more than 70 environments. Covers traditional and Agile methods in single solution.

Project ready, enterprise scale: whether for a team of ten, or an enterprise of tens of thousands, HP solutions offer proven configurability and scalability.

Complete application lifecycle: no other company offers the integrated product coverage of HP to support applications from beginning to end.