Leadership of Software Quality Assurance · CTAL-Full, CSTE, PMP, ASQ CMQ/OE • Leading cross-enterprise collaboration in the DFW area among Software Quality practitioners for the

Leadership of Software Quality AssuranceLeadership of Software Quality Assurance

argodata comargodata.com

Mark Bentsen, CTAL CSTE PMP ASQ CMQ/OE

© 2016 ARGO Data Resource Corporation. All rights reserved.

CTAL, CSTE, PMP, ASQ CMQ/OEQA Manager

2

Customer-Focused Staff350 Employees (as of 6/30/2015)

Richardson, TX; Memphis, TN; Toronto. ONARGO At A Glance Founded in 1980 Founded in 1980 Provides Mission Critical Software and Services Financial Services Best Practice Competency

Lending Sales & Service

T ll P Research &

Corporate Administration

10%

Customer Support

9%

Sales & Marketing

3%

Teller Payments Fraud

Healthcare Best Practice Competency Patient Credit Enterprise Master Patient Identity Production Footprint

Research & Development

47%Implementation32%

p y Patient Transition Care

Financial Strength Revenue: $47 Million Balance Sheet: $192 Million Assets

and No Debt 139

Production Footprint32,500 Retail Offices / 6,000 Contact Center Seats301,500 Workstations100,408,800 Daily Transactions

and No Debt 31% of Revenue invested in R&D over

the past five years

790

170

471 320

20

24

16

25

474 2,569

61

53594

125

* Based on ARGO Audited Financial Statements, FYE June 30, 2015

Experienced Management Team

435181

236

2,757105

65

16

157

548

1,498

1,9046051,587

451

927

158 1,029

286

1,038

87

878

Experienced Management TeamExecutiveMax MartinChairman & CEO

David EngebosPresident & COO

Product ManagementIrene Shippee, VPCraig YorkCalin Sandru, VPRandy Wynn

100

665

2,271

203 415927

599474

600

366

1,050

1,260 130President & COO

Todd RobertsonSVP Business Development

Corporate AdministrationMelissa KirkhamVP Accounting & Finance

y y

TechnologyTed Martin, VPCyrus BavarianJerry Bowman

Implementation & Support


5922 2,578EDMS Monitoring22,613 Financial Centers in 40 States103,230 Workstations and Servers69,338,000 Daily Transactions

Brian JobeVP General Counsel

Jolene HerschDirector HR

Brent TompkinsDon PerryDaniel Baez

3

Wh t W DWhat We DoWith research and development investments, we continuously i d d fi d i i h l improve our products and find inventive ways to use technology to achieve efficient operations.

HealthcareHealthcareCommercialCommercial Fraud & Fraud & RetailRetailEMPIEMPILendingLending BSA/AMLBSA/AMLLendingLending

Sales &Sales &Teller Teller Predictive Predictive Early DetectionEarly Detection


ServiceServicePaymentsPayments AnalyticsAnalyticsyy

Monitoring Monitoring ServiceService

4

Customers


ARGO has over 100 additional customer installations through our resale partners.

5

Mission Critical Support (EDMS) – the ResultsMission Critical Support (EDMS) the Results

Audit Checklist Audit Checklist Prevention

62%ARGO Detector/First Responder

PredictRespond

62%

81%

p

81%

80%


Resolution & Recovery < 1 Hour

6

EDMS Monitoring


7


8

PresenterMark BentsenMark Bentsen

• Quality Assurance Manager, ARGO DataCTAL-Full, CSTE, PMP, ASQ CMQ/OE

• Leading cross-enterprise collaboration in the DFW area among Software Quality practitioners for the past seven years. Gathering corporately on a quarterly basis and on field trips to one another’s work places, the QA field trips to one another s work places, the QA Trailblazers are pushing each other to expand the capabilities of the modern software testing organization.

• QA Manager of ARGO Data, a software development company providing mission-critical and analytical solutions for financial services and healthcare.

• Mark & his wife Melissa are the President Couple of Better Marriages Texas and have been active in Marriage Enrichment since they said “I do” in 2001. Prior to working in technology, he worked in YWAM & Mercy Ships in Switzerland and Namibia.


8

• Contact: [email protected]

9

What is Testing?Fundamentals of Testing

Testing is NOT QA

g

The Coffee Analogy

The problem:

You have coffee grounds in the coffee.

Solution:

□ Tweeze the grounds out□ Tweeze the grounds out

□ Use a better filter

□ Learn how to use your tool


□ Learn how to use your tool

10

What do we do?

Reduce Risk & Eliminate Waste

Effective software testing teams:□ Build confidence□ Build confidence□ Reduce “Risk & Surprises”□ Detect defects early□ Detect defects early□ Provide valuable information


10

11

What do we do?

Reduce Risk & Eliminate Waste

Effective Quality Assurance teams:□ Identify risks□ Identify risks□ Prevent defects□ Focus on continuous improvement of SDLC quality□ Focus on continuous improvement of SDLC quality□ Guard the company brand


11

12

What is Testing?Fundamentals of Testing

The Types of Testing (Green Circles)

g


Figure copyright and courtesy of Rex Black.Advanced Software Testing Vol.2

13

The two essentials of leading technical professionalsprofessionals


14

Priority management and scaling your effectiveness of project deliveryof project delivery


15

Credibility of the testing OrganizationTest Management

Credibility

y g g

Building and keeping credibility

– Credibility is based on trust built over timeCredibility is based on trust built over time

– It can be lost in a moment

– Credibility is not perfectionCredibility is not perfection

– Be quick to admit mistakes and slow to make assumptions

Damage Control – Rebuilding Lost CredibilityDamage Control Rebuilding Lost Credibility

Be honest and open

– Allow time to recover trust– Allow time to recover trust

– Keep relationships and lines of communication open

Be able to explain your position Don’t argue

Copyright © 2015 ARGO Data Resource Corporation. All rights reserved.

– Be able to explain your position. Don t argue.

– Document your findings carefully

16

Test Leadership

I share my yexpectations for the team of testers.

Then they have the homeworkto share their to share their expectations of me.


me.

17

Fundamentals of TestingFundamental Test Process

A Project = Who does What by When


18

Five Whys

Work backward from the problem to identify the root cause.

Ask “Why does this happen?” For each answer ask why again. Continue until the reason is no longer related to the Continue until the reason is no longer related to the

problem. Typically requires asking “Why” five times Typically requires asking Why five times.


19

Non-technical Example

I have a flat tire Because I have a nail in my tire Because I drove through a construction site on my way g y y

to work. Because it’s the only way to get to workBecause it s the only way to get to work. Root Cause: I have a flat tire because I drove through

a construction site on my way to work and drove over a a construction site on my way to work and drove over a nail.


20

Appendix I - Primary Contributing Cause"Primary Contributing Cause", found in the Quality Control Status tab, captures the root cause for the defect. Additional supporting info is to

be included in the defect’s Comments. Primary Contributing Cause is to be assigned during or before dev’s Fixed status.

© 2016 ARGO Data Resource Corporation. All rights reserved. 20

21

Appendix I – Decision Flow to Assign Primary Contributing Cause

© 2016 ARGO Data Resource Corporation. All rights reserved. 21

22

Sequential Development V-ModelTesting Throughout the Software Life Cycle

Testing in the lifecycle

q p


23

Scrum Overview


23

http://www.mountaingoatsoftware.com/scrum_figures

Agile Testing V Model

Test PlanP

r cutio

nnning andreparation Te

st E

xec

d n T


25

Defect Flow Client Implementationsp


26

Appendix III - Defect Governance & SLAs

To identify Issues in ALM that exceed defined service level agreementTo place additional focus on older issues that may not be valid due to product direction or implemented enhancements.


27

Requirements QualityRequirements Quality

The Key to Quality


27

28

Static Techniques and the Test ProcessStatic Techniques

Most defects are introduced in the requirements

q

Typically, the defects introduced in the requirements remain undetected until the test execution phase or worse still until the

Requirements56% execution phase, or worse still, until the

developed system is delivered to the customer, because the original undetected defect also d i i t d i d d l t d

56% Design27%

drives incorrect design, code development, and test case development.

The amount of effort (and the corresponding cost) that it takes to fix Other

10% defects whose origin can be traced to the requirements is even higher at 82%

10%Code7%


29

Static Techniques and the Test Process Static Techniques

Relationship between Requirements, design, and code

q

An error in requirements must be corrected not only in the requirements themselves, but also in the design, the code, and the test requirements themselves, but also in the design, the code, and the test cases. In other words, the rework effort can almost equal the initial design, development and testing effort.


Requirements Design Code

30

Static Techniques and the Test ProcessStatic Techniques

Relationship between Requirements, design and code

q

Unit Test 50%Inspection/

Team Review

Integration Test 18%

85%

Team Review

65 - 90%

Integration Test 18%

System Test 12%

UAT 5%UAT 5%Delivered

to Customer

15%

Testing10 - ~35%

Delivered toCustomer 015%

Approx.10-35%

The typical defect discovery rate on projects that rely exclusively on code-

Customer .015%


level testing to validate application quality, and do not perform rigorous reviews of upstream deliverables is 85%.

31

Cost of DefectsCost of Defects

Myths & Realities


31

Cost of Defects at Different Stages of the SDLCCost of Defects at Different Stages of the SDLC

Capers Jones, Software Assessments, Benchmarks, and Best Practices ,

Addison-Wesley, 2000

Source:


32

33

Phase That a Defect Is Corrected

05.28.

08

McConnell, Delivering Software Project Success: 10 Myths of Rapid Development, 2001


page 33BST Defect Cost Analysis

Pressman Cost Model

Requirements: 1XC d 10XCode: 10XEarly Test: 15-40XLate Test: 30-70XProduction: 40-1000X

Pressman, R.S. Software Engineering: A Practioners Approach, Sixth Ed., McGraw Hill, New York, 2005


35

IBM Cost Model

05.28.

08

Relative Costs to Fix Software DefectsRelative Costs to Fix Software Defects

IBM Systems Sciences Institute


BST Defect Cost Analysis

36

Components of Cost of Software Quality (CoSQ)

C t f P ti C t f A i lCost of Prevention• Solid requirements• Management of quality &

i t

Cost of Appraisal• Work product reviews• Code reviews

T tiprocess improvement• Training• Automation

• Testing• Audit and compliance

activities

Cost of Internal Failure• Analysis• Defect repair

Cost of External Failure• Service Failures• Reputation impactDefect repair

• Crisis management –• Project Time/Costs

• Re-testing

Reputation impact• Crisis management - Ops• App Support• Customer Service callsRe testing

• Opportunity costs related to missing launch dates

Customer Service calls• Defect remediation• Regulatory non-compliance


36

37

Defect Analysis- Cost per Phase 05

$10,000

Avg Defect Cost per Phase

$7 000

$8,000

$9,000 Low/Medium

High Critical

$5,000

$6,000

$7,000

$2,000

$3,000

$4,000

$-

$1,000

$ ,

Requirements L1 Defect Cost L2 Defect Cost L3 Defect Cost Production*q

* EstimateOn average, 10 people touch each defect


38

Defect Cost Analysis Results05

.

Cost

Low/Medium High Critical Average

Total Average Defect Cost $ 293 $ 353 $ 700 $ 449

Requirements $ 50 $ 75 $ 100 $ 75

L1 D f C $ 120 $ 200 $ 300 $ 207L1 Defect Cost $ 120 $ 200 $ 300 $ 207

L2 Defect Cost $ 340 $ 380 $ 800 $ 507

L3 Defect Cost $ 420 $ 480 $ 1 000 $ 633

*Production used factor of 100x as an estimate Production includes defect correction customer

L3 Defect Cost $ 420 $ 480 $ 1,000 $ 633

Production* $ 5,000 $ 7,500 $ 10,000 $ 7,500

Production used factor of 100x as an estimate, Production includes defect correction, customer impact, & lost revenue. Utilizing industry standard- low end weighting

All values averages and rounded to nearest whole number


39

January Analysis- Detecting Defects Earlier

05.28.

08

50% defect shift saves $2.14MM per Corp Release□ Finding 50% of each phases defects in earlier phase

$4.0MM6000 $2 14MM

$3.0MM

$3.5MM

$4.0MM

4000

5000

6000

cts

Current State Costs

Future State Costs

Current State (Jan'08) Defect Count

Future State (50% shift) $7.87MM$8.00MM$9.00MM

$2.14MM Savings per

Corp Release

$1.5MM

$2.0MM

$2.5MM

2000

3000

4000

unt o

f Defec

( )

$5.72MM

$4 00MM$5.00MM$6.00MM$7.00MM$8.00MM

$0.5MM

$1.0MM

1000

2000

Cou

$0 00MM$1.00MM$2.00MM$3.00MM$4.00MM

$0.0MM0 $0.00MMCurrent State

CostsFuture State

Costs


Phase

40

Test Progress Monitoring and Control Test Management

The Value of Testing – Capers Jones

g g


41

The value proposition of The value proposition of unit/component testing

How to Get development Peers ‘On-Board’ with Q lit P ti i D l t W kflQuality Practices in Development Workflows.


41

42

Structure-based or White-box TechniquesTest Design Techniques

Describe the concept and value of code coverage

q

Structural Test Coverage Levels


43

Equivalence & Boundary | Positive & Negative


43

44

The Riskiest of the RisksThe Riskiest of the Risks

“It ain't what you don't know that gets you intoIt ain t what you don t know that gets you intotrouble. It's what you know for sure that just ain't so.”

Mark Twain


Mark Twain

44

45

Fundamentals of TestingFundamental Test Processes

There are five fundamental test activities and respective tasks from planning to closure


46

Your ByproductsYour Byproducts

Increase the Effectiveness of Your Test Coverage & I C d Q litImprove Code Quality


46

47

NO!!NO!!

D ’t Ag t th I iblDon’t Agree to the Impossible.


47

48

Communication Heuristics

Misconception is that status and metrics only go out in email.

If you depend on email, you have one “where” and one “how” in your communication tool belt. There are a lot of other tools available to the wise test manager.


48

49

Test Management Trifecta

What have you completed? What did you learn? What remains?

© 2016 ARGO Data Resource Corporation. All rights reserved.XBOSoft, Inc. All Rights 49

50

BLUF

Bottom Line Up Front Follow with a “Headline” Impact to the triple constraint?

© 2016 ARGO Data Resource Corporation. All rights reserved.XBOSoft, Inc. All Rights 50

51

Communication Effectiveness

“Wh d t “When documents are mostly to enable handoffs, they are evil. When they capture a When they capture a record of a conversation that is best not forgotten, they are valuable ”they are valuable.- Tom Poppendieck


51 Mont

1500 N. Greenville Avenue, Suite 500Richardson, TX 75081

Mark Bentsen, QA ManagerMark Bentsen, QA ManagerCTAL, CSTE, PMP, ASQ CMQ/OE

[email protected]


53

Unit Test - Maturity Model

CMM Unit Test Level Details

el 1 al

Level 0 - UnawareUnaware of unit testing concepts or missing fundamental skills to develop unit test.

Level 1 - IgnoredA belief that not enough time is available for unit testing or that it would not bring benefit to the specific

k t h d

Leve

Initi

Level 1 Ignored work at hand.

Level 2 - ExperimentalExperimentation of basic unit test concepts, typically positive scenarios. Missing strategy as to coverage areas. Typically used by creator of test and not others within the organization. Likely not maintained for reuse..

le Level 3 - IntentionalIntentional effort to build some unit test in places throughout the development lifecycle. May not

i id i i (h h) i

Leve

l 2R

epea

tab l Level 3 Intentional represent test scenarios outside positive (happy path) testing.

Level 4 - Positive/Negative TestIntentional effort to build positive and negative unit test throughout the development lifecycle. Understanding of testing principals beyond positive (Happy Path) testing techniques.

Level 5 - Positive/Triangulation TestSpecific test with different input and expected results than the positive test to ensure no hard coded return results.

Leve

l 3D

efin

ed

return results.

Level 6 - Positive/Negative/Boundary TestIntentional effort to build effective unit test leveraging appropriate testing principals such as Positive, Negative and Boundary testing. Effective communication channels in place between development and QA.

Level 7 - Mocks and StubsMocks and Stubs in place to replicate dependent functionality.

L D

Level 8 - Designed for TestabilityCode that is easier to test due to development design. Clear delineation and simplicity in design.

Level 9 - Test Driven DevelopmentBegin development process by building unit test which evolve with primary code development. Designed for testability. Red, Green, Refactor. Never write a line of code that doesn't have a failing test.

Leve

l 4M

anag

ed

Level 10 - Code CoverageIntentional effort to build unit test to measurably cover functionality, logic and lines of code across the development.

Level 11 - Unit Test in the Build Automated unit testing during the build process (CI). All Unit Test must pass in order to consider the build successful.Awareness of Unit Test code coverage across an organizations landscape ensuring consistency in testing


53

Level 12 - Code Coverage Awarenessa e ess o U t est code co e age ac oss a o ga at o s a dscape e su g co s ste cy test g

practices. High level dashboards showing metrics down to individual projects regarding code coverage and last execution times.

Leve

l 5O

ptim

izin

g

Level 13 - Automated Builds and TasksFully automated build and reporting process. Bringing awareness to the collective and individual health of the SDLC process.

Documents

Leadership of Software Quality Assurance · CTAL-Full, CSTE, PMP, ASQ CMQ/OE • Leading cross-enterprise collaboration in the DFW area among Software Quality practitioners for the