The Quality Quotient – Insightful Strategies for Quality IT Project

Releases

October 24, 2015

History

On September 9, 1947, the Mark II technical team of

Harvard University found a moth in the wiring of the

computer system. Computer scientist and US Navy Rear

Admiral Grace Hopper removed the moth, and taped it

in the logbook where she made an entry: "Relay #70

Panel F (moth) in relay... First actual case of bug being

found."

2

https://en.wikipedia.org/wiki/Grace_Hopper

Quality Defined

• Quality is a perceptual, conditional, and somewhat subjective

attribute and may be understood differently by different people.

• The degree to which the product/service was produced correctly, is

reliable, maintainable or sustainable.

• How good or bad something is.

Quality is when expectations are either met or exceeded (depending on how high your expectations were!)

3

Perceptions of Quality

4

Journey from perception to reality is unique - it includes scientific methods and emotional connects

Quotients, Quotients, Quotients

5

Examples of Quotients:

IQ Intelligence Quotient – How bright you are

TQ Technical/Operational Quotient – How able are you to get things done

MQ Motivational Quotient – How driven are you to achieve and grow

XQ The experience Quotient – How many of the requisite kinds of experience you’ve had

PQ People Quotient – How well you handle yourself at work with others (sometimes referred to as EQ)

LQ Leaning Quotient - How deftly you adopt new skills, behaviors and beliefs

QQ Quality Quotient – How well you incorporate strategy, capabilities, process,

talent and testing rigor

Quality Quotient FormulaQuality includes two key parameters:

“Quality of Design” and “Quality of Conformance”

Because of the two perspectives on quality, the de-facto definition

of quality needs to be enhanced to reflect practitioner and

empirical views

q + Q + S = Quality Quotient

where

q = operationally limited to product’s defect rate and reliability

Q = includes product quality, process quality and customer

satisfaction

S = establishing the right strategies, capabilities, heuristics and organizational talent

6

Top Challenges

7

• Diverse Technologies and evolving Application landscape

• Agility to meet Business Strategy and Cost Governance

• Organizational Maturity and Change Management

— Process & Standards

— Vendor Governance (outsourcing

partners, software products)

— Culture of Innovation

• Talent / Skill Mix

Use Case

Very Large Investment Fund Accounting Project

Quality was the #1 priority for the project team

8

• Total Defects = 5600; almost 2700 found by “Model Office”

• Vendor defects accounted for 17% with 44% of those classified as critical / high severity

• Four years in the making

• High complexity

• Large Integration testing

effort

• Several new teams created

This project was perceived as extremely successful due to high quality

Key Attributes of Quality Quotient

9

1. Strategy and Planning

2. Engage Early, Test Early, Test Often

3. Talent & Skill Mix

4. Process and Tools

5. Influential Communication

1. Strategy and Planning

10

• How are we influencing Scope - business acceptance, white box view, defined roles – getting agreements!

• Are our Test Objectives aligned to SMART principles

Is my Quality Strategy aligned

How is my Test Strategy governing Product and Release level - See the Forest through the Trees

How am I instilling Heuristics and Innovation– “How can we ..” culture

Are the Tools supplementing the life cycle

Are we incorporating Lessons Learned?

With 15% increase in planning efforts, test execution efforts can be reduced by 25 – 30%

Test Objectives

11

Sample Test Objectives

Test Level Objective Metric

Requirements 100% of functional, non functional requirements are testable, assessed for impact, have full test coverage and are approved by stakeholders

Execution 100% completion of high and medium test cases and 80% of low priority (time to market driven project). Include Defect Detection % - 2/3rd

defects found in 1/3rd of test execution, % of acceptable defects by phase

Traceability 100% of requirements are risk rated (assigned a risk score); all high riskrequirements are traced to test cases

Root Cause By test level (requirements, code, design, data, etc. defects found in each test level) use a guidepost to determine expectation. Industry examples: 20% requirements, 30% design, 35% coding, 15% other

Automation Percent of automated testing (API vs Non API, Regression), failure rates, # sprints behind

Non Functional Performance results meet X% variance to baseline, % improvement, % of x-priority SLA/SLO met per non functional test type

2. Engage Early, Test Early, Test Often

12

Early Engagement – Influence ecosystem and multifaceted testing needs

• Vendor assessment and proof of concept phases

• Contract phase

Test Early

• Influence requirements (functional / non-functional) - strengthens collaboration between Quality, Business and Dev teams

• Test design techniques integrated with requirements

• Testability, traceability and impact assessment for business process integration testing

Test Often

• In Sprint API automation

• Integrated “Day In The Life” as well as business regression testing

• Integrated Continuous Integration/Continuous Delivery (Jenkins/Hudson)

Sample Techniques

13

MindMap for pair-wise testing

Use cases / Test

Scenarios / Data reqs

Technique/Tool (Pairwise Algorithm,

etc..)

Unified view of disparate entities

3. Talent & Skill Mix

14

You don’t just need testers…..

• Accountable

• Analytical

• Interpersonal and politically savvy

• Problem solver

• Sixth Sense – extra sensory perception

• Influential Communicator (written, verbal, visual)

• Strategic – sees the forest

• Business Domain knowledge

• Developer, ETL and Data testing skills

• Project Management

Your natural aptitude

Your ability that comes from your knowledge, practice or

aptitude

“We know that our best people are scattered everywhere but we don’t know exactly who knows what.”

BA Tester SDET Strategic

Ta

len

tS

kil

ls

Skill Development Plans

Faster BetterSmarter

HO

W?

Setup Knowledge Sharing & adoption

• Dedicated Knowledge Management (KM) focus group - KSM and SDP(Knowledge Silo Metrics, Skill Development Plans)

• Test life cycle integration with KM framework, standards adaptation

Injection of New Roles & Tools

• Introduction of Test Architect, Technical Program Manager, Test Data Architect,

SDET’s (Software Developer Engineer in Test)

• Impact assessment, business regression, RBT, Pairwise

Injection of new skills

• Enabling non-UI Automation with SDET skills, up-skill to develop hybrid testers

• Performance Engineers

• Test Data Architects

15

4. Process and Tools

16

• (Prescriptive) Standards – lightweight with guiding principles

• Move from “Silo” department functions to a culture of embedded/integrated mindshare

• Process – focus on objectives, allow innovation with some balance• Define table stakes (i.e. IT Controls)

• Tools – Process and Engineering(continuous build and release, code coverage, etc.)

• Continuous Improvement

“We cannot have islands of excellence in a sea of slovenly indifference to standards” - John W. Gardner

Scorecard Maturity Levels

17

• Level -1 – Basic— Does not meet criteria for level 2

• Level – 2 – Operational— Example – Quality is governed by Test Approach

which is base-lined, approved and continuously refined…

• Level – 3 – Insightful— Example– Enablement of upstream defect

prevention, refined using test optimization techniques…

• Level – 4 – Influential— Example– Quality approach is a key influencer,

focus on expected business outcomes and strongly advocated…

• Level – 5 – Aspirational— Example– Quality influences IT and business

process with a “Best in QM” view…

A transformation maturity framework, focused on behavior and practices,

incorporating CMMi, TMMi framework standards

5. Influential Communication

• Quality does not stop at defect prevention

• Reminders on cost to value

• Large transformation programs may have overlapping Integration phases

• Go/No Go recommendations

18

Pre-ProductionQuality

— Architecture and Design — Build stability (First Pass Yield), root cause patterns,

corrective actions, defect rejections— Quality gates— Metrics driven reporting with dashboards— Operational readiness

Post ProductionQuality

— Production checkouts— 30 / 60 / 90 day user acceptance

Applied Example

1. Strategy and Planning— Well Defined Scope, roles and responsibilities agreement— Test objectives matured over time (defined well with vendor)— Heuristics matured over time, leveraged lessons learned

2. Engage Early, Test Early, Test Often— Leveraged mindmaps, well defined acceptance criteria, static testing and impact assessment— Traditional and API automation, maturing operational readiness, production checkouts— Business flow and Day in the Life testing

3. Talent & Skill Mix— Good use of talent, quality efficiencies helped drive integration strategy

4. Process and Tools— Leveraged maturing test and development standards and process

5. Influential Communication— Excellent communication throughout (vendor trips, business, dev, integration, architecture,

EPMO), good use of stage gates

Large Investment Fund Accounting Project

19

Conclusion

20

Quality IS in the eye of the beholder. You have a unique perspective that can influence the perceptions of quality.

• Start with a shared understanding of the objectives to align teams to the common vision.

• Put the time and effort into the Quality Strategy

• Find your own Quality Quotients that work for your organization

• Disrupt!

Quality Quotient is not about tools or people or changing technology landscape, it is about …U and I.

QUALITY