Indwelling and outflowing – applying the jidoka technique

Collabera Collabera

Indwelling and Outflowing – Applying the JIDOKA Technique

Moses & Smitha

Collabera solutions

Bangalore

1

Collabera

2

Collabera

3

Collabera

4

Collabera

5

Collabera

6

Controllable Input Lead

Non Controllable Output

Lag

6

Collabera

Requirements structure

Test case adequacy Code

structure

Structural application profile JIDOKA

7

Collabera

8

Collabera

9

Collabera

10

Collabera

11

Collabera

Population : 20.5 M Richest city in India, Per-capita income in 2009 was 486,000 (US$10,692) ( 3 times the national average) 6% of India’s GDP & Highest GDP of any City in South/West/Central Asia Commercial and entertainment capital of India & one of the world's top 10 centers of commerce 70% of capital transactions to India's economy Home to premier financial, scientific & nuclear institutions A potpourri of many communities and cultures.

12

Collabera

Figuring out

13

Requirements

Architecture

Design & Coding

Unit Testing

System Testing C

ust

om

er

Cu

sto

me

r

? ? ? ? ?

Requirements Review defects

Architecture review defects

Code Review defects

Unit Testing defects

System testing defects

Effort

Schedule

Quality Control Metrics

Project Management

Engineering Metrics

Collabera

14

Collabera

15

Pre-SIT SIT UAT

Cost of fixing 3.5 18 80

0

20

40

60

80H

ou

rs f

or

fixi

ng

Cost of fixing defect

Collabera

JIDOKA-Step By Step

Requirements structure

Test case adequacy Code

structure

Structural application profile

16

Collabera

CASE Study : Correlating use case to SIT

Use Case

Volatility

Detailing

Verification

Complexity

# of change after first baseline

Level 1 Basic & alternate flows Level 2 Additional details

Level 1 No SME reviews Level 2 SME reviews

As defined in UCP template Based on # of transactions

SIT defects Mapped to each use case

UAT defects Mapped to each use case

Production defects

Mapped to each use case

Functional Defects

α

17

Collabera

JIDOKA in Requirements

“Can I say my requirements are at a level “ X” which signifies the quality of my requirements?” “Can I set thresholds and monitor my requirements that enables me to do course corrections?”

18

Collabera

Gold – Silver - Bronze Rating

Basic Flow Available, documented and reviewed.

Alternate scenarios added and reviewed

Special considerations of non functional and interface aspects documented and reviewed. Validated through prototypes

19

Collabera

Requirements Qualification matrix

Planning & Tracking iterative assessment (Score Card)

Business Requirements assessment (Score Card)

Functional Requirements assessment (Score Card)

Non Functional Requirements assessment (Score Card)

Requirement documentation assessment (Score Card)

Requirement Phase Design Phase Proposal / SOW / Discovery 20

Collabera

TS N TS 2 TS 1

Requirements structure- A deeper insight

Category 1

Category 2

Category 3

Category 4

Category 5

80% 60% 20%

# of use cases and # of reports

% Use cases with Alternate / exceptional flows

% use cases without business rules

% use cases with Open points

% Use cases with screens/Prototypes % of use case with QOS criteria

21

Collabera

Figured out - Requirements

22

Requirements

Architecture

Design & Coding

Unit Testing

System Testing C

ust

om

er

Cu

sto

me

r

% UC with Alt flows % UC with Open points % Ucs without business rules % Use cases with screens/Prototype

? ? ? ?

Requirements Review

Architecture review

Code Review Unit Testing System testing

Effort

Schedule

Quality Control Metrics

Project Management

Engineering Metrics

Collabera

JIDOKA-Step By Step

Requirements structure

Test case adequacy Code

structure

Structural application profile

23

Collabera JIDOKA - Architecture

Attributes Baseline Arch

Candidate Arch 1

Candidate Arch 2

Candidate Arch n

Map all Stakeholder Concerns

Identify Architecturally Significant Requirements

Architecture Strategy & Style

Architecturally Significant Decisions

Architectural Views

Architectural Analysis & Evaluation

Number of ASRs

% coverage of BR to ASRs

# of architectural styles/patterns

Ratio of ASRs covered through structured decision making techniques ( to tradeoffs, risks & sensitivities )

Completeness of architectural views

Quality attribute ranking

24

Collabera

Figured out - Architecture

25

Requirements

Architecture

Design & Coding

Unit Testing

System Testing C

ust

om

er

Cu

sto

me

r

% UC with Alt flows % UC with Open points % Ucs with more than 2 changes % Use cases with screens/Prototype

% coverage of BR to ASRs # of arch styles /patterns Ratio of ASD t o tradeoffs, r isks & sensitivities

? ? ?

Requirements Review

Architecture review

Code Review Unit Testing System testing

Effort Variance

Schedule Variance

Quality Control Metrics

Project Management

Engineering Metrics

Collabera

JIDOKA-Step By Step

Requirements structure

Test case adequacy Code

structure

Structural application profile

26

Collabera

Cyclomatic complexity

%Branch Statements

Depth of Inheritance

Calls/method

Methods / Class

Class Coupling

Maintainability Index Boolean Expression

complexity

Class Data Abstraction coupling

Class Fan Out Complexity

Npath Complexity

Java NCSS

% comment Lines Density of comment lines

Public undocumented API

Duplicated lines

Duplicated blocks

Number of children

Afferent couplings

Lack of cohesion methods

Package dependencies to cut

Violations

Complexity distribution by method

Complexity distribution by class

File dependencies to cut

The problem of plenty

27

Collabera

II

Slide 28

% Branches Block Depth

Cyclomatic Complexity

Correlating SIT with engineering metrics

% Branches Block Depth

Cyclomatic Complexity 28

Collabera

Case Study

92%

4%

2% 1% 1%

Number of Class vs SIT defects

8% of classes is contributing to 100% of SIT defects

Study from European university

Key parameters measured

Cyclomatic complexity No. of methods No. of Calls LOC/method

Our performance is above UCL on

LOC/method

29

Collabera

Sample Java Class and Code toxicity analysis

30

Detailed study of the data shows most of the SIT & AT defects are occurring amongst the top four Java classes shown in the sample data above which have highly toxic code with high method length

Correlates to AT & SIT defects

Collabera

2.3.10

2.3.9.2 2.3.6.2

2.3.53.2

2.3.52.2

2.3.5.2 2.3.4.2 2.3.36.2

2.3.35.2

2.3.34.2

2.3.33.2

2.3.32.2 2.3.30.2 2.3.20.2 2.3.19.2

2.3.10.2

2.3.7.2

2.3.50.2

2.3.49.2

2.3.20.2

2.3.13.2 2.3.44.2 2.3.43.2 2.3.42.2 2.3.41.2

2.3.40.2

2.3.39.2

2.3.38.2

2.3.37.2

2.3.11.2

2.3.12.2

2.3.27.2

2.3.2.2

2.3.1.2

2.3.48.2

2.3.47.2

2.3.46.2

2.3.45.2

2.3.26.2

2.3.25.2

2.3.23.2

2.3.22.2

2.3.18.2

2.3.17.2

2.3.16.2

2.3.15.2

UC complexity vs. max code complexity

31

Collabera

32

Collabera

Unit test measures

33

Unit Test coverage Test LOC Vs SLOC Test Pass percentage

Collabera

Figured out - Coding

34

Requirements

Architecture

Design & Coding

Unit Testing

System Testing C

ust

om

er

Cu

sto

me

r

% UC with Alt flows % UC with Open points % Ucs with more than 2 changes % Use cases with screens/Prototype

% coverage of BR to ASRs # of arch styles /patterns Ratio of ASD t o tradeoffs, r isks & sensitivities

?

Requirements Review

Architecture review

Code Review Unit Testing System testing

Effort

Schedule

Quality Control Metrics

Project Management

Engineering Metrics

Cyclomatic complexity Methods / Class Calls Per Method % Branch Statements

Test Coverage Test LOC/SLOC Test cases failed per build

Collabera

JIDOKA-Step By Step

Requirements structure

Test case adequacy Code

structure

Structural application profile

35

Collabera

Variant -1

Application TC TV TI Others

1 Application 1 24 24 9 6

2 Application 2 36 54 27 9

3 Application 3 6 6 6 3

4 Application 4 9 12 0 6

Before

After

No Application TC TV TI Others

1 Application 1 30 45 15 18

2 Application 2 45 75 36 24

3 Application 3 9 9 6 12

4 Application 3 12 15 6 15

TC TV TI Others 1 1.25 1.25 2.3

36

Collabera

Variant – 2

% Navigation test cases

% Presentation test cases

% Behavioral test cases

% UI test cases

% Accessibility test cases

% Security test cases

37

Collabera

Variant - 3

Scenario Level

• % of end to end business scenarios

• % of alternate business scenarios

• % of untestable scenarios

Test case Level

• # of test cases per use case

• % of positive test cases

• % of negative test cases (exceptional cases, abnormalities)

• % of interface test cases

• % of UI test cases

• % of non functional test cases

• % of requirements covered through test cases

38

Collabera

JIDOKAA in software

39

Requirements

Architecture

Design & Coding

Unit Testing

System Testing C

ust

om

er

Cu

sto

me

r

% UC with Alt flows % UC with Open points % Ucs with more than 2 changes % Use cases with screens/Prototype

% coverage of BR to ASRs # of arch styles /patterns Ratio of ASD t o tradeoffs, r isks & sensitivities

% of end toend scenarios % of untestable scenarios % of positive test cases % of negative test cases

Requirements Review

Architecture review

Code Review Unit Testing System testing

Effort

Schedule

Quality Control Metrics

Project Management

Engineering Metrics

Cyclomatic complexity Methods / Class Calls Per Method % Branch Statements

Test Coverage Test LOC/SLOC Test cases failed per build

Collabera

Shifting to the better - I

40

Collabera

Improved Pre-SIT Detection

Pre_SIT Defects %

Dec 2009 Last 1 year

70 %

49 %

100% of development and enhancement were above 2011 pre-SIT target Most development and enhancements had slipped defects ratio less than 2%

III

41

Collabera

Shift in defect distribution

Requirements & Design review

Code Review Unit Test System Test Post Release

Pre Post

42

Collabera

What Did We See….

Overall Satisfaction Quality On time

CSAT performance

Mar-10 Sep-10 Mar-112009 2010 2011

43

Collabera

Qualitative Benefits

These metrics and dashboards are easier to understand

Improve ability to isolate and deal with the defective ones

Improved risk management and transparency

44

Collabera

Thank you

45

Mosesraj R - (mosesrajr@collabera.com)

Smitha T – (smithat@collabera.com)