Reviews and Inspections

Reviews and Inspections

Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 13

Required Readings

• D. Gause and G. Weinberg, “Making Meetings Work for Everybody,” Exploring Requirements: Quality Before Design, Dorset House, 1989.

• M. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, Vol. 15, No. 3, July 1976.

(cont’d)

Required Readings (cont’d)

• B. Grady and T. Van Slack, “Key Lessons in Achieving Widespread Inspection Use,” IEEE Software, July, 1994.

• Sauer, et al., “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research,” IEEE TSE, Vol. 26, No. 1, 2000.

About Meetings

• “Reviews and Inspections” are a form of human-based testing that involves people working together cooperatively.

• Thus, we begin with a few basic axioms (ala Gause and Weinberg) regarding meetings…

D. Gause and G. Weinberg, “Making Meetings Work for Everybody,” Exploring Requirements: Quality Before Design, Dorset House, 1989.

Meetings can be terrible...

• Ever been to a really BAD meeting?

• Meetings as measurements: If your meetings are terrible, then your entire process is probably sick.

• In order for meetings to be effective, they need to be made safe…

Safe to attend...

• Establish a no-interruption policy, but also set time limits for individual speakers so that everyone will be able to participate.

• Outlaw personal attacks and put-downs.

• Finish on time, but schedule a continuation of the meeting if business isn’t finished.

• Use a related issues list and ensure follow-up for important off-topic matters that come up.

And safe NOT to attend...

• To remove uncertainty about what will be covered, publish an agenda and stick to it.

• Handle “emergency issues” in a way that will not hurt people who were not invited.

• Be sure people who should attend are identified and explicitly invited in advance.

• Gently confront those present who should not attend immediately but unobtrusively – preferably before the meeting starts.

Other Meeting Axioms

• Meetings should be as small as possible, but no smaller.

• Keep the agenda short. (A meeting that tries to do too many things does none well.)

• Design meetings to have the appropriate structure and pace.

• Identify someone to act as a facilitator.

• Be prepared! (95% of meetings that fail do so because of inadequate preparation.)

M. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, Vol. 15, No. 3, July 1976.

What are Reviews & Inspections?

• Involve people examining a work-product / document (requirements, user manuals, design, test plan, test cases, source code, etc.) with the aim of discovering anomalies and defects.

• More formal than walkthroughs or desk-checking

• Can be more effective than (machine-based) testing after system implemen-tation. (As demonstrated in many studies.)

Quotes and Notes from Fagan’s Paper

• “An ingredient (of inspections) that gives max-imum play to the planning, measurement, and control elements (of software development) is consistent and vigorous discipline.”

• “...finding errors by inspection and reworking them earlier in the process reduces the overall rework time and increases productivity…”

(cont’d)

Quotes and Notes from Fagan’s Paper (cont’d)

• “…we first describe…places at which inspections are important. Then we discuss factors that affect productivity and the operations involved with inspections. Finally, we compare inspections and walk-throughs…”

Proposed Inspection Types and Levels

• Product

– I0: component level design

– I1: unit level design (“design complete”)

– I2: code inspection

• Test

– IT1: test plans

– IT2: test cases

• Publications

– PI0 - PI2

Update on Inspection Types and Levels

• In addition to those originally proposed by Fagan, IBM now undertakes the following inspections:

– DR1 - DR3 inspections on requirements, product-level design, and system-level design, and

– IT1/2 inspections expanded to unit, component, product, and system testing levels.

Code Inspections – the People Involved

1. Moderator:

– the key person; the coach

– technically competent, but preferably someone working on a different project

– Trained and experienced in facilitation

2. Designer

3. Coder/Implementer (author/owner)

4. Tester

I1 (“Design Complete”) Inspection Process

1. Overview (whole team)

2. Preparation (individual) using…

– ranked distributions of error types

– checklists of clues on finding errors

(cont’d)

I1 (“Design Complete”) Inspection Process (cont’d)

3. Inspection (whole team)

– a “reader” is chosen by the moderator†

– every element of logic and every branch is considered

– objective is to find errors; no specific solution hunting is permitted

– moderator prepares written report within one day

† Which team member should NOT assume this role?

(cont’d)

I1 (“Design Complete”) Inspection Process (cont’d)

4. Rework (owner)

5. Follow-up (moderator)

– if > 5% of material has been reworked, the entire element is re-inspected

Other Considerations/Observations

• “...people have to be taught or prompted to find errors effectively.”

• (inspection results) “...should not under any circumstances be used for programmer performance appraisal.”

• (acceptance of inspections) “...is related to the success of the inspections (people) have experienced, the conduct of the trained moderator, and the attitude demonstrated by management.”

(cont’d)

Other Considerations/Observations (cont’d)• “The most marked effect of inspections on the

development process is to change the old adage that design is not complete until testing is completed…”

• “...in one case...approximately two thirds of all errors reported during development (were) found by...inspections...”

Inspections versus Walkthroughs: “Comparison of Key Properties”

Properties Inspection Walk-through

Formal moderator training Yes No

Definite participant roles Yes No

Who “drives” the process Moderator Owner

Use checklists? Yes No

Formal follow-up Yes No

Analysis process problems improvements

Yes No









Yes No









Yes No









Yes No









Yes No









Yes No









Yes No

Inspecting Modified Code

• “Since most modifications are small...they are often erroneously regarded as trivially simple and handled accordingly; ...In the author’s opinion, all modifications are well worth inspecting...”

• “Human tendency is to consider the ‘fix,’ or correction, to a problem to be error-free itself. ...The number of bad fixes can be...reduced by some simple inspection after clean compilation of the fix.”

B. Grady and T. Van Slack, “Key Lessons in Achieving Widespread Inspection Use,” IEEE Software, July, 1994.

Notes from the Grady and Van Slack Paper

• On the value of Inspections:

– ROI: some can achieve 100:1; based on HP data, you can expect 10:1.

– “We estimate that roughly 1/3 of all HP software costs are rework, and inspections can save 60% of these costs.”

– Lessons at HP suggest inspections are appropriate for ALL software development organizations.

Influences on Inspections at HP

• History of hardware reviews (first software reviews were walk-throughs)

• Fagan’s “very influential” paper which introduced the term “software inspection.”

• Tom Gilb’s 1988 SE Manamement book

– focus on EARLY life-cycle artifacts– measures of the inspection process itself– use of these measures for process

improvement

HP’s Inspection Process

• Focus on software “documents” as opposed to code

• Goal: to ensure that documents are clear, self-explanatory, correct, and consistent with all related documents

• Five roles: inspectors, scribe, owner, moderator, chief moderator (process owner and champion)

HP’s Inspection Process (cont’d)

• Seven steps:

1. planning (moderator and owner plan inspection and create packet)

2. kickoff (brief team meeting)

3. preparation (individuals find issues)

4. logging meeting (team meets to find and log issues)

(cont’d)


• Seven steps:

1. planning (moderator and owner plan inspection and create packet)

2. kickoff (brief team meeting)

3. preparation (individuals find issues)*

4. logging meeting (team meets to find and log issues)*

* cf. Fagan’s description of these steps.

(cont’d)


• Seven steps: (cont’d)

5. cause/prevention (brainstorm causes and recommendations)

6. rework (verify and fix defects)

7. follow-up (moderator)


• Seven steps: (cont’d)

5. cause/prevention (brainstorm causes and recommendations)*

6. rework (verify and fix defects)

7. follow-up (moderator)

* “The process step that varies the most…” Why?

Technology Transfer Lessons

• Importance of communicating successes

• Clearly defining who is responsible for process improvement

• A high-level, compelling vision (shape-up or else...)

• Readily available training (including MANAGEMENT training)

(cont’d)

Technology Transfer Lessons (cont’d)

• Consulting to create an environment for success BEFORE training is begun

– adapt the objectives/process as needed

– find a chief moderator/champion

– benchmark the current process

– follow-up to ensure success

Sauer, et al., “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research,” IEEE TSE, Vol. 26, No. 1, 2000.

Questions Posed by Sauer, et al., Regarding Technical Reviews1. What makes reviews effective at defect

detection?

2. How can we improve review performance within the current review design?

3. What design alternatives would theory predict to be more effective?

Approach: applying behavioral theory of group performance

Controversy Regarding the Inspection / Logging Meeting

“…20 years after Fagan’s seminal paper, debate continues over whether a key component of inspections, the group meeting, is necessary for defect detection because it is not clear whether, in the context of a process in which individual preparation focuses on defect detection, significant numbers of defects are discovered through reviewers interacting (synergy).”

Results from the Behavioral Theory of Group Performance Suggest:• Increasing inspectors’ defect detection

expertise (e.g., through appropriate selection an/or training) should have the largest impact on performance.

• The interacting group meeting does not improve performance in discovering new defects.

(cont’d)

Results from the Behavioral Theory of Group Performance Suggest: (cont’d)• The performance advantage of an interacting

group is a function of the level of false positives discovered by individuals.

• An expert pair performs the discrimination task (identifying false positives) as well as any larger group.

• Above a critical limit, performance declines with group size.

Implications for Practice

• Under current review designs, performance may be improved by:

– selecting better reviewers,

– providing (better) training,

– fine-tuning the size of reviews, and

– providing aids to expertise (e.g., error checklists).

(cont’d)

Implications for Practice (cont’d)

• Using alternative designs, performance may be improved by separating the tasks of defect collection and defect discrimination.

• Managing Reviews: past view that review performance should be collective, not individualistic, should change. Individuals’ review performance should be assessed in staff appraisals.


Summary

Reviews and Inspections Summary

• Disciplined, human-based testing (e.g., inspections) can be very effective.

• Human-based testing is applicable to code, design, requirements, testware, publications, etc.

• The exclusive objective of inspections should be defect detection.

(cont’d)

Summary (cont’d)

• Inspection results are useful for process tracking and improvement.

• Performance may be improved by increasing expertise and fine-tuning group size.

• Separating the tasks of defect collection and defect discrimination may be a useful design alternative for reviews.

(cont’d)

Summary (cont’d)

• Inspection results should never be used in programmer performance appraisals...

• Individuals’ review performance should be assessed in staff appraisals...

(Can you reconcile these two statements?)


Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 13

Documents

Reviews and Inspections