By Tom Gilchrist, CSQA, CSQE, Software Process Reviews/Audits Process Overview

by

Tom Gilchrist, CSQA, CSQE,

Software Process Reviews/AuditsSoftware Process Reviews/Audits

Process OverviewProcess Overview

SASQAG 10/17/2002 [email protected] 2

Before we start…

• SQA Context

• Overview of SW Audit Process

• SW Audit Examples

Information in this presentation are my opinions and not necessary those of my employer.


Some Terms/Ideas

• Process

• Deterministic vs. Non Deterministic

• Quality vs. Value


Software Quality Assurance

• Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Reviews or Audits)

• Review and measure the quality of software products and processes throughout development. (Dynamic & Static Testing)

• Provide software project management (and other appropriate parties) with the results of reviews and process checks.

• Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.


Formal Definition

Audits provide an independent evaluation of software products or processes to ascertain compliance to standards, specifications, and procedures based on objective criteria that included documents that specify:

– The form or content of the product to be produced

– The process by which the products shall be produced

– How compliance to standards or guidelines shall be measured.

IEEE STD 1028, (1988)


Audit Types

• First Party AuditFirst Party Audit– Within you company or organization

• Second Party AuditSecond Party Audit– Sometimes called “external audits”– By a Customer on his Supplier– By a Supplier on you.

• Third Party AuditThird Party Audit– Outside third party is contracted to do

the audit.


Audit/Process Review Principles

• Conducted by individuals who are organizationally independent of the developers.

• Begin early in the requirements phase and continue throughout the development process.

• Professionally planned, conducted and documented.

• Follow-up on corrective action.• Project Management is involved in the Audit

process and is responsible for rework and process improvements.


What Software Audit Should Do

• Determine:• Compliance to requirements• Conformance to plans, policies, procedures, and

standards• Drive process improvement based on:

• Adequacy of plans, policies, procedures, and standards

• Effectiveness and efficiency of plans, policies, procedures, and standards

• Assess personnel familiarity to requirements and documentation

• Assure availability, use and adherence to software standards


What Triggers an Audit?

• Quality Assurance Plan• Event• Date

• Requests from management• Requests from developers• Requests from customers• Integration with process improvement activities• Outside requirements — regulatory• Gut feel


Scope: Requirements, Time, and Target

Audit

Target

External

Standards

Organizational

Procedures and

Methods

• Spread around organization

• Cover all functions and activities

• Try to hit things early

• Move towards process audits


Process Review/Audit Process

OK

PrepareAudit

Developers Project ManagerAuditor

ConductAudit

Write-upReport &Findings

Follow-upAudit

Re-Work

Findings?

NO

YES

CloseoutAudit &

File END

Reviewwith

Manager

Plan(Requirements,

Scope, & Checklist)Start

CorrectiveActions


Identify Requirements

• Policies/Standards Corporate, Group, IEEE• Processes/Plans SCMP, SQAP, SDP, Project Plan• Procedures Change Management, Design

Reviews, Document Standards,

Testing • Task Instructions Library updates, unit testing, peer

reviews

• Success of an audit is directly proportional to preparation, research and analysis conducted before the audit is performed.


Requirement Types

• Functional (ascertainably true or false)• Quality (range of acceptable values)


Types of Audits (Internal)

• Quality System Audits

• Product Audit

• Process Audit

• Project Audit

• CM Audit


Evidence Collection

• Collect Factual Information• Analyze and Evaluate the Evidence• Draw Conclusions• Generate Findings


Corrective Action of Findings

• Determine Action– Immediate Remedial Action– Process Improvement/Fix– Acceptable Risk

• Identify Root Cause• Corrective Actions Plan • Manage CA Plan to completion• Analyze Effects of CA


Develop Audit Checklist

• Focus on clear requirements (or unclear to fix)

• Select subset of requirements• Focus on important steps/products• Write clear concise questions• Canned checklist vs. straw horse


Checklist Sample

Requirement Checklist Item Details Observations Results (P/F)

Company Standard ABC-234, page 7

Does project QA plan will have a list of deliverables subject to Peer Reviews?

Check SQA document for a list of approved peer reviews and which documents are to be reviewed. (if no documents are found, then fail. If no peer review procedures are referenced, then fail)

Project SQA Plan

Were the number of audits completed equal to the number planned?

Check to see which audits were planned for the last 60 days. Check for evidence that the audit was completed and if there were findings, that a CA plan was signed.

Project SQA Plan

Were the number of peer reviews completed equal to the number planned?

For each peer review type, check the CM records for the past 60 days to see if the document type specified in the QA plan was checked into CM for the first time. If so, check for records of the peer review being completed as per peer review process cited in SQA plan.


Interviewing

• Ask open-ended questions• Know the types of answers expected• Focus on Process and not People• Seek Corroboration and Evidence


Sample Interview Questions

• How do you track your progress?• Do you have a CM Plan?• Tracing

– What are you working on?– Is it a configured item?– Do you have an approved CR or PR?– Is the version you are working on

checked out of CM?


Desirable Auditor Characteristics

• EmotionalEmotional• Interviews• Group

dynamics• Oral reports• Empathy• Don’t take

things personally

• MechanicalMechanical• Sampling• Root Cause

Analysis• IntellectualIntellectual

• Writing• Planning• Speaking• Detail

Oriented• Concise


Desirable Auditor Characteristics(Cont.)

• Knowledge of Audit process• Knowledge of target (SW) processes• Knowledge of techniques • Professional attitude• Good listener• Inquisitive/analytical• Communicates at all levels• Detailed Notes and Observations• Diplomatic

Documents

By Tom Gilchrist, CSQA, CSQE, Software Process Reviews/Audits Process Overview