Upload
jarrett-rasbury
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
1
Software Quality Assurance
• Industry statistics for 1994 (WW. Gibb):– For every 6 systems that work, 2 are cancelled– It is worse for large systems: 3 out of every 4 large systems were
reduced in scope or did not work at all (e.g. Denver airport)– Still bad today for low maturity software development
• Good quality software doesn’t just happen, however good the team producing the software and however good the development techniques they use.
• Good software development needs procedures and management.
That is what QA is all about
2
Why are procedures and management necessary for quality?
• Software development is complex: much to remember– Hence standards, procedures and guidelines which embody best practice
– These also help training of new staff
• When the pressure’s on… something has to give...– Hence defined points at which quality of work so far is assured
– Hence being explicit with the customer about cost-time-scope-quality trade-off: ‘if you want software of this quality...’
• Some defects may go unnoticed by the author– Hence reviews… which wouldn’t happen without management!
• Helps assure the customer that the software is of sufficient quality
3
This lecture
• What is quality?– Quality factors
• What is quality assurance?– Quality manuals and plans
• Standards for quality manuals: the ISO 9000 family
• Group Project Standards
• Reviews– Kinds of review: particularly the Formal Technical Review
– How to do a formal review
4
What is quality?
• ISO definition (ISO 8402:1994)– “The totality of features and characteristics of a product, process or
service that bears on its ability to satisfy stated and implied needs.”
• Or…– Software quality is meeting the requirements of the user
• With the caveats that…– The user requirements may not be precisely stated
– Some user requirements may not be stated at all, because of ‘shared assumptions’ between user and developer
– The developer may also have requirements which may not be part of the user requirements (e.g. testability, maintainability)
5
Quality factors (ISO 9126:1992)
• Functionality– Satisfying stated or implied needs (regarding the software’s functions)
• Reliability– Maintaining performance under stated conditions for stated period of time
• Usability– The effort needed for use, and the individual assessment of such use by a stated or
implied set of users
• Efficiency– The relationship between performance and resources required
• Maintainability– The effort needed to make specified modifications
• Portability– The ability of software to be transferred from one environment to another
6
What is Quality Assurance?
• Quality Assurance is the overall management of quality. • Within a company, it includes:
– Specifying a quality policy: General quality objectives and commitments
– Maintaining a quality manual, which contains details regarding:• What constitutes good quality: Standards, procedures, guidelines• Quality controls: methods of assuring that quality is good
– Producing quality plans for each project• The parts of the quality manual which apply to this project
– Assigning specific people to quality assurance, often including a project-independent quality assurance group
– Planning (schedule/budget), performing, evaluating quality assurance activities
7
Recognised standards ofQuality Assurance
General Quality Standards
QualityManual
Project 1Quality Plan
Project nQuality Plan
Project 2Quality Plan
Increases customer’s confidence that acompany produces quality products
e.g. ISO 9000
8
The ISO 9000 family
• ISO 9000: Guidelines for selection and use of the other documents in the family.
• ISO 9001: Model for QA in design, development, production, installation and servicing.
• ISO 9002: Model for QA in production, installation and servicing.
• ISO 9003: Model for QA in final inspection and test.
• ISO 9004: Guidelines on quality management and quality system elements.
• Also ISO 9000-3: Application of ISO 9001 to the development, supply and maintenance of software. Direct correspondence between clauses of ISO 9001 and ISO 9000-3.
9
Some example sections from ISO 9000-3
• Management Responsibility.
• Quality System.
• Internal quality system audits.
• Corrective action.
• Contract review.
• Purchaser’s requirements spec.
• Development planning.
• Quality planning.
• Design and implementation.
• Testing & Validation.
• Acceptance.
• Maintenance.
• Configuration Management.
• Document Control.
• Measurement.
• Rules, practices, conventions.
• Training.
• Purchasing.
ISO 9001/9000-3 are short (10-15 pages): open to interpretation
10
ISO 9001 versus ISO 9000-3:‘quality planning’ section
ISO 9001
… Quality planning shall be consistent with all other requirements of a supplier’s quality system and shall be documented in a format to suit the supplier's method of operation. The supplier shall give consideration to...
ISO 9000-3
Quality planning should address…
• quality requirements, expressed in measurable terms, where appropriate;
• the life cycle model to be used for software development
• defined criteria for starting and ending each project phase
• identification of types of reviews, tests and other verification and validation activities to be carried out
• identification of configuration management procs to be carried out
11
ISO 9000:A standard for quality manuals
• Companies can, and do, get certification that:– Their quality manual meets the ISO 9000 standards
– They run their projects with adherence to their quality manual
• A company is ISO certified by an independent body– The company first receives training in ISO 9000 etc.
– The quality manual is sent to the body, which assesses it and reports back
– An assessment team visits the company to ensure that it follows its manual
– The body then awards the company ISO 9000 status unconditionally, conditionally, or not at all
– After certification, an assessment team visits unannounced 2-4 times per year - and conducts a full reassessment every 3 years
12
Software Engineering Institute’s Capability-Maturity Model (SEI CMM)
Level 1Initial
Unstable, ad-hoc processes;project success depends on staff skills/experience (‘heroics’)
Level 2Repeatable
Stable project planning, tracking of cost/schedule/functionality, problem identification, project standards;can ‘repeat the success of previous projects’
Level 3Defined
Stable process model includingentry/exit criteria; reviews, qualitytracking, full process training program
Level 4Managed
Management by measurement: quantitativetargets for each project; productivity andquality measures; measurementdatabase
Level 5Optimizing
Organisational commitmentto continuous processimprovement; defect prevention
13
Comments on the CMM
• CMM versus ISO 9000– CMM much more prescriptive (500 pages versus 10-15 pages)– Companies rated ISO 9000 are probably partly level 2, partly level 3;
but they may only be at level 1 (because ISO open to interpretation)– CMM is improvement-oriented, ISO 9000 is ‘minimum quality’
• Disadvantages of the CMM– Nothing about technology or risk analysis– Originally to help DoD select contractors: large, slow-moving systems– Key areas a bit arbitrary: 80% level 2 + 70% level 3 = level 1!– Too bureaucratic for small companies?
• At least it collects together some best practices
14
A project’s quality plan
• Should be defined at an early stage of the project • Can use quality factors to determine which parts of the quality
manual are particularly appropriate to this project– Is portability important? – How long will this product live?
• Define activities to be performed. Answer questions for each activity:– What is checked and against what?– Who performs the check and who approves the checks? – Who takes the final decision if there is controversy?– How are results recorded?– How are corrective actions controlled?
15
Quality controls: The general process
1 Define things which are considered ‘good quality’.
2 Define the procedure for checking quality.
3 Carry out the procedure on a specified item.
4 Record the result.
5 Take and record any corrective action – Regarding both the item and the quality system
16
Group Project Standards
• A cut down version of what you might see in industry
• They cover the following areas:– QA.01 Quality plan - see previous slide on ISO-9001 Quality planning
– QA.02 Project management - how we will run the project
– QA.03 Documentation - what they will cover/look like
– QA.05 Design Spec standards
– QA.06. Test Spec standards
– QA.07 Review standards - will cover now...
– QA.08 Configuration management
– QA.09/10 Coding standards for Java/C#
– QA.11 Final report / maintenance documentation
17
An important quality control:the review
• Bugs are caught more efficiently by review than by testing
• A review is…– A meeting at which some aspect(s) of the project are inspected and
assessed by a group of people
• All reviews have an element of formality, and their outcome is minuted
• Formal Technical Review– To identify defects in a workproduct (design document, code listing, …)
• Walkthrough– To identify defects in a workproduct, but with the emphasis on learning
and evaluating alternatives - less formal
18
Formal Technical Review: process
• Author submits document or code to the review convenor– Briefly examined before accepting it for review
• Review preparation– Review team formed
– Criteria formed by which the document/code will be examined
– All necessary documents sent to reviewers, who read them
• The review meeting (2 hours maximum)– Issues (possible defects) are detected, but not corrected
• Follow-up: document changed in response to issues highlighted in review meeting, and resubmitted if necessary
19
Documents needed for a review
• The item under review!
• The document(s) from which the item is derived– For test spec: requirements, user interface design, design?
– For code: detailed design (or for us, list of classes/attributes/methods)
• The relevant quality plan documents
• A review checklist: questions to answer– Summarises what to look for when reviewing
– Derive from the quality plan documents and (to an extent) from the documents from which the item is derived
– Try to fit on one sheet of paper, to concentrate on major issues
20
Example review checklist: design
• Do the reviewers understand the design?
• Will the design meet the requirements?
• Does the design satisfy the relevant standards?
• Can the design be simplified?
• Will the design lead to a maintainable system?
• Could the design be modified so as to offer more opportunity for reusing existing software?
The more specific the quality plan, the easier to create checklistsSE.QA.07 has a more specific design checklist for group project.
21
Example review checklist: code
• Are all variables initialised before their values are used?
• Have all constants been named?
• Is each loop certain to terminate?
• Are upper and lower bounds of arrays sensible?
• Have pointers been correctly reassigned after modification?
• Does code conform to QA code standards for the language?
22
The review meeting
• The item being reviewed is ‘walked through’ from start to end
• Issues are highlighted as they occur– Includes issues found before, as well as during, the meeting
• The author evaluates each issue, and…– Accepts as a defect, or explains why not a defect, or answers a
question which a reviewer has
• Minutes are taken by the scribe (QA Manager for us!)
23
Conclusions
• QA is the management of doing things in such a way to ensure that end results meet the users requirements.
• Steps must be appropriate & economically realistic.
• Aim is not highest possible quality, but is a compromise between– Time
– Cost
– Quality
– Scope