1

CHAPTER 8: SOFTWARE QUALITY ASSURANCE AND CONFIGURATION MANAGEMENT

8.1 Quality Assurance Over-view

What is quality assurance- is the forecasting and prevention of quality related problems.

o Every effort is taken to meet the customer quality requirements. Examples of quality assurance tools include: Policies and procedures Supervision Periodic checking and rechecking Reviews Checklists Walk-through

 Examples of quality control tools include: Inspection tools Measurement gauges Acceptance tests

Software Quality Assurance

Software quality can be defined as:

Conformance to explicitly-stated functional and performance requirements, explicitly-documented development standards, and implicit characteristics that are expected of all professionally developed software.

This definition thus emphasizes on three important points:

Quality will be measured with Software Requirements, i.e. lack of conformance to these requirements will mean lack of quality.

Specified standards define a set of development criteria that guide the manner in which software is engineered. If the criteria is not followed, lack of quality will result.

The set of implicit requirements, e.g. good maintainability, must still be followed. If these are not met, software quality is suspect.

Software quality assurance(SQA)

- is a planned systematic pattern for all actions necessary to provide adequate confidence that the product, or process by which the product is developed, conforms to established requirements.

- The primary goal of software quality assurance is to enhance the quality of software. To this end, a new concept, Clean Room engineering, is evolving where by a software is developed to assure quality.

Software quality is thus a mix of factors that will vary across different applications and different customers. The sections below will thus be concerned about:

2

Identification of Software Quality Factors

Human activities required to achieve them

Quality management activities1 Quality assurance

Establish organisational procedures and standards for quality. is concerned with the forecasting and prevention of quality problems. It is pro-active.

2 Quality planning Select applicable procedures and standards for a particular project and modify these as

required.3 Quality control

Ensure that procedures and standards are followed by the software development team. is concerned with detecting and reporting quality-related problems. It is reactive.

Quality management should be separate from project management to ensure independence.

8.2 Software Quality Factors

In order to help us categorize software quality factors, McCall proposes a categorisation which focuses on three important aspects of a software product:

Product Operations : A software product's operational characteristics;

Product Reunion: Its ability to undergo change,

Produced Transition : Its adaptability to new environments.

8.2.1 Product Operations

Correctness

Does it do what I want?

- The extent to which a program satisfies its specification and fulfils the mission objectives.

Reliability

Does it do it accurately all the time?

- The extent to which a program can be expected to perform its intended function with required precision.

3

Efficiency

Will it run on my hardware as well as it can?

- The amount of computing resources and code required by a program to perform a function.

Integrity

Is it secure?

- The extent to which access to software or data by unauthorised persons can be controlled.

Usability

Is it designed for the user?

- The effort required to learn, operate, prepare input, and interpret output of a program.

8.2.2 Product Revision

Maintainability

Can I fix it?

- The effort required to locate and fix an error in a program.

Flexibility

Can I change it?

- The effort required to modify an operational program.

Testability

Can I test it?

- The effort required to test a program to ensure that it performs its intended function.

8.2.3 Product Transition

Portability

Will I be able to use it on another machine?

- The effort to transfer the program from one hardware and/or software system environment to another.

Reusability

Will I be able to reuse some of the software?

- The extent to which a program (or parts of a program) can be reused in other applications-related to the packaging and scope of the functions that the program performs.

Interoperability

Will I be able to interface it with another system?

- The effort required to couple one system to another.

4

8.3 Software Quality Assurance Major Activities

Software Quality Assurance (SQA) is a planned and systematic pattern of actionsª that are required to ensure quality in software.

The SQA group in any company serves as the customer's in-house representative. That is, the people who perform SQA must look at the software from the customer's point of view.

Does the software adequately meet the quality factors from the customer's point of view?

Does the software adequately meet the quality factors described earlier?

Has software development been conducted according to pre-established standards?

Have technical disciplines properly performed their roles as part of the SQA activity?

8.3.1 SQA Activities

SQA is comprised of a variety of tasks associated with seven major activities:

Application of Technical Methods

- SQA begins with a set of technical methods and tools that help the analyst to achieve a high-quality specification and the designer to develop a high-quality design.

Conduct of Formal Technical Review

- Activity that accomplishes quality assessment for the specification and the design. The FTR is a stylised meeting conducted by technical staff with the sole purpose of uncovering quality problems.

Testing of Software

- Combines a multistep strategy with a series of test case design methods that help ensure effective error detection.

Enforcement of standards

- Degree in which this is applied varies from company to company. In many cases, standards are dictated by customers or regulatory mandates. In other situations, standards are self-imposed.

Control of Change

- Every change to software has the potential of introducing errors or creating side effects that propagate errors. The change control process thus contributes directly to software quality by formalising requests for change, evaluating the nature of change, and controlling the impact of change.

Measurement

- Track software quality and assess the impact of methodological and procedural changes on improved software quality.

Record keeping and recording

- Collection and dissemination of SQA information. The results of reviews, audits, change control, testing, and other SQA activities must become part of a historical record for a project and should be disseminated to the development staff on a need-to-know basis.

5

8.4 Formal Technical Reviews

Formal technical review is

A class of reviews that include walkthroughs, inspections, round-robin reviews, and other small group technical assessments of software

A planned and controlled meeting attended by a group of diversified people

8.4.1 Objectives of FTR

To uncover errors in function, logic, or implementation for any representation of the software

To verify that the software under review meets its requirements

To ensure that the software has been represented according to predefined standards

To achieve software that is developed in a uniform manner

To make projects more manageable

8.4.2 Effects of FTR

Early discovery of software defects so the development and maintenance phase is substantially reduced

Serves as a training ground, enabling junior engineers to observe different approaches to software analysis, design, and implementation

Serves to promote backup and continuity because a number of people become familiar with parts of the software that they may not have otherwise seen

8.4.3 Guidelines for the Organisation and Preparation of FTR

Should involve between three and five people

Advance preparation should occur but should not require no more than 2 hours of work for each person

The duration of the review meeting should be less than 2 hours

Focus on a components of the software (eg. a portion of requirements specification, a detailed module design, a source code listing for a module)

Producer should report his/her progress

A recorder should actively record all issues as

What was reviewed?

Who reviewed it?

What were the findings and conclusions?

The attenders must make a decision at the end of the session as to

Accept the product

Reject the product

Accept the product provisionally, subject to further modification

6

8.4.4 Review Guidelines during the FTR

Review the product, not the producer

Tone of the meeting should be loose and constructive, and intent should not be to embarrass or belittle.

Set an agenda and maintain it

One main problem in all types of meetings is the tendency to drift. The FTR must be kept on track and on schedule.

Limit debate and rebuttal

There may not be universal agreement on some issues. Rather than spending time debating the issue, the issue should be recorded for further discussion off-line.

Enunciate problem areas, but don't attempt to solve every problem noted

A review is not a problem-solving session. The solution of a problem can often be accomplished by the producer alone. Problem solving should be postponed until after the review meeting.

Take written notes

Makes notes on a wall board so that wording and prioritisation can be assessed by the other reviewers as information is recorded.

Limit the number of participants and insist upon advance preparation

Keep the number of people involved to the necessary minimum.

Develop a checklist for each product that is likely to be reviewed

A checklist helps the review leader to structure the FTR meeting and helps each reviewer to focus on important issues. Checklists should be developed for analysis, design, code and even test documents.

Allocate resources and time schedule for FTRs

For reviews to be effective, they should be scheduled as tasks during the software engineering process.

Conduct meaningful training for all reviewers

To be effective, all review participants should receive some formal training. The training should stress both process-related issues and the human psychological side of reviews.

Review your early reviews

Debriefing can be beneficial in uncovering problems with the review process itself.

8.5 Software Reliability

Most hardware-related reliability models are predicated on failure due to wear rather than failure due to design defects. In hardware, failures due to physical wear (e.g. the effects of temperature, corrosion, shock) are more likely than a design-related failure. The opposite is true for software: in fact, all software failures can be traced to design or implementation problems; wear does not enter into the picture.

7

Software Reliability: The probability of failure free operation of a computer program in a specified environment for a specified time.

A simple measure of reliability is mean time between failure (MTBF), where

MTBF = MTTF + MTTR

In addition to this reliability measure, a measure of availability can be calculated.

Software availability is the probability that a program is operating according to requirements at a given point in time and is defined by:

Availability =

8.6 Software Quality Assurance Approach

At the low end of the scale, quality is the sole responsibility of the individual who may engineer, review, and test at any comfort level. At the high end of the scale, an SQA group is charged with the responsibility standards and procedures for achieving software quality and ensuring that each is followed.

Before formal quality assurance procedures are instituted, a software development organisation should adopt software engineering procedures, methods, and tools. This methodology, when combined with an effective paradigm for software development, can do much to improve the quality of all the software produced by the organisation.

Manager/managers and practitioners are not interested in establishing formal SQA functions as:

Managers are reluctant to incur the extra up-front cost

Practitioners feel they are already doing everything that needs to be done

No one knows where to put such a function organisationally

Everyone wants to avoid the "red tape" that SQA is preceived to introduce

8.6.1 Benefits of SQA Software will have fewer latent defects, resulting in reduced effort and time spent

during testing and maintenance Higher reliability will result in greater customer satisfaction Maintenance costs (a substantial percentage of all software costs can be reduced) Overall life cycle cost of software is reduced

8.6.2 Constraints of SQA Diffcult to institute in small organisations, where available resources to perform the

neccessary activities are not available SQA represents cultural change - and change is never easy SQA required the expenditure of dollars that would not otherwise be explicitly

budgeted to software engineering or quality assurance

MTTFMTTF + MTTR

* 100%

8

Configuration management

Configuration management (CM) is concerned with managing evolving software systems: system change is a team activity; CM aims to control the costs and effort involved in making changes to a system.

Somehow, all modifications to the software system (including all documentation, data, and training material) must be identified, organized, and controlled.

Involves the development and application of procedures and standards to manage an evolving software product.

May be seen as part of a more general quality management process.

Principles of Software Configuration ManagementSoftware configuration management has a number of component parts:

1. Configuration Identification – Knowing what items are under configuration control and how to identify them;

2. Configuration Control – How to handle Change Request, emergencies, procedures describing how to build releases and control variants;

3. Status Accounting – Tracking the items and managing transitions between one status and another;

4. Configuration Auditing – Proving the above are being done properly.

When released to CM, software systems are sometimes called baselines as they are a starting point for further development (i.e. further change).

baselines

A "baseline" is A specification or product that has been formally reviewed and agreed upon, that thereafter serves as the basis for further development, and that can be changed only through formal change control procedures.

The sequence of events for a change might look something like:

i. Someone identifies a change they feel is needed (e.g. adaptive/corrective/enhancement maintenance) and generates a change request

ii. The request is evaluated by a member of the development group and generates a change report iii. The change report is evaluated by the change control authority (e.g. project admin, or task leader),

and either the requested change is denied (and the initiator informed) or is approved

The rest of this sequence assumes the change request has been approved

a. Individuals are assigned to those objects under configuration control which must be changed

b. The individuals "check out" the objects (so no one else can alter the objects at the same time)

c. The changes are applied d. The changes are reviewed/audited

9

e. The objects are "checked in" f. The changes are tested g. The software is rebuilt if appropriate h. The cumulative set of changes to all the configuration objects is reviewed/audited i. The changes are included in the new version j. The new version is distributed

Some of the items that may fall under formal change control include

The project plan The requirements and specifications document(s) The user manual(s) The design specification document(s) The source code listings The test plan, test cases, and recorded results The operation and installation manual(s) The executable program The database description document(s) The bug reports, maintenance requests, and change orders The standards and procedures documents

Reviews/audits are geared towards the following

Ensuring the requested change has been made, and that only the requested change has been made

Ensuring a formal technical review has assessed the change Ensuring that all the defined standards have been followed Ensuring that the change has been clearly identified in the appropriate documentation and all

appropriate configuration items updated Ensuring that the change information (author, date, etc) has been properly recorded

Standards

CM should always be based on a set of standards which are applied within an organisation. Applied standards will cover both the process followed and the product resulting (i.e.

standards regarding methods, and standards regarding things like document/code formats) CM standards should define how items are identified, how changes are controlled and how

new versions are managed. These may be based on external CM standards (e.g., IEEE standard).

The most common existing cm standards are based on a waterfall model, effective new standards are needed for other techniques such as evolutionary development.

Document standards might specify the sections which must be included in the document, content items that must be addressed in each section, the order of sections, forms which must be included, the document layout (fonts, page formats, footnote and bibliography styles), the support tools used (e.g. specific word processing packages, spell checkers, etc)

Configuration management planning

10

All products of the software process may have to be managed: o specifications, o designs, o programs, o test data, o user manuals.

Thousands of separate documents are generated for a large software system. Starts during the early phases of the project. Must define the documents or document classes which are to be managed. Documents which might be required for future system maintenance should be identified and

specified as managed documents.

Configuration item identification

Large projects typically produce thousands of documents which must be uniquely identified. Some of these documents must be maintained for the lifetime of the software. Document naming scheme should be defined so that related documents have related names. A hierarchical scheme with multi-level names is probably the most flexible approach.

CM database

All CM information should be maintained in a CM database. Should allow queries about configurations to be answered. Who has a particular system

version? What platform is required for a particular version? What versions are affected by a change to component X? How many reported faults in version T?

Derivation history

Record of changes applied to a document or code component. Should record, in outline, the changes made, the rationale for the change, who made the

change and when it was implemented. May be included as a comment in code. If a standard prologue style is used for the derivation

history, tools can process this automatically.

Version and release management

CM should allow a user to specify a variety of possible configurations of the software by selecting from different available versions - each supported by all the appropriate tools and documentation.

Invent identification scheme for system versions. Plan when new system version is to be produced. Ensure that version management procedures and tools are properly applied. Plan and distribute new system releases.

Version/variants/releases

Version: an instance of a system which is functionally distinct in some way from other system instances.

Variant: an instance of a system which is functionally identical but non-functionally distinct from other instances of a system.

11

Release: an instance of a system which is distributed to users outside of the development team.

Attributed version identification

Attributes can be associated with a version with the combination of attributes identifying that version.

Examples of attributes: Date, Creator, Programming Language, Customer, Status, etc. More flexible than an explicit naming scheme for version retrieval, can cause problems with

uniqueness. Needs an associated name for easy reference.

Risk and risk management

Because software projects represent a substantial business investment, and because there is tremendous potential for cost and schedule overruns or failed delivery of requirements, risk management is an extremely important aspect of software engineering projects

We can consider five steps in risk management:

1. Establish a framework for categorizing types of risk and severity of risks 2. Evaluate the risks associated with the project 3. Establish means to avoid the assessed risks, with emphasis on the risks which are most

likely to occur and/or have the most severe impact 4. Establish metrics which can be used to monitor the increasing/decreasing potential for

different risks as the project progresses 5. Establish contingency plans for mitigating risks should they actualize during the project

life cycle

Not all risks can be planned for, but sound risk management strategies can salvage many projects that would be doomed otherwise

1. Frameworks for categorizing risks

We can categorize risks by their severity, e.g.: catastrophic, critical, marginal, or negligible

We can also categorize risks by the aspect of the project they impact, e.g.: system performance, product support, cost/budget, and schedule

A simple table helps ensure everyone has similar ideas about the severity classification of different types of risk, e.g.:

Risk type: Category

Performance Support Cost Schedule

Catastrophic Unable to meet core requirements

Product will be unsupportable

Significant cost overruns/budget

Core delivery date(s) unachievable

12

shortfalls expected

Critical

Product performance impaired, some requirements not met

Slow/difficult maintenance/support expected

Some cost overruns/budget shortfalls expected

Delivery date in danger

Marginal Minor requirements not met

Reasonable support achievable

Costs/budget adequate

Schedule achievable

Negligible All requirements achievable

Easily supportable product

Under-budget Potential for early delivery

2. Evaluation of specific risks

We now try to categorize all forseeable risks. For each risk identified, establish a rough probability, and classify the severity of its impact.

E.g. We estimate there is a 40% chance that upper management will re-assign two of our key personnel because of problems on another project.

This would have a critical impact on both the product and the schedule, due to loss of their expertise.

Some of the common risks, and factors affecting their severity and probability, are outlined below.

1. Organizational risks

a) The strength of organizational committment to the project in terms of budget, staffing/resources, decision-making, has a substantial impact on risk.

What is the likelihood of staff/resource/funding cutbacks or delays, and what is the impact of each?

Factors to consider here include:

the potential impact of the product on company revenue, the visibility of the product to senior management, the regulatory constraints on the project, the costs of late or incomplete delivery, management awareness of these costs and constraints, etc.

b) What is the likelihood of delays/technical deficiencies due to staff instability - i.e.

are the best people available? will they remain available? do they have appropriate training?

2. Customer-related risks

13

a. Lack of support, or actual obstruction, from the customer can also represent a substantial risk to software projects.

In assessing the customer-related risks for a project, the factors to consider include:

i. Does the customer understand the technical aspects of their requests and the software development process?

ii. Is the customer willing to participate in the review and evaluation process and give timely feedback to the software team?

iii. Does the customer have a clear idea of their requirements?

Your confidence in assessing these factors will also differ depending on how close and how frequent your previous contact with the customer has been.

A lack of familiarity means the risk factor is higher (since there is less confidence in original assessments of the nature of the customer)

3. Risks associated with the development process

b. The effectiveness and appropriateness of the development process obviously has a significant impact on the successful and timely completion of the project.

Factors to consider when evaluating process-associated risks include:

i. Is the process clearly specified, in writing, for the entire team, management, and the clients?

ii. Are the team members, management, and the client genuinely willing/able to follow the process

iii. Does the process account for regular reviews, standards, and CM? iv. Is there adequate software tool support for the process?

c. Delays or technical deficiencies can be a result of using new or unusual development processes. The risk assessment should thus consider

i. Do the requirements call for unconventional development methods? ii. Do the customer requirements call for new (to the team and/or

organization) analysis, design, or test methods?

4. Risks associated with the product a. Product size can be estimated in many ways:

i. lines-of-code, ii. number of files,

iii. number of transactions, iv. size of database, v. number of users, etc

What is the likelihood that the end product will be catastrophically larger than the original estimates?

What would be the impact on performance/support/cost/schedule?

Influencing factors to consider include

14

i) the stability of requirements,

ii) the amount of re-use expected,

iii)the level of technical experience on the part of the original estimators, etc.

b. How stable are the product requirements?

I.e. what is the risk of substantial requirements changes forcing substantial product changes and redevelopment?

In part this depends on the nature of the product and its dependence on the external environment.

In part this depends on the clarity, focus, and consistency of the customer's view of the requirements.

c. What are the risks with respect to new/unfamiliar techniques/technology?

Factors to consider include:

i. is the technology new to your organization and/or team members? ii. Does the product interact with new/unproven hardware or software?

iii. Is a specialized/unconventional user interface required? iv. How restrictive are the performance constraints?

3. Risk avoidance

Once risks have been identified, the team should evaluate which risks most need attention - based on probability of occurence and severity of impact

The team then develops strategies to help avoid the specific risks targetted - this will be very risk-specific.

Example:

If staff turnover leading to delayed production is identified as a key risk, possible avoidance strategies include:

o Reducing turnover through efforts to improve staff morale incentive plans, flexible hours, improved working conditions, listening to staff and acting on feedback

o Taking measures to reduce lost time when/if turnover does occur: developing training plans to bring new personnel up to speed quickly, ensuring that for each key area at least two people have a working understanding

of the material ensuring that documentation is clear, thorough, and up-to-date

15

4. Risk monitoring

Once a risk has been identified, we can begin monitoring relevant information to give us ample warning if the risk shows signs of materializing

Example:

Considering the staff turnover risk again, relevant data to monitor might include: o Monitoring the availability of jobs in other areas of the company or outside the company

(i.e. what is the potential for turnover) o Monitor the typical conditions/salaries of competing jobs o Monitor the general attitude of team members, particularly in response to project

pressures and personal interactions

5. Contingency planning

Assuming efforts to avoid the risk have failed and the risk has become a reality, contingency planning is needed to counteract the effects of the risk

Example:

Continuing the staff turnover example, assume key people have left the project (and that we've followed our "avoidance" policies above)

o since at least two people have been associated with each critical area, there is a back-up person to move into the vacated area

o since there is adequate training material and documentation, a new person can be brought into the project and brought up to speed in reasonable time

o assuming the departing personnel have given some advance notice, they spend their final time documenting key information their replacement must learn, and/or training that person directly