A Proposal for a Quality Improvement Framework for Small Organizations

Johan Redestig, pt95jre@student.hk-r.seErik Starck, pt95est@student.hk-r.se

University of Karlskrona/RonnebyDepartment of Software Engineering and Computer Science

Ronneby, SwedenMaster thesis in software engineering

2008-09-06Supervisor: Conny Johansson

AbstractThe goal of this thesis is to present a quality framework for small organizations that are developing software. This framework includes a quality manual that could be used as first steps for small software developing organizations that wish to work in a process oriented manner and thereby improve the quality in the organization.

To be able to do this, an empirical study is performed, in which we tries to find the areas in which small organizations normally fail. The quality framework focuses on these areas, thereby making the quality framework as compact as possible. This makes it suitable as an introduction to a process oriented way of working for small software developing organizations.

2

AcknowledgementsThe authors would like to thank all that have participated in the development of this thesis in one way or another. Especially our supervisor Conny Johansson for his excellent remarks and support. We would also like to thank the people that have participated in the interviews for their time and effort.

Johan Redestig

Erik Starck

3

Table of contents1INTRODUCTION........................................................................................................................................6

1.1ADDRESSED PROBLEMS..................................................................................................................................61.2GOALS........................................................................................................................................................7

2STRUCTURE AND ORGANIZATION OF THIS PAPER......................................................................8

3QUALITY......................................................................................................................................................9

3.1DEFINING QUALITY FOR SMALL ORGANIZATIONS..............................................................................................93.2EXISTING QUALITY MODELS AND SMALL ORGANIZATIONS...............................................................................143.3COMPANY QUALITY MANUAL, CQM...........................................................................................................173.4CONCLUSION.............................................................................................................................................17

4METHOD....................................................................................................................................................19

4.1QUALITATIVE STUDY...................................................................................................................................194.2RELIABILITY..............................................................................................................................................214.3ANALYSIS OF DATA.....................................................................................................................................224.4CONCLUSION.............................................................................................................................................22

5PROBLEMS AND HYPOTHESES..........................................................................................................24

5.1VIEW OF SOFTWARE DEVELOPMENT................................................................................................................245.2PLANNING.................................................................................................................................................255.3MANAGEMENT...........................................................................................................................................265.4SELECTING TECHNICAL SOLUTIONS.................................................................................................................275.5CONFIGURATION MANAGEMENT AND VERSION CONTROL....................................................................................275.6TESTING...................................................................................................................................................285.7REQUIREMENTS..........................................................................................................................................295.8CONCLUSION.............................................................................................................................................30

6INTERVIEW ANALYSES.........................................................................................................................31

6.1OVERVIEW................................................................................................................................................316.2COMPANY I...............................................................................................................................................326.3COMPANY II..............................................................................................................................................366.4COMPANY III............................................................................................................................................416.5COMPANY IV............................................................................................................................................466.6COMPANY V..............................................................................................................................................50

7ANALYSIS CONCLUSION......................................................................................................................54

7.1THE HYPOTHESIS........................................................................................................................................547.2CONCLUSION.............................................................................................................................................57

8QUALITY FRAMEWORK FOR SMALL ORGANIZATIONS...........................................................59

8.1QUALITY DEFINITION..................................................................................................................................598.2QUALITY MODEL.......................................................................................................................................618.3QUALITY MANUAL ....................................................................................................................................628.4CONCLUSION.............................................................................................................................................67

9FUTURE WORK........................................................................................................................................68

9.1THE CQM...............................................................................................................................................689.2COMPARISON TO A BIG COMPANY..................................................................................................................689.3THE NEXT STEP..........................................................................................................................................68

10REFERENCES.........................................................................................................................................69

11DEFINITIONS..........................................................................................................................................72

4

11.1SMALL ORGANIZATION...............................................................................................................................7211.2QUALITY.................................................................................................................................................7211.3ISO 9000..............................................................................................................................................72

12APPENDIX A - QUESTIONS.................................................................................................................73

12.1INITIAL...................................................................................................................................................7312.2VIEW OF SOFTWARE..................................................................................................................................7312.3PLANNING...............................................................................................................................................7412.4MANAGEMENT.........................................................................................................................................7412.5SELECTING TECHNICAL SOLUTIONS...............................................................................................................7412.6CONFIGURATION MANAGEMENT...................................................................................................................7512.7TESTING ................................................................................................................................................7512.8REQUIREMENTS........................................................................................................................................75

13APPENDIX B – OVERVIEW OVER THE ANSWERS TO THE QUESTIONS..............................76

13.1QUALITY.................................................................................................................................................7613.2PLANNING ..............................................................................................................................................7613.3SELECTING TECHNICAL SOLUTIONS ..............................................................................................................7713.4CONFIGURATION MANAGEMENT...................................................................................................................7713.5TESTING.................................................................................................................................................7813.6REQUIREMENTS .......................................................................................................................................78

5

1 Introduction

Many software-developing organizations try to enhance the quality of the products that they produce by focusing on a process-oriented approach of working [HAMMER+]. There exist different models of process oriented approaches that companies can be certified in, one common model is ISO 9000. According to Bo Manelius at ISOQA are small organizations underrepresented among the certified organizations. The reasons are the cost and the extra work that a certification involves.

We believe that small organizations are intimidated by the size of the existing quality models, such as ISO 9000, CMM or TickIT. The reasons for this can be that the quality models may have a bad reputation and are perceived to be bureaucratic and time consuming. The goal of this paper is to construct a quality framework that can be used by small organizations. This framework includes a quality definition, a quality model and a quality manual. The manual can be used as a first step towards a more process oriented way of working. It should be far from as large as CMM or ISO 9000, to make sure that the size does not become an intimidating factor.

To be able to do this, we must focus on the aspects of software development that can be improved with the least amount of work and bureaucracy. We assume that there are such aspects that can be greatly improved with little effort, for example, only one third of the programmers are aware of the concept of configuration management [BECK+]. Another example: One manager found software development to be like “Jumping from a cliff with closed eyes, and hoping that one lands on something soft”. The problem that he was referring to was that of choosing what technology to base the products on. If the chosen technology did not measure up, then the entire product had to be rewritten. That organization did not believe that there was any time to evaluate competing technologies. If developers are not even aware of basic control methods such as configuration management or know how to evaluate technical solutions, then we believe that it is possible to increase the quality in many organizations by focusing on a few carefully selected areas. To find these areas we make an empirical study on small software developing organizations in the south of Sweden. In this study we will try to find the areas in which small organizations normally fail, but also the areas in which they do not fail, as this does not have to be a part of the quality manual.

We will also examine what the concept of quality means for small organizations, so we know what such organizations want to achieve. We studied a number of employment opportunities where various organizations were looking for quality managers. All of these organizations were really looking for test engineers. The conclusion is that quality still is interpreted to be testing. This paper will show that quality management is about much more than pure testing even though testing is important.

We believe that it is possible to develop high quality software within small organizations. In fact, there is a great potential in smaller organizations. Organizations and projects can gain a lot from being kept small [LIBERTY], [BROOKS], [MAGUIRE]. They can adopt fast when circumstances change, they can use the potential of their team members or employees to a greater extent, and they lack much of the overhead work that is typical for larger organizations. Introducing formalized processes can help the organization bring out its full potential [ZAHRAN].

The intended reader of this paper is a person that is familiar with the basic concepts of quality management in software developing organizations. It could be someone working on a small company that wishes to introduce quality improving processes into their organization.

1.1 Addressed problemsWe aim to address the following problems, in the context of software development in small organizations.

6

What are the most important quality improving measures? What are the most difficult problems in developing software? We believe that it is important that the quality improving measures can be easily incorporated into the organization, and that they make as much improvement to the development cycle as possible. Otherwise they will not be accepted by the organization and, as a consequence, not used. By focusing on the most difficult problems and then use quality improving methods to cope with them, we believe that quality can be increased. The problem is to find the most rewarding quality improving efforts for small organizations.

How do small companies define quality? We believe that a firm definition of the term ‘quality’ can help when striving for higher quality. Therefore the problem is to construct a quality definition that suits small organizations.

Is it possible to construct a small framework for achieving higher quality in smaller organizations? The problem is to construct a small quality framework that can be used in small organizations with a desire to raise their quality. This is built on the conclusions that we drew from the previous problems.

1.2 GoalsThe goals of the paper is to:

Study awareness and usage of basic quality control methods in small software developing organizations and locate the areas in which they can be improved with minimum effort.

Produce a set of guidelines and recommendations of the most important quality enhancing methods. The guidelines must be easy to incorporate into a small organization, they must be flexible and the organization must be able to extend them when their quality maturity is increasing.

7

2 Structure and organization of this paper

This paper is organized in the following sections:

Quality – discusses the quality concept, and provides various definitions thereof. The quality concept is the foundation for any further discussion about improving quality.

Method – Description of the method that we used during the empirical study of software developing organizations. There is also a models used for analyzing the result of the study.

Problems – The problems that we believe small software developing organization faces. These are the hypothesis for the empirical study. The problems discussed here are partially based on the personal experience of the authors.

Interview– Contains a summary of each interview.

Analysis – Analysis of the empirical study, the result and various discussions.

Model – The conclusions drawn from the empirical study used to discuss what quality improving activities are most important, and examples of such processes are given.

Appendix A – Contains the questions that we asked during the interviews.

Appendix B – Answers to the questions, these have been generalized.

8

3 Quality

The chapter gives a background and discussion on the concept of quality in general. The goal is to provide a foundation for the discussion of quality as it applies for small organizations. The basic software engineering principles [GILB] are practiced in order to construct a basis for the quality concept. The principles states that in order to construct a system, the following must be known:

The goals of the system. “Projects without clear goals will not achieve their goals clearly” [GILB].

The requirements on the system.

How to fulfill the requirements.

The actual implementation of the requirements.

We aim to construct the quality concept by the guidelines that [GILB] describes by the goals, requirements and fulfillment. This chapter is divided into these sections:

Defining Quality for Small Organizations. The purpose of the quality definition is to document the goals of quality improvement activities in general and a quality model in particular. Quality is often defined in a single sentence or a short paragraph. The chapter provides an overview of definitions available. ISO, for example, has one quality definition and Total Quality Management is, in a way, a quality definition in itself. This is the goal of the quality framework.

Existing Quality Models and Small Organizations. A quality model is the requirement specification for quality improvement. Examples of quality models include ISO 9001 and CMM; the chapter discusses these in further detail. They are auditable standards, which means that a company can be reviewed to see whether they comply with the standard or not. These are the requirements of the quality framework.

Company Quality Manual, CQM. The CQM is a strategy for fulfilling the requirements of a quality model. The CQM is often company specific and contains well-defined processes that are followed to achieve a certain result and can (or should) be used directly as working instructions. This describes how to fulfill the requirements.

The documents that are produced or the results that become of following the CQM is the most concrete level, and will not be covered by this thesis.

The purpose of this chapter is to focus on the first two parts of our quality model, and to a lesser extent the CQM. It contains a general discussion over the concept of quality, and attempts to give an overview that leads to a definition of quality that can be the ground on which the rest of this thesis can stand on. It also contains an overview of existing quality models and their relation to small companies.

3.1 Defining Quality for Small OrganizationsThe definition of quality should provide the goals of quality work. “If quality is to be managed, it must first be understood.” (David Garvin). This chapter contains an elaboration on the definitions of the concept.

9

After the industrial revolution, quality became an important aspect of every company’s way to success [ENCYC]. It was no longer enough to be productive and effective, it was necessary to produce high quality products. Michael Hammer and James Champy express it like this: “Adequate is no longer good enough. If a company can't stand shoulder to shoulder with the world's best in a competitive category, it soon has no place to stand at all.” [HAMMER+]

We believe that it is fundamental for further work to agree upon a quality definition, since it is the base for the next step in our model of quality models. When a quality definition has been established we know what to strive for. There are, as we shall see, some major problems when agreeing on a quality definition - it is in fact a very difficult concept to grasp. There is a plethora of quality definitions available (see chapter Subjective experience below), we have chosen those that we find the most interesting. We have also studied how the quality models that we give a more in depth view on below (see Existing Quality Models and Small Organizations) define quality.

To summarize, these are the sources we used for our quality definition:

Subjective experiences – The view of quality from the individual point of view of the authors of this document, as well as collected opinions from different sources.

Total Quality Management - TQM has many insightful views on quality.

Existing quality models – Each model has its own definition, and they do not always overlap.

Conclusion – Bringing the pieces together.

3.1.1 Subjective experienceOne common set of quality definitions focuses on the subjective experience of the individual. There are many similarities with existentialism and this type of quality perception, accepting that there are objects of various qualities, but it is the individual perception that matters. This means that there are different quality definitions, based on the point of view of the person that makes the definition. Quality for the manager is not necessarily the same thing as quality for the customer.

In [GARVIN:1] David Garvin identifies five different types of quality definitions: transcendent, product-based, user-based, manufacturing-based and value-based. Below are examples of different quality definitions that fit into the different types of definitions.

Transcendent Definition

“Quality is neither mind nor matter, but a third entity independent of the two... Quality is a characteristic of thought and statement that is recognized by a non-thinking process. Because definitions are a product of rigid, formal thinking, quality cannot be defined. But even though Quality cannot be defined, you know what Quality is!” (Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance)

“The meaning of the word quality, like beauty, is in the eye of the beholder. ” (Binnie Perper, How to sell quality... in other words)

“Everyone has an idea of what quality is, but no one can clearly and unambiguously define it.” (B. Edvardsson, Kvalitetsutveckling : ett managementperspektiv)

Product-based Definition

"Differences in quality amount to differences in the quantity of some desired ingredient or attribute." (Lawrence Abbott, “Quality and Competition”)

10

User-based Definition

"Quality consists of the capacity to satisfy wants." (C.D. Edwards, Quality Progress, October, 1968)

"Quality is the degree to which a specific product satisfies the wants of a specific customer." (H.L. Gilmore, Quality Progress, June, 1974)

"Quality is any aspect of a product...which influences the demand curve." (Dortman and Steiner, American Economic Review)

"In the final analysis of the marketplace, the quality of a product depends on how well it fits patterns of customer preference." (Kuehn and Day, Harvard Business Review, 1962)

Manufacturing-based Definition

"Quality means conformance to requirements." (Phil Crosby, Quality is Free)

Value-based Definition

"Quality is the degree of excellence at an acceptable price and the control of variability at an acceptable cost." Broh, Managing Quality, 1982.

"Quality means best for certain customer conditions. These conditions are (a) the actual use and (b) the selling price of the product." (A. Feigenbaum, Total Quality Control)

These quality definitions focus on the concept depending upon the situation. David Garvin has built a model consisting of eight dimensions of quality, based on this fact [GARVIN:2]. The eight dimensions are performance, features, reliability, conformance, durability, serviceability, aesthetics and perceived quality. The problem, according to Garvin [GARVIN:2], is that the customer often is talking about another dimension of quality than the manager is thinking about.

3.1.2 The Total Quality Management model quality definition Total Quality Management (TQM) is a popular concept [TINGEY]. It is more of a philosophy than a quality model, but the ideas behind it can very well be interesting for small organizations. It is the implementation of it that might differ between a big and a small organization. CMM and ISO 9000 are all different ways of achieving Total Quality Management.

TQM was introduced by Dr. Feigenbaum in 1983 and became widely spread because of Japanese companies who recognized the benefits of the model [ENCYC]. Today, the term TQM lacks a universally agreed upon definition, but it basically consists of ten basic benchmarks, which together form a view on the concept of quality. The benchmarks are:

1. Quality is an organization-wide process. This is means that having a quality department that is separate from the rest of the company, or by focusing on for example just testing, optimal quality will not be the result.

2. Quality is what the customer says it is. If there was no customer; no one who had the requirements for the system, quality would not even be required. Since it is the customer who the product is produced for, he or she is the most important part of any production chain.

3. Quality and cost are a sum, not a difference. Quality will, in the end, save money.

11

4. Quality requires both individual and teamwork zealotry. Not only must quality be an organization-wide process, but the different parts of the company must also be able to work together in their efforts for higher quality. A well-built infrastructure is required.

5. Quality is a way of managing. Good management means bringing out the full capacity of everyone in the organization.

6. Quality and innovation are mutually dependent. Some organizations seem to want the creative process, before construction; to be as chaotic as possible, consequently bringing out the maximum of creativity from the people involved. But the quality must be a partner of development, from the beginning.

7. Quality is an ethic. No processes, numbers or metrics are enough to motivate people. What they need to feel is that the work they are doing is done well. With a constant focus on quality, they can be certain it will be.

8. Quality requires continuous improvement. You can never have enough quality. Quality is a constantly upward moving target.

9. Quality is the most cost-effective, least capital-intensive route to productivity. This is why process improvement is important. The race for quality never reaches the goal. There is always a better way to do something.

10. Quality is implemented with a total system connected with customers and suppliers. Based on the fact that the company can not operate without the customers or the suppliers. They are a unit, a complete system. Implementing quality without including all parts of the system is misleading.

TQM is not a quality model that you can use directly to follow processes for your work. One can perceive TQM as the requirements that need to be set on a quality model for it to be complete. The ten benchmarks described above makes quality a way of totally focusing the organization towards serving the customer, as well as market success, employee satisfaction and cost leadership. This has been summarized in shorter quality definitions of TQM, for example the following:

“A comprehensive set of tools, management philosophies, and improvement methods including: customer orientation, the empowerment of employees, participative management, data-based decisions, continual improvement, a ‘process’ orientation, and a set of quantitative tools for process improvement.” [KLEIN]

But other definitions exist, for example:

”The infrastructure, tools, methods and rules which result in increased business success and customer satisfaction by enabling continuous improvements to people, processes and products.” [BROCKA]

3.1.3 The ISO 9000 model quality definitionMuch has been written about ISO 9000. This chapter attempts to give an overview of the core values and quality definitions that ISO 9000 is built upon, as well as an introduction to the requirements that ISO 9000 puts on an organization.

ISO 9000 in itself does not define quality, but the ISO definition of quality is:

“The totality of features and characteristic of a product or service that bears on its ability to satisfy stated or implied needs.” [ENCYC].

12

Since this is the ISO definition, one would expect that it is the purpose of ISO 9000 to fulfill this. This definition does not mention for example process improvement or employee satisfaction. ISO 9004 introduces Organizational Goals [TINGEY] that includes the formulation:

“In order to meet its objectives, the organization should ensure that the technical administrative, and human factors affecting the quality of its products will be under control, whether hardware, software, processed materials, or services. All such control should be oriented towards the reduction, elimination, and, most importantly, prevention of quality nonconformities.”

This formulation implies that quality is about having control of the working process.

There is also an addition to the ISO 9000:1994 standard, which is called Quality Management Principles [ISOTC176]. A quality management principle is defined as: “...a comprehensive and fundamental rule or belief, for leading and operating an organization, aimed at continually improving performance over the long term by focusing on customers while addressing the needs of all other stakeholders.” There are eight different Quality Management Principles [ISOTC176]:

Principle 1: Customer-Focused Organization. “Organizations depend on their customers and therefore should understand current and future customer needs, meet customer requirements and strive to exceed customer expectations.” [ISOTC176]

Principle 2: Leadership. “Leaders establish unity of purpose and direction of the organization. They should create and maintain the internal environment in which people can become fully involved in achieving the organization's objectives.”[ISOTC176]

Principle 3: Involvement of People. “People at all levels are the essence of an organization and their full involvement enables their abilities to be used for the organization's benefit.” [ISOTC176]

Principle 4: Process Approach. “A desired result is achieved more efficiently when related resources and activities are managed as a process.” [ISOTC176]

Principle 5: System Approach to Management. “Identifying, understanding and managing a system of interrelated processes for a given objective improves the organization’s effectiveness and efficiency.” [ISOTC176]

Principle 6: Continual Improvement. “Continual improvement should be a permanent objective of the organization.” [ISOTC176]

Principle 7: Factual approach to decision making. “Effective decisions are based on the analysis of data and information.” [ISOTC176]

Principle 8: Mutually beneficial supplier relationships. “An organization and its suppliers are interdependent, and a mutually beneficial relationship enhances the ability of both to create value.” [ISOTC176]

These principles are somewhat similar to the benchmarks in TQM (see above).

3.1.4 The CMM model quality definition There exist no written quality definition in CMM, but since it is a five level scale where an immature organization is at level one, and a mature organization is at level five, in a sense the CMM define quality to be maturity. The more advanced your processes are, the better you are, according to CMM.

13

3.2 Existing Quality Models and Small OrganizationsA quality model is a useful tool for a company to control their working processes and in the end increase their quality. Without a process, improvement becomes harder [HUMPHREY] since the same mistakes can be repeated. It also becomes easier to incorporate new people to the organization. The principle 4, Process Approach [ISOTC176] in Quality Management Principles, also states that processes make the organization more efficient.

There are many advantages with small organizations compared to large ones. Organizations, and projects can gain a lot by being kept small [LIBERTY], [BROOKS], [MAGUIRE]. They can adopt fast when circumstances change, they can use the potential of their team members to a greater extent, and they lack much of the overhead work that is typical for larger organizations. Also, all of the advantages of being small might just as well turn against the organization if special considerations are not taken. Quick communication and high speed in taking decisions might lead to less time for reflections and long term planning.

A study performed in 1995 [STANDISH] showed that only 19% of all software projects finished on schedule and budget, and one of the main reasons for this was poor documentation and requirements handling. That figure is quite low, and there are obviously big potential profits to be made by having it increased. We believe that quality models can possibly aid in doing that.

Our personal experience is that the quality problem is as serious in small organizations as in large ones, but the ways of controlling it are different. This becomes apparent when one studies the requirements that CMM or ISO 9000 put on an organization. For example the need of independent reviewers, or even a quality team, is practically hard to satisfy in an organization of very few employees. This chapter attempts to enlighten the difficulties for small organizations to take advantage of the existing quality models. It focuses on the following quality models:

CMM – Software developing quality model. Chosen because it focuses solely on software development.

ISO 9000 – Large quality model in Sweden and most western countries. Chosen because it is widely used and well known.

3.2.1 ISO 9000ISO 9000 consists of two types of standards. The first type is a range of guidelines and is not auditable. Amongst these are ISO 9000-1:1994, ISO 9004-1:1994 and others [TINGEY]. The second type consists of auditable standards. This includes ISO 9001, ISO 9002 and ISO 9003. Of these three, ISO 9001 is the broadest in scope.

ISO 9001 consists of 20 quality system requirements, and these are further broken down into 46 subsections [TINGEY]. The quality requirements include: “Management responsibility”, “Quality System”, “Contract Review”, “Design Control”, “Purchasing”, “Control of Customer-Supplied Product”, “Product Identification”, “Process Control”, “Inspection and Testing”, “Control of Inspection, Measuring and Test Equipment”, “Inspection and Test Status”, “Control of Nonconforming Product”, “Corrective and Preventative Action”, “Handling, Storage, Packaging, Preservation and Delivery”, “Control of Quality Records”, “Internal Quality Records”, “Training”, “Servicing” and “Statistical Techniques”. They are not divided into maturity levels as CMM.

The assessment is a 4-step process [TINGEY]:

1. Preaudit (optional). Preparation for the next step.

2. Audit (pass/fail). Detailed audit of the company’s quality management systems. Can last several weeks.

14

3. Registration. If passed in the previous step, the company is registered.

4. Surveillance audit (semi-annual/on-going). Continuous monitoring and verification.

ISO 9000-3 contains guidelines for adopting the ISO 9001 to a software developing organization.

The quality definition of ISO (see The ISO 9000 model quality definition above) implies a very customer-oriented approach. But the ISO 9000 model is simpler than that. It can actually be summarized in the following steps [LJUNGBERG]:

1. Say what you do (procedures).

2. Do what you say (responsibility).

3. Document what you did (document).

4. Show it (certifying/revision).

This means that as long as you document what you do, you can basically do anything. There are some regulations in ISO 9000 that states that laws etc. must be followed, but very little beyond that. In a sense ISO 9000 is thus a quality model that does not focus upon the quality of the product being produced but more on the process of producing the product, which might seem strange. A new version of ISO 9000, called ISO 9000:2000 is currently being developed [LJUNGBERG], where they focus upon this and (and other) issues.

3.2.2 SEI CMM for SoftwareThe Capability Maturity Model is aimed specifically towards software developing organizations. It was developed by The Software Engineering Institute, which is operated by Carnegie Mellon University. It is a model for software process assessments and it is broken into 5 maturity levels. The maturity levels (except the first) are further divided into 18 Key Process Areas, which are further broken down into 52 goals and 316 key practices. CMM have (intentionally?) avoided the quality concept, they are referring to maturity.

The following levels are included in the CMM-model [PAULK+]:

1. Initial. None, or few, defined processes. Ad hoc development. Success depends on individuals who perform “heroic” acts of programming.

2. Repeatable. Documented and stable estimating, planning and commitment processes are available. People are more trained on software engineering concepts.

3. Defined. Integrated management and engineering processes are used across the organization. Data are collected and used in all defined processes. People are constantly trained and educated.

4. Managed. Processes are quantitatively understood and stabilized. Sources of individual problems are focused on and eliminated. Data are used to understand the process and stabilize it.

5. Optimizing. Processes are continuously and systematically improved. Common sources of problems are understood and eliminated. Everyone is involved in process improvement. Data are collected to evaluate and select process improvements.

15

Very few companies reach higher than level 3 and a minority reaches level 3, or even level 2. During 1987 and 1994, 379 CMM assessments were conducted by SEI in USA. They gave the following results from SEI/CMM:

Level 1: Initial 73,0%

Level 2: Repeatable 16,0%

Level 3: Defined 10,0%

Level 4: Managed 0,6%

Level 5: Optimizing 0,3%

0Table 1 CMM assessments 1987-1994

But the situation has improved. Recent assessments from 1993 to 1999, performed in 697 organizations, give the following results from SEI/CMM:

Level 1: Initial 55,2%

Level 2: Repeatable 25,8%

Level 3: Defined 15,6%

Level 4: Managed 2,7%

Level 5: Optimizing 0,6%

0Table 2 CMM assessments 1993-19999

This indicates a major improvement, and might be an indication of a change in the software industry towards higher process maturity and quality.

Level 2 is the first level that puts any requirements on the organization. The Key Process Areas in level 2 are:

Requirements management. Includes, amongst other, processes for documenting requirements, keeping project plans and documents consistent with changing requirements and reviewing new requirements before adopting them into the project [PAULK+].

Software project planning. Includes, amongst other, processes for documentation and derivation of estimates, commitments to work, documenting project planning, identifying a software lifecycle with predefined stages of manageable size, recording software planning data and deriving the schedule for the project [PAULK+].

Software project tracking and oversight. Includes for example progress reporting, agreeing to changes of commitments, communication of the status of the project, process for taking corrective actions, process for project schedule tracking and basic risk management [PAULK+].

Software subcontract management. Includes, amongst other, a process for selecting software subcontractors, holding periodic technical reviews and interchanges with the subcontractor, tracking the progress of the subcontractor, a process for inspecting the subcontractors software engineering capabilities, conducting acceptance testing as part of the

16

delivery of the products from the subcontractor and evaluating the performance of the subcontractor on a periodic basis [PAULK+].

Software quality assurance. The purpose of this key process area is to make sure that management is provided appropriate visibility into the process being used by the project and the products built. It includes the forming of an SQA group, reviews of the projects development plan, standards and procedures and measuring of the SQA groups activities to determine cost and schedule status etc [PAULK+].

Software configuration management. Includes a commitment to control changes to identified software work products and following a written organizational policy for implementing software configuration management, establishing a configuration management group, initiating, recording, reviewing, approving and tracking change requests and problem reports for all configuration items according to a documented procedure and conducting software baseline audits according to a documented procedure, amongst other [PAULK+].

We will not go into detail in the higher levels.

3.3 Company Quality Manual, CQMThe Company Quality Model describes how the company will fulfill the requirements of the quality model. It is company-specific documentation of the processes being used in the company. Absence of such processes can lead to ‘fire-fighting’ [ZAHRAN] which minimizes the team’s capability of achieving the common goal.

A process can be defined in many ways. [ZAHRAN] presents a holistic view of the definition of a process. In this view, a process has three aspects:

The Document specifying the process

The Knowledge in people’s brains to drive their behavior, and

The Results of the Process.

The CQM provides the first aspect of this view. The last two aspects are not covered by this thesis, but would fall under the forth layer in our quality framework.

3.4 ConclusionBy following basic software engineering principles [GILB] we have constructed a model of a quality framework, divided into the following parts:

A quality definition.

A quality model.

A documented process, or a CQM.

The documents that are produced or the results that become of following the CQM.

The task now becomes to establish a quality definition and a quality model and from this construct a CQM for small organizations.

17

By looking at different existing quality definitions, as well as relying on our own experience, it has been shown that quality is a subjective experience and very difficult to define in general. If you want to define it, you must take a set of different aspects into consideration. David Garvin introduces 5 different types of quality definitions [GARVIN:1]. They are Transcendent Definition, Product-based Definition, User-based Definition, Manufacturing-based Definition and Value-based Definition. The existing quality models’ quality definitions often lack important aspects, such as process improvement, employee or customer satisfaction or business success.

The quality models ISO 9000 and CMM include a wide range of requirements, which all must be fulfilled by an organization in order to be certified [TINGEY], [PAULK+].

The documented process, or CQM, describes how to fulfill the requirements of the quality model. It is a documentation of the processes being used in the company. Absence of such processes can lead to ‘fire-fighting’ [ZAHRAN].

18

4 Method

This chapter describes the method that is used during the scientific research and empirical studies, in order to gather and analyze new information. The material in these chapters has largely been based on [EJVEGARD] and [ELY+].

There are many types of methods available for scientific research, examples are case studies, descriptions, classification, quantification, etc. The method that we choose to implement the paper on is a qualitative study and hypothesis. Hypotheses are validated through interviews.

4.1 Qualitative studyWe study the awareness of quality models and also the problems that small software developing organization face, by performing a qualitative study based on interviews. The most accurate way to study this is probably by conducting some type of active research [IVERSEN+] or an ethnographic study. This is not a feasible way for us due to the short time available to produce the paper. The best alternative that we have found is to perform interviews with a set of representative organizations. The main problem with interviews is that we can not expect the organizations to simply answer the questions “what problems are there?” There is a risk that such questions would give the answer “there are no problems!” To remedy the situation we have developed a set of problems that we suspect that these organizations face. The questions that make up the interview have been designed in such a way that they can test the validity of the expected problems. It is possible that this method does not reveal all problems that these organizations face, but we hope that it reveals sufficiently many. The study is not going to measure the impact that the problem may or may not have in the organization, it will only find presence or absence of the problem.

The empirical study is hence conducted as formal interviews. Formal interviews mean interviews with a set of predefined questions. There is room for exploration within the interview, but the goal is to stick to the questions. The questions are asked in the same order for each organization (where applicable) to avoid that the order affects the answers of the questions. The purpose of the interviews is to determine the validity of our hypothesis regarding small organizations.

During the interview, some steps are taken to ensure that the interview is not unnecessarily interrupted. If it is possible we will conduct the interviews on some type of ‘neutral’ ground where there are, for example, no telephones.

We plan to perform five interviews from five unique companies, with two persons from each company.

4.1.1 DocumentationWe are going to use tape recordings to document each interview, provided that the persons being interviewed agree on being taped. If we may not use tape recordings then notes will be taken by hand. We have chosen not to employ a video camera due to practical reasons, a video camera would probably unnecessarily disturb the interview since there are so few persons present.

The tape recordings will later (preferably the same day as the interview) be transcribed into a log. This log will be edited for size and later be sent to the persons that were interviewed for acceptance.

4.1.2 SelectionHow are the interviewed organizations selected? The purpose of the paper is to examine small software developing organizations, so there are limits of the size of the organization. Small is defined as less than or equal to 20 persons developing software. This criterion is approximate so the organization may be larger if otherwise interesting. The organization must be developing software products. It does not matter what type of software they are developing. It is also a

19

purpose of this paper to examine organizations that currently are not certified according to some quality model. Due to practical reasons only organizations in the south of Sweden are selected. The organizations probably has an office in Malmö/Lund, or at least in Skåne.

It would be difficult to ensure objectivity if we have been working within the organization, so it is desirable that we are not already familiar with the organization. By organization we do not necessarily mean a company, an organization may be the software department within a company that does other things, e.g. hardware development, than software products. The resulting requirements for selecting organizations are:

Develops software products

Size, less than 20 persons developing software

Not certified according ISO 9000, CMM, or similar

Not already familiar with the organization

Located in the south of Sweden

A list of possible organizations that fulfill our requirements is compiled and then we randomly select five from that list. Two persons from each company will be interviewed. The first person that we are interested in interviewing has responsibility for the end product. He/she is developing software, and is communicating with the customer. Ideally the person is some type of project leader. The other person is a developer, a programmer, which probably has less contact with the customer. To keep objectivity, we are not interested in interviewing people that we have had previous personal contact with.

First person from the organization:

Responsibility for end product

Involved in software development

Customer contact

No previous personal contact

Second person from the organization:

Developing software, close to the code

Probably less customer contact

No previous personal contact

These selection criteria are the default, if we find an organization/person that does not fulfill all the requirements and still is interesting this will be documented in the interview.

4.1.3 TimeTotal time for each interview is estimated to be one hour. That time will be kept rather tightly, as it is important that the interview stays focused and does not disappear into irrelevant discussions. Time is valuable, and we do not wish to allocate the interviewee longer than necessary.

20

4.1.4 ConfidentialityThe purpose of the paper is to analyze problems with current state of practice in small organizations. It is thus necessary to discuss problems within certain organizations. The organizations will probably not want to participate in an interview that results in a paper where the organization is described as having problems. Therefore is it necessary to ensure that the organization and person participating in the empirical study are guaranteed anonymity. The interviews must be confidential. The questions have been designed in such a way that they do not obviously reveal what organization that has answered them. In cases where products or product descriptions are mentioned, that might lead to readers being able to figure out which company was interviewed, these are changed to a more general form.

4.1.5 AccessWe gain access to the organization by contacting them via e-mail. If there is interest within the organization to participate in the interview then a time and a location is decided. The original e-mail will describe who we are and the purpose of the interview; we will also describe the type of person that we are interested in interviewing.

4.1.6 QuestionsThe questions that are not applicable to the organization at hand are ignored, but it is documented what questions that have been skipped.

The questions have been designed in such a way that they will try to answer if the hypotheses are correct, or wrong. It is easy to ask questions like “do you have problems with configuration management?” this has been avoided since such questions are closed. The aim has been to design as open questions as possible that can be used to determine the validity of our hypotheses.

4.1.7 OutlineThe interview will begin with a presentation of the work that we are performing. After that the person being interviewed is asked to describe the organization and what type of work that he/she is performing in the organization. These are hopefully easy questions and will, hopefully, remove some tension. The main questions are then worked through in the order they appear in the question document. There will be room for small exploratory paths away from the questions, but the goal is to stick to the questions.

When the interview is finally over the person that has been interviewed will be asked if he/she is interested in receiving a copy of the final report.

4.2 ReliabilityThe paper is based mostly on the findings from the interviews. If we do not manage to produce reliable results from the interviews the entire paper will become highly unreliable. We have identified some aspects of reliability that we will try to resolve.

4.2.1 ObjectivityObservation can never be entirely objective [ELY+], but we will try to obtain as much objectivity as possible. There is a problem with interviewing people working within software developing organizations, since we are familiar with the environment and culture that often is present there. That familiarity might make us see things the way we expect them to be, not how they are. We believe that awareness of the problem of objectivity is the best way to ensure it.

21

4.2.2 Risks during the interviewInterviews are by no means an exact science. There are many possible ways that interviews can result in erroneous conclusions. We have identified a set of risks, and strategies for minimizing or avoiding them:

The person being interviewed says what we want to hear. This could possibly be avoided by performing the interview on neutral ground like a cafeteria.

We are told how things should be, not the actual state of practice. There have probably been discussions within the organization about how things should really be. There has however never been any time to realize these plans. The questions must be designed in such a way that they result in the current state of practice, what they are actually doing.

We do not interpret the person being interviewed correctly. Possible reasons for this might be that we do not share the same definitions for the concepts that are discussed. It is not obvious what requirement, test or problem actually means. The first step avoid this risk is to tape the interview, that will minimize the risk of us not hearing what they are saying. The other risk is more difficult, but we must remember to ask for definitions of the ‘fancier’ words.

The questions are designed in such a way that they lead the answer. There is a balance between asking open enough questions, and asking specific questions. Too open questions will not provide the answer that we are interested in and too specific questions risk leading the answer. We have designed rather open questions, and we will ask counter questions if it is obvious that it was misinterpreted.

4.3 Analysis of dataThe following process is followed when analyzing the data:

1. The interviews are translated into English.

2. The interviews are sent to the interviewees for approval.

3. Each interview is analyzed and an analysis document is prepared. The analysis is based upon the interview and the hypothesis regarding software development. The analyses are summarized in a conclusion of the interview, which includes a grade to which we feel that the hypothesis fit into the organization. This grade is from 1 to 5, where 1 means that the hypothesis did not fit in at all, and 5 means we were right on the spot.

4. Each answer to each question is studied and generalized in order to find patterns among the answers. These answers and patterns are available in Appendix A.

5. Finally, a conclusion of all the interviews are made and the grades for each hypothesis and interview are added. A matrix of the hypothesis is produced and an average of the grades for each hypothesis is calculated. An average value of 3.0 or more indicates that we were correct.

6. For each hypothesis, a conclusion is written.

4.4 ConclusionA qualitative study based upon short formal interviews with a set of small Swedish software developing organizations. The interviewed organizations are develops software products with less than 20 persons, are not quality certified and that we are not already familiar with.

22

Two persons for each organization are interviewed where one person is some type of project manager and the second person is more like a programmer. The purpose of the interviews is to be able to validate a set of hypotheses.

The result of the interviews is analyzed with the objective of establishing compliance or non-compliance with hypotheses. The strategy is to assign a score (1-5) to each hypothesis for each company. If the score is more or equal to 3 then the corresponding hypothesis is validated for that organization. The average of all scores is presented in the interview conclusion.

23

5 Problems and hypotheses

This chapter contains descriptions and discussions about the hypotheses that we have regarding software development in small organizations. These hypotheses are used as the basis for the qualitative study.

The hypotheses are based partially on our personal experience from developing software in the industry and partially on literature studies; they may or may not be valid, that is what the empirical study is going to show. Some of the organizations that we have encountered were not aware of configuration management, no project plans were established products were not tested before delivery, requirements were not documented and bugs were not corrected. Many organizations realize that it is necessary to make nightly backups, but how many have actually verified that these backups are correct? We wish to examine if our personal experiences are generally true. We have identified the general problem domain, and some hints to be looking for in the analysis of the interviews. We believe that small organizations face problems in these domains:

View of software development – Development is too short sighted.

Planning – No or poor project plans are produced and maintained.

Management – Is not working with the developers.

Selecting technical solutions – The solutions are based upon prejudices and commercials.

Configuration management and version control – There is no control over versions of work products.

Testing – Basic ad-hoc testing is performed, but little is documented.

Requirements – It is not known what the customer really wants.

The domains are similar with the key process areas in CMM level 2 (repeatable) [PAULK+]. That leads us to believe that these problems are those that are most necessary to correct before more rigorous quality models can be introduced in the organization.

Each hypothesis description begins with a theoretical background. This background serves the purpose of motivating the existence of the hypothesis and the indications therein. After the background is a description of the hypothesis itself. The hypothesis describes a hypothetical state in an organization. The hypothesis is concluded with a set of indications that the hypothesis is correct; these indications will be used later on when analyzing results from interviews.

5.1 View of software developmentBackground. Developing new software products is an investment. It takes time and effort to develop and the end product will in many cases survive for years to come. This is indicated by the large cost of maintenance. Maintenance costs are as much as 40-70 percent of the cost during the total lifecycle [BOEHM], where maintenance is defined to be post-delivery changes.

Hypothesis. The hypothesis is that the organization does not consider the long-term nature of software when they are developing it. Planning is focused on the next deadline; or the next visit by the customer. They are often aware that they should document, test etc, but they feel that there is no time for such things. In the very short view of software lifetime, there is no time. When the goal is to finish the current project and not long-term success then haste and stress follows. Small

24

organizations can probably not develop strategic plans for the next ten years, but it should be possible to predict that there will be another project. There is a problem with maintenance of old products.

If the organization realized that they should have a more long-term view of their software investment than we believe that the next deadline would not be as important as continuos improvement. With a longer-term view of software development, quality systems and continuos improvement is easier to motivate.

One of the most important aspects of software development is often time-to-market due to the competitive advantage of getting to market sooner [CROW]. So there is no time for too much control, or too formal processes. When the focus on time-to-market is high and a project risk becoming late, management hires more people and adds them to the project. Adding more peoples to a late project risk making it later [BROOKS].

Indications.

Quality System. There is currently no existing quality system concerning the entire organization and the processes by which they produce software.

View of ISO 9000. Their view of for example ISO 9000 is based on prejudices. They think that it will cost too much to formalize the way they work into a process. It may or may not be true that the cost is too high, the focus here is that the organization does not know if this is the case. The reason that we have chosen to focus on ISO 9000 and no other quality models such as CMM in the hypothesis is that we believe that ISO 9000 is the only major well known model in small software organizations. If the view of ISO 9000 is based on prejudices we conclude that the same would be the case of other models as well.

Experience preservation. They do not in a formalized way preserve experiences from previous projects. When experiences are not taken care of the organization risk facing the same problems in the next project.

Brooks Law. When late for a deadline they add people to the project or work overtime in stead of discussing a more long-sighted solution with the customer. Brooks law: “Adding manpower to a late software project makes it later” [BROOKS].

5.2 PlanningBackground. Poor planning is the source of problems more than any other [METZGER]. Large projects must be broken down into smaller manageable tasks (divide and conquer), called activities or miniature milestones. Activities are only a few days worth of time and defined to be ‘done’ or ‘not done’ [MCCONNEL]. The reason to break down projects into smaller activities is to be able to track progress, “Software progress monitoring was so poor that several well-known software disasters were not anticipated until the very day of expected deployment” [JONES:2]. We conclude that formal progress reporting is essential for any software project. Without control on daily basis the project risk being delayed, “How does a project get to be a year late? One day at a time.” [BROOKS].

There is a risk in putting too much faith in a detailed project plan as well; the anti pattern [BROWN+] discusses this risk and calls it Death by planning. “The deliverables plans should be updated weekly to ensure appropriate planning and controls that reduce project risks” [BROWN+].

Hypothesis. The hypothesis is that there are no or vague project plans. The delivery dates of the products are decided by the management, which seldom asks the employees first. A plan is often produced at the beginning of a project, but it is not maintained during the project. There is no way

25

to tell if the project is running late, or if the deadlines will be held. The plan is not based on activities that have been produced by dividing the project into smaller, manageable pieces. It is difficult to know if the project is on schedule or not, since no progress reporting is performed.

Indications.

Divide and conquer. Projects are not divided into activities (or miniature milestones). The project is planned as a whole, work tasks are identified but these are very large and involve many peoples. It is difficult to measure progress in such big activities. Project plans are not revised. Establishing a plan early in the project is not worth much if it is not revised to mirror reality as time goes by.

Progress reporting. Progress reporting is non-existent. Project progress is based upon gut feelings, thus is it difficult to know the current state of the project. There is a risk to fall for the 80-20 problem. That is, when one believe to be 80% finished, only 20% is actually.

Deadlines and milestones. There exists no deadlines or milestones. Given enough time will make any problem go away. A project with no deadlines can never be late. The deadlines are decided by management, with little or no input from the people responsible for implementing the project.

Use of estimates. Estimations that are the basis for project planning are based solely upon experience and gut feelings. It is difficult for new employees who lack experience to produce reasonably reliable plans.

5.3 ManagementBackground. We have not found any theoretical background for this hypothesis; it is based solely upon our personal experiences developing software in small organizations. This makes the hypothesis clearly weaker than the others presented in this chapter. The reason that we have chosen to include it in the paper is that it is a personal experience and we wish to examine if this is generally the case.

Hypothesis. The hypothesis regarding management is that the organizations has been founded by one ‘older’ male with excellent knowledge of some domain and that this person is controlling the organization like a despot. This person hired a few ‘programmers’, and they all set out to develop new software. The management person had a very hard time to realize the software development specific problems, and the programmers are not trained in seeing the greater picture. The programmers are excellent ‘coders’, but they do not see anything beyond technical solutions. It is possible that the organization lacks a project manager, or some type of authority between the management and the ‘coders’. The manager is not taking active part in the development of the project, but believes that the developers should be left alone, which is odd since it is he that sets the deadlines. As the projects proceeds and the manager gets new ideas for the end product new requirements are added to the end product.

Indications.

Project managers. There are not any project managers. Absence of a person who is responsible to managing communication between higher management and developers indicate this problem.

Management part of development. Management does not take active part of development and has therefore little knowledge of software development problems. Despite this he is setting the deadlines, defining requirements and signing the contracts. This leaves the developers in an impossible situation since someone else decides when they are going to be finished.

26

Adding requirements. Requirements are added without the influence of the developers. The manger is in contact with the customer, and promises new features.

Developers domain knowledge. The developers have poor knowledge of the end user environment that they are developing software for. The manager who has this knowledge assumes that the developers are as familiar with the domain as he is.

5.4 Selecting technical solutionsBackground. Solutions are the means to provide implementations of the requirements on a software product. The technological solutions that are selected define the end product. In order to select such solution must the requirements be know; the qualities of a suggested solution must be known, along with the drawbacks of the solution and the resource consumption required [GILB+]. When a set of possible solutions have been found an evaluation must be performed in order to find the solution that best suits the problem [GILB+]. “A solution worth considering is one whose positive contribution to your requirements outweighs its negative ones” [GILB+].

Hypothesis. The hypothesis is that the technical solutions that are selected are the result of brief discussions, or selects products from company X alone, because they have always done so. There is no technical reason why they select the solutions they do. The organization is limited by the technologies they have experience in, and they are applying the known technologies for any problem. Programmers refuse to consider alternative languages, or database solutions. When new technologies are introduced into a project is it unknown if the product will meet the necessary requirements. The decision to select a specific solution is not taken by the project as a whole but rater by management, with little input from developers.

Indications.

Vendor lock-in. A company is using product X from vendor Y because they have always done so. Technologies are selected mainly from previous experience; this results in problems to adopt new technologies. The concept has been borrowed from the anti-pattern with the same name [BROWN+].

Evaluating technologies. New technologies are not evaluated. New untested technologies are accepted as solutions without evaluating them or comparing them with similar technologies. When using new technologies is not validated that it will perform according to the requirements. New technologies are not subjected to proof of concept tests in advance of the selection. It is thus unknown if the technology is sufficient.

Golden hammer. A single solution is applied to all problems. The company uses the same solution for any problem. The concept has been borrowed from the anti-pattern with the same name [BROWN+]

New technologies evaluated by management only. Management takes decisions of technologies on strategic meetings, without developers’ influence.

5.5 Configuration management and version controlBackground. Software configuration management is the process by which change and evolution of software products is controlled. Configuration management improves productivity and product quality [FEILER]. Configuration management reduces overall software development costs [TOMAYKO]. “Lack of automated source-code control is a common and irksome inefficiency” [MCCONNEL].

27

One of the most important configuration management problems is management of simultaneous updates [TOMAYKO]. If this is not managed then there is a risk that different changes distort the end product and thus causes software failure. [TOMAYKO]. There are configuration management strategies that allow simultaneous access to source code or other documents. This strategy requires manual effort to merge changes made by different persons. This is the strategy used by cvs (http://www.cyclic.com/) for example. During the interviews we will ask how they ensure atomic file access, if the answer is that this is not necessary since they accept merging changes then the indication is wrong.

Hypothesis. Our hypothesis is that problems occur when the company has many customers and many versions of the same software at different customer sightings. What if customer A finds a bug in a specific version, and some other customers are affected as well. Does the company know which version of the product is installed where? The organization has problems with ensuring that only one person is working with a file at a given time. It is not possible to restore the last version of a file, except by restoring backup tapes.

Indications.

CM tool. Without tools the management of versions and files are ignored or managed by hand. Managing these issues by hand is time consuming, difficult and boring.

Working version control. Rely on backup copies for previous versions of files. There are many problems with backups for example (1) there is no history information (2) information is backed up only once or twice a day (3) it is time consuming to restore backups.

Delivery control. The company does not know which version of the software is installed where. It is not even sure that the customer knows what version he/she has installed. Lack of this information makes it difficult to restore the customer configuration for finding defects. It is not possible to reproduce versions being delivered to the customer. It is not possible to restore the customer configuration in the office. Since the company is unable to do this, they have very hard time to find defects.

File control. It is possible that two persons work in the same file simultaneously. They do thus risk ruining each other’s work, or produce inconsistencies in the files.

5.6 TestingBackground. It is necessary to test software products. There are developers that try to avoid testing by developing defect-free software using various methods but “attempts to construct perfect software are doomed to fail” [HAMLET]. What testing method is then the best to use? “No one method, no one set of methods, can guarantee finding all bugs” [BEIZER]. Even though it is impossible to find all bugs in a software product, it is not possible to avoid testing. In all test strategies is it not only necessary to document how to test but also to predict the outcome of the test. If the outcome can not reliably be predicted then there are misconceptions as to what it is the routine should and is doing [BEIZER]. The purpose of testing is to find the important defects, in order to remove them, before they make their way to the customer. Testing the actual end product is not the only work product to test. If defects can be found and eliminated before they make their way to the end product large savings can be done; “the cost of removing defects from software increases dramatically with the time the defects remain in the product” [BOEHM]. Software inspections have reported test execution cost reduction of 5 to 10 times, since there are fewer defects to find [GILB+].

Hypothesis. Some type of testing is performed, that is the source code is compiled and executed. Possibly some type of ad hoc testing as well. There is no formal test plan. When the customer inspects partial deliveries, or even the end product, everyone is very nervous that major defects will be detected. No regression testing is performed; it is common that defects that have been

28

corrected reappear again and again. The organization is not familiar with simple testing concepts such as coverage tests, or statistical testing.

Indications.

Testing strategies. None or just one testing strategy is followed. The testing strategy is probably input validation. No single testing strategy can find all defects, relying only on input validation is thus not sufficient.

Test cases documentation. Test cases are not documented. With undocumented test cases the process of finding defects is ad-hoc. Each test run is different from the previous.

Binary testing focus. Only binaries are tested. Other types of tests such as peer reviews or inspections are not performed. Inspections can be performed earlier in the development cycle than tests on binaries and can therefore potentially save time and money.

Validation. The result of a test is not known before the test. If a test is performed, and the result of the test is not known in advance, than any result is acceptable. Some defects such as program crashes can be found but more subtle defects are difficult to find.

Root cause analysis. Root cause analysis, the process by which one attempts to find the real reason to a problem. Found defects can reappear or produce follow-up errors, since no root cause analysis is performed.

5.7 RequirementsBackground. Requirements for software products are the description of the desired properties of a system. The requirements define what to be implemented. Requirements are also one of the most difficult aspects of software development. Incomplete and changing requirements are two of the main reasons to why project are delivered late, over budget and with less functionality than desired [STANDISH]. A European study also showed that the management of customer requirements was one of the principal problem areas in software development [KOTONYA+]. The risk of adding features that are “good to have” is known as feature creep. Projects that fail to control creeping requirements are highly accessible to excessive schedule pressure, it can even destabilize the product to such a degree that is can not be finished at all [JONES].

Hypothesis. Our hypothesis is that the products developed are based upon the customer specifications, but little time is spent on analyzing the requirements to ensure that they are the correct. The customer is maybe not sure what he/she really needs. The set of existing requirements is in constant flux. Also, it is very difficult to know that you have interpreted the customers’ demands correctly. The customer often has very deep domain knowledge, but is uncertain about the terms used in software development or the industry in general, which leads to misunderstandings and communication problems. Requirement changes are discussed over the phone or mentioned in meetings. These changes are not managed, and thus easily forgotten or misinterpreted.

Indications.

Requirements specification. Initial requirements are poorly documented. The initial requirements are unknown and correspondence with the customer is based on phone calls and e-mail, the information from this communication is not documented. It is thus not known what to build.

Feature creep. New requirements are poorly documented which makes feature creep a problem. The requirements are in constant flux. The customer constantly changes his/her mind; the reason to this is that they do not really know what they want.

29

Requirements verification. There exist no formal way of communicating requirements with the customer. The requirements analysis is based upon ad-hoc processes, it is not known if the agreed upon requirements are the right requirements.

5.8 ConclusionThe chapter presents a set of hypotheses that are used as the basis for the empirical study, these are partially based upon personal experiences of the authors and partially on literature studies.

View of software development. The hypothesis is that the organization does not consider the long-term nature of software when they are developing it. Planning is focused on the next deadline, or the next visit by the customer. Longer-term quality achievements are not prioritized or encouraged.

Planning. There are no or vague project plans. A plan is often produced at the beginning of a project, but it is not maintained during the project. There is no way to tell if the project is running late, or if the deadlines will be held.

Management. One single male person that controls all aspects of the development manages the organization. This person is controlling the organization like a despot.

Selecting technical solutions. The organization is limited by the technologies they have experience in. Programmers refuse to consider alternative languages, or database solutions. When new technologies are introduced into a project is it unknown if the product will meet the necessary requirements.

Configuration management and version control. There exist no plan for controlling configuration of source code, documentation, releases or any other work product. It is not possible to retrieve older versions of work products.

Testing. There is no formal test plan. When the customer inspects partial deliveries, or even the end product, everyone is very nervous that major defects will be detected. No regression testing is performed; it is common that defects that have been corrected reappear again and again.

Requirements. Products developed are based upon the customer specifications, but little time is spent on analyzing the requirements to ensure that they are the correct. The customer is maybe not sure what he/she really needs. The set of existing requirements is in constant flux.

30

6 Interview analyses

This section contains the analysis of each interview performed. The actual transcriptions from the interviews are not available in this document. The analyses are based on our hypotheses and on the answers from the interviews. The analysis has been as objective as possible, but we have probably been affected by our impressions from the meeting as well.

Two companies declined the opportunity to participate in the interview series. The stated reason was that they could not spare the time. The total time for the interview was two hours for each company. The companies that could not plan for two extra hours in the next three weeks probably had important views upon the software quality concept. There is hence a risk that we have not interviewed the companies that are most important.

6.1 OverviewEach interviewed company is analyzed in a separate chapter. The chapter begins with a short description of the company and its domain, and the persons interviewed. The analysis is based on the hypothesis presented earlier; the goal here is to find if those hypotheses are consistent with the interviews or not. To do this, we make a judgment of how well the hypothesis stood up against reality for each company. To quantify this judgment, a score from 1 to 5 measures the consistency, where a score of 5 indicates full consistency and 1 means no consistency. Since there is a range between 1 and 5 we have chosen to set the value 3 as the limit for when our hypothesis was right or wrong. This means that when all the interviews are summarized the hypothesis’s that received an average value of 3.0 or higher are verified.

These scores are the results of evaluating the indications of the hypothesis with evidence found during the interview. Since the interviews only lasted for an hour each, these scores are affected by our subjective impression of the company, as well as the answers they gave to our questions.

The general structure of an analysis is as follows.

6.1.1 Name of hypothesisIndication 1…n. Describes how the indication stands up against reality.

Conclusion. A conclusion of the hypothesis and a final motivation for the grade.

Name of hypothesis #grade

31

6.2 Company IOnly one person is interviewed in this company. The company consists of two persons so we do not believe that a second person would have contributed with much extra information. The interview was conducted in the home of the interviewee.

This company is more a hobby than a full time work. It might not be fair to compare them to other software developing companies, but we have chosen to include them in the survey since this is the way that many larger companies have been founded. When the company grows it probably becomes more necessary to structure the work, but the more people involved the more difficult it probably is.

6.2.1 Presentation of the companyThe company is developing and maintaining software products and web pages. Two persons founded the company in 1996. The company is focusing on database front-ends in Visual Basic. It is a small organization, where the employees are the owners and do not depend on the company’s income for their own living. It is closer to a hobby than a full time job.

6.2.2 Presentation of the interviewedPerson I

He is 40 years old and is working for Company I at a part time basis only. Does not have any formal education in software development. The other person working within the organization has a formal education as a programmer.

6.2.3 The hypothesisView of software development

Company I’s view of quality is based on the assumption that it is the software in itself that carries the quality; the customer is involved only as an end user of the product. Usability and stability are the main concerns. These two make quality in software development difficult. By usability they mean a “...clean user interface”. This is based on personal experience and the domain knowledge of the company. The second most important factor is the structure of the database, and ensuring the relations. The problems are hence more or less technical by nature.

Quality System: The company had not been discussing how or if they needed to structure their work. The company currently does not use any quality system, although they think that doing so could improve the way they work.

Experience preservation: No project conclusions are made. Experience gained is focused on technologies and programming.

View of ISO 9000: They have no or little relations to ISO 9000, but believe that it might help software-developing organizations to structure their work.

Brooks Law: The are only two persons working within the company, and they do not hire more people even if they would be late, so they do not risk falling for Brooks law.

Conclusion: There is no quality system implemented in the organization, nor are there any concrete plans to do so. Problems addressed in earlier projects are not preserved. There exist no plans for improving the way that they work. This is because Company I is producing software because they think it is fun, and as a hobby. They may not have many of the problems that larger

32

organizations do, but there is no long-term plan or view of software development so the hypothesis scores 5.

View of software development 5

Planning

Divide and conquer: The project is not broken down into activities; the short term planning is managed with TODO lists, which is a common way of keeping track of activities. The fact that they do not have any deadlines makes further planning unnecessary since they can never become late. This also leads to the fact that estimates are unnecessary and measurement of progress seems to be non existent. The schedule is based upon what windows and tables that have been designed.

Progress reports: Progress is not documented, or measured.

Milestones and Deadlines: Since the projects the company has been involved in were fully developed before being marketed, they did not have a deadline in the normal sense of the word so the problems involved with having a deadline to work against was non existent. They develop a basic plan for the project, but there are no real deadlines. They did not promise any external customer any release dates, so they do not find a more thorough plan necessary.

Use of estimates: Size in length or time is not estimated.

Conclusion: Projects are not divided into activities, nor is any project plan established. Progress is not measured. This is absence of planning, which the hypothesis suggests, and the score is therefore 5.

Planning 5

Management

This hypothesis does not apply for this type of organizations, since the two employees are also developing their products.

Management N/A

Selecting technical solutions

Vendor lock-in: Although they did not explicitly state it, we believe that a certain major company is providing all their development tools. For example, when asked how they preserve experiences they said: “Microsoft is improving the standard both on development tools and web browsers.” They also used Microsoft Access as their database “...because we believe that it is fastest”. When talking about a project that they have made, it was a rework of an older program: “We decided to develop an alternative to this application with modern tools such as Access and Visual Basic.”

Evaluating technologies: They do not perform any evaluation of the tool that they select to use, the only requirement is that of experience. No validation of the technologies used was performed. They used Visual Basic as the programming language “...because we are most experienced with that”.

Golden hammer: The company has been involved in too few projects for us to make a conclusion about this.

New technologies evaluated by management only: Not applicable.

33

Conclusion: Even though the number of projects that they company has been involved in are too few to draw any real conclusions. We still believe that our hypothesis is true and give it a 5, since the situation complies with that described by the hypothesis.

Selecting technical solutions 5

Configuration management and version control

CM Tool: The organization does not use any CM tool.

Working version control: There is no CM tool, which means that it is very difficult to revert to older versions of existing files.

Delivery control: The fact that their products have only been delivered to single customers reduces the problem of distributing updates so less version control might be necessary.

File control: The company consists of two persons, and they had only one computer to work with. So it is impossible to have to people working on the same file simultaneously.

Conclusion: The organization is not using any CM tool, which the hypothesis suggests. There is no control for managing delivered products, few products has been delivered so this might be a non-issue. The situation is like that discussed in the hypothesis, even though the implications are not so difficult.

Configuration management and version control

5

Testing

Testing strategies: No formal tests are performed; the only type of test that seems to be performed is rudimentary input validation.

Test cases documentation: There are no test plans, and the result of a test case is not known in advance of the test. The tests are performed by hand and no regression testing is performed. Bugs, defects or deviations are not classified or documented.

Binary testing focus: Only binaries are tested.

Root cause analysis: No root cause analysis is done.

Conclusion: The organization is not preparing any formal test plans. The result of the tests executed is not known in advance, and is based solely on the binaries. Defects discovered are not subject to root cause analysis, which could prevent them from reoccurring. This is the situation that the hypothesis suggests so the score is 5.

Testing 5

Requirements

Requirement specification: One of the projects they have done is a conversion of an older program into a new Windows-based modern application. This made the requirements handling rather easy, since they had an original product to base their new work on. They used this older program as a base, and added their own experience of the domain with modern development tools.

Feature creep: Since they had no deadlines and no specified requirements, there was no risk for feature creep.

34

Requirement verification: A customer, with newer requirements, was never involved. Since the applications were developed with no customer involved and no real deadline, they had much freedom in doing their work. It was totally up to them selves to decide. They are developing prototypes as they way of communicating requirements with the customer.

Conclusion: They developed the product for company where one of them had previously worked and had thus deep knowledge of the domain. Since they know their domain very well, and can use this knowledge to build user-friendly products. Our hypothesis about requirements handling does not apply and get a 1.

Requirements 1

35

6.3 Company IIThis interview was conducted on site, in a meeting room. Each person was interviewed individually, beginning with Person I.

Company II is a software developing organization that is actively working with introducing a quality system. The system is not currently a part of each software developers everyday work, which means that the hypothesis regarding configuration management for example seem to hold.

6.3.1 Presentation of the companyCompany II is a company with very high domain knowledge in a certain technical domain. When they started, 15 years ago, they did not do any software development at all, but over the years this has increased and now a larger portion of the company is involved in software development. They have customers nation-wide and in more than 30 countries around the world. The general education level is master of engineer in their domain. Some of the employees are systems engineer, but they are a minority.

The company has been working together with a large, international consultant company, and has learnt a lot about project management from them.

6.3.2 Presentation of the interviewedPerson I

Person I have a Technical Ph.D. and has worked in the company for 3 years. She is working as a technical consultant and project manager. She does not write any code, but do large-scale system design on some products. Customer contact is a large part of her work.

Person II

Person II is a master of engineer in a domain closely related to the companies. She is currently working mostly as a software developer.

6.3.3 The hypothesisView of software development

Quality System. There is a company quality manual that describes project management over larger software projects. This manual seems to focus on project management and customer relation’s aspects, but there seems to be less focus on the developers. The developers are not even using the manual.

View of ISO 9000. “ISO 9000 contains a lot that our organization would benefit from. But the risk is that it takes longer time, and it can not take extra days to write a document that the customer has to pay for.” There seems to be some doubts about the usage of ISO 9000 and that it might lead to too much paperwork. Paperwork in itself is not a bad thing though, in the context of avoiding misunderstandings is the strategy to document. “We try to get it on paper. A lot of paper work, but you have to do it. We do status reports that the customer has to sign. That is our goal.” The perception about ISO 9000 is not based on studies or actual experience.

Experience preservation. Formal conclusion reports are produced after longer projects, which is another sign of their project management maturity. The developers do not seem to be involved in the process, so the programming work are still not as developed as the project management processes.

36

Brooks Law. When deadlines can not be held as agreed upon the solution is to work more. “Then you work overtime and you get really stressed up.”

Conclusion. There was a company quality manual, it was not based on ISO 9000, but that is not a bad thing. This manual was mainly used by management, the development teams had little contact with it. Our hypothesis is regarding the entire software developing organization so there was some compliance with it. The fact that conclusion reports were produced is contrary to the hypothesis. Finally was the hypothesis more or less wrong, due to some compliance did it make the score 2.

View of software development 2

Planning

Divide and conquer. TODO-lists for short term planning of what to do during the day, and in shorter projects. Projects are not broken down into small activities, at least not smaller projects. There were large differences between large and small projects, where larger projects were more formal.

Progress reporting. There is no formal way of measuring progress in a project, yet it is important to be able to predict when the current plan is going wrong. The reason for the overtime is: “Lack of resources. Even though we try to plan, you do not always have the resources you need. Perhaps we do not have anybody to put in to work.”

Deadlines and milestones. There are deadlines and additional milestones, these are planned and the project plan is revised when it is necessary.

Use of estimates. There are planning strategies but they are often difficult to implement, the main reason is that estimation is difficult. “You think that everything will go much smoother than it does.”

Conclusion. Project plans are established and these are updated when necessary. The project is not divided into activities in the sense; TODO lists perform short term planning. This is consistent with the hypothesis. The fact that it is necessary to work a lot of overtime during some periods imply weaknesses in project planning. Estimations are not based on metrics or other formal method. It is definitely not full compliance with the hypothesis, but some indications are correct, so an average score is reasonable.

Planning 3

Management

Project managers. There are project managers that are responsible for the larger projects. The requirements on the product are negotiated with the customer. The developers are taking active part in the project.

Management part of development. The CEO or other management person performs no extra control over the project. This weakens our hypothesis, that there is one person on the company that decides everything.

Adding requirements. There have been problems to reach project conclusions, the customer was never satisfied, more and more issues had to be corrected. This was not the result of higher management intervention though.

Developers domain knowledge. Company II started as a company within a specific domain, and has gradually moved over into becoming a software developing organization. This means that the

37

developers have a very high degree of knowledge in the domain, though there is less software engineering expertise.

Conclusion. The organization is clearly not governed by a despotic manager, which the hypothesis states. No compliance with any of the indications was found, so a very low score is natural.

Management 1

Selecting technical solutions

Vendor lock-in. The solutions used in projects vary, languages etc. depend on the project at hand. “We change for example language and tools between projects.” The solutions are not limited to previous experiences. There is also no risk that a single solution is applied to any problem.

Evaluating technologies. Technologies are not evaluated and verified that they will work in practice and solve the problems stated by the requirements. “You do not. It can be too slow. You do not know that.”

Each employee have a personal development plan, when there is nothing scheduled then the time is spent on learning for instance new languages. This ensures that new knowledge is obtained by the organization.

Golden hammer. Some solutions are limited due to lack of alternatives, there is only one main vendor in the domain and it is necessary to use their products.

New technologies evaluated by management only. Each project is responsible for selecting suitable solutions. This means that management is not involved the way our hypothesis suggests.

Conclusion. There are little indications that our hypothesis is correct here. Solutions vary from project to project, depending on the requirements at hand. However, many projects require products from a single large vendor. New technical solutions that have not previously been employed are not evaluated in advance, which the hypothesis predicts. There is thus little correct indications so the score is as low as 2.

Selecting technical solutions 2

Configuration management and version control

CM tool. Person I’s view: “Most often, it is one-person projects, and then there are no problems. If it is bigger ones, then it is divided so that you do not share components between persons. Every person is responsible for his or her piece. You do not share files.” There is no CM-tool in use, but since files are never shared this was not a problem. Person II answer differs: “In the beginning that was not a problem, but then problems occurred. [...] But it’s important, it’s so very important. It’s dangerous to change things at the same time.”

Working version control. Reverting to older versions of documentation and source code is possible by retrieving backups. The developer usually saves temporary versions locally “You may have an older copy left, but it is not always that you do.”

Delivery control. The state of the product before delivery is saved, with the purpose of keeping track of what version is stored where. It does happen though that changes are made to this state. “But if changes are coming in after the delivery, we are a bit sloppy with documenting it.”

File control. There is no official way of ensuring atomic usage of source code, or other documents.

38

Conclusion. Based upon the words by the developer interviewed we conclude that many of the indications are correct. A CM plan and tool is desired by the organization. Older versions of source code are retrieved from backups and there is no control on the file level. It is not always possible to reproduce customer’s versions since late changes are not always documented. This indicates full compliance with the hypothesis.

Configuration management and version control

5

Testing

Testing strategies. There are three types of tests involved (1) simple tests, (2) unit test and (3) integration testing. Simple tests are performed by the programmer during the development of the product.

Test cases documentation. These tests do not follow any formal plan, nor are the results of the test known in advance. A person other than the programmer performs unit tests; these are also undocumented with unknown results. The final type of testing, integration testing is more formal with prepared test procedures but still with unknown results.

Binary testing focus. The testing strategies employed test the produced binaries alone.

Validation. Expected results from tests are not documented.

Root cause analysis. Little regression testing is performed to ensure that old defects do not reoccur, or that patches do not produce more defects. Defects do not result in a root cause analysis.

Conclusion. There was a set of different testing strategies; this is directly inconsistent with the hypothesis. Most of the tests are undocumented with unknown results, though. No inspections or test other than binaries are performed and subsequent versions are not subject to regression tests. We still believe that the score is average since so many different testing strategies are performed.

Testing 3

Requirements

Requirements specification. The product requirements are documented. “A lot of paper work, but you have to do it.” There have been situations where changed requirements have not been documented. The result of such changing requirements has been feature-creep.

Feature creep. There has been situations like “If the customer has changed his mind, and we say “yes, yes”, and then we have not documented it and it becomes expensive.”

Requirements verification. The requirements are communicated with the customer with the help of prototypes, milestones etc. The use of evolutionary delivery decreases problems related to requirements handling.

Conclusion. Requirement specifications are rigorously documented, which is inconsistent with the hypothesis. There have however been indications of feature creep. The strategy to avoid misunderstandings is to develop increments or prototypes so the customer can observe development progress. There are risks associated with evolutionary prototyping [CONNELL+] development or evolutionary delivery [GILB]. One of the largest is that the next deadline becomes too important, and you forget about the long-term lifetime of the software that you are developing [MCCONNELL]. To have a strong design, for example, becomes secondary in comparison to have an executable binary on the next delivery. The score is low due to requirement specification and incremental delivery.

39

Requirements 2

40

6.4 Company IIIThe interview was performed at the office in a meeting room. The two persons were interviewed simultaneously.

6.4.1 Presentation of the companyThe organization is developing products and is consulting on EDI (electronic data interchange) solutions. The primary platform is the IBM AS/400.

6.4.2 Presentation of the interviewedPerson I

Is working, as the manager for the consult department, is also responsible for two internal products. He has a shorter academic education, but there was not much computer education available at the time. There have naturally been many courses within the organization as well.

Person II

Works as an on site consultant is also responsible for one product. He is usually adapting existing solutions, but may spend time developing products on site. He has one year of academic education in designing computers and software, also internal education within the organization. The main knowledge has been obtained according to learn-by-doing.

6.4.3 The hypothesisView of software development

Quality System: The interviewed has a customer oriented quality perception that is also partially existential; “It is an experience”. They mean that quality is personal and what some people perceive as quality, may not be quality for some one else. Project planning is perceived to be one of the major obstacles when it comes to quality in software development and testing is the second major problem factor. Later in the interview it becomes apparent that they have no type of formal testing.

The company is clearly interested in improvement. “We are in a process where we are getting used to the idea of improving quality if that process leads to ISO 9000 or something else we do not know.” By improvement they mean to document more of the way they work, “It might be necessary to become more formal and spend more time on documentation.” They also state that: “The larger the organization becomes the more important it is to document what one is doing.”

The company wants to introduce some quality framework but do not know which are available. They feel that they should improve themselves but they have not taken any serious steps in that direction as of yet. The current projects and work-tasks leave little time for focusing on documenting processes etc.

View of ISO 9000: The general perception of ISO 9000 is that it is too bureaucratic, “...many of our customers are certified. The fact that they are certified leads to an enormous paperwork each time there is a deviation report. For small companies this leads to very much administrative work the times that I have seen it.” They also seem a little skeptical about ISO 9000 since it does not help them develop better products.

Experience preservation: There is currently little possibility to preserve experiences from one project to the next project, and this has caused some problems for the company. Not preserving

41

these experiences indicates that they are more focused upon the current project, than long-term success, which verifies our hypothesis. Only the problems of today are important. There seem to be some interest in improving this situation though. One solution is to keep a log of events occurring within each project.

Brooks Law: Since they have no deadlines, they have no projects that are late for deadlines and Brooks’ law is not applicable.

Conclusion: As all the others, the company is interested in improvement, but does not have a plan for implementing it. They believe that ISO 9000 is too bureaucratic and does not believe that it will help them produce better products. They do not preserve experience, which has caused some problems in the past.

All in all, this strengthens our hypothesis of short-term view of software development, the current project is more important than long term improvement.

View of software development 4

Planning

Divide and conquer: Short term planning is based upon TODO lists that are managed individually by each person. There is no central planning but group meetings ensure that workload is equally divided. Longer scope planning is based upon contracts or projects; there are no shorter activities.

Progress reports: They do not have any deadlines, but lack of deadlines does implicate that it is not necessary to report progress, “one feels where one is.”

Milestones and Deadlines: The most interesting aspect of project planning is that they did not establish any deadlines. They rather agreed upon “what time in the year that we will be finished.” The reason that they worked this way was that they are primarily selling consulting and resources, not developing their own software. Since they prioritize paying customers they believe that it is impossible to set any deadlines for the internal projects. Without deadlines the company risks feature-creep and becoming late. “The time plan for our internal projects is not very strict, we use it as a general aim. There is thus no ‘deadlines’ when we have to be finished.”

Use of estimates: No estimates are done.

Conclusion: The company does consulting services and on-sight development, which means that the internal project’s suffered in planning. Project planning was perceived by the company to be one of the major obstacles when it comes to quality in software development. If they do not know where they want to go, then it does not matter what path they chooses.

This strengthens our hypothesis and we give it a 4.

Planning 4

Management

Project managers: The company had product responsible that in a way acted as project managers.

Management part of development: It is a collective responsibility to select projects. There do not seem to be any despot on the top that decides what to do. This is probably because of the high technical competence in the organization.

42

Adding requirements: New requirements were added with the influence of the developers.

Developer’s domain knowledge: The developer’s domain knowledge was very high.

Conclusion: The developers, as well as the managers, have very good knowledge of the end users environments. It is a collective responsibility to select projects. The developers take part when adding new requirements. Adding it all together makes our hypothesis wrong.

Management 1

Selects technical solutions

Vendor lock-in: All software is developed for the AS/400 server from IBM, which is vendor lock-in. This is the situation described in the hypothesis but it is on the other hand one of the business ideas of the organization.

Evaluating technologies: “There are strategic meetings within the organization that select major technical solutions.” They discussed an example of such strategic decision and the arguments for that technology was that “It is very well documented” and “It is a big known provider” - it is good because other people use it. Some type of internal, thorough evaluation of the product before the organization decides to implement all of their software with it, is not performed.

Golden hammer: “When we have selected to tool to be used for development then all must projects adhere to that.” To select one solution that all projects must use is clearly similar to the golden hammer [BROWN+], where a single solution is applied to any problem, and strengthens our hypothesis.

New technologies evaluated by management only: This was not true in this organization.

Conclusion: All software is developed for specific server software, which is vendor lock-in. They have strategic meetings, deciding on new technologies, but does not seem to make any thorough evaluation. Management does not decide on new technologies, but the facts that they are trusting brand or lack of alternatives strengthens our hypothesis.

Selects technical solutions 4

Configuration management and version control

CM Tool: No CM tool is used. The reason that they have not evaluated any version control system is “there is not any versioning or generations management tool automatically available for AS/400.” This does not change our hypothesis. The reason for not using a configuration management tool does not matter to our hypothesis.

Working version control: Their strategy for reproducing older versions of their products is based upon a backup system. The backup system goes three weeks back, but anything older than that is difficult to reproduce. The fact that they rely on backup copies for older versions of files strengthens our hypothesis.

Delivery control: They do not keep track of what version is installed where, and they do not preserve older versions of their products, only the latest version is available at the office. Recently a feature has been added that allows the customer to see which version they have installed. This leads us to believe that they have had problems with this earlier. Corrections to detected defects are distributed in so called program temporary fixes (PTF). These are distributed to the customers that have reported the defect.

43

File control: They have not had any problems with overwriting each other’s files, since they are working in separate files. There is no way of ensuring that people actually are working in separate files other than oral communication.

Conclusion: The company does not use any CM tool. They do not have any overall strategy for CM handling and does not keep control of where different versions of their software were installed. This has made them adding a new feature to their programs that lets the customer see which version they are running, probably because they have had that kind of problems before. There is no formal way of ensuring that files are not corrupted as a consequence of overwriting each others work.

All of our indications were right, giving the hypothesis a 5.

Configuration management and version control

5

Testing

Testing strategies: The main concern about testing is that a non-technical person should perform it. This person does not follow any formal test plan; he tests based on previous experience. There are hence little formal test plans, or test protocols.

Test cases documentation: The result of a test is not known in advance of the test. Some type of regression testing is performed on their PTF before they are delivered to their customers. Little or no tests are automated.

Binary testing focus: Only binaries are tested.

Root cause analysis: They do not perform any root cause analysis of the defects that they find, so they risk repeating the problems over and over. The defects are not documented and used in future tests, so there is a risk that they will return. There have been problems with defect corrections that lead to new defects.

Conclusion: There are no test specifications, no peer reviews etc. The simplest form of monkey testing is performed, they only condition is that the person who is testing should be a non-technical person. Adding it all together, our hypothesis is verified and strengthened, which gives a high score.

Testing 5

Requirements

Requirement specification: The company is mainly maintaining their product, and the requirement analysis made is based upon customer criticisms on previous versions. The product has matured over the years, and there are industry standards that must be followed, so they do not find the requirement analysis to be any problem.

Feature creep: They are aware of the risk with feature creep but do not associate it with requirement analysis.

The facts that they have existing products which they continuously add PTF:s to might indicate that feature creep has occurred. They do not however, make new products or are involved in new projects. This means that our hypothesis indication of Feature creep does not really apply.

44

Requirement verification: The product responsible handles requirement verification. “As responsible for a product you have to be able to view things from the customer perspective.”

Conclusion: Their deep domain knowledge makes it easier for them to understand customer requirements and suggestion. There are no formal processes for managing requirements, but we do not find our hypothesis strengthened due to their domain knowledge.

Requirements 2

45

6.5 Company IV

6.5.1 Presentation of the companyDevelops interactive media in the form of web services and CD-ROM. The customers are average to large sized companies. Examples of products are shockwave games, presentation support, sales support and education. The company consists of two persons, but they hire resources during larger projects. Large projects employ up to four people.

Company IV strong side seems to be their way of communicating with the customer. They want the customer to feel at home with what they do. This makes the customer feel safe about their investment.

They have not been in contact with software engineering, and the skills they have are from experience and ideas from other companies.

6.5.2 Presentation of the interviewedPerson I

Is working with project management, but is also responsible for the graphics and is a 3D specialist. Has unfinished studies at LTH and some miscellaneous jobs behind him. Has largely been learnt by doing.

Person II

Is responsible for programming. Do sometime work as the project manager. In his academic background there is a degree in system engineering and a short course in multimedia.

6.5.3 The hypothesisView of software development

Quality System: Since the company is developing web services and multimedia applications, they are somewhat separated from the other companies, since much of the work is done by graphic tools, and WYSIWYG (What You See Is What You Get) editors. This decreases the complexity generally found in code-based products, and the importance of writing maintainable code.

They do not have a complete quality system, but they do follow simple processes concerning for example customer contacts. They do not write conclusions of projects, although they realize that as they grow, this will become increasingly important.

Experience preservation: When asked if they document their experiences they say: “We should really document more.” They continue: “If we grow to a certain point of size such methods will become important, but there is currently no such method. “

View of ISO 9000: They think that ISO 9000“...feels too big and nasty”. This indicates that their view of ISO 9000 is based on prejudices.

Brooks Law: This is not applicable since their projects are so small. They can not afford to add more people. If the project is too big, they turn it down.

Conclusion: Most of the companies we have interviewed states that they will get better at documenting what they do, and Company IV is no exception. It is impossible for us to know if this is a sincere desire to improve their processes or something they know that we want to hear.

46

They follow some simple processes, but do not have a complete quality model and they do not document their experiences. The general impression we got from the company, added together with their answer’s give our hypothesis a 3 when applied to Company IV.

View of software development 3

Planning

Divide and conquer: When asked how they make their project planning they answer: “When we are planning a project, we divide it into phases. Each phase is then estimated for length. The sum of all phases indicates how long the entire project is. A phase can be a few days .” This means that they divide projects into smaller parts and do estimates on the parts.

Since they do not have any progress reporting, one can assume that the project plan is not updated.

Progress reporting: They do not have any kind of progress reporting – “It is like cooking - you see what happens”.

Deadlines and milestones: Deadlines and milestones are used.

Use of estimates: They use estimates, so the indication was wrong.

Conclusion: In a small company such as Company IV, long-sighted planning is very difficult. Still, Company IV seems to have an organized way of working. They follow simple processes from PROMISE (Producers Of Multimedia In SwEden) when contacting customers, and since they teach classes and hold lectures, they must follow some sort of scheme.

The company has some sort of a planning strategy. Three of our indications in our hypothesis were wrong, but since they are only two people some of the indications might not apply. Adding it together gives our hypothesis a 2.

Planning 2

Management

This hypothesis does not apply for this type of organization. The company is too small.

Management N/A

Selecting technical solutions

Vendor lock-in: When working with multimedia and web based development, new tools and techniques are constantly emerging. It is also important that you use the latest tools in order to be able to do the most advanced effects and implement the latest features. This means that you are in some way stuck to the tool vendor that you are most familiar with.

Evaluating technologies: When asked how to choose technology, they answer: “We choose the technology that we know best. It takes too long to learn a new programming language for each project so we stick to the technology that we are competent in using.” This would strengthen our hypothesis, but in the next sentence they also add: “If we do not know the required technology we are forced to turn down the customer.” They can not afford to evaluate alternatives, but they evaluate the ones they are familiar with. Still, this is a verification of our hypothesis.

When it comes to whether or not the product will work in practice, they regulate that in the requirements. This means they are at least aware of the problem.

47

Golden hammer: Company IV seems to know their limitations and does not take jobs that are not inside their domain. They also know what their tools are capable of, and turn down jobs that are not possible to implement with the technologies used. This implies that our hypothesis indication of Golden Hammer is not correct. They use a hammer, but only to hammer nails.

New technologies evaluated by management only: Not applicable due to the size of the organization.

Conclusion: Being a multimedia creator, Company IV is very closely dependent on the tools on the market. It is crucial to know the right tool and to use it optimally. Company IV seems to be stuck to the technologies they are using, but still they know their limitations. This means that our hypothesis is almost correct and is given a 4.

Selecting technical solutions 4

Configuration management and version control

CM Tool: The Company does not use a CM tool. They have some sort of version control but it is manually managed. They do not work in a networked environment, which could possibly speak against investing in a tool, but a configuration management tool brings with it a lot of useful side effects. Also, the network can be seen as a part of the CM tool itself. Today the two persons in the company have different version control strategies. They both rely on backup copies for older versions of source files.

Working version control: Person II says: “There has never been any version management in that way, and there has never been any problem with version management. There is no need of such control. There have however been problems with your own versions, so some formalization might be valuable.” This indicates that they might improve their configuration management process. They do not currently have any version handling process.

Delivery control: Each product is only delivered to one customer, so this is not applicable.

File control: Since they do not work in a networked environment, two persons can not access the same file simultaneously but that does not mean that the same document can not end up on both of their computers.

Conclusion: Even though the company seems to have a good control of the files and versions of products, they lack a common CM strategy and do not use a tool. It is up to the employee to keep track of changes or different versions of files. They only deliver to one customer, so they have no delivery plans. This gives our hypothesis a 4.

Configuration management and version control

4

Testing

Testing strategies: When making a CD-ROM product, thorough testing is of the essence. Once printed, the CD-ROM is irreversible. This has lead to a testing process that seems to capture most of the errors in the end product. It is not, however, documented or formalized.

Test cases documentation: More than one testing strategy was followed, but only binaries were tested. The result of a test is known before the test, during the user testing. Test protocols were used, and reapplied when errors are found.

Binary testing focus: Only binaries are tested. No testing or reviewing of source code or documents is done.

48

Root cause analysis: Found defects are not subjected to root cause analysis, which means that there is no way of preventing them from reoccurring.

Conclusion: The company follows a testing strategy that appears to work very well, but it is not documented. Only binaries are tested. No root cause analysis is done but more than one testing strategy is followed, and they appears to be working well. This results in a 2 to our hypothesis.

Testing 2

Requirements

Requirements specification: We asked what the difficulties involved in software development related to quality was, and Person II answered: “It is to understand what the customer wants, or rephrasing the customer needs into a requirement specification. This is one of our strong sides; we are good at explaining to the customer what the customer really wants.” Requirements are thoroughly documented and no work is done until consensus has been achieved.

Feature creep: New requirements are incorporated in the requirement specification, which prevents feature creep.

Requirement verification: They had a complete view on the software lifecycle, when it comes to testing and verifying requirements. They were for example always sure that new requirements are verified before beginning to work on them. But still the process was not formalized in any way. It was totally based on the experience of Person I and II.

Conclusion: Company IV knows a lot about how to do things wrong, because they have been working as teachers, teaching multimedia. They have seen their students do all the classic mistakes and they have also learnt how to speak to people that does not know as much about technology as they do.

Our hypothesis indications were wrong at most parts. Even though they do not have a formalized requirement’s handling, they follow informal processes that work well. This gives our hypothesis about poor requirement’s management a 2.

Requirements 2

49

6.6 Company VThe interview took place in a meeting room at the companies office. The two representatives of the company were interviewed at the same time.

Company V was the last company we interviewed and the only that had come into any contact with ISO 9000. They were very pleased with it, and showed us their quality manual, which was easy to read, and very small (a couple of pages). They saw no problem with “too much documentation” or “too much bureaucracy” that the other companies were concerned about.

6.6.1 Presentation of the companyCompany V developed their own products and offered their customer extra resources. There were ten employees in Company V at the time of the interview. The typical customers were smaller producing companies in Scandinavia. The main product that Company V developed was a production-planning tool that can be connected to the customers business systems. They also had customers in trotting business.

6.6.2 Presentation of the interviewedPerson I

Person I was the CEO of the company. He had been working there from the start and have been studying economy, but has no formal degree.

Person II

Person II is quality responsible, and a programmer. He has been working at Company V for two years and has no academic degree, although he did study computer science for some time.

6.6.3 The hypothesisView of software development

Quality System. The organization is currently in the process of developing a quality system, with the aim of becoming ISO 9000 certified.

View of ISO 9000. The view of ISO 9000 is based upon facts and hand on experience. The process of becoming certified is actively supported by external expertise.

Experience preservation. The view is that documenting a process is not enough; it must be improved upon with time according to the experiences learnt. “We must continuously improve our processes.”

Brooks Law. When late for a deadline is nights and weekends utilized, rather than removing requirements. The main strategy is to call the customer and work something out, though.

Conclusion. The hypothesis is wrong on most points here. There is a quality system actually being developed, and that system is based on ISO 9000. The organization is taking courses led by authorized ISO 9000 personnel and the view of ISO 9000 is thus not based on prejudices. The organization finds it necessary to preserve experiences in order to improve upon the processes. This makes the hypothesis score very low.

View of software development 1

50

Planning

Divide and conquer. The project-plans are prepared and these are based upon earlier experience, and in-house expertise. Project-planning and cost budgeting are some of the most important aspects of software development. These plans are partially broken down into smaller sub-processes, or activities. Each sub-process is a rather large part of the entire project.

Progress reporting. There is no formal way of measuring progress on the work tasks. “One learns from experience how long time that each work task should take.”

Deadlines and milestones. Deadlines and milestones schedule the project. These are defined in the project-plans and are necessary to structure the work.

Revision of project plan. The plan is revised when necessary, no plan can last forever and is a living document.

Use of estimates. The plans and deadlines are not based upon previous measured metrics or formal method. The method used is based solely on earlier experiences.

Conclusion. The hypothesis is mostly wrong in these aspects. Formal project-plans are produced, and are considered as important aspects of a project as a whole. These plans are broken down into sub-processes, deadlines and milestones. The sub-processes are large parts of a project are not activities, as we perceive them. There is no way of reporting progress except verbally in meetings. The project-plan is updated throughout the project, based on informal progress reports. The initial plan is based on estimates from earlier projects, no metrics etc. are collected. Project-plans are important and used, the lack of progress reporting and formal estimations makes the score 2 though.

Planning 2

Management

Project managers. There are project managers who are responsible for ensuring the project schedule, and allocates resources for work tasks.

Management part of development. The management is not directly in control over the requirement of the software products. There are formal methods for controlling projects.

Adding requirements. Small changes are accepted without any formal discussion, but if a change would imply many hours of work than this must be formally documented and planned for.

Developers domain knowledge. The developers have a varying background not necessarily from the domain that the organization is developing products within.

Conclusion. The hypothesis is clearly wrong in all indications. There are project managers and no senior management person dictates the requirements in the project. The developers have a varying background. The formal quality structures are inconsistent with our hypothesis. All of this implies that the score should become very low.

Management 1

Selecting technical solutions

Vendor lock-in. “We usually select Microsoft products, since they can be trusted to survive the next few years. Microsoft is also the only company that develops interesting technologies.”

51

Technologies are selected based upon what vendor have developed them. Previous experience is also very important when selecting solutions in projects.

Evaluating technologies. There is no way of knowing if the selected technology will work in practice, since they are not evaluated and no proofs of concept are developed. Selecting a new technology is taking a ‘chance’.

Golden hammer. It is important to follow the technological evolution to keep up with current events. It is however each developers’ responsibility to study new technologies in their spare time.

New technologies evaluated by management only. Each project is in control over the selection of solution and no higher management decides what technologies are used.

Conclusion. Stating that Microsoft is the only company that develops new and interesting technologies seems as a high conformance with vendor lock-in. New technologies are accepted without formal evaluations, they take chances in stead. Expecting that the personnel study new technologies seems to risk that they do not, and thus sticking to old technologies. It is however the projects responsibility to select techniques, not management. This makes the score high, but not full.

Selecting technical solutions 4

Configuration management and version control

CM tool. A CM tool is used, the tool is Visual Source Safe delivered by Microsoft.

Working version control. The tool makes it unnecessary to rely on backups for version control.

Delivery control. The products are delivered to unique customers, this makes it relative easy to keep track over what version that is installed where. The possibility to make labels in the tool also makes it easy to retrieve older versions of entire products.

File control. The tool makes it easy to ensure unique file access by file-locks.

Conclusion. This company was the only one interviewed that used a CM tool. This tool is greatly appreciated. They deliver their software to unique customers, so the version control problem is non existent. All of the hypothesis indications are wrong and the score can only be 1.

Configuration management and version control

1

Testing

Testing strategies. There is a set of different testing types: functionality testing, flow testing, beta testing and provocative testing.

Test cases documentation. Only beta testing is formally documented, with expected result for each test case. The other testing strategies are performed by the person responsible for the code, or his/hers peers.

Binary testing focus. Only the binaries are tested. Most test strategies are variations of input testing.

Root cause analysis. Found defects do not result in root cause analysis, and can hence return in a form or another.

52

Conclusion. There were different types of strategies for testing and this is contrary to our hypothesis. They appeared to have very good testing strategies, which weakens out hypothesis. However, the testing was based on the binaries only, which conforms to the hypothesis. Finally, beta tests are documented with expected results and thus the score is rather low.

Testing 2

Requirements

Requirements specification. Requirements are formally documented and the customer signs all specifications. Prototypes are used as the basis for communication with the customer, and to ensure that misunderstandings are avoided.

Feature creep. Changing requirements are documented as far as possible; the customer must also sign all changes. Smaller changes may be implemented without extra paperwork. The requirements changes but in an orderly manner.

Requirements verification. The requirements are documented in a requirements specification, these are discussed with the customer using prototypes and other means of communication.

Conclusion. Requirements are documented and verified with the customer. Prototyping is used to ensure that the requirements are correct. Changing requirements are documented. None of the indications for this hypothesis were right, the final score was thus 1.

Requirements 1

53

7 Analysis conclusion

This chapter contains the conclusions that we draw from the analysis of the interviews. Table 3 contains the resulting averages from the interviews. The management hypothesis could not be performed on two organizations; the reason to this was that those company where too small to have any apparent management.

The grades in Table 3 are from 1 to 5 and the corresponding hypothesis is considered to be true if the average value is more than or equal to 3.0. The grades are based on how many of the indications in the hypotheses that the company showed signs of, so 3.0 would mean 60% compliance with the hypothesis. For the purpose of this paper, that is a sufficient indication of the hypothesis. Two hypotheses are thus not ‘true’ and those are Management and Requirements. The hypotheses that are the most ‘true’ are the ones regarding Configuration Management and Selecting Technical Solution.

Managem

ent

Requirem

ents

View

of softw

are dev.

Planning

Testing

Selecting technical solution

CM

and version control

Cmp. I N/A 1 5 5 5 5 4

Cmp. II 1 2 2 3 3 2 5

Cmp. III 1 2 4 4 5 4 5

Cmp. IV N/A 2 3 2 3 4 4

Cmp. V 1 1 1 2 2 4 1

Average

Table 3, the averages of the companies in the survey

7.1 The hypothesisEach hypothesis is discussed in further detail in this chapter. The grade is motivated and examples and quotes from the interviews are presented.

7.1.1 View of software developmentWe have shown that these companies have a too shortsighted view of software development. The current project or activity is more important than long term improvement. Introducing quality systems into an organization takes time and effort and these organizations do not believe that they have the time or the resources to do so. It is common that the companies have an internal discussion regarding quality improvement, and they feel that it is necessary to document more, but little actual activity is performed in order to achieve this.

There are examples of companies that have successfully introduced, or are in the process of introducing, quality systems into their organizations. None of these organizations believed that such system consumed too much time or effort. We have not found that ISO 9000 is not suited as a quality system for small organizations. Some believed that ISO 9000 was too bureaucratic and time consuming but their views was built on believes and no actual experience. The company that

54

was in the process of becoming certified according to ISO 9000 meant that there is nothing wrong with ISO 9000, since you are yourself responsible for developing processes.

We asked the companies what quality, when related to software, was for them. The most common answers during the interviews were some form of “customer satisfaction” or “well-tested software”. Three of the companies said that software quality is that the program does not crash. We did not find any company with a stated quality definition. These quality definitions are from the organization point of view. More accurately, from the person that received the question point of view. The definitions are seldom concerned with quality of what.

There are dangers in focusing only on these two. It is, for example, not always clear whom the customer is, the one paying for the product, or the one using it. For how long should the customer be satisfied, one hour or ten years? Such definitions easily neglect the long-term perspective of software development. The products that were developed 20 years ago, and today suffer from the Y2K problem, are they of low quality? Were they not enough tested when they were released? To achieve customer satisfaction it is necessary to know what the customer expects. The budget or time schedule may not be as important as fulfillment of requirements for some customers. Such customer may very well be satisfied with the result, even if it was delivered late, or cost more than estimated.

One of the companies we interviewed gave us the following answer: “It is to meet the customer’s demand. [...] Quality is the right thing, on time and on budget. There is no good or bad quality, you have to be at the same level as the customer.” The person who gave us this answer was a project manager at the company. The company had been discussing quality and how to increase their quality for a longer period of time. It is interesting to note that the developer at the same company gave us the following reply on the same question: “[...] it is well tested. It never becomes bug-free, but that it is tested is very important.” This clearly shows that different persons, even at the same company, have different views on quality.

Another company answered “Quality when it comes to software development is that everyone is working according to the same principles and that everything is documented.” This company was on the way of getting ISO 9000 certified, and this answer was the only one where testing or customer satisfaction was not even mentioned. If this indicates process maturity or too much influence from the ongoing ISO 9000-certification process is impossible to say, but it does show yet another perspective of quality.

7.1.2 PlanningMany software projects are rather roughly planned. There are examples of companies that do not issue deadlines or milestones in their projects. Most companies do however perform high level planning, but we have not found any company that breaks the larger scale plan down into smaller activities (such as reviewing a document, meetings, finishing a certain module of software, testing a certain module of software etc.). The short-term schedule is instead managed ad-hoc using TODO lists. There are no examples of companies who are measuring progress in their projects. Some uses their large-scale project plan to identify progress in their projects.

On the single developer level progress is based upon experience, which makes it difficult for new employees with little experience to estimate progress. Large project happens one day at a time, one hour at a time [BROOKS], it is thus necessary to keep track of the smaller activities. The estimations for larger activities are also based upon experiences, which might be less a problem since there are probably experienced persons available for that, but a more formal estimation process could potentially increase estimations correctness.

7.1.3 Management Our personal experience with small software developing organizations was that the management did pay little attention to the developers. Management controlled requirement changes and project

55

deadlines without discussing this with the people responsible for developing the products. Our study has shown that this is not generally the case. In the larger companies there were project managers who are responsible for the project and communication with higher management. In smaller organization all of the employees were part of the development process.

7.1.4 Selecting technical solutionsThe solutions that are used to implement the project requirement are selected with an ad-hoc process, and most often came from the same vendor (3 of the companies). None of the companies was able to know that the selected solution would actually work in practice. The selection of technology does thus seem like chancing. The selection is often based upon previous experience, or by trusting some vendor.

All companies agree that technology changes very fast and that it is important to be aware of new emerging technologies. In that light it does seem dangerous to trust personal experience and a specific vendor. There is one example of a company that made a strategic decision regarding what technology their entire project was to be developed with.

Selecting technologies to implement requirements is an important part of any project, if a technology is chosen that is not capable of solving the problems and hand are must work wasted. It does seem sensible that selecting technologies based upon the requirements in the project, and an evaluation of the possible solutions. The evaluation would then contain a proof of concept to ensure that the technology does actually deliver the desired requirements. It may very well be the case that a single vendor develops suitable products, but neglecting to evaluate alternatives may lead to the usage of inferior solutions.

7.1.5 Configuration management and version controlWe have only found one company that used a configuration management tool. All companies were aware of the existence of such type of tool, but were not using them. The reason for not using such a tool was usually that they could not find any reason to do so. Every company felt that it was necessary to keep track of older versions of the product, but that was conveniently managed using some backup strategy.

The developers kept personal older copies of the source code locally on their own workstations. The personal backup strategy is risky, since it depends on human remembering to update files back and forth. Two companies did not work in a network environment and did not believe that there was any risk of overwriting each other’s work. All companies were thus subject to the problem that a tool solves, and they were trying to solve these problems manually. Tools are cheap and easily installed, so there are no reason that a software developing organization should not use them.

Most companies developed products that was unique to each customer, and it was thus no problems of keeping track of what versions was installed where.

7.1.6 TestingWe have shown that the testing processes in small software developing organizations have insufficiencies. Most of the companies have some sort of testing strategy, but it has been shown that one testing strategy is not enough [BEIZER] to find a maximum amount of defects.

Only binaries, i.e. the functionality of the program as seen by the user, are tested. No reviews or inspections of code are performed. No coverage testing is performed. The most common way of testing seems to be to have someone follow a test protocol, report and fix the deviations, and iterate until no more defects are found. When errors are found outside the testing protocol, they

56

are seldom incorporated in the testing process. Root cause analysis is seldom made, error fixing is almost ad hoc.

We believe that most companies are unaware of other testing strategies than for example input validation (the program should not crash on erroneous input) and a validation against the system or requirement specification (to make sure the program does what it is supposed to do).

7.1.7 RequirementsMost of the companies we interviewed were highly competent in their specific domain. This made requirements handling less of a problem than we had expected. Most of the companies were also very good at documenting the requirements. Also, the companies told us stories about how they had failed with their requirement handling previously and learnt a lot from this. It seems as if this is one of the first fields that companies get better in. If you run a company you learn early that documenting the requirements are important for your own survival.

Prototypes were used extensively, which might lead to other problems [MCCONNELL], but they help a lot in discussing requirements with the customer.

But it was not just positive. Most of the companies lacked a formalized process for requirements handling, and new requirements were often added without being documented which leads to feature creep.

None the less, our hypothesis did not stand up against the actual reality, which is clearly shown by the low grade of the requirement hypothesis.

7.2 ConclusionThe study has shown that companies have shortsighted view of software development, the hypothesis “View of software development” received the grade 3.0. The organizations are not considering the long-term nature of software while developing it. The organizations do not believe that they have the time or the resources to introduce a quality plan. Planning is focused on the next deadline only. Even though the time to market is becoming increasingly important [CROW], one can not forget about the long-term quality of the software being produced. This is very important, as late changes are much more expensive than earlier fixes [BOEHM]. In fact, fixing an error while the software is in operation can be as much as 1000 times more expensive than if the error is corrected in the first stages of development [BOEHM]. We believe that by introducing formal processes in the form of a CQM, this can be remedied, a suggestion for such CQM is presented in the next chapter.

Project planning is inadequate and the hypothesis received the grade 3.2. The hypothesis is that there are no or vague project plans. According to [METZGER] poor planning is the most common source of problems. During the interviews we found that many software projects are rather roughly planned. Some companies do not issue deadlines or milestones in their projects. None of the companies are measuring progress in their projects. We believe that this can be improved by introducing project-planning processes.

Companies select technical solutions without evaluating them. The organizations are limited by the technologies they have experience in. This hypothesis “Selecting technical solutions” received the grade 3.8. It was based on the anti-patterns [BROWN] Vendor Lock-In and Golden Hammer and our own experience of the industry. We believe that by introducing processes for evaluating technical solutions this can be improved.

Configuration management is inadequate. The hypothesis is that problems occur when the company has many versions of the same software at different customer sightings. There are problems with ensuring that only one person is working with a file at a given time. It is not

57

possible to restore the last version of a file, except by backup. The hypothesis received the grade 3.8, which means that it is verified by our study. A CM tool and a CM process can improve the situation at most companies [TOMAYKO], [FEILER].

Testing is inadequate. The hypothesis regarding software testing is that some type of testing is performed, but there exist no formal test plan. No regression testing is performed; it is common that defects that have been corrected reappear again and again. The organization is not familiar with simple testing concepts such as coverage tests, or statistical testing. This hypothesis received the grade 3.6. We believe that by introducing a testing process, this can be improved.

All of the above hypotheses were verified by our study. The following hypotheses, however, proved to be wrong.

The hypothesis “Requirements” received the grade 1.6. The hypothesis is that the products developed are based upon the customer specifications, but little time is spent on analyzing the requirements to ensure that they are the correct and that changing requirements are poorly handled. This is serious because it can destabilize the product to such a degree that it can not be finished at all [JONES]. But since most of the companies we interviewed were highly competent in their specific domain, it made requirements handling less of a problem than we had expected. This seems to be one of the first areas that companies get better in.

The hypothesis “Management” received the grade 1.0, which means that is was completely wrong. This hypothesis is that one person on the company decides everything as a despot. He (it is often a he) sets the deadlines and adds new requirements, but can not understand the software development specific problems. The programmers on the other hand are not trained in seeing the project from the managers’ point of view. It is possible that the organization lacks a project manager, or some type of authority between the management and the developers. This does not seem to be the case in the industry today, according to our study.

We will now use this knowledge to construct a basic Quality Manual for small organizations. It will focus on the 5 issues that proved to be a problem in existing organizations.

58

8 Quality framework for small organizations

The purpose of this chapter is to use the results from the analysis in chapter: 7, and construct a quality framework from this, that can be used to construct a CQM structure especially focused on small organizations.

In chapter 3 we discussed the concept of quality. We constructed a 4-level model of quality abstractions, which included:

Quality Definition. A quality definition describing goals.

Quality Model. A quality model documenting the requirements.

Quality Manual. A CQM that is implementing the requirements.

Produced documents.

In order to construct a CQM for small organizations, we must take the previous two levels into consideration, thus the first task becomes to find a quality definition. From this quality definition, we will construct a quality model and from the quality model build a quality manual. The idea is that the quality manual should be constantly updated and improved, while the quality model is a more stable document, which provides the ground on which the manual stands on. The quality definition on the other hand, should very seldom be modified.

Why not use ISO 9000 or CMM? ISO 9000 suffers from a bad reputation. During our interviews, nearly all the companies replied that ISO 9000 was “too much bureaucracy” or “too much documentation”. But this does not have to be the case. One of the companies, which was on their way to be ISO 9000-certified, showed us their quality manual, and it was as thin as a video manual. But even though ISO 9000 might be better than its reputation, the reputation itself lowers the use of it. It is also quite large, and it might be a big first step for a company with no formalized quality organization.

The way that CMM divides companies into maturity levels makes it tempting to think that CMM is excellent for small organizations, since companies of all sizes can “fit in” to the 5-level scale. But the statistics presented in chapter 3 clearly shows that this is not the case. More than 50 percent of the companies does not even reach level 2! CMM might be good in a later stage, but the first step is obviously too steep.

The existing quality models are thus unsuitable to be used as a first step. They might be useful in a later stage, but in the initial phases of quality improvement, they seem to be too large and incomprehensible. What we are looking for is something that can be used in the next (or even the current) project, not something that takes years to develop.

8.1 Quality DefinitionOne of our hypotheses about problems in small organizations is that they are not considering the long-term perspective of software development. This was strengthened by our empirical study. To avoid that such problems is introduced by a weak quality definition we have chosen to slightly alter the definition that companies seems to agree on. The definition is quite similar to one of the Total Quality Management definitions, but we do not see this as a drawback. Our definition of quality is divided into two parts. The first part is:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

59

This definition includes 3 aspects of quality: business success, customer satisfaction and continuous improvement. The company should keep their customers happy, make money doing so, and keep getting better at it. Simply stating that quality is customer satisfaction might feed the misconception that the customer should be happy only on the delivery day. The customer should of course be happy on the delivery day, but the customer should also be happy a year after the delivery day. The delivered product should withstand changing requirements, maintenance etc. The long-term perspective must be reflected in the quality definition. This perspective is caught with the “continuous improvements”.

To summarize, the three aspects of our first part of our quality definition are:

Customer satisfaction. Customers are satisfied when they receive the expected product and service. The customer may expect the product or service to be delivered on time, within budget and according to the requirements, but it does not have to be so. The customer probably expects the written agreements to be fulfilled, but the contents of them depend on the customer and project. It difficult to generalize on customer’s expectations, it is something that has to be explored for every new customer.

Business success. Business success is the ultimate goal for most companies. It is crucial for its survival to make money, but business success is not simply making quick money. Business success is a long term survival and expansion of an organization.

Continuous improvements. It is important to make continuous improvements to the organization’s quality plan. This is crucial for many reasons:

The quality manual must be flexible. If one part of it is not good, or not used, then remove it or rewrite it!

Learning from previous mistakes is a sure way of becoming really good at something.

Processes that lead to mistakes must be removed or changed.

You gain a competitive advantage by always aiming for higher quality.

The world is changing.

The idea of continuous improvement does also prevent the idea that quality is something that can be achieved – which of course it is not. If the organization believes that it has “achieved quality” then what is the point in becoming better?

The quality models available does not simply concentrate on customer satisfaction; they are concerned with, more or less, the entire organization. This is where the second part of our quality definition comes in. It is not possible to achieve long-term success if half the organization is counteracting the purpose. For example: if not enough effort is put into the well being of the employees, they will not produce as well as they could potentially do. Our definition of quality for the entire organization is a recursive definition:

“The quality of a system, is the quality of its sub systems.”

Quality can increase when each subsystem is increasing their quality. We believe that many organizations, especially small ones, focus too much on one or two subparts, forgetting the other. This is reflected in our hypothesis.

60

8.1.1 ConclusionOur empirical study showed that companies have a shortsighted view of software development. When me make our quality definition, it is important to take a more longsighted view of software developing into account. We now have a quality definition, which is the first part of the 4-level model. The quality definition is:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

And:

“The quality of a system, is the quality of its sub systems.”

The task now becomes to define the system, i.e., the organization, and its subparts. This will become our quality model in the 4-level abstraction from chapter 3.

8.2 Quality ModelThe purpose of this chapter is to describe a Quality Model that we can use to build a Quality Manual from. The model should build on the Quality Definition that we defined in chapter 59. It must also be as simple as possible so that it can easily be incorporated in a small company. To fulfill this, we make a model of a small company, including all the parts of the company. Then we set up quality values for each part of the company, so that we can fulfill the definition that: “The quality of a system, is the quality of its sub systems.”

This philosophy is quite different from for example ISO 9000 or CMM, which merely lists the requirements of the quality manual [PAULK+], [TINGEY]. We have chosen not to do this, because we want a more flexible model, which suits the dynamic structure of a small organization.

8.2.1 The OrganizationAn organization, in our model, consists of producers, owners, customers and products.

Producers:

Employees (developers)

Management (short sighted economical interests)

Project leaders

Owners:

Owners (long sighted economical interests)

Customers:

Customers, in a very wide sense of the concept. Customers are those that are buying the product and the people that are going to use the product.

Product:

One or more software product(s)

In small organizations, one person may have many roles, for example being an owner, an economical manager and a developer. Each role has its own perspective of quality.

61

8.2.2 Perspectives of QualityThe concept of quality depends on ones point of view. The various stakeholders have different quality expectations:

Employees: The employees, especially the developers, are an essential part of a software producing organization. The employees of the organization value personal growth, a comfortable working environment and a reasonable salary.

Management: Economical managers have economical responsibility for one specific software project. Their values include a low price, a controlled development process that allows for planning and effective development.

Project leaders: Project leaders value well-defined specifications, sufficient resources, time and clear goals. They want control of the project.

Owners: The owners make long term investments in the organizations, and therefore their values include making a profit, having a clear strategy and satisfied customers.

Customers: The customer is the strongest driver for quality. Without a customer, quality would not even be required. The customer wants the expected software at the expected time and for an expected price.

Product: The product does not, of course, have any values of its own. There are however attributes of the product that are not of main interest of the customer. Such attributes are testability, maintainability, reusability etc. These internal requirements are easily forgotten if too much focus is spent on ‘customer satisfaction’.

According to our quality definition (see chapter Quality Definition above), a system has quality when its sub systems have quality. In our model, this means that when all quality perspectives are taken into account, the result is an organization that can fulfill the other quality definition:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

8.2.3 ConclusionSince we have shown in chapter 3 that the definition of quality is dependent on the subjective view of the person making the definition, we built our quality model around a model of an organization that suits most small companies. This will ensure that, as many views of quality as possible will be taken into account. We then set up quality values for each subpart of the organization, thus ensuring that each part of the organization can have its view of quality fulfilled. This leads to conflicts – it is not always possible to satisfy everybody.

8.3 Quality Manual It has been shown that small companies have weaknesses in the following activities: project planning, selecting technical solutions, configuration management and testing. This chapter provides a suggestion for an initial quality manual that attempts to reduce those problems. A complete CQM would have to cover many more areas that those that we are suggesting here, but we focus on the issues that was found during the interviews. The processes suggested here is a suggestion of what processes could look like, they are by no means absolute.

It is important that the manual is simple to use and can easily be extended. It should be used as a first step towards a bigger manual for a small company, and can not bring with it a lot of extra “unnecessary” work. Otherwise the manual risks ending up on the shelves collecting dust. “A small organization can lose the spirit of the CMM (or any other process model) while attempting to apply the model to the letter, introducing excessive documentation and formality that can

62

impede project work” [WIEGERS]. It is also important that the manual can be used directly in the organization, preferably in on-going projects. Many of the companies that we interviewed said that they were about to develop a quality manual for their organization, but they did not take the first step towards doing so. By keeping the manual small, the first step becomes an easier one.

One of the interviewed companies that was in the process of developing a CQM of their own claimed that processes should be documented as flowcharts. The reason to this was to reduce the amount of text and thus making the CQM smaller. This is probably an excellent way to document the CQM; we have however documented our process suggestions as text. The processes are in the following format:

Name: The name of the process.

Forces: Why does the process exist? It is important that the people using the process understand why it must be followed.

Preconditions: What needs to be done in order to start the process.

Process: A list of activities that may or must be followed.

Result: The expected result of the process.

Rationale: Explains why the process looks like it does and how it relates to the quality model. Which subparts of the organization in our quality model are involved? How are their quality values affected? It also contains references to the interviews.

8.3.1 Divide and estimateName: Divide and estimate

Preconditions: None.

Assisting material: Data from previous similar projects.

Process: The Project Managers and the Employees have a meeting in which they together divide the project into as small activities as possible. To make the estimates more accurate, let all the people estimate on all the activities. Then note invariance and let the people who made the estimates argue for it. Modify the numbers and repeat this until consensus has been reached. The person who is responsible for the activity always has the final say. This should be done once, at the beginning of the project, and at major milestones. The estimation meeting should not take longer than approximately 1-2 hours.

Activities are only a few days worth of time and defines to be ‘done’ or ‘not done’ [MCCONNEL]). When all activities are short enough, and the participant’s feel assured that no activities have been missed, the process is finished.

Result: A list of activities, and a time for each activity.

Rationale: By dividing the project into smaller activities, estimating becomes simpler. It is also easy to forget large parts of work, if the work is not divided into smaller tasks. Estimating the time each activity will take is crucial for a realistic project plan.

The Employees must know what to do and what is expected from them. By being a part of the project planning they feel that they have a saying in what they are supposed to do. The interviews showed that this was not always the case.

63

Project Managers must be able to make correct plans. By asking the Employees the plans will most probably be more correct. This means that the Managers can give the Customers accurate price estimations.

It also becomes simpler for new Employees to make correct estimations if they make the estimations in the company of more experienced people.

8.3.2 Progress reportingThe interviews showed that some companies totally neglected to measure progress in the projects. By not doing this, they risk that the project is running out of control. A project gets late one hour at the time.

Name: Progress reporting

Preconditions: A list of activities.

Assisting material: None.

Process: A project plan must be kept, and updated, during the entire project. If an activity takes too long time, it must be re-estimated and the plan must be updated accordingly.

An activity has two states: Working and Finished.

Result: A project plan that is consistent with the actual progress of work in the project.

Rationale: During the project, it is important that activities does not take too long time. This will make the entire project later. The plan must be updated during the entire project.

8.3.3 Milestones and deadlinesTwo of the interviewed companies did not have any deadlines. Overtime was used a lot during some periods of time in two companies. Planning was proven to be a problem in the companies we interviewed. One of the companies that we asked for interviews could not even participate because they were working too much.

Name: Milestone and deadlines

Preconditions: A project plan.

Assisting material: Data from previous similar projects.

Process: The project is divided into a number of deadlines. Each deadline is divided into a range of miniature milestones. The difference between deadlines and miniature milestones is that the deadlines are project-wide while the miniature milestones are personal for each Employee or for a small team of Employees. Each deadline and miniature milestone should have a clear definition of what needs to be done in order to pass the milestone or deadline.

Result: A list of deadlines and miniature milestones, and the requirements for each.

Rationale: By focusing on miniature milestones, the project is kept on track and keeps the correct focus. Deadlines help increase the pressure on Management and Developers. It is however very important that the Developers have a say when deciding the deadlines.

64

By keeping miniature milestones, the Developers know what is expected of them. Since they have themselves agreed on the milestone, a commitment culture is grown. Managers can more easily track the progress of the project by seeing how many milestones are kept.

If a deadline is not kept, precautions must be taken. It is dangerous to believe that the time lost will be regained in a later stage [MCONNELL]. More likely, the entire project will be delayed by the same percentage of time that the deadline was delayed. The best way of ensuring that deadlines are kept is to remove functionality requirements [MCONNELL]. The interviews showed that this is seldom done. In stead, overtime and stress is used to try to get the project on its feet. This will lead to lower quality of the Product and burned out Employees.

8.3.4 Evaluation techniqueMany of the companies in the interviews used the same tool for many different tasks. They also used tools from a specific vendor consistently. As a small company, this might be the simplest solution. But in the long run, it is important to keep an open mind towards new technologies.

Name: Quantify the Needs

Preconditions: The system requirements of the system.

Steps performed: Make a matrix with different tools on the Y-axis and non-functional requirements on the X-axis. Examples of non-functional requirements include performance, ease-of-use, portability, maintainability etc. Put grades on each tool and non-functional requirement. Base the grade on experience, reviews from technical magazines and where it is possible, experiments and facts. All the people involved in using the tool should be able to grade it. The tool with the highest score is the most suitable technical solution. The grades may be weighted if some requirements are more important than other are.

Result: A matrix with grades of the different tools.

Rationale: Selecting a good technical solution can greatly enhance the quality and lifetime of the Product.

8.3.5 Configuration managementOnly one of the companies used a CM tool, the other companies used local backups or relied on system backups for older versions of files. It is our conclusion that a tool can greatly improve or at least simplify CM. A tool alone does not provide perfects CM, but it is a step in the right direction.

Name: Configuration Management

Preconditions: A CM tool simplifies matters a lot.

Steps performed: Every project has a list of configuration items, which must be under CM tool control. All of these items must have a version number, and changes to them must be documented. After a delivery (see Milestones and deadlines above), all items included in the are labeled with an unique identifier. This enables the organization to retrieve old deliveries.

Result: N/A

Rationale: The Employees will have better control of the configuration items. Managers and Employees will be able to trace changes on the items.

65

8.3.6 Line coverageName: Line coverage

Preconditions: Software source code has been produced.

Steps performed: A set of scenarios is executed through a tool that monitors the level of line coverage. It is likely that all lines of code can not be executed at once. It is not a goal to reach 100 percent coverage since this is often not realistic. Coverage as close to 100 percent as possible should be a goal, though.

Result: Software code that is ready for statistical testing.

Rationale: Line coverage is the method by which one ensure that all lines of software code has been executed at least once. This method does naturally not find all defects, but there is very good tool support, which makes the activity easy to perform. It thus makes improvements for the Developers, making it easier for them to find errors, and increases the quality in the Product.

8.3.7 Statistical testingThe second most problematic aspect of software development is ensuring that “The program does not crash”. The most common strategy to prevent program crashes was to perform software tests. Most people interviewed said that it is difficult to test the system the way that the user is going to use it. “The user is incredibly creative in finding ways to use the system that we could not anticipate”.

Name: Statistical testing

Preconditions: Software deliverables has been produced.

Steps performed: A model over usage behavior is established, this model is a Markov diagram with probabilities on the arcs. The model is then used to ‘generate’ test specifications that are similar to the way the user would use the products.

Result: A document describing the defects found.

Rationale: The idea behind statistical testing is to test software the way that the user is using the system. This results in that the most important defects are found first, thus improving quality in the Product.

8.3.8 Peer reviewsBeing able to reduce the number of defects found in the deliverables could potentially reduce the testing effort. Full-blown software Inspections as [GILB+] describes are probably too much for smaller software developing organizations, but the peer reviews of CMM [PAULK+] are probably realistic. The reason for reviews is that no organization interviewed tested anything but the binaries. What work items that are subject to Reviews is defined by the organization.

Name: Peer reviews

Preconditions: Software work item has been prepared, along with a checklist containing information on what defects to look for.

Steps performed: The person(s) performing the review checks the work item for the possible defects described in a checklist. The defects that he/she finds are documented in a defect log.

Result: Document describing defects found.

66

Rationale: It has been shown [GILB+] that reviews or inspections greatly increase the quality of the work produced. It makes the Product better, and improves the work performed by the Developers.

8.3.9 ConclusionThis chapter presented a Company Quality Manual to be used as a first step towards a process-oriented approach of working. It contained processes for all the areas under which the empirical study showed that small companies fall behind on.

8.4 ConclusionThe goal of this thesis was to present a quality manual made for small organizations that are developing software. To be able to do this, an empirical study was performed, and it shows that small companies have weaknesses in the following activities: project planning, selecting technical solutions, configuration management and testing. From the conclusions drawn from this, we constructed a quality framework that was presented in this chapter.

The quality framework consisted of a quality definition, a quality model and a quality manual. The manual should be used as a first step, and will probable have to be reworked to fit into the organization that uses it. This is not a drawback, though. A quality manual must be constantly updated and improved. The quality framework presented in this chapter should be used as a very first step only. It is more important to begin than to get it right the first time.

67

9 Future Work

This chapter gives suggestions as to what future work could be interesting in the light of this thesis.

9.1 The CQMOur proposal for a CQM has not been tested in the industry. It would be interesting to implement this into some small organizations and observing its usefulness. The findings would be use to iterate over the design of the CQM in order to produce one that is more efficient for its purpose.

9.2 Comparison to a big companyThe intent is that the CQM should be used in a small organization. But how does a larger organization differ from a small one in this context? Are the same problems applicable there?

9.3 The next stepThis thesis provides the first step in moving towards higher quality for a small organization. But there is still a long way to go before all the requirements for an ISO 9000 certification can be achieved. A proposal for future work is to make The Next Step, which would include a more detailed CQM, possibly including activities such as Risk Management, Software Metrics and Inspections.

68

10 References

[BECK+] Back, Leland L., and Thomas E. Perkins; A Survey of Software Engineering Practice: Tools, Methods, and Results; 1983; IEEE Transactions on Software Engineering SE0

[BEIZER] Boris Beizer; Software System Testing and Quality Assurance; 1984; Van Nostrand Reinhold

[BOEHM] B. W. Boehm; Software Engineering Economics; 1981; Prentice-Hall, Inc.

[BROCKA] Bruce Brocka and M. Suzanne Brocka; Quality Management: Implementing the Best Ideas of the Masters; 1992; Irwin Professional Pub

[BROOKS] Brooks, Fredrik P; The Mythical Man Month: Essays on Software Engineering; 1995; Addison-Wesley Pub Co.

[BROWN+] Brown, William H. Brown, William J Malveau, Raphael C; Antipatterns; 1998; John Wiley & Sons (C)

[CONNELL+] John Connell, Linda Shafer; Object oriented rapid prototyping; 1995; Yourdon Press

[CROW] Kenneth A. Crow; Improving Time-to-Market Through Planning and Resource Management; 1996; DRM Associates (http://www.npd-solutions.com/resources.html)

[EDVARDSSON] B. Edvardsson, B. Thomasson; Kvalitetsutveckling : ett managementperspektiv; 1992 ; Studenlitteratur AB

[EJVEGARD] Rolf Ejvegard; Vetenskaplig metod; 1996; Studentliteratur

[ELY+] Margot Ely et.al; Kvalitativ forskningsmetodik i praktiken. – circklar innom circlar; 1993; Studentliteratur

[ENCYC] John Marciniak (editor); Encyclopedia of Software Engineering; 1994; John Wiley & Sons, New York

[FEIGENBAUM] A. Feigenbaum; Total Quality Control; 1991; McGraw Hill

[FEILER] Peter H. Feiler; Software Configuration Management: Advances in Software Development Environments; 1990; Carnegie-Mellon University, Software Engineering Institute

[GARVIN:1] David Garvin; What does product quality really mean?; 1984; Sloan Management Review, 26,1, 1984

[GARVIN:2] David Garvin; Five Definitions of Quality; 1988; Free Press

[GILB] Tom Gilb; Principles of Software Enginnering Management; 1988; Addison-Wesley

[GILB+] Tom Gilb, Dorothy Graham; Software Inspections; 1996; Addison-Wesley

69

[HAMMER+] Michael Hammer, James Champy; Reengineering the Corporation : A Manifesto for Business Revolution, 1994; Harperbusiness

[HAMLET] Dick Hamlet; Are we testing for true realiability?; 1992; IEEE Software

[HUMPRHEY] Watts S. Humphrey; Introduction to the Personal Software Process; 1996; Addison-Wesley

[ISOTC176] ISO/TC 176/SC 2/WG 15/N131;Draft Brochure on Quality Management Principles and Guidelines On Their Application; 1997; ISO

[IVERSEN+] Jakob H, Iversen, Peter Axel Nielsen, Jacob Norbjerg; Problem Diagnosis in Software Process Improvement; 1998; CTI, Technical University of Denmark

[JONES:2] Capers Jones; Patterns of Large Software Systems: Failure and Success; 1987; IEEE Software

[JONES] Capers Jones; Assessment and Control of Software Risks; 1994; Yourdon Press

[KLEIN] Robert A. Klein; Achieve Total Quality Management; 1991; Chemical Engineering Progress

[KOTONYA+] Gerald Kotonya, Ian Sommerville; “Requirements Engineering”; 1998; John Wiley & Sons Ltd.

[LIBERTY] Liberty, Jesse; Clouds to code; 1997; Wrox Press Inc.

[LJUNGBERG] Lars Ljungberg; Buzzwords in Software; 1999; Q-Labs

[LÖÖV+] Peter Lööv, Gunnar Arnberg; Approaching CMM Level 2; 1996; Department of Computer Science and Business Administration, University College of Karlskrona/Ronneby

[MAGUIRE] Steve Maguire; Debugging the development process: Practical Strategies for Staying Focused, Hitting Ship Dates, and Building Solid Teams; 1994; Microsoft Press

[MCCONNELL] Steve McConnell; Rapid Development; 1996; Microsoft Press

[METZGER] Philip Metzger; Managing a Programming Project, 2d ed; 1981; Prentice Hall

[PAULK+] Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V. Weber.;Capability Maturity Model for Software;Version 1.1. CMU/SEI-93-TR-24. Pittsburgh: Software Engineering Institute;1993.

[PERPER] Binnie Perper; How to sell quality... in other words. 1996; Writing By Design

[STANDISH] Sample research paper (http://www.standishgroup.com/chaos.html); The CHAOS Report; 1995; Standish Group

[TINGEY] Michael O. Tingey; Comparing ISO 9000, Malcolm Balbridge and the SEI CMM for software; 1997; Prentice Hall

[TOMAYKO] James E. Tomayko; Software Configuration Management; 1990; Carnegie-Mellon University, Software Engineering Institute

70

[WIEGERS] Karl E. Wiegers; Software Process Improvement: Eight Traps to Avoid; May 1996; Software Development Magazine

[ZAHRAN] Sami Zahlan; Software Process Improvement; 1998; Addison-Wesley

[WIEGERS] Karl E. Wiegers; Software Process Improvement: Eight Traps to Avoid; May 1996; Software Development Magazine

71

11 Definitions

11.1 Small organizationThis thesis studies small organizations. By this we mean small software developing organizations. Small in this context are less than 20 persons actively taking part in development of software products. An organization is not necessarily a company, there can be several organizations within a larger company. The studies companies consisted of only one organization each, though.

11.2 QualityThere are many definitions of the quality concepts, some of them are discussed in the quality chapter. Other quality related concepts that are defined in that chapter are system, framework and model.

11.3 ISO 9000When we refer to ISO 9000 in this paper, we refer to the family of such quality certifications, with ISO 9000-3 in particular.

72

12 Appendix A - Questions

This section contains the actual questions that are asked during each interview. The headings serve the purpose of organizing the questions, but these will not be communicated to the interviewee. Each question is commented for the purpose of that question, we will use these comments during the interview to clarify if necessary. These comments will also be used when the interview is later analyzed.

The questions generally ask for consciousness about software development problems. There is only time for one hour of questions so it is not possible to go to any depth in any area. The strategy then is to ask for issues that either we have personal experience from or are discussed in frameworks such as CMM.

12.1 InitialThese questions are discussed initially. They do not serve any scientific purpose but they are here to get the discussion going.

Describe your role in the organization, time of employment, education?

What is the average level of education in the organization (average number of study points) (what type of people are working within the organization. Are there mostly self-learned, or do they have some formal education?)

12.2 View of softwareWhat is quality? Many organizations say proudly that they are developing high quality products. If they have a real interest in high quality, then they should be aware of what quality is, or at least given the concept some thought.

What are the problems that affect product quality? We are interested in the spontaneous answer to this question. What aspect of software development is perceived to be most difficult? If the spontaneous answer is “bugs”, then the organization might be fire fighting. If requirements and understanding the customer is central then the organization probably have reached a higher level of maturity.

What is your relation to ISO 9000? Why? (It is probably safe to use ISO 9000 as a concept, everyone has heard of it?) We need to know the perceived problems with existing models, and then we know what is expected of our model. It is also interesting to find if the feeling about ISO 9000 is based upon prejudices and hand on experiences.

Have you studied other certification frameworks? Which? Having studied other quality frameworks clearly indicated awareness of other frameworks. Such awareness can probably be interpreted as more than brief interest in improving quality.

How do you ensure that the experiences from one project are taken care of in the next? If an organization does not explicitly document their experiences from each project, then how do they share their experiences? Experiences are one of the bases for improvement, and if that is not taken care of, how do they improve?

73

12.3 PlanningHow do you know what to do when you come to work each morning? Some organizations say proudly that when you come to work each morning you never know what to do! This indicates that there is little control over what and when activities are performed.

Do you know what to do next week, next month? How long in advance do the organization plan its projects. It is probably better to have a small or weak plan than not having any at all. However, the plan needs to be consistent with reality so it needs to be updated regularly. Do the organization do this? Try to find out if they have longer plans than simply this project. Short plans do basically have two implications; either do they not have any future customers, or do they not believe to have any time except for the current project.

How do you know if you are going to meet a deadline? Are they measuring progress? Are people reporting what and how much they have done. Have they been collecting some type of metrics, or even experiences that help them to know how the project is going? The opposite is a very naïve approach where all deadlines will be held.

What happens if you are late for a deadline? There are many possible strategies for managing late projects, for example: Adding more people to the project and risking breaking Brooks law [BROOKS], or simply ignoring the problem.

How much overtime does each employee use each month? Approximately none, 1, 2 or many hours per month. Much overtime indicates that the organization have difficulties in planning. If the answer is that it depends, then they are sometimes using very much overtime, and that indicates poor planning.

Who is going to use the documentation? (Is the documentation actually used, or is it produced because they believe that it is necessary?)

What happens if a given person gets sick or quits? (Are some people more valuable than other? How do they manage the transition from one developer to another?)

12.4 ManagementHow come you are developing the type of products you are? (The thesis is that a person with much domain knowledge once founded the organization. That person did then hire a set of programmers to develop products. This indicates that the organization do not find software engineering problems as main problems.)

What control does management have over the development? What role does the management person have? Is he/she involved with the actual development, or does he/she only provide customers and requirements specifications?

12.5 Selecting technical solutionsHow do you select technical solutions? Is it a conscious choice, or are they using XYZ because they have always done so.

How do you know if the product will work in practice? Have they performed some evaluation among possible choices? Is it a rational choice or are they working with ‘Microsoft’.

74

12.6 Configuration managementHow do you ensure that two persons are not working in the same file? This is CM in its purest form. Lack of CM means that they spend time merging files every now and then. If they are using some CM tool then they must be considered to be aware of this problem.

How do you do if you want to revert to last weeks state or that before lunch?

Can you reproduce an old version of a product? Being able to reproduce old versions means that they have some formalized method for managing the versions that they deliver.

Do you know what version is installed where? Lack of management of installed versions may lead to problems with updating client’s software.

12.7 Testing What is a bug? (Defect? Error?) (Is there a difference between them? Is a bug only a system crash?)

Describe the testing cycle. (Is there for example a preset stability ambition, or is the products just tested until they do not seem to crash anymore.)

How do you ensure that old defects do not return? (Someone said, “all bugs have to be fixed three times”, while this may or may not be true, lack of regression testing can harm the project.)

Describe the defect cycle. (Is the new version sent to all the customers?)

12.8 RequirementsHow do you know that you are developing the right application? What type of analysis has been performed in order to reach the requirements specification? Is the organization simply accepting customer requirements, are they developing prototypes, or what?

How do you avoid misunderstandings? The provider and the customer might very well reach consensus, but how are they sure that they both agree on the same thing? From personal experience we have found that if specifications are too technical the customer might not really understand what they mean, but accept them anyway.

How are changing requirements managed? The requirements for any software product will inevitably change [KOTONYA+]. These changes risk leading to feature creep. Projects that fail to control creeping requirements are highly susceptible to excessive schedule pressure [JONES].

75

13 Appendix B – Overview over the answers to the questions

The answers of the questions have been generalized and are presented here. A person may have answered many things, and a person may have not answered a question. It is noted for each answer what organization and person that gave the answer, these are presented within brackets.

The answers to the questions concerning management varied from company to company and we failed to find any interesting patters, and are therefore not listed in this chapter.

What is the average level of education in the organization?Does not have any formal education, perhaps some courses [1, 3.1, 3.2, 4.1, 4.2 and 5]. The most common reason is that there were no or little opportunities for formal education when they started. Has an exam from a university [2.1, 2.2, 5.1]

13.1 QualityWhat is quality?The program does not crash [1.1, 2.2, 4.2]Meeting customer requirements [2.1, 2.2, 3.2, 4.1]Is difficult to grasp, or define [3.1]Working according to principles and everything is documented [5.1, 5.2]

What are the problems that affect product quality?Bugs and testing [2.1, 2.2, 3.1]Requirements, understanding the customer and avoiding feature creep [1.1, 2.1, 4.1, 4.2]Planning and documentation [2.2, 3.1, 3.2, 5.1, 5.2]

What is you relation to ISO? Why?It is too bureaucratic [2.1, 2.2, 3.1, 3.2, 4.1, 4.2]It would probably help us [1.1]It is helping us [5.1, 5.2]Something, but not ISO, would help us. [2.1, 3.1]It does not help us to develop, just to document [3.1](The most common perception is thus that ISO is too bureaucratic. However as seen in the next answer is that the people that find ISO that have not had any real experience with it either.)

Has actual experience of ISO [1.1, 5.1, 5.2]Have not worked with ISO [2.1, 2.2, 3.1, 3.2, 4.1, 4.2]

Have you studied other certification frameworks? Which?No [1.1, 2.2, 3.1, 3.2]Yes (Not CMM, TQM or Malcom Balbridge though) [2.1, 4.1, 4.2, 5.1, 5.2]

How are you ensuring that the experiences from one project are taken care of in the next?One learns new things during each project [1.1, 2.2, 5.1, 5.2]Conclusion reports [2.1]We do not [3.1, 3.2, 4.1, 4.2]

13.2 Planning How do you know what to do when you come to work each morning? (Short term planning)TODO lists [1.1, 2.1, 2.2, 3.1, 4.1]

76

One person is responsible for delegating work tasks [5.1, 5.2]

Do you know what to do next week, next month?There is no problem to know what to do [2.1, 3.1, 3.2, 5.1]Project plans [4.2]As it happens [4.1, 2.2]

How do you know if you are going to meet a deadline?We have never been late for a deadline [1.1, 3.1]Experience [3.1, 3.2]Project plans and systems specifications [2.1, 5.1, 5.2]You do not [4.1](No one answered that they measured progress)

(Company 1 and 3 did not plan for any deadlines)

What happens if you are late for a deadline?Overtime [2.1, 2.2]Discussion with the customer, cutting down requirements for example [2.1, 4.1, 4.2]Take resources from other projects [5.1, 5.2]

How much overtime does each employee use each month?Very much during some periods [2.1, 2.2]We preferably work weekends [3.1, 3.2]Does not work any (project-related) overtime [1.1, 4.1, 4.2, 5.1, 5.2]

What happens if a given person gets sick or quits?Our time-plan is not so tight that we can not handle that [3.1, 3.2]The organization is two or less [1.1, 4.1, 4.2]Our quality system enables us the share work [5.1, 5.2]Someone else has to take over [2.1, 2.2]

13.3 Selecting technical solutions How do you select technical solutions?We use applications from a major software vendor [1.1, 3.1, 3.2, 5.1]It depends on the project and the customer [2.1, 2.2]The technologies that we are experienced in [4.1, 4.2, 1.1, 5.2]

How do you know if the product will work in practice?We do not know that [2.1, 2.2, 5.1, 5.2]We use applications from a major software vendor [3.1, 3.2, ]It is specified in the requirement specification [4.1, 4.2]

13.4 Configuration managementHow do you ensure that two persons are not working in the same file?We do not [1.1, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2]We are using a configuration management tool that ensures this [5.1, 5.2]

How do you do if you want to revert to last weeks state or that before lunch?It is not possible [1.1, 2.1, 2.2, 4.1]By restoring yesterdays backup [3.1, 3.2, 4.2]From our configuration management tool [5.1, 5.2]

Can you reproduce an old version of a product?Yes [1.1, 2.1, 2.2, 4.1, 4.2, 5.1, 5.2]

77

No [3.1, 3.2]

Do you know what version is installed where?Yes, we develop customer unique applications so it is easy to keep track of versions [1.1, 2.2, 4.2, 5.1, 5.2]Yes, but we have had problems with that. [2.2]No [3.1, 3.2]

13.5 TestingDescribe the testing cycleThere are no (or little) formal test-plans or specifications. Each developer is responsible for his/her work [1.1, 3.1, 3.2, 2.1]Alpha/Beta strategies [5.1, 5.2]The person that has written the code does not test it. [3.1, 2.2]There are test specification, with documented expected results [5.1, 5.2]It is not possible to perform a complete system test. The user is incredibly creative in finding ways to use the system that we could not anticipate [2.1, 2.2, 3.1, 3.2, 4.2]

How do you ensure that old defects do not return?Once we have fixed a problem, it does not return [1.1, 3.1, 3.2, 4.1, 4.2]We do not know if old defects return [2.2, 2.1]We have a bug-tracking system, and write deviation reports [5.1, 5.2]

Describe the defect cycleThe customer reports the defect, and we fixe it as soon as possible [1.1]The above but, if the defect is major then the test specifications have to be re-executed [3.1, 3.2, 5.1, 5.2](Most people did not answer this question)

13.6 Requirements How do you know that you are developing the right application? (and how do you avoid misunderstandings?)Have long experience in the domain, so we know [1.1,3.1]Prototypes [4.1, 2.1, 5.2, 1.1]Meetings, email and phone [2.1, 2.2, 4.1, 4.2]Documentation and specifications [2.1, 3.1, 5.1,5.2]External standards [3.2]

How are changing requirements managed?Have never experienced changing requirements [1.1]That is regulated in contracts and specifications [4.1, 4.2, 5.1, 5.2]There is no specific way of managing changes [2.1, 2.2, 3.1, 3.2]

78