Craft or Science? Software Engineering Evolution

Views from the bridge

Craft or Science? Software Engineering Evolution

(c) Alcatel, 1994

ABSTRACT

The last thirty years of the technological revolution have seen the development of a number ofhigh-technology industries. The computer industry saw an explosion in its software needs as long as 25 yearsago, and has been evolving accordingly ever since. Other hardware industries such as telecommunicationswhich have felt more recently the eects of this revolution are learning the same lessons: experiencing theevolution of software development from a craft (a creative act based partially on rules of thumb andhypothesis), to a true branch of engineering using scientic and mathematical principles. This article presentshow far software development has come, explaining how, although software engineering not yet a trulyengineered process, it is certainly en route to becoming so. Finally it gives examples of how otherengineering disciplines can help software engineering along its evolutionary path.

THE SOFTWARE EXPLOSION

The concept of software has been around for as long as there have been procedures to automate. We could goback to Herman Holleriths punched card census system, or Charles Babbages dierence engines. Lists ofinstructions, or programs, in the form of punched cards, could be said to date right back to Jacquardslooms (as programmed by Ada Lovelace, daughter of Lord Byrno, who gave her name to the Adaprogramming language). Software drives hardware: software programs are there to enable the hardware torun.

The twentieth century is seeing a technological revolution without precedent. Cars, telephones, radio,computers and satellites have been invented with barely a pause for breath. And there are no signs ofanything slowing down: the innitely small becomes smaller, and what was impossible to do yesterday hasbecome impossible to do without. One basic need has not changed: to get the most out of all this modernmachinery, procedures must be wri;en and programmed. The technological revolution has not dismissed theneed for software. However the increase in hardware complexity has obliged the development of ever moreelaborate software packages.

More recently a specic type of machine the computer has evolved with the primary purpose of runningsoftware programs. Software programs started to exist in their own right, independent of the hardware:companies were started that produced only software, and a new industry was born. Specialised techniqueshave been, and continue to be dened for software development. To ensure that necessary consideration isgiven to these mechanisms, a new discipline called Software Engineering is being developed which provides arm framework based on engineering principles.

Repeatedly pioneers of the hardware revolution are nding that software plays an ever more important rolein their products. The software no longer comes second; it becomes as important as the hardware on which it

Craft or Science? Software Engineering Evolution | Views from the bridge http://viewsfromthebridge.wordpress.com/tech-publications/craft-or-scie...

1 of 10 11/28/2013 02:28

runs. Software packages cease to be rmware instructions blown onto eproms, becoming complex,workstation based graphic systems involving many person years of development eort. In short, the samesoftware engineering principles used by the software industry are becoming necessary for the hardwaremanufacturers as well.

As hardware companies start down the path of large-scale software production they risk making mistakeswhich have already been made once over. The still relatively immature, trial and error state of softwaretechnology adds to this danger. As it evolves towards a true engineering discipline, the understanding offundamental software concepts improves, the approach to development becomes more disciplined and thetechniques employed become be;er dened and used. But it is not there yet.

Perhaps the established software companies are further along this evolutionary path. But by noting theadvances they have already made, and by avoiding their mistakes, hardware manufacturers may join inwithout losing time.

ENGINEERED SOFTWARE?

As software products have grown in complexity, so have the techniques and procedures used to developthem. The term Software Engineering was coined to suggest an approach with the necessary discipline tocontrol the many activities involved. But is the software we produce really the result of an engineeredprocess? Can we justify this use of the term at all, or is it wishful thinking? Maybe this question can beanswered by looking at the denition of engineering itself.

Engineering has been dened as the application of science and mathematics to the design and constructionof artefacts which are useful to humanity .

In other words, engineering provides a means to create something useful based on sound scientic principles.Can we apply this denition to software production? It implies :

1. The creation of software which fulls its requirements of :

* suitability of purpose, or how appropriately the software matches the needs agreed between user andanalyst* evolubility, maintainability, the abilities of the package(s) to stay up to date without implying anever-increasing support overhead* timeliness, particularly as a late product can often mean an obsolete one.* elegance of solution. The product should be ecient and pleasant to use

2. A development process which :

* is based on proven scientic and mathematical principles* uses formally dened and commonly understood techniques* enables the mathematical proof of the output of any stage.

If the above were true we could condently use the term Software Engineering. This looks feasible but weknow it is not true. Virtually every available so-called software engineering textbook opens by gleefully listingexamples of failed large-scale software projects. And the so-called Software Crisis in not over yet : as therst chapter of DeMarcos book Peopleware [*] cheerfully reminds us, somewhere, at this very minute, aproject is in the process of failing

So what is going wrong? According to Peopleware , it is due to not treating developers with respect.Browns Software Engineering Environments [*] implies that a fully integrated tool set will solve many ofthe problems. Conger [*] explains that it is the methodology that must rst be properly dened and thenfollowed. The answer is of course that they are all right, at least partially. There are a number of principles


2 of 10 11/28/2013 02:28

and techniques available for software development, dealing with human, environment and quality aspects. Itis not these principles in themselves which are at fault; more likely it is their use which is lacking. As explainsBaber [*], By its inherent nature, software development is an engineering discipline. It is not currentlypractised as such.

So why are these techniques so dicult to apply? The reasons may be linked to the newness of the wholedomain of software development.

THE STUMBLING BLOCKS

There are several good reasons why we fail to apply correctly software engineering principles. Brown lists theproblems as follows :

* Complexity, both of the product and the development processes* Diculty of establishing requirements* (Supposed) changeability of software* Invisibility* Development of a theory for the problem domain

These issues are detailed below.

Complexity

Application requirements. The complex nature of software applications stems from the fact that what wea;empt to do with software is far more dicult than what we a;empted to do without it. In general the mostdesired products are those which enable things which would not otherwise have been possible.

Development activities. Software development involves the interaction of a number of design andimplementation activities, each of which will look at the application in a dierent way. For example a systemdecomposition may be done according to architecture, functions or processes. The design must guaranteecoherence between the dierent types of decomposition: this requirement becomes more and more dicult asthe application increases in size.

Product interfacing. The added requirement that two applications work together indicates further complexity.This is a special problem for hardware manufacturers who not only have to ensure correct communicationbetween dierent software modules, but also between software and hardware devices.

Organisational complexity. The co-ordinated development of software and hardware increases theco-ordination required between dierent development groups or even separate companies, each with theirown understanding of the application eld.

Diculty of establishing requirements

To develop a package, the supplier must rst identify the needs of his customers. This is very dicult as theanalyst is unlikely to understand fully either the clients domain, or the way the client talks about it. To makeit worse, there is a reasonable chance that the client does not really know exactly what he wants.

Changeability of Software

We have all made the mistake (at least once) of thinking that software is easy to modify. The apparentsimplicity of changing software, adding functions, making that quick x is undeniable. But we all know aswell that it is fatal to think of software in this way. But we still do.

A spin-o problem is that, at the beginning of a software project certain a;ributes (for example,


3 of 10 11/28/2013 02:28

performance) may be left out with the intention of adding them later. Similarly the client often gives lessconsideration than necessary to the requirements denition, thinking that he can request a modication later.

Invisibility

The intangibility of software is linked to the immaturity of the eld, and our own lack of experience. Thismakes it hard to comprehend the steps to take. We still think of programming when we think of software,when in reality programming makes up no more than one third of the total development eort.

This intangibility can cause one program to look much like another. The risk is, for example, that theprototype of an application is used as a real product. The dierences between a thrown-together prototypeand an engineered software package become evident when modications are necessary later.

Development of a theory for the problem domain

Computer systems are often required to model the situations in which they are used. A failure to analysecorrectly the domain of a set of requirements will result in fundamental aws in the implementation.

SYMPTOMS OF IMMATURITY

Each of the above problems may be linked either to the immaturity of the eld, or the lack of understandingof the problem domains to which software is applied. And these problems have no easy solutions, linked asthey are to our ongoing accumulation of knowledge. Furthermore, as we enter new problem domainsrequiring software, for example network management, we nd untreated problems which require newmethods of analysis. The evolutionary process continues.

Given these diculties, it is no surprise that mistakes are made. Common errors are :

Reinvention. To start from scratch, not proting from the proven work of the past, is a frequent error. It ishard to know what has been done already and what is new, especially as dierent problems presentthemselves in new ways. This is highlighted when experts leave a given company and their work must berepeated. The mistake of reinvention is often made purposely as individuals do not trust or understand theprogress of others: at a company level this is referred to as the NIH ( Not Invented Here ) principle.

Misjudgement. This is the art of applying the wrong solution to a problem, either through misunderstandingor lack of competence. For example, to use a database when a at le would do, or to apply object-orientedtechniques to a particularly non-object-oriented problem. Remember the adage : A real programmer can writeFORTRAN in any language.

To prevent these kind of blunders, a more formal approach must be applied to the whole of the developmentprocess. In this way the decisions of each stage can be veried and proved. Fortunately for softwaredevelopers, a number of denitions and standards exist to ensure this, such as the IEEE standard for softwaredevelopment processes: this is described in detail below.

THE SOFTWARE DEVELOPMENT PROCESS

It is not an easy task to develop software. The newcomer to the software industry is faced with a plethora ofdierent life cycles, complex methodologies, overlapping methods and incomplete design techniques. Addedto this, a large variety of design and development tools with similar overlaps and discrepancies, serve toconfuse the issue. Further confusion arises from the tool set providers sales claims. Maybe it is possible tofollow the methodology SSADM using Oracles CASE tool set, but that certainly wasnt the original intentionof the tool.

In this environment of contradiction and overlap it is easy to lose sight of the fundamental elements of


4 of 10 11/28/2013 02:28

development which apply whatever methodology, design technique or tool is in use. Whichever way theactivities of a software project are performed, it remains mandatory that each one exists and produces theexpected output.

This may be considered as the lowest level of the software development process. It may be built on as follows:

Level 0: The mandatory activities and objectives of each part of the project are xed.

Level 1: These activities are organised into processes and grouped as the phases of a software life cycleappropriate to the product in hand.

Level 2: The expected outputs of each activity, process or phase are formalised by the denition of amethodology

Level 3: To aid the production of such outputs, and to ensure their coherence, the use of proven designmethods are dened.

Level 4: To accelerate and co-ordinate the production processes, software design, development and supporttools are used, possibly integrated to form an environment. As complexity grows such tools become essential.

It is clearly essential to x the lower level denitions before proceeding higher. It would be dangerous to x atool set before the methodology is dened, especially as tools often impose restrictions on the methodology oreven the life cycle.

As Sue Conger [*] notes, No one tool is ideal or complete. The software engineer knows how to select thetools, understanding their strengths and weaknesses. Most of all, a software engineer is not limited to a singletool he or she tries to force-t to all situations .

Problem solving must be handled at the lowest possible level. Incoherence in a lower level may result in awshigher up, if it is not treated. The cause of a given problem should be checked for at the lowest level ratherthan curing higher level symptoms of problems occurring lower down.

Level 0 activities are mandatory for the development and maintenance of all software packages withoutexception, small or large. A standard from the IEEE computer society: IEEE Standard for DevelopingSoftware Life Cycle Processes [*] denes the minimal set of these activities: a total of 65 mandatory activitiesare specied, organised into 17 processes. The processes themselves are divided into 7 dierent process types,as follows :

The standard imposes only the tasks to be performed. Each activity works on provided inputs (which must beavailable before the activity can be started) to generate output information. The form of this information is notimposed, nor is the method used to obtain it; the actual ordering of activities into life cycle phases, and thedocuments to be produced, are left to the project in its Level 1 and 2 process phases.

Maybe, further down the evolutionary path we will see standards dened for the higher levels of the softwaredevelopment process. For the moment we must content ourselves with Level 0. If all software projectsconformed to this lowest level standard, we would already be a lot further towards the achievement of theengineering discipline we so need.

THE PATH TO ENGINEERING

By examining the essential a;ributes of other engineering disciplines, we can both discover how much furthersoftware engineering has to go, and benet from their experience. So what are the required a;ributes of abranch of engineering? A summary of Babers [*] essential aspects of any engineering discipline are listed


5 of 10 11/28/2013 02:28

below, together with their relevance to software development.

1. The existence of a substantial body of directly relevant scientic, theoretical knowledge

The theory of the development of software has been studied for almost fty years now. Signicant progresshas been made but some necessary areas are seriously lacking. For example, in the eld of metrics andreliability measurement, a report presented by IBM La Gaude in 1991 [*] explained that software reliabilityis not a eld applied [by La Gaude], which considers that the theory has not yet reached a sucient level toproduce anything of substance .

In order that research be continued and expanded upon, knowledge should come from a wide variety ofrelated (but dierent) areas. This is not a problem for software engineering science which overlaps with anumber of disparate elds. As notes Baber, many important unanswered questions remain, ensuring thatresearch will continue.

2. The thorough learning of such knowledge by practitioners and their management.

It seems too obvious to say that software engineers should be competent in what they do. Other engineeringdisciplines require many years of study before acceptance. However, given the diculties of complexity andintangibility this is often not the case for software. Even once employed, training (one of the IEEEsmandatory activities) can often get missed out. The La Gaude report explains its inability to use project costevaluations such as COCOMO by saying that the principal factor which upsets the model is the high level ofvariation in the quality of the programmers .

At the management level, a lack of technical make recruitment and task allocation dicult.

3. The regular, systematic application of such knowledge to their work.

It is not enough to understand only theory: software researchers are frequently chastised for their lack ofpracticality. However, the work of practitioners with only a limited theoretical base is unlikely to beprogressive, and will probably result in wasted time as theory is reinvented on the job.

4. The individual responsibility of engineers and their management to clients and public for correct, safedesigns

Each person working on a project has individual responsibility for the reliability of his work.

As each phase of the software life cycle is normally worked on by a dierent set of people, this aspect impliesthat a group can present their outputs condent that they have been veried both systematically andanalytically against their requirements.

5. The existence of professional associations having sucient knowledge and experience prerequisites formembership.

Professional bodies exist to enable the sharing of knowledge between professionals. They also provide acommonly agreed measurement for the competence of their members.

What Benets?

The benets of an engineering approach are noted as :

* a reduction of errors (to the point where software failures become a noteworthy event)

* an increase in eciency, as unnecessary overheads are minimised or eliminated.


6 of 10 11/28/2013 02:28

SOFTWARE IS ART?

The traditional craftsman uses handed down principles and rules of thumb as the basis of his work:experience gained over many years of apprenticeship. The engineer proves his designs at every stage withscientic laws and mathematics. Craft and engineering dier in their techniques but they have a common goalremain the same : to provide ecient, useful artefacts.

In Computer Programming as an art [*], Knuth summarises this by saying computer programming is anart, because it applies accumulated knowledge to the world, because it requires skill and ingenuity, andespecially because it produces object of beauty . The knowledge base that we have built up over thirty yearsis agreed to be incomplete. Reliance is often placed on software craftsmen, commonly known as gurus orhackers (in the programming world, a hacker is seen as a programmer of high esteem whereas it is thecracker who breaks into computer systems) the reputation of some gives them almost hero. Baber [*]suggests that when software development progresses to be a true engineering discipline, such characters willdisappear to be replaced by members of professional bodies. If this is going to happen it is a long way oyet.

And what of software itself? Is software an art form? Knuth refers to beautiful programs, and it is true thatan elegantly structured section of code may be a pleasure to look at, at least to another programmer! Thispoint should not be taken too far: a motorway intersection may well be an object of beauty to anotherconstruction engineer, but it is probably not for the rest of us. Artistic qualities do however have practicalvalue for software : an elegant program is reasonably likely to be a well-wri;en, maintainable one. It would beworth considering the addition of a li;le art at all stages of the software life cycle. For example :

1. In the specication phase, a common language document is used to show the producers understanding ofthe clients needs. A well-wri;en specication will increase the chances that such requirements have beennoted; a readable specication may well ensure that both parties read it at all.

2. Software architectures can be elegant or impractical. An elegant architecture is one which ensures that itssubsystems work together smoothly and eciently; an inecient architecture will be the source of anunnecessary performance overhead.

3. Algorithms can be uid, smooth in motion or clunky and slow. A well-wri;en algorithm will be moreecient than a badly wri;en one.

4. The code itself can read like a good book or a childs rst essay. If the code is not readable it will quicklybecome unmaintainable, especially if its author is no longer available to explain it. Code should beself-documenting. If it is not clear exactly what a given code section is doing then it should be rewri;en. In themaintenance phase, the code should also explain how it has been modied, by whom and when.

The quality of the product is dependent on the experience of the software developers and their management at their craft. Such craftsmanship must be learned. And as we have already seen, there is not always the timeor resources available to provide such training. Peopleware makes the point that developers should beconsidered as experts rather than resources, and should be given enough space and facilities to permit themto excel. In DeMarco and Listers experience, it is this which ensures the optimal productivity of developmentgroups.

Todays software developer is somewhere between a craftsman and an engineer, using both science andcommon sense in his work. Tomorrows developer may well be the perfectly trained engineer provingeverything as he goes. But he is not there yet. At least for the moment, gurus are a necessity and softwarebeauty keeps our minds on the goal.

5. TAKING IT ALL TOO FAR?


7 of 10 11/28/2013 02:28

As the evolution towards true engineering continues, we a;empt to increase our understandings of theworkings of software by comparing it to the real-world. Such modelling happens at all levels, for example :

* Algorithms are wri;en based on boiling and cooling techniques to enable the shaking down of data(simulated annealing).

* Prototyping of software products reects the tradition of building scale models to demonstrate the feasibilityof a design.

* Specication paradigms such as neural networks are used to dene learning algorithms.

Approaches to engineering production have also been mapped onto the software world. To indicate thediversity of such models, three such techniques are presented here : the rst two come from the world ofelectronics the modelling of programmed modules as discrete components, each with a dened behaviourand interfaces (like Integrated Circuits) and the creation of an error-free environment (or cleanroom ) forsoftware production. Finally, measurement techniques have been borrowed from materials engineering as ameans of increasing the tangibility of software during its development.

- Software Components and ICs

The principle behind Software Components was rst presented by Brooks back in 1975. Using the sameprinciples as in electrical engineering, low-level programmed modules are formalised with a dened externalbehaviour and interfaces. An example of this approach is the NAG library of programmed functions fornumerical computation (or the PHIGS 3D graphics library). More recently, with the advent of object orientedtechnology this approach has been extended to the concept of Software Integrated Circuits ICs which arebinary les which implement objects.

As writes Brad Cox [*], the possibility that the software industry might obtain some of the benets that thesilicon chip brought to the hardware industry; the ability of a supplier to deliver a tightly encapsulated unit offunctionality that is specialised for its intended function, yet independent of any particular application . It isindeed a nice thought, but is it possible in practice? Note that within the chip, it is a dierent story as the sameproblems as with software occur between the dierent modules on the silicon.

A related technology is that of the code reuse approach to software development, in which software isdesigned with an explicit aim to use previously wri;en software components, and any new softwarecomponents are designed and programmed in a way to ensure their capacity for reuse. Note that this is verydierent to developing a new software product based on a previously developed, seemingly similarapplication. this approach often ends up costing more than it would to rewrite the supposedly sharedpackages.

- Cleanroom software Engineering

Possibly the best way to go but not so sure that we are ready for this, yet. It is evolution, not revolution, that isnecessary.

- Software Metrics

It is not enough to transpose the rules of electrical engineering onto software, to reapply the samemanagement techniques and to expect them to work.

What is interesting is that such techniques, proved and re-proved, have met with li;le interest. This isprobably due to the NIH syndrome again. Coupled with this is the lack of understanding about software ingeneral, and so methods of optimising parts of the process will not necessarily be taken into account.


8 of 10 11/28/2013 02:28

7. CONCLUSION

It is clear is that the software production will become an engineering discipline. In its current state it is notlacking in itself but in the lack of sound underlying principles, both for software and for the understanding ofthe problems to be solved. As to whether we can call Software Engineering by that name, the debatecontinues; what is sure is that the implications of the term go a long way to ensure that software meets ourexpectations of it, both as suppliers and clients. Software Development NEEDS to be an engineeringdiscipline, with all its implications.

REFERENCES

[] Brad J. Cox, Andrew J. Novobilski, Object Oriented Programming: An Evolutionary approach , 2ndEdition Addison Wesley 1991

[] Sue A. Conger, The new software engineering , 1st Edition 1994 International Thompson Publishing

[] Donald E. Knuth, Computer Programming as an Art , ACM Turing Award Lectures: The First TwentyYears, pp. 34-46, (C) 1987 ACM Press

[] M. F. Devon, Methodologie de Developpement des logiciels : lexperience IBM La Gaude , Conference aucampus Thompson le 29 Mai 1991

[] Brooks, The mythical man-month , 1975

[] Tom De Marco, Timothy Lister, Peopleware :

[] IEEE standard for developing software engineering processes.Craft or Science? Software Engineering Evolution

2 Responses

Crossing the Agile Development experts Total Immersion, on January 26, 2009 at 8:21 pm said:[...] Craft or Science? Software Engineering Evolution [...]

ReplyLooking back: Software is Art? Views from the bridge, on May 12, 2010 at 8:10 am said:[...] [[And the punchline: this was wri;en in 1994, as part of a bigger piece entitled Craft or Science -Software Engineering Evolution]] [...]

Reply

Blog at WordPress.com. The Digg 3 Column Theme.

Follow

Follow Views from the bridge


9 of 10 11/28/2013 02:28

Powered by WordPress.com


10 of 10 11/28/2013 02:28

Documents

Craft or Science? Software Engineering Evolution