CSCE 431: Software Lifecycle Some material from B. Meyer, M. Oriol, B. Schoeller, ETH Zurich

CSCE 431: Software Lifecycle

Some material from B. Meyer, M. Oriol, B. Schoeller, ETH Zurich

CSCE 431 Software Lifecycle

Outline

• Early ideas about improving software quality

• Software development process

• Process models

• Process maturity

• Organizational structure that supports a process

• Summary


Idea: Better Languages

• Working with machine languages was tedious• Especially tedious to write efficient programs

• High accidental complexity

• No portability, each machine programmed differently

• As a consequence, low productivity• Extreme example:

• Microcode developers typically wrote 2 lines of debugged code per day – 1 before lunch, 1 after lunch

• Idea: translate programs from a higher-level description

• Backus, 1954: FORTRAN (FORmula TRANslator)• First HLL implementation• 20x reduction in statements


Idea: Better Languages

• Idea of a better language applied many times, still being applied• FORTRAN as an example:

• FORTRAN I (1957)• 32 instructions

• FORTRAN II (1958)• Added procedures, COMMON, arithmetic IF – my first language, on punched cards!

• FORTRAN IV (1962)• Added logical expressions, logical IF – my first college language, still punched cards!

• FORTRAN 66• ANSI standard

• FORTRAN 77• ELSE, DO WHILE, (previously had to use GOTO)• Taught in freshman engineering until 1999

• FORTRAN 90, 95• Modules, abstract data types, dynamic memory allocation, FORALL vectorization

• FORTRAN 2003, 2008• Objects, procedure pointers

• FORTRAN 2015 – C interoperability, new parallel features

• Other successful languages have had similar progress over the decades


Idea: Structure Programs Better

• Böhm, Jacopini, 1966• Any program can be rewritten as a structured program• Using only sequence, iteration, selection control constructs• One entry point

• Dijkstra, 1966: Argued against the use of GOTO in his Notes on Structured Programming

• Structured programming makes it more feasible to develop convincing arguments of a program’s correctness (proofs)

• Adaptability and manageability of a program improves• E. W. Dijkstra, Go To Statement Considered Harmful, CACM,

Vol. 11, No. 3, March 1968, pp. 147-148 http://www.cs.texas.edu/~EWD/ewd02xx/EWD215.pdf

http://www.cs.texas.edu/~EWD/ewd02xx/EWD215.pdf


Managing Complexity: First Ideas of Modularity

• Top-down design

• N. Wirth’s Stepwise Refinement• Decompose problem into high-level units• Decompose each unit recursively to smaller units• When at code-level for each unit, stop

• Exemplified in• N. Wirth, Program Development by Stepwise

Refinement, CACM, Vol. 14, No. 4, April 1971, pp. 221-227, http://sunnyday.mit.edu/16.355/wirth-refinement.html

http://sunnyday.mit.edu/16.355/wirth-refinement.html

http://sunnyday.mit.edu/16.355/wirth-refinement.html


Stepwise Refinement Example

• A program that removes the “single line” comments from C++ source code

• Read the file• Open the file• Read line-by-line and put in an array of strings

• Remove the comments• Iterate through the array

• In each line, look for first occurrence of “//”• Remove the rest of the line

• Write the modified file back• Iterate through the array, store each line

• Code


Pros and Cons

+ Modularity

+ Big picture visible

+ Systematic, nothing missed

– Testable only at the end

– Code reuse unlikely

– Simplistic

• Not a realistic approach for today’s SW scale


Bottom-Up Design

• Lowest-level subsystems specified and implemented first

• Gradually composing and building larger subsystems

• System emerges from smaller subsystems

+ Reuse possible

– “Big picture” may get lost

• In practice, both approaches in use• Design, mostly top-down• Component/algorithm use and composition from libraries, bottom-up

• Highly-productive programmers tend to mix top-down and bottom-up methods all the time, working wherever most progress can be made

• I use the term “top-down design with bottom-up feedback”• Used in chip design too


Object-Oriented Design

• Simula 67, 72, Dahl and Nygaard• My first OO programming language, 1978

• Point of OO: objects modularize data and code by co-locating code and data relevant to each other

• Reuse (hopefully)• Modularity• Encapsulation – Dave Parnas’ information hiding• Visual notation

• Ideally modularization of design matches modularization of code

• Eiffel: specification and code within the same formalism


About the Early Ideas

• Languages have surely improved, but a language alone is not a full answer to SE problems

• No silver bullet• Simula was REALLY SLOW

• Process ideas in small scale, possibly useful, but certainly not enough

• Design ideas helpful, but not sufficient

• More complete picture of the activities of software construction and the artifacts produced, and of organizing them, is necessary


Outline



• Process models



• Summary


Process

• Loose definition: a series of activities, involving constraints and resources, that produce an intended output

• Prescribes activities• Uses resources subject to constraints to produce (intermediate or

final) products• May be composed of subprocesses• Each activity has entry and exit criteria, so that it is known when

the activity begins and ends• Activities organized (in a sequence, usually) so that it is clear when

one activity is performed in relation to the other activities• Guiding principles that explain the goals of each activity• Constraints or controls may apply to activities (time limits, budget

limits)


Process

• When a process is for building a product, it is often called a lifecycle

• A software lifecycle describes the life of software from an idea to a deployed and maintained application


What is Your Personal Process?

• Code and run

• Code, run, fix defects and try again

• Design, code, run, fix defects and try again

• Design, code, test, fix, run

• Design, code, test, fix, document, run

• Specify, design, code, test, fix, document, run

• Requirements, specify, design, code, test, fix, document, run• Contains ~all components of established SW processes• Different component organizations lead to different processes

• When not organized (with minimal specify, design, document phases), we have the “no-method” process


No-Method

• Sketch an idea

• Code immediately

• Get feedback from self or customer

• Modify

• Repeat ad infinitum


No-Method

• Does not scale

• Other processes are reactions against no-method

• Still often used• Especially in crises when the chosen process breaks

down and panic and desperation take over• In projects where developers have domain knowledge

(experts of user domain as well)

• Not good to follow, but good to recognize, and to know predicted outcome

• No-method process is often practiced in CSE education


Best Software Process?

• Proponents of a particular SW process believe that it reduces the discrepancies of what a program is supposed to do and what it does (at an acceptable cost)

• Success in one setting may convert user to “believer”, but application in another environment/project/team may disappoint

• A really good hammer does not make everything a nail


Best Software Process?

• Conjecture: there is no universal best SW process

• Approach and tools depend on problem, team, quality requirements,…

• An SE skill is ability to select appropriate approach and tools


Outline



• Process models



• Summary


Process Models

• Next: brief introduction to some well-known process models that have had an impact on SE

• Practical SW processes for particular teams and applications can be tailored from process models


Waterfall Model

• Originally suggested by Royce, 1970• Proposed as an initial (and flawed) concept, argued that

feedback from each phase to its predecessor needed

• Popular in “the beginning”, was sometimes required, e.g. DoD contracts

• Archetype of all inflexible sequential models• But still used in many organizations

Requirements

Design

Implementation

Verification

Maintenance


Waterfall Model with Feedback

• Feedback may go to any predecessor phase• E.g. maintenance problem may lead to new

requirements

Requirements

Design

Implementation

Verification

Maintenance


About the Waterfall Model• Context: computer time was scarce, attempt to minimize

coding time• Focus on documents, carefully reviewed after every phase

• Every phase has completion criteria, document “sign off”

• Too document-intensive• Documentation update effort >> code update effort• See: The Mythical Man-Month, Fred Brooks

• Long development cycles• Models green field SW development

• Reality: SW developed with a lot of iteration/versions

• Shortcoming: treats SW development as manufacturing process, not problem-solving process

• Curtis et al, On Building SW Process Models Under the Lamppost, ICSE, 1987, pp. 96-103.


Waterfall Model and Requirements

• Key assumption: possible to define/control requirements fully and precisely

• Reality: requirements evolve and emerge as understanding of the system increases and environment evolves

• Estimate (Bernstein & Yuhas): 40-60% of requirements can be known during requirements elicitation phase

• Rest are emergent requirements (Barry Boehm)


Risk in Waterfall Model


More Phases in More Recent Waterfall Model


Variation in Waterfall: V-Model

• Development phases → code → test phases• Each development phase has corresponding test phase

• Test planning a parallel activity • Intent: mitigate risk, improve & guarantee quality, reduce total

project/lifecycle cost, improve communications between all stakeholders


Variant of V-Model

• Implementation and design guided by testing


Sequential vs. Iterative Models

• Difficulties understanding a SW system before it has been built lead to iterative models

• Many variations, basic idea is that certain process phases are repeated, until convergence


Evolutionary Model

+ Prototype can be evaluated and design changed

+ User can evaluate prototype and give feedback

– Code twice may be too much to ask

– Still a sequential model


Prototype Model

• Iterate until convergence


Spiral Model


Spiral Model

• Iterations typically 6 months to 2 years

• Generates a succession of prototypes (or versions)

+ Focus on continuous assessment of risk

+ Estimates become increasingly accurate

+ Manages change

+ SEs start their work earlier

– Not a “lean” process - ~overkill for small projects


Processes Abound

• Boehm’s WinWin Spiral

• Development Plan approach (L. Bernstein)

• RUP (Rational Software Corp. – now IBM)

• …

• Processes arise from• Organization’s need to have control over SW

system development, accountability for delays• Desire to guarantee SW quality• Desire to rely on codified process to help

organization plan/manage resources


Agile Processes

• Current fashion

• Probably for good reason – success

• Misconception: agile process == no-method process

• “Those who brand proponents of … agile methodologies as hackers are ignorant of both the methodologies and the original definition of the term hacker” (Bernstein & Yuhas)


Agile Processes

• Everything in moderation• Embraces modeling, but not to be filed in a dusty

corporate repository• Embraces documentation, but not to waste paper in

never-maintained/used tomes• Insists on planning, but recognizes its limits in

changing environments

• Core emphasis: schedule• Driver organizations to schedule containment


Examples of Agile Processes

• Extreme Programming (XP)

• Scrum

• Crystal

• Dynamic Systems Development Method (DSDM)

• Lean Development

• Feature-Driven Development (FDD)


Example: Scrum


Example: Scrum


Some Typical Practices

• Test-first programming

• Regular refactoring

• Continuous integration

• Pair programming

• Shared code (no-one claims ownership)

• Coding standard to which everyone adheres

• A common work area

• Customer on site

• User stories

• “Manager-servant”

• Timeboxing


Iterative Development

• Each iteration is a mini-project

• Iteration goal: a release• stable, integrated, tested, partially-complete system

• All produced code from all teams integrated into a release

• Most iteration releases internal (to the team)

• During an iteration, no changes from external stakeholders

• Can make changes between iterations


Some Typical “No-Nos”

• Big up-front requirements specifications

• Binding documents• More of a collaboration agreement with the

customer than a legally binding contract of a feature list and specs

• Diagrams (e.g. UML)

• Non-deliverable products• Iterations should be deliverable, even if only

partially-complete


Why Minimize Up-Front Requirement Specification Effort?• Details too complex for people to grasp

• Many details only revealed during development

• Stakeholders not sure what they want

• They have difficulty stating it

• As product develops, stakeholders will change their minds

• External forces (environment) cause changes and extensions


How Many Requested Features Are Used?

• Jim Johnson, 2002:Never 45%Seldom 19%Occasionally 16%Often 13%Always 7%

• Reinertson, Thomke, 1998:• Requirements always change• Complete requirements to developers in 5% of cases• On average, design starts with 58% of requirements

specified in detail


What is Needed for Architecture Design?

• Agile answer: • Architect needs:

• Most nonfunctional or quality requirements (internationalization, load, response time, etc.)

• Subset of functional requirements

• to design a good core architecture


Agile Manifesto

• Individuals and interactions >> processes and tools

• Working SW >> comprehensive documentation

• Customer collaboration >> contract negotiations

• Responding to change >> following a plan

• http://agilemanifesto.org

http://agilemanifesto.org/


Principles Behind the Agile Manifesto

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

• Business people and developers must work together daily throughout the project.

• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

• Working software is the primary measure of progress.

• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

• Continuous attention to technical excellence and good design enhances agility.

• Simplicity–the art of maximizing the amount of work not done–is essential.

• The best architectures, requirements, and designs emerge from self-organizing teams.

• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.


Intermediate Summary

• Evolution of languages has helped to reduce number of programming defects

• Implementation effort has been reduced thanks to abstractions, libraries, better languages

• Result: projects have grown to tackle more complex tasks

• Processes have emerged as an attempt to make SW projects more predictable

• Brief look at early process ideas• Brief overview of some processes in use today


Outline



• Process models



• Summary


Maturity of a SW Development Process

• A software development process is mature• If the development activities are well defined and• If management has some control over the quality, budget

and schedule of the project

• Process maturity is described with• A set of maturity levels and• The associated measurements (metrics) to manage the

process

• Assumption:• With increasing maturity the risk of project failure

decreases

• CMM: Capability Maturity Model (SEI, Humphrey)


CMM Levels1. Initial Level

• Also called ad hoc or chaotic, may require “heros”

2. Repeatable Level• Process at least documented

3. Defined Level• Process is institutionalized (sanctioned by management)

4. Managed Level• Quantitative, agreed upon metrics, provide feedback for resource

allocation (process itself does not change)

5. Optimizing Level• Process allows feedback of information to change the process itself

• Danger of viewing CMM as form of accreditation (also ISO 9001, quality certificates, …)


Experimental Evidence: Boeing

• SW-CMM for software process improvement in use in Boeing since 1991

• Over 200 organizations assessed

• Over 11,000 people trained in SW-CMM

• Several higher levels of maturity organizations (Level 3, 4 & 5)

• PSP/TSP techniques in use• Humphrey’s Personal/Team Software Process


Evidence That CMM Works

• Effort Estimates


Evidence That CMM Works

• In reports about Boeing experiments, all measured quantities positive

• Reuse• Productivity (reduced staff support per system)• Quality (fewer post release defects)• Schedule control (reduced cycle times, schedule

estimate variance)• Employee satisfaction

• Some consider level 3 sufficient, but increase continues all the way to level 5


Drowning in Processes

• S. A. Sheard, Evolution of Frameworks Quagmire, Software Management, 2001.


Outline



• Process models



• Summary


Organizational Structure• A dual challenge of designing a process is to design the organization to

execute the process

• Two main possibilities:• A functional organization

• Each functional group specialized to a particular task

• Serves many projects

• + People work on what they are good at

• - Communication cost high

• - Commitment to product, reward of accomplishment possibly low

• A project organization• Each product served by one group

• - Possibly less efficient

• + Good for humans (personal growth, less burn out, more interesting)

• + Fosters commitment, responsibility

• No right answer. Perhaps• Functional organization beneficial in more established product, and• Project organization beneficial in a new less defined project.


Outline



• Process models



• Summary


Summary• No best process, (almost) any process will help

• Forces team to write things down• Helps with “writer’s block” — lists of questions that are worthwhile

going through even though many irrelevant to particular project• What also helps, probably more than any process, is good practices

like automated tests, automated builds, version control, good release model, automated documentation generation, defect tracking,… discipline and automation in general

• Two contradictory risk mitigation approaches• Plan everything carefully, because the later a problem is discovered,

the costlier (exponentially) it is to fix• Do not plan too far ahead (in details), because it is impossible to get

the requirements right, thus effort will be wasted

• Experimental data suggests that improving the maturity of an organization’s software process improves results in practically all metrics

Documents

CSCE 431: Software Lifecycle Some material from B. Meyer, M. Oriol, B. Schoeller, ETH Zurich