Formal Versus Agile: Survival of the Fittest? (Paul Boca)

Agile versus Formal Methods: Survival of the Fittest?

Paul Boca

Collaborators

• Sue Black

• Mike Hinchey

• Jonathan Bowen

• Jason Gorman

(Scrum mistress)

(Formal Methodist)

(Zzzzzzzzzzz)

About Me

• PhD in program transformation (categorical stuff) • Hardware Compilation R&D (industry light)• High-level synthesis (start up)• Was R&D manager at UK static analysis vendor• Currently Quality Engineering Manager (SME)• Formal Methods enthusiast• Organizer of formal methods seminars

Outline

• Agile background• Formal Methods background• Co-existence of Formal Methods and Agile• The Cost of Agile• Summary

But first...

Quiz #1

• First Prize: Toyota Prius• Second Prize: Sony PlayStation 3• Runner up: Toyota Prius and PlayStation 3

Quiz #1

• 63,000 ?

• 2,000+ ?

• 700 ?

• 280210 ?

The number of bugs in Windows 2000

The number of pages of Z specifications in a certain air-traffic control system

Number of lines of code written per month by an agile developer

Sony PlayStation 3 bug revealed!

What do each of the numbers below correspond to

Agile Manifesto

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

http://agilemanifesto.org/

Approaches + Techniques

• Approaches– Extreme Programming– Scrum– DSDM

• Techniques– Pairwise Programming– Test Driven Development

Quiz #2

Agile is about producing software quickly

Agile is about being responsive to change

Some Agile Success Stories

But Agile is not perfect

• Lacks comprehensive documentation

• Writing tests upfront is great, but it’s not possible to test everything

• Rapidly changing requirements can be hard to trace.

• Refactoring code can introduce defects

We’ll return to these later...

Formal Methods

The application of a fairly broad variety of theoretical computer science fundamentals, in particular logic calculi, formal languages, automata theory, and program semantics, but also type systems and algebraic data types to problems in software and hardware specification and verification

http://en.wikipedia.org/wiki/Formal_methods

Formal Methods Timeline

Z1977

B1986

CSP1978

Circus2000

DbC1986

1970 1980 1990 2000 2010

X machines1974

VDM70s

Alloy1997

FM + Agile2003

not complete but sound

Formal Methods Success Stories

Formal Methods Success Stories

Quiz #3• Agile will phase out Formal Methods

• Formal Methods will phase out Agile

• Formal Methods and Agile can coexist

What’s more, Agile and Formal Methods need one another

Agile is not perfect




• Refactoring code can introduce defectsIt's like déjà vu all over againYogi Berra






Why we need documentation

• Code bases need to be maintained and extended

• Some systems will be in operation for decades• Knowledge transfer to prepare for personnel

leaving– Pairwise programming = Knowledge in silos

Design by Contract to the rescue

• Preconditions and postconditions provide information on the intent of a function

• Generating loop invariants can help with retrospective documentation

• Contracts can be checked (semi-)automatically






To test or not to test?

• Writing tests is definitely a good thing• When is enough enough? • Can only test the finite, and you can’t test

everything• Testing sequential programs is hard enough,

so what happens when parallelism is introduced?

• What about safety critical applications?

Formal Methods to the Resue

• Automated generation of test cases (e.g. Perl, Python, Functional Programming languages)

• Use static analysers (e.g. Coverity)– Finds issues in code, but can’t find everything– Complements dynamic testing

• Use model checkers (e.g. FDR)• Mutation testing

– Modify the code base introducing “mutants”– See whether the test suite “kills” the mutant– Helps to identify gaps in test suites




• Rapidly changing requirements can be hard to trace





• Rapidly changing requirements can be hard to trace


Refactoring Code

• Refactor to– Remove code smells– Beautify code

• IDEs can carry out certain refactoring steps automatically.

• Refactoring to change underlying algorithms is a manual process – needs a “Eureka”

• Manual refactoring is error prone

Refactoring = program transformation

• Correctness preserving transformations are present under the hood of refactoring systems

• New transformations can be added

• Proved sound

• Completeness issues

• Decidability issues

Avoiding Continuous Disintegration!

• Continuous Integration– Frequent checkins to source control system– Build frequently (triggered after commit)– Run tests– Fix defects immediately

Avoid the wrath of Agnes!!

Continuous Integration (CI)

Subversion server

build server

test server

Feedbackmechanism

commit

commit

commit

check

Model checker

TheoremProver

Analysis tools

Enabling CI+FM

• Faster tools• Harness multi core architectures• Deploy machines on demand in the cloud

– No need to use physical machines– Provision servers in the cloud to meet demand– Cost effective solution

The Cost of Agile

Time Time

Eff

ort

Eff

ort

Formal development XP development

The Cost of Agile

Time Time

Eff

ort

Eff

ort

Formal development Scrum development

The Cost of Agile

Time Time

Eff

ort

Eff

ort

Formal development DSDM

Agile and FM: survey

• We’re putting together a survey to gain further insight into how formal methods and agile can work together

• Areas of interest include:– Suitability of FMs Suitable Agile approaches– Application areas Success Stories– Demographics Reasons for Failure– Team sizes Team skills– Tools used Lessons learned– Defect rates Maintainability of code

Summary

• Agile has drawbacks, but these can be overcome with formal methods

• Formal methods opens up areas for agile, e.g. Safety Critical Systems

• Continuous Integration can be supplemented with formal methods, harnessing cloud computing

• Dispelled myths about the costs of Agile and Formal Methods

• Surveying the landscape: formal methods and agile usage in industry

Take Away Messages

Formal Methods can add value in the agile domain, acting as a sanity check and safety net

Formal methods provides reliability, assurance and good documentation, whilst agile provides flexibility, customer satisfaction and tangible progress

Agile and FM: survey

• If you would like to participate in the survey, please email me at [email protected]

• I look forward to hearing from you

• Thank you in advance.

mailto:[email protected]

FM + AM 2010

• 17 September, Pisa, Italy

• One of the workshops at SEFM 2010:– http://www.sefm2010.isti.cnr.it/

• See http://fm-am-2010.tripod.com/index.html for further details of the workshop

• SUBMIT SUBMIT SUBMIT SUBMIT SUBMIT SUBMIT (even though you’ll be reducing the chances of my paper being accepted!)

http://www.sefm2010.isti.cnr.it/

http://fm-am-2010.tripod.com/index.html

<shameless plug/>

20% reduction for delegates.

If you are interested, please pick upa flyer.

All royalties ploughed back into seminarseries

AGILE

FM

The perfect partnership

Technology

Formal Versus Agile: Survival of the Fittest? (Paul Boca)