Software Engineering Issues in System Dependability

Dr Martyn Thomas CBE FREngDIRECTOR, MARTYN THOMAS ASSOCIATES

60 Years of Computing 40 Years of Software Engineering

The first stored program computer ran

its first program in Manchester, UK in

June 1948.

The first commercial software was the

Bakeries Valuations for the Lyons

Electronic Office (UK 1951)

The first software engineering

conference was held in Garmisch,

Germany 40 years ago in October

1968

Privacy, Security and Software Engineering

To process and manage confidential information, we need secure, dependable

computer systems.

Creating secure and dependable systems cost-effectively needs strong software

engineering.

Commercial software development is usually not good enough to create secure

systems with very few errors.

A Study by the US National Security Agency found that methods based on mathematics

can provide a cost-effective solution.

The Challenges of Complex IT Projects

By Secure Systems I mean systems that provide the required service and do not permit unauthorised access or interference.

By Dependable Systems I mean systems that can be trusted to operate as required, because there is strong evidence that they do not contain critical errors.

Most commercial software contains many errorsResearchers have reported fault densities of 1 to 30 defects per KLoC (thousand lines of source program) in commercial software in service.

The UK Ministry of Defence found over 20 defects per KLoC in the operational software in the C130J military transport aircraft (including many safety critical faults).Sources: German & Mooney, Air Vehicle Static Code Analysis - Lessons Learnt. 2001.

Pfleeger & Hatton, IEEE Computer, pp33-42, February 1997.Fenton & Ohlsson, Quantitative Analysis of Faults and Failures in a Complex Software System,

http://www.md.kth.se/RTC/SC3S/papers/FentonOhlsson105280_final.pdf

Security errors have caused great damage“Several individual viruses and worms have caused events where damage was assessed at over $1billion each”

Source: Software for Dependable Systems - Sufficient Evidence? National Academies Press, 2007

Many security vulnerabilities arise from simple programming errors, such as “buffer overflows” where the length of input into an array is not checked and this allows a malicious user to overwrite the program.

Testing alone can never be enough to show that a program is dependable

To test all possible paths with all possible data is completely impractical.

Finding bugs and fixing them cannot lead to secure, dependable software.

Critical software errors may remain hidden for decades.

Testing software tells you that the tests work – not that the software works

Continuous behaviour means you can interpolate between test results

Discrete behaviour means that you can’t!

An Experiment by the NSA

A recently published experiment by the US National Security Agency shows that the use of mathematically formal methods can produce dependable software, cost effectively.

TOKENEER ID StationProtectedEnclave

Portal

Display FingerprintReader

TokenReader

TISAlarm

Admin

Source: Slides by Randolph Johnson, NSA

The Tokeneer experiment

• Praxis developed software for the Tokeneer ID Station using their Correct by Construction (C by C) methods

• NSA asked an independent team to find errors• Praxis then trained a group of NSA staff in C by C• The NSA staff modified the TIS to add new functions• All the specifications, designs, tests, analyses, proofs and

other documentation were collected• Productivity was analysed and compared with previous

projects

Correct by Construction Software Engineering source: Peter Amey, Praxis HIS

A combination of formal methods and agile development

Precise notations and incremental development

Three basic principles:

• Make it hard to introduce errors (Z, SPARK)

• Find any errors almost immediately (SPARK Examiner)

• Generate evidence of dependability throughout development, as a natural by-product of the process

C by C Development

Source: Peter Amey, Praxis HIS

C by C Guiding Principles

Write right

Step, don’t leap

Say something once

Check here before going there

Explain your decisions

Use the right tools for the job

Source: Peter Amey, Praxis HIS

Write Right

Capture information, requirements, designs and code unambiguously and accurately.

Use notations with strong semantics

Step, Don’t Leap

Make each development step semantically small, to reduce errors and improve checking.

Say something once

Make each document have a clear purpose and write information in only one place. This aids clarity and avoids inconsistency.

Check here before going there

Verify each step before moving on. Check for consistency against something else – usually the previous design step.

Explain your decisions

Document design decisions and explain why they are justified – that will help you be sure they are right, and help you and others to understand them later.

Use the right tools

Use the appropriate verification method for each property, whether tool-supported proof, informal peer review, static analysis or testing.

Tokeneer was written in SPARKSPARK enforces type security and enables deep static

analysis:

• no buffer overflows

• complete elimination of several classes of errors

• complete analysis of program flow and data flow

• SPARK formal annotations make it possible to prove that the software implements the specification

http://www.praxis-his.com/sparkada/

Statistics of Tokeneer SystemSPARK

program

Source Lines

Productivity

LOC/day

Core 9,939 38

Support 3,697 88

Defects found

by NSA

reviewers

Zero

Source: R Johnson, NSA

Statistics of Tokeneer SystemSPARK

program

Source Lines

Productivity

LOC/day

Spark

annotations

Core 9,939 38 16,564

Support 3,697 88 2,240

Defects found

by NSA

reviewers

Zero

Source: R Johnson, NSA

C by C is a practical approach

The NSA found that the Correct by Construction methods could be successfully taught to postgraduate engineers in a few weeks.

Summer intern students successfully added new features to the Tokeneer ID Station using C by C, with high productivity and zero defects

All the Tokeneer materials have been publishedhttp://www.adacore.com/tokeneer

The materials allow the entire project to be repeated, including all the static analysis steps and all the proofs.

The NSA has licensed the materials under the GPL (GNU General Purpose Licence).

IT projects often overrun

IT projects often overrunStandish “Chaos Chronicles” (2004 edition):

• 18% of projects “failed”; (cancelled before completion)• 53% of projects “challenged” (operational, but seriously over budget and/or over time with

fewer features or functions than initially specified)

Typical Standish figures:

• Cost overruns on 43% of projects; and• Time overruns on 82% of projects.

Oxford University/Computer Weekly 2003 study:

• 10% of UK projects “failed”; and• 75% of UK projects “challenged”.

Getting the Requirements rightIn the UK, many Government IT projects have failed.

Our software industry is very optimistic; so are our politicians.

This can lead to “irrational exuberance” (Alan Greenspan’s phrase)

There is no such thing as an IT Project - only Business Change enabled by IT. The IT costs are typically < 50%

The Royal Academy published a study in 2004: The Challenges of Complex IT Projects which recommended the use of systems architects

How do you get the right technical solution to a business requirement?

USE AN ARCHITECT!

http://www.raeng.org.uk/news/publications/list/reports/Complex_IT_Projects.pdf

A specialist in requirements, high-level designand working with engineers

Summary• Software that needs to be secure must be dependable• Dependability can only be achieved with the use of mathematically

formal methods• Formal methods are now practical and cost-effective for use in

commercial and industrial software development• The Tokeneer experiment allows software developers to benchmark

their own methods against Correct by Construction and to experiment with developing near zero-defect software, at no increase in cost.

• Our engineering Academies need to promote the use of scientifically valid methods and tools, to reduce the number of project failures and security disasters.