8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson

8/6/2002 Safeware Engineering Corporation 1

The Safety Risk of The Safety Risk of Requirements IncompletenessRequirements Incompleteness

Jeffrey Howard

Patrick Anderson

8/6/2002 Safeware Engineering Corporation

2

Requirements IncompletenessRequirements Incompleteness

Many incidents and accidents have been linked to flaws in real-time embedded system software

Software-related errors are most often requirements errors, particularly incompleteness

A specification is incomplete if required behavior is omitted or subject to more than one interpretation


3

Completeness CriteriaCompleteness Criteria

Professor Nancy Leveson has compiled over 60 completeness criteria to address this problem, covering:

– Human-Computer Interface– Trigger Events– Robustness– Nondeterminism– Values and Timing– Data Age– Feedback– And More

Validated at JPL and used at Safeware

SpecTRM-RL (SpecTRM Requirements Language) enforces these criteria


4

Today’s Example AccidentToday’s Example Accident

The importance of the criteria is easily demonstrated when they are ignored

No one wants their embarrassing stories told in a conference session

Everything you see here is falseEverything you see here is trueThe ElectroShear 2000 Accident


5

ElectroShear 2000 SchematicElectroShear 2000 Schematic


6

ElectroShear 2000 Shearing PenElectroShear 2000 Shearing Pen

Shearing pen, where shearing is done

Entry and exit gates – Gate position sensors– Gate actuators– Gate locks

Four mechanical arms mounted with electric trimmers

Three sheep detection sensors– Digital camera– Weight plate– Thermal sensor

Trimmer head sensors– Wool sensor– Skin flush-fit

sensor


7

Normal OperationNormal Operation

The system begins with entry gate open and exit gate closed

Workers load a sheep and close the entry gate

At least two of the three sheep detection sensors agree on the sheep’s presence

The system shears, adjusting trimmer position using the skin flush-fit sensor

The wool detection sensor is ignored - the software detects its own completion

After shearing, the exit gate opens

Collect wool and repeat


8

The AccidentThe Accident

A technician replaced the trimmer blades in a pen, then greased the entry gate

While manually moving the gate, he lowered it to the point of closing it

The system exited standby mode and began a shearing cycle

The technician was caught in the pen and sheared

The system behaved erratically during shearing, and three of the four mechanical arms were damaged


9

Technician’s StatementTechnician’s Statement

“My next work order was pen #22. The guys working with it had complained that the entrance gate was moving slowly and making some noise. As long as I was there, I was supposed to replace the trimmer heads. They were overdue. I got there and the guys unloaded the sheep they were putting into the pen. They put the pen into standby, so I lifted the exit gate, disconnected the weight plate, and went in to replace the trimmer heads. After that, I sprayed some grease on the gate tracks and worked it by hand a little to get the grease spread out. The machine just went crazy on me. It was a close shave.”


10

The InvestigationThe Investigation

ElectroShear’s documentation jumbled requirements and design

Accident investigators used SpecTRM-RL to explore the system’s behavior

SpecTRM-RL uses text attributes and AND/OR tables to represent software behavior


11

SpecTRM-RLSpecTRM-RL


12

SpecTRM-RL (2)SpecTRM-RL (2)


13

Why did the system leave Standby Mode?Why did the system leave Standby Mode?

Gates do not require frequent maintenance

Maintenance procedures require the gates to stay open during maintenance

Designers didn’t anticipate entrance gate closings during standby mode

Entrance gate closing during standby mode moves the pen into loaded mode


14

Shearing Pen Mode LogicShearing Pen Mode Logic


15

Criterion: NondeterminismCriterion: Nondeterminism

“The behavior of the state machine should be deterministic (only one possible transition out of a state is applicable at any time.”

Automated tools can check this


16

Was the technician a ram?Was the technician a ram?

The system classified the technician as a sheep

Two of the three sensors must agree– Digital Camera– Thermal Sensor– Weight Plate

The camera mistook the human on all fours as a sheep

The software still had obsolete input data queued from the disconnected weight plate


17

Weight Plate InputWeight Plate Input


18

Criterion: Data AgeCriterion: Data Age

“All inputs used in specifying output events must be properly limited in the time they can be used (data age).”

In SpecTRM-RL, all inputs have an Obsolete value


19

Why was the exit gate open?Why was the exit gate open?

If the exit gate is open, the shearing cycle shouldn’t start

During the accident, it was open

No escape for the technician

When the system went into standby mode, exit gate position sensors were ignored

The system came out of standby mode with an incorrect system model


20

Exit Gate Position LogicExit Gate Position Logic


21

Criterion: State CompletenessCriterion: State Completeness

“The internal software model of the process must be updated to reflect the actual process state at initial startup and after temporary shutdown.”

SpecTRM-RL requires states to have an Unknown state value


22

What about the wool sensor?What about the wool sensor?

The wool sensor didn’t detect wool being sheared

That didn’t stop the shearing cycle

System engineers provided a wool sensor to detect the end of shearing

The software keeps track of shearing completion as progress along the planned shearing path

The software ignores the sensor, because it’s easier to detect the end of shearing as running out of planned shearing path


23

Criterion: Input Variable CompletenessCriterion: Input Variable Completeness

“All information from the sensors should be used somewhere in the specification.”

SpecTRM-RL has an “Appears In:” attribute to identify orphaned inputs


24

Why were the arms flailing?Why were the arms flailing?

Mechanical shearing arm motion became increasingly erratic

By the end of the accident, three of the four arms were damaged by the controller’s commands

The shearing arm fine-adjustment sensor doesn’t handle struggling humans well

The data bus was flooded with commands and telemetry


25

Criterion: Environmental CapacityCriterion: Environmental Capacity

“For the largest interval in which both input and output loads are assumed and specified, the absorption rate of the output environment must equal or exceed the input arrival rate.”

SpecTRM-RL’s attributes address timing behavior


26

Why couldn’t the operator help?Why couldn’t the operator help?

An operator finally noticed the calamity

The operator issued a stop command to the shearing pen

The shearing pen didn’t stop

The designers didn’t anticipate high communication load

The stop command is just another order on the bus

The operator had no way to know the order was lost


27

Criterion: Output FeedbackCriterion: Output Feedback This problem actually

touches on a number of criteria– Inadequate display of

state to operators– Inability to preempt

lower priority tasks– Lack of feedback

For the moment, focus on the lack of feedback to the operators

SpecTRM attributes on outputs make feedback paths easy to check


28

Why didn’t the entry gate open?Why didn’t the entry gate open?

When the operators realized the system wouldn’t shut down, they commanded the gate open

It didn’t open

Keeping gates closed during shearing is a safety feature

The command that closes the gate isn’t reversible.

No notice was given to the operator.


29

Criterion: ReversibilityCriterion: Reversibility

“Output commands should usually be reversible.”

SpecTRM-RL outputs have an attribute linking to the output that reverses their command


30

Investigation FindingsInvestigation Findings

There was no operator error in this accident.

There were no component failures in this accident.

Even the software didn’t “fail.” It met its requirements, such as they were.

The Electroshear 2000 was found to be unsafe.

The culprit cited was the shearing pen control software.

Software problems stemmed from incomplete requirements.


31

Completeness Criteria (2)Completeness Criteria (2)

The ElectroShear accident demonstrates several completeness critera– Nondeterminism– Data Age– State Completeness– Input Variable

Completeness– Environmental Capacity– Output Feedback– Reversibility

Consideration of these criteria could have prevented and/or reduced the severity of the accident


32

SummarySummary

The example may be fanciful, but the problems illustrated are quite real

The completeness criteria were compiled from decades of research, accident and incident reports, and specification review

SpecTRM-RL builds the criteria into a state of the art, analyzable, and executable requirements language


DiscussionDiscussion

And/Or Questions


The EndThe End

Documents

8/6/2002Safeware Engineering Corporation1 The Safety Risk of Requirements Incompleteness Jeffrey Howard Patrick Anderson