Embed Size (px)
Citation preview
1
Software Reliability Assurance for Real-time
Systems
Joel Henry, Ph.D.
University of Montana
NASA Software Assurance Symposium
September 4, 2002
2
Overview
• System development
• Testing problems
• Solution approach
• Results and conclusions
• Practical application
3
• Controls devices that control:– Wind generator– Model support– Tunnel atmosphere
• Emphasizes reliability and safety
• Utilizes multiple development strategies
• Based on simple structure
System DevelopmentExample: Wind Tunnel Software
4
System Development Simple Structure
ControllingComputers
ExternalDevices
ALGORITHMSample InputsRun SoftwareUpdate Outputs
5
• Engineer builds graphical model in MATLAB
• Models enter simulate-debug-simulate-debug phase
• Engineer auto-generates source code• Source code is compiled, linked, and then
deployed• Hardware/software integration begins
System Development
6
• Recall simple model– Input variables – sampled over time– Outputs variables – produced over time– Sample time – variable or set frequency
• Consider test requirements– Input file/matrix– Output file/matrix– Analysis tools
Testing ProblemsTest Size
7
• Consider an example– 100 input variables
– 50 output variables
– 100 millisecond sample time
• Assume you want to test a one hour operation period– 100 inputs*10 per second*3600seconds = 3,600,000 values
– 50 outputs*10 per second*3600seconds = 1,800,000 values
• Ignore issues of useful inputs and defect detection
Testing ProblemsTest Size
8
• Domain determinants– Input variable – minimum, maximum, and accuracy– Output variable – minimum, maximum, and accuracy
• Consider test requirements– Input file/matrix with all possible values for input– Output file/matrix much more complex problem
Testing ProblemsDomain Coverage
9
• Consider an example for input variable– Input variable for pressure in a tank
• Min – 0
• Max – 999.999
• Accuracy – 3 (decimal places)
– 1000*1000 = 1,000,000 possible values
• Ignore issues of legal sequencing and combinations
Testing ProblemsDomain Coverage
10
• Automation to:– Generate large input matrices/files– Perform simulation and/or test autogenerated code– Analyze output matrices/files
• Methods to:– Evaluate domain coverage– Aid debugging– Evaluate results
Solution ApproachOverview
11
Command and Control Algorithm
Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results
MATLAB/Simulink Environment
Source Code
Executable Code
Verification and Validation Methodology
Solution Approach
Suite of testing tools
12
Generate Tests
Simulate Model
Test Auto-code
Detect Faults
Evaluate Results
Suite of testing tools
Verification and Validation Methodology
Command and Control Algorithm
MATLAB/Simulink Environment
Model Information
Test Data
Test Results
ExecutableCode
Test Data
Test Results
Solution Approach
13
Solution Approach
Generate Tests
Simulate Model
Test Auto-code
Detect Faults
Evaluate Results
Suite of testing tools
Verification and Validation Methodology
Data Graphs
Raw Value Files
Completeness, MTTF,Reliability File
14
• Execute multiple tests– Evaluate testing effectiveness– Track trends in model reliability
• Automate and evaluate– Specify effectiveness and reliability goals– Evaluate on a per test case basis– Track through testing phase over all tests
Results and Conclusions
15
Results and ConclusionsExample – Multiple Tests
System MTTF
0
1000
2000
3000
4000
5000
6000
1 2 3 4 5
Test Runs
MT
TF
(Sec
on
ds)
Mean Time To Failure
Test Interval = 1000 ms
16
Results and ConclusionsExample – Multiple Tests
Bucket Coverage Improvement
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10
Input Percentiles
No
of
Hit
s Test Run 1
Test Run 2
Test Run 3
Test Run 4
Bu
cket
Co
vera
ge
(%)
17
Results and ConclusionsExample – Multiple Tests
Prob Of System Exception
0
0.2
0.4
0.6
0.8
1
1 2 3 4 5
No Of Test Runs
Pro
bab
ilit
y o
f S
yste
m
Exc
epti
on
Prob Of Exception
18
Results and ConclusionsExample – Multiple Tests
Output Variables
0
0.1
0.2
0.3
0.4
0.5
1 2 3 4 5
Test Runs
Pro
babi
lity
of E
xcep
tion
in a
n ou
tput
var
iabl
e Prob Of Excp in Output 1
Prob Of Excp in Output 2
Prob Of Excp in Output 3
Prob Of Excp in Output 4
Prob of Excp in any output
19
• Can do domain testing supported with automation
• Can set quantitative goals
• Can evaluate progress toward goals
• Can measure MTTF, domain coverage, confidence percentages, and reliability
• Can create an organizational history
Practical Application What?
20
• Invest in automation
• Integrate domain coverage with application specific testing
• Establish goals and collect data
• Calculate MTTF, domain coverage, and reliability
• Use common sense with quantitative data
Practical Application How?
21
Questions and Contact Info
• Joel Henry– [email protected]– MATT and RATT– http://www.cs.umt.edu/RTSL/design992/links/index.htm
• MATLAB users -
