Upload
lionel-briand
View
58
Download
0
Embed Size (px)
Citation preview
.lusoftware verification & validationVVS
Automated Testing of Hybrid Simulink/Stateflow Controllers
Industrial Case Studies
Reza MatinnejadShiva NejatiLionel BriandSnT Centre/University of Luxembourg
Software Development in Automotive
• Software development is largely model-based
• Automotive software models have dynamic behaviors
• Mathematical models capturing plants/hardware
• Controllers
2
Two Types of Controllers
• Open loop controllers
• Closed loop controllers
3
Controller
Actuator Sensor
Plant
Input
Controller
Actuator
Plant
Input
Disturbances
Disturbances
Open Loop vs Closed Loop• Closed loop controllers – PID controllers
• More expensive
• More accurate and self-adaptive
• Always present in large and critical cyber-physical systems
• Open loop controllers – State-based models
• Less expensive
• Often controls timing behaviors
• Is combined with closed-loop controllers4
Simulink/Stateflow Models
• Heterogeneous
• Continuous behavior
• Are used for
• simulation
• algorithm design testing
• code generation 5
Time-ContinuousSimulink Model Hardware
Model
Network Model
Existing Simulink Testing Tools• Control theory techniques
• Synthesis of linear PID controllers
• Automated test case generation
• Based on (formal) assertions or structural code coverage
• Automated verification
• Model checking or theorem proving
6
Limitations
• Automotive models are rarely linear
• Automatable test oracles may not be available or sufficient
• Test oracles are in many cases manual
• Structural coverage may not help reveal faults
• White box approaches have incompatibility issues
• Scalability issues7
Black Box Search Based Testing of Simulink
8
Solution Generation
Fitness computation
• Explorative • Exploitative
Model InputSpec
ModelSimulation
InputSignals
OutputSignal(s)
Input Signals
• Simulate the model• Compute Fitness functions
on outputs
SimulinkModel Fitness
values
ModelSimulation
InputSignals
OutputSignal(s)
[SSBSE 2013, ASE 2014, ESEC/FSE 2015,IST J 2015, ICSE 2016]
Fitness Functions – Closed Loop• Generic requirements
• Stability, responsiveness and smoothness
• Maximizing (quantitative) fitness functions to generate failure
9
Initi
al D
esire
d(ID
)
Desired ValueI (input)Actual Value (output)
Fina
l Des
ired
(FD
)
timeT/2 T
Smoothness
Responsiveness
Stability
Fitness Functions – Closed Loop• Specific requirements
• E.g., ``The contact between caliper and disk should occur within 32ms’’
caliper position à disk position à
10
⌃[0,32]⇤((x x0 + ✏) ^ (x � x0 � ✏))
x
x0
Min{Max{Max{|x(t)� (x0 + ✏)|, |x(t)� (x0 � ✏)|}}t0tT }
Translation [Abbas et. al. TECS 2013]
Fitness Functions – Open Loop • Failure patterns
• Output diversity
11
0.0 1.0 2.00.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time Time
0.0
0.25
0.50
0.75
1.0
Output of Our Approach• Failure Explanation
• A characterization of the input space showing under what input conditions the system is likely to fail
• Visualized by diagrams or regression trees
• Failure Detection
• Individual test cases revealing failures
• A set of test input signals
12
Case studies• A mixed of closed loop and open loop controllers, and plant
models
• Developed by BOSCH
• Publicly available
• A large plant model – a mathematical continuous model
• Developed by an automotive company13
Failure Explanation – HeatmapDiagram
14
L
R
Failure Explanation – Regression Tree
15
All PointsCount MeanStd Dev
2384 1.016e+104.898e+11
c_gear>=1.0279Count MeanStd Dev
1997 25167.822135651.79
Count MeanStd Dev
387 6.257e+101.216e+12
t0>=0.0029462Count MeanStd Dev
1631 4550.450255698.046
t0<0.0029462Count MeanStd Dev
366 117044.69
276423.68
c_gear<1.0279
Failure Detection – Closed Loop
16
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.080.00.042
0.044
0.046
0.048
0.050
0.052
⌃[0,32]⇤((x x0 + ✏) ^ (x � x0 � ✏))Test Input violating: time
position
Failure Detection – Open Loop
17
0 1.0 2.0 0 1.0 2.0
0
4.0
-4.0
-8.0
0
1.5
-1.5
1.0
-1.0
0.5
-0.5
2.0
-2.0
-6.0
105
106
Time Time
Test inputs exhibiting ``instability” and ``grow to infinity” failure patterns
Summary of Lessons Learned• Generating test cases is not enough
• It is important to help engineers with input space explorationand failure explanation
18
All PointsCount MeanStd Dev
2384 1.016e+104.898e+11
c_gear>=1.0279Count MeanStd Dev
1997 25167.822135651.79
Count MeanStd Dev
387 6.257e+101.216e+12
t0>=0.0029462Count MeanStd Dev
1631 4550.450255698.046
t0<0.0029462Count MeanStd Dev
366 117044.69
276423.68
c_gear<1.0279
Summary of Lessons Learned• Engineers do not always have specific and precise
requirements at hand
• We generate test cases that reveal violation of both specific requirements and (estimated) failure patterns
19
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.080.00.042
0.044
0.046
0.048
0.050
0.052
0 1.0 2.0
0
1.5
-1.5
1.0
-1.0
0.5
-0.5
105
Time
Summary of Lessons Learned
• Incompatibility issues or manual overhead is a major obstacle for adoption of current Simulink testing tools
• Our approach is black box and has no overhead
• Time performance of our approach is acceptable
20
Our Tools• SimCoTest
Simulink Controller Tester
https://sites.google.com/site/simcotesttool/
• CoCoTestContinuous Controller Tester
https://sites.google.com/site/cocotesttool/21