Reza Matinnejad Shiva Nejati Lionel Briand Software Verification and Validation Group SnT Center, University of Luxembourg Thomas Bruckmann Delphi Automotive Systems, Luxembourg
MiL Testing of Highly Configurable Continuous Controllers:
Scalable Search Using Surrogate Models
Closed-loop Continuous Controllers An Example: Conveyor Belt Controller
Controller
1
+ -‐
2
Closed-loop Controller
Closed-loop Continuous Controllers An Example: Conveyor Belt Controller
Closed-loop Continuous Controllers
3
Desired Value
Controller Model
System Output
Model-based Development of Embedded Software
Plant Model
+ -
Model-in-the-Loop Stage (MiL)
Hardware-in-the-Loop Stage (HiL)
4
Software-in-the-Loop Stage (SiL)
Configuration 1
Configuration 2 Configuration 3
Controller Model
Plant Model
5
MiL Testing Continuous Controllers (1) Instantiating the Controller Model
• Step1: Identifying an assignment of values to the configuration parameters of the controller model
Des
ired/
Act
ual V
alue
Des
ired/
Act
ual V
alue
Time Time
MiL Testing Continuous Controllers (2) Running Model Simulation
6
Test Input 1
• Step2: Running simulations of the controller model and examining the output signals
Test Input 2 / Output 1 / Output 2
Controller Requirements (1)
Time Time
Act
ual V
alue
• Stability: The actual value shall stabilize at the desired value
Act
ual V
alue
7
Stability Violation
Stability Violation
Controller Requirements (2)
• Smoothness: The actual value shall not abruptly change when it is close to the desired value
Time
Act
ual V
alue
8
Smoothness Violation
Controller Requirements (2)
9
Time
Act
ual V
alue
Responsiveness Violation
• Responsiveness: The controller shall respond within a certain time-limit
Stability Objective Function: Fst
• Fst (Test Case A) > Fst (Test Case B) • We look for test input that maximizes Fst
Test Case A Test Case B
Act
ual V
alue
Act
ual V
alue
Time Time
10
Smoothness Objective Function: Fsm
• Fsm (Test Case A) > Fsm (Test Case B) • We look for test input that maximizes Fsm
Act
ual V
alue
Time Time
Test Case A Test Case B
11
Act
ual V
alue
Responsiveness Objective Function: Fr
• Fr (Test Case A) > Fr (Test Case B) • We look for test input that maximizes Fr
Act
ual V
alue
Time Time
Test Case A Test Case B
Act
ual V
alue
12
Our Approach: Search-based Testing of Continuous Controllers
13
• Step 1 : Exploration
• Step 2 : Search
Controller-Plant Model
Controller BehaviorOver the
Input Space
1.Exploration
Plant ⌃
Controller
Worst-Case Scenarios
CriticalOperatingRegions of
the Controller
2.Search
FD
ID FD
ID
Our Earlier Work: Search-based Testing of Continuous Controllers
with fixed values for the configuration parameters
Initial Desired (ID)
Fina
l Des
ired
(FD
)
Smoothness HeatMap Smoothness Worst-case Scenario
Controller-plant model
HeatMap Diagram
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration 2.Single-statesearch
Controller Input Space
14
Fst Fsm Fr
Published in [IST Journal 2014, SSBSE 2013]
Controller-plant model
HeatMap Diagram
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration 2.Single-statesearch
Challenges When Searching the Entire Configuration Space
1. The input space of the controller is larger • 6 configuration parameters in our case study
4. Search becomes slower • It takes 30 sec to run each model Simulation
2. Not all the configuration parameters matter for all the objective functions
3. Harder to visualize the results
15
Our approach for Testing the Controller in the Entire Configuration Space
Dimensionality ReducLon to focus the exploraLon on the variables with significant impact on each objecLve funcLon
VisualizaLon of the 8-‐dimension space using Regression Trees
Surrogate Modeling to avoid running simulaLons for parts of the input space where the surrogate model is the most accurate.
16
Controller-plant model
+Configuration
parametersranges
Regression Trees
Worst-Case Scenarios
List of Critical Regions
DomainExpert
1.Exploration with Dimensionality
Reduction
x<0.2
y<0.3
z<0.1
2. Search with Surrogate Modeling
Exploration with Dimensionality Reduction Elementary Effects Analysis Method
• Goal: IdenLfying variables with a significant impact on each objecLve funcLon
17
Exploringall the
dimensions
Exploring the significant
dimensions
ProportionalGainDerivativeGain
IntegralGainIntgrResetErrLim
...
SmoothnessProportionalGainIntgrResetErrLim
...
ElementaryEffects
AnalysisMethod
...
Regression Tree
• Goal: Dividing the input space into homogenous parLLons with respect to the objecLve funcLon values
Smoothness Regression Tree
18
All Points
ID < 0.7757
Count MeanStd Dev
Count MeanStd Dev
ID >= 0.7757Count MeanStd Dev
FD >= 0.5674 FD < 0.5674Count MeanStd Dev
Count MeanStd Dev
1000 0.007
0.0049
574 0.0059
0.004
426 0.01034250.0049919
182 0.0134
0.005
244 0.0080.003
FD < 0.1254 FD >= 0.1254Count MeanStd Dev
Count MeanStd Dev
182 0.01345550.0052883
244 0.00802060.0031751
Smoothness HeatMap
FD<0.5674
FD>=0.1254
Smoothness Critical Partition
Search With Surrogate Modeling Supervised Machine Learning
• Goal: To predict the values of the objecLve funcLons within a criLcal parLLon and speed up the search
19
• Surrogate model predicts Fsm with a given confidence level
A
B • During the search, surrogate model predicts Fsm for the next point: 1. Not confident about Fsm(B) and
Fsm(A) relation • Run the simulation as before
2. Confident that Fsm(B) > Fsm(A) • Run the simulation as before
3. Confident that Fsm(B) < Fsm(A) • Avoid running the Simulation
Search With Surrogate Modeling Supervised Machine Learning
• Different Machine Learning techniques: 1. Linear Regression 2. ExponenLal Regression 3. Polynomial Regression (n=2) 4. Polynomial Regression (n=3)
• Criteria to compare different techniques: 1. R2 ranges from 0 to 1
• R2 shows goodness of fit 2. Mean RelaLve PredicLon Error (MRPE)
• MRPE shows predicLon accuracy
20
Evaluation: Research Questions
• RQ1 (Which ML Technique): Which Machine Learning technique performed the best?
21
• RQ2 (Effect of DR): What was the effect of Dimensionality Reduction?
• RQ3 (SM vs. No-SM): How did the search with Surrogate Modeling perform comparing to the search without Surrogate Modeling?
• RQ4 (Worst-Case Scenarios): How did our approach perform in finding worst-‐case test scenarios?
RQ1: Which Machine Learning Technique?
22
• The best technique to build surrogate models for all our three objecLve funcLons is Polynomial Regression (n=3) • PR (n=3) has the highest R2 ( goodness of fit ) • PR (n=3) has the smallest MRPE ( predicLon error )
• The surrogate models are accurate and predicLve enough for Smoothness and Responsiveness (R2 was close to 1, MRPE was close to 0) • In future, we want to try other Supervised Leaning
techniques, such as SVM
RQ2: Effect of Dimensionality Reduction
23
• Dimensionality reducLon helps generate beber surrogate models for Smoothness and Responsiveness • Smaller MRPE: More predicLve surrogate model
Smoothness Responsiveness 0.03
0.02
0.01 No DR DR
MR
PE
0.02
0.03
0.04
MR
PE
Mean Relative Prediction Error (MRPE)
No DR DR
RQ3: Search with vs. without Surrogate Modeling
24
• For responsiveness, the search with SM was 8 Lmes faster
0.16
0.165
0.17 H
ighe
st v
alue
of
Fr
After 300 Sec After 2500 Sec
• For smoothness, the search with SM was much more effecLve
Hig
hest
val
ue
of F
sm
After 800 Sec After 2500 Sec
SM No SM
0.22
0.225
0.23
SM No SM 0.22
0.225
0.23
SM
No SM
SM
0.16
0.165
0.17
SM No SM
No SM
RQ4: Worst-Case Scenarios (1)
25
• Our approach is able to idenLfy criLcal violaLons of the controller requirements that had neither been found by our earlier work nor by manual tesLng
MiL Testing Varied Configurations
(Current work)
MiL Testing Fixed Configurations
(Earlier Work)
ManualMiL Testing
(Industry Practice)
Stability
Smoothness
Responsiveness
2.2% deviation - -24% over/undershoot
170 ms response time
20% over/undershoot
80 ms response time
5% over/undershoot
50 ms response time
RQ4: Worst-Case Scenarios (2)
26
• For example, for the industrial controller we idenLfied the following violaLon of the Stability requirement within the given ranges for the calibraLon variables
ID = 0.36
FD = 0.22
Flap
Pos
ition
Time
2.2 % Deviation
27
Conclusion
28
Reza Matinnejad PhD Candidate Software Verification and Validation Group SnT Center, University of Luxembourg Emails: [email protected] [email protected]
MiL Testing of Highly Configurable Continuous Controllers:
Scalable Search Using Surrogate Models
Alternative Heatmap
Stability Responsiveness
ü Our approach was implemented at Technical University of Munich too
ExisLng Tool Support for TesLng Simulink/Stateflow Models
• None of the exis,ng tools specifically test the con,nuous proper,es of Simulink output signals
Related Work
Technique Examples Limitations and Differences
Mixed discrete-continuous modeling techniques
• Timed Automata • Hybrid Automata • Stateflow
• Require model translation • Scalability issues • Verification of logical and
state-based behavior
Search-based testing techniques for Simulink models
• Path coverage or mutation testing
• Worst-case execution time
• Mainly focus on test data
generation • Only applied to discrete-event
embedded systems
Control Engineering • Ziegler-Nichols Tuning Rules
• Design/configuration
optimization • Signal analysis and generation
Commercial tools in automotive industry
• MATLAB/Simulink Design Verifier
• Reactis tool
• Combinatorial and Logical
Properties • Lack of documentation
31
CoCoTest Results
Stability
Violation
32