Sensing and actuating for Human -in-the-Loop systems: the ...ssie.dei.unipd.it/wp-content/uploads/2018/07/SSIE18_06... · business? n Experts ... Intelligence champions n Intelligence:

+

Sensing and actuating for Human-in-the-Loop systems: the autonomous driving case

Alessandro Beghi, Mattia Bruschetta

+Outline

n Part I: Autonomous driving, the scenarion Autonomous vehicles: Hype or real business?

n Getting rid of the human: sensing and actuating

n Human in the loop

n Part II: Driving Simulatorsn Driver Centric Automotive Engineering

n Model Predictive Control

n Virtual Driver

n Motion Cueing Algorithms

HIL systems & autonomous driving A. Beghi, M. Bruschetta

2

+Part I: Autonomous driving, the scenario

A. Beghi, M. Bruschetta HIL systems & autonomous driving 3

+Autonomous vehicles: Hype or realbusiness?n Among the trends that are going to shape the automotive

industry in the coming years, autonomous driving stands out as having the potential to completely change the automotiveindustry as we know it.

n Auto-makers around the world are unanimous in predictingthe emergence of autonomous driving sometime in the nexttwo decades

n The prevailing view is that the speed of these breakthroughsis limited by regulatory constraints and liability issuesmore than by the underlying technologies.

n On the technology side, making functions operate seamlesslyto ensure driver comfort and reducing the cost of advanced systems are key challenges.


4

+Autonomous vehicles: Hype or realbusiness?

n Experts forecast that there will be a mixture of semi-autonomous and fully automated vehicles on the roads in the next decade, with steadily increasing penetration of both.

n With a minimum starting cost of approximately US$10,000 to equip a midsize vehicle with fully autonomous drivingcapability, the technology is expected to be found mostly in the premium segment and on commercial vehicles atfirst.

n On the other hand, the evolution is driving a strong market penetration of Advanced Driver Assistant Systems (ADAS) on «standard» vehicles.

n Market value hard to estimate: from hundreds of billions to trillions USD.


5



6


n 2017 claims:

n VAG – Fully Self-Driving Cars by 2020n GM – Rumors of Self-Driving by 2018n Ford – True Self-Driving by 2021n Honda – Self-Driving on the Highway by 2020n Toyota – Self-Driving on the Highway by 2020n Renault-Nissan – 2020 for Autonomous Cars in Urban Conditions,

2025 for Truly Driverless Carsn Volvo – Self-Driving on the Highway by 2021n Hyundai – Highway 2020, Urban Driving 2030n Daimler – Nearly Fully Autonomous by Early 2020’sn FCA – expects there to be some self driving on the road

by 2021n BMW – Fully self-driving possible by 2021n Tesla – End of 2017


7



8



9


HIL systems & autonomous driving A. Beghi, M. BruschettaSource: Roland Berger

10

+Autonomous vehicles as ArtificialIntelligence champions

n Intelligence: what features?n Memory

n Learning

n Generalization

n Abstraction

n Planning

n Problem solving

n Fully AUTONOMOUS!

n «Hard AI» approach


11

+Autonomous vehicles as Cyber-Physical Systems (CPS)

n ʺA cyber-physical system (CPS) integrates computing, communication and storage capabilities with monitoringand/or control of entities in the physical world, and must do so dependably, safely, securely, efficiently and in real-time.“ (S. Shankar Sastry, UC Berkeley)

HIL systems & autonomous driving A. Beghi, M. BruschettaSource: Vincenzo Liberatore, NetworkedCyber-Physical Systems: An Introduction, 2007

12

+Getting rid of the human: actuating and sensing


+Getting rid of the human


14

Sensing

Actuating

+Getting rid of the human


§ Vehicle's location and environment: highly precise and real-time information of a vehicle's location and its surrounding environment(e.g., road signs, pedestrian traffic, curbs, obstacles, traffic rules).

§ Prediction & decision algorithms: advanced concepts based on unsupervised/deep learning, machine learning will be needed to create systems to detect, predict and react to the behavior of otherroad users, including other vehicles, pedestrians, and animals.

§ High accuracy, real-time maps: detailed and complete mapsmust be available to provide additional and redundant information for the environmental models that vehicles will use for path and trajectory planning.

§ Vehicle driver interface: self-adapting interface with smoothtransition of control to/from the driver, mechanisms to keep the driver alert and a flawless ride experience for user acceptance

15

+Actuating = going «by-wire»


§ "by-wire": replacing a pure mechanical system of motion or control with a sensor, an actuator, and some computing power (CPS)

§ X-by-wire technology is referred to primarily three categories: Throttle-by-wire, Brake-by-wire, and Steer-by-wire (in order of their perceived complexity and risk, from lowest to highest)

§ «dry system»: the hydraulic fluids are no longer necessary. Thisleads to a simpler and more easily maintained system.

§ An x-by-wire system can also be easily upgraded or adjusted to different needs as it is software-defined (download the new software, no parts must be replaced).

§ Sensing the vehicle!

16



§ Example: Steer-by-wire

17

Source: Bertacchini et al.. ISIE 2006; P isu et al., Journal o f dynam ic system s, m easurem ent, and control, 2006



Source: Kissai et al, ECMR 2017

18



§ Full X-by-wire for EVs with four In-Wheel Motors: over actuating

19

Source: Schaeffler

+Sensing the environment


20



Source: 2025 AD

21



Source: 2025 AD

22



Source: 2025 AD

23



Source: 2025 AD

24



Source: 2025 AD

25

+The human in the loop


+But … can we really get rid of the human?


27

§ Humans will still play a crucial role in the design of control systemsfor automated vehicles

§ Fully autonomous cars: human is just a passenger

User experienceComfortSickness…



§ Semi-autonomous cars: cooperative guidance & control§ Shared and cooperative guidance and control entails at least two

autonomous entities that need to work together

28

Source: Altendorf et al, IFAC-PapersOnLine, (2016)



§ H-Mode (Flemisch) automation scheme is based on the H(orse)-metaphor, which describes the roles of human and automationsimilar to the roles of a rider and a horse or a driver and a horse-cart.

29

Source: Altendorf et al, IFAC-PapersOnLine, (2016)

+Human-in-the-Loop


30

§ In automated driving, the human has to be considered a systemcomponent in the control loop

§ As a passenger, it is a part of the system that indirectly affects the control action, as it is associated with «outputs» that do not concernthe task to be executed (i.e., taking the vehicle somewhere)

§ As a part of the (cooperative) control architecture, it activelycollaborates to the task execution

§ Main issues:

1) Modeling2) Sensing

+Human-in-the-Loop


31

§ Modeling:§ Cognitive processes (including stress and cognitive load)§ Human cognitive representation of the execution of the driving

task§ Perceptual organs (vision, vestibular, somatosensory, …)

§ Perception§ Sensing:

§ Eye-tracking§ BCIs and Functional Near-Infrared Spectroscopy (fNIRS)§ ECG and EEG§ SPR (Skin Potential response)§ …

+Human-in-the-Loop


32

§ As measuring cannot be intrusive, the key is the derivation of observers and estimators

§ Experiments can be performed in controlled scenarios, with dedicated sensing architectures

§ Models are then derived and used in model-based control approaches, the most promising of which is Model PredictiveControl (MPC)

§ The presence of highly automated vehicles on the streets will notbecome a reality without their control algorithms being thoroughlytested and 'trained' in real-/almost real-life traffic conditions

+Testing is crucial


33

+Testing is crucial


§ Real fleets of physical test cars are going to be indispensable§ However, a lot of knowledge can be gained by using virtual

enviroments:§ Controlled§ Safe

§ Repetible§ More than 95% of the training can be done in the virtual world:

§ Waymo is simulating some 25,000 vehicles simultaneously in virtual versions of actual Austin, Mountain View, and Phoenix streets and on a number of different test routes.

§ More than 12 million Km in the virtual world. § In 2016: virtual Km driven > 3.5 billion§ Over the past 8 years: real Km driven ~ 5 million

34

+Part II: Driving simulators


+Outline

n Part I: Autonomous driving, the scenarion Autonomous vehicles: Hype or real business?

n Getting rid of the human: sensing and actuating

n Human in the loop

n Part II: Driving Simulatorsn Driver Centric Automotive Engineering

n Model Predictive Control

n Virtual Driver

n Motion Cueing Algorithms


36

+

A. Beghi, M. BruschettaHIL systems & autonomous driving

37

The Dynamic Driving Simulator

Professional Driving Simulator: moving mechanical platform acting on a car cockpit (driver), and inducing stimuli (cues) on the driver to faithfully reproduce the motion sensations of a real vehicle.

+


38

Driver Centric Automotive Engineering

Teamwork between Design

and Test Engineers

Strategic ADAS/AD Verification

Explore Robotic Acceptance

Vehicle-Driver-

Environment Interface

Assessment

Objective-Subjective Test

Comparison

Driver at the Center

+

n Control strategies for dynamic driving simulators are called Motion Cueing Algorithms (MCA)

n MCA is a problem in the problem: to make available a testing tool for HMI control system, a complex HMI control system is needed

n MCA is an extremely challenging Human Machine Interaction problem

n Accounting for human behavioral and cognitive aspects

n Ensuring safety (e.g. limitation on actuators)

n Handling switch of control authority

n Strict Real-time requirements


39

Driving Simulators as a control HMI problem

+

n New paradigm: collaborating with human to improve performance

n Challenges: n Handling switch of control authority

n Ensuring safety (e.g. limitation on forces)

n Accounting for human behavioral and cognitive aspects

n Strict Real-time requirements

n Promising Framework: Model Predictive Control

M. BruschettaDriving Simulator as HMI problem

HIL scenarios

Collaborative roboticsAutonomous driving Dynamic Driving Simulator

+Model Predictive Control


+MPC principles


42

+

n Consider the system

n And the sequence of control variables in the future horizon at time k

n Minimizing the performance index

n Where

n And N is the so-called prediction horizon


Linear MPC: basics43

+

n The state equation in the prediction horizon is

n and define

n The future state variables are given by


Linear MPC: Matrix formulation44

+

n Moreover defined

n Then the optimization problem can be written as

n Which is Quadratic Programming (QP) problem


Linear MPC: Matrix formulation45

+Nonlinear MPC (MATMPC)

discrete QP

ProblemDirect Multiple Shooting

discrete NLP

Problem

linearization

optimal solution

Non Linear Model

Objective Function

bounds and

constraints

qpOases

Sequential Quadratic Programming (SQP)

discretization QP solution

NLP Problem QP sub-problem


46

+Direct method: Multiple shooting strategy

n Multiple shooting is a numerical method for the solution of boundary value problem (ODE Solution)

n The method divides the prediction horizon into several smaller intervals

n Solves an initial value problem for each of the smaller intervals

n Imposes additional continuity conditions to form a solution on the whole interval.

+NMPC Optimization: Multiple shooting strategy

SQP iteration until continuous solution is obtained

+NLP formulation: the discretized problem

The NLP problem is formulated as

where

and

Original problem

+MPC exampleAutonomous Driver


+

n Testing of passive driving on simulators requires an “autonomous driver”

n Goal is that of taking decisions based on:n Vehicle model (more detailed reference model → more skilled

driver)n Future trajectory and velocity reference n Cost function

n Main advantages:n Coupling of throttle and steer inputs n Driving style tunable through cost function

n Challenge:n Solution of a non linear optimization problem at each time step

Real-Time NMPC based Virtual Driver


52

+Model concept

n 4 wheel vehicle model

n longitudinal forces are linear

n ! is the normalized throttle\braking effortn " is the friction coefficientn #$ is the normal force

n lateral forces through Pacejka

n B, C, D, E are tire-dependent parametersn % is the slip angle of the wheel

n dynamic formulation in path s

n introduction of lateral error &' and heading error &(n references not dependent on time

#) = ! ⋅ " ⋅ #$

#, = -sin(2 tan56(7% −9(7% − tan56(7%))))


53

+

n input vector ! = #$, &n #$ is the derivative of the steering anglen & is the derivative of the normalized throttle\braking effort

n state vector ' = (, ), *, +,, +-, #$, &n (, ) and * are longitudinal, lateral and rotational velocities

n +, and +- are the tracking errorsn #$ and & are the actual inputs for the simulation model

n dynamic equationsn ( =)*+ /

0 ∑2,3456,7 −49n ) =−(*+ /

0 ∑2,34-6,7n *= /

:; < ∑34-=,7 −> ∑34-?,7 +@ ∑2456,? −∑2456,A

Model equations


54

+

n weight matrix is diagonal → " = $%&'()*)

n optimization variables ℎ = -, /0, /̇0, /2+ 4, /̇2n - = 56 + 76: vehicle velocity

n /0 and /2 + 4: position errors (↑ weights → smaller traj tracking error)

n /0 and /2: velocity tracking errors (↑ weights → more conservative behaviour)

Objective Function

The objective function is quadratic

: = ;<=>

?@ℎ−B/C D"(ℎ−B/C)+ ;

<=>

?@EF

GDHG


55

+

n Constant reference velocity (25m/s)

n Controller reduces velocity to balance tracking error and velocity error

Double Lane Change


57

+

n Changing weights on trajectory error

n !_"≪ 1 → perfect velocity tracking, greater trajectory error

n !_"≫1 → better trajectory tracking, greater velocity error

Weights influence


58

+

n performance comparison

Weights influence


59

+MPC Driver vs Basic Driver


60

+Motion Cueing Algorithm


+MPC for Motion Cueing

n Model-based: linearized vestibularsystem models

n Cost function: minimization of

perceived acceleration/velocity error

n Constrained, optimal problem: explicithandling of performance and workingarea constraints

Features


Vehicle Model

Vestibular Model

Model Predictive Control based MC

Vestibular Model

Cost FunctionOptimizator

PLC

Driver

Platform

a, v

r

d

u

s

Preprocessing

63

Semicircular channels

Otholits

+Vestibular SystemLinear Models

Semi-circular channels linear Transfer Function

Otoliths linear Transfer Function


65

+Vestibular System

n Under steady condition the human being is incapable of di-stinguishing translational acceleration from gravity force with the otoliths ⇒ rotational and/or visual sensorial information is needed

n This feature can be exploited when developing motion cueing algorithms: Low frequency components of the acceleration can be reproduced by using Tilt Coordination

Tilt coordination


66

+MPC: Objective function

The structure of the objective function defines the control objectives

n y are the tracking signals

n r are the reference signals to be tracked, constant along the horizon

n qy , qu , q∆ are the weights, M are the platform limits

n x are the states

n u are the inputs for the platform control system

Parameters


67

+

n The reference signal in well tracked within the platform limits


Experimental data68

+

n The reference signal in well tracked within the platform limits

n Tilt coordination is obtained without filtering


Experimental data69

+Time-varying PredictiveMCA


+Prediction strategy: Non look Ahead (NLA) vs Look Ahead (LA)

A. Beghi, M. Bruschetta

n A short constant reference is usedPros:n Reduced computational burdenCons:n Limited platform exploitation

n Use a long prediction window using an estimation of the future driver’s behavior

Pros:n Virtual enlargement of the platform

operational spaceCons:n Knowledge about the future driver behavior is

needed

NLA

LA

HIL systems & autonomous driving

71

+


The reference manipulation

!a# $ + & = ( )a* $ & = 1 … -.$/0 !1 $ + & & = -. … -2,/0

)a $ :reference from current driver behavior!1:reference from benchmark lap$/0: gain to modulate the reference !1 , $/0 ∈ [0 ÷ 1]-.: samples in the constant reference part-2,//0: samples in the overall reference

n The reference is composed by two partsn A constant one equal to the

current driver behavior )a* $n A time varying one equal the

expected driver behavior !1 $ + &

n LA on behalf of NLA: if the time varying phase is set to zero, only the current information is used by the algorithm, as in the NLA case.


72

+A new cost function

n The cost function must be rewritten accordingly

n A new set of tuning parameters (STP) can thus be defined for the LA case

!"#$% = 'a) ,+ ',) ,+ '-) ,+./

'a) ,$% ',) ,$% '-) ,$%.01

2 34


The optimal tuning set should be capable of maximizing performance if information on future driver behavior is available, and to guarantee a NLA-like robustness if information is not reliable.

Optimal Tuning Set (OTS)


73

+

n Assume K to be the input gain to the MPC based MC: greater is K more effective is the MCA

n If !"# = 1 the future information is used and a greater K is expected to be allowed

n If !"# = 0 the algorithm is required to properly track the current driver behavior, i.e.

n A Multi-Objective optimization can be performed on


Metrics for Multi-Objective optimization

'"# (), +) ,-."# ≜ max3∈ℝ6

78. :. ;<= − ?@A B

≤ 0.05

E"#{(),7|,-."#} ≜ ;I<= − I@A B

CL: Current LapBL: Benchmark LapMCL: Most Challenging Lap0,05J/8L: perception threshold


Vehicle Model

Vestibular Model

Model Predictive Control based MC

Vestibular Model

Cost FunctionOptimizator

PLC

Driver

Platform

a, v

r

d

u

s

Gain K

arg minQRSTU

VW,1VL

VW = E"# {X()|.-,"# }

VL = '"# X(), X() .-,"#

75

+

n The result of the two-objective optimization is a Pareto Front

n Small tracking errors if !"# = 0are possible only reducing the performances of the LA

n The best LA performance is obtained with &"#<0.05, hence it is also the OTS

n The Multi Objective optimization has been performed with the NSGA-II algorithm in a parallel computing framework


Pareto Front and Optimal Tuning Set (OTS)

'( = &"#{*+,|./0"#}'2 = 3"# *+,, *+, ./0"#

arg min;<=>?'(,1'2

A/00.35

0.3

0.15

0.2

0.25

OTS


76

+


Data set (DS)

n 12 laps on the Calabogie track have been performed with a virtual driver varying:

n Trajectory:

n Center line

n Corner Cutting

n Corner Cutting on Kerbs

n Grip : {1, 0.8, 0.6, 0.4} are used where 1 is the standard level. With small grip values unexperienced drivers are emulated. Using

grip 0.4 the lap time is increased of almost 20 s.

n The MCL is the Corner Cutting on Kerbs with grip 0.8


77

+


Performance of Optimal Tuning Set

n !"#% is the percentage gain while

using the %&' and the MCL

!"#% = !"# − !*"#!*"#

n !"#% decreases if the CL is far from the BL

n For all the laps in the data set an advantage w.r.t. the NLA is achieved

!"#% +,, .+, %&'"#+, ∈ 0'

MCL


78

+

n The validation has been performed with a human standard driver lap

n The advantage in using prediction w.r.t. the NLA on the VL results to be


Validation Lap (VL)

LA Acceleration tracking Vertical Displacement comparison

!"#% %&, ()& *+,"# = 43%


79

+

n Hierarchical architecture:n Auto-controllern Human Operator

n Advisory controller

n Automatic controller Linear MPC:n Force constraints

n Linear modelsn Predictive features

n Advisory controllerSwitches control authority:n Application dependent strategies


A step ahead: HMI control architecture

[1] W. Li et al. "Synthesis for human-in-the-loop control systems“, in TACAS 2014, Springer.

Picture taken from [1]

80

+

n Autonomous driving is a very relevant control application, both for its societal and market impact, and for its research content

n Sensing and actuating strategies have to be interpreted in a non-trivial way, due to the presence of the human in the loop

n Control architectures for cooperative guidance & control pose new challenges, associated with modeling the human/robot interraction for exchange of control authority

n MPC is a valuable tool for tackling many of the aforementioned problems

n Challenges include real-time performance and inclusion of non standard (e.g. learning based) models in the MPC framework


81

Conclusions

Documents

Sensing and actuating for Human -in-the-Loop systems: the ...ssie.dei.unipd.it/wp-content/uploads/2018/07/SSIE18_06... · business? n Experts ... Intelligence champions n Intelligence: