Helsinki University of TechnologyAdaptive Informatics Research CentreFinland

Variational Bayesian Approach for Nonlinear Identification and Control

Matti Tornio and Tapani Raiko

October 9, 2006

Introduction

System identification and control in nonlinear state-space models

Continues the work by Rosenqvist and Karlström (Automatica 2005)

Our background is in machine learning Uncertainties taken explicitly into account

by using Variational Bayesian treatment

Why nonlinear state-space models?

System identification using a hidden state has many benefits:

More resistant to noise Observations (without history) do not always

carry enough information about the system state

Finds a representation of the state that is more suitable for approximating the dynamics

System identification in nonlinear state-space models

We use a state-of-the-art tool by Valpola and Karhunen (Neural Computation 2002)

Parameters, states, and observations are modelled with Gaussian distributions

Less prone to overfitting (than the prediction error method)

Can determine the dimensionality of the state space etc.

Properties of the method

The model scales well to higher dimensions Can model very complex dynamics Natural conjugate gradient algorithm is used for

fast system identification

Nonlinearities by MLP networks

f(x(t),θ)=B tanh[Ax(t)+a] + b + noise The parameters θ include the weight matrices,

bias vectors, noise variances etc. Note that the policy mapping does not fix the

control signal (because of the noise model)

Variational Bayesian treatment

Posterior probability p(x,θ|y) is approximated by q(x,θ)

q is assumed to be Gaussian with limited dependencies

The fit of q to p is measured by a cost function Both identification and prediction can be done

by minimising the misfit by adjusting the parameters defining q (means, variances, covariances)

Control

Current state is estimated with extended Kalman filter (EKF)

Control signals u(t) are selected to minimise the expected cost E{J} over the distribution q

Quasi-Newton algorithm for optimisation Compare to dual control:

estimation errors increase the expected cost

Control (cont.)

Prediction with variances is ~5 times slower too slow for some applications, the method can still

be used for system identification Learning is done offline

online learning possible as well, leads to different exploration strategies

Optimistic inference control

Alternative control scheme Observations at some point in the future are

fixed and the states leading to this desired future are inferred

Allows the direct use of inference algorithms Conceptually very simple, but not as versatile

as NMPC constraints hard to model

Experiments

Assume the cart-pole system to be unknown Dynamics are identified from only 2500

samples 6-dimensional state space x(t) was used

Results

Very high success rate was reached even under high noise

Partially observed system is hard to control

4 obs

2 obs

4 obs, f=I

2 obs, f=I

0 20 40 60 80 100 120

4 obs

2 obs

4 obs, f=I

2 obs, f=I

0 10 20 30 40 50 60 70 80 90 100

Low noise High noise

Results (initialisation)

Good initialisations are important Local minima are the biggest problem

Internal forward model can provide reasonable initialisations without significant extra computation

Conclusion

Learning nonlinear state-space models seems promising when observations about the system state are incomplete

or the dynamics of the system are not well known

Variational Bayesian treatment helps against overfitting to determine the model order