An introduction to optimal control problem - · PDF file2 Optimal control problems 3 Numerical methods J. Loh eac (BCAM) An introduction to optimal control problem 06-07/08/2014 4

An introduction to optimal control problemThe use of Pontryagin maximum principle

Jerome Loheac

BCAM

06-07/08/2014

ERC NUMERIWAVES – Course

J. Loheac (BCAM) An introduction to optimal control problem 06-07/08/2014 1 / 41

Generalities

The aim of this course is to give basis to solve analytically or numerically optimalcontrol problems.In full generality, we consider a system governed by the dynamic:

x = f (x , u) . (1)

with x ∈ Rn is the state variable and u ∈ Rm is the control variable. The controlproblems is:

Problem (Control problem)

Given T > 0, x i , x f ∈ Rn does it exists u : [0,T ]→ Rm such that systems (1)steers x i to x f in time T .

For optimal control problem, we consider a cost function:

J(x , u) =

∫ T

0

f0(x(t), u(t)) dt + g(x(T ), u(T ))

and the aim is to find a control u which realize the control problem andminimize J.


References (books)

E. Trelat Controle optimal : theorie et applications

J.-M. Coron Control and Nonlinearity

E. B. Lee and L. Markus Foundations of optimal control theory

A. A. Agrachev and Y. L. Sachkov Control theory from the geometricviewpoint

L. S. Pontryagin The maximum principle in optimal control

A. D. Ioffe and V. M. Tihomirov Theory of extremal problems

J.-L. Lions Controlabilite exacte, perturbations et stabilisation de systemesdistribues

M. Tucsnak and G. Weiss Observation and control for operator semi-groups

H. O. Fattorini Infinite-dimensional optimization and control theory


1 Controllability results

2 Optimal control problems

3 Numerical methods


Controllability results

1 Controllability resultsLinear control problemsNonlinear control problems


3 Numerical methods


Controllability results Linear control problems

Linear control problems IAutonomous systems

We consider the system:

x = Ax + Bu x(0) = x i . (2)

with A ∈Mn(R) and B ∈Mn,m(R).Proving the controllability of the system (2) is equivalent as proving theobservability of the adjoint system:

ϕ = −A∗ϕ ϕ(T ) = ϕf , (3)

with the observation operator B∗.That is to say, proving that there exists a constant C > 0 such that:

‖ϕf ‖2Rn 6 C

∫ T

0

‖B∗ϕ(t)‖2Rm dt (ϕf ∈ Rn) .



Linear control problems IIAutonomous systems

Theorem (Kalman rank condition)

Ifrank

[B AB . . . An−1B

]= n ,

then for every T > and every x i , x f ∈ Rn, there exists u ∈ L2(0,T ;Rm) such thatsystem (2) steers x i to x f in time T . In addition, the control of minimalL2 − norm is given by:

u(t) = B∗ϕ(t)

with ϕ the solution of the adjoint problem (3) with final condition ϕf and whereϕf realise the minimum of:

J(ϕf ) =1

2

∫ T

0

‖B∗ϕ(t)‖2Rm dt + 〈x i , ϕ(0)〉Rn − 〈x f , ϕf 〉Rn

(where here ϕ is the solution of the adjoint problem (3) with final condition ϕf ).



Linear control problems IIIAutonomous systems

Example

x = −x + u



Linear control problemsNon-autonomous systems

We consider here the system:

x = A(t)x + B(t)u (4)

and we assume that A and B are smooth.

Theorem (Kalman rank condition for non-autonomous linear systems)

Let us define for every i ∈ N the matrices Bi ∈Mn,m(R) by:

B0(t) = B(t) , Bi (t) = Bi−1(t)− A(t)Bi−1(t) (t ∈ [0,T ]) .

If there exists t ∈ [0,T ] so that:

Span Bi (t)u , u ∈ Rm , i ∈ N = Rn

then the system (4) is controllable.


Controllability results Nonlinear control problems

Nonlinear control problems ILinear test

We consider here the general system

x = f (x , u) . (5)

Let assume that f ∈ C 1(Rn × Rm,Rn). Let us consider a trajectory (x , u), i.e.˙x = f (x , u), the linearizing along this trajectory leads to the linear system:

x = ∂x f (x , u) x + ∂uf (x , u) u . (5’)



Nonlinear control problems IILinear test

Theorem

If system (5’) is controllable, then the system (5) is locally controllable along thetrajectory (x , u).

That is to say:For every ε > 0, there exists η > 0 such that for every x i , x f ∈ Rn with

‖x(0)− x i‖Rn , ‖x(T )− x f ‖Rn < η ,

there exists a trajectory (x , u) of (5) so that

x(0) = x i , x(T ) = x f and supt∈[0,T ]

‖u(t)− u(t)‖Rm 6 ε .



Nonlinear control problems IIILinear test

Example (Spring)

x = −x − x3 + u

Example (Brockett integrator)

x1 = u1

x2 = u2

x3 = x1u2 − x2u1



Nonlinear control problems ISystems without drift

Let us now consider the control system:

x =m∑i=1

fi (x)ui , (6)

where the fi are smooth vector fields on Rn.To these vector fields, we associated the Lie algebra Lief1, . . . , fm generated bythe iterated Lie brackets:

[fi , fj ](x) = ∂x fj fi (x)− ∂x fi fj(x) (x ∈ R) .



Nonlinear control problems IISystems without drift

Theorem (Chow)

Let us assume:rankLief1, . . . , fm(x) = n (x ∈ Rn) .

Then,

1 If the set of admissible controls Ω ⊂ Rm is nonempty and contains 0 in itsinterior, then there exists T > 0 such that the system (6) is controllable.

2 If the admissible set Ω is Rm, for every T > 0, the system (6) is controllable.


x1 = u1

x2 = u2

x3 = x1u2 − x2u1



Nonlinear control problems IDrift systems

Let us now consider the control system with drift:

x = f0(x) +m∑i=1

fi (x)ui , (7)

where the fi are smooth vector fields on Rn and we assume that f (0) = 0 (i.e.(0, 0) is an equilibrium point).For such a system, it is also natural to consider the Lie algebra generated byf0, . . . , fm.But the Lie algebra rank condition is not enough to obtain controllability,

Example

x1 = x22

x2 = u



Nonlinear control problems IIDrift systems

To end up with this problem, we define the set Br f0, . . . , fmof formal iteratedLie brackets generated by f0, . . . , fm. And for every h ∈ Br, we define δi (h) thenumber of times that fi appears in h.Let us also define

σ(h) =∑

π∈Perm

π(h) ,

with Perm the group of permutation of 0, . . . ,m such that for every π ∈ Perm,π(0) = 0.For θ ∈ [0,∞], we will say that the control system (7) satisfies the Sussmanncondition S(θ) if

1 it satisfies the Lie algebra rank condition at 0,

2 for every h ∈ Br with δ0(h) odd and δi (h) even for every i ∈ 1, . . . ,m,σ(h)(0) is in the span of the g(0)’s such that

g ∈ Br and

θδ0(g) +

∑mi=1 δi (g) < θδ0(h) +

∑mi=1 δi (h) if θ ∈ [0,∞) ,

δ0(g) < δ0(h) if θ =∞ .



Nonlinear control problems IIIDrift systems

Theorem (Sussmann)

If for some θ ∈ [0, 1], the control system (7) satisfies the condition S(θ), then it issmall time locally controllable.

That is to say:For every τ > 0, there exists η > 0 such that for every x i , x f ∈ Rn, with

‖x i‖, ‖x f ‖ < η ,

there exists u : [0, τ ]→ Rm such that the trajectory of (7) satisfies:

x(0) = x i , x(τ) = x f and ‖u(t)‖ 6 τ (t ∈ [0, τ ]) .

Example

x1 = x32 , x2 = u



Nonlinear control problems IVDrift systems

We can extend this result to more general problems. Consider the general controlsystem:

x = f (x , u) , (8)

with f (0, 0) = 0.To (8), we associate:

x = f (x , y) , y = u . (8’)

Theorem

If for some θ ∈ [0, 1], the control system (8) (i.e. (8’)) satisfies the conditionS(θ), then it small time locally controllable.


Optimal control problems


2 Optimal control problemsLinear control problemsGeneral control problems

3 Numerical methods


Optimal control problems Linear control problems

Linear control problems ILinear-quadratic theory


x = A(t)x + B(t)u , x(0) = x i (9)

and given T > 0, we want to minimize the cost:

J(u) = x(T )>Qx(T ) +

∫ T

0

(x(t)>W (t)x(t) + u(t)>U(t)u(t)

)dt , (10)

with x the solution of (9)In the above, the matrices Q and W (t) are symmetric nonnegative and U(t) ispositive. In addition, we assume that A,B,W and U are L∞ in time.



Linear control problems IILinear-quadratic theory

Theorem (Existence)

Assume there exists α > 0 such that:∫ T

0

u>(t)U(t)u(t)dt > α

∫ T

0

u>(t)u(t)dt (u ∈ L2([0,T ],Rm)) ,

then there exists a unique control u ∈ L2([0,T ];Rn) minimizing J.



Linear control problems IIILinear-quadratic theory

Theorem (Optimality condition)

The trajectory x associated to the optimal control u is optimal if and only if thereexists and adjoint vector p such that:

p = −A(t)>p + W (t)x(t) , p(T ) = −Qx(T ) .

More other, the optimal control u is:

u(t) = U(t)−1B(t)>p(t) .

Example

x = u , x(0) = x i

with the cost J(u) =

∫ T

0

(x(t)2 + u(t)2

)dt.



Linear control problems ITime optimality


x = A(t)x + B(t)u , x(0) = x i (11)

Given x f ∈ Rn, the aim is to find the minimal time T > 0 such that there exists acontrol u : [0,T ]→ Rm steering x i to x f in time T and such that:

u(t) ∈ Ω (t ∈ [0,T ]) ,

with Ω a compact set of Rm.

Theorem (Existence)

If there exists a time T > 0 such that x i can be steered to x f with a control uwith u(t) ∈ Ω then, the minimal time exists.



Linear control problems IITime optimality

Theorem (Optimality condition)

Assume that the previous existence theorem hold and let us define T the minimaltime. Then the control u : [0,T ]→ Rm is a control in time T if and only if thereexists p(t) a non trivial solution of:

p = −A(t)>p ,

such that u satisfies:

p(t)>B(t)u(t) = maxv∈Ω

p(t)>B(t)v (t ∈ [0,T ] a.e.) .



Linear control problems IIITime optimality

Let us focus on the autonomous control system:

x = Ax + Bu , (12)

with A ∈ Rn×n and B ∈ Rn×1.

Theorem (Bang-Bang property)

Assume that the pair (A,B) satisfies the Kalmann rank condition ant thatΩ = [−1, 1] then,

1 If all the eigenvalues of A are real, then the time optimal control has at mostn − 1 commutations;

2 If A has a non real eigenvalue, then for every N ∈ N, there exists points x i

and x f such that the time optimal control has at least N commutations.

Example

x = −x + u


Optimal control problems General control problems

Existence results IGeneral systems

Let us consider the control system:

x = f (t, x , u) , (13)

with a control u such that u(t) ∈ Ω, with Ω a compact set of Rm. The aim is tojoin a set M i ⊂ Rn to a set M f ⊂ Rn by minimizing the cost: with the cost:

J(u) =

∫ t(u)

0

f0(t, x(t), u(t))dt + g(t(u), x(t(u))) .



Existence results IIGeneral systems

Theorem

Assume that M i is accessible from M f , i.e. for every x f ∈ M f , there existsx i ∈ M i , u : R+ → Rm with u(t) ∈ Ω such that there exists t = t(u) > 0 forwhich the solution of (13) with initial condition x(0) = x i satisfiesx(t(u)) = x f .

Assume in addition that for all of these control u, there exists a constantb > 0 (independent of u) such that:

t(u) + ‖x(t)‖ 6 b (t ∈ [0, t(u)]) .

Assume also that for every s ∈ R and y ∈ Rn, the set(f (s, y , v)

f0(s, x , v) + γ

), v ∈ Ω , γ > 0

is convex.

Then there exists a control u, an initial condition x i ∈ M i and a time t(u) > (u)such that u steers x i to x f in time t(u) and u minimize J.



Existence results IIIGeneral systems

Remark

The boundedness property for u (u with values in a compact set Ω) and for x canbe replaced by some growth conditions.See for instance Harlt, Sethi and Vickson, A survey of the maximum principle foroptimal control problems with state constraints, SIAM review, 1995.



Pontryagin maximum principle IGeneral result

Theorem (Pontryagin maximum principle)

If u is optimal, then there exists an application p : [0,T ]→ Rn and p0 6 0 suchthat:

1 (p(·), p0) is nontrivial;

2 for almost every t ∈ [0,T ], we have:

x = f (t, x , u)

p = −∂xH(t, x , p, p0, u)

with:H(t, x , p, p0, u) = 〈f (t, x , u), p〉Rn + p0f0(t, x , u)

and the control u is such that:

maxv∈Ω

H(t, x(t), p(t), p0, v) = H(t, x(t), p(t), p0, u(t)) (t ∈ [0,T ] a.e.) .



Pontryagin maximum principle IIGeneral result

RemarkOne can also notice that:

d

dtH(t, x(t), p(t), p0, u(t)) = ∂tH(t, x(t), p(t), p0, u(t)) .

and hence if f and f0 are independent of t, then H(t, x(t), p(t), p0, u(t)) isconstant.



Pontryagin maximum principle ITransversality conditions

Theorem (Transversality conditions)

In addition,

1 If the final time T is not fixed, then,

maxv∈Ω

H(T , x(T ), p(T ), p0, v) = −p0∂tg(T , x(T )) .

2 If x(0) and x(T ) are not fixed but satisfies (x(0), x(T )) ∈M with M asub-manifold of Rn × Rn, then,

(p(0), p(T )− p0∂xg(T , x(T ))) ⊥ T(x(0),x(T ))M .



Pontryagin maximum principle IITransversality conditions

Example1 If x(0) is free, then, p(0) = 0;

2 If we want periodic trajectories, i.e. x(0) = x(T ) and if g = 0, then,p(0) = p(T ).



Pontryagin maximum principle IApplications

Example (Linear control problem)

x = Ax + Bu

T fixed, x(0) = x i , x(T ) = x f , Ω = Rm, g = 0, f0(t, x , u) = 12‖u‖

2;

T fixed, x(0) = x i , x(T ) ∈ Rn, Ω = Rm, g(t, x) = x>Qx ,f0(t, x , u) = x>Wx + u>Uu;

T free with T > 0, x(0) = x i , x(T ) = x f , Ω = BRm(0, 1), g = 0,f0(t, x , u) = 1.



Pontryagin maximum principle IIApplications


x = uy = 〈Mx , u〉 (x(0), y(0)) = (0, 0)

Minimize T such that (x(T ), y(T )) = (0, y f ), with Ω = BRm(0, 1).Assuming M> 6= M.

Example (Zermelo’s problem)

x = v cos(u) + c(y)y = v sin(u)

(x(0), y(0)) = (0, 0) .

T > 0 free, y(T ) = 1, x(T ) ∈ R, f = 0, g(t, x) = x2.Assuming c(y) > v for every y ;

Minimize T such that y(T ) = 1 and x(T ) ∈ R.


Numerical methods



3 Numerical methodsIndirect methodsDirect methods


Numerical methods Indirect methods

Shooting methods I

We remind that the optimality system is:

x = ∂pH(t, x , p, p0, u) (14a)

p = −∂xH(t, x , p, p0, u) (14b)

where:H (t, x(t), p(t), p0, u(t)) = max

v∈ΩH (t, x(t), p(t), p0, v) (15)

In general, p0 can be chosen to −1 if it is not possible, this means that theadmissible trajectories are independent of the cost function.

Let us assume that u(t) is uniquely determined by (15), i.e., u = u(t, x , p). Thus,writing z = (x , p)>, (14) can be written as:

z = F (t, z) .



Shooting methods II

We chose an initial condition z(0) = z i and we look for z i such that transversalityconditions are fulfilled. The transversality condition can be expressed has thezeros of a function G , defined by G (z i ) = R(z i , z(T ; z i )).

The aim is to find z i such that:

G (z i ) = 0 .

this can be done for instance with a Newton method.

Remark

If T is free, we can add T in the state variable, with the equation T = 0 andrescale the problem on [0, 1].

Remark

This procedure can be parallelized. Instead of having an unknown z i we can take asubdivision 0 = t0 < · · · < tN = T and take as unknown, z(ti ). In addition to thetransversality condition, we will have the continuity of z at points t1, . . . , tN−1.



Shooting methodsNewton method

We recall the Newton method.

Given G : Rd → Rd the aim is to find z ∈ Rd such that G (z) = 0. If zk is close toz , then we have:

0 = G (z) = G (zk) + dG (zk) · (z − zk) + o0(z − zk) .

Then we chosezk+1 = zk + dk ,

with dk ∈ Rd solution of:

G (zk) + dG (zk) · dk = 0 .


Numerical methods Direct methods

Full discretization I

For this method, we tackle directly the optimization problem:

min J(T , x , u)

x = f (ti , x , u)u(t) ∈ Ωx(0) =? , x(T ) =?

(16)

More precisely, we will discretize it, i.e., given a subdivision0 = t0 < · · · < tN = T , we consider the unknowns xi ' x(ti ) and ui ' u(ti ). Andwe write X = (x0, . . . , xN) and U = (u0, . . . , uN).We also consider a quadrature formula for J and x = f (x , u), i.e.J(T , x , u) ' J(T ,X ,U) and x = f (t, x , u) becomes constraints of the typeci (X ,U) = 0 for every i ∈ 1, . . . ,N.


Numerical methods Direct methods

Full discretization II

Thus the minimisation problem (16’) becomes:

min J(T ,X ,U)

ci (X ,U) = 0 i = 1, . . . ,Nui ∈ Ω i = 0, . . . ,Nx0 =? , xN =?

(16’)

which is a finite dimensional minimisation problem with constraints.This problem can be solved with a penalty method or with a sequential quadraticprogramming method.

RemarkOther methods are based on Halminton-Jacobi equation.We refer to Bardi and Capuzzo-Dolcetta, Optimal control and viscosity solutionsof Hamilton-Jacobi-Bellman equation, Birkhauser, 1997.


Conclusion

Thanks for you attention


Documents

An introduction to optimal control problem - · PDF file2 Optimal control problems 3 Numerical methods J. Loh eac (BCAM) An introduction to optimal control problem 06-07/08/2014 4