Introduction to Optimal Control

Daniel A. (MSc. & Science Ambassador)

Department of Department of Electromechanical Engineering

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Introduction to Optimal Control

Optimal control deals with the most celebrated problem of modern control theory,
”determination of best control strategy, selecting optimum control strategy, selecting
optimum control vector u, that minimize cost function or performance index”.
Consider an LTI system with initial condition x0 :

ẋ = Ax + Bu
y = Cx + Du (1)

The objective of optimal control is to determine the control vector u that forces the behavior
of the dynamic system at the required final state by maximizing the required cost function and
at the same time fulfills the physical constraints.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Cost Function

The cost function has a general representation of:

Z tf
t=tf
J = θ[x(t), t] |t=t0 + φ(x(t), u(t), t)dt (2)
t0

t=t
The first term θ[x(t), t] |t=t0f is called terminal error optimization problem (Terminal cost
Rt
function) and the second term t0f φ(x(t), u(t), t)dt is called optimal cost function over the
entire trajectory.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Performance Indices

The cost function (performance index) may be a function that optimizes energy consumed,
optimize the time required to reach the desired state, or minimize the tracking or terminal
value error. Depending on the desired optimization problem we have five performance indices.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Minimum Time Control Problem

The aim is to determine a control vector that derives the state of the dynamic system from its
initial value x0 to its final value xf in minimum time. The best example is car racing where
finishing time is the only required parameter to be minimized. The cost function for minimum
time is given as
Z tf
J= dt (3)
t0

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Minimum Terminal Error Control Problem

The aim is to determine a control vector u that minimizes terminal error. Consider a system
with state x we want to reach xd after a certain time t, in terminal control problem we need to
determine control signal u that minimize ξ = xf − xd . The best example is rocket launching,
where the emphasis is put on target only. The cost function of this control problem is:

J = (xf − xd )T S(xf − xd ) (4)

where S is n × n, symmetric positive definite matrix.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Minimum Effort/Energy Control Problem

The aim is to determine the minimum amount of energy that is required to derive the system
from x0 to xf . The best example is car braking where stopping time is required to be
achieved with minimum energy. The cost function is:
Z tf
J= uT Ru dt (5)
t0

where R is m × m, symmetric positive definite matrix.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Servo-Mechanism or Tracking Problem

This control problem aims to determine a control vector u that minimizes the tracking error
when driving the system from any initial state x0 to the desired state (final state). If the
desired trajectory is xd over the course from t0 to tf , the control vector u is required to
minimize e = x − xd . The best example is space shuttle tracking and control, where
tracking error, energy consumption, and the difference between desired terminal point and the
shuttle final point are all required to be minimized. The cost function for this control problem
is:
Z tf
J= ((x − xd )T Q(x − xd )) dt (6)
t0

where Q is n × n, symmetric positive definite matrix.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Servo-Mechanism or Tracking Problem (Cont.)

If we want to minimize the energy required by the system, in addition to minimizing tracking
error the cost function becomes the sum of tracking error and minimum effort cost function.
Z tf Z tf
J= ((x − xd )T Q(x − xd )) dt + uT Ru dt
t0 t0
Ztf
= ((x − xd )T Q(x − xd ) + uT Ru) dt (7)
t0

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Servo-Mechanism or Tracking Problem (Cont.)

If we add terminal error minimization to the problem in addition to minimizing tracking error
and energy optimization, the cost function takes general form,
Z tf
J = (xf − xd )T S(xf − xd ) + ((x − xd )T Q(x − xd ) + uT Ru) dt (8)
t0

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Regulator Problem

This control problem is a special case of optimal tracking problem where the desired final state
is a point in space, particularly an origin xd = 0. The best example is restoring the equilibrium
point of any dynamic system. The tracking optimal problem cost function is:
Z tf
T
J = xf Sxf + (xT Qx + uT Ru) dt (9)
t0

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Control Problem Formulation

• Optimize a cost function J while fulfilling the state dynamics of the system.
• Determine a control signal u that minimizes:
Z tf
Υ(x, ẋ, u) dt
t0

subject to:

ẋ = Ax + Bu

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

General Function Including Constraints

• Define a general function that includes the constraint and the function to be optimized:

φ(x, ẋ, u) = Υ(x, ẋ, u) + λT (Ax + Bu − ẋ)

• The optimization problem reduces to unconstrained optimization:

Z tf
Υ(x, ẋ, u) + λT (Ax + Bu − ẋ) dt


t0

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Euler-Lagrangian Equation for Optimum

• For the function to have an optimum value, the Euler-Lagrangian equations must be
satisfied:
 
∂φ d ∂φ
− =0
∂x dt ∂ ẋ

 
∂φ d ∂φ
− =0
∂u dt ∂ u̇

 
∂φ d ∂φ
T
− =0
∂λ dt ∂ λ̇T

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Problem Statement

The problem is determining control law u that minimizes the cost function:
Z tf
xT Qx + uT Ru + λT (Ax + Bu − ẋ) dt


J=
t0

subject to:

ẋ = Ax + Bu

with the final state xf = 0.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Cost Function and Weights

• Q: Weighing matrix for state error.

• R: Weighing matrix for input.
• Design choices:
• Minimize state error: Increase Q.
• Minimize input: Increase R.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Choosing Weights

Possible starting points for Q and R:

• Simplest choice: Q = I, R = ρI. Vary ρ for desired response.
• Diagonal weights:
 
q1 0 0 0
 0 q2 0 0  1
 0 0 q3 0  ,
Q=  qi =
x̃2i
0 0 0 q4

• Output weighting: z = Hx, Q = xT x, R = ρI.

• Trial and error on weights.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Euler-Lagrangian Equations

Using Euler-Lagrangian equations:

 
∂φ d ∂φ
− =0
∂x dt ∂ ẋ
 
∂φ d ∂φ
− =0
∂u dt ∂ u̇
 
∂φ d ∂φ
− =0
∂λT dt ∂ λ̇T

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Control Law

Let λ = P x, where P is a positive definite symmetric matrix. The control law is:

u = −Kf x = −R−1 B T P x

where Kf is the optimal feedback gain matrix.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Riccati Equation

The Riccati equation is:

Q + P A + AT P − P BR−1 B T P = 0

Solving for P gives the optimal control law.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Closed-Loop System

The closed-loop system becomes:

ẋ = (A − BKf )x

where Kf = R−1 B T P .

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Block Diagram

ud + u + x y
- B ʃ C
+

Kf

Figure: Block diagram representation of LTI system with optimal state feedback controller.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Steady State Reference Tracking

In control systems, it is often desired to achieve fast convergence of the controlled output y to
a specific nonzero constant set-point value r. The set-point values (xd , ud ), xd , represents the
desired state of the system and while ud represent the desired control signal that the output of
system require to follow the reference input.

ẋ = Ax + Bu
y = Cx + Du (10)

At this point, the rate of change of the state variables will be zero, and the output is expected
to be equal to the reference values. Hence, for a pair (xd ,ud ), equation 10 will be:

Axd + Bxd = 0
r = Cxd + Dud (11)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Matrix Representation

    
−A B −xd 0
= (12)
−C D ud r

This corresponds to an LQR criterion of the form:

Z tf
J= (ỹ T Qỹ + ũT Rũ) dt (13)
t0

where ỹ = y − r and ũ = u − ud .

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Three Distinct Cases, Case-I

To understand when this is possible, three distinct cases should be considered:

• Case-I: When the number of inputs m, is strictly smaller than the number of controlled
outputs p, we have an under-actuated system. In this case, the system of equation 12
generally does not have a solution because it presents more equations than unknowns.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Case-II

• Case-II: When the number of inputs m is equal to the number of controlled outputs p,
equation 12 always has a solution as long as the Rosenbrock’s system matrix
 
sI − A B
P (s) = (14)
−C D

is non-singular for s = 0. This means that s = 0 should not be an invariant zero of the
system and therefore it cannot also be a transmission zero of the transfer function
C(sI − A)−1 B + D.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Case-III

• Case-III: When the number of inputs m is strictly larger than the number of controlled
outputs p, we have an over-actuated system and equation 12 generally has multiple
solutions.
When P (0) is full row-rank, i.e., when it has n + p linearly independent rows, the
(n + p) × (n + p) matrix P (0)P (0)T is non-singular and one solution to equation 12 is
given by
   
xd T T −1 0
= P (0) (P (0)P (0) ) (15)
ud r

Also in this case, s = 0 should not be an invariant zero of the system because otherwise
P (0) cannot be full rank.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Set-Point Problem

The optimal set-point problem can be reduced to that of optimal regulation by considering an
auxiliary system with state x̃ = x − xd , ũ = u − ud whose dynamics are:

x̃˙ = A(x − xd ) + B(u − ud )

x̃˙ = Ax̃ + B ũ (16)

We can then regard equation 13 and equation 16 as an optimal regulation problem for which
the optimal solution is given by:

ũ = −Kf x̃ (17)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal Control for Set-Point Problem

Going back to the original input and state variables u and x, we conclude that the optimal
control for the set-point problem defined by equation 10 and equation 13 is given by:

u = −Kf (x − xd ) + ud (18)

Since the solution to equation 12 can be written in form:

xd = F r (19)
ud = N r (20)

The values of F and N are determined using equation 11, by replacing for xeq = F r and
xd = N r:

Axd + Bud = 0 ⇒ AF r + BN r = 0 ⇒ (AF + BN ) = 0 (21)

Cxd + Dud = r ⇒ CF r + DN r = r ⇒ (CF + DN ) = I (22)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Solving for Matrices N and F

Solving for F from equation 21 and replacing in equation 22, the values for matrices N and F
is:

N = (D − CA−1 B)−1 (23)

F = −A−1 B(D − CA−1 B)−1 (24)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Steady State Reference Tracking

ud
N

r xd + u + x y
F + -
Kf + B ʃ C
+

Figure: Steady state reference tracking

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Alternative Method: Steady State

At steady state:

u = −Kf x + Ko r (25)

Replacing for u in the state space equation:

ẋ = Ax − Bu ⇒ ẋ = (A − BKf )x + BKo r (26)

As time goes to infinity (t → ∞), the state x goes to the desired state xd and the input u
tends toward desired input ud , having dynamic equation:

ẋd = Axd + Bud ⇒ ẋd = (A − BKf )xd + BKo r (27)

ud = −Kf xd + Ko r (28)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Tracking Error

The tracking error x̃ = x − xd is then:

x̃˙ = (A − BKf )x̃ (29)

Since ẋd is zero at t → ∞, then the dynamic equation 27 can be written as:

xd = −(A − BKf )−1 BKo r (30)

Replacing equation 30 in equation 28, we have:

ud = Kf (A − BKf )−1 BKo r + Ko r

= (Kf (A − BKf )−1 BKo + Ko )r (31)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Output at Steady State

The output of the dynamic system y goes to r as t → ∞:

y = Cxd + Dud = r (32)

From equation 32, by replacing for ud and xd , the value of matrix Ko is:

Ko = −(C(A − BKf )−1 B − D((Kf (A − BKf )−1 B + I))−1 (33)

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Steady State Reference Tracking (Alternative)

r u + x y
Ko
+ B ʃ C
- +

Kf

Figure: Steady state reference tracking

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

State Observer

In cases where the states of the dynamic system are not directly accessible, a state observer
can be utilized to estimate the state of the system, thereby allowing the use of the estimated
state for control purposes.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal State Estimation

The state feedback controller gain matrix design problem involves finding a gain matrix that
can stabilize a given linear system by controlling its states. On the other hand, the state
estimator gain matrix design problem involves finding a gain matrix that can estimate the
states of a given linear system based on its input and output signals. Interestingly, these two
problems obey a duality property.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Duality Property

This means that the solution to one problem can be used to solve the other problem. In this
context, the problem of finding the state feedback controller gain matrix using a linear
quadratic regulator (LQR) can be used to design an optimal state estimator gain matrix for
linear time-invariant systems.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

LQR Approach

The LQR approach is a widely used method for designing state feedback controllers. It
involves minimizing a quadratic cost function that captures the system’s performance and
control effort. The optimal state feedback controller gain matrix is obtained by solving the
associated algebraic Riccati equation.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal State Estimator Design

The problem of designing an optimal state estimator gain matrix involves minimizing a similar
quadratic cost function that captures the difference between the actual state and the
estimated state. The optimal estimator gain matrix is obtained by solving another algebraic
Riccati equation.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Duality in Cost Functions

The duality property between these two problems arises because the same cost function is used
in both cases, but with different weighting matrices. In other words, the solution to one
problem can be used to solve the other problem by simply swapping the weighting matrices.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Optimal State Estimator Cost Function

To design an optimal state estimator gain matrix using LQR, we define a similar cost function
as follows:
Z tf
(x̂ − x)T Q(x̂ − x) + (y T Ry y) dt


J=
t0

where Q and Ry are weighting matrices that capture the relative importance of the output
and state estimation error, respectively.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Algebraic Riccati Equation for Estimator

The optimal estimator gain matrix Lc is obtained by solving the following algebraic Riccati
equation:

Q + P A + AT P − P C T Ry−1 CP = 0

where P is any symmetric positive definite matrix. Once Lc = P C T Ry−1 is obtained, it can be
used to estimate the system’s state accurately.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Duality in Mathematical Structure

The duality property between the two problems arises from the fact that they share the same
mathematical structure. Specifically:
• The state feedback controller gain matrix design problem can be formulated as a
quadratic optimization problem using the LQR technique.
• The state estimator gain matrix design problem can be formulated as a quadratic
optimization problem using the LQE technique.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

LQR and LQE Techniques

The LQR and LQE techniques are based on minimizing a cost function that measures:
• The deviation of the system’s states from a desired trajectory (LQR).
• The deviation of the estimated states from the actual states (LQE).
The cost function is formulated as a quadratic function of the system’s states and inputs.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)

Class Examples

For the class examples

• Design a Linear quadratic regulator for the systems.

Introduction to Optimal Control Daniel A. (MSc. & Science Ambassador)