III.

Stochastic Optimization in Continuous Time

The optimization principles set forth above extend directly

to the stochastic case.

The main difference is that to do

continuous-time analysis, we will have to think about the right

way to model and analyze uncertainty that evolves continuously
with time.

To understand the elements of continuous-time

stochastic processes requires a bit of investment, but there is a

large payoff in terms of the analytic simplicity that results.
Lets get our bearings by looking first at a discrete-time
stochastic model. 11

Imagine now that the decision maker

maximizes the von Neumann-Morgenstern expected-utility indicator

dth
(19) E0 s
U[c(t),k(t)]h,
te
------

t=0

where EtX is the expected value of random variable X conditional

on all information available up to (and including) time t. 12
Maximization is to be carried out subject to the constraint that

(20)

k(t+h)

------

k(t) = G[c(t),k(t),q(t+h),h],

k(0) given,

------------------------------------------------------------------------------------------------------------------------------------------------------------

11An encyclopedic reference on discrete-time dynamic programming

and its applications in economics is Nancy L. Stokey and Robert
E. Lucas, Jr. (with Edward C. Prescott), Recursive Methods in
Economic Dynamics (Cambridge, Mass.: Harvard University Press,
1989). The volume pays special attention to the foundations of
stochastic models.
12Preferences less restrictive than those delimited by the von
Neumann-Morgenstern axioms have been proposed, and can be handled
by methods analogous to those sketched below.
21

where {q(t)}8t=-8 is a sequence of exogenous random variables with

a known joint distribution, and such that only realizations up to
and including q(t) are known at time t.

For simplicity I will

assume that the q process is first-order Markov, that is, that

the joint distribution of {q(t+h), q(t+2h), ...} conditional on
{q(t), q(t h), ...} depends only on q(t).
------

For example, the AR(1)

process q(t) = rq(t h) + u(t), where u(t) is distributed

------

independently of past qs, has this first-order Markov property.

Constraint (20) differs from its deterministic version, (6),
in

that

the

transition
factor.
for

this

time

interval

function,

but

not

appears

as

an

argument

necessarily

as

arises

because

q(t+h)

is

the

multiplicative

Thus, (20) is somewhat more general than (6).

generality

of

The need

meant

to

be

"proportional" to h in a sense that will become clearer as we

proceed.
Criterion (19) reflects inherent uncertainty in the
realizations of c(t) and k(t) for t > 0.

Unlike in the

deterministic case, the object of individual choice is not a

single path for the control variable c.
of contingency plans for c.

Rather, it is a sequence

Now it becomes really essential to

think in terms of a policy function mapping the "state" of the

program to the optimal level of the control variable.

The

optimal policy function giving c*(t) will not be a function of

the state variable k(t) alone, as it was in the last section;
rather, it will depend on k(t) and q(t), because q(t) (thanks to
the first-order Markov assumption) is the piece of current
22

information that helps forecast the future realizations q(t+h),

q(t+2h), etc. Since k(t) and q(t) evolve stochastically, writing

c*(t) = c[k(t);q(t)] makes it clear that from the perspective of

any time before t, c*(t) will be a random variable, albeit one
that depends in a very particular way on the realized values of
k(t) and q(t).
Bellmans principle continues to apply, however.

To

implement it, let us write the value function--again defined as

the maximized value of (19)--as J[k(0);q(0)].

Notice that q(0)

enters the value function for the same reason that q(t)
influences c*(t).

If q is a positive shock to capital

productivity (for example), with q positively serially

correlated, then a higher current value of q leads us to forecast
higher qs for the future. This higher expected path for q both
raises raises expected lifetime utility and influences the
optimal consumption choice.
In the present setting we write the Bellman equation as

J[k(t);q(t)] = max

(21)

(
)
{U[c(t),k(t)]h +e dhEtJ[k(t+h);q(t+h)]},
9
0
------

c(t)

where the maximization is done subject to (20).

The rationale

for this equation basically is the same as before. The

contingent rules for {c(s)}8s=t+1 that maximize
8
dsh
Et s
U[c(s),k(s)]h subject to (20), given k(t) and the
te
------

s=t

23

optimal choice c*(t), will also maximize

8
s
Et t e dshU[c(s),k(s)]h subject to (20), given the probability
------

s=t+1

distribution for k(t+h) induced by c*(t).

Equation (21) is the stochastic analogue of (7) for the case
of first-order Markovian uncertainty.

The equation is

immediately useful for discrete-time analysis: just use (20) to

eliminate k(t+h) from (21) and differentiate away.
concern here is with continuous-time analysis.

But our

We would like to

proceed as before, letting the market interval h go to zero in

(21) and, hopefully, deriving some nice expression analogous to
(9).

Alas, life is not so easy.

If you try to take the route

just described, you will end up with an expression that looks

like the expected value of

J[k(t+h);q(t+h)]
h

------

J[k(t);q(t)]
.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This quotient need not, however, converge (as h

defined random variable.

0) to a well-

One way to appreciate the contrast

between the present setup and the usual setup of the calculus is
as follows.

Because J[k(t);q(t)] is a random variable, a plot of

its realizations against time--a sample path--is unlikely to be

differentiable. Even after time is carved up into very small
intervals, the position of the sample path will change abruptly
from period to period as new realizations occur.

Thus,

expressions like the quotient displayed above may have no well24

defined limiting behavior as h

0.

To proceed further we need a

new mathematical theory that allows us to analyze infinitesimal

changes in random variables.

The stochastic calculus is designed

to accomplish precisely this goal.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Stochastic Calculus
Let X(t) be a random variable whose change between periods t
1 and t, DX(t) = X(t)

------

To

simplify

------

X(t

------

1), has mean m and variance

matters

Ill

assume

that

distributed, although

this

is

at

not

DX(t)

all

s .

is normally

necessary

for

the argument. 13
We are interested in the case where DX(t), the change in
random variable X over the period of length 1 between t

------

1 and

t, can be viewed as a sum (or integral) of very small (in the

limit, infinitesimal) random changes.

We would also like each of

these changes, no matter how small, to have a normal

distribution.

Our method, as in the usual calculus, is to divide

the time interval [t

------

1, t] into small segments.

But we need to

be sure that no matter how finely we do the subdivision, DX(t),

the sum of the smaller changes, remains N(m,s2).
To begin, carve up the interval [t
subintervals, each of length h = 1/n.

------

1, t] into n disjoint

For every i e {1,2,...,n},

------------------------------------------------------------------------------------------------------------------------------------------------------------

13For a simplified yet rigorous exposition of

25

these

matters,

let n(i) be a N(0,1) random variable with En(i)n(j) = 0 for i $

j. Suppose that DX(t) can be written as
n

(22) DX(t) = s
t

mh + sh

Then

1/2

n(i)

i=1

since

nh

1,

(22)

is

consistent

hypothesis that EDX(t) = m and VDX(t) = s2.

n

VDX(t) = s

s
t

with

our

initial

For example,

s En(i)n(j)/n = s En(i)2/n =
t
t

i=1 j=1

s .

i=1

Equation (22) expresses the finite change DX(t) as the sum

of tiny independent normal increments of the form mh + sh1/2n. It
is customary to denote the limit of such an increment as h
mdt

sdz,

where

for

any

instant

t,

dz(t)

0 by

limh 0h1/2n(t).
L

When this limit is well-defined, we say that X(t) follows the

Gaussian diffusion process

(23)

dX(t) = mdt + sdz(t),

which means, in notation that is suggestive but that I will

not attempt to define rigorously, that
t

idz(s) = X(t) + m(t t) + s[z(t) z(t)]

X(t) = X(t) + m(t t) + sj
------

------

------

for all t < t. 14

------------------------------------------------------------------------------------------------------------------------------------------------------------

14Again, see Merton, op. cit., for a more rigorous treatment. To

make all this more plausible, you may want to write (22) (for our
26

Think of X(t) as following a continuous-time random walk

with a predictable rate of drift m and an instantaneous rate of
variance (variance per unit of time) s2.

When s = 0, we are back

in the deterministic case and are therefore allowed to assert

that X(t) has time derivative m: dX(t)/dt = m.

But when s > 0,

X(t) has sample paths that are differentiable nowhere.

So we use

a notation, (23), that does not require us to "divide" random

differences by dt.

Because we are looking at arbitrarily small

increments over arbitrarily small time intervals, however, the

sample paths of X(t) are continuous.
Now that we have a sense of what (23) means, I point out
that this process can be generalized while maintaining a
Markovian setup in which todays X summarizes all information
useful for forecasting future Xs.

(24)

For example, the process

dX = m(X,t)dt + s(X,t)dz.

------------------------------------------------------------------------------------------------------------------------------------------------------------

earlier case with t = t

------

1) as

DX(t)

------

s n(i)/en ,
= t
q------

i=1
where n = 1/h is the number of increments in [t
1, t]. We know
from the central-limit theorem that as n
8, the right-hand side
above is likely to approach a limiting normal distribution even
if the n(i)s arent normal (so my assumptions above were
stronger than necessary). Obviously, also, X(t) X(t
h) will
2
be normally distributed with variance hs no matter how small h
is. But X(t)
X(t
h) divided by h therefore explodes as h
0
2
(its variance is s /h). This is why the sample paths of
diffusion processes are not differentiable in the usual sense.
27
------

------

------

------

------

allows the drift and variability of dX to be functions of

the level of X(t) itself, which is known at time t, and of
time.
There is a further set of results well need before tackling
^
the one major theorem of stochastic analysis applied below, Itos
chain rule.

We need to know the rules for multiplying stochastic

differentials.

Were familiar, from the usual differential

calculus, with the idea that quantities of order dt are

important, whereas quantities of order dtm, m > 1, are not.

For

example, in calculating the derivative of the function y2, we

compute h-1 times the limit of (y + h)2

------

y2 = 2yh + h2 as h

0.

The derivative is simply 2y, because the term h2 goes to zero

even after division by h.
stochastic calculus.

The same principle will apply in

Terms of order greater than h are

In particular dt2 = limh 8h2 will be set to zero,

just as always.

discarded.

What about something like the product dzdt?

the limit of h3/2n as h

Since this is

8, it shrinks faster than h and

accordingly will be reckoned at zero:

(25)

dzdt = 0.

Finally, consider dz2 = limh 8hn2. This is of order h, and thus

does not disappear as h gets very small. But the variance of
L

this term can be shown to be 2h2, which is zero asymptotically. 15

------------------------------------------------------------------------------------------------------------------------------------------------------------

15To prove this, note that because n is N(0,1), Vhn2 = E(hn2

28

------

By Chebyshevs inequality, hn2 thus converges in probability to

its expected value, h, as h

0, and so we write

dz2 = dt.

(26)

^
Lets turn now to Itos
famous lemma.

Suppose that the

random variable X(t) follows a diffusion process such as (24).

^
The basic idea of Itos
Lemma is to help us compute the
stochastic differential of the random variable f[X(t)], where
f(Q) is a differentiable function.

If s(X,t) _ 0, then the chain

rule of ordinary calculus gives us the answer: the change in f(X)

over an infinitesimal time interval is given by df(X) = f(X)dX =
f(X)m(X,t)dt.

If s(X,t) # 0 but f(Q) is linear, say f(X) = aX

for some constant a, then the answer is also quite obvious:

in

this special case, df(X) = f(X)dX = am(X,t)dt + as(X,t)dz.

Even if

f(Q) is

nonlinear, however,

there is

often

simple answer to the question weve posed:

^
Itos
Lemma.

be

twice

Let X(t) follow a diffusion process, and let f:R

continuously

differentiable.

differential of f(X) is

(27)

df(X) = f(X)dX +

1
f"(X)dX2.
2
------

------------------------------------------------------------------------------------------------------------------------------------------------------------

h)2 = E(h2n4

------

2h2n2 + h2) = 3h2

------

2h2 + h2 = 2h2.
29

The

stochastic

Comment.

If X follows the diffusion process (24), then,

using rules (25) and (26) to compute dX2 in (27), we get

(28)

df(X) = [m(x,t)f(X) +

s(X,t)2
-----------------------------------

f"(X)]dt + s(X,t)f(X)dz.

Youll notice that (28) differs from the "naive" chain rule only
in modifying the expected drift in f(X) by a term that depends on
the curvature of f(Q).

If f"(X) > 0 so that f(Q) is strictly

convex, for example, (28) asserts that Etdf(X) = Etf[X(t+dt)]

------

f[X(t)] is greater than f(X)m(X,t)dt = f(X)EtdX = f[EtX(t+dt)]

------

f[X(t)].

But anyone who remembers Jensens Inequality knows

that Etf[X(t+dt)] > f[EtX(t+dt)] for convex f(Q), and that the
^
opposite inequality holds for concave f(Q). So Itos
Lemma
should not come as a surprise.16

------------------------------------------------------------------------------------------------------------------------------------------------------------

16In case you dont remember Jensens Inequality, heres a quick

30

^
Motivation for Itos
Lemma.

^
The proof of Itos
Lemma is quite

subtle, so a heuristic motivation of this key result will have to

suffice. 17

Once again Ill rely on a limit argument.

interval length h, Taylors theorem 18

f[X(t+h)]
+

implies that

f[X(t)] = f[X(t)][X(t+h)

------

1
f"{X(t) + x(h)[X(t+h)
2
------

where x(h) e [0,1].

For an

------

X(t)]

------

X(t)]}[X(t+h)

------

X(t)]2,

It may look "obvious" to you that this

converges to (27) as h

0.

Beware.

It turns out to be quite a

chore to ensure that the right-hand side of this expression is

well behaved as h

0, largely because of the complicated

dependence of the term f"{X(t) + x(h)[X(t+h)

Fortunately, as h

------

X(t)]} on h.

0, the randomness in this term does disappear

quickly enough that we can safely equate it to f"[X(t)] in the

limit.

The result is (27).

It should now be clear how one would

------------------------------------------------------------------------------------------------------------------------------------------------------------

sketch of a proof. Recall that a convex function has the

property that gf(X1) + (1 g)f(X2) > f[gX1 + (1 g)X2] Ag e [0,1].
------

------

It is easy to extend this to the proposition that

sp f(X ) >
t i i
i

f(s
tpiXi) for (p1,...,pn) in the unit simplex.

(Try it.)

So for

finite discrete probability distributions were done. (Obviously

concave functions work the same way, with the inequalities
reversed.) Now consider the case in which the random variable X
has an arbitrary continuous density function p(X). We can
approximate Ef(X) by sums of the form

sf(X )p(X )h, each of which

i
i
t
i

must be at least as great as f[s

tXip(Xi)h] if we choose the
i

17For Taylors theorem with remainder, see any good calculus text.
31

^
motivate a multivariate version of Itos
Lemma using the
multivariate Taylor expansion.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The preceding digression on stochastic calculus has equipped

us to answer the question raised at the outset:

What is the

continuous-time analogue of (21), the stochastic Bellman

equation?
To make matters as simple as possible, in analogy with
section IIs time-stationary setup,

Ill

= X(t+h)

the

------

X(t), where X(t) follows

assume
simple

that

q(t+h)

diffusion

process (23), dX = rdt + sdz, for constant r and s.

Under this

assumption Etq(t+h) = rh always, so knowledge of q(t) gives us no

information about future values of q.
depends on

(29)

Thus the value function

the state variable k alone.

J[k(t)] = max

Now (21) becomes

(
)
{U[c(t),k(t)]h + e dhEtJ[k(t+h)]}.
9
0
------

c(t)

Lets carry on by adapting the last sections strategy of

subtracting J[k(t)] from both sides of (21) and replacing e dh by
------

------

dh.

(We now know we can safely ignore the terms in hm for m

> 2.) The result is

(
0 = max {U[c(t),k(t)]h + EtJ[k(t+h)]
c ( t )9
Now let h

0.

------

J[k(t)]

------

dE J[k(t+h)]h}.
t

According to (20), dk = G(c,k,dX,dt), and I

assume that this transition equation defines a diffusion process

32

^
for k. Itos
Lemma then tells us that

(30)

dJ(k) = J(k)dk +

Thus as h

1
J"(k)dk2,
2
------

0, EtJ[k(t+h)]

1
J"[k(t)]Etdk(t)2.
2
------

------

J[k(t)]

Furthermore, as h

J[k(t)]Etdk(t) +
0, EtJ[k(t+h)]

J[k(t)].

So we end up with the following:

PROPOSITION III.1.

(Continuous-Time Stochastic Bellman Equation)

Consider the problem of maximizing E0i80 e dtU(c,k)dt subject to a
------

diffusion process for k controlled by c, and given k(0).

At each

moment, the optimal control c* satisfies the Bellman equation

(31)

0 = U(c*,k)dt + J(k)EtG(c*,k,dX,dt)
1
J"(k)EtG(c*,k,dX,dt)2
dJ(k)dt
2
(
)
1
= max {U(c,k)dt + J(k)Etdk + J"(k)Etdk2
dJ(k)dt}.
2
c ( t )9
0
+

------

------

------

------

Equation (31) is to be compared with equation (9), given in

Proposition II.1.
III.1 is
Define

(32)

quite

Indeed, the interpretation of Proposition

similar

to

that

of

Proposition

II.1.

the stochastic Hamiltonian [in analogy to (10)] as

H(c,k)

_ U(c,k) + J(k)

Etdk
----------------------

dt

Etdk2
1
+ J"(k)
.
2
dt

33

------

--------------------------

The Hamiltonian has the same interpretation as (10), but with a

stochastic twist.

The effect of a given level of "savings" on

next periods "capital stock" now is uncertain.

Thus the

Hamiltonian measures the expected flow value, in current utility

terms, of the consumption-savings combination implied by the
consumption choice c, given the predetermined (and known) value
of k.

The analogy will be clearer if you use (30) to write (32)

as 18

H(c,k)

and

if

= U(c,k) + EtdJ(k) ,
dt

you

----------------------------------------

use

the

ordinary

chain

rule

to write the
Q = U(c,k) +
deterministic Hamiltonian (10) as U(c,k) + J(k)k
dJ(k)/dt.
The stochastic Bellman equation therefore implies the same
rule as in the deterministic case, but in an expected-value
sense. Once again, optimal consumption c* satisfies (11),

H(c*,k)

= max {H(c,k)} = dJ(k).

c

Rather than proceeding exactly as in our deterministic

analysis, I will sacrifice generality for clarity and adopt a
specific (but widely used) functional form for the continuous------------------------------------------------------------------------------------------------------------------------------------------------------------

19The notation in (32) and in the next line below is common.

Since Etdk, for example, is deterministic, (Etdk)/dt can be
viewed as the expected rate of change in k. Since diffusion
processes arent differentiable, Et(dk/dt) is in contrast a
nonsensical expression.
34

time version of (20), dk = G(c,k,dX,dt).

I will assume the

linear transition equation

(33)

dk = kdX

------

cdt

= (rk - c)dt + skdz

(since dX = rdt + sdz).

What form does (31) now assume?

this we have to calculate Etdk and Etdk2.

that Etdk = (rk
that dt2
2 2

s k dt.

(34)

To see

It is clear from (33)

Invoking (25) and (26), and recalling

= 0, we see that dk2 = Etdk2 = k2dX2
2ckdXdt + c2dt2 =
------

c)dt.

------

We thus conclude that c* must solve

(
{U(c,k) + J(k)(rk
c ( t )9

0 = max

------

c) +

1
J"(k)k2s2
2
------

------

dJ(k)}.

In principle this equation is no harder to analyze than was

(9): the two are identical [if G(c,k) = rk

c] aside from the

^
additional second derivative term in (34), due to Itos
Lemma.
So

we

proceed

as

before,

starting

off

------

by

maximizing

the

Hamiltonian.
Since k

is

predetermined and known at

each moment, the

necessary condition for c* to maximize the right hand of (34) is

(35)

Uc(c*,k) = J(k),

which is the same as (12) because Ive assumed here that Gc = 1.

We can also define the optimal policy function c* = c(k),
------

just as before.

By definition c(k) satisfies the equation

35

0 = U[c(k),k] + J(k)[rk

(36)

------

c(k)] +

1
J"(k)k2s2
2
------

------

dJ(k).

One would hope to understand better the implied dynamics of c

by differentiating with respect to the state variable.

The

result is

(37)

Uk(c*,k) + J(k)(r
+

d) + J"(k)ks

------

+ J"(k)(rk

------

c*)

1
J(k)k2s2 = 0,
2
------

where Ive already applied the envelope condition (35).

It is tempting to give up in the face of all these second
and third derivatives; but it is nonetheless possible to
interpret (37) in familiar economic terms.

Lets again define

the shadow price of k, l, by

l _ J(k).

This shadow price is known at time t, but its change over the
interval from t to t + dt is stochastic. Equation (37) differs
from (13) only by taking this randomness into account; and by
writing (37) in terms of l, we can see precisely how this is
done.
To do so we need two observations.

^
First, Itos
Lemma

discloses the stochastic differential of l to be

36

(38)

dl = dJ(k) = J"(k)(kdX

------

cdt) +

1
J(k)k2s2dt
2
------

(verify this), so that

(39)

Etdl = J"(k)(rk

------

1
J(k)k2s2.
2

c) +

------

---------------------

dt
Second, the term J"(k)ks2 in (37) can be expressed as

(40)

J"(k)ks2 =

where R(k) _

------

------

J(k)R(k)s2,

J"(k)k/J(k) should be interpreted as a coefficient

of relative risk aversion.

Using (39) and (40), rewrite (37) in terms of l = J(k) as

Uk(c*,k) + l[r

------

R(k)s

------

d] +

Etdl
---------------------

dt

or, in analogy to (14), as

(41)

Uk + l[r

------

R(k)s2/2] + [(Etdl)/dt

------

lR(k)s2/2]

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

= d,

To compare (41) with (14), notice that under the linear

transition equation (33), r corresponds to the expected value of
Gk; we adjust this expectation downward for risk by subtracting
the product of the risk-aversion coefficient and s2/2.
37

An

identical risk adjustment is made to the expected "capital gains"

term, (Etdl)/dt.

Otherwise, the equation is the same as (14),

and has a corresponding "efficient asset price" interpretation.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Example
An individual maximizes the expected discounted utility of
consumption, E i8e dtU(c)dt, subject to a stochastic capital
------

0 0

accumulation constraint that looks like (33):

dk = rkdt + skdz

------

cdt,

k(0) given.

What is the meaning of this savings constraint?

Capital has a

mean marginal product of r, but its realized marginal product

fluctuates around r according to a white-noise process with
instantaneous variance s2.

The flow utility function is

c1 (1/e)
1
U(c) =
,
1
(1/e)
------

------

----------------------------------------------------------------------------

as in the second part of the last sections example.

To solve the problem Ill make the same guess as before,
that the optimal consumption policy function is c(k) = hk for an
appropriate h.

As will be shown below--and as was the case in a

deterministic setting--the value function J(k) is a linear

function of k1 (1/e), making the risk aversion coefficient R(k)
------

38

defined after (40) a constant, R _ 1/e.

For now I will assume

this, leaving the justification until the end.

How can we compute h in the policy function c(k) = hk?

The

argument parallels our earlier discussion of the nonstochastic

case, which you may wish to review at this point.
Start

by

thinking

about

the

implications

of

the

postulated policy function for the dynamics of capital.

If

c(k) = hk, then

dk = rkdt + skdz

------

c(k)dt = (r

------

h)kdt + skdz.

But as optimal c is proportional to k,

dc = (r

------

h)cdt + scdz.

Above we defined l as J(k); but first-order condition (35)

^
implies that l = U(c) = c-1/e. Application of Itos
Lemma to l
= c-1/e leads to

( )
dl =

2
------2

1 c-1 (1/e)dc +
2

------

------

9 0
2

( )( )(

1 1
1 c-2-(1/e)dc2.
1 +
e
9209 e 09
0
2

22

22

2
22
22
----------2

22

22

------

2
2

Because weve already established that Etdc = (r

------

h)cdt

that dc2 = s2c2dt, we infer from the equation above that

E t dl
c-(1/e)#h
=
dt
e
3

-----------------------

--------------------------------

------

r +

&1*&
1 * 2$
72871 + e 8s 4
------

------

39

and

But there is an alternative way of describing the dynamics

of l:

equation (41) can be written here as

Etdl
---------------------

dt

= l[d

------

(r

Rs2)] = c-1/e[d

------

(r

------

------

s /e)].

So we have derived two potentially different equations

for (Etdl)/dt; clearly the two are mutually consistent if
and only if

( )

( )
1* 2
r + 1 &
1
+
s
e8
2 7

[d

(r

------

------

s /e)] =

22

22

------

9 0
2

22
2

------

------

------

e
2

2
2
2
c

or, solving for h, if and only if

h = r

e(r

------

------

d) +

The implied consumption

(e

------

1)

-----------------------------------------

2e

s .

rule is similar to the one that arose in

the nonstochastic example analyzed earlier, but it corrects for

the unpredictable component of the return to capital.
that we again obtain h = d if e = 1.)

(Notice

The analogy with (16) will

be clearest if the rule is written as

(42)

In

h = (1

(42),

------

e)(r

------

appears

1 2
Rs ) + ed.
2
------

as

the

weighted
40

average

of

the

time-

preference

rate

and

risk-adjusted

expected

return

on

investment.
Problems still arise if h < 0.

In these cases an optimum

fails to exist, for reasons essentially the same as those

discussed in section IIs example.
As a final exercise lets calculate the value function J(k)
and confirm the assumption about its form on which Ive based my
analysis of the optimal consumption policy function.

In the
^
process well learn some more about the importance of Itos
Lemma. One way to approach this task is to calculate the (random)
path for k under an optimal consumption plan, observe that the
optimal contingency rule for consumption is c = hk, and then use
this formula to compute the optimal (random) consumption path and
lifetime expected utility.

Indeed, we took a very similar tack

in the deterministic case.

So we start by asking what the

optimal transition equation for the capital stock, dk = (r

h)kdt + sdz, implies for the

level of k.

------

[Throughout the

following discussion, you should understand that h is as

specified by (42).]
Observe

first

that

the

optimal

capital-stock

transition equation can be written as

dk/k = (r

------

h)dt + sdz.

A crucial warning.

You might think that dk/k is the same thing

as dlog(k), as in the ordinary calculus.

41

If this were true, we

would conclude that the capital stock follows the stochastic

process
t

log[k(t)] = log[k(0)] + (r

------

i
h)t + s dz(s),
j
2

or, equivalently, that

k(t) = k(0)e(r h)t + s[z(t) z(0)].

------

------

But this is incorrect.

(dk/k)

------

1 2
s dt = (r
2
------

------

^s Lemma tells us that dlog(k) =

Ito
------

1 2
s )dt + sdz.
2

------

[The reason for this

divergence is Jensens Inequality--log(Q) is a strictly concave

function.] It follows that the formula for k(t) below is the
right one:
(43)

At

2
k(t) = k(0)e(r h s /2)t + s[z(t) z(0)].
------

an

optimum,

------

k(t)

------

will

be

conditionally

distributed, with an expected growth rate of r

e(r h)t. 20

------

lognormally

h: E k(t)/k(0) =
0

------

As a result of (43), the value function at t = 0 is

------------------------------------------------------------------------------------------------------------------------------------------------------------

20If X is a normal random variable with mean m and variance s2, eX

is said to be lognormally distributed. The key fact about
lognormals that is used repeatedly is that when X is normal,
2
EeX = em+s /2.

For a proof, see any good statistics text.

42

J[k(0)] =

-1

------

------

2
z

2
2
c

(
E0{ i
e dt[hk(t)]1 (1/e)dt
j
9
------

------

------

------

)
}
0

-1

(i dt #
{je E03hk(0)e(r
9

e
2

2
------

------

2
z

2
c

------

1 (1/e)
dt
4

h s /2)t+s[z(t) z(0)]$
2

------

------

------

)
}
0

------

------

------

e
2

------

2
z

------

(
1 (1/e) i dt [1 (1/e)](r
{ hk(0)
je e
9

-1

2
2
c

------

------

------

1)

h s /2e)t

------

------

dt

------

------

d0

------

------2

2
z

2
c

[hk(0)]1 (1/e)

------

-1

------------------------------------------------------------------------------------------------------------------------------------------------------

9
2

------

(e

------

1)(r

------

Rs /2

------

d)

------

------

}.
2

Youll recognize the final product above as the same formula

for J[k(0)] that we encountered on p. 16 above, with the sole
amendment that the risk-adjusted expected return r

------

Rs2/2

replaces r everywhere [including in h; recall (42)]. 21

d

------

(e

------

1)(r

------

Rs2/2

------

Because

d) = h, h > 0 ensures convergence of the

integral defining J(k). Finally, J(k) is a linear function of

k1 (1/e), as claimed earlier.
------

There is another, more direct way to find the value

------------------------------------------------------------------------------------------------------------------------------------------------------------

21To move from the second to the third equality above, I used the
fact that the normal random variable [1
(1/e)]s[z(t)
z(0] has
2 2
mean zero and variance [1
(1/e)] s t conditional on t = 0
information.
43
------

------

------

function, one that also applies in the deterministic case.

[Had

we known the value function in advance, we could have used (35)

to compute the consumption function without trial-and-error
guesses.]

By (35), the optimal control must satisfy

c(k) = J(k) e.
------

Thus by (34),

[J(k)]1 e
0 =
+ J(k)[rk
1
(1/e)
------

---------------------------------------------------------------

1
J(k) e] + J"(k)k2s2
2
------

------

------

------

dJ(k).

This is just an ordinary second-order differential equation which

in principle can be solved for the variable J(k).

You may wish

to verify that the value function J(k) we derived above is indeed

a solution.

To do the nonstochastic case, simply set s2 = 0.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The similarities between this example and its deterministic

analogue are striking.

They are not always so direct.

Nonetheless, it is noteworthy that for the linear state

transition equation considered above, there exists a stochastic
version of Pontryagins Maximum Principle.
problem in full generality, 22

One could attack the

but as my goal here is the more

modest one of illustrating the basic idea, I will spare you this.

------------------------------------------------------------------------------------------------------------------------------------------------------------

22As
does
Jean-Michel
Bismut,
"Growth
and
the
Optimal
Intertemporal Allocation of Risks," Journal of Economic Theory 10
(April 1975): 239-257.
44

PROPOSITION

III.2.

(Stochastic

Maximum

Principle)

Let

c*(t) solve the problem of maximizing

d(s t)
E0i
U[c(s),k(s)]ds
je
------

------

subject to the transition equation

dk(t) = rk(t)dt + sk(t)dz(t) - c(t)dt,

k(0) given,

where z(t) is a standard Gaussian diffusion.

Then there exist

costate variables l(t) such that if z(t) is the instantaneous

conditional covariance of l(t) and z(t), the risk-adjusted
Hamiltonian
H~[c,k(t),l(t),z(t)] _ U[c,k(t)] + l(t)[rk(t)

------

c] + z(t)sk(t)

is maximized at c = c*(t) given l(t), z(t), and k(t); that is,

(44)

~
dH
(c*,k,l,z) = Uc(c*,k)
dc
------------

------

l = 0

at all times (assuming an interior solution).

Furthermore,

the

differential

costate

variable

obeys

the

stochastic

equation

(45)

dl = lddt
= lddt

------

------

~
dH
(c*,k,l,z)dt + zdz
dk
[Uk(c*,k) + lr + zs]dt + zdz

------------

45

for dk = rkdt
To

------

understand

c*dt + skdz and k(0) given

how

this

proposition

follows

from

our

earlier discussion, observe first that because l will again

equal J(k), the instantaneous conditional covariance of l(t)
and z(t) can be seen from (25), (26), and (38) to be

(46)

z = (E dldz)/dt = J"(k)sk.
t

Thus, with reference to the definition (32) of the unadjusted

stochastic Hamiltonian, given here by

H(c,k)

= U(c,k) + J(k)(rk

------

c) +

1
J"(k)s2k2,
2
------

we have

H~(c,k,l,z)

H(c,k)

1
J"(k)s2k2 =
2

------

H(c,k)

------

lR(k)s k/2,

where R(k) is the relative risk-aversion coefficient defined

~ as the expected
above. Accordingly, we can interpret H
instantaneous flow of value minus a premium that measures the
riskiness of the stock of capital currently held.
With (46) in hand it is easy to check the prescriptions of
the Stochastic Maximum Principle against the results weve
already derived through other arguments.
corresponds directly to (35).

Clearly (44)

Likewise, if you multiply (37) by

46

dt and combine the result with (38), you will retrieve (45).
IV.

Conclusion
These notes have offered intuitive motivation for the basic

optimization principles economists use to solve deterministic and

stochastic continuous-time models.

My emphasis throughout has

been on the Bellman principle of dynamic programming, which

offers a unified approach to all types of problems.

The Maximum

Principle of optimal control theory follows from Bellmans

approach in a straightforward manner.
I have only been able to scratch the surface of the topic.
Methods like those described above generalize to much more
complex environments, and have applications much richer than
those I worked through for you.

The only way to gain a true

understanding of these tools is through "hands on" learning:

must apply them yourself in a variety of situations.

As I noted

at the outset, abundant applications exist in many areas of

economics.

I hope these notes make this fascinating body of

research more approachable.

47

you