This is just mathematical background needed to understand the book.It contains the Math Appendix and Ch5 (the Math tools chapter). I will collect substantive results from the book in a separate post.
For further reference:
- Bertsekas-Shreve 1978: Stochastic Optimal Control, The Discrete Time Case (fairly advanced due to its generality and abstractness, to my surprise)
- Yong-Zhou 1999: SMP & HJB
- Fleming-Soner 2006: Controlled Markov Processes and Viscosity Solutions
- Oksendal-Sulem 2007: Applied Stochastic Control of Jump Diffusions
- Pham 2010: Cont-time Stochastic Control and Optimization with Fin Applications
- Touzi 2013: Optimal Stochastic Control, Stochastic Target Problems, and Backard SDE (advanced)
Appendix: Stochastic Calculus
definition of Brownian Motion;
definition of stochastic (Ito) integral. Shorter way:
for adapted, L2 proecss , define
this is called an Ito process, and often written as . Can show that this SP is a martingale.
More generally, Ito process can be written as
generally, we just need and to be adapted and satisfy certain integrability conditions. But in the special case where , the equation is called SDE. But namewise, book also mentioned that Ito processes are stochastic processes satisfying SDEs with Brownian noise terms.
definition of stochastic (Ito) integral. Rigorous way:
- define Ito integral for simple functions
- prove that any can be approximated, in , by sequence of simple functions
- define Ito integral for as the limiting value of the Ito integral of the sequence of simple functions
Ito isometry: for adapted ,
infinitesimal generator: the generalization of derivative of a function, to make it applicable to stochastic process.
This is the generator of an Ito process satisfying a certain SDE, e.g., .
Jump Processes
Poisson process
, valued in
, with intensity param
, is a SP s.t.:
- , a.s.
- has Poisson distro with param :
- has independent increments: is independent of
- has stationary increments:
Classic result 1: time between successive jumps of are independent, and exponentially distributed.
compensated Poisson process:
with
. Note that this is a martingale.
as with BM, we can define stochastic integrals wrt compensated Poisson processes in a way that the resulting integral process is a martingale.
let be adapted, define stochastic integral of wrt by:
, where the ’s are jump times.
- need , not , to make integral a martingale.
- alt. def: replace first term with , where , which in this case is either 0 or 1. sum over a continuum of , what is the formal def?
Ito formula for Poisson process
recall we can write
Ito’s formula for such process
is:
suppose
satisfies
for differentiable
. Then:
or written in compensated poisson process.
we also see from this (compensated version) formula that the generator of the process is
jump diffusion
Ito formula for jump diffusion for the above , let be defined by , then:
again, common to write it using compensated poisson process .
compound Poisson process is built out of:
- a Poisson process with intensity
- a collection of iid RVs , with common distro
. The process jumps when Poisson event arrives, but the jump size is drawn from .
We can show, as before:
defined by is a martingale
we can define stochastic integral wrt compound Poisson too.
note that
a Ito’s formula for where
Doubly Stochastic Poisson Processes (Cox process)
these are jump processes which has stochastic intensity
given counting process , we want its intensity process be stochastic
the approach is to give a way to compute the probability that an event arrives at , given info we have at time : to define , where is the natural filtration generated by .
this means:
the driver of the intensity process can be diverse, leading to Feller/OU/Hawkes processes.
as before, can define its compensated version which is a martingale; can define stochastic integral wrt the compensated doubly stochastic Poisson process, and can derive a Ito’s formula for such integral processes, and from which we can derive an expression for the generator of the joint process .
Feynman-Kac
certain linear PDEs are related to SDEs.
Let be an Ito process satisfying: .
The generator of is then where
Now suppose we try to solve PDE: , with terminal condition ,
then we have a probabilistic representation of solution :
(note there is a typo in the book)
Consider the simplest example:
Now introduce a BM , and define
is a martingale, and Markov: for some
use Ito’s lemma to write out
devide the above by and take limit, we get
by definition, . Thus, satisfies the PDE. Recall .
Ch5 Stochastic Optimal Control and Stopping
A few motivating examples (just to be familiar with notation)
Merton Problem
value function: , where:
- at , place dollars in risky asset
- wealth level is
state dynamics follow:
is the admissible set, the set of -predictable, self-financing strategies satisfying . (to prevent doubling strategies)
Optimal Liquidation
state dynamics follow:
- (note the sign)
is the set of -predictable, non-negative bounded strategies (excluding repurchasing of shares, and keep liquidation rate finite)
optimal Limit Order placement identical value function expression, just change the to , which means that agent posts a LO at when current stock price is .
state dynamics:
- denotes market orders