Introduction to stochastic processes

Objective

This course is an introduction to discrete-time martingales and Markov chains and their applications in statistics. A Markov chain is a dynamic model (easy to simulate) which links the present state of one or more variables with their recent past. Markov chains appear naturally in a large number of statistical models, in finance, biology (in the study of population dynamics) and economics (VAR model). The autoregressive models studied in time series are a simple linear example. Due to their flexibility, they have also become a powerful simulation tool (the Markov chain Monte Carlo or MCMC method). We will first present a few concepts and theorems that are important for martingales. We will then look more specifically at the behaviour of finite-state Markov chains: communication properties, the existence of a stationary law (ergodic theorem). The continuous case enables us to introduce Nummelin's powerful renewal and extension tools. Applications in law simulation (MCMC algorithm) and stochastic optimisation algorithms (simulated annealing algorithm, Robin-Moroe) will be examined in lectures and tutorials.

Assessment:

The overall grade for the course will be the average of the continuous assessment (“contrôle continu”, CC) grade (25%) and the written final exam (75%).

The continuous assessment grade is made up of two elements, each graded out of twenty points: (i) the grade for attendance at tutorial sessions (TD), whose attendance is mandatory, and (ii) the grade for participation in tutorial sessions. The CC grade is calculated as follows: 50% of the attendance grade + 50% of the participation grade.

The attendance grade is calculated according to the scale available on the school's intranet.

Planning

1. Basic concepts, examples -Process, definitions, examples of Markov chains
2. Introduction to discrete-time martingales -Stopping time, martingales versus Markov chains, fundamental theorems, applications to stochastic optimisation algorithms
3. Discrete-state Markov chains -Definitions, properties, classification of Markov chains (recurrence, transience), probability/invariant measure
4. General-state Markov chains -Strong Markov properties, concepts of atom and small set, Nummelin extension, ergodic theorem and central limit theorem, applications: law simulation (MCMC)/simulated annealing.

References

BREMAUD P. (1999). Markov Chains, Gibbs Fields, Monte Carlo Simulation, and Queues, Springer. [21 BRE 00 A]
WILLIAMS D. (1997). Probability with Martingales, Cambridge University Press. [16 WIL 00 A]