Advanced Macroeconomics: Dynamic Programming and Reinforcement Learning

Objectif

Recursive methods are the cornerstone of dynamic optimization. The objective of this course is to offer an intuitive, yet rigorous, introduction to recursive tools and their applications in economics. After having presented the mathematical foundations of dynamic programming, we will illustrate its application by studying a series of foundational papers on growth theory, labor economics and learning. In the second part of the course, we will study recent advances in model free algorithms and reinforcement leaning.

On completion of this course, students should know how to:

• Write dynamic optimization problems in recursive forms;
• Determine whether the problem is well-defined and characterize its optimality conditions;
• Numerically solve policy functions;
• Implement model-free algorithms.

Plan

Part I: Dynamic Programming

Lecture 1: Principle of Optimality

Lecture 2: Dynamic Programming 

• Application: Optimal Growth

Lecture 3: Optimal Stopping

• Application: Job Search

Lecture 4: Bayesian Learning 

• Application: Matching

Part II: Reinforcement Learning

Lecture 5: The Exploration-Exploitation Dilemma

• Application: Multi-Arm Bandit

Lecture 6: Reinforcement Learning

Références

We will cover a series of papers whose list will be communicated during the course. Most of the material will be contained in the slides, but students are encouraged to also consult the following books.

Main references

• Ljungqvist, Lars et Thomas Sargent, Recursive macroeconomic theory, MIT Press
• Stokey, Nancy, Lucas, Robert et Prescott Edward, Recursive Methods in Economic Dynamics, Harvard University Press
• Sutton, Richard and Andrew Barto. Reinforcement Learning: An Introduction. Bradford Books; second edition.

Additional references

• Fleming, Wendell et Raymond Rishel, Deterministic and Stochastic Optimal Control, Springer-Verlag, New-York 
• Kamien, Morton et Nancy Schwartz, Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management, Elsevier
• Luenberger, David, Optimization by Vector Space Methods, John Wiley & Sons 
• Liberzon, Daniel, Calculus of Variations and Optimal Control Theory: A Concise Introduction, Princeton University Press