5 of 5 written

Optimal Control

Computing controls for the systems a physics engine simulates, from LQR and dynamic programming to iLQR and MPC, with the Bellman bridge to reinforcement learning.

Part 0

LQR

The LQR Problem

The optimal-control problem with a closed-form answer: balance a pole by solving a Riccati equation.

LQR

Part 1

Dynamic Programming

Dynamic Programming and the Bellman Equation

Value functions and the Bellman equation, the shared root of optimal control and reinforcement learning.

Bellman

Part 2

Trajectory Optimization

iLQR and Trajectory Optimization

Local LQR around a trajectory, iterated into a swing-up: the workhorse of model-based control.

iLQR

Part 3

MPC

Model Predictive Control

Re-solve a short-horizon optimal control problem every step: the method behind modern legged and manipulation control.

MPC

Part 4

Continuous Optimal Control

Pontryagin and Hamilton-Jacobi-Bellman

The continuous-time optimality conditions, and how HJB becomes the Bellman equation RL solves by sampling.

Pontryagin