RL by Tsitsiklis 【Notes待完成】

The lecture of John Tsitsiklis: Reinforcement learning
https://www.youtube.com/watch?v=fbmAsxbLal0

The value function indicates the value of the current state:
$V^*(s^t) = min_\pi(E(\sum_{i=t..T}\gamma^t\cdot cost(\pi,s^i))$
$= min_a(cost(a,s) + \gamma \cdot \sum_{s'}(P(s'|s,a)V^*(s')))$ (bellman’s)
is a minimization over policies (functions). And what we want at the end is just the best policy.

• Some can be solve in Poly time by reformulating it as LP (check MAB Gittins 79)
• The curse of dimensionality
• Intractable -> APPROXIMATE it (the function). Linearly or not (Neural Network)
  3 approaches
• Policy network
• Value network
  – From value to policy : $min_a c(a,s) + E(s')$
  – Look ahead: Monte-Carlo Tree Search
  – Actor-Critic methods: given $\pi$ learn $V$ then aid to improve $\pi$

Approximate Policy Iteration

• given $\pi_0$, do simulation to get global reward
• from the loss, train the value function $V$ (by NN or whatever)
• now from $V$, update $\pi$
  

Discrete actions -> ocillations :
incremental methods: update V little by little.