【論文講義】正交策略梯度法和自動駕駛應用

【講義】正交策略梯度法和自動駕駛應用

本文是論文Orthogonal Policy Gradient and Autonomous Driving Application的講解講義,本文中我們從一個關於獎勵函數的回報梯度定理出發,證明了"當策略梯度向量和Q-Value向量正交時,獎勵函數值爲極大值",由此得出了一種實時逼近最優值的方法,並實現了這種方法且應用在了智能自動駕駛上.

---

關於獎勵函數的回報梯度的定理

在此先證這個定理:在MDP(Markov Decision Process)中: 平均回報函數$\rho$和策略$\pi$及Q函數$Q^{\pi}$滿足:

$\frac{\partial \rho}{\partial \theta} = \sum_{a}d^\pi(s) \sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a)$

證明：

累計獎勵可寫作:$V^{\pi}(s) = \sum_a \pi(s,a) Q^\pi(s,a)$

可得:
$\frac{\partial V^{\pi}(s)}{\partial \theta} =\frac{\partial}{\partial \theta} \sum_a \pi(s,a) Q^\pi(s,a)$
$=\sum_a (\frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) + \pi(s,a)\frac{\partial Q^\pi(s,a)}{\partial \theta} )$
$=\sum_a (\frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) + \pi(s,a)\frac{\partial }{\partial \theta}(R^s_a-\rho(\pi)+\sum_{s'}P^a_{ss'} V^\pi(s')) )$
$=\sum_a (\frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) + \pi(s,a)(-\frac{\partial \rho}{\partial \theta}+\sum_{s'}P^a_{ss'} \frac{\partial V^\pi(s')}{\partial \theta}) )$

由$d^\pi$項累加可得:
$\sum_s d^\pi(s) \frac{\partial \rho}{\partial \theta} = \sum_s d^\pi(s) \sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) + \sum_s d^\pi(s) \sum_a \pi(s,a) \sum_{s'}P^a_{ss'} \frac{\partial V^\pi(s')}{\partial \theta}) - \sum_s d^\pi(s) \frac{\partial V^{\pi}(s)}{\partial \theta}$

再由$d^\pi$的恆定性我們可得:
$\sum_s d^\pi(s) \frac{\partial \rho}{\partial \theta} = \sum_s d^\pi(s) \sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) + \sum_{s' \in S} d^\pi(s') \frac{\partial V^{\pi}(s')}{\partial \theta} - \sum_{s \in S} d^\pi(s) \frac{\partial V^{\pi}(s)}{\partial \theta}$
$\Rightarrow \frac{\partial \rho}{\partial \theta} = \sum_{a}d^\pi(s) \sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a)$
得證.

---

有了這個定理我們不難發現,

$\sum_a \frac{\partial \pi(s,a)}{\partial \theta} Q^\pi(s,a) =\vec{ \frac{\partial \pi}{\partial \theta}} \odot \vec{Q}$

這意味着只需要使$||\vec{ \frac{\partial \pi}{\partial \theta}} \odot \vec{Q}||=0$即正交即可滿足達到極值點。

---

用於自動駕駛

我們的實驗環境是Torcs.《The Open Racing Car Simulator》(TORCS)是一款開源3D賽車模擬遊戲.是在Linux操作系統上廣受歡迎的賽車遊戲.有50種車輛和20條賽道,簡單的視覺效果.用C和C++寫成,釋放在GPL協議下.

整體架構是,由策略網絡負責給出具體的駕駛操作,由Q值網絡負責給出對駕駛操作達到的效果做預期評估.(這和人類的智能行爲類似,是標準的Deep Reinforcement learning套路:行動能力和思考能力兼備).