Reinforcement Learning Exercise 5.13

最新推荐文章于 2026-04-18 09:20:44 发布

原创最新推荐文章于 2026-04-18 09:20:44 发布 · 394 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#reinforcement learning

收录于

reinforcement learning 专栏收录该内容

37 篇文章

订阅专栏

本文详细解析了强化学习中从公式(5.12)到(5.14)的推导过程，展示了如何利用马尔科夫决策过程的独立性属性，将复杂策略评估表达式简化为更简单的形式。

Exercise 5.13 Show the steps to derive (5.14) from (5.12).
$\rho_{t:T-1}R_{t+1} = \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}\frac{\pi(A_{t+1} \mid S_{t + 1})}{b(A_{t+1} \mid S_{t + 1})}\frac{\pi(A_{t+2} \mid S_{t + 2})}{b(A_{t+2} \mid S_{t + 2})}\cdots\frac{\pi(A_{T-1} \mid S_{T - 1})}{b(A_{T-1} \mid S_{T - 1})}R_{t+1} \qquad{(5.12)}$
According to the property of Markov Procedure, each action step is independent, so we have
$\mathbb E(\rho_{t:T-1}R_{t+1})=\mathbb E( \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}R_{t+1})\mathbb E(\frac{\pi(A_{t+1} \mid S_{t + 1})}{b(A_{t+1} \mid S_{t + 1})})\mathbb E(\frac{\pi(A_{t+2} \mid S_{t + 2})}{b(A_{t+2} \mid S_{t + 2})})\cdots\mathbb E(\frac{\pi(A_{T-1} \mid S_{T - 1})}{b(A_{T-1} \mid S_{T - 1})})$
And according to equation (5.13), the expectation of each step is 1, except the first.
$\mathbb E\Biggl[ \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)}\Biggr] \doteq \sum_a b(a\mid S_k) \frac{\pi(a \mid S_k)}{b(a \mid S_k)} = \sum_a \pi(a \mid S_k) = 1 \qquad{(5.13)}$
So,
$\begin{aligned} \mathbb E(\rho_{t:T-1}R_{t+1})&=\mathbb E( \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}R_{t+1})\\ &=\mathbb E(\rho_{t:t}R_{t+1}) \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad{(5.14)} \end{aligned}$