标准版的策略梯度算法(Vanilla Policy Gradient)

喜欢ヅ旅行 2022-11-30 04:11 150阅读 0赞

### 前言 ###

来源于： [标准版的策略梯度算法(Vanilla Policy Gradient)][Vanilla Policy Gradient]

--------------------

## 1 背景知识 ##

策略梯度(Policy Gradient, PG)方法的核心思想在于是能获得更好的回报的动作的采样概率不断提高，使获得更少回报的动作的采样概率不断降低，从而达到一个最优的策略。

## 2 知识速览 ##

*  标准的策略梯度算法(Vanilla Policy Gradient, VPG)属于在策略(on-policy)算法
 *  VPG算法可以被用到离散和连续动作空间中
 *  Spinning Up中的实现支持使用MPI的并行执行

## 3 关键表达式 ##

记 ![\[公式\]][equation_tex_5Cpi_5Ctheta]为以 ![\[公式\]][equation_tex_5Ctheta] 为参数的策略函数， ![\[公式\]][equation_tex_J_28_5Ctheta_29] 表示该策略在有限时间步长下不打折扣的期望回报值。 ![\[公式\]][equation_tex_J_28_5Ctheta_29] 的梯度值如下所示：

![\[公式\]][equation_tex_5Cnabla_7B_5Ctheta_7D_J_5Cleft_28_5Cpi_7B_5Ctheta_7D_5Cright_29_3D_5Cunderset_7B_5Ctau_5Csim_5Cpi_7B_5Ctheta_7D_7D_7B_5Cmathrm_7BE_7D_7D_5Cleft_5B_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cnabla_7B_5Ctheta_7D_5Clog_5Cpi_7B_5Ctheta_7D_5Cleft_28a_7Bt_7D_7C_s_7Bt_7D_5Cright_29_A_5E_7B_5Cpi_7B_5Ctheta_7D_7D_5Cleft_28s_7Bt_7D_2C_a_7Bt_7D_5Cright_29_5Cright_5D_5C_5C]

其中 ![\[公式\]][equation_tex_5Ctau] 表示依据策略 ![\[公式\]][equation_tex_5Cpi_7B_5Ctheta_7D] 所得到的轨迹， ![\[公式\]][equation_tex_A_5E_7B_5Cpi_5Ctheta_7D] 表示当前策略的优势函数。

策略梯段算法是通过对策略的性能进行梯度上升从而更新策略的参数来更新整个策略的：

![\[公式\]][equation_tex_5Ctheta_7Bk_2B1_7D_3D_5Ctheta_7Bk_7D_2B_5Calpha_5Cnabla_7B_5Ctheta_7D_J_5Cleft_28_5Cpi_7B_5Ctheta_7Bk_7D_7D_5Cright_29_5C_5C]

通常的策略梯度算法的实现过程中计算的优势函数估计是基于无限时间步长的折扣奖励值的，尽管许多场景的公式描述是有限时间步长的非折扣奖励。

## 4 探索和利用 ##

VPG利用在策略的方法来训练随机策略，这意味着其通过在最近版本的策略函数中采样动作来探索。采样动作的随机性来源于设置的初始条件和训练的过程。在不断的训练过程中，由于更新规则使得策略更倾向于选择具有更高回报值的动作，因此策略的随机性将不断降低。然而这可能导致策略最终陷入一个局部最优的策略。

## 5 伪代码 ##

**算法 1** 标准的策略梯度算法

1.  输入：初始的策略函数的参数 ![\[公式\]][equation_tex_5Ctheta_0] ,初始的值函数的参数 ![\[公式\]][equation_tex_5Cphi_0]
2.  **for** k = 0, 1, 2, ... **do**
3.  通过在环境中执行策略 ![\[公式\]][equation_tex_5Cpi_7Bk_7D_3D_5Cpi_5Cleft_28_5Ctheta_7Bk_7D_5Cright_29] 来搜集到轨迹的集合 ![\[公式\]][equation_tex_5Cmathcal_7BD_7D_7Bk_7D_3D_5Cleft_5C_7B_5Ctau_7Bi_7D_5Cright_5C_7D] .
4.  计算后续折扣奖励值 ![\[公式\]][equation_tex_5Chat_7BR_7D_7Bt_7D] .
5.  基于当前的值函数 ![\[公式\]][equation_tex_V_7B_5Cphi_7Bk_7D_7D] 计算优势函数的估计值 ![\[公式\]][equation_tex_5Chat_7BA_7D_7Bt_7D] (可以利用任何优势函数值估计方法)
6.  利用如下的表达式估计策略梯度,  
    ![\[公式\]][equation_tex_5Chat_7Bg_7D_7Bk_7D_3D_5Cfrac_7B1_7D_7B_5Cleft_7C_5Cmathcal_7BD_7D_7Bk_7D_5Cright_7C_7D_5Csum_7B_5Ctau_5Cin_5Cmathcal_7BD_7D_7Bk_7D_7D_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cnabla_7B_5Ctheta_7D_5Clog_5Cpi_7B_5Ctheta_7D_5Cleft_28a_7Bt_7D_7C_s_7Bt_7D_5Cright_29_7C_7B_5Ctheta_7Bk_7D_7D_5Chat_7BA_7D_7Bt_7D._5C_5C]
7.  计算策略的更新，可以使用标准的梯度上升算法， ![\[公式\]][equation_tex_5Ctheta_7Bk_2B1_7D_3D_5Ctheta_7Bk_7D_2B_5Calpha_7Bk_7D_5Chat_7Bg_7D_7Bk_7D] ，也可以使用  
    其他的梯度上升算法比如Adam
8.  通过回归下面的均方损失来拟合值函数：  
    ![\[公式\]][equation_tex_5Cphi_7Bk_2B1_7D_3D_5Carg_5Cmin_7B_5Cphi_7D_5Cfrac_7B1_7D_7B_5Cleft_7C_5Cmathcal_7BD_7D_7Bk_7D_5Cright_7C_T_7D_5Csum_7B_5Ctau_5Cin_5Cmathcal_7BD_7D_7Bk_7D_7D_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cleft_28V_7B_5Cphi_7D_5Cleft_28s_7Bt_7D_5Cright_29-_5Chat_7BR_7D_7Bt_7D_5Cright_29_5E_7B2_7D_5C_5C]  
    这一过程通常利用梯度下降算法。
9.  **end for**

## **6. PyTorch实现(Spinningup中的vpgpg代码解读)** ##

这里对Spinnningup中的vpg源代码进行分析，暂时忽略打印日志模块和程序的多线程执行，主要分析程序的整个运行结构。

### 1 实现中用到的其他背景知识 ###

我们定义当前时刻t的reward\_to\_go如下所示:

![\[公式\]][equation_tex_5Chat_7BR_7D_7Bt_7D_5Cdoteq_5Csum_7Bt_5E_7B_5Cprime_7D_3Dt_7D_5E_7BT_7D_5Cgamma_5E_7Bt_5E_5Cprime-t_7Dr_5Cleft_28s_7Bt_5E_7B_5Cprime_7D_7D_2C_a_7Bt_5E_7B_5Cprime_7D_7D_2C_s_7Bt_5E_7B_5Cprime_7D_2B1_7D_5Cright_29_5C_5C]

在GAE中，定义 ![\[公式\]][equation_tex_5Cdelta_7Bt_7D_5E_7BV_7D_3Dr_7Bt_7D_2B_5Cgamma_V_5Cleft_28s_7Bt_2B1_7D_5Cright_29-V_5Cleft_28s_7Bt_7D_5Cright_29] 。于是 ![\[公式\]][equation_tex_5Cdelta_7Bt_7D_5E_7BV_7D] 可以看作是对当前动作 ![\[公式\]][equation_tex_a_t] 的优势的估计。我们定义如下的多步的优势函数估计形式，

![\[公式\]][equation_tex_5Cbegin_7Baligned_7D_26_5Chat_7BA_7D_7Bt_7D_5E_7B_281_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_V_5Cleft_28s_7Bt_2B1_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_282_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Cgamma_5E_7B2_7D_V_5Cleft_28s_7Bt_2B2_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_283_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Cgamma_5E_7B2_7D_r_7Bt_2B2_7D_2B_5Cgamma_5E_7B3_7D_V_5Cleft_28s_7Bt_2B3_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_28k_29_7D_3A_3D_5Csum_7Bl_3D0_7D_5E_7Bk-1_7D_5Cgamma_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Ccdots_2B_5Cgamma_5E_7Bk-1_7D_r_7Bt_2Bk-1_7D_2B_5Cgamma_5E_7Bk_7D_V_5Cleft_28s_7Bt_2Bk_7D_5Cright_29_5Cend_7Baligned_7D_5C_5C]

当 ![\[公式\]][equation_tex_k_5Crightarrow_5Cinfty] 时，有 ![\[公式\]][equation_tex_5Chat_7BA_7D_7Bt_7D_5E_7B_28_5Cinfty_29_7D_3D_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_5Cgamma_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2B_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_5Cgamma_5E_7Bl_7D_r_7Bt_2Bl_7D] 。GAE( ![\[公式\]][equation_tex_5Clambda_2C_5Cgamma])的定义可以看作是对上述优势函数的指数加权平均：

![\[公式\]][equation_tex_5Cbegin_7Baligned_7D_5Chat_7BA_7D_7Bt_7D_5E_7B_5Cmathrm_7BGAE_7D_28_5Cgamma_2C_5Clambda_29_7D_26_3A_3D_281-_5Clambda_29_5Cleft_28_5Chat_7BA_7D_7Bt_7D_5E_7B_281_29_7D_2B_5Clambda_5Chat_7BA_7D_7Bt_7D_5E_7B_282_29_7D_2B_5Clambda_5E_7B2_7D_5Chat_7BA_7D_7Bt_7D_5E_7B_283_29_7D_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Clambda_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cright_29_2B_5Clambda_5E_7B2_7D_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Clambda_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cright_29_2B_5Clambda_5E_7B2_7D_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_5Cleft_281_2B_5Clambda_2B_5Clambda_5E_7B2_7D_2B_5Cldots_5Cright_29_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cleft_28_5Clambda_2B_5Clambda_5E_7B2_7D_2B_5Clambda_5E_7B2_7D_2B_5Cldots_5Cright_29_5Cright._5C_5C_26_5Cleft._5Cquad_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cleft_28_5Clambda_5E_7B2_7D_2B_5Clambda_5E_7B3_7D_2B_5Clambda_5E_7B4_7D_2B_5Cldots_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B1_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B_5Clambda_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B_5Clambda_5E_7B2_7D_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_28_5Cgamma_5Clambda_29_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_5Cend_7Baligned_7D_5C_5C]

### **2 命令行参数传递** ###

实现中的命令行参数可以通过argparse模块获取，具体的方法如下。

import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--env', type=str, default='HalfCheetah-v2') # 环境名称
    parser.add_argument('--hid', type=int, default=64) # 神经网络隐藏层的维度，默认为64
    parser.add_argument('--l', type=int, default=2) # 神经网络隐藏层的层数，默认为2个隐藏层
    parser.add_argument('--gamma', type=float, default=0.99) # 奖励折扣因子
    parser.add_argument('--seed', '-s', type=int, default=0) # 随机数发生器种子
    parser.add_argument('--cpu', type=int, default=1) # cpu数量，原始默认为4，这里为了方便分析改为1
    parser.add_argument('--steps', type=int, default=4000) # 每个Epoch中保存的"状态-动作对"的数量
    parser.add_argument('--epochs', type=int, default=50) # Epoch的个数
    parser.add_argument('--exp_name', type=str, default='vpg') # 当前的实验名称
    args = parser.parse_args()

获取到参数后，便将所有参数传入到vpg函数中，传递的方式如下：

vpg(lambda : gym.make(args.env), actor_critic=core.MLPActorCritic,
           ac_kwargs=dict(hidden_sizes=[args.hid]*args.l), gamma=args.gamma, 
           seed=args.seed, steps_per_epoch=args.steps, epochs=args.epochs,
           logger_kwargs=logger_kwargs)

这里的lambda : gym.make(args.env)定义了一个匿名函数，其返回一个标准的gym环境，actor\_critic模块采用的是core文件中的MLPActorCritic模块。

### 3 存储数据的VPGBuffer类 ###

在进行下一步的分析之前，我们首先看一下存储数据的VPGBuffer这一模块，这一模块不仅存储智能体与环境直接进行交互所产生的数据，还用GAE-Lambda方法来计算状态动作对的advantage值。

(1) \_*init*\_函数

其中的 \_*init*\_函数定义如下所示：

def __init__(self, obs_dim, act_dim, size, gamma=0.99, lam=0.95):
        self.obs_buf = np.zeros(core.combined_shape(size, obs_dim), dtype=np.float32)
        self.act_buf = np.zeros(core.combined_shape(size, act_dim), dtype=np.float32)
        self.adv_buf = np.zeros(size, dtype=np.float32)
        self.rew_buf = np.zeros(size, dtype=np.float32)
        self.ret_buf = np.zeros(size, dtype=np.float32)
        self.val_buf = np.zeros(size, dtype=np.float32)
        self.logp_buf = np.zeros(size, dtype=np.float32)
        self.gamma, self.lam = gamma, lam
        self.ptr, self.path_start_idx, self.max_size = 0, 0, size

这一函数中，obs\_dim表示观测空间的维数，act\_dim表示动作空间的维数，size表示该缓存区的容量。gamma表示奖励折扣因子，lam表示在GAE-Lambda中的系数。obs\_buf中存储的为观测值，act*buf中存储的为动作值，*adv*buf中存储的为当前动作a的advantage值，rew\_buf中存储的为当前动作的即时reward值,ret\_buf中存储的是reward to go，即当前状态下的折扣奖励值，val\_buf存储的是当前状态的值V(s\_t)。*

注意到用到了core.combined\_shape()这个函数，其定义如下所示：

def combined_shape(length, shape=None):
        if shape is None:
            return (length,)
        return (length, shape) if np.isscalar(shape) else (length, *shape)

这一函数的作用是针对不同的传入的shape参数，返回不同的元组。如果传入的shape=None那么直接返回(length, )，如果传入的shape为一个元组列表等，则返回对应的元组(比如传入的length=10, shape=\[3,4,5\],那么这一函数的返回值为(10, 3, 4, 5))。于是这一函数可以用来帮助在VPGBuffer中开辟空间(比如当观测是一个向量时，传入的shape可以为观测项链的长度,当观测是一个图片时，传入的shape包含观测图像的通道，长，宽等信息)。

(2）strore函数

然后就是类中的store函数，其定义如下所示

def store(self, obs, act, rew, val, logp):
 """
 Append one timestep of agent-environment interaction to the buffer.
 """
 assert self.ptr < self.max_size # buffer has to have room so you can store
 self.obs_buf[self.ptr] = obs
 self.act_buf[self.ptr] = act
 self.rew_buf[self.ptr] = rew
 self.val_buf[self.ptr] = val
 self.logp_buf[self.ptr] = logp
 self.ptr += 1

其作用为将一个timestep的数据存到Buffer中。

(3) finish\_path函数

接下来就是finish\_path函数，其定义如下所示：

def finish_path(self, last_val=0):
 path_slice = slice(self.path_start_idx, self.ptr)
 rews = np.append(self.rew_buf[path_slice], last_val)
 vals = np.append(self.val_buf[path_slice], last_val)
 
 # the next two lines implement GAE-Lambda advantage calculation
 deltas = rews[:-1] + self.gamma * vals[1:] - vals[:-1]
 self.adv_buf[path_slice] = core.discount_cumsum(deltas, self.gamma * self.lam)
 
 # the next line computes rewards-to-go, to be targets for the value function
 self.ret_buf[path_slice] = core.discount_cumsum(rews, self.gamma)[:-1]
 
 self.path_start_idx = self.ptr</code></pre></div>这个函数当每个轨迹结束或者在epoch终止的时候被调用。其作用是回到当前轨迹的开始时刻，并利用整个轨迹的奖励值和值函数的估计来计算GAE-Lambda并且计算每个状态的reward_to_go，以作为价值函数的目标。我们首先计算每个时刻的 <img src="https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D%3Dr_%7Bt%7D%2B%5Cgamma+V%5Cleft%28s_%7Bt%2B1%7D%5Cright%29-V%5Cleft%28s_%7Bt%7D%5Cright%29" alt="[公式]" eeimg="1" data-formula="\delta_{t}^{V}=r_{t}+\gamma V\left(s_{t+1}\right)-V\left(s_{t}\right)"> 。注意到self.rewbuf中存储的为所有的单步奖励值，于是首先需要将当前轨迹所对应的奖励值取出来放到rew列表中，将对应的 <img src="https://www.zhihu.com/equation?tex=V%28s_t%29" alt="[公式]" eeimg="1" data-formula="V(s_t)"> 取出来放到val列表中。计算 <img src="https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D" alt="[公式]" eeimg="1" data-formula="\delta_{t}^{V}"> 的代码为<div class="highlight"><pre><code class="language-text">deltas = rews[:-1] + self.gamma * vals[1:] - vals[:-1]</code></pre></div>这行代码能够得到当前轨迹所有时刻的 <img src="https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D" alt="[公式]" eeimg="1" data-formula="\delta_{t}^{V}"> 值列表。然后利用这一deltas列表可以得到GAE-Lambda估计值：<div class="highlight"><pre><code class="language-text">self.adv_buf[path_slice] = core.discount_cumsum(deltas, self.gamma * self.lam)</code></pre></div>这一计算过程依赖于core中的discount_cumsum函数，其定义如下：<div class="highlight"><pre><code class="language-text">def discount_cumsum(x, discount):
 return scipy.signal.lfilter([1], [1, float(-discount)], x[::-1], axis=0)[::-1]</code></pre></div>这一函数若输入为一向量[x0, x1, x2],则其输出为[x0 + discount * x1 + discount^2 * x2, x1 + discount * x2, x2]。对其最简单的验证为向其中传入x=[1,2,3],discount=0.9发现其输出为[5.23,4.7,3]，以证实其起到的作用。其具体分析为：<code>scipy.signal.lfilter</code>(b,a,x,axis=-1,zi=None) 是一个滤波器函数，其进行如下的滤波计算<img src="https://www.zhihu.com/equation?tex=a%5B0%5D%2Ay%5Bn%5D+%3D+b%5B0%5D%2Ax%5Bn%5D+%2B+b%5B1%5D%2Ax%5Bn-1%5D+%2B+...+%2B+b%5BM%5D%2Ax%5Bn-M%5D-+a%5B1%5D%2Ay%5Bn-1%5D+-+...+-+a%5BN%5D%2Ay%5Bn-N%5D" alt="[公式]" eeimg="1" data-formula="a[0]*y[n] = b[0]*x[n] + b[1]*x[n-1] + ... + b[M]*x[n-M]- a[1]*y[n-1] - ... - a[N]*y[n-N]"> 其中M,N分别为1维数组中b,a中的元素数目减一(degree of b and a)，该函数的返回值为y。在这里，我们b=[1], a=[1, -discount],这意味着M=1,N=2则带入上述计算过程，对于传入的一维向量x，其输出结果的代表<img src="https://www.zhihu.com/equation?tex=y%5Bn%5D+%3D+x%5Bn%5D+%2B+discount+%2A+y%5Bn-1%5D%5C%5C" alt="[公式]" eeimg="1" data-formula="y[n] = x[n] + discount * y[n-1]\\"> 即有，<img src="https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+y%5B0%5D+%26%3D+x%5B0%5D%5C%5C+y%5B1%5D+%26%3D+x%5B1%5D+%2B+discount+%2A+y%5B0%5D%5C%5C+y%5B2%5D+%26%3D+x%5B2%5D+%2B+discount+%2Ay%5B1%5D+%3D+x%5B2%5D+%2B+discount+++x%5B1%5D+%2B+discount+%5E2+%2A+x%5B0%5D%5C%5C+%26%5Ccdots%5Ccdots%5C%5C+y%5Bn%5D+%26%3D+x%5Bn%5D+%2B+discount+%2A+y%5Bn-1%5D%5C%5C+++++++++%26%3D+x%5Bn%5D+%2B+discount+%2A+%28x%5Bn-1%5D+%2B+discount+%2A+x%5Bn-2%5D%29%5C%5C+++++++++%26%3D+%5Ccdots%5C%5C+++++++++%26%3D+x%5Bn%5D+%2B+discount%2Ax%5Bn-1%5D+%2B+discount%5E2%2Ax%5Bn-2%5D%EF%BC%8B%5Ccdots%EF%BC%8Bdiscount%5En%2Ax%5B0%5D+%5Cend%7Baligned%7D%5C%5C%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80%E3%80%80++" alt="[公式]" eeimg="1" data-formula="\begin{aligned} y[0] &= x[0]\\ y[1] &= x[1] + discount * y[0]\\ y[2] &= x[2] + discount *y[1] = x[2] + discount x[1] + discount ^2 * x[0]\\ &\cdots\cdots\\ y[n] &= x[n] + discount * y[n-1]\\ &= x[n] + discount * (x[n-1] + discount * x[n-2])\\ &= \cdots\\ &= x[n] + discount*x[n-1] + discount^2*x[n-2]＋\cdots＋discount^n*x[0] \end{aligned}\\　　　　　　　　　　　　　　　　 "> 注意到传入这一函数的 <img src="https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D%3D%5BV%28s_0%29%2C+V%28s_1%29%2C+V%28s_2%29%2C+%5Ccdots%2C+V%28s_T%29%5D" alt="[公式]" eeimg="1" data-formula="\delta_{t}^{V}=[V(s_0), V(s_1), V(s_2), \cdots, V(s_T)]">被逆置,那么传入的后的函数返回值<img src="https://www.zhihu.com/equation?tex=adv%5Bn%5D+%3D+V%28s_n%29+%2B+discount++%2AV%5Bs%7Bn%2B1%7D%5D+%2B+discount%5E2+%2AV%5Bs%7Bn-2%7D%5D+%2B+%5Ccdots+%2Bdiscount%5E%7BT-n%7D+%2A+V%5Bs_T%5D%5C%5C" alt="[公式]" eeimg="1" data-formula="adv[n] = V(s_n) + discount *V[s{n+1}] + discount^2 *V[s{n-2}] + \cdots +discount^{T-n} * V[s_T]\\"> 这样得到的输出为 <img src="https://www.zhihu.com/equation?tex=%5Badv%5BT%5D%2C+adv%5BT-1%5D%2C+%5Ccdots%2C+adv%5B0%5D%5D" alt="[公式]" eeimg="1" data-formula="[adv[T], adv[T-1], \cdots, adv[0]]">然后再对其进行逆序操作即可得到GAE-Lambda的计算值。利用这一函数同样可以计算reward_to_go的值。(4) get函数这一函数在每个epoch的结束阶段调用，用来获取buffer中的所有数据并重置buffer中的相关指针。返回一个包含当前Buffer中所有数据的字典用于后续的执行。<div class="highlight"><pre><code class="language-python3"> def get(self):
 assert self.ptr == self.max_size # buffer has to be full before you can get
 self.ptr, self.path_start_idx = 0, 0
 # the next two lines implement the advantage normalization trick
 adv_mean, adv_std = mpi_statistics_scalar(self.adv_buf)
 self.adv_buf = (self.adv_buf - adv_mean) / adv_std
 data = dict(obs=self.obs_buf, act=self.act_buf, ret=self.ret_buf,
 adv=self.adv_buf, logp=self.logp_buf)
 return {k: torch.as_tensor(v, dtype=torch.float32) for k,v in data.items()}</code></pre></div>这里需要注意的是其将advantange值归一化到了均值为0，方差为1的正态分布。<h3>4 MLPActorCritic模块</h3>(1) MLPActorCritic类的总体架构在__init__方法中，首先判断动作空间是连续动作空间还是离散动作空间，如果是连续动作空间，那么策略为高斯策略，否则为CategoricalActor策略，然后构造Critic。<div class="highlight"><pre><code class="language-python3">class MLPActorCritic(nn.Module):
 def __init__(self, observation_space, action_space, 
 hidden_sizes=(64,64), activation=nn.Tanh):
 super().__init__()
 
 obs_dim = observation_space.shape[0]
 
 # policy builder depends on action space
 if isinstance(action_space, Box):
 self.pi = MLPGaussianActor(obs_dim, action_space.shape[0], hidden_sizes, activation)
 elif isinstance(action_space, Discrete):
 self.pi = MLPCategoricalActor(obs_dim, action_space.n, hidden_sizes, activation)
 
 # build value function
 self.v = MLPCritic(obs_dim, hidden_sizes, activation)
 
 def step(self, obs):
 with torch.no_grad():
 pi = self.pi._distribution(obs)
 a = pi.sample()
 logp_a = self.pi._log_prob_from_distribution(pi, a)
 v = self.v(obs)
 return a.numpy(), v.numpy(), logp_a.numpy()
 
 def act(self, obs):
 return self.step(obs)[0]</code></pre></div>(2) Actor类Actor类由一个名为Actor的父类和其两个子类MLPCategoricalActor与MLPGaussianActor构成,父类的实现如下所示：<div class="highlight"><pre><code class="language-text">class Actor(nn.Module):
 
 def _distribution(self, obs):
 raise NotImplementedError
 
 def _log_prob_from_distribution(self, pi, act):
 raise NotImplementedError
 
 def forward(self, obs, act=None):
 # Produce action distributions for given observations, and 
 # optionally compute the log likelihood of given actions under
 # those distributions.
 pi = self._distribution(obs)
 logp_a = None
 if act is not None:
 logp_a = self._log_prob_from_distribution(pi, act)
 return pi, logp_a</code></pre></div>子类中必须根据动作是离散还是连续实现私有方法_distrbution和_logprob_from_distribution用来计算策略的分布和动作的对数似然值。MLPCategoricalActor()类继承了上述Actor类用来处理离散的动作空间，其定义如下所示：<div class="highlight"><pre><code class="language-text">class MLPCategoricalActor(Actor):
 
 def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
 super().__init__()
 self.logits_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)
 
 def _distribution(self, obs):
 logits = self.logits_net(obs)
 return Categorical(logits=logits)
 
 def _log_prob_from_distribution(self, pi, act):
 return pi.log_prob(act)</code></pre></div>该类的初始化方法中定义了执行策略的神经网络logits_net，首先转到神经网络的定义如下：<div class="highlight"><pre><code class="language-text">def mlp(sizes, activation, output_activation=nn.Identity):
 layers = []
 for j in range(len(sizes)-1):
 act = activation if j < len(sizes)-2 else output_activation
 layers += [nn.Linear(sizes[j], sizes[j+1]), act()]
 return nn.Sequential(*layers)</code></pre></div>可以看出这个函数定义了非常简单的全连接神经网络，注意默认情形下最后输出没有经过激活函数，仅仅是logits。再回到初始化方法中<div class="highlight"><pre><code class="language-text">self.logits_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)</code></pre></div>这一句实际上是初始化了一个全连接的神经网络，其输入维度为obsdim,隐藏层维度由hidden_sizes决定，输出的维度为动作空间的维数，activation为激活函数，没有指定输出层的激活函数。MLPCategoricalActor类中的distribution方法将观测值传入策略网络并且将输出的logit值转为概率分布，_log_prob_from_distribution方法则是输出当前动作的对数似然值。注意此时的pi为一个Categorical类，该类中已经封装好了log_prob方法，所以才会出现return pi.log_prob(act)这条语句。MLPGaussianActor()类中的函数基本与上述保持一致，其实现如下所示：<div class="highlight"><pre><code class="language-text">class MLPGaussianActor(Actor):
 
 def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
 super().__init__()
 log_std = -0.5 * np.ones(act_dim, dtype=np.float32)
 self.log_std = torch.nn.Parameter(torch.as_tensor(log_std))
 self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)
 
 def _distribution(self, obs):
 mu = self.mu_net(obs)
 std = torch.exp(self.log_std)
 return Normal(mu, std)
 
 def _log_prob_from_distribution(self, pi, act):
 return pi.log_prob(act).sum(axis=-1) </code></pre></div>这里的分布的方差是人为指定的并不是学习得到的。(3) Critic类Critic类构造的事一个当前状态的状态值函数，定义如下：<div class="highlight"><pre><code class="language-text">class MLPCritic(nn.Module):
 
 def __init__(self, obs_dim, hidden_sizes, activation):
 super().__init__()
 self.v_net = mlp([obs_dim] + list(hidden_sizes) + [1], activation)
 
 def forward(self, obs):
 return torch.squeeze(self.v_net(obs), -1)</code></pre></div><h2>5 计算策略的损失和Critic的损失</h2><h3>1 策略的损失</h3>这里其实需要计算的是上述伪代码中第6行中的公式，即<img src="https://www.zhihu.com/equation?tex=%5Chat%7Bg%7D_%7Bk%7D%3D%5Cfrac%7B1%7D%7B%5Cleft%7C%5Cmathcal%7BD%7D_%7Bk%7D%5Cright%7C%7D+%5Csum_%7B%5Ctau+%5Cin+%5Cmathcal%7BD%7D_%7Bk%7D%7D+%5Csum_%7Bt%3D0%7D%5E%7BT%7D+%5Cnabla_%7B%5Ctheta%7D+%5Clog+%5Cpi_%7B%5Ctheta%7D%5Cleft%28a_%7Bt%7D+%7C+s_%7Bt%7D%5Cright%29%7C_%7B%5Ctheta_%7Bk%7D%7D+%5Chat%7BA%7D_%7Bt%7D.%5C%5C" alt="[公式]" eeimg="1" data-formula="\hat{g}_{k}=\frac{1}{\left|\mathcal{D}_{k}\right|} \sum_{\tau \in \mathcal{D}_{k}} \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} | s_{t}\right)|_{\theta_{k}} \hat{A}_{t}.\\"> 计算代码如下所示：<div class="highlight"><pre><code class="language-text"> def compute_loss_pi(data):
 obs, act, adv, logp_old = data['obs'], data['act'], data['adv'], data['logp']
 
 # Policy loss
 pi, logp = ac.pi(obs, act)
 loss_pi = -(logp * adv).mean()
 
 # Useful extra info
 approx_kl = (logp_old - logp).mean().item()
 ent = pi.entropy().mean().item()
 pi_info = dict(kl=approx_kl, ent=ent)
 
 return loss_pi, pi_info</code></pre></div>可以看出这一函数不仅计算量策略的损失，还能够输出其他有关当前策略的有用信息。值得注意的是由于采用的是梯度下降算法，因而loss_pi需要在前面加个负号。<h3>2 评论家的损失</h3>这里对应着伪代码中的第8行，即要回归如下的表达式：<img src="https://www.zhihu.com/equation?tex=%5Cphi_%7Bk%2B1%7D%3D%5Carg+%5Cmin+_%7B%5Cphi%7D+%5Cfrac%7B1%7D%7B%5Cleft%7C%5Cmathcal%7BD%7D_%7Bk%7D%5Cright%7C+T%7D+%5Csum_%7B%5Ctau+%5Cin+%5Cmathcal%7BD%7D_%7Bk%7D%7D+%5Csum_%7Bt%3D0%7D%5E%7BT%7D%5Cleft%28V_%7B%5Cphi%7D%5Cleft%28s_%7Bt%7D%5Cright%29-%5Chat%7BR%7D_%7Bt%7D%5Cright%29%5E%7B2%7D%5C%5C" alt="[公式]" eeimg="1" data-formula="\phi_{k+1}=\arg \min _{\phi} \frac{1}{\left|\mathcal{D}_{k}\right| T} \sum_{\tau \in \mathcal{D}_{k}} \sum_{t=0}^{T}\left(V_{\phi}\left(s_{t}\right)-\hat{R}_{t}\right)^{2}\\"> 其中 <img src="/images/20221124/3abc544e133d4c7bb5ebba935561af1e.png+%5Cdoteq+%5Csum_%7Bt%5E%7B%5Cprime%7D%3Dt%7D%5E%7BT%7D%5Cgamma+%5E%7Bt%5E%5Cprime-t%7Dr%5Cleft%28s_%7Bt%5E%7B%5Cprime%7D%7D%2C+a_%7Bt%5E%7B%5Cprime%7D%7D%2C+s_%7Bt%5E%7B%5Cprime%7D%2B1%7D%5Cright%29%5C%5C" alt="[公式]" eeimg="1" data-formula="\hat{R}_{t} \doteq \sum_{t^{\prime}=t}^{T}\gamma ^{t^\prime-t}r\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)\\"> 为当前状态的reward-to-go。<h3>3 优化过程</h3>首先设置两个网络的优化器：<div class="highlight"><pre><code class="language-text"> pi_optimizer = Adam(ac.pi.parameters(), lr=pi_lr)

vf\_optimizer = Adam(ac.v.parameters(), lr=vf\_lr)

接下来就是update()函数，在该函数中，首先通过

data = buf.get()

获取Buffer中的数据，然后在进行更新之前得到损失值和相关的一些信息

pi_l_old, pi_info_old = compute_loss_pi(data)
    
    pi_l_old = pi_l_old.item()
    
    v_l_old = compute_loss_v(data).item()

然后利用单步的梯度下降来训练策略网络：

pi_optimizer.zero_grad()
    
    loss_pi, pi_info = compute_loss_pi(data)
    
    loss_pi.backward()
    
    pi_optimizer.step()

训练值函数(Critic)的网络如下所示：

for i in range(train_v_iters):
    
    vf_optimizer.zero_grad()
    
    loss_v = compute_loss_v(data)
    
    loss_v.backward()
    
    vf_optimizer.step()

其中的train\_*v\_*iters是指在一个epoch中对值函数进行梯度下降的次数。

## 6 训练过程主循环 ##

首先初始化环境：

start_time = time.time()
    
    o, ep_ret, ep_len = env.reset(), 0, 0

接下来进入主循环，搜集数据并且在每个epoch对网络的参数进行更新。首先将观测值传入到策略网络和Critic网络，得到所应该采取的动作，状态值，以及对数似然，函数调用如下：

a, v, logp = ac.step(torch.as_tensor(o, dtype=torch.float32))

然后将输出的动作值环境中进行执行,并将得到的结果保存到Buffer中，并更新当前的时刻所处的状态

next_o, r, d, _ = env.step(a)
    
    ep_ret += r
    
    ep_len += 1
    
    buf.store(o,a,r,v,logp)
    
    o = next_o

接下来进行超时，当前轨迹终止以及epoch结束的条件判断及处理过程：

timeout = ep_len == max_ep_len
                terminal = d or timeout
                epoch_ended = t==local_steps_per_epoch-1
    
                if terminal or epoch_ended:
                    if epoch_ended and not(terminal):
                        print('Warning: trajectory cut off by epoch at %d steps.'%ep_len, flush=True)
                    # if trajectory didn't reach terminal state, bootstrap value target
                    if timeout or epoch_ended:
                        _, v, _ = ac.step(torch.as_tensor(o, dtype=torch.float32))
                    else:
                        v = 0
                    buf.finish_path(v)
                    if terminal:
                        # only save EpRet / EpLen if trajectory finished
                        logger.store(EpRet=ep_ret, EpLen=ep_len)
                    o, ep_ret, ep_len = env.reset(), 0, 0

最后保存模型并且进行参数的更新

# Save model
    if (epoch % save_freq == 0) or (epoch == epochs-1):
        logger.save_state({
        'env': env}, None)
    
    # Perform VPG update!
    update()

\[1\] [https://spinningup.openai.com/en/latest/algorithms/vpg.html][https_spinningup.openai.com_en_latest_algorithms_vpg.html]

\[2\] [High Dimensional Continuous Control Using Generalized Advantage Estimation][], Schulman et al. 2016

[Vanilla Policy Gradient]: https://zhuanlan.zhihu.com/p/106006748
[equation_tex_5Cpi_5Ctheta]: https://www.zhihu.com/equation?tex=%5Cpi_%5Ctheta
[equation_tex_5Ctheta]: https://www.zhihu.com/equation?tex=%5Ctheta
[equation_tex_J_28_5Ctheta_29]: /images/20221124/30a3fb4ed2af44719300abf22713c054.png
[equation_tex_5Cnabla_7B_5Ctheta_7D_J_5Cleft_28_5Cpi_7B_5Ctheta_7D_5Cright_29_3D_5Cunderset_7B_5Ctau_5Csim_5Cpi_7B_5Ctheta_7D_7D_7B_5Cmathrm_7BE_7D_7D_5Cleft_5B_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cnabla_7B_5Ctheta_7D_5Clog_5Cpi_7B_5Ctheta_7D_5Cleft_28a_7Bt_7D_7C_s_7Bt_7D_5Cright_29_A_5E_7B_5Cpi_7B_5Ctheta_7D_7D_5Cleft_28s_7Bt_7D_2C_a_7Bt_7D_5Cright_29_5Cright_5D_5C_5C]: https://www.zhihu.com/equation?tex=%5Cnabla_%7B%5Ctheta%7D+J%5Cleft%28%5Cpi_%7B%5Ctheta%7D%5Cright%29%3D%5Cunderset%7B%5Ctau+%5Csim+%5Cpi_%7B%5Ctheta%7D%7D%7B%5Cmathrm%7BE%7D%7D%5Cleft%5B%5Csum_%7Bt%3D0%7D%5E%7BT%7D+%5Cnabla_%7B%5Ctheta%7D+%5Clog+%5Cpi_%7B%5Ctheta%7D%5Cleft%28a_%7Bt%7D+%7C+s_%7Bt%7D%5Cright%29+A%5E%7B%5Cpi_%7B%5Ctheta%7D%7D%5Cleft%28s_%7Bt%7D%2C+a_%7Bt%7D%5Cright%29%5Cright%5D%5C%5C
[equation_tex_5Ctau]: https://www.zhihu.com/equation?tex=%5Ctau
[equation_tex_5Cpi_7B_5Ctheta_7D]: https://www.zhihu.com/equation?tex=%5Cpi_%7B%5Ctheta%7D
[equation_tex_A_5E_7B_5Cpi_5Ctheta_7D]: /images/20221124/88fec0c004454817ad01088d0c2020cc.png
[equation_tex_5Ctheta_7Bk_2B1_7D_3D_5Ctheta_7Bk_7D_2B_5Calpha_5Cnabla_7B_5Ctheta_7D_J_5Cleft_28_5Cpi_7B_5Ctheta_7Bk_7D_7D_5Cright_29_5C_5C]: https://www.zhihu.com/equation?tex=%5Ctheta_%7Bk%2B1%7D%3D%5Ctheta_%7Bk%7D%2B%5Calpha+%5Cnabla_%7B%5Ctheta%7D+J%5Cleft%28%5Cpi_%7B%5Ctheta_%7Bk%7D%7D%5Cright%29%5C%5C
[equation_tex_5Ctheta_0]: https://www.zhihu.com/equation?tex=%5Ctheta_0
[equation_tex_5Cphi_0]: /images/20221124/4cbb41771a464aa49cb10228d4f34203.png
[equation_tex_5Cpi_7Bk_7D_3D_5Cpi_5Cleft_28_5Ctheta_7Bk_7D_5Cright_29]: https://www.zhihu.com/equation?tex=%5Cpi_%7Bk%7D%3D%5Cpi%5Cleft%28%5Ctheta_%7Bk%7D%5Cright%29
[equation_tex_5Cmathcal_7BD_7D_7Bk_7D_3D_5Cleft_5C_7B_5Ctau_7Bi_7D_5Cright_5C_7D]: /images/20221124/a762794193624d8dafeb70b7a6f25bcb.png
[equation_tex_5Chat_7BR_7D_7Bt_7D]: /images/20221124/3abc544e133d4c7bb5ebba935561af1e.png
[equation_tex_V_7B_5Cphi_7Bk_7D_7D]: https://www.zhihu.com/equation?tex=V_%7B%5Cphi_%7Bk%7D%7D
[equation_tex_5Chat_7BA_7D_7Bt_7D]: /images/20221124/44cc6097939c4a8893247f6f5cc59535.png
[equation_tex_5Chat_7Bg_7D_7Bk_7D_3D_5Cfrac_7B1_7D_7B_5Cleft_7C_5Cmathcal_7BD_7D_7Bk_7D_5Cright_7C_7D_5Csum_7B_5Ctau_5Cin_5Cmathcal_7BD_7D_7Bk_7D_7D_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cnabla_7B_5Ctheta_7D_5Clog_5Cpi_7B_5Ctheta_7D_5Cleft_28a_7Bt_7D_7C_s_7Bt_7D_5Cright_29_7C_7B_5Ctheta_7Bk_7D_7D_5Chat_7BA_7D_7Bt_7D._5C_5C]: https://www.zhihu.com/equation?tex=%5Chat%7Bg%7D_%7Bk%7D%3D%5Cfrac%7B1%7D%7B%5Cleft%7C%5Cmathcal%7BD%7D_%7Bk%7D%5Cright%7C%7D+%5Csum_%7B%5Ctau+%5Cin+%5Cmathcal%7BD%7D_%7Bk%7D%7D+%5Csum_%7Bt%3D0%7D%5E%7BT%7D+%5Cnabla_%7B%5Ctheta%7D+%5Clog+%5Cpi_%7B%5Ctheta%7D%5Cleft%28a_%7Bt%7D+%7C+s_%7Bt%7D%5Cright%29%7C_%7B%5Ctheta_%7Bk%7D%7D+%5Chat%7BA%7D_%7Bt%7D.%5C%5C
[equation_tex_5Ctheta_7Bk_2B1_7D_3D_5Ctheta_7Bk_7D_2B_5Calpha_7Bk_7D_5Chat_7Bg_7D_7Bk_7D]: https://www.zhihu.com/equation?tex=%5Ctheta_%7Bk%2B1%7D%3D%5Ctheta_%7Bk%7D%2B%5Calpha_%7Bk%7D+%5Chat%7Bg%7D_%7Bk%7D
[equation_tex_5Cphi_7Bk_2B1_7D_3D_5Carg_5Cmin_7B_5Cphi_7D_5Cfrac_7B1_7D_7B_5Cleft_7C_5Cmathcal_7BD_7D_7Bk_7D_5Cright_7C_T_7D_5Csum_7B_5Ctau_5Cin_5Cmathcal_7BD_7D_7Bk_7D_7D_5Csum_7Bt_3D0_7D_5E_7BT_7D_5Cleft_28V_7B_5Cphi_7D_5Cleft_28s_7Bt_7D_5Cright_29-_5Chat_7BR_7D_7Bt_7D_5Cright_29_5E_7B2_7D_5C_5C]: https://www.zhihu.com/equation?tex=%5Cphi_%7Bk%2B1%7D%3D%5Carg+%5Cmin+_%7B%5Cphi%7D+%5Cfrac%7B1%7D%7B%5Cleft%7C%5Cmathcal%7BD%7D_%7Bk%7D%5Cright%7C+T%7D+%5Csum_%7B%5Ctau+%5Cin+%5Cmathcal%7BD%7D_%7Bk%7D%7D+%5Csum_%7Bt%3D0%7D%5E%7BT%7D%5Cleft%28V_%7B%5Cphi%7D%5Cleft%28s_%7Bt%7D%5Cright%29-%5Chat%7BR%7D_%7Bt%7D%5Cright%29%5E%7B2%7D%5C%5C
[equation_tex_5Chat_7BR_7D_7Bt_7D_5Cdoteq_5Csum_7Bt_5E_7B_5Cprime_7D_3Dt_7D_5E_7BT_7D_5Cgamma_5E_7Bt_5E_5Cprime-t_7Dr_5Cleft_28s_7Bt_5E_7B_5Cprime_7D_7D_2C_a_7Bt_5E_7B_5Cprime_7D_7D_2C_s_7Bt_5E_7B_5Cprime_7D_2B1_7D_5Cright_29_5C_5C]: /images/20221124/3abc544e133d4c7bb5ebba935561af1e.png+%5Cdoteq+%5Csum_%7Bt%5E%7B%5Cprime%7D%3Dt%7D%5E%7BT%7D%5Cgamma+%5E%7Bt%5E%5Cprime-t%7Dr%5Cleft%28s_%7Bt%5E%7B%5Cprime%7D%7D%2C+a_%7Bt%5E%7B%5Cprime%7D%7D%2C+s_%7Bt%5E%7B%5Cprime%7D%2B1%7D%5Cright%29%5C%5C
[equation_tex_5Cdelta_7Bt_7D_5E_7BV_7D_3Dr_7Bt_7D_2B_5Cgamma_V_5Cleft_28s_7Bt_2B1_7D_5Cright_29-V_5Cleft_28s_7Bt_7D_5Cright_29]: https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D%3Dr_%7Bt%7D%2B%5Cgamma+V%5Cleft%28s_%7Bt%2B1%7D%5Cright%29-V%5Cleft%28s_%7Bt%7D%5Cright%29
[equation_tex_5Cdelta_7Bt_7D_5E_7BV_7D]: https://www.zhihu.com/equation?tex=%5Cdelta_%7Bt%7D%5E%7BV%7D
[equation_tex_a_t]: /images/20221124/db0044a6dbe84fdcb2a887d4a6ad8198.png
[equation_tex_5Cbegin_7Baligned_7D_26_5Chat_7BA_7D_7Bt_7D_5E_7B_281_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_V_5Cleft_28s_7Bt_2B1_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_282_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Cgamma_5E_7B2_7D_V_5Cleft_28s_7Bt_2B2_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_283_29_7D_3A_3D_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Cgamma_5E_7B2_7D_r_7Bt_2B2_7D_2B_5Cgamma_5E_7B3_7D_V_5Cleft_28s_7Bt_2B3_7D_5Cright_29_5C_5C_26_5Chat_7BA_7D_7Bt_7D_5E_7B_28k_29_7D_3A_3D_5Csum_7Bl_3D0_7D_5E_7Bk-1_7D_5Cgamma_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2Br_7Bt_7D_2B_5Cgamma_r_7Bt_2B1_7D_2B_5Ccdots_2B_5Cgamma_5E_7Bk-1_7D_r_7Bt_2Bk-1_7D_2B_5Cgamma_5E_7Bk_7D_V_5Cleft_28s_7Bt_2Bk_7D_5Cright_29_5Cend_7Baligned_7D_5C_5C]: https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%26%5Chat%7BA%7D_%7Bt%7D%5E%7B%281%29%7D%3A%3D%5Cdelta_%7Bt%7D%5E%7BV%7D+%3D-V%5Cleft%28s_%7Bt%7D%5Cright%29%2Br_%7Bt%7D%2B%5Cgamma+V%5Cleft%28s_%7Bt%2B1%7D%5Cright%29%5C%5C+%26%5Chat%7BA%7D_%7Bt%7D%5E%7B%282%29%7D%3A%3D%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D+%3D-V%5Cleft%28s_%7Bt%7D%5Cright%29%2Br_%7Bt%7D%2B%5Cgamma+r_%7Bt%2B1%7D%2B%5Cgamma%5E%7B2%7D+V%5Cleft%28s_%7Bt%2B2%7D%5Cright%29%5C%5C+%26%5Chat%7BA%7D_%7Bt%7D%5E%7B%283%29%7D%3A%3D%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%2B%5Cgamma%5E%7B2%7D+%5Cdelta_%7Bt%2B2%7D%5E%7BV%7D%3D-V%5Cleft%28s_%7Bt%7D%5Cright%29%2Br_%7Bt%7D%2B%5Cgamma+r_%7Bt%2B1%7D%2B%5Cgamma%5E%7B2%7D+r_%7Bt%2B2%7D%2B%5Cgamma%5E%7B3%7D+V%5Cleft%28s_%7Bt%2B3%7D%5Cright%29%5C%5C+%26%5Chat%7BA%7D_%7Bt%7D%5E%7B%28k%29%7D%3A%3D%5Csum_%7Bl%3D0%7D%5E%7Bk-1%7D+%5Cgamma%5E%7Bl%7D+%5Cdelta_%7Bt%2Bl%7D%5E%7BV%7D%3D-V%5Cleft%28s_%7Bt%7D%5Cright%29%2Br_%7Bt%7D%2B%5Cgamma+r_%7Bt%2B1%7D%2B%5Ccdots%2B%5Cgamma%5E%7Bk-1%7D+r_%7Bt%2Bk-1%7D%2B%5Cgamma%5E%7Bk%7D+V%5Cleft%28s_%7Bt%2Bk%7D%5Cright%29+%5Cend%7Baligned%7D%5C%5C
[equation_tex_k_5Crightarrow_5Cinfty]: https://www.zhihu.com/equation?tex=k+%5Crightarrow+%5Cinfty
[equation_tex_5Chat_7BA_7D_7Bt_7D_5E_7B_28_5Cinfty_29_7D_3D_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_5Cgamma_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_3D-V_5Cleft_28s_7Bt_7D_5Cright_29_2B_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_5Cgamma_5E_7Bl_7D_r_7Bt_2Bl_7D]: /images/20221124/44cc6097939c4a8893247f6f5cc59535.png%5E%7B%28%5Cinfty%29%7D%3D%5Csum_%7Bl%3D0%7D%5E%7B%5Cinfty%7D+%5Cgamma%5E%7Bl%7D+%5Cdelta_%7Bt%2Bl%7D%5E%7BV%7D%3D-V%5Cleft%28s_%7Bt%7D%5Cright%29%2B%5Csum_%7Bl%3D0%7D%5E%7B%5Cinfty%7D+%5Cgamma%5E%7Bl%7D+r_%7Bt%2Bl%7D
[equation_tex_5Clambda_2C_5Cgamma]: /images/20221124/20466911fb6a416c8422ade482286ea2.png
[equation_tex_5Cbegin_7Baligned_7D_5Chat_7BA_7D_7Bt_7D_5E_7B_5Cmathrm_7BGAE_7D_28_5Cgamma_2C_5Clambda_29_7D_26_3A_3D_281-_5Clambda_29_5Cleft_28_5Chat_7BA_7D_7Bt_7D_5E_7B_281_29_7D_2B_5Clambda_5Chat_7BA_7D_7Bt_7D_5E_7B_282_29_7D_2B_5Clambda_5E_7B2_7D_5Chat_7BA_7D_7Bt_7D_5E_7B_283_29_7D_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Clambda_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cright_29_2B_5Clambda_5E_7B2_7D_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Clambda_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cright_29_2B_5Clambda_5E_7B2_7D_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_5Cleft_281_2B_5Clambda_2B_5Clambda_5E_7B2_7D_2B_5Cldots_5Cright_29_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cleft_28_5Clambda_2B_5Clambda_5E_7B2_7D_2B_5Clambda_5E_7B2_7D_2B_5Cldots_5Cright_29_5Cright._5C_5C_26_5Cleft._5Cquad_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cleft_28_5Clambda_5E_7B2_7D_2B_5Clambda_5E_7B3_7D_2B_5Clambda_5E_7B4_7D_2B_5Cldots_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_281-_5Clambda_29_5Cleft_28_5Cdelta_7Bt_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B1_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cgamma_5Cdelta_7Bt_2B1_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B_5Clambda_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cgamma_5E_7B2_7D_5Cdelta_7Bt_2B2_7D_5E_7BV_7D_5Cleft_28_5Cfrac_7B_5Clambda_5E_7B2_7D_7D_7B1-_5Clambda_7D_5Cright_29_2B_5Cldots_5Cright_29_5C_5C_26_3D_5Csum_7Bl_3D0_7D_5E_7B_5Cinfty_7D_28_5Cgamma_5Clambda_29_5E_7Bl_7D_5Cdelta_7Bt_2Bl_7D_5E_7BV_7D_5Cend_7Baligned_7D_5C_5C]: https://www.zhihu.com/equation?tex=%5Cbegin%7Baligned%7D+%5Chat%7BA%7D_%7Bt%7D%5E%7B%5Cmathrm%7BGAE%7D%28%5Cgamma%2C+%5Clambda%29%7D+%26%3A%3D%281-%5Clambda%29%5Cleft%28%5Chat%7BA%7D_%7Bt%7D%5E%7B%281%29%7D%2B%5Clambda+%5Chat%7BA%7D_%7Bt%7D%5E%7B%282%29%7D%2B%5Clambda%5E%7B2%7D+%5Chat%7BA%7D_%7Bt%7D%5E%7B%283%29%7D%2B%5Cldots%5Cright%29+%5C%5C+%26%3D%281-%5Clambda%29%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Clambda%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%5Cright%29%2B%5Clambda%5E%7B2%7D%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%2B%5Cgamma%5E%7B2%7D+%5Cdelta_%7Bt%2B2%7D%5E%7BV%7D%5Cright%29%2B%5Cldots%5Cright%29+%5C%5C+%26%3D%281-%5Clambda%29%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Clambda%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%5Cright%29%2B%5Clambda%5E%7B2%7D%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%2B%5Cgamma%5E%7B2%7D+%5Cdelta_%7Bt%2B2%7D%5E%7BV%7D%5Cright%29%2B%5Cldots%5Cright%29+%5C%5C+%26%3D%281-%5Clambda%29%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%5Cleft%281%2B%5Clambda%2B%5Clambda%5E%7B2%7D%2B%5Cldots%5Cright%29%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%5Cleft%28%5Clambda%2B%5Clambda%5E%7B2%7D%2B%5Clambda%5E%7B2%7D%2B%5Cldots%5Cright%29%5Cright.%5C%5C+%26%5Cleft.%5Cquad%2B%5Cgamma%5E%7B2%7D+%5Cdelta_%7Bt%2B2%7D%5E%7BV%7D%5Cleft%28%5Clambda%5E%7B2%7D%2B%5Clambda%5E%7B3%7D%2B%5Clambda%5E%7B4%7D%2B%5Cldots%5Cright%29%2B%5Cldots%5Cright%29+%5C%5C+%26%3D%281-%5Clambda%29%5Cleft%28%5Cdelta_%7Bt%7D%5E%7BV%7D%5Cleft%28%5Cfrac%7B1%7D%7B1-%5Clambda%7D%5Cright%29%2B%5Cgamma+%5Cdelta_%7Bt%2B1%7D%5E%7BV%7D%5Cleft%28%5Cfrac%7B%5Clambda%7D%7B1-%5Clambda%7D%5Cright%29%2B%5Cgamma%5E%7B2%7D+%5Cdelta_%7Bt%2B2%7D%5E%7BV%7D%5Cleft%28%5Cfrac%7B%5Clambda%5E%7B2%7D%7D%7B1-%5Clambda%7D%5Cright%29%2B%5Cldots%5Cright%29+%5C%5C+%26%3D%5Csum_%7Bl%3D0%7D%5E%7B%5Cinfty%7D%28%5Cgamma+%5Clambda%29%5E%7Bl%7D+%5Cdelta_%7Bt%2Bl%7D%5E%7BV%7D+%5Cend%7Baligned%7D%5C%5C
[https_spinningup.openai.com_en_latest_algorithms_vpg.html]: https://link.zhihu.com/?target=https%3A//spinningup.openai.com/en/latest/algorithms/vpg.html
[High Dimensional Continuous Control Using Generalized Advantage Estimation]: https://link.zhihu.com/?target=https%3A//arxiv.org/abs/1506.02438