作者：王语其 蒋思卿 闫宇深

本篇文章记录笔者和其团队在完成Gym库（OpenAI发布）的LunarLander-v2地图时，基于传统强化学习算法Actor-Critic架构以及TD(1)时序差分算法（offline-learning）、epsilon-greedy策略、MBGD+AdamW训练（学习率恒定，无学习率调度器）的实战，通过迁移性的架构设计（FFN、LSTM）和超参数调优，取得了连续100轮均分+104.38分的成果。另外，根据动作比率曲线分析，No Action曲线比例过低，而Main Engine比例偏高，通过该地图自带的动画窗口（"human"）发现着陆器虽然可以平稳低速降落，但和回归式训练的用于预测gamma*max_a(ADV(s_next,a))+R(s_current,a_current)里面的ADV(s_next,a)的延迟更新DQN-2号（延迟更新轮数一般在10轮左右，不可太大或太小）和 DQN-1号（ADV(s,a)=Q（s,a）- mean_a(Q(s,a)）)、Double DQN（ADV（s,a）=V(s) + Q（s,a）- mean_a(Q(s,a)）)、Double Dueffling DQN（状态回归和状态-动作回归双模型共享模型基座）相比，DQN（纯基于FFN或者FC网络）基于online-learning、缓存池和随机（时间步step不连续）批量抽样训练，可以取得连续100个Episode序列得分平均超过280分，是现代强化学习的常用技巧，还存在较大的优化空间。

当然不是说Actor-Critic架构过时了，第一是因为这个方法在LunarLander地图少有实现或者说应用，而主流或者AI给出的方法都是基于常见的三种DQN架构；第二是因为A2C+MC采样（offline-learning）架构具有Low Bias,High Variance的特性，同时也是现代大模型基于人类反馈的强化学习（RLHF）后训练方法的核心策略之一（每个样本的输出中间的R（s,a）都为0，最终生成结束有一个奖励模型的评分，算是A2C的现代发展之一，也是Episode-level+评论家、决策模型同步更新），具身智能（自动控制）可能在Q-learning方面应用更多。

Content

Environment Clarification

Actions and States

Reward and Termination Conditions

Transition

Initialization

Training Setting

Sampling

Training

Model Saving and Termination Condition

Evaluation Results

Architecture and Hyperparameters

Asynchronous Updated FFN

Synchronous Updated LSTM-FFN

Hyperparameters

---

Environment Clarification

This study is based on OpenAI’s gym package. The map is LunarLander-v2, which needs reinstallation of box2d and local compilation. If compiling tool swig is missing, refer to Download SWIG to download and configure the tool.

Actions and States

There are 4 available actions(0 for doing nothing, 1 for left jet propulsal,2 for main engine propulsal,3 for right jet propulsal) and the lander state consists of 8 parameters: x,y, θ（orientation angle), vx,vy,w(angular speed),left_foot_ground_contact_detect（bool） and

right_foot_ground_contact_detect（bool）. The state space is infinite and traditional MDP solving schedule by dynamic programming is out of date, here we introduce non-linear models to learning relationship between state space and action space, i.e., an appropriate mapping.

Reward and Termination Conditions

For the reward part, an episode finishes if the lander crashes because of overspeed landing or landing with inappropriate angle(1), goes out of the screen without contacting the ground(2) or comes to rest(3) , receiving an additional -100 or +100 points, respectively, on top of the base reward. Each leg ground contact is worth +10 points. Firing the main engine incurs a -0.3 point penalty and firing the orientation engines incurs a -0.03 point penalty, for each occurrence. It is also important to notice that it is flexible and supportive to do slight changes on step rewards to obtain better scores (calculated with original method).

Transition

The transition occurs with a basic 50Hz frequency which can not be altered by doing one of the 4 actions (definite and accurate transition by physical computation).

Initialization

The probing spacecraft is initialized to be set at the top of the screen with y=0 and x a small random value around 0, guaranteeing initialization randomness and landing robustness.

Training Setting

Training is divided into sampling process and neural network parameter update process.

Sampling

In the first process, epsilon-greedy policy is employed with ε=0.01+0.99*exp(-(count-1)*0.01), action is determined or randomly selected based on epsilon and quiet(used to guarantee larger possibility of choosing action 0 in random mode). The state, reward and action sequences are recorded at this section. Discount rate gamma is set to 0.99 in the study, for an episode with 300 or more steps, the future far end of the reward sequence discounts to less than 5% of itself while reward 100 steps away only discounts to about 37% of itself.

Training

For RL part, use epsilon-greedy algorithm combined with Monte-Carlo reward design(Episode Sampling-Training).

Since the study employs Neural Network design, the parameter update process consists of gradient accumulation, learning rate scheduling and optimizer updating. If the network supports one-time sequence input of states, then forward process is executed once as a whole for every episode, otherwise do forward propagation and backward propagation one state by one state. The gradient is accumulated for one or multiple episodes based on input argument size and network weights are then updated.

In loss computation part, action sequence is used to select corresponding elements of output logits. By the way, it is used to record ratios of 4 actions following the whole training process.

A speed constraint is added by hand into the sampled reward for training rather than demonstrated reward.

Model Saving and Termination Condition

If average score (no discount) based on last 100 consecutive episodes exceeds input argument target or over maximum epochs limit (1000 or 2000 in the study), then the training process terminates.

If average score (no discount) based on size exceeds 0 and is higher than any other previous eligible record, the model weights is saved as .pth file.

Figure 1  Training Score Curve of LSTM-FFN A2C System (synchronous updated base LSTM framework) for 2000 epochs with 20000 episodes, highest episode-level mark about 277

Figure 2  Training Score Output in Console of FFN A2C System (asynchronous updated framework), with an average of 57.34 MC score over 50 consecutive episodes

Figure 3  Saved model weights for FFN A2C System

Evaluation Results

Evaluation is based on 100 consecutive episodes to obtain 100 separate scores (without discount). The action chosen procedure is the same as that of Sampling process in part II. See several total-score graphs below for different architecture and hyperparameter settings.

Models which don’t perform as well as the target requires however acquires total MC score (without discount) over 0 can also be evaluated.

Figure 4  Evaluation Score and Action Ratio Curves of FFN A2C System for 100 consecutive episodes with average MC score 104.38

Architecture and Hyperparameters

Considering relatively high-tech RL architecture——Actor and Critic(A2C) System. It is a two model based system where actor predicts and chooses action with maximum probability corresponding to current state and critic regresses discounted accumulated MC reward from current step to the end of the episode. Advantage is calculated by subtracting predicted MC reward from true MC reward acquired from sampling and substitutes true MC reward in one model architecture.

 

We mainly introduce and employ two systems-Asynchronous Updated FFN & Synchronous Updated LSTM-FFN combined neural network.

Asynchronous Updated FFN

Synchronous Updated LSTM-FFN

Hyperparameters

For discount rate gamma, it is recommended to set to 0.99 but not 1. Lower gamma has demonstrated obviously much worse result for the same architecture under comparative experiments.

For size=50,epoch=1, gamma=0.999 or 0.9 and size=50,epoch=1,gamma=0.99, the latter one obtains much better result.

For gradient accumulation controlling parameter size, a smaller size=1 causes rapid fluctuation and a bigger size=200 causes complex and not outstanding characteristics. Size such as 10,20 and 50 are better choices.

For network depth, 2 hidden layers structure is recommended for FFN and LSTM-FFN A2C systems. What’s more, total amount of parameters for FFN is 256x256+256x12+64x64+…=73825 and for LSTM-FFN is about 20000.