Skip to content

Instantly share code, notes, and snippets.

@pat-coady

pat-coady/README.md

Last active Aug 17, 2017
Embed
What would you like to do?
Proximal Policy Optimization with Generalized Advantage Estimation

Proximal Policy Optimization with Generalized Advantage Estimation

By Patrick Coady: Learning Artificial Intelligence

Summary

The same learning algorithm was used to train agents for each of the ten OpenAI Gym MuJoCo continuous control environments. The only difference between evaluations was the number of episodes used per training batch, otherwise all options were the same. The code is available in the GitHub repository. The exact code used to generate the submissions is in the aigym_evaluation branch.

The README.md file in the GitHub repository provides additional details on the algorithm and usage instructions.

Here are the key points:

  • Proximal Policy Optimization (similar to TRPO, but uses gradient descent with KL loss terms) [1] [2]
  • Value function approximated with 3 hidden-layer NN (tanh activations):
    • hid1 size = obs_dim x 10
    • hid2 size = geometric mean of hid1 and hid3 sizes
    • hid3 size = 5
  • Policy is a multi-variate Gaussian parameterized by a 3 hidden-layer NN (tanh activations):
    • hid1 size = obs_dim x 10
    • hid2 size = geometric mean of hid1 and hid3 sizes
    • hid3 size = action_dim x 10
    • Diagonal covariance matrix variables are separately trained
  • Generalized Advantage Estimation (gamma = 0.995, lambda = 0.98) [3] [4]
  • ADAM optimizer used for both neural networks
  • The policy is evaluated for 20 episodes between updates, except:
    • 50 episodes for Reacher
    • 5 episodes for Swimmer
    • 5 episodes for HalfCheetah
    • 5 episodes for HumanoidStandup
  • Value function is trained on current batch + previous batch
  • KL loss factor and ADAM learning rate are dynamically adjusted during training
  • Policy and Value NNs built with TensorFlow

Results can be reproduced as follows:

./train.py Reacher-v1 -n 60000 -b 50
./train.py InvertedPendulum-v1
./train.py InvertedDoublePendulum-v1 -n 12000
./train.py Swimmer-v1 -n 2500 -b 5
./train.py Hopper-v1 -n 30000
./train.py HalfCheetah-v1 -n 3000 -b 5
./train.py Walker2d-v1 -n 25000
./train.py Ant-v1 -n 100000
./train.py Humanoid-v1 -n 200000
./train.py HumanoidStandup-v1 -n 200000 -b 5

References

  1. Trust Region Policy Optimization (Schulman et al., 2016)
  2. Emergence of Locomotion Behaviours in Rich Environments (Heess et al., 2017)
  3. High-Dimensional Continuous Control Using Generalized Advantage Estimation (Schulman et al., 2016)
  4. GitHub Repository with several helpful implementation ideas (Schulman)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.