How to stream OpenAI Gym environment rendering within a Jupyter Notebook

jupyter_gym_render.md

Open jupyter with

$ xvfb-run -s "-screen 0 1400x900x24" jupyter notebook

In Jupyter

import matplotlib.pyplot as plt
%matplotlib inline

After each step

def show_state(env, step=0):
    plt.figure(3)
    plt.clf()
    plt.imshow(env.render(mode='rgb_array'))
    plt.title("%s. Step: %d" % (env._spec.id,step))
    
    plt.pause(0.001)  # pause for plots to update

    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())

not doing display.display(plt.gcf()) made it much better to view.