SageMaker RL에서 Ray로 Roboschool Reacher 풀이

rl_roboschool_ray_distributed_simple.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker로 다중 노드들간 분산 RL을 이용해 Roboschool 에이전트를 훈련하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 주의) 로컬 노트북 인스턴스에서 실행시 권한\n",
    "- SageMaker 노트북 인스턴스는 SageMaker에 맞게 설정된 IAM Role을 이용\n",
    "- 로컬 노트북 인스턴스는 로컬에서 설정된 IAM User를 사용한다\n",
    "- IAM User로 이 노트북을 실행하기 위해서는 다음과 같은 권한이 필요\n",
    "  - AmazonSageMakerFullAccess\n",
    "  - AmazonEC2ContainerRegistryFullAccess\n",
    "  - IAMFullAccess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 사용할 Roboschool 과제 선택"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "roboschool_problem = 'reacher'  # 쉬움\n",
    "#roboschool_problem = 'hopper'\n",
    "#roboschool_problem = 'humanoid'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 먼저 필요한 것들\n",
    "\n",
    "### 외부 모듈 import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "import sys\n",
    "import os\n",
    "import glob\n",
    "import re\n",
    "import subprocess\n",
    "from IPython.display import HTML, Markdown\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "sys.path.append(\"common\")\n",
    "from misc import get_execution_role, wait_for_s3_object\n",
    "from docker_utils import build_and_push_docker_image\n",
    "from sagemaker.rl import RLEstimator, RLToolkit, RLFramework\n",
    "from markdown_helper import generate_help_for_s3_endpoint_permissions, create_s3_endpoint_manually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### S3 버킷 설정\n",
    "\n",
    "체크포인트와 메타데이터를 위한 S3 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
     ]
    }
   ],
   "source": [
    "sage_session = sagemaker.session.Session()\n",
    "s3_bucket = sage_session.default_bucket()  \n",
    "s3_output_path = 's3://{}/'.format(s3_bucket)\n",
    "print(\"S3 bucket path: {}\".format(s3_output_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 변수 정의\n",
    "\n",
    "훈련 작업을 위한 접두어와 *컨테이너를 위한 이미지 경로(BYOC일 때만 필요)*를 정의"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'rl-roboschool-distributed-reacher'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 설명적인 작업 이름 만들기\n",
    "job_name_prefix = 'rl-roboschool-distributed-' + roboschool_problem\n",
    "aws_region = boto3.Session().region_name\n",
    "job_name_prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 훈련 장소 설정\n",
    "\n",
    "SageMaker 노트북 인스턴스나 로컬 노트북 인스턴스에서 훈련 가능. 노트북 인스턴스에서 훈련하는 것을 **로컬 모드**라 함. 로컬 모드에서는 로컬 컨테이너에서 코드 실행을 위해 SageMaker Python SDK를 이용. 로컬 모드는 초기 개발 및 더비깅에 적합. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 로컬 모드(노트북 인스턴스에서 훈련) 여부\n",
    "local_mode = False\n",
    "\n",
    "if local_mode:\n",
    "    instance_type = 'local'\n",
    "else:\n",
    "    # SageMaker에서 훈련시, 인트턴스 타입 선택 \n",
    "    instance_type = \"ml.c5.2xlarge\"\n",
    "    \n",
    "# 훈련용 인스턴스 수    \n",
    "train_instance_count = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IAM 역할 생성\n",
    "\n",
    "SageMaker 노트북 인스턴스에서 role = sagemaker.get_execution_role() 또는 로컬 노트북 인스턴스에서 유틸리티 함수 role = get_execution_role()를 호출해 실행 역할을 얻음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except:\n",
    "    role = get_execution_role()\n",
    "\n",
    "print(\"Using IAM role arn: {}\".format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 로컬 모드를 위한 도커 설치 \n",
    "\n",
    "로컬 모드를 위해서는 아래와 같은 도커 환경이 설치되어 있어야 함:\n",
    "- docker\n",
    "- docker-compose\n",
    "- nvidia-docker (로컬 GPU 장비를 위해)\n",
    "\n",
    "SageMaker 노트북 인스턴스에서는 아래의 스크립트 수행하면 설치됨."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SageMaker 노트북 인스턴스에서만 실행\n",
    "if local_mode:\n",
    "    !/bin/bash ./common/setup.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 도커 컨테이너 빌드\n",
    "\n",
    "아래 과정으로 Roboschool이 설치된 도커 컨테이너를 빌드해야 한다:\n",
    "\n",
    "1. 기본 컨테이너 이미지 얻음 \n",
    "2. Roboschool과 의존 패키지 설치 \n",
    "3. 새 컨테이너 이미지를 ECR에 업로드 \n",
    "\n",
    "3~4분 정도 걸림"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING! Using --password via the CLI is insecure. Use --password-stdin.\n",
      "Login Succeeded\n",
      "Logged into ECR\n",
      "Building docker image sagemaker-roboschool-ray-cpu from Dockerfile\n",
      "$ docker build -t sagemaker-roboschool-ray-cpu -f Dockerfile . --build-arg CPU_OR_GPU=cpu --build-arg AWS_REGION=ap-northeast-2\n",
      "Sending build context to Docker daemon  464.4kB\n",
      "Step 1/14 : ARG CPU_OR_GPU\n",
      "Step 2/14 : ARG AWS_REGION\n",
      "Step 3/14 : FROM 520713654638.dkr.ecr.${AWS_REGION}.amazonaws.com/sagemaker-rl-tensorflow:ray0.5.3-${CPU_OR_GPU}-py3\n",
      " ---> b522ab9d6e52\n",
      "Step 4/14 : WORKDIR /opt/ml\n",
      " ---> Using cache\n",
      " ---> 888e0f77a0c8\n",
      "Step 5/14 : RUN apt-get update && apt-get install -y       git cmake ffmpeg pkg-config       qtbase5-dev libqt5opengl5-dev libassimp-dev       libtinyxml-dev       libgl1-mesa-dev     && cd /opt     && apt-get clean && rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/*\n",
      " ---> Using cache\n",
      " ---> 6751e6cb26af\n",
      "Step 6/14 : RUN apt-get update &&     apt-get install -y libboost-python-dev\n",
      " ---> Using cache\n",
      " ---> 7cdde49abdda\n",
      "Step 7/14 : RUN apt-get update     && apt-get install -y --no-install-recommends python3.6-dev     && ln -s -f /usr/bin/python3.6 /usr/bin/python     && apt-get clean     && rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/*\n",
      " ---> Using cache\n",
      " ---> 545c2184394b\n",
      "Step 8/14 : RUN curl -fSsL -O https://bootstrap.pypa.io/get-pip.py &&     python get-pip.py &&     rm get-pip.py\n",
      " ---> Using cache\n",
      " ---> 72d3593543de\n",
      "Step 9/14 : RUN pip install --upgrade     pip     setuptools\n",
      " ---> Using cache\n",
      " ---> 47deb5234f9d\n",
      "Step 10/14 : RUN pip install sagemaker-containers --upgrade\n",
      " ---> Using cache\n",
      " ---> 0f48f773151d\n",
      "Step 11/14 : RUN pip install roboschool==1.0.46\n",
      " ---> Using cache\n",
      " ---> b7dd4076a46c\n",
      "Step 12/14 : ENV PYTHONUNBUFFERED 1\n",
      " ---> Using cache\n",
      " ---> 108f959cf33e\n",
      "Step 13/14 : RUN python -c \"import gym;import sagemaker_containers.cli.train;import roboschool; import ray; from sagemaker_containers.cli.train import main\"\n",
      " ---> Using cache\n",
      " ---> 36f65ce4adf3\n",
      "Step 14/14 : WORKDIR /opt/ml/code\n",
      " ---> Using cache\n",
      " ---> c2d2e27d72fc\n",
      "Successfully built c2d2e27d72fc\n",
      "Successfully tagged sagemaker-roboschool-ray-cpu:latest\n",
      "Done building docker image sagemaker-roboschool-ray-cpu\n",
      "ECR repository already exists: sagemaker-roboschool-ray-cpu\n",
      "WARNING! Using --password via the CLI is insecure. Use --password-stdin.\n",
      "Login Succeeded\n",
      "Logged into ECR\n",
      "$ docker tag sagemaker-roboschool-ray-cpu 415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu\n",
      "Pushing docker image to ECR repository 415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu\n",
      "\n",
      "$ docker push 415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu\n",
      "The push refers to repository [415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu]\n",
      "c4e8cce86b64: Preparing\n",
      "b438630f4a71: Preparing\n",
      "83256b64f5df: Preparing\n",
      "6cbdea9e59f6: Preparing\n",
      "725ba5f36903: Preparing\n",
      "65fed3034f32: Preparing\n",
      "d1d3b9d7fb4e: Preparing\n",
      "3ad59e27f6bf: Preparing\n",
      "9702a85a1269: Preparing\n",
      "79e4b43308b4: Preparing\n",
      "60a4c6264060: Preparing\n",
      "6ee8fe30b55b: Preparing\n",
      "36cb14e2633d: Preparing\n",
      "c9bbcdfaaa98: Preparing\n",
      "09b011f6bf18: Preparing\n",
      "296294040c48: Preparing\n",
      "302af95b9e68: Preparing\n",
      "0ecacb4d5bb1: Preparing\n",
      "d6db7d05e3b2: Preparing\n",
      "8d999119430c: Preparing\n",
      "fe4ed9a0e78a: Preparing\n",
      "3db5746c911a: Preparing\n",
      "819a824caf70: Preparing\n",
      "647265b9d8bc: Preparing\n",
      "41c002c8a6fd: Preparing\n",
      "65fed3034f32: Waiting\n",
      "d1d3b9d7fb4e: Waiting\n",
      "3ad59e27f6bf: Waiting\n",
      "9702a85a1269: Waiting\n",
      "79e4b43308b4: Waiting\n",
      "60a4c6264060: Waiting\n",
      "6ee8fe30b55b: Waiting\n",
      "36cb14e2633d: Waiting\n",
      "c9bbcdfaaa98: Waiting\n",
      "fe4ed9a0e78a: Waiting\n",
      "3db5746c911a: Waiting\n",
      "819a824caf70: Waiting\n",
      "09b011f6bf18: Waiting\n",
      "296294040c48: Waiting\n",
      "302af95b9e68: Waiting\n",
      "0ecacb4d5bb1: Waiting\n",
      "d6db7d05e3b2: Waiting\n",
      "8d999119430c: Waiting\n",
      "647265b9d8bc: Waiting\n",
      "41c002c8a6fd: Waiting\n",
      "b438630f4a71: Layer already exists\n",
      "83256b64f5df: Layer already exists\n",
      "725ba5f36903: Layer already exists\n",
      "6cbdea9e59f6: Layer already exists\n",
      "c4e8cce86b64: Layer already exists\n",
      "65fed3034f32: Layer already exists\n",
      "d1d3b9d7fb4e: Layer already exists\n",
      "3ad59e27f6bf: Layer already exists\n",
      "79e4b43308b4: Layer already exists\n",
      "9702a85a1269: Layer already exists\n",
      "60a4c6264060: Layer already exists\n",
      "6ee8fe30b55b: Layer already exists\n",
      "c9bbcdfaaa98: Layer already exists\n",
      "09b011f6bf18: Layer already exists\n",
      "36cb14e2633d: Layer already exists\n",
      "296294040c48: Layer already exists\n",
      "302af95b9e68: Layer already exists\n",
      "0ecacb4d5bb1: Layer already exists\n",
      "d6db7d05e3b2: Layer already exists\n",
      "8d999119430c: Layer already exists\n",
      "41c002c8a6fd: Layer already exists\n",
      "819a824caf70: Layer already exists\n",
      "fe4ed9a0e78a: Layer already exists\n",
      "3db5746c911a: Layer already exists\n",
      "647265b9d8bc: Layer already exists\n",
      "latest: digest: sha256:f98f6ab259615f76be64336bd34b18ed9ec72cce2dae6eaf488780a9c44d9f6f size: 5573\n",
      "Done pushing 415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu\n",
      "Using ECR image 415742736303.dkr.ecr.ap-northeast-2.amazonaws.com/sagemaker-roboschool-ray-cpu\n",
      "CPU times: user 147 ms, sys: 52 ms, total: 199 ms\n",
      "Wall time: 3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cpu_or_gpu = 'gpu' if instance_type.startswith('ml.p') else 'cpu'\n",
    "repository_short_name = \"sagemaker-roboschool-ray-%s\" % cpu_or_gpu\n",
    "docker_build_args = {\n",
    "    'CPU_OR_GPU': cpu_or_gpu, \n",
    "    'AWS_REGION': boto3.Session().region_name,\n",
    "}\n",
    "custom_image_name = build_and_push_docker_image(repository_short_name, build_args=docker_build_args)\n",
    "print(\"Using ECR image %s\" % custom_image_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 학습 코드 작성\n",
    "\n",
    "훈련 코드는 `src/` 디렉토리의 `train-reacher.py` 파일에 저장되어 있음. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mgym\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mray\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mray.tune\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m run_experiments\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mray.tune.registry\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m register_env\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mroboschool\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msagemaker_rl.ray_launcher\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m SageMakerRayLauncher\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mcreate_environment\u001b[39;49;00m(env_config):\n",
      "     \u001b[37m# 워커 프로세스를 위해 이 import는 메소드 내에서 수행되어야 함.\u001b[39;49;00m\n",
      "    \u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mroboschool\u001b[39;49;00m\n",
      "    \u001b[34mreturn\u001b[39;49;00m gym.make(\u001b[33m'\u001b[39;49;00m\u001b[33mRoboschoolReacher-v1\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "\n",
      "\u001b[37m# SageMakerRayLauncher는 Ray-RLLib을 이용한 SageMaker RL 어플리케이션을 위한 기본 클래스.\u001b[39;49;00m\n",
      "\u001b[37m# 사용자는 이것을 상속받아 필요한 메소드를 구현하고, train_main()을 호출하여 훈련 프로세스를 시작.\u001b[39;49;00m\n",
      "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mMyLauncher\u001b[39;49;00m(SageMakerRayLauncher):\n",
      "\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32mregister_env_creator\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m):\n",
      "        register_env(\u001b[33m\"\u001b[39;49;00m\u001b[33mRoboschoolReacher-v1\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, create_environment)\n",
      "\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32mget_experiment_config\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m):\n",
      "        \u001b[34mreturn\u001b[39;49;00m {\n",
      "          \u001b[33m\"\u001b[39;49;00m\u001b[33mtraining\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: {\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33menv\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[33m\"\u001b[39;49;00m\u001b[33mRoboschoolReacher-v1\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33mrun\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[33m\"\u001b[39;49;00m\u001b[33mPPO\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33mstop\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: {\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mepisode_reward_mean\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m18\u001b[39;49;00m,\n",
      "            },\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33mconfig\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: {\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mgamma\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m0.995\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mkl_coeff\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m1.0\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mnum_sgd_iter\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mlr\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m0.0001\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33msgd_minibatch_size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m1000\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mtrain_batch_size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m25000\u001b[39;49;00m,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mmonitor\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[36mTrue\u001b[39;49;00m,  \u001b[37m# 비디오 저장\u001b[39;49;00m\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mmodel\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: {\n",
      "                \u001b[33m\"\u001b[39;49;00m\u001b[33mfree_log_std\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[36mTrue\u001b[39;49;00m\n",
      "              },\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mnum_workers\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: (\u001b[36mself\u001b[39;49;00m.num_cpus-\u001b[34m1\u001b[39;49;00m),\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mnum_gpus\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[36mself\u001b[39;49;00m.num_gpus,\n",
      "              \u001b[33m\"\u001b[39;49;00m\u001b[33mbatch_mode\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[33m\"\u001b[39;49;00m\u001b[33mcomplete_episodes\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "    MyLauncher().train_main()\n"
     ]
    }
   ],
   "source": [
    "!pygmentize src/train-{roboschool_problem}.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ray 균질(homogeneous) 스케일링 - train_instance_count을 1 보다 크게 설정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "균질 스케일링은 같은 타입의 다중 인스턴스를 이용하게 해줌. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training job: rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580\n"
     ]
    }
   ],
   "source": [
    "# Ray의 훈련 척도 이용 \n",
    "metric_definitions = RLEstimator.default_metric_definitions(RLToolkit.RAY)\n",
    "    \n",
    "# 훈련 시작    \n",
    "estimator = RLEstimator(entry_point=\"train-%s.py\" % roboschool_problem,\n",
    "                        source_dir='src',\n",
    "                        dependencies=[\"common/sagemaker_rl\"],\n",
    "                        image_name=custom_image_name,\n",
    "                        role=role,\n",
    "                        train_instance_type=instance_type,\n",
    "                        train_instance_count=train_instance_count,\n",
    "                        output_path=s3_output_path,\n",
    "                        base_job_name=job_name_prefix,\n",
    "                        metric_definitions=metric_definitions,\n",
    "                        hyperparameters={\n",
    "                          # 여기에서 Ray 알고리즘의 인자들을 설정할 수 있음.\n",
    "                            \n",
    "                            # 8코어의 m4.2xl 인스턴스 3 대. 1개의 코어는 Ray 스케쥴러를 위한 것.\n",
    "                            \"rl.training.config.num_workers\": (8 * train_instance_count) - 1                            \n",
    "                            \n",
    "                          #\"rl.training.config.horizon\": 5000,\n",
    "                          #\"rl.training.config.num_sgd_iter\": 10,\n",
    "                        }\n",
    "                    )\n",
    "\n",
    "# 훈련 시작(로컬 모드: Sync, 리모트 모드: Async)\n",
    "estimator.fit(wait=local_mode)\n",
    "job_name = estimator.latest_training_job.job_name\n",
    "print(\"Training job: %s\" % job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 시각화\n",
    "\n",
    "RL 훈련은 시간이 많이 걸리기에 훈련 과정 추적이 필요. 훈련 중 중간 출력물이 S3에 저장되기에, 그것을 캡쳐."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job name: rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580\n",
      "S3 job path: s3://sagemaker-ap-northeast-2-415742736303/rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580\n",
      "Output.tar.gz location: s3://sagemaker-ap-northeast-2-415742736303/rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/output.tar.gz\n",
      "Intermediate folder path: s3://sagemaker-ap-northeast-2-415742736303/rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/\n",
      "Create local folder /tmp/rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580\n"
     ]
    }
   ],
   "source": [
    "print(\"Job name: {}\".format(job_name))\n",
    "\n",
    "s3_url = \"s3://{}/{}\".format(s3_bucket,job_name)\n",
    "\n",
    "if local_mode:\n",
    "    output_tar_key = \"{}/output.tar.gz\".format(job_name)\n",
    "else:\n",
    "    output_tar_key = \"{}/output/output.tar.gz\".format(job_name)\n",
    "\n",
    "intermediate_folder_key = \"{}/output/intermediate/\".format(job_name)\n",
    "output_url = \"s3://{}/{}\".format(s3_bucket, output_tar_key)\n",
    "intermediate_url = \"s3://{}/{}\".format(s3_bucket, intermediate_folder_key)\n",
    "\n",
    "print(\"S3 job path: {}\".format(s3_url))\n",
    "print(\"Output.tar.gz location: {}\".format(output_url))\n",
    "print(\"Intermediate folder path: {}\".format(intermediate_url))\n",
    "    \n",
    "tmp_dir = \"/tmp/{}\".format(job_name)\n",
    "os.system(\"mkdir {}\".format(tmp_dir))\n",
    "print(\"Create local folder {}\".format(tmp_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 훈련 롤아웃 비디오를 얻음\n",
    "\n",
    "훈련중 일부 롤아웃 비디오가 S3에 저장됨. 여기에서는 최근 10개 비디오를 가져오고, 그중 마지막을 플레이."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for s3://sagemaker-ap-northeast-2-415742736303/rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/...\n",
      "Only downloading 10 of 143 files\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000000.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000001.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000008.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000027.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000064.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000125.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000216.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000343.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000512.mp4\n",
      "Downloading rl-roboschool-distributed-reacher-2019-05-08-10-20-27-580/output/intermediate/training/PPO_RoboschoolReacher-v1_0_2019-05-08_10-23-4188iwjt0k/openaigym.video.0.136.video000729.mp4\n"
     ]
    }
   ],
   "source": [
    "recent_videos = wait_for_s3_object(s3_bucket, intermediate_folder_key, tmp_dir, \n",
    "                                fetch_only=(lambda obj: obj.key.endswith(\".mp4\") and obj.size>0), limit=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./src/tmp_render/last_video.mp4\" controls autoplay></video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_video = sorted(recent_videos)[-1]  # Pick which video to watch\n",
    "os.system(\"mkdir -p ./src/tmp_render/ && cp {} ./src/tmp_render/last_video.mp4\".format(last_video))\n",
    "HTML('<video src=\"./src/tmp_render/last_video.mp4\" controls autoplay></video>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 훈련 작업의 성능 척도를 그림\n",
    "\n",
    "CloudWatch에 저장된 알고리즘 척도를 이용해 실행중인 훈련의 리워드를 볼 수 있음."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtIAAAFACAYAAACRLJt/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcneP9//HXJ7tIBCO2EIk9lqKiQVGkltprj6U02tTSopZv/eiirZYWpdYKsZVSWwmxNqRRShtLa1dLQxAmaVaR/fr9cZ/pTCYzyck459xzZl7Px+M85px77rnvz7mNmXeu+dzXFSklJEmSJC2bDnkXIEmSJFUjg7QkSZLUAgZpSZIkqQUM0pIkSVILGKQlSZKkFjBIS5IkSS1gkJYkSZJawCAtSZIktYBBWpIkSWqBTnkXUKxVVlkl9evXL+8yJEmS1MY999xzk1JKvZe2X9UE6X79+jFu3Li8y5AkSVIbFxHji9nP1g5JkiSpBQzSkiRJUgsYpCVJkqQWqJoe6abMmzePCRMmMHv27LxLqYhu3bqx1lpr0blz57xLkSRJaveqOkhPmDCBnj170q9fPyIi73LKKqXE5MmTmTBhAv3798+7HEmSpHavqls7Zs+eTU1NTZsP0QARQU1NTbsZfZckSWrtqjpIA+0iRNdpT+9VkiSptav6IC1JkiTlwSBdATfeeCNnnXVW3mVIkiSphKr6ZkNJkiSppRYuhGnTYPJkmDQp+zh5cvFf32aC9KmnwosvlvaYW24Jl17a/OcPPfRQTj/9dAYNGsS8efPYYost2GmnnXjzzTeZNm0a++yzDz/96U8X+ZrVV1+diRMnAvC73/2OiRMncu6553LLLbdw9dVX06lTJ/r378+IESPo2LFjad+QJElSGzV/Pvz3v4sG4sYBufHnJk/OwnRLtZkgnYfvfe97DB8+nEGDBvGnP/2J/fbbj6OPPpqNN96Y8ePHs8022ywWpJvy1ltvcdlllzF27Fi6devGGWecwR133MGQIUMq8C4kSZJal9mzly0QT5qUjSw3p2tXWGUVqKnJHptvXv+8pmbRz9XUwAYbFFdnmwnSSxo5Lpcdd9yRs846i+nTpzNixAiuvfZazjnnHFZbbTX69etHhw5LbkFfsGABAC+++CITJ05kzz33BGDmzJmsueaaZa9fkiSpnFKCGTOWLRBPngyzZjV/zJ49Fw3A66/fdBhu+Lp7dyjH5GdtJkjnZejQoZx77rn06NGDu+++mwEDBnD22WfzxhtvcNpppy22f6dOnZg5cybdu3fnwQcfZJtttmGTTTZhwIABPPTQQ3To0IHp06czb968HN6NJElS01KCmTPh44+htrb4EePmIk0ErLRSfejt0we+8IUlB+KaGujSpbLve0kM0p/TEUccwdprr80dd9zBqquuypAhQxg1ahRbbbUVK6644mL7n3322eywww6svfba/xt13mSTTTj44IPZYYcd6Nq1K126dOGaa66hpqam0m9HkiS1IynB1KlZOG78+OSTxbd99lnTx+nUadHAu+GGi75uKhyvuCJU++1gkVLKu4aiDBw4MI0bN26Rba+99hoDBgzIqaJ8tMf3LEmSirdwYTYS3FwYbhyW585d/BgdO0Lv3rDaatlj1VXrn9e9bhiQe/YsT+tEXiLiuZTSwKXt54i0JElSKzd/ftYusaRQXPeorYXCbViL6Nx50UC8+eaLhuOGj5oaWMqtXsIgLUmSlIu5c4sbMf744yxEN9VE0K1bffjt2xe22abpYLzqqlk/clsaNW4NDNKSJEklMmtWffhdWkieMqXpY/ToUR+A118fvvzl5keO21pLRbWp+iCdUiLayXdQtfSzS5LUVsybl7VKfPJJ9mj4vPGjthY+/bTp4/TqVR9+N9sMBg9uPhx3717Z96iWq+og3a1bNyZPnkxNTU2bD9MpJSZPnky3bt3yLkWSpKq1cGG2+l0xofiTT5ofNa7rN657bLjhoq8b36jnr++2qaqD9FprrcWECROora3Nu5SK6NatG2uttVbeZUiS1GrULfjRXChuvH3SpKZvxIvIZqDo3TsLvltssWgwrgvHdc979bKlQlUepDt37kz//v3zLkOSJJXQ7NlNB+PmwvKcOU0fZ4UV6oPveuvBdtstGoYbPmpqqn9OY1VeVQdpSZLU+s2dm40E19bWP5YUjGfMaPo4XbvWt0qsumrWa9xUKO7dO3vYTqFyM0hLkqRlMnfu4oG4qZBc95g6tenjdOiw6Ajxl77UdBtF3aNHD9sp1LpUJEhHxEbADcB7KaXDI6If8AzwemGXmSmlfSpRiyRJWtScOcWH4k8+genTmz5Ox46L9hl/8Yv1o8N12xq+XnllF/1QdavUiPQg4DLggAbbHk4pHVuh80uS1G589tnigbi5UFxb23wrRadOWTCuC8ADBzYdiOu2rbiiwVjtS0WCdErp5ojYudHmwRHxV2AucGlKaWQlapEkqdrMmlV8KK6thZkzmz5O586Lht/+/ZsPxb17Z8HYVgqpeXn1SI8H+qaUUkT0BR6LiDdSSm803CkihgHDAPr27ZtDmZIkldf8+fD++/DOO/Duu9njnXfgP/+BiROXvMhHly6LBuANNlhyK4VTtkmllUuQTg2W6EspvRcRfwY2Bd5otN9wYDjAwIEDXdZPklR1UsrCcOOgXPfx/fcXnde4Y0dYZ51stHiHHZoPxauu6vLQUt5yCdIRsSHwfkrps4hYCdgRuCCPWiRJ+rxmzqwPyY2D8rvvZq0ZDa22WhaUt98++9i/P6y7bvZxrbWy3mRJrV9e/6uuCVwfEQuAzsA5KaX3c6pFkqQlmjcvGzluGI4bPm+8wG6PHlkoXn992G23RYNyv36w/PK5vA1JJVaxIJ1SGgOMafB8h0qdW5KkJUkpu1mvuaDcuP2iUyfo2zcLxwccUB+S6wJzTY0tF1J74B+PJEntQl37RVNBuan2i9VXr2+/aByU+/Sx/UKSQVqS1EbUtV807k+uez5p0qL79+xZ336x++6Lt190757L25BURQzSkqSqMH06TJhQ//jgg2yKuIazXyxcWL9/p07Z7BfrrgsHHrhoUO7f3/YLSZ+fQVqSlKuUstHiunDcMCg3DM5NLTKy+upZON5hh0WDcl37RceOlX8/ktoPg7QkqWwWLMgWFWkuHH/wQfaYM2fRr+vYEdZYI5sKbrPNYI89sud1jz59YM01oWvXfN6XJIFBWpLUQnPm1AfhxuG47vnEiYvOdgFZ+K0Lw9tuu2g4rnu+2mqOJktq/QzSkqTFzJzZfDiue9547mTIbuCrC8O7775oOK57bm+ypLbCIC1J7UhK8N//LnkUecKE7Ma+xlZZpT4Qf+lLi48i9+kDK6xQ+fckSXkxSEtSGzNzJrzyCrz0Erz11uJBefbsRffv0CG7aW+ttWDjjWHw4MX7kfv0gW7d8nk/ktRaGaQlqUrNmwdvvpkF5pdfzj6+9FI2HVydzp3rw/DAgdkqfI1Hkldf3cVFJKkl/NEpSa1cSvDee/VBuS40v/56FqYhuzFvo42ylouhQ2HzzbNHv37ZiLMkqfQM0pLUikyatOjo8ssvZ48ZM+r36ds3C8l7751NDbf55lmIdio4Saosg7Qk5WDWLHj11cVHmSdOrN9n5ZWzkHzMMfWBedNNoVev/OqWJNUzSEtSGc2fD//+9+KjzG+/nbVsQHYT36abwp571gfmzTfPepedJk6SWi+DtCSVQErZjBiNA/Nrr9Wv2tehA2ywAWy5JRx9dH1oXnddFx+RpGpkkJakZTRlyqLtGHV9zFOn1u/Tp08WknfbrT4wDxjgFHKS1JYYpCWpGZ99lo0oNx5l/uCD+n169cpC8pAh9YF5s81gpZXyq1uSVBkGaUnt3oIFWc9y4/mY33oLFi7M9unaNRtR3nXX+rC8+ebZyLN9zJLUPhmkJbV5dctiv/de9hg/vv75229ns2fUrfYXAeutl4Xkww+vD8zrr++iJZKkRflrQVLVmzs3a7doKijXPT79dNGv6dYtm4+5Xz848cT6wLzJJtC9ey5vQ5JUZQzSklq1lLKb+BqG4sZB+cMP66eSq7PqqllQHjAgm1aub9/6xzrrwCqr2JIhSfp8DNKScjV//qKjyU0F5Yar+gF06VIfinfbLQvGDYPy2mvDcsvl834kSe2HQVpSWU2fvngwbvj6gw/qb+irs8oqWSDeYAMYPHjxoLzqqtmczJIk5ckgLanFFizI2ioa9yM3DMrTpi36NZ07ZyPGffvCLrvUt1o0HE1efvl83o8kScvCIC1pid58M5vZoqmQPGFCFqYbWmmlLBj37w9f+criQXn11R1NliS1DQZpSYtZuBDuuw8uvBD+9rf67Z06wVprZYF4xx0XvXmvbjS5Z8/86pYkqZIM0pL+57PP4Oab4eKL4d//zqaGu+QS+NKXsqC8xhrQsWPeVUqS1DpUJEhHxEbADcB7KaXDC9t+AewCBPD/UkpjKlGLpMVNngxXXQWXXw61tbD11nD77XDQQS5CIklScyr1K3IQcBlwAEBE7ApsmVLaPiLWBB6PiM1SSvMrVI8k4N134Te/geuvh1mzYK+94Mwzs95m51iWJGnJKnLLT0rpZmBig02DgTsLn/sQGA9sVIlaJMG4cXDYYdmy19dcA4ceCi+9BKNGwc47G6IlSSpGXn+07Q00uIWJSYVti4iIYcAwgL59+1amMqmNWrgQHn44u4FwzBhYYQU44ww4+WTo0yfv6iRJqj55TUL1KdCrwetewJTGO6WUhqeUBqaUBvbuvVjOllSEOXPghhtg881h773hrbfgoovg/ffhV78yREuS1FJ5BenRwH4AEbEKWVvHGznVIrVJU6dmQXnddWHo0Gy2jZtvhnfegdNPz0akJUlSy+XV2jEK2D0iniYL86eklGbnVIvUprz/Plx6KVx7LcyYAV/9anYz4e672/ssSVIpVSxIF6a3G1N4noCTK3VuqT345z+zlo3bb4eUspsJzzgDttoq78okSWqbnCFWqmIpwejR2Q2Ejz4Kyy8P3/0unHpqttqgJEkqH4O0VIXmzYM778wC9Isvwuqrwy9/CccfDyutlHd1kiS1DwZpqYrMmAEjRmTLdr/3Hmy8MVx3HRx1FHTtmnd1kiS1LwZpqQp89FG2fPfVV2ezcey4I1xxRTadXYe85t6RJKmdM0hLrdhrr2U3EN5yS9bOceCB2RLegwblXZkkSTJIS61MSvDkk1n/8wMPQLducNxxcNpp2ZLekiSpdTBIS63EggXwpz9lAfrvf4eaGvjJT+Ckk8CFPSVJan0M0lLOZs2CG2+E3/wG3n4b1lsPrroKjjkGunfPuzpJktQcg7SUk9pauPLK7DFpUtb3/KtfwQEHZMt5S5Kk1m2pQToiegDfBVYBLgTWSCm9WO7CpLbqrbey0ecbboDZs2HffbMbCHfYwSW8JUmqJsVMnHUTMAPYHpgCXFTWiqQ26pln4OCDYcMNs7mgjzwSXn0VRo7MprMzREuSVF2KCdIrppSuBGanlOYW+TWSgIUL64Pydttly3mfdRb85z/ZQioDBuRdoSRJaqlieqQ7R8RKABHRBehc3pKk6jd7djb388UXw+uvQ9++2WqExx0HPXvmXZ0kSSqFYoL0ecBfyXqknyy8ltSEKVOy1Qcvuww+/hi22gr+8IespaOz/wSVJKlNWWqQTik9GhHbAhsC76SUppS/LKm6jB+fjThfdx18+inssUd2A+Guu9r7LElSW9VskI6IG4DUxHZSSkPLWpVUJaZMgVNPhVtvzQLzkCFwxhnwhS/kXZkkSSq3JY1I3174eBTwNvAMsBOwQbmLkqrBiy/CQQfBe+/BKadkgXrttfOuSpIkVUqzQTql9AhARJySUjq6sPnhiHisIpVJrdgNN8CJJ2bLeI8dm83IIUmS2pdiprKrKSzKQkR0I7vpUGqXZs+GYcNg6FDYfnt4/nlDtCRJ7VUxs3b8GnguIl4BNgeuKG9JUuv07rvZ7BvPPw9nnw0/+5lLeUuS1J4VM2vH3RExmqw3+sOU0gflL0tqXR58EI46qn6BlX33zbsiSZKUt6W2dhSmvvsbcC3wWEQMKntVUiuxYAH8+Mew996wzjrw3HOGaEmSlCl2QZZdUkoTI2IN4FZg1/KWJeVv0iQ44gh47DH45jfhyithueXyrkqSJLUWxQTpSClNBEgpfRQRi80tLbU1zz4LhxwCn3wC116bLe3twiqSJKmhYmbtmBcRmwJExObA/PKWJOUnJbjqKthxx+xGwqeegm99yxAtSZIWV8yI9GnAiIhYBxgPHFfekqR8fPopHH883HIL7LUX/P73sPLKeVclSZJaq2Jm7XgV2A4gIjYB/l3uoqRKe/PNbJXCV17JprU75xzoUMzfayRJUrtVzKwdt0fE2hFxEDAcuKn8ZUmVc889MHAgfPQRPPww/OhHhmhJkrR0xcSF1VNK7wM7pZR2APqU6uQRMTUixjR49CjVsaWlmT8fzjwzG4keMCBbaGX33fOuSpIkVYtieqR7RsT3gJeW4WuK9WJKaecSHk8qysSJcNhhMHYsnHACXHIJdO2ad1WSJKmaFDMifTbQG7g+ItYH7i/h+TeNiLGFx2I3MUbEsIgYFxHjamtrS3hatWdPPglbbQX/+AfcfHM2S4chWpIkLatIqelpoSOiV0ppWkR0afy5lNLckpw8okNKaWFE1ACjgHNSSqOb2nfgwIFp3LhxpTit2qmUspHn//s/WHdduPtu2HzzvKuSJEmtTUQ8l1IauLT9ljQi/evCxzeA1wsf656XREppYeHjZOBuYItSHVtqaPp0OPRQOP102G+/bDTaEC1Jkj6PZvudU0rfKXzsHxEdgJqUUsn6KwrzUk9LKU2NiOWAfYHTS3V8qc4rr2Q3FL71Flx4YRamXWBFkiR9Xku9cbAw7d2lwKSI6Al8p7n2i2W0AnBjRHQEOgPXpZT+UYLjSv9z223ZyoQ9e8Lo0fCVr+RdkSRJaiuKmYHjHGDrlNInEbE2cBcw6POeOKX0ErDL5z2O1JS5c7OR5yuugB12gDvugDXWyLsqSZLUlhQza8e0lNInAIX5pKeWtyTp83n//Wzk+Yor4LTT4PHHDdGSJKn0ihmR/kdEfBd4ANgMmBgRG9Z9MqX0ZrmKk5bVn/8MQ4bA7Nlw551w8MF5VyRJktqqYoL0NoXHQQ22XVP4mIBdS12UtKwWLoTzz8+W9x4wIFv2e6ON8q5KkiS1ZUsN0imlXSKiM/VLhUutypQp8I1vwAMPZKPRw4dDDxeblyRJZbbUHunCrB0vAg9HxDoRcXH5y5KK88ILsPXW8MgjcPnlcOuthmhJklQZxdxseBqwNfBJSml84bmUu+uvh+22y2bo+Mtf4LvfdX5oSZJUOcUE6Tkppdlk/dBS7mbPhm9/G447Lpva7oUXskAtSZJUScUE6f9ExJlA94g4EphQ5pqkZr37Lnz5y3DddXDOOVlLR+/eeVclSZLao2KC9ElAd6AW2BQ4vqwVSc0YNQq++EV4+20YORLOOw86dsy7KkmS1F4VM2vHZ8BPK1CL1KQFC+Dcc7PgvOWWcPfdsO66eVclSZLau2LmkZZyM2kSHHEEPPYYfPObcOWVsNxyeVclSZJU3PR3Z1eiEKmxZ5/NWjnGjs16oq+/3hAtSZJaj2J6pL9S9iqkBlKCq66CHXfMeqCffjqboUOSJKk1Kaa1Y2pE3AE8DiwESCkNL2tVarc+/RS+851sYZW994abb4aVV867KkmSpMUVE6RfLXxcrZyFSG++CQceCK++Cj//OZx9NnQo5m8mkiRJOShm1o6fRsS2wFrAqGK+RlpW99wDxx4LXbpkc0PvtlveFUmSJC1ZMTcb/hI4HTgb6AHcXO6i1H7Mnw9nngkHHQQDBsDzzxuiJUlSdSjmD+c7pJQOAaallGqBXmWuSe3ERx/B4MFw0UVw4onZ7Bx9++ZdlSRJUnGKadOIiAggFV53KWM9aifGjoXDDoNp0+D3v4ejjsq7IkmSpGVTzIj0vcBIoE9EXAs8XN6S1JalBBdfDLvuCj17ZnNFG6IlSVI1KuZmw4sjYhywNfBySunR8peltmj6dBg6NFvi+8AD4YYbYIUV8q5KkiSpZZYapCNiE+D/gH7AvyLijZTS+HIXprbl5ZezGwrffjvriT7tNIjIuypJkqSWK6a14zrgcuDLwN3AiLJWpDbnjjtg0KCsH3r0aDj9dEO0JEmqfsUE6fkppYdTSlNTSncBRiAV7b77YMgQ2GoreOEF+IoLzkuSpDaimCD9TER8ASAiVgXeK29JaiueegoOPxy23jpbZGWNNfKuSJIkqXSKmf7uOOD7EfEB0BuYHREfko1Mp5TSmuUsUNXp1Vdh331h7bVh1ChYfvm8K5IkSSqtYmbtqKlEIWo7JkyAPfeErl2zkejevfOuSJIkqfSKae0om4j4bkT8LSKeiYjD8qxFpTFlCnztazB1Kjz4IPTvn3dFkiRJ5VFMa0dZRMR6wFBgW6Ar8PeIeDSlNCWvmvT5zJ4N++8Pb7wBDz2U3WAoSZLUVjUbpCNip+Y+l1IaW4Jz7wqMTCnNBeZGxFhge2BUgxqGAcMA+vbtW4JTqlwWLIAjj4Qnn4TbboPBg/OuSJIkqbyWNCJ9QuHj2kAN8E+y1Q3/Cwwqwbl7A5MavJ5U2PY/KaXhwHCAgQMHphKcU2WQEpx8MtxzD1xySTZThyRJUlvXbJBOKQ0BiIh7gK+mlGZHRHfgnhKd+1OgV4PXvQDbOqrQL38JV10FZ54Jp56adzWSJEmVUczNhjUppdkAKaVZQPcSnXs0sFdEdIyI5YCdgWdLdGxVyPXXww9/CEcdBRdckHc1kiRJlVPMzYavR8QVwKNkPczTSnHilNLLEfEA8DSQgN+klCaW4tiqjAcegGHDYPfdYcQI6JDrHDCSJEmVVUyQPgfYH9gDeB84qlQnTymdD5xfquOpcp55Bg49FLbcEu66C7p0ybsiSZKkyiomSI9MKW0PjCh3MaoOr78Oe+8Na66ZrVrYs2feFUmSJFVeMX+Mfy4iNi17JaoKH36YrVrYqVO2auFqq+VdkSRJUj6KCdLHAC9FxMSI+CgiPix3UWqdpk3LVi2cNClbtXC99fKuSJIkKT9Lbe1IKa1QiULUus2ZAwccAK++mrVzbL113hVJkiTla6lBOiJ6AbvRYNq7lNLN5SxKrcvChXD00TBmDPz+99ksHZIkSe1dMTcb3g98BHQFepBNf2eQbidSyhZZufNOuPDCbL5oSZIkFdcjvQA4HniSbAq8bmWtSK3Kr38Nl18O3/8+nH563tVIkiS1HsUuoTEV2CiltABYo4z1qBW56SY46yw4/HC46CKIyLsiSZKk1qOYIH0psAHwTkS8Dvy7vCWpNXjoITjuOBg8GG680VULJUmSGiumR/q/wNsppQsi4ncppanlLkr5+vvf4eCDYfPN4Z57oGvXvCuSJElqfYoZZzwU+EtE3AEcGBG2drRh//53tmrhaqtlo9IrOPmhJElSk4qZR/p7ABHRF7gEuIxs9g61MRMnwh57ZM8ffhhWXz3feiRJklqzYuaR3gMYAqwE3Ad8s9xFqfKmT4e99oKPP4YnnoANN8y7IkmSpNatmB7p3wKjgR+nlN4rcz3Kwdy5cOCB8K9/wf33w5e+lHdFkiRJrV8xrR0bR8RA4JSIWBsYnVK6pvylqRIWLoRjj4XRo7PZOb72tbwrkiRJqg5LvdkwIrqQLQ8+HVgR2KfcRalyzjwTbrsNzj8fjjkm72okSZKqRzGtHS8AjwOjgF+llGaXtyRVysUXw29+A9/7HvzgB3lXI0mSVF2WOiKdUtoUuJXCTB0R0bPcRan8br0VzjgDDjkELrnEVQslSZKWVTGzdvySbGXD9YCxwDXA18tcl8ro0Uezvuidd4abb4aOHfOuSJIkqfoUsyDLDimlQ4BpKaVPgF5lrkll9NxzcNBBsMkmcO+90K1b3hVJkiRVp2KCdEREAKnwuksZ61EZvfVWNivHyitnqxb28p9EkiRJLVZMkL4XGAn0iYhrgYfLW5LK4eOPYc89YcECeOQRWHPNvCuSJEmqbsXM2nEdMA4YCLycUnqkvCWp1GbMgL33hg8/zOaL3njjvCuSJEmqfsUE6YdSStsDfyl3MSq9uXPh4IPhxReznujttsu7IkmSpLahmNaO5yJi07JXopJbuBCGDs1m6Rg+HPZxKR1JkqSSKSZIHwP8KyImRsRHEfFhuYtSaZx1VjZf9HnnZYFakiRJpbPU1o6U0gqVKESldcklcOGFcOKJcPbZeVcjSZLU9hTTI10WETGm0aYfppT+mkctbc3tt8Npp8GBB8Jll7lqoSRJUjnkFqQBUko753n+tmj0aPjGN2CnnbK2DlctlCRJKo9ilgjvBnwNWAUIgJTS8BKce35EPAF0I1t6/McppTklOG679cIL8PWvw0YbwX33uWqhJElSORVzs+EoYG9gTWANYPUSnXv3lNIuwM7A8sAPG+8QEcMiYlxEjKutrS3Radumd9/NVi1cccVs1cIVV8y7IkmSpLatmNaOTimlb5X6xCmlhYWPcyLiJuBHTewzHBgOMHDgwNT488rU1sIee2RzRj/+OKy1Vt4VSZIktX3FjEi/EREblPKkEdEjItZvsOkwsvYOLaOZM7NVC99/H+6/HzbZJO+KJEmS2odiRqS/DBwZEa8AC4BUWOnw8wjgVxGxWuH5OOC3n/OY7c68eXDoofDcc3DPPfDlL+ddkSRJUvtRTJDeq9QnTSnNAA4q9XHbk5Tg29/O+qGvuQb23z/viiRJktqXYoL0BGAroHuDbePLU46Kdc45cNNNcO65MGxY3tVIkiS1P8UE6XuBlYCFQA0wERhczqK0ZJdfDuefnwXoH/8472okSZLap2JuNuwFfB34E7ANMK2sFWmJ7rwTTjkla+W48kpXLZQkScpLMUF6YUqpFlg3pTQL6FPmmtSMMWPgqKNgu+3gttugU67rUkqSJLVvxQTpWyNiADAnIh4BZpa5JjXhX//KRqHXWy+b5m655fKuSJIkqX1b6phmSunaiOhENj1dDfBm2avSIsaPhz33hJ7p6LnFAAARK0lEQVQ94eGHYeWV865IkiRJSx2RjoiDgH8CDwNTgJ+XuyjVmzw5W7Vw1qwsRPftm3dFkiRJguJaO04DtgY+SSmNLzxXBcyaBfvsA//5D4wcCZttlndFkiRJqlPM7WpzUkqzIyKVvRr9z/z5cNhh8OyzcNddsNNOeVckSZKkhooJ0v+JiDOB7hFxJNkCLSqjlOD44+GBB+Cqq+DAA/OuSJIkSY0V09pxEtmqhrXApsDxZa1I/OQnMGIE/PCHcMIJeVcjSZKkphQza8dnwE8rUIuAq6+Gn/8chg6Fn/0s72okSZLUnGaDdEQsAN4nWxr8f5uBlFJat9yFtUf33AMnnZTdYHjNNa5aKEmS1JotaUT6IuCrwCPAiJTS25UpqX168kk44ggYNAj++EdXLZQkSWrtmo1rKaUfRERHYC/gwohYHrgJuCulNLdSBbYHL78M++0H/fplqxZ27553RZIkSVqaJd5smFJakFK6P6V0IPAtYBjO2lFStbVZK8dyy2ULrqyySt4VSZIkqRhLbSCIiM3IQvS2wEhgSLmLai/mzoWDD4aJE7PWjn798q5IkiRJxVrSzYbfAY4ku+HwupTSqRWrqh1ICb73PRg7Fm65BbbZJu+KJEmStCyW1NpxFdATWBP4UUQ8XveoTGlt29VXw/Dh8IMfwJFH5l2NJEmSltWSWjuc4q5MnngCTjkF9t4bfvGLvKuRJElSSyxp1o7xlSykvXjnHTjkEFh/fbj1VujYMe+KJEmS1BLFLBGuEpkxA/bfHxYsgJEjoVevvCuSJElSS7nsR4UsXAhHHw2vvppNc7fBBnlXJEmSpM/DIF0hP/kJ3HcfXHop7LZb3tVIkiTp87K1owLuuAPOOw+GDoWTT867GkmSJJWCQbrMXngBjj0Wtt8erroKIvKuSJIkSaVgkC6jjz/Obi6sqYF77oGuXfOuSJIkSaVS9iAdEStFxLURMbHR9oMi4pmI+FtEtLlVE+fMgQMPhEmTst7o1VbLuyJJkiSVUiVGpLcGrm64ISJWAM4H9gB2BIZExMYVqKUiUoITT4Snn4YbboAvfjHviiRJklRqZZ+1I6X0Z4BYtDl4EPB0Smla4XP3AoOB18tdTyVcfjlcfz2ccw4cdlje1UiSJKkc8uqR7g1MavB6UmHbIiJiWESMi4hxtbW1FSvu8/jzn+G007Le6J/9LO9qJEmSVC55BelPgYbr+vUCpjTeKaU0PKU0MKU0sHfvxXJ2q/PWW3DoobDxxvD730MHb+WUJElqs/KKek8BX46I7hHRAdgXeDKnWkpi+nTYb79seruRI6Fnz7wrkiRJUjnlsrJhSmlSRPwK+AswH3ggpfR8HrWUwoIFcMQR8Oab8NhjsO66eVckSZKkcqtYkE4prd7o9U3ATZU6fzn98IcwahRceSXsskve1UiSJKkS7OL9nP7wB7jgAhg2DE44Ie9qJEmSVCkG6c9h3Dg47jjYccdsyjuX/5YkSWo/DNIt9NFHcMAB2YqFd98NXbrkXZEkSZIqKZebDavd7Nnw9a/DlCnZ6oVVMDOfJEmSSswgvYxSguOPh2efhbvugi22yLsiSZIk5cHWjmV0ySVw001w7rlw0EF5VyNJkqS8GKSXwcMPw5lnZgH6Rz/KuxpJkiTlySBdpDfegMMPh802gxtvdPlvSZKk9s44WISpU2H//aFzZ7jvPujRI++KJEmSlDdvNlyKBQtgyBB4+20YPRr69cu7IkmSJLUGBumlOOusrDf6mmtgp53yrkaSJEmtha0dS3DzzXDRRXDSSdkS4JIkSVIdg3QznnkGvv1t2GWXbMo7SZIkqSGDdBM++CBbubBPH7jzzuwmQ0mSJKkhe6Qb+eyzLETPnAmPPQY1NXlXJEmSpNbIIN1ASlk7xz/+Affem80ZLUmSJDXF1o4GLrwQbr0VzjsvmzdakiRJao5BumDUqGyqu8MOg7PPzrsaSZIktXYGaeC117JFV7bcEq6/HiLyrkiSJEmtXbsP0lOmwH77wXLLZct/d++ed0WSJEmqBu36ZsP587NWjvHjYcwYWHvtvCuSJElStWjXQfrMM7Mp7kaMgO23z7saSZIkVZN229px/fVw6aVwyikwdGje1UiSJKnatMsg/dRTcPzx8NWvwkUX5V2NJEmSqlG7C9LvvQcHHgjrrAN//CN0atfNLZIkSWqpdhUjZ82CAw7IlgEfMwZWXjnviiRJklSt2k2QTinrhX7xRbj/fhgwIO+KJEmSVM3K3toREStFxLURMbHR9tcjYkyDx/rlrOP887NWjgsugL33LueZJEmS1B5UYkR6a+BqYN9G2yemlHauwPm57z445xw48shsyjtJkiTp8yp7kE4p/RkgFl93e5WIGAN0Bu4HfpVSSqU+/8svw1FHwcCBcO21Lv8tSZKk0shz1o4vFEak9wAGAYvN5hwRwyJiXESMq62tXeYTTJ6cLf/dowfce2+2DLgkSZJUCrkF6ZTSwsLHmcAfgC2a2Gd4SmlgSmlg7969l+n48+bBIYfAhx9mIbpPn5KULUmSJAE5BemIWDUi1ig87wgcDIwt5TlOOw2eeAKGD4dBg0p5ZEmSJCm/6e+6ANdFxPKFGkallO4q1cGHD4crroAzzoBvfKNUR5UkSZLqVSxIp5RWb/B8ArBnOc4zdiycdBLsuWc21Z0kSZJUDm1qifDx4+Ggg2C99eC226Bjx7wrkiRJUlvVZoL0zJnZDB3z5mXzRq+4Yt4VSZIkqS1rE0uEL1wIxx6bzRn94IOw0UZ5VyRJkqS2rk0E6fPOg7vvhosvhj32yLsaSZIktQdV39pxzz3wk5/AMcfA97+fdzWSJElqL6o6SP/zn3D00bDttvC737n8tyRJkiqnaoN0bS3svz+stFI2Kt2tW94VSZIkqT2pyh7puXPh4IPh44/hySdhjTXyrkiSJEntTVUG6ZNPzhZeufVWGDgw72okSZLUHlVda8fVV8M118BZZ8ERR+RdjSRJktqrqgrSTzyRjUbvs0825Z0kSZKUl6oJ0nPmwCGHwAYbZC0dLv8tSZKkPFVNkH7rrWwFw5EjYYUV8q5GkiRJ7V3V3Gw4ezY88ACsv37elUiSJElVNCK90UYweHDeVUiSJEmZqgnSPXrkXYEkSZJUr2qCtCRJktSaGKQlSZKkFjBIS5IkSS1gkJYkSZJawCAtSZIktYBBWpIkSWoBg7QkSZLUAgZpSZIkqQUM0pIkSVILGKQlSZKkFoiUUt41FCUiZgBv5F1HO7MKMCnvItoZr3nlec0rz2teeV7zyvOaV14pr/k6KaXeS9upU4lOVglvpJQG5l1EexIR47zmleU1rzyveeV5zSvPa155XvPKy+Oa29ohSZIktYBBWpIkSWqBagrSw/MuoB3ymlee17zyvOaV5zWvPK955XnNK6/i17xqbjaUJEmSWpNqGpGWJEmSWo2qCNIR8d2I+FtEPBMRh+VdT1sSERtFxNMRcXuDbb8obPtbROxc2NY5IoZHxJMRMTYiNsut6CoWEctHxFUR8feI+EdE/LKw3WteJhGxYkTc0eBnyGmF7V7zMovMYxFxY+G117yMImJqRIxp8OjR3O/Ppv5baNlFxDoRMbpwLf8aEd38Pi+fiDil0ff4hxHxlTyveauf/i4i1gOGAtsCXYG/R8SjKaUp+VbWZgwCLgMOAIiIXYEtU0rbR8SawOOFb76jgfkppR0jYkuyPqTt8yq6iq0I/CGldGJEdABei4iX8ZqXU1fg3JTSqxHRieyaT8BrXgknAi8DK/mzpSJeTCntXPeiud+fwFY08d8ipTQ/j6KrVUR0BP4IfDOl9Frh9Vfw+7xsUkq/BX4LEBFdgKeBHuR4zathRHpXYGRKaW5KaQYwFr/5SialdDMwscGmwcCdhc99CIwHNipsv6Ow/UWgJiKWr2y11S+l9EFK6a+Fl8sDc4Gt8ZqXTUrp45TSq4WXvYH5ZP+A9JqXUUT0A/Yi+4c6+LOlEjYtjLyNjYjjaP73Z3P/LbRsvka2UNwvIuIp4AT8Pq+kbwB3k31P53bNqyFI92bRVWomFbapPJq73v53KKHCyMXNwJlAT7zmZRcRFwCvAL/Ba15WERFkAfpkoO6Odn+2lN9qKaWdgK8D3wZWxWteThsDA4BjgN2BbwLb4TUvu8Lv0OOBK8n5Z0s1BOlPgV4NXvcCbOson+aut/8dSiQiOgO3ALenlB7Ga14RKaWzgLXJRjE2wGteTscDj6SU3m6wze/zMkspLSx8nEw2UtcJr3k5LSAb8Z+RUvoU+DPQH695JRxO9jNmOjn/bKmGID0a2CsiOkbEcsDOwLP5ltSmjQb2A4iIVcj+PPJGo+0bkfUdTcuryGpV6Om6neyH7x8Lm73mZRTZDbV1oxCzgGnApXjNy2kbYKfIbmL+HVnf6BS85mVTuOltxcLz5YB9gcdp+vdncz9ztGz+CuxcuL6dgC8DI/D7vKwKf/E6heznOOT8O7TV32yYUno5Ih4gayhPwG9SShOX8mVquVHA7hHxNNk/tE5JKc2OiBHAiIh4EgiyPxtq2X2L7JdZTUR8p7DtdOBjr3nZzAeuiYheQHeyX34jgcFe8/JIKQ2te164g/5Y4OfAb73mZbMCcGPhT96dgetSSk829fszIpr8OZ9b5VUqpfSPiHgMGAfMIRsk+S1+n5fb14G/pZRqC69zzS0uyCJJkiS1QDW0dkiSJEmtjkFakiRJagGDtCRJktQCBmlJkiSpBQzSkiRJUgsYpCVJkqQWMEhLkiRJLWCQlqTPKSIejYgxEfF6RLxXeD6mwYqKS/v6oyNi/yV8fq2IuLS5zxd5ji0j4quF58tFxA2FxTtKJiIOabDQUOPP/TAidi3l+SQpby7IIkklEhHHAhunlM7Ku5bGyl1bRNQAdwO7pCZ+sUREF7Ilq7/qKnqS2gpHpCWpjCJiRkRcERHPRMQaEXFbRDwRES9ExLDCPudGxPGF51Mj4teFEe1nI2LNiOgXEc8UPn9jRFwZEY9FxEsRcURh+9CIeD4iHoqI30XE7Q1q+ApwFnBERDxU2DaxwblvKBzvlYjYJyJuj4h/RMTlhX16R8TdEfF4RDxVOF5j3wb+kFJKEbFV4evHRsT9EbFSSmku8ChwaLmutSRVmkFaksqrBzA8pbQt8F/gR8DuwPHAqU3s3wt4MKW0M/AAMKSJfboXjrEX8P8Ko8FnAzumlL4GTG64c0rpL8AFZEH3a00cb0VgD2AY8HvgDOBLwL4RsQJwIXBHSmlX4ADgt00c42tkI84AXwSeTCntBAwF5ha2Pw7s2cTXSlJV6pR3AZLUxtWmlP5VeL4W8HPgP8B4skDc2OSU0pjC8/eBjZrY5+5C+8T7EbEKsB7wckrp08LnRxe2FeuBlNLCiPigcJwJ8L9R65WAgcC6EXFCYf/uEdExpbSgwTFWBz4pPL8eOLQwov0yMLyw/RNgjWWoS5JaNYO0JJXXwgbPTwbuTyn9YQk33s1v9DqWsk+QBe5NImK5lNJnwPZNfM2SbohZ0MzzuuO/DFyZUnoSICL6NwrRABOBVYHpZCH+jpTSHyPi1kJ9DwKrAR8uoQ5Jqiq2dkhS5dwH/Dgi/kLWmlESKaWPyNotnoqIR8hGsRsH5+eBvSJiZEQs66jwacAZhd7uscAhTezzIFD3j4MdC7WMAWqAZwrbdwEeXsZzS1Kr5awdktTGRMRBwG4ppeMreM6VgXtY+qwdg1NKcypVlySVkyPSktQGFG44JCKCbGaM5yt5/pTSf4EryW5YbMr/AT82REtqSxyRlqQ2oHBj3xZAN7Ke5hMMrZJUXgZpSZIkqQVs7ZAkSZJawCAtSZIktYBBWpIkSWoBg7QkSZLUAgZpSZIkqQX+PyljFnnzaDLLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from sagemaker.analytics import TrainingJobAnalytics\n",
    "\n",
    "df = TrainingJobAnalytics(job_name, ['episode_reward_mean']).dataframe()\n",
    "num_metrics = len(df)\n",
    "if num_metrics == 0:\n",
    "    print(\"No algorithm metrics found in CloudWatch\")\n",
    "else:\n",
    "    plt = df.plot(x='timestamp', y='value', figsize=(12,5), legend=True, style='b-')\n",
    "    plt.set_ylabel('Mean reward per episode')\n",
    "    plt.set_xlabel('Training time (s)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}