NEWT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NEWT Python Tutorial\n",
    "\n",
    "In the tutorial we go over some simple examples using python to communicate with the NEWT API to access NERSC. More details on NEWT can be found at http://newt.nersc.gov\n",
    "\n",
    "We use the `requests` python module (See http://www.python-requests.org) for this tutorial since it really simplifies interactions over HTTP, and allows you to hold on to authenticated sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from requests import Session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize a session object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s = Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we're going to do the simplest possible call to NEWT. This is the \"Hello World Example\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = s.get(\"https://newt.nersc.gov/newt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the content of the response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"status\": \"OK\", \"output\": {\"text\": \"Welcome to NEWT\", \"version\": \"0.3.2\"}, \"error\": \"\"}'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we check our current authentication status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = s.get(\"https://newt.nersc.gov/newt/auth\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"username\": null, \"session_lifetime\": 0, \"auth\": false, \"newt_sessionid\": null}'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we authenticate. Just provide a dict with of the form `{\"username\": \"myusername, \"password\": \"mypassword\"}`. Here mypassword will be your NERSC password and your MFA string combined. `POST` it to the https://newt.nersc.gov/newt/auth URL "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "password = 'XXXXXX' + 'YYYYYY' # XXXXXX is NERSC passwd, YYYYYY is NERSC MFA\n",
    "r = s.post(\"https://newt.nersc.gov/newt/auth\", {\"username\": \"shreyas\", \"password\": password})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check our status code and the content to make sure it succeeded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"username\": \"shreyas\", \"session_lifetime\": 43198, \"auth\": true, \"newt_sessionid\": \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"}'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a new batch job on cori. Post the path for the SLURM job submission file on the target system. This should be a `post` request to https://newt.nersc.gov/newt/queue/cori/ using a dict of the form `{\"jobfile\": \"/path/to/file/on/cori\"}`. You can also post a `\"jobscript\"` (instead of a `\"jobfile\"`) containing the raw contents of the script. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = s.post(\"https://newt.nersc.gov/newt/queue/cori/\", {\"jobfile\": \"/project/projectdirs/osp/newt.sbatch\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The JOB id is in the resulting `\"jobid\"` field. Note that the requests object has a `json()` method that will automatically convert json output into python objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"status\": \"OK\", \"error\": \"\", \"jobid\": \"1250312\"}'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "jobid = r.json()['jobid']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use this jobid to query the queue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = s.get(\"https://newt.nersc.gov/newt/queue/cori/%s\" % jobid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"status\": \"PD\", \"repo\": \"mpccc\", \"rank_bf\": \"N/A\", \"qos\": \"normal\", \"name\": \"newt.sbatch\", \"timeuse\": \"0:00\", \"hostname\": \"cori\", \"jobid\": \"1250312\", \"queue\": \"regular\", \"submittime\": \"2016-03-02T10:50:04\", \"reason\": \"Priority\", \"user\": \"shreyas\", \"memory\": \"0\", \"nodes\": \"1\", \"timereq\": \"1:00\", \"procs\": \"1\", \"rank_p\": \"5607\"}'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get everything for user `shreyas`. Note you can query for any field in the job object with `?key=value`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "r = s.get(\"https://newt.nersc.gov/newt/queue/cori/?user=shreyas\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "j = r.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample shreyas job (the first job in the returned list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'hostname': u'cori',\n",
       " u'jobid': u'1250267',\n",
       " u'memory': u'0',\n",
       " u'name': u'newt.sbatch',\n",
       " u'nodes': u'1',\n",
       " u'procs': u'1',\n",
       " u'qos': u'normal',\n",
       " u'queue': u'regular',\n",
       " u'rank_bf': u'N/A',\n",
       " u'rank_p': u'5580',\n",
       " u'reason': u'Priority',\n",
       " u'repo': u'mpccc',\n",
       " u'status': u'PD',\n",
       " u'submittime': u'2016-03-02T10:38:01',\n",
       " u'timereq': u'1:00',\n",
       " u'timeuse': u'0:00',\n",
       " u'user': u'shreyas'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "j[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}