This is a IPython Notebook on how to provision an IPython Cluster on the Joyent Public Cloud. It is a companion for this blog post: http://atl.me/2013/joyent-ipython-cluster . It may be viewed on the Notebook Viewer: http://nbviewer.ipython.org/5107045 .
{ | |
"metadata": { | |
"name": "launcher" | |
}, | |
"nbformat": 3, | |
"nbformat_minor": 0, | |
"worksheets": [ | |
{ | |
"cells": [ | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"This Notebook is an example session launching a sample IPython cluster in the Joyent Public Cloud.\n", | |
"\n", | |
"It is a part of a blog post on [automatic deployment of an IPython parallel compute cluster on Joyent](http://atl.me/2013/joyent-ipython-cluster), so contextual details are there.\n", | |
"\n", | |
"You should install the Python [SmartDC](http://pypi.python.org/pypi/smartdc) libraries first with:\n", | |
"\n", | |
" pip install smartdc" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"import time, sys\n", | |
"import json\n", | |
"from smartdc import DataCenter\n", | |
"import paramiko" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 1 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"Select your datacenter, select your username and key identifier:" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"sdc = DataCenter('eu-ams-1', key_id='/username/keys/keyname', verbose=True)" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 2 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"I chose the most up-to-date base 64-bit image and the smallest package available.\n", | |
"\n", | |
"The boot script is available on [this gist](https://gist.github.com/atl/5106787). " | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"ipc = sdc.create_machine(dataset='sdc:sdc:base64:', package='Extra Small 512 MB', \n", | |
" tags={'cluster': 'ipython', 'role': 'controller'},\n", | |
" boot_script='./startup-controller.sh', name='ipcontroller')\n", | |
"ipc.poll_until('running')" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:07:49.372832\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines\n", | |
"2013-03-07T10:07:58.617559\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:02.459808\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:05.942330\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:09.668697\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:13.213666\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:16.890286\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:20.323838\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:23.840497\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:27.553048\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:08:31.015873\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
} | |
], | |
"prompt_number": 3 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"This function is a roadblock, pausing further operation until the boot script completely finishes running, and we have some confidence that the controller\u2019s engine configurations have been written." | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"def wait_for_svc(connection, fmri, interval=3, timeout=315):\n", | |
" SERVICE_POLL = 'svcs -H -o STA,NSTA %s' % fmri\n", | |
" for _ in xrange(timeout//interval):\n", | |
" states = tuple(connection.exec_command(SERVICE_POLL)[1].read().strip().split())\n", | |
" if states == ('ON', '-'):\n", | |
" print >>sys.stderr\n", | |
" return True\n", | |
" elif states == ('MNT', '-'):\n", | |
" raise StandardError('Bootscript failed: now in maintenance mode')\n", | |
" elif states == ('OFF', 'ON'):\n", | |
" # heartbeat\n", | |
" print >>sys.stderr, '.',\n", | |
" else:\n", | |
" # slightly unusual state\n", | |
" print >>sys.stderr, '?',\n", | |
" time.sleep(interval)\n", | |
" raise StandardError('Timeout')\n" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 4 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"ssh_conn = paramiko.SSHClient()\n", | |
"ssh_conn.set_missing_host_key_policy(paramiko.AutoAddPolicy())\n", | |
"ssh_conn.connect(ipc.public_ips[0], username='root')" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 6 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"wait_for_svc(ssh_conn, 'svc:/smartdc/mdata:execute')" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 7, | |
"text": [ | |
"True" | |
] | |
} | |
], | |
"prompt_number": 7 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"The key command in this sequence is the following, where we retrieve the metadata that the engines need in order to connect to the engine." | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"_, rout, _ = ssh_conn.exec_command('cat /opt/local/share/ipython/profile_default/security/ipcontroller-engine.json')\n", | |
"ipcontroller = json.load(rout)" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 8 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"This is similar to the controller provisioning, except that it uses the engine boot script from the gist linked above, it defines a different role, and doesn't bother naming the engines." | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"for _ in xrange(4):\n", | |
" sdc.create_machine(dataset='sdc:sdc:base64:', \n", | |
" package='Extra Small 512 MB', \n", | |
" metadata={'ipython.url': ipcontroller['url'], \n", | |
" 'ipython.key': ipcontroller['exec_key']},\n", | |
" tags={'cluster': 'ipython', 'role': 'engine'}, \n", | |
" boot_script='./startup-engine.sh')" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:11:45.336342\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines\n", | |
"2013-03-07T10:12:13.032310\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:12:19.690627\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:12:26.242430\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
} | |
], | |
"prompt_number": 9 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"ipc.public_ips" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "pyout", | |
"prompt_number": 10, | |
"text": [ | |
"[u'37.153.99.232']" | |
] | |
} | |
], | |
"prompt_number": 10 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"ssh_conn.close()" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 11 | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"At this point, you could SSH into the IPython Controller node, and test your cluster. Note that the `IPYTHONDIR` environment variable should be set to `/opt/local/share/ipython`." | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"---\n", | |
"\n", | |
"Once you have finished with the cluster, you will want to shut it down at some point. Here is a convenient way to delete the entire cluster:" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"cluster = sdc.machines(tags={'cluster': 'ipython'})\n", | |
"cluster" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:12:33.045438\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 12, | |
"text": [ | |
"[<smartdc.machine.Machine: <ipcontroller> in <DataCenter: eu-ams-1>>,\n", | |
" <smartdc.machine.Machine: <4eac1ad> in <DataCenter: eu-ams-1>>,\n", | |
" <smartdc.machine.Machine: <d8f2ae6> in <DataCenter: eu-ams-1>>,\n", | |
" <smartdc.machine.Machine: <19666c2> in <DataCenter: eu-ams-1>>,\n", | |
" <smartdc.machine.Machine: <873fde1> in <DataCenter: eu-ams-1>>]" | |
] | |
} | |
], | |
"prompt_number": 12 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"from operator import methodcaller" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [], | |
"prompt_number": 13 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"map(methodcaller('stop'), cluster)" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:19:18.615269\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7\n", | |
"2013-03-07T10:19:22.007255\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c767e541-25a9-48c7-af2c-292a78489d7d" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:23.819118\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines/89b08e35-a7c5-44fc-90be-7056e802bdde" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:25.664591\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines/193f2748-7e7c-483d-8dde-3ffc73b1ed03" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:27.198584\tPOST\thttps://eu-ams-1.api.joyentcloud.com/my/machines/cc8addf7-baeb-4913-8f4b-5a0ba3aa4f76" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 14, | |
"text": [ | |
"[None, None, None, None, None]" | |
] | |
} | |
], | |
"prompt_number": 14 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"map(methodcaller('poll_until', 'stopped'), cluster)" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:19:28.741289\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7\n", | |
"2013-03-07T10:19:32.274923\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:35.843260\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:37.647634\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c767e541-25a9-48c7-af2c-292a78489d7d" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:39.183008\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/89b08e35-a7c5-44fc-90be-7056e802bdde" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:40.722421\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/193f2748-7e7c-483d-8dde-3ffc73b1ed03" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:42.255855\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines/cc8addf7-baeb-4913-8f4b-5a0ba3aa4f76" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 15, | |
"text": [ | |
"[None, None, None, None, None]" | |
] | |
} | |
], | |
"prompt_number": 15 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"map(methodcaller('delete'), cluster)" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:19:43.798491\tDELETE\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c8be2243-ae5d-413d-a568-d73ac9e07ff7\n", | |
"2013-03-07T10:19:45.330310\tDELETE\thttps://eu-ams-1.api.joyentcloud.com/my/machines/c767e541-25a9-48c7-af2c-292a78489d7d" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:46.698550\tDELETE\thttps://eu-ams-1.api.joyentcloud.com/my/machines/89b08e35-a7c5-44fc-90be-7056e802bdde" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:48.107172\tDELETE\thttps://eu-ams-1.api.joyentcloud.com/my/machines/193f2748-7e7c-483d-8dde-3ffc73b1ed03" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n", | |
"2013-03-07T10:19:49.630606\tDELETE\thttps://eu-ams-1.api.joyentcloud.com/my/machines/cc8addf7-baeb-4913-8f4b-5a0ba3aa4f76" | |
] | |
}, | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 16, | |
"text": [ | |
"[None, None, None, None, None]" | |
] | |
} | |
], | |
"prompt_number": 16 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [ | |
"sdc.machines()" | |
], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"stream": "stderr", | |
"text": [ | |
"2013-03-07T10:20:15.112498\tGET\thttps://eu-ams-1.api.joyentcloud.com/my/machines\n" | |
] | |
}, | |
{ | |
"output_type": "pyout", | |
"prompt_number": 18, | |
"text": [ | |
"[]" | |
] | |
} | |
], | |
"prompt_number": 18 | |
}, | |
{ | |
"cell_type": "code", | |
"collapsed": false, | |
"input": [], | |
"language": "python", | |
"metadata": {}, | |
"outputs": [] | |
} | |
], | |
"metadata": {} | |
} | |
] | |
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment