Create an iPython HTML Notebook on Amazon's AWS Free Tier from scratch.

roll_ipython_in_aws.md

What

Roll your own iPython Notebook server with Amazon Web Services (EC2) using their Free Tier.

What are we using? What do you need?

• An active AWS account. First time sign-ups are eligible for the free tier for a year
• One Micro Tier EC2 Instance
• With AWS we will use the stock Ubuntu Server AMI and customize it.
• Anaconda for Python.
• Coffee/Beer/Time

Why?

I had been looking for complete tutorials on setting up my own iPython Notebook on EC2 from scratch and I couldn't quite find what I was looking for. I hope someone finds this useful!

Why Anaconda over Enthought?

I've used the Enthought Free distribution on my own system for a while but it was missing some packages that I loved. For future uses I wanted a distribution that I didn't have to modify out of the box. I looked around and stumbled across Continuum's package. I liked where they were taking their products. Not to mention their free distribution had many, many awesome packages already included. Why not?

How

AWS Console and EC2 Instance Launch.

• Head over to the AWS Console and log in.
• We're going to Launch an Instance. If this is your first time at the Rodeo you won't have any instances listed.

• We're going to use the Classic Wizard for the Instance Creation

• There are many OS's to choose from. All the eligble OS's eligible for the Free Tier are marked by big, bold stars. We're going to use Ubuntu Server 12.04.1 LTS.

• The free tier is only eligible for the T1 Micro. I will use all defaults here.

• Again in the Advanced Instaces options I leave all options at their default settings.

• You can create tags for machines. I haven't found the need to yet. Defaults!

• The Wizard is going to ask you to create a public key (or use an existing one if this isn't your first time). Create and download that key and save it in a nifty (read: memorable) spot.

• The Security Group step is a very important step. This step can only be configured when you launch the instance! Under the Create New Security Group.

• You should have rules for SSH(22): 0.0.0.0/0, HTTPS(443): 0.0.0.0/0, and 8888: 0.0.0.0/0. Where 0.0.0.0/0 means access from ANY outside I.P. address. We will use port 8888 for our iPython Notebook server. Your rules should look like the right hand size of the photo below.

• This is the last page of the Wizard! Review the options you set and if they look good the you're all set.

SSH-ing into your Instance

• I'm going to assume you're using the Terminal in Mac OS X (or something vaguely Unix-y).
• Right-Click on the instance you just created from the Instances panel then click Connect

• The Connect diaglog box will give you some code you can use to log in to your instance. In your terminal Navigate to where you saved your PEM key and run the command from there. WATCH OUT #1: If you haven't changed the permissions on your key then the server is going to reject the connection. CHMOD 400 (your PEM). WATCH OUT #2: Ubuntu wants you to login as ubuntu rather than root. Change that.
• Here's an example of my terminal ouput with both of those "gotcha's" happening.

• If all goes well then you're logged in!

Setting up your Python Environment

• Now it's time to set up your python environment with Anaconda. Use wget to download the Anaconda installer to your server

• Run the Bash script. I've will leave everything with its default setting. Anaconda will install everthing in an folder in your home directory.

• After a few minutes the install will finish and tell you to put the folder that was just created at the top of your $PATH. Modify your .bashrc.

• Make sure to reload your .bashrc with the source command. You can double check which Python you are using by calling: $ which python

• Say whaaaat? Now you've got a pretty good base to do all your Scientific Computing in Python.

Setting up iPython Notebook

• We're almost there. I can feel it. We've got a few more steps: Create a iPython Profile for our server; Create a password for our notebook log in; Create a self-signed SSL certificate for HTTPS access; and finally modify our ipython_notebook_configuration.py file. Much of (if not all) of this code follows the iPython Notebook doc very closely.

• From anywhere type: $ ipython profile create nbserver

• Next, we're going to create a password for your notebook server. I'm going to do everything from within iPython right now. You can access the shell commands by prepending your commands with "!". Some commands like "cd" and "ls" don't need an "!" in front. It's pretty awesome. See "Magic Functions" in the resources section.

• The output of passwd() is going to be used in the notebook configuration file later. So save/remember it!

• Up next we're going to create a directory in our home folder called "certificates". In this folder we're going to save a self signed SSL certificate. Complete the questions as best you can.

• We're going to need the name of the certifcate and the absoulte path for the notebook configuration file.

• Last, but not least, it's time to modify our ipython_notebook_config.py file.
• Cruise over to ~/.ipython/profile_nbserver/ (or whatever you decided to call your profile) and open up the ipython_notebook_config.py file.

• You can either go through the config file and uncomment each of these one by one, or just use the iPython HTML Notebook docs and copy the code to the top of the file like I did. You'll need to change "certfile" and "password" to the values you just got from the previous steps. Change the "port" to 8888.
• Save!

Launching you iPython Notebook

• From anywhere run the following: $ ipython notebook inline --profile=nbserver

• If you're successful you should have output like the above! WATCH OUT: make sure your server is running at "all ip addresses on your system" rather than just 127.0.0.1 (or something like that). You WILL have a bad time.

Logging in to your Notebook from the Browser

• In order to log in via the browser we need the Public DNS for our sever. Cruise over to your AWS Console and select your instance from the Instances page. Under description you should find this:

• Using your public DNS go to your fav browser and type: https://your-Instance's-public-DNS:8888

• If successfull you'll get a warning about the self signed certificate. It's ok! Click Continue.

• You're in! Enter your password, create a notebook, and start doing...stuff!

What next?

A few things:

• Why not associate an Elastic IP to your instance(s)? Now you've got a static ip no matter what kind of instance you start.
• Use your Elastic IP and set up forwarding from a domain that you own. Instead of having a random IP address (albeit static) you can access your notebook via a memorable domain name. E.g. https://theres_always_money_in_the_banana_stand.com/notebook
• Create a Snapshot of your AMI. Now you can use this AMI on ANY instance type you like.
• Create an EBS volume to store all your data/work. Use your newly created Python AMI and run a massive instance for a few hours. Your setup Python setup will be the same but your hardware will be pretty beefy.
• Dive in to the .ipython folder in your home directory (or where you installed it) and modify your ipython_configuration.py. There are lots of options for configuring what packages are imported and what they are called at startup.

Resources and Further Reading

1. iPython HTML Notebok Documentation
2. iPython on EC2 Screen Cast
3. EC2 Screencast on YouTube
4. Contiuum.io Anaconda Download
5. Anaconda Package Index
6. iPython Tutorial -> Magic Functions

@AlanZZhao:

Off the top of my head you could use Route53 for your DNS with a domain like alanzhao.com or even just a subdomain like compute.alanzhao.com.

Have that domain (or subdomain!) point to an elastic IP.

Create a script for starting up your compute cloud with the AWS SDK in your favorite language. This script should: create a new server instance and associate it with your elastic ip. Now your domain points to your server! Your shutdown script does the reverse: deallocate a server.

Cons: you have to keep your elastic ip around and AWS charges.

You could allocate an elastic ip in your startup script, but that's a call you'll have to make depending on how fast Route53 propagates that information.

Instead of the Ubuntu image, use http://docs.continuum.io/anaconda/images.html (page also includes info on Azure and Docker) and skip the Python install/setup step. Caveat: I don't know what is included/configured in that image, so all of the ipython stuff might still apply.

Also, one should probably:

• Run:
  • Update (conda update foo):
  • conda - Highly recommended to get the latest Anaconda package manager. Also will probably update to latest Python / openssl / maybe other important stuff.
  • ipython - New version 3.0 has some cool new features (also requires jsonschema package).
  • sympy - New version 0.7.6 has cool new features and many bug fixes.
  • numpy / scipy / matplotlib
  • Install (conda install foo):
    • terminado - A nice addition to ipython 3.0 which will run a server-side terminal session in a browser tab
    • Whatever else one might want.

I highly recommend updating ipython to 3.0. In addition to lots of bug fixes, ipython is now powered by the Jupyter backend, which allows many different frontends (Oh, my!).

@imatypeofwalrus: I find byobu much easier to use than screen: do sudo apt-get install byobu, then byobu gives you a terminal that can stay alive after you're gone. Exit byobu with F6, get byobu help with F9, F2 creates another byobu terminal tab, F3 and F4 swich byobu tabs. (don't exit by logging out of the terminal - you want it to stay alive!). You can run ipython in a tab and leave it going when your'e gone :)

I just want to say how fantastically useful this was, super clear

This was absolutely wonderful. Thanks!

One question I have is how to create multiple notebook instances on the same AWS server but accessed on different ports. I've tried multiple ways to get this to work, but nothing seems to do the trick (and I can't find an appropriate source online). Any suggestions? Thanks!

Edit: I figured it out. You have to add a new inbound rule in the security settings for the other ports.

Just wanna thank you very much!

Just stumbled on this, looks great :)
I tried doing something similar with a digitalocean vm http://mseth.me/you-own-private-anaconda-server/

#Configuration file for ipython-notebook.

c = get_config()

#ipython Kernel Configs
c.IPKernelApp.pylab = 'inline'
c.NotebookApp.certfile = '~/.ssh/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:bcd259ccf... your hashed password here'
# It's a good idea to put it on a known, fixed port
c.NotebookApp.port = 8888

for Jupyter AKA ipython 4.0 you need to use
jupyter-notebook --config='~/.ipython/profile_nbserver/ipython_notebook_config.py'

When I type: ipython notebook inline --profile=nbserver
I got the warning
" Unrecognized alias: '--profile=nbserver', it will probably have no effect. "

I need to use
ipython notebook ./ inline --config="~/.ipython/profile_nbserver/ipython_config.py"

and it seems to work

@halmingzhao
completed path in config worked for me as well.
Thanks !

Fails on a couple of fronts:

1. in my nbserver directory there is ipython_kernel_config.py and ipython_config.py but no ipython_notebook_config.py

I created the ipython_noetbook_config.py and put the stuff that you instructed in.

I then tried:
python notebook inline --profile=nbserver
and got.
[W 21:57:28.352 NotebookApp] Unrecognized alias: '--profile=nbserver', it will probably have no effect.
[C 21:57:28.353 NotebookApp] No such file or directory: /home/ubuntu/.ipython/profile_nbserver/inline
ubuntu@ip-172-31-61-235:~/.ipython/profile_nbserver$

Thanks! this guide was super helpful. I was total beginner to UNIX too.
For others who are new to UNIX like me

1. To edit config files you can use vi file.py
   in vi, you can toggle between editing mode/command mode by ESC. you can start editing by i
   to exit without save :q! enter to exit with save :x
2. How to get full path of file in command line
   http://stackoverflow.com/questions/5265702/how-to-get-full-path-of-a-file
3. missing ipython_notebook_config.py
   i had this issue, but ended up modifying the jupyter config file instead.
   I guess this doesn't really utilize profiles, but i didn't need in my case.
   http://stackoverflow.com/questions/31962862/ipython-ipython-notebook-config-py-missing

Hi,

After i have tried to launch the ipython notebook, the webpage is not simply not available at "http://ec2-54-169-4-157.ap-southeast-1.compute.amazonaws.com:8000/"

Anyone know how tot solve this problem?

[ec2-user@ip-172-31-31-24 ~]$ ipython notebook --config="/home/ec2-user/.ipython/profile_nbserver/ipython_config.py"
[I 07:25:13.993 NotebookApp] Serving notebooks from local directory: /home/ec2-user
[I 07:25:13.994 NotebookApp] 0 active kernels
[I 07:25:13.994 NotebookApp] The IPython Notebook is running at: https://[all ip addresses on your system]:8000/
[I 07:25:13.994 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

I got all the way to the end but got the same problem as Sushiyan above.
Helps?

@jonschull: Try changing http to https. Also make sure the inbound rules on your instance are set properly to allow connections to the particular port on which your server is running.

Thanks for posting your notes, it was quite handy !

/Ram

If you find that the configuration file ipython_notebook_config.py is missing, here's why:

Things have changed in jupyter since "the big switch", where notebook configurations have migrated to the .jupyter folder....

if your notebook config is missing (which was my case when starting ipython to create a profile), you can generate a jupyter config with the following:

jupyter notebook --generate-config

this should create a ~/.jupyter/jupyter_notebook_config.py file with relevant notebook settings ready to be uncommented/modified such as ports and passwords.

Hat tip: http://stackoverflow.com/a/33702208/3345375

Note: This advice is a little different from what townie mentions above. It worked for me, though.

@jgharris7 @carolingfish

I had the same issue as @haimingzhao and @amolsharma99....

Unrecognized alias: '--profile=nbserver', it will probably have no effect.

This is because Jupyter Notebook no longer supports profiles, only config.

See the comments by marbre and Carreau here: ipython/ipython#8769

Hi,
Thanks for a great article. Unfortunately after going through all the steps successfully, I am unable to create a new notebook. It returns me an error :Permission denied: Untitled.ipynb

Any suggestion?

Thanks,
K.Vikraman

@Vikramank If you don't have the right permissions for a certain file or directory, you may need to change its permissions or its owner. To do this on the command line, use chmod or chown respectively.

Or sometimes you can override permissions. Just precede a command with the word sudo and that'll get things done.

Great article !!! also thanks to Townie for updated line to configure notebook.

Awesome article!

Very nice and informative, preparing for AWS certification, recently joined Kerneltraining.com for amazon aws tutorial, can any one suggest me some books and videos on this.

Thank you! Could you fix the problems of the images? I think they are all not found.

Exactly what i was looking for. How feasible is it for you to make a video tutorial on youtube for it?