Skip to content

Instantly share code, notes, and snippets.

@sananand007

sananand007/Run_Jupyter_Notebook_Conda_aws.md

Last active March 4, 2018 15:32
Show Gist options
  • Save sananand007/a074f63490c395133ade8cc53b796c1a to your computer and use it in GitHub Desktop.
Save sananand007/a074f63490c395133ade8cc53b796c1a to your computer and use it in GitHub Desktop.
Download ZIP
Steps to Run a Jupyter Notebook using AWS Putty Console and Anaconda env with Windows and Chrome
Raw
Run_Jupyter_Notebook_Conda_aws.md

How to Login into the Jupyter Notebook through AWS and Conda Environment created in AWS :

Installing Anaconda on Ubuntu :

    - https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
       cd /tmp
    - curl -O https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
    - check the sha256sum: sha256sum Anaconda3-5.1.0-Linux-x86_64.sh
    - bash Anaconda3-5.1.0-Linux-x86_64.sh
    - Press "Enter for All of it till installation is complete"
    - source ~/.bashrc [to activate the installation]
    - conda list [this should work]
    
     Creating Environments : [Basically install all that is required , custom, all greatest and latest can be loaded !!]
    
    - conda create --name tensorflow_p364 python=3.6.4
    - conda info --envs
    - source activate tensorflow_p364
    - Installing requirements:
     - conda install tensorflow
     - conda install scipy
     - conda install matplotlib
     - conda install tqdm

Configure Jupyter Notebook on Your EC2 Instance

  • Connect to the EC2 instance and then complete the following procedure
  • To configure the Jupyter notebook server on your EC2 instance, you create a configuration file. In the configuration file, you set some of the values to use for web authentication, including the SSL certificate file path, and a password.
    To configure the Jupyter notebook server on your EC2 instance, you create a configuration file. In the configuration file, you set some of the values to use for web authentication, including the SSL certificate file path, and a password.

    Connect to the EC2 instance and then complete the following procedure. If you have set up a cluster of EC2 instances, connect to the master node.

    1. Configure the Jupyter server

    Create an SSL certificate.

    $ cd
    $ mkdir ssl
    $ cd ssl
    $ sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout "cert.key" -out "cert.pem" -batch
    Create a password. You use this password to log in to the Jupyter notebook server from your client so you can access notebooks on the server.

    2. Open the iPython terminal.

    $ ipython
    At the iPython prompt, run the passwd()command to set the password.

    iPythonPrompt> from IPython.lib import passwd 
    iPythonPrompt> passwd() 
    You get the password hash (For example, sha1:examplefc216:3a35a98ed...).

    Record the password hash.

    Exit the iPython terminal.

    $ exit
    
    3. Create a Jupyter configuration file.

    $ jupyter notebook --generate-config 
    The command creates a configuration file (jupyter_notebook_config.py) in the ~/.jupyter directory.

    4. Update the configuration file to store your password and SSL certificate information.

    Open the .config file.

    $ vi ~/.jupyter/jupyter_notebook_config.py
    Paste the following text at the end of the file. You will need to provide your password hash.

    c = get_config()  # Get the config object.
    c.NotebookApp.certfile = u'/home/ubuntu/ssl/cert.pem' # path to the certificate we generated
    c.NotebookApp.keyfile = u'/home/ubuntu/ssl/cert.key' # path to the certificate key we generated
    c.IPKernelApp.pylab = 'inline'  # in-line figure when using Matplotlib
    c.NotebookApp.ip = '*'  # Serve notebooks locally.
    c.NotebookApp.open_browser = False  # Do not open a browser window by default when using notebooks.
    c.NotebookApp.password = 'sha1:fc216:3a35a98ed980b9...'  
    This completes Jupyter server configuration.

    - Go to the directory where you want to open
    - mkdir ssl
    - cd ssl
    - sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout "cert.key" -out "cert.pem" -batch
    - ls ssl [you have created two files]
    - open the IPython terminal
      ipython
      from IPython.lib import passwd
      passwd
      ## set the passwd - #e#er#t#on#7 [Out[3]: 'sha1:ae72cbbf933b:006e6f9b36315f7a1718f03593e665f7e8283ab1']
      ## Get the sha1
      exit #exit the terminal

Configure a Windows Client (that is what I have !!)

To connect your Windows client to the Jupyter server, do the following:

  • Configure proxy settings

  • Configure your Internet browser to use an add-on, such as FoxyProxy, to manage your Socket Secure (SOCKS) proxy settings. The proxy management add-on enables you to limit the proxy settings to domains that match the form of the public DNS name of your EC2 instance. The add-on automatically handles turning the proxy on and off when you switch between viewing websites hosted on the EC2 instance and those on the Internet.

  • Set up an SSH tunnel to your EC2 instance

  • This is also known as port forwarding. If you create the tunnel using dynamic port forwarding, all traffic that is routed to a specified unused local port is forwarded to the Jupyter server on the EC2 instance. This creates a SOCKS proxy.

Topics

Step 1: Prepare
Step 2: Configure Browser Proxy Settings
Step 3: Set Up an SSH Tunnel to Your EC2 Instance on the Client

Step1:Prepare [Configure proxy settings for Chrome]

Download and install the Standard version of FoxyProxy from http://getfoxyproxy.org/downloads.html.

Using a text editor, create a file named foxyproxy-settings.xml. Save the following to the file:

<?xml version="1.0" encoding="UTF-8"?>
<foxyproxy>
   <proxies>
      <proxy name="ec2-socks-proxy" id="2322596116" notes="" fromSubscription="false" enabled="true" mode="manual" selectedTabIndex="2" lastresort="false" animatedIcons="true" includeInCycle="true" color="#0055E5" proxyDNS="true" noInternalIPs="false" autoconfMode="pac" clearCacheBeforeUse="false" disableCache="false" clearCookiesBeforeUse="false" rejectCookies="false">
         <matches>
            <match enabled="true" name="*ec2*.amazonaws.com*" pattern="*ec2*.amazonaws.com*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*ec2*.compute*" pattern="*ec2*.compute*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" />
            <match enabled="true" name="*.compute.internal*" pattern="*.compute.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false"/>
            <match enabled="true" name="*.ec2.internal* " pattern="*.ec2.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false"/>
	  </matches>
         <manualconf host="localhost" port="8157" socksversion="5" isSocks="true" username="" password="" domain="" />
      </proxy>
   </proxies>
</foxyproxy>
You use these settings file in the next step when configuring FoxyProxy. The settings do the following:

Port 8157 is the local port number used to establish the SSH tunnel with the Jupyter server. This must match the port number you used in PuTTY or in the terminal.

The *ec2*.amazonaws.com* pattern matches the public DNS name of clusters in AWS US Regions.

The *ec2*.compute* pattern matches the public DNS name of clusters in all other AWS Regions.

Configure FoxyProxy.

Choose Customize and Control Google Chrome, choose Tools (or More Tools), and then choose Extensions.

Next to FoxyProxy Standard, choose Options.

Choose Extensions, choose Options, and then do the following.

Import the settings file.

On the Import/Export page, choose Choose File, and then open the foxyproxy-settings.xml file that you saved in the preceding step. If you are prompted to overwrite the settings, do so.

Choose Proxy mode, and then choose Use proxies based on their predefined patterns and priorities.

If the installation and configuration is successful, you see the FoxyProxy icon next to the URL address bar in the browser.

Go through and complete all the steps present at :

After the steps:

The jupyter notebook is running at : https://ec2-54-187-16-50.us-west-2.compute.amazonaws.com:8888/

  • Go past the steps of putty as below

Set up an SSH tunnel using dynamic port forwarding on Windows

  • Start PuTTY by double-clicking putty.exe.

  • In the Categories section, choose Sessions and for Host Name, type ubuntu@MasterPublicDNS. For example:

  • ubuntu@ec2-##-###-###-###.compute-1.amazonaws.com

  • In the Categories section, choose Connection. Choose SSH, and then choose Auth.

  • For Private key file for authentication, choose Browse, and then choose the .ppk file.

  • In the Categories section, choose Connection-. Choose SSH, and then choose Tunnels. Configure the tunnel.

    • For Source port, type 8157 (an unused local port).

    • Choose Dynamic and Auto.

    • Choose Add, and then choose Open.

  • This opens the tunnel.

Create a directory on the EC2 instance for storing Jupyter notebooks. This is your Jupyter workspace.

$ mkdir ~/mynotebooks
$ cd ~/mynotebooks
`
+ Start the Jupyter notebook server.
$ jupyter notebook

  • By default, the server runs on port 8888. If the port is not available, it uses the next available port. The Jupyter terminal shows the port on which the server is listening.

  • once you Type the about url on chrome, get past the warning by going to the "Advanced section"

  • Type the Password you created when configuring the Jupyter server

Test the server connection

  • If the connection is successful, you see the Jupyter notebook server home page.
  • Type the password that you created when you configured the Jupyter server.
  • Now you have access to the Jupyter notebook server that is running on the remote EC2 instance.
  • You can create new notebooks and write Apache MXNet code in Python.

Install tqdm forced

  • sudo /home/ubuntu/anaconda3/bin/python3.6 -m pip install tqdm

Force install conda env

  • conda create -n tensoflow_v1p1p0 python=3.6.4 tensorflow=1.1.0 matplotlib=2.0.0 scipy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment