julia_aws.Rmd

# Julia and R on Amazon EC2
#### This is basically forked from: https://github.com/magerton/ec2-for-R-and-julia

## Purpose

This guide explains how to set up a simple Amazon Linux 2 instance on EC2 for use with Julia + GPU.

The guide assumes basic familiarity with a UNIX-like systems (e.g., navigating the file structure, copying, moving, ssh, etc). Note that Windows 10 now includes the "Windows Subsytem for Linux" (WSL), which provides a very nice terminal environment ([MSDN setup guide](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide)). The Git Bash terminal is also a good choice for Windows. macOS users may make use of the built-in macOS terminal.

If you have suggestions, pull requests & edits are welcome!

## Launch an EC2 Instance

1. Sign up for an Amazon AWS account. Sign up for a [GitHub education pack](https://education.github.com/pack) if eligible and you may get some free Amazon AWS credits.
2. Spin up an Amazon Linux 2 instance. The "compute optimized" tier is recommended, and you should not need more than 16GB of storage.
3. Create a new SSH key pair when prompted, or choose one already saved in your AWS account. If you create a new pair, you will be asked to download your private key.
    - **Your private key must be kept secure**. By convention it should be placed in your local `~/.ssh` directory and be protected by either `0400` or `0600` permissions [(note)](https://superuser.com/questions/215504/permissions-on-private-key-in-ssh-folder). Your `~/.ssh` directory should have `0700` permissions.
    - Private keys may take several different forms:
        + `*.pem` - Standard file format for cryptographic keys/certificates. AWS uses this format.
        + `*.key` - Alternate file extension for a PEM file only containing a private key.
        + `*.ppk` - Proprietary PuTTY format for private keys. PuTTY does not support the PEM format.
    - Public keys utilize the `*.pub` extension, but when copied to a server are appended to your remote `~/.ssh/authorized_keys` file. The presence of your public key in this **remote** file grants you access to the server.
        + If you are manually copying a new public key to an instance you already have access to, use the `ssh-copy-id` command [(note)](https://askubuntu.com/questions/4830/easiest-way-to-copy-ssh-keys-to-another-machine). Otherwise, the AWS setup guide handles this process for you.
4. Connect to your EC2 instance via SSH. You can find the IP address/hostname of your instance in your AWS dashboard.
    - Append the following to your local `~/.ssh/config` file, substituting the appropriate values as necessary:
        ```shell
        Host your_server_name
            HostName your_ip_address_or_hostname
            User ec2-user
            IdentityFile ~/.ssh/your_private_key.pem
        ```
    - Then, SSH into the server with `ssh your_server_name`. 
    - Alternatively, you can skip the instructions above and connect directly with:
        ```shell
        ssh ec2-user@your_ip_address_or_hostname -i ~/.ssh/your_private_key.pem
        ```

## Install Software

### Git
- Install Git
    ```shell
    sudo yum install git
    ```
- To push and pull from GitHub over SSH, you will need another public/private key pair that is tied to your GitHub account [(note)](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/). If you do not have a key pair, generate one on your EC2 instance with `ssh-keygen` and add the public key to your GitHub account. If you already have an authorized key pair, copy the private key to your EC2 instance and place it in your remote `~/.ssh` directory:
    + On the local machine, navigate to your directory with relevant keys (usually `~/.ssh` or `%USERPROFILE%/.ssh`).
    + Use `sftp` to put your `github_rsa` private key on the remote server.
    + Exit `sftp`, and then `ssh` back into the server.
    + Move the private key into .ssh: `mv github_rsa .ssh/`.
    + Check that the permissions are correct: `ls -al .ssh`.


### Intel MKL
- Intel MKL is [available for free](https://software.intel.com/en-us/articles/how-to-get-intel-mklippdaal) from Intel. The [yum repository](https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-yum-repo) can be easily added on an Amazon Linux 2 system:
    ```shell
    sudo yum-config-manager --add-repo https://yum.repos.intel.com/mkl/setup/intel-mkl.repo
    sudo rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
    ```
- Multiple versions of MKL are available, but the latest can be easily installed:
    ```shell
    sudo yum install intel-mkl
    ```

### Julia (Build From Source)

These instructions are for building Julia from source. Binary files are also available on the [Julia Downloads page](https://julialang.org/downloads/), and installation instructions are available [here](https://julialang.org/downloads/platform.html)

- First, install the necessary dependencies:
    ```shell
    sudo yum groupinstall 'Development Tools'
    sudo yum install make gcc gcc-c++ libatomic python gcc-gfortran perl wget m4 patch pkgconfig
    sudo yum autoremove cmake # default version is too old
    ```
- Download the Julia source code:
    ```shell
    wget https://github.com/JuliaLang/julia/archive/v1.0.3.tar.gz
    ```
- Extract Julia source code tarball and move to `/usr/local`:
    ```shell
    tar -xzvf v1.0.3.tar.gz
    mv julia-1.0.3/ /usr/local/julia-1.0.3/
    cd /usr/local/julia-1.0.3/
    ```
    ```
- Use `make` to compile Julia:
    ```shell
    ./contrib/download_cmake.sh # force Julia to build an updated version of cmake
    make -j4 # where '4' is the number of available CPU threads
    ```
- Symlink Julia to `/usr/local/bin`:
    ```shell
    ln -s /usr/local/julia-1.0.1/julia /usr/local/bin/julia
    ```
### Julia Packages
- Open up a julia prompt and install packages into the default folder
    ```bash
    ]add AxisAlgorithms BenchmarkTools Calculus CategoricalArrays DataFrames Distributions FileIO Formatting GLM GR Gadfly IndirectArrays Interpolations JLD2 MixedModels NLSolversBase NLopt Optim PkgDev Plots Primes Profile ProgressMeter PyPlot RData Ratios StatsBase StatsFuns StatsModels
    ```