rahul-c1

AWS EMR bootstrap to install RStudio Server along with sparklyr

How to use the bootstrap

1. You will first have to download the gist to a file and then upload it to S3 in a bucket of your choice.
2. Using the AWS EMR Console create a cluster and choose advanced options.
3. In Step 1 make sure you check the Spark x.x.x checkbox if you want to make use of the sparklyr library in RStudio. You can customize the Spark version by choosing a different emr Release version.
4. In Step 3 you can configure your bootstraps. Choose to Configure and add a Custom action
   • For the Name you can fill something like Install RStudio Server
   • For the Script location you will need to point to where you have uploaded the gist (Eg. s3://my-bucket/emr/bootstrap/install-rstudio-server.sh)

rahul-c1

AWS EMR bootstrap to install RStudio Server along with sparklyr

How to use the bootstrap

1. You will first have to download the gist to a file and then upload it to S3 in a bucket of your choice.
2. Using the AWS EMR Console create a cluster and choose advanced options.
3. In Step 1 make sure you check the Spark x.x.x checkbox if you want to make use of the sparklyr library in RStudio. You can customize the Spark version by choosing a different emr Release version.
4. In Step 3 you can configure your bootstraps. Choose to Configure and add a Custom action
   • For the Name you can fill something like Install RStudio Server
   • For the Script location you will need to point to where you have uploaded the gist (Eg. s3://my-bucket/emr/bootstrap/install-rstudio-server.sh)

rahul-c1

import pandas as pd
import numpy as np
import re

# ================================================
class option_parser:
    def __init__(self, symbol, response):
        self.symbol = symbol
        self.response = response
    # ------------------------------------------------

rahul-c1

import pandas as pd
import numpy as np
import re

# ================================================
class option_parser:
    def __init__(self, symbol, response):
        self.symbol = symbol
        self.response = response
    # ------------------------------------------------

rahul-c1

import asyncio
import aiohttp

# ================================================
# for first run only
class first_async_scraper:
    def __init__(self):
        pass

    async def _fetch(self, symbol, url, session, headers):

rahul-c1

import os
import sys
import pandas as pd
import pandas_datareader.data as web
import numpy as np
import time
import asyncio
from fake_useragent import UserAgent
'''set path variables'''
project_dir = "YOUR/PROJECT/DIR"

rahul-c1

rahul-c1

'''
Netfonds import 5 days of intraday data
'''

import numpy as np
import pandas as p
from pandas.tseries.offsets import *
import datetime as dt
import matplotlib.pyplot as plt
size = (14,10)

rahul-c1

#Example syntax for Secure Copy (scp)

##What is Secure Copy?

scp allows files to be copied to, from, or between different hosts. It uses ssh for data transfer and provides the same authentication and same level of security as ssh.

###Examples

Copy the file "foobar.txt" from a remote host to the local host

rahul-c1

Software

Please make sure you are using the latest version of R (current version is 3.4.1) - the final digit is the minor version number. The essential numbers are the first two. You can check the version of R you are running via

R.version.string

Please install the latest version of RStudio (https://www.rstudio.com/products/rstudio/download/).