Development environment setup guide for etl_pipeline

ETL_Pipline_Setup.md

Setting Up Your ETL Pipeline Development Environment

Step 1. Install Homebrew

Homebrew is a package manager for OS X. It's the easiest way to install and keep up to date a lot of the software that you'll need to work on ETL pipeline.

First run:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Then to verify it installed successfully run:

brew --help

You should see something like this.

~/ ᐅ brew --help
Example usage:
  brew [info | home | options ] [FORMULA...]
  brew install FORMULA...
  brew uninstall FORMULA...
  brew search [foo]
  brew list [FORMULA...]
  brew update
  brew upgrade [FORMULA...]
  brew pin/unpin [FORMULA...]

Troubleshooting:
  brew doctor
  brew install -vd FORMULA
  brew [--env | config]

Brewing:
  brew create [URL [--no-fetch]]
  brew edit [FORMULA...]
  https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Formula-Cookbook.md

Further help:
  man brew
  brew home

Step 2. Install lots of necessary software using Homebrew

We are going to install everything you need to run the ETL Pipeline app locally.

• MySQL
• git
• Redis
• RVM

For most of this we'll use Homebrew. The one exception is RVM (Ruby Version Manager).

    brew install homebrew/versions/mysql56
    brew install redis
    brew install git

Setup MySQL to launch automatically on startup (see Technical Onboard Documentation).

    ln -sfv /usr/local/opt/mysql56/*.plist ~/Library/LaunchAgents 
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql56.plist   

Step 3. Install RVM

Your Mac comes with Ruby installed, but it's an older version. Also, each app you build will likely require ruby gems and with system ruby you have to install them globally. RVM (or another Ruby version manager like rbenv) will let you install gems just for your app.

    \curl -sSL https://get.rvm.io | bash -s stable

Step 3.1 Install Ruby 2.2.2

    rvm install ruby-2.2.2
    rvm use ruby-2.2.2

Step 4. Get the ETL pipeline code

    git clone https://github.com/bkr/etl_pipeline.git

Step 5. Install the app dependencies

We are going to create a gemset for our app so that it's dependencies are installed separately. Then we are going to use bundler to install the dependency gems.

    rvm gemset create etl_pipeline
    rvm gemset use etl_pipeline
    gem install bundler
    bundle install

Step 6. Create Development Databases

Rails needs a couple of databases to run test and run the app locally. You may notice the weird bundle exec command in front of everything. that ensures that all your apps dependencies are loaded and accessible when you run the command.

Create the user

    $> mysql -u root
    
    create user 'br_db_user'@'localhost' identified by 'br_db_pass';
    grant all on *.* to 'br_db_user'@'localhost' with grant option;
    quit

Create the schema.

    bundle exec rake db:create RAILS_ENV=development
    bundle exec rake db:create RAILS_ENV=test

Update the schema to the latest verison.

    bundle exec rake db:migrate RAILS_ENV=development
    bundle exec rake db:migrate RAILS_ENV=test

Run the tests

    bundle exec rake