skozz

#Heroku, Ruby on Rails and PhantomJS

In this post, I’m going to show you how to modify an existing Ruby on Rails app running on Heroku’s Cedar stack to use PhantomJS for screen scraping. If you’ve never heard of PhantomJS, it’s a command-line WebKit-based browser (that supports JavaScript, cookies, etc.).

Let’s get started. This is a high-level overview of the required steps:

• Modify your app to use multiple Heroku buildpacks.
• Extend your app to use both the Ruby as well as the PhantomJS buildpacks.
• Confirm that everything worked.

skozz

#!/usr/bin/env bash

# Install Rails

trap 'ret=$?; test $ret -ne 0 && printf "failed\n\n" >&2; exit $ret' EXIT

set -e

log_info() {
  printf "\n\e[0;35m $1\e[0m\n\n"

skozz

INSTALL JAVA
$ sudo apt-get update && sudo apt-get install default-jre

INSTALL ELASTIC SEARCH https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch
$ sudo update-rc.d elasticsearch defaults 95 10
$ sudo service elasticsearch restart
$ sudo service elasticsearch status

skozz

## just some ways to check if a url exists

# method 1 - from Simone Carletti
require "net/http"
url = URI.parse("http://www.google.com/")
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)

# method 2 - from some kid on the internet
require 'open-uri'

skozz

rm /Volumes/TADA68\ \ /FLASH.BIN
cp ~/Downloads/FLASH.bin /Volumes/TADA68\ \ /FLASH.BIN

skozz

"""
Code that goes along with the Airflow tutorial located at:
https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py
"""
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.generic_transfer import GenericTransfer
from airflow.contrib.hooks import FTPHook
from airflow.hooks.mysql_hook import MySqlHook

skozz

class Person < ActiveRecord::Base

  # This manual SQL query...
  scope :allowed_to_eat_cheese, joins(
    <<-SQL
    INNER JOIN cities ON people.city_id  = cities.id
    INNER JOIN states ON cities.state_id = states.id
    SQL
    ).where('states.allows_cheese_consumption' = 'Yeppers')

skozz

Project Title

One Paragraph of project description goes here

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

skozz

###Sketch trial non stop

Open hosts files:

$ open /private/etc/hosts

Edit the file adding:

127.0.0.1 backend.bohemiancoding.com

127.0.0.1 bohemiancoding.sketch.analytics.s3-website-us-east-1.amazonaws.com

skozz

- Add `gem 'tzinfo-data'` to the Gemfile. 
- Add `tzdata` to the Dockerfile
```
RUN apk update \
    && apk add build-base \
       tzdata
```
- Build `docker-compose build` or `docker-compose up --build`
- Enjoy the life