#Heroku, Ruby on Rails and PhantomJS
In this post, I’m going to show you how to modify an existing Ruby on Rails app running on Heroku’s Cedar stack to use PhantomJS for screen scraping. If you’ve never heard of PhantomJS, it’s a command-line WebKit-based browser (that supports JavaScript, cookies, etc.).
Let’s get started. This is a high-level overview of the required steps:
- Modify your app to use multiple Heroku buildpacks.
- Extend your app to use both the Ruby as well as the PhantomJS buildpacks.
- Confirm that everything worked.
David Dollar has created Heroku buildpack that allows you to use multiple Heroku buildpacks for your app. ;-) Install it by setting an environment variable:
$ heroku config:set BUILDPACK_URL=https://github.com/ddollar/heroku-buildpack-multi.git
Extend your app to use both the Ruby as well as the PhantomJS buildpacks
Create a .buildpacks
file in the root directory of your app:
$ touch .buildpacks
Within your .buildpacks file, specify that your app uses both the Ruby and the PhantomJS buildpacks. This prevents you from having to (cross-)compile PhantomJS yourself. Make the contents of your .buildpacks file:
https://github.com/heroku/heroku-buildpack-ruby
https://github.com/stomita/heroku-buildpack-phantomjs
PhantonJS has a dependency on libQtWebKit.so.4, which the PhantomJS buildpack installs on Heroku in/app/vendor/phantomjs/lib. Modify your Heroku app’s LD_LIBRARY_PATH to include this directory:
$ heroku config:set PATH="/usr/local/bin:/usr/bin:/bin:/app/vendor/phantomjs/bin"
$ heroku config:set LD_LIBRARY_PATH=/usr/local/lib:/usr/lib:/lib:/app/vendor/phantomjs/lib
Confirm that everything worked
First things first - load your app in your web browser and confirm that it still comes up. :-) Next, launch a Heroku bash shell:
$ heroku run bash
Within the bash shell, invoke the PhantomJS executable to ensure that it runs:
$ vendor/phantomjs/bin/phantomjs —version
Good to go! Now you’re ready to call PhantomJS from your Ruby code.
I almost died today trying to solve this so hopefully this helps that poor sole that had to go through what I did today. Basically did everything that was said and I'll just recap what I did.
Overview of what needs to be done:
heroku config:set BUILDPACK_URL=https://github.com/ddollar/heroku-buildpack-multi.git
Create a .buildpacks file in your app for rails (touch .buildpacks should do it but you can also write click on your main project directory and create it)
In the .buildpacks file add
https://github.com/heroku/heroku-buildpack-ruby
https://github.com/stomita/heroku-buildpack-phantomjs
Then in your terminal set the configs:
$ heroku config:set PATH="/usr/local/bin:/usr/bin:/bin:/app/vendor/phantomjs/bin"
$ heroku config:set LD_LIBRARY_PATH=/usr/local/lib:/usr/lib:/lib:/app/vendor/phantomjs/lib
running "heroku run bash" to make sure everything was setup doesn't work until you deply to heroku. Keep that in mind if you are getting an error when you run "vendor/phantomjs/bin/phantomjs —version", deploy then it'll most likely work.
New stuff
Great, now your heroku app is being stupid when you go to your domain name and it's probably giving you an error such as:
The reason I found was because the dynos weren't setup anymore to know what to do. So you need to create a Procfile, make sure you spell it exactly like Procfile and DONT lowercase the P:
Then put these commands in the Procfile (these commands below for some reason get erased in heroku thanks to buildpacks so you are basically setting your application back to what it was doing before you used buildpacks):
web: bin/rails server -p $PORT -e $RAILS_ENV
worker: bundle exec rake jobs:work
Sweet, commit that and push to heroku. One more step..
Make sure you have a dyno actually running for web:
you can also just run this in your terminal:
heroku ps:scale web=1
Which sets your dyno web dyno to 1. Anyways, after that everything worked for me. Hopefully that helps!