Heroku Iron docs update

iron_mq.md

IronMQ is a massively scalable messaging queue that makes it easy for you to keep a resilient and synchronised message-passing system in the cloud, between any number of clients, without server set-up or maintenance. It makes managing data and event flow within your application simple.

Iron has partnered with Heroku to make using both services together even easier.

Get Started

It's quick and easy to get IronMQ set up and running on Heroku. You just need to install the IronMQ add-on for Heroku. You can do this with the heroku command:

:::term
$ heroku addons:add iron_mq:rust
-----> Adding iron_mq to cold-winter-5462... done, v10 (free)

This will add the starter level add-on for IronMQ, which will let you test the add-on and play around a bit. There are other levels of the add-on, as well.

Configuration

Now that you've added the add-on, you need to retrieve your token and project ID. The token functions as a password, so please keep it secure! Each app has a different project ID. You can get the token and project ID by running the following command:

:::term
$ heroku config | grep IRON
IRON_MQ_PROJECT_ID => 123456789
IRON_MQ_TOKEN      => aslkdjflaksuilaks

You can also get your token and project ID from the Iron.io HUD. To get to the Iron.io HUD, go to your apps panel for Heroku, choose your app, expand the add-ons drop-down, and click on IronMQ. This will bring you to the Iron.io HUD, where you can see your project ID and token listed.

IronMQ has clients for a lot of languages, and you can always use the REST API (or write your own!). This means your existing Heroku stack should work without any changes. The remainder of this article will be using the Ruby library, but all of the libraries have analagous calls (that should be well-documented).

We're going to need to install the Ruby gem, for development purposes:

:::term
$ gem install iron_mq
Fetching: iron_mq-1.4.0.gem (100%)
Successfully installed iron_mq-1.4.0

Heroku automatically adds the token and project ID to your production environment variables. You need to take care of your development environment yourself, however. Simply add the following in config/environments/development.rb:

:::ruby
ENV['IRON_MQ_TOKEN'] = 'YOUR TOKEN'
ENV['IRON_MQ_PROJECT_ID'] = 'YOUR PROJECT ID'

If you’re building for Rails 3, add the following to your Gemfile:

:::ruby
gem 'iron_mq'

If you’re building on Rails 2, add the following to your environment.rb file:

:::ruby
config.gem 'iron_mq'

Basic Example

We're ready to start working with our message queue. We're going to build a sample application that pulls tweets from Twitter's search function every time a URL is hit. We're then going to push those tweets onto an IronMQ queue. Then we'll pull tweets off the queue, one at a time, and display them to the user. We'll then let the user delete them from the queue.

Pulling the Tweets

To make life a little easier on ourselves, let's use the Twitter gem. Install it:

:::term
$ gem install twitter
Fetching: twitter-2.1.0.gem (100%)
Successfully installed twitter-2.1.0

If you're building for Rails 3, add the following to your Gemfile:

:::ruby
gem 'twitter'

If you're building for Rails 2, add the following to your environment.rb file:

:::ruby
gem 'twitter'

Now that we have a convenient wrapper for the Twitter API, let's do a search. We're going to search for "IronMQ", but you could search for mentions of your company, your product, or your favourite cat video.

Run the following command:

:::term
$ rails generate controller tweets get view

Rails will generate a bunch of files for you, giving us a skeleton we can work in. Go ahead and open app/controllers/tweets_controller.rb. Modify it to look like this:

:::ruby
class TweetsController < ApplicationController
  require 'iron_mq'
  require 'twitter'
 
  def get
    tweets = Twitter.search("IronMQ")
    render :json => tweets.inspect
  end

  def view
  end
end

Easy, right? Load up your-heroku-app.herokuapp.com/tweets/get, and you'll see the results of your Twitter search.

Adding the Tweets to IronMQ

It's time to actually make use of those tweets. We're going to push them to an IronMQ queue, so we can deal with them whenever we feel like it and know we're not missing any of them. Let's modify app/controllers/tweets_controller.rb some more:

:::ruby
class TweetsController < ApplicationController
  require 'iron_mq'
  require 'twitter'
 
  def get
    tweets = Twitter.search("IronMQ")
    ironmq = IronMQ::Client.new("token"=>ENV["IRON_MQ_TOKEN"], "project_id"=>ENV["IRON_MQ_PROJECT_ID"], "queue_name"=>"tweets")
    tweets.map do |tweet|
      ironmq.messages.post("#{tweet.id}")
    end
    render :json => tweets.inspect
  end

  def view
  end
end

The IronMQ gem is pretty simple to use. You just instantiate a new client (IronMQ::Client.new) and pass it some information. The token and project_id are the variables we set up earlier, and the queue_name is just an identifier for the message queue you want to use. You can have multiple queues per project, and these identifiers help tell them apart. After that, posting a message is as easy as ironmq.messages.post(), passing the message we want to post (the ID of the tweet, in this case).

If you load up your-heroku-app.herokuapp.com/tweets/get, the tweets you pull from Twitter will be stored in IronMQ. You can tell it worked by going to your Iron.io HUD (click on the IronMQ addon in your Heroku dashboard) and selecting Queues. This will display a list of your queues, along with the number of messages waiting in them.

Retrieving Tweets from IronMQ

Putting information into IronMQ is great, but it's a little useless if we can't get it out again. We're going to write a method that pulls a single Tweet from IronMQ and displays it. Let's modify app/controllers/tweets_controller.rb again:

:::ruby
class TweetsController < ApplicationController
  require 'iron_mq'
  require 'twitter'
 
  def get
    tweets = Twitter.search("IronMQ")
    ironmq = IronMQ::Client.new("token"=>ENV["IRON_MQ_TOKEN"], "project_id"=>ENV["IRON_MQ_PROJECT_ID"], "queue_name"=>"tweets")
    tweets.map do |tweet|
      ironmq.messages.post("#{tweet.id}")
    end
    render :json => tweets.inspect
  end

  def view
    ironmq = IronMQ::Client.new('token'=>ENV['IRON_MQ_TOKEN'], 'project_id'=>ENV['IRON_MQ_PROJECT_ID'], 'queue_name'=>'tweets')
    tweet = ironmq.messages.get()
    Twitter.configure do |config|
      config.consumer_key = "CONSUMER_KEY"
      config.consumer_secret = "CONSUMER_SECRET"
      config.oauth_token = "ACCESS_TOKEN"
      config.oauth_token_secret = "ACCESS_SECRET"
    end
    render :inline => Twitter.oembed(tweet.body.to_i).html
  end
end

As you can see, we now have a view method defined. This method configures an IronMQ client (just like we did before) and pops a message off the queue. When a message is popped off the queue, it isn't removed from the queue automatically. 60 seconds after a message is popped off the queue, it's returned to the queue if it hasn't been deleted yet. This ensures that messages are only removed when they've been properly handled.

After we get the message, we configure a Twitter client. The only reason we do this is to get around the rate limiting that Twitter uses on unauthenticated apps. Twitter rate-limits unauthenticated requests based on the IP, and depending on your app, you may have problems with other apps sharing your IP and eating up your API requests. You may be able to avoid this step.

Finally, we get the tweet ID from the body attribute of our message, convert it to an integer with .to_i so that it will play nice with Twitter's oembed function, and get the html output of the oembed function. This gives us a nice, embeddable HTML version of the tweet, which we render.

Load up your-heroku-app.herokuapp.com/tweets/view, and you'll see the most recent tweet that matched your search query.

Deleting Tweets from IronMQ

Once we've dealt with the tweet, we need a way to remove it from the queue. We're going to add a new action to our Rails app to take care of this for us. Edit app/controllers/tweets_controller.rb again:

:::ruby
class TweetsController < ApplicationController
  require 'iron_mq'
  require 'twitter'
 
  def get
    tweets = Twitter.search("IronMQ")
    ironmq = IronMQ::Client.new("token"=>ENV["IRON_MQ_TOKEN"], "project_id"=>ENV["IRON_MQ_PROJECT_ID"], "queue_name"=>"tweets")
    tweets.map do |tweet|
      ironmq.messages.post("#{tweet.id}")
    end
    render :json => tweets.inspect
  end

  def view
    ironmq = IronMQ::Client.new('token'=>ENV['IRON_MQ_TOKEN'], 'project_id'=>ENV['IRON_MQ_PROJECT_ID'], 'queue_name'=>'tweets')
    tweet = ironmq.messages.get()
    Twitter.configure do |config|
      config.consumer_key = "CONSUMER_KEY"
      config.consumer_secret = "CONSUMER_SECRET"
      config.oauth_token = "ACCESS_TOKEN"
      config.oauth_token_secret = "ACCESS_SECRET"
    end
    render :inline => Twitter.oembed(tweet.body.to_i).html + "<br /><br /><a href=\"/tweets/acknowledge/#{tweet.id}\">Acknowledge</a>"
  end

  def acknowledge
    ironmq = IronMQ::Client.new('token'=>ENV['IRON_MQ_TOKEN'], 'project_id'=>ENV['IRON_MQ_PROJECT_ID'], 'queue_name'=>'tweets')
    render :json => ironmq.messages.delete(params[:id])
  end
end

You'll notice we've added a link to /tweets/acknowledge/#{tweet.id} to our /tweets/view output. When you get a message off your queue, you get a bunch of useful metadata with it, beyond just the data you passed in. One bit of this metadata is the message ID, which we use to delete the message from the queue.

The acknowledge action is simple: we create an IronMQ::Client again, using the same code we used in the other two actions. Once it's created, we call .messages.delete on it, passing the ID we put in the URL. We render the JSON response we get from the delete call.

That's all it take takes to delete a message. To wire up our new action, edit config/routes.rb and add the following line:

:::ruby
match "tweets/acknowledge/:id" => "tweets#acknowledge"

This tells Rails to match our new action and that we expect an ID parameter.

Next Steps

Congratulations! You've written your first application using IronMQ. To get into more advanced uses of it, you may want to check out the API docs or a more advanced implementation that ties in IronWorker, TweetWorker.

Troubleshooting

Issues should get logged with Heroku Support. You're also welcome to stop by the Iron.io support chat room and chat with Iron's staff about issues. You can also find more resources on the Iron.io support site and more documentation on the Iron.io documentation site.

iron_worker.md

IronWorker is a massively scalable task queue/job queue that makes it easy for you offload front end tasks, run background jobs, and process many tasks at once -- all in the cloud and with no servers to set-up and manage. It can also function as a cron-in-the-cloud service, running tasks on a schedule you define.

IronWorker has partnered with Heroku to make using both services together even easier.

Get Started

It's quick and easy to get IronWorker set up and running on Heroku using your language of choice. Note that Ruby IronWorker currently requires Ruby 1.9 or later. Please check the Heroku documentation to find out whether your stack is supported and how to select a supported stack.

Once you have a stack selected, you need to install the IronWorker add-on for Heroku. You can do this with a quick command:

:::term
$ heroku addons:add iron_worker:starter
-----> Adding iron_worker to strong-mountain-405... done, v29 (free)

This will add the starter level add-on for IronWorker, which will let you test the add-on and play around a bit. There are other levels of the add-on, as well.

Configuration

Now that you've added the add-on, you need to retrieve your token and project ID. The token functions as a password, so please keep it secure! Each app has a different project ID. You can get the token and project ID by running the following command:

:::term
$ heroku config | grep IRON
IRON_WORKER_PROJECT_ID => 123456789
IRON_WORKER_TOKEN      => aslkdjflaksuilaks

You can also get your token and project ID from the Iron.io HUD. To get to the Iron.io HUD, go to your apps panel for Heroku, choose your app, expand the add-ons drop-down, and click on IronWorker. This will bring you to the IronWorker HUD, where you can see your project ID and token listed.

Heroku automatically adds the token and project ID to your production environment variables. You need to take care of your development environment yourself, however. Simply add the following in config/environments/development.rb:

:::ruby
ENV['IRON_WORKER_TOKEN'] = 'YOUR TOKEN'
ENV['IRON_WORKER_PROJECT_ID'] = 'YOUR PROJECT ID'

Then create a file at config/initializers/iron_worker.rb and put the following into it:

:::ruby
IronWorker.configure do |config|
  config.token = ENV['IRON_WORKER_TOKEN']
  config.project_id = ENV['IRON_WORKER_PROJECT_ID']
end

If you're building for Rails 3, add the following to your Gemfile:

:::ruby
gem 'iron_worker'

If you're building on Rails 2, add the following to your environment.rb file:

:::ruby
config.gem 'iron_worker'
config.load_paths += %W( #{RAILS_ROOT}/app/workers )

Basic Example

Now we're ready to create a worker and show how it gets invoked within an application. For demonstration purposes, we're going to make a FibonacciWorker that will calculate the Fibonacci sequence up to a given number.

Writing the Worker

Here's the worker that calculates the sequence. You can copy and paste and place in in a file called fibonacci_worker.rb.

:::ruby
require 'iron_worker'

class FibonacciWorker < IronWorker::Base

  attr_accessor :max

  def run
    values = Array.new
    num1 = 0
    num2 = 0
    nextNum = 1
    while nextNum < max
      num1 = num2
      num2 = nextNum
      nextNum = num1 + num2
      values << num2
    end
    log "#{values}"
  end
end

This example uses a basic implementation of a Fibonacci calculation up to the max attribute we set, then logs the values.

We require the iron_worker gem and sub-class IronWorker::Base. Then we set up an attribute accessor to hold the number we want to calculate up to. Finally, we define a run method and place the code to execute in there. This is what will run when tasks are taken off the queue and processed within IronWorker.

Testing the Worker

It's time to make use of those hard-won credentials. Testing the worker is just a matter of running it on your local machine. Doing so is simple. Put the following in run_fibworker.rb:

:::ruby
require 'iron_worker'
require_relative 'fibonacci_worker'

IronWorker.configure do |config|
  config.token = 'YOUR TOKEN HERE'
  config.project_id = 'YOUR PROJECT ID HERE'
end

worker = FibonacciWorker.new
worker.max = 1000
worker.run_local

We require the iron_worker gem, use require_relative to load up the worker file, set the credentials we pulled from the IronWorker HUD, set any attributes, and call run_local. The run_local command just runs the worker on your local machine and prints any puts statements to STDOUT. To see it in action, enter ruby run_fibworker.rb in your console.

Queuing the Worker

Once the worker is tested to your satisfaction, it's time to offload it to the IronWorker cloud. Edit run_fibworker.rb and change the last line (worker.run_local) to worker.queue. Enter ruby run_fibworker.rb in your console again, and you'll see the following output:

:::term
IronWorker initialized.
Uploading FibonacciWorker, code modified.
file size to upload: 283625
Uploading now...
Done uploading.
Queuing FibonacciWorker...

Your Worker is now running in the cloud.

Running from Heroku

Now that we know the worker runs and uploads from your machine, we want to run it from within your application within Heroku.

First, you need to get an app running on Heroku. Heroku has some great documentation on how to do that. Once you get the Heroku app running, integrating it with our Fibonacci Worker is simple. Create an app/workers directory, and move fibonacci_worker.rb into it. Then just take our run_fibworker.rb code, and move it into a Controller. For example, WorkerController may look like this:

:::ruby
def run
  worker = FibonacciWorker.new
  worker.max = 1000
  resp = worker.queue
  render :json => resp
end

Deploy the app to Heroku, and load up your-app.herokuapp.com/worker/run to see the success message.

Next Steps

This is just the tip of the iceberg. IronWorker has a robust API that allows for a lot more interaction with your Workers. You may want to try:

• Checking the status of your worker.
• Getting the logs from your worker.
• Scheduling tasks for your worker that will be run after a delay, repeating, or both.

You can also check out some example workers:

• TweetWorker, an app that pulls tweets and displays them. It uses IronWorker, IronMQ, and Sinatra, all while being hosted on Heroku.
• We also have a full repository of IronWorker examples for Rails on Github.

Troubleshooting

When trying to troubleshoot a Worker, the best first step is to try and run the Worker locally. If the Worker runs locally, it should run on the cloud. You can also access your Worker logs through the Iron.io HUD. These logs will show you any errors thrown or debug messages you log while the worker is running.

The most common source of Worker errors is a mismatch between your local environment and the cloud's environment. Double-check your Gemfile and your Ruby version -- workers run under Ruby >1.9. Also, make sure your Gemfile.lock has been updated. Run bundle install to make sure.

Also note that IronWorker is not able to to connect with Heroku's shared databases. At this point, Heroku's shared databases do not allow for direct connections. We are working to remedy this situation. The suggested workaround is to pass the data back through your application, post-processing.

Issues should get logged with Heroku Support. You're also welcome to stop by the Iron.io support chat room and chat with Iron's staff about issues. You can also find more resources on the Iron.io support site and more documentation on the Iron.io documentation site.