In my last article I introduced you to Propono - a pub/sub wrapper for Ruby. If you missed it, then I'd recommend going and having a read before you move on to this. To recap, Propono makes it exceptionally easy to publish and subscribe to messages between services using two simple methods:
# Service 1
Propono.listen_to_queue(:user) do |message|
p "Message Received"
p message
end
# Service 2
Propono.publish(:user, {foo: 'bar'})
It runs on your own AWS account, and relies on no infrastructure beyond their exceptionally reliable cloud services. Your data stays within your control, and both configuration and application are very easy.
As we started to use Propono at Meducation I discovered that we were wrting the same pub/sub patterns over and over again. We treat our services in the same way we treat classes - each having a single responsibility. This means we're constantly creating new daemons and writing a lot of the same bootstrapping code. I therefore decided it would make sense to extract a lot of this common code into a wrapper around Propono, that both lives within it, and bootstraps a new daemon for us. We called this Larva.
In this article I'm going to take you through how to create a new daemeon powered by Larva, how to use the features of Larva to make services ridicuously quickly, and then take a look at what's happening under the hood.
In this article, we're going to build a daemon that's responsible for sending emails upon certain events. It's workflow is very simple:
- Listen for messages about a series of events.
- Work out what email to send.
- Send the email.
The first email/event we care about is sending a friendly welcome email when a new user signs up to our website. Let's presume we have a website that's got Propono plugged into it and that publishes events about users onto the "users" topic. When a user signs up, it publishes a message like this.
Propono.send :user, {entity" 'user", action: 'created", id: 123125, name: "Jeremy Walker", email: "jez.walker@gmail.com"}
In a standard service with Propono in, we would listen to the user queue as follows:
Propono.listen_to_queue(:user) do |message|
case message.action
when "created"
# ... Do something
end
end
Larva is centered around the concept of processors. A processor listens on a topic, expecting messages in a certain format and knowing what to do for each message. Rather than directly reference Propono in our Larva daemon we would create a processor. To replicate the behavour in the case statement above, we'd create a processor like this:
class UserProcessor < Larva::Processor
def user_created
# ... Do something
end
end
The processor looks for two keys in the message: entity and action, and then looks for a method in the format of "#{message[:entity]}#{message[:action]}". If we also wanted to listen to events about users updating their profile photo, we could send messages from the website such as:
Propono.send :user, {entity" 'photo", action: 'updated", url: "http://cloudfront...."}
and listen by adding another method to the processor:
class UserProcessor < Larva::Processor
def user_created
# ... Do something
end
def photo_updated
# ... Do something else
end
end
These processors are the core building block of a Larva daemon, and make building functionality that listens to messages extremely easy.
Let's actually write our mailer dameon. Creating a daemon is very simple. Firstly, let's install Larva:
gem install larva
You now have the larva gem installed and the larva command at your disposal. Let's get larva to spawn a mailer dameon for us:
larva spawn mailer
OK. With that simple command lots just happened. A new directory has been created called mailer. If you cd into it and run ls you'll see a pretty standard directory structure. You'll also notice this is a git project. In fact, if you check the git log ("git log") you'll see there's already a commit with this basic structure in place. You'll also notice a test directory with some sample tests in. Let's check they pass:
bundle install
bundle exec rake test
You should see two test have been run and one assertion that has passed. Let's start by taking a look at the processor test and see what's going on. Open up the test/processors/user_processor.rb file. It will look something like this:
module Mailer
class UserProcessorTest < Minitest::Test
def test_everything_is_wired_up_correctly
message = { entity: "user", action: "created", foo: 'bar'}
Mailer::UserProcessor.any_instance.expects(:do_something).with('bar')
Mailer::UserProcessor.process(message)
end
end
end
This test is checking that a sample processor is wired up correctly. It's crafting a message similar to the one I described above and calling process on the UserProcessor with it. We're then checking that a "do_something" message is getting called with the 'bar' string. Let's take a look at the code that it's testing. Open up the processor in lib/mailer/processors/user_processor.rb. It'll look like this:
module Mailer
class UserProcessor < Larva::Processor
def user_created
do_something(message[:foo])
end
private
def do_something(foo)
end
end
end
This processor is really similar to what we looked at earlier. It's expecting a message with entity of "user" and action of "created" then calling the private method "do_something" with the :foo part of the message. There's one remaining part of the jigsaw here. Take a look at lib/mailer/daemon.rb, which looks something like:
module Mailer
class Daemon < Larva::Daemon
def initialize(options = {})
processors = {
user: UserProcessor
}
super(processors, options)
end
end
end
The daemon's initializer is where we set up the piping between a topic :user and the processor class. This code means that any messages received on the :user topic will be passed to the UserProcessor which will then call the relevant method. In Propono code that means any message sent like this:
Propono.publish :user, {....}