blog.md

Receiving mail or replies

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user's responses to email or action items quicker.

At a high level, there are a few different layers to this. The topmost layer is the email service, and this book's case, Mailgun. This service handles sending outbound emails, as well as routing incoming emails to an address/domain name specified in their interface. Once the email is routed, it will be redirected to a route and processor file in your application. This file will be responsible for parsing the incoming email address and using logic to decide what to do with it.

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Mailgun's email routing service and ngrok, an HTTP tunneling service. ngrok is very useful for a few solution throughout the remainder of this book. It allows you to have a web accessible URL, which tunnels(connects) to ports within your local machine. Meaning, that you will have a http://somesubdomain.ngrok.com that will forward to your computer, and a specific port specified when you start ngrok. This allows you to test with external services such as Mailgun, Stripe(later), Github(later) and more!

Let's get started with some setup:

• Download and use ngrok

  • Download ngrok
  • Unzip and move the executable file to where you would like.
  • In *nix OS's, open ngrok with: path/to/ngrok HTTP start 3000. This will be dependent of the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.
  • Optionally, if you upgrade to a paid version of ngrok, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart ngrok.

• In Mailgun go to the Routes tab to create the email route(modeled after MVC routes like Rails) that will send the email. Enter the following settings:

  • Chose Match Recipient
  • Enter development.standup.*@app.yourdomain.com for the recipient field. Mailgun routes allow wildcard character matching, which will allow you input extra characters in the email's reply-to. Meaning, adding something like a user's hash_id to have identifiable information from the incoming email.
  • Check forward and enter http://yoursubdomain.ngrok.io\linebreak/email_processor as the destination of that forward.
  • You can leave the priority alone and give the route a name, before pressing the submit button.

Now with that done, we can begin to modify the application. The application changes will consist of three main parts. First, a new gem(and a companion adapter gem) will be added. griddler is the main library to handle incoming email easily; griddler-mailgun is the Mailgun specific adapter that allows the griddler functionality to work with Mailgun as an incoming mail router. Next, Griddler will need to add a few parts of configuration application-wide to make sure some basic configuration is met. Lastly, an email processor file will be added to handle recieving and parsing the email.

The great part about adding Griddler to your current application is that if you are using the Gemfile from Chapter 3, you already have it installed. If not, just add gem 'griddler' and gem 'griddler-mailgun' to your Gemfile and run a bundle install.

Next to configure and setup Griddler in the application you will need to add a new file in the initializer folder setting a few configuration values. Then, add a quick line to add the default Griddler routing into the routes.rb file.

First the Griddler configuration:

\begin{codelisting} \codecaption{config/initializers/griddler.rb} \label{code:init_griddler} https://gist.github.com/3727d4fd972b108fe9d510bd3d4a07b4

\end{codelisting}

This will set the text the griddler library will look for in the email and tell it that it will use the installed Mailgun adapter.

Next, add a line to mount the library based routes into your application. Adding the line right abobve the root to: is fine:

\begin{codelisting} \codecaption{config/routes.rb} \label{code:griddler_routes} https://gist.github.com/05d188b1054b19ff8c3537ba0317a754

\end{codelisting}

By mounting Griddler with that syntax, it will automatically add a route to your application that will route from a specified endpoint to a Griddler based controller:

https://gist.github.com/79d0945803bdb605400d5fffcdd8b038

There are settings in Griddler's GitHub documentation to change the defaults, but unless you want to get creative with route paths or email processor class names, it can be unnecessary.

The default settings expect a class EmailProcessor to exist and to handle parsing the incoming email with a method process. Griddler, however, does not can where the actual file is placed, but that the class exists and is loaded. Personally, I find that email processing fits most into the services definition and can be placed there.

To allow Griddler and its files to capture text from incoming replies, there are a few changes that will be needed.

First we will update the EmailReminderMailer to create a unique reply-to address and include that email address as part of the outgoing email:

\begin{codelisting} \codecaption{app/mailers/email_reminder_mailer.rb} \label{code:ch8_reminder_mailer} https://gist.github.com/d32f1bc16a076717890bcbbfebe6fa2f

\end{codelisting}

Here we are using building the reply_to string by adding development if the current Rails environment is your local Rails application. This way, the separate Mailgun route made for development can route different from a production route you will add before deploying your application.

Next we will updated the mailer template to have ##- Please type your reply above this line -## and some text letting the email recipient know they can add a standup by replying:

\begin{codelisting} \codecaption{app/views/email_reminder_mailer/reminder_email.html.slim} \label{code:ch8_email_reminder_template} https://gist.github.com/b634c62c7a18162870abff2b4d29a714

\end{codelisting}

Lastly, we will need to add an extra column to the Standups table to track the Message-ID coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

https://gist.github.com/c8ef18519cd760b8ef93798ca9bd1ac8

Next, before the end of the newly created migrations change method, you will want to add add_index :standups, :message_id. This index will allow quick lookups as the Standups table grows. Finally, migrate the actual change:

https://gist.github.com/cf2d143bf2d1b456f7c3a4924e551217

With those changes out of the way we can now add the new EmailProces\linebreaksor class that will parse the incoming email:

\begin{codelisting} \codecaption{app/services/email_processor.rb} \label{code:email_processor} https://gist.github.com/6cab831ec95829bbb95082e513b6f2f8

\end{codelisting}

The class is relatively simple, but let's go over it section by section:

https://gist.github.com/ab0b74fd5fbf0e11f825042e615523e9

Here we are initializing the object when the class is called by Griddler, setting the email to a local email variable. Additionally, we are creating a hash to later use in the text content to Task type conversion.

https://gist.github.com/48cfea61509efe19ec126652ce17646c

This just adds some additional logging if the mail is processed in the local development environment.

https://gist.github.com/9ea050fb46cc35f265acbe05be09afd2

Here we are grabbing some information used in the parsing, as well as giving the process method chances to exit early if the incoming email is not sufficient for processing and Standup creation. In the first section, the incoming email address is parsed to find the user's hash_id. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers not guaranteeing "just once" delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current user and current time.

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation here) searches for lines that begin with [d], [t] or [r]. If those are present, it captures the content to the end of the line. The .scan method on the content's body, allows it to catch all occurrences of the above pattern. If the scan's output is empty, the process method exits.

https://gist.github.com/3960f9416d229a33eb00dee220521f29

The last section here is a culmination of all of the stored information so far to be saved into a new Standup. The user, tasks_from_body, today and message_id are all passed into a method that will hand the actual save. The build_and_create_standup method creates a new Standup, with the user's ID, date and message_id. Once the object is created, the tasks strings are iterated over to build the Task with a type and assigned as child objects with the << syntax. Finally the new Standup object with children Tasks will be saved with standup.save

Lastly, you can test this all works if you reply back to an email(that was sent through Mailgun SMTP and not letter_opener) with the following text:

https://gist.github.com/d0101a998c390befb0cf11527b24794f

Testing this will require a new spec file with quite a few it blocks to test all the branches the EmailProcessor may encounter.

First, it would be best if we create a factory to be able to quickly generate an email to be used within the EmailProcessor's spec. This way the email can have defaults and then we can use the FactoryGirl .build commands to create a new email object with any different attributes when needed to test the processor.

The factory itself is pretty simple:

\begin{codelisting} \codecaption{spec/factories/email.rb} \label{code:email_factory} https://gist.github.com/f1bbf8645b8e402e7f0c93b517758d39

\end{codelisting}

Now, with a factory available, the email_processor_spec, will be able to easily spin up new email objects as needed with specific changes to test all of the processors' conditional branches.

\begin{codelisting} \codecaption{spec/services/email_processor_spec.rb} \label{code:email_processor_spec} https://gist.github.com/e4e34c1943feba18ba4b51be2ef6ccab

\end{codelisting}

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then tests each failing path that doesn't create a standup, in order as those paths appear in the EmailProcessor's .process method.

A quick run of the whole rspec suite should show no failing tests and nearly perfect test/code coverage:

rspec spec

................................................................................
........................................................................

Finished in 25.78 seconds (files took 10.16 seconds to load)
152 examples, 0 failures

Coverage report generated for RSpec to standup_app/coverage. 428 / 431 LOC
(99.3%) covered.