Two weeks later

2014-02-07.md

Jan 24 -> Feb 7 2014

Progress

• 3 prototypes started: "Turbomodules", AngularJS, Vanilla/jQuery
• Rails routing for all approaches
• No model layer duplication

Turbomodules

For lack of a better name, we've called our first prototype Turbomodules for their lightweight module system that sits on top of a standard rails + turbolinks stack. This approach has many core principles and assumptions worth repeating:

• Server side rendering of all (almost) initial state
• Rails routing + ERB rendering
• ActiveRecord model use in ERB templates (no presenter objects)
• "Server knows best". Most recent data, most capable of answering questions asked in building views
• SJR (Server-generated JavaScript Responses) are not wanted
• Declarative style of defining page capabilities and data
• The admin is mostly a document store
• DOM driven pages
• Minimal but structured JS for rich client interactions
• No javascript routing, modeling
• JS formatting, bindings, event handling
• White magic

Each page is a Document

One early decision we made with this approach was to look at each page in a more traditional document approach and to augment it with its own app functionality as needed. We've felt that over time most attempts at keeping state client side have resulted in poor practices more than valuable gain.

By throwing away the requirement for a stateful client, we can take a lot out of Javascript right off the start. Furthermore, as we move the majority of our display logic to the server side, we cut out a large middle layer of "management code" that does all the inbetween our current Batman framework needs to get the data from the server on to the page.

Not MVC

The "turbomodules" approach does not really bring any conventions you'd expect to see with a JS MVC framework, as it's not one. There is no templating, no client side rendering, no routing system and no model layer.

There are however bindings and controllers, which I'll describe later.

Custom bindings and controllers... again?

Re-implementing a binding system from the start again screamed NIH. After some thought, we decided to continue building our own minimalistic binding system for a few reasons:

• Avoid biasing ourselves to any existing implementation
• Discover what we really need out of a binding system
• Write down what we think the simplest solution to each problem we face is, only look for abstractions once the patterns present themselves
• Even as we approach very similar concepts to what is out there already, we come to those decisions on our own terms and have a much better understanding of why we do or don't want what we have

We've stayed with most of our initial decisions (https://gist.github.com/kristianpd/41a9bd0749197741b824#bindings), with a few exceptions/clarifications:

• The first node that is bound to a keypath that has a value is taken as that binding's value
• Only direct keypath bindings can set values, functions merely apply a filter to a previously defined binding
• We are still living with the <define bind="page.id"><%= @page.id %></define> compromise of getting server data in to the binding system that we don't use anywhere but in our code (no display). There have only been 2 cases of needing to use this so far.

It's also important to note that while we are still open to taking one of the mature binding systems off the shelf, this existing system hasn't had to be augmented too much and is less than 150 lines of JS. Combined with the white magic of Turbomodules described below, we're still less than 200 lines of framework JS.

scopes

We've formalized the concept of scopes a little more. Scopes are basically the controller and data store for each page. All binding system constructs use the scope to resolve their values and actions and it can be seen as the single place to store everything you'll need to work with in javascript for the page.

The 3 refreshes and your decision

As we've flushed out our integration with turbolinks, we've come to 3 ways of changing the state of your page:

1. Navigation
2. Form submissions
3. Refresh/Redraws

Navigation

Navigations are simply <a> tags that lead to another area of the admin. Turbolinks prevents the default behaviour of navigations by XHR GET'ing the DOM behind the scenes and replacing the current body + layout, js and css tag diffs. We do nothing special here other than make sure we clean our state between these somewhat stateful navigations.

You will use a navigation when you need to go from one page to another, or in our way of thinking, one app to another.

Form submissions

These are about as Web 1.0 as you can get. Full page (hard) refreshes, and a server side redirect or render. No state is kept around.

While you may stay within a page / app on a form submission, this implies the server has taken some action that merrits refreshing the entire page. Since the server will simply render the page in the correct state on response, we don't need to do any magic here.

Soft-refresh/redraw

Here's where you actually need to make a decision. In the first two examples, we either didn't need to keep any state around because we left the area or because everything changed and we may as well pull down the new Document.

While this satisfies a large portion of the admin, many of our pages also bring a more modern/rich client experience that hard refreshes won't satisfy.

For example:

1. Any page where you have input elements + some form of operation that can happen that is not related to those elements.
2. You want to perform a bulk operation on a selection of elements, but return to that selected list after the action has been performed.
3. Live / responsive searching as you type

Since we can't simply do a hard-refresh form submission (we'd lose the other form's semi-filled out data), we need a way of performing an operation server side then updating our DOM with the changes.

Alternative rendering strategies

There are a several approaches we can take here, I'd like to give some reasoning for why we went with our own approach:

Imagine a simple Add comment form:

<ul id="comments">
  <%= render @comments %>
</ul>

<form action="/comments">
  <textarea name="comment[body]"></textarea>
  <input type="submit"></input>
</form>

SJR (formerlly RJS)

This is the conventional rails way of doing it. You perform an operation (form submission, etc), the server responds with a JS format and you get something like this:

# create.js.erb
$('#comments').append("<%= render @comment %>");

This, in the most basic example seems pretty straight forward and simple to understand. Our concern with this approach, is that over time it often starts looking more like this:

# create.js.erb

<% if flash[:error] %>
  Flash.error("<%= flash[:error] %>");
<% else %>
  <% if high_priority %>
  $('#high_priority_comments').append("<%= render @comment %>");
  <% else %>
  $('#comments').append("<%= render @comment %>");
  <% end %>

  $('#comment_count').html("<%= @total_comment_count %>");
<% end %>

We feel this approach couples the view implementation details to the JS too much and results in a lot of independently innocent jQuery snippets that compound to a messy, less desirable situation that the freedom bindings + client side rendering have given us.

XHR + jQuery + JSON

You could approach the problem in a similar way by returning JSON from the server and updating your DOM manually with jQuery. You may be able to use a templating library to make this easier, but we feel like this would become even more burdensome than the SJR approach as a large amount of view details that are handled by the rails partials would innevitably creep in.

XHR + jQuery + Rails rendered partials

This approach was often our first consideration (hasn't been ruled out, Thibaut is exploring this still). Simply structure our views into nicely separated sections, broken down with Rails' existing partial system.

As opposed to having an SJR response do the $().append, you could move that to a controller sitting in JS land that would grab the HTML template returned by the server. Perhaps a better example would be as follows:

<table id="comments">
  <% comments.each do |comment| %>
    <tr>
      <td><input type="checkbox" name="comment_ids" value="<%= comment.id %>"></input></td>
      <td><%= comment.body %></td>
    </tr>
  <% end %>
</table>
<a onclick="doBulkAction('magic')">Magic!</a>

// something that resembles a controller

function doBulkAction() {
  var selection = $('#comments').find("input[type=checkbox].checked") // forgive my bad selector
  var operation = $.post(/* stuff */);
  operation.done(function(response){
    $('#comments').html(response);
  });
}

# comments controller

def show
  # a bunch of stuff to set up show view ivars
end

def bulk_operation
  do_it
  render partial: 'comments', @blog.comments
end

This approach is fairly straightforward to understand, seems to have a fairly repeatable pattern of $('#').html() replacement and may even help force devs to break their code up even better.

We had a few concerns with this approach initially:

1. The number of partial end points will likely skyrocket
2. If the partials start to cross-cut concerns, common set up may start to get messy between the actions. In this example, both show and bulk_operation would need to load @popular_stuff if all of a sudden the comments partial wanted to use it.
3. If a given action needs to update many areas of the screen, you either need to refresh larger areas or have multiple partial round-trips server side to refresh your screen. This would all have to be managed client side and we felt it may be rougher albeit a more rare scenario.

There can only be one! (way to render a page)

Just in case we've lost track, we're talking about soft-refreshing a page. This is the case where you need to update a portion of the page with some new data (think bulk actions) while leaving another form on the page (search box) untouched. We also would like to have the items (if they're still around) checked off when we get back from our action.

This approach returns to some of our early goals:

• Server knows best on how to render stuff
• We want one "draw loop", one canonical path to follow to see how a page is rendered
• We don't want to manage JS snippets that replace sections of the page

To follow the more complicated bulk actions example above, here's what our current situation looks like:

<table id="comments" refresh="comments">
  <% comments.each do |comment| %>
    <tr>
      <td><input type="checkbox" name="comment_ids" value="<%= comment.id %>" bind="comments.<%= comment.id %>.selected"></input></td>
      <td><%= comment.body %></td>
    </tr>
  <% end %>
</table>
<a click="bulkActions.doBulkAction('magic')">Add a comment</a>

// something that resembles a controller

function doBulkAction() {
  var resourceIds = // grab array of ids
  var operation = $.post(/* do the action */)
  // on success, refresh
  operation.done(function(response){
    Shopify.refresh({ refreshKeys: ['comments'] })
  });
}

What the hell is a Shopify.refresh?

Shopify.refresh can easily been imagined as a redraw on a back buffer. In fact, this is all that Turbolinks really does before it replaces the entire body contents with the new page. In this case, we're being a little more selective.

We want the full redraw of the page to happen server side as it removes any need for us to care about how the DOM is structured and what parts of it will change. Unfortunately we can't just replace the entire body in this case because we may have a form field that we've started to fill out before performing our bulk operation and we don't want to lose that data.

Enter the refresh key. refresh is a way of hooking a DOM node or a set of DOM nodes, to be optionally refreshed when we perform a back buffer redraw by round tripping a simple /show action to the server. In the example above, any DOM node with the refresh=comments attribute will be grabbed from the back buffer and replaced in the live screen. There's currently an extra requirement that all elements with the refresh key must also have a unique id.

This would also allow you to have a heading like this anywhere else in your DOM (not just the comments partial) and update it by adding the refresh key hook.

<h2 id="total_comments" refresh="comments">Your total comment count is: <%= @total_comment_count %></h2>

Another hidden benefit is that each module has a chance to apply transient state from the previous page to the newly rendered sections. We'd like to carry over our selected state, in our modules we have something like this:

  if (options.lastScope) {
    var resources = options.lastScope.get(resourceName)
    for (i in resources) {
      if (resources[i].selected) {
        scope.set(resourceName + '.' + i + '.selected', true);
      }
    }
  }

We feel this does a nice job of decoupling the need to know what DOM elements may be updated, or to make sure you get enough data to redraw the screen as you'll always have the whole data to choose from.

Testing?

Testing has admittedly taken a back seat so far in the prototyping under the following assumptions:

1. A large portion of tests we currently imagine when thinking admin will go away since there is no API layer, no model layer, no routing layer
2. Our modules are very straightforward and could easily be tested in any JS unit testing framework.
3. We want to keep our JS to a minimum with this approach, therefore there will likely be orders of magnitude less JS testing than we currently imagine.
4. More testing will be done in rails (functional tests include ERB rendering)
5. We have not solved the E2E problem yet and are not trying to solve it with this specific prototype.