Near the end of 2013, serious concerns about the scalability of Batman and the Admin over the next 5 years arose. While it may have appeared sudden and drastic, these concerns had been brewing for some time. In mid January, we resparked conversations that seriously considered the future of both Batman and the Admin at Shopify and came up with a plan to solve them.
Technological and cultural hurdles aside, I will outline several of the key problems that we find ourselves facing with Batman and the Admin today and then proceed to define our goals and current outcome of an attempt to rewrite the Admin using Rails and a selective amount of JS.
Although it may be tempting to revist the past 3-4 years of Batman development to list problems and challenges encountered, our scope was only in defining our current state and determining the best way for us to move forward.
While the current Admin is feasible and could be maintained and developed for years to come, it is important to constantly assess and revisit our stack and not be afraid to challenge our core assumptions; no matter how scary the prospect of doing so may seem.
To remain as objective as possible, I'd going to list what I believe to be our biggest challenges with the current approach and some clear goals for any successful alternative. In turn, I'll compare both our current and proposed future approach to these criteria and give any reasoning I can for where we stand today.
Challenges
- Heavy duplication of state in the Shopify object model between JS and RB
- Extraction of core business logic in to the client that is missing server side
- Large toolchain
- Effectively two stacks, equally maintained and highly coupled in development process
- Isolation, though artificial, between the admin team and core Ruby/Rails developers
- High cost of onboarding new developers in to Batman framework
- Hesitation to spread Batman across Shopify
This section is aimed at providing a brief history of the current admin stack in order to show a it of the process we took to get to our current state. While many opinions of JS MVC listed below admittedly represent a Batman biased view of them, I've tried my best to outline challenges we'd likely face in any JS MVC approach.
In the earlier stages of the Admin, the toolchain was layered in on top of core Rails technologies. This meant using the asset pipeline to build our asset bundles, serving up the initial page via an ERB action and using JSON endpoints for any interaction. While this appeared simple, several challenges arose with this approach:
- Slow deploy times due to asset compilation
- Difficulty in isolating JS bundles for test suites
- Slower development cycle as asset pipeline would often have to recompute the assets
In short, it always felt as though we had crammed our JS MVC app on top of Rails in a way that worked, but was never really built to do so.
This led us to seek out a more modern JS MVC toolchain in an attempt to simplify and decouple our processes.
In September of last year, the Admin team began isolating the Admin from the core Rails back end. There were several goals in doing so including:
- Increased development speed & response time
- Increased test suite speed
- Decoupling Admin from Rails
- More modern JS tooling
- Longer term isolation and deployment of admin client independent of rails
The majority of these objectives were realized as we moved our toolchain to a Lineman / Grunt based workflow. We removed our dependency on the rails asset pipeline in favor of a more modern toolchain better suited to the tasks at hand. Our test suite spin up increased dramatically, had better reporting and was generally easier to extend than previously. We nearly entirely decoupled the admin, with only a single ERB configuration dependency keeping the two stacks tied together.
While this separation of concern was appealing, we never felt the time was right to move the entire admin physically out of the Rails app. The belief is and was that it would always be easier to work on the Admin if it was in people's faces and there are serious concerns throughout the company about pulling it in to a separate repo.
I'd now like to review our challenges listed above and give more insight in to what each one means in the current Admin.
Heavy duplication of state in the Shopify object model between JS and RB
Due to the very nature of the Batman.Model system, we've set ourselves up for a huge amount of duplication between the server and client. The client is left with a near complete representation of the same object model the server has, while dangerously opening the door for us to differ or extend that model in a way that may alienate us from the true source of information: the server.
This model is not adopted by all frameworks (Angular has no rich model layer for example) and some of the challenges or frustrations we have are of our own making.
While a light-weight object model (just JSON, no JS objects) may seem the real answer, I believe they too will share many of the same problems.
-
Any JSON or presenter object translation will result in some level of duplication or definition of the object model that can become out of sync or require constant update.
-
While this in itself is not a bad pattern to follow , I believe that more often than not this layer would act as a 1-1 translation that would change constantly as the underlying object model changes.
Presented with the ever evolving object model that Shopify holds, I think the theoretical gains of this separation, have not been realized and may be difficult to fully realize in any JS MVC approach. has_one's become has_many's, properties are renamed or removed and core behaviour is added on. These types of changes affect the API at a fundamental layer, and I do not see them benefiting from the abstraction.
Extraction of core business logic in to the client that is missing server side
Developers often take the path of least resistance or work in a domain model of which they may not have full context. They often work with what is present in front of them and do not question, or know to question the underlying implementation of something they are working with.
As a result, over time we've noticed that several areas of the Admin have separated themselves from the server and begun to build up their own complex object model to represent Shopify's ever evolving domain model.
One can simply state that this is not the way it should be done and developers should know how to make the best decision all the time, or code reviewers should catch it. In reality, the situation is far more complex.
Take the TransactionTrees example in the Admin for example. The current state evolved over a few iterations:
- Use the existing order
/cancel&/refundendpoints as you only ever have to worry about the amount or the action - Introduction of partial refunds
- Modifications made to allow multiple refund transactions to show up for a given order
- Ask to extend the core API to support these changes at an order level
- Told to make use of the existing Order Transactions API
- Multiple payment methods introduced with POS and Gift cards
- Decided (Admin + API) to keep using the Order Transactions API
- Order Transactions API is built in a single transaction manner, no nesting or parent contexts are given
- Increase the complexity of the client model to be able to build up these Order Transactions for use with the API
- Many challenges encountered with various auth, void, refund states an order can be in. Introduce Trees
This is a very brief summary, but it shows how each decision compounds to a more complicated model. While many of these problems originate from wrong decisions server side, the client abstraction allows us to move forward, perhaps when we shouldn't.
Large toolchain
The move away from Rails asset pipeline was done with good intentions, and in my opinion is a better state than where we were before. While we're not realizing the full intent of this move, the decoupling of technologies is nearly complete.
The argument for this separation is warranted from an Admin team point of view. Faster tooling, easier changing, self managed deploy process all have tangible benefits, many of which have been realized. In general, this tooling has made developing the Admin a smoother experiene, if you know what you're doing.
The challenge we face is that many still do not understand the JS toolchain and as much as we try to cram it in to Rails, the differences can be frustrating. It's also important to note that we've never fully realized even our own initial goals with this tooling. The main reason for this is that we've frankly been in a state of limbo since last year on major technological decisions as we bring in to question our original path.
Effectively two stacks, equally maintained and highly coupled in development process
Model duplication problems outlined above, combined with a RESTful routing scheme that originated from a rails app has left many layers of the application duplicated between client and server. While one can argue the routing of the application is independent of your API, our Rails server is capable of doing both at once.
Let's look at a change in Shopify's domain model and how it would be applied to the admin. Adding a middle name to a customer would require the following:
- LHM to add column to customers table
- RB model
attr_accessibleand any unit tests needed - RB model
serialization_optionschange with test - Controller API functional test asserting
to_jsoncall or keys - Batman customer model encoder with associated tests
- Batman view / form changes
Steps 3-5 are necessary due to JS MVC stack and 3&4 would be likely required for most JS MVC solutions.
Isolation, though artificial, between the admin team and core Ruby/Rails developers
For many reasons listed above, JS MVC or just large amounts of Javascript has presented both cultural and technical hurdles in having others adopt the tooling and work with the admin. Many developers at Shopify still cringe when they hear "Node" and "Javascript". While this opinion is mostly due to lack of experience, the admin has become a Javascript developer role and not a role everyone feels they should be part of.
I make no argument for or against the value of this position, but only present it as a current way of life.
High cost of onboarding new developers in to Batman framework
Building our own custom MVC framework has come with many challenges. A lingering one is the constant cost of onboarding new developers. Many of these problems stem from the fact that Batman is not widely adopted in industry, effectively making it an in house solution.
While the conventions are very similar to rails, it is different enough that many spend a few weeks of upfront time learning the ins and outs before they become comfortable. Perhaps the greater cost is that of people not jumping in to it because of the artificial barrier of entry.
Hesitation to spread Batman across Shopify
While self inflicted, I believe this should be treated as a problem. This wraps up feelings described above along with many other biases and opionions formed over the years that have resulted in lack of confidence in the framework. This is hard if not impossible to break once set in.
With all this taken in to consideration, here are the high level goals we plan to achieve with this prototype:
- Simplify admin technology stack
- Eliminate costly state duplication
- Speed up merchant experience on all platforms (mobile included)
- Speed up development onboarding
- Make the right thing easy and the wrong hard