At FlashFunders, our product team consists of five engineers working on our rails apps, a PM and two designers. We are constantly reevaluating our process for getting new features into production. One of the signficant improvements we've made is Flash Flow, a tool we use to keep our acceptance environment up to date. We are open sourcing Flash Flow in hopes that it is useful to you and that you might help improve it.
In this post we discuss what Flash Flow does and the problems it solves.
- Product manager (PM) writes story.
- Engineer creates feature branch from master to work on story.
- Feature branch gets merged to acceptance environment for PM to review, feature branch gets pushed to Github for pull request review.
- PM accepts story, code passes review, code gets merged to master and deployed to production.
Of course the PM's story is always perfectly clear, concise, and unambiguous. And the engineer’s code always meets the functional requirements for every edge case on the first try. And the code is so beautifully written that the reviewer can only sit, mouth agape, marveling at the fact that the coder has created the platonic ideal for code that “sends a user a reminder after two days of not verifying their email address”. And of course during all of this every automated test of our 100.1% test coverage passes in zero or fewer seconds, et voila, our site has a new feature.
Admittedly, once in a while, things don’t follow this path as precisely as they should.
One of the frustrating problems we’ve had at FlashFunders is merging code that’s ready for PM approval to a shared environment. The way we do it at FlashFunders is that code has to pass code review and get accepted by our PM before it goes to production. Like most things in life, the happy path is trivial; We have a branch called acceptance and all work gets merged to that branch and it gets deployed to our acceptance environment. Easy, right?
Except that you want to rebase your branch, and when you merge to acceptance git is convinced that you have merge conflicts with the old version of your branch. Or work makes it to acceptance, and then it’s decided that it shouldn’t have been done in the first place. Or one of your co-workers breaks the build right before he leaves for his three week vacation.
In each of those cases, our shared environment has wound up in a state where code exists that is either broken or obsolete. The way we used to handle this situation was by having an engineer stop what she’s working on, asking everyone else if it’s okay to re-build the acceptance branch, and then branching from master and merging all the code into acceptance that’s supposed to be there. There was a lot of wasted time and effort, plus uncertainty at any given point around what exactly was in the acceptance branch.
What we realized is that conceptually acceptance is not a branch in the traditional sense, it’s really just meant to be a snapshot of all the works in progress at a moment in time. The only time it was guaranteed to be in a known and correct state was exactly at the moment when we had re-built it by starting from master and merging all the feature branches in. So we decided to try to make it so that acceptance is always in that state.
Our flash_flow gem automates the creation of the acceptance branch, so now when you are ready to deploy your feature to acceptance, you run flash_flow and all active work is merged anew into acceptance and pushed to github. We define active work as all open pull requests on github. This is what flash_flow does:
- Creates a new branch based on master
- Merges all pull requested branches into this new branch, remembering and resolving merge conflicts along the way (more on this below).
- Pushes the result to Github (which kicks off our build and auto-deploy to acceptance)
This way, by running flash_flow, we have our acceptance branch in a happy state all the time. If someone were to commit directly to acceptance, the next time flash_flow is run that code would get wiped out.
One problem that becomes obvious very quickly with this scheme is merge conflicts. Let’s say you have two feature branches, A and B. flash_flow starts with master and merges A into it. If there’s a conflict here, it’s A’s fault, because it should get itself up to date with master and not conflict, so we can ignore that case. But then B gets merged into master+A, and if B conflicts with A, B is out of luck. flash_flow doesn’t let you resolve the conflict, because if you did your change would just get wiped out the next time someone ran flash_flow anyway (back to this point in a minute).
And that is how we treated conflicts initially. We would say if your code has a conflict when we run flash_flow, you have to wait until A has been merged to master so that you can get B up to date with master, and only then can it be merged successfully with flash_flow. An advantage of doing things this way is basically that you pretty much rid yourself of issues that arise from merge conflicts. Especially when two people are working on the same thing in parallel for a longer period of time dealing with conflicts all the time can be very time-consuming and error prone. The clear drawback of this is that it forces you to serialize work in acceptance that’s actually being done in parallel, and if the work needs to be serialized that should be handled in your issue tracker not your deploy tool.
Enter git rerere, the amazing feature of git that none of us had heard of before. To really understand what it does you should read that post, but even then it takes some time to sink in. In brief, what rerere does is remember how you resolved a particular merge conflict in a particular file. So from the example above, the reason it wasn’t worth it to resolve conflicts between A and B was because the next time flash_flow ran whoever ran it would just have to resolve it all over again, every single time. With rerere, you resolve the conflict once and the resolution gets saved and re-used every time that you encounter the conflict. So you merge B into master+A, resolve your conflicts, and then commit your changes. Now the next time you run flash_flow it recognizes that rerere knows how to fix those conflicts and B will merge successfully into acceptance and your formerly serialized features are back to living happily side by side.
There’s a little more to it than that, because rerere really only works in a single repo, but flash_flow copies all your local conflict resolutions into the acceptance branch, and copies them down the next time someone runs flash_flow, so every flash_flow user winds up with a superset of the rerere patches. The hidden benefit of this is that when that same branch is ready to be merged to master, the conflicts are often dealt with the very first time they come up in flash_flow, and you get to re-use the same resolution that worked in your acceptance environment.
And that’s the thrust of our process. Flash Flow does a little more than that, because it actually pushes your local branch to Github and creates a pull request automatically, but the important part is how it handles merging all the feature branches into one common branch for deployment to our acceptance environment.
Flash Flow wasn't built in isolation, and we'd be remiss to not mention the other approaches that came before which informed our decisions. In particular, Gitflow, and Github flow are popular git branching strategies. A lot of people have noted the similarity between this and git flow, and the first reaction is often that our acceptance branch is really just the develop branch in the git flow model. But the major difference is that while in Gitflow develop lives forever and is in fact where you cut release branches from, with Flash Flow the assumption is that you are cutting branches to be released directly from master, and when each individual branch is ready it will be merged back to master. In fact, this is identical to Github Flow, the only addition being the notion of the acceptance branch, due to the way we work with our product team.
In our process, it is the job of the product manager to write stories, it’s our job to implement the specified functionality, and then it is up to the PM to verify that our code does what she wants it to do. By creating the acceptance branch from scratch every time we have effortless control over the code that we deliver to our PM.
Flash Flow works well for us. In addition to the basics above, Flash Flow optionally integrates with Pivotal Tracker so that it can auto-deliver stories and mark them when they're deployed to production, and it can also integrate with Hipchat to notify us if a branch doesn't merge.
The codebase is production ready, we've been using it on our team for over six months now. We are now most interested in seeing how it works for other teams, so give it a try or let us know why you wouldn't.
Sorry it took so long to get to this. I like it. It's clear to me but I'm a tainted reader at this point since I understand our process. How about adding a diagram showing open pull requests (some marked Do Not Merge) to make it a bit clearer what ends up in acceptance? The ability to easily exclude a PR seems like an important feature to highlight.