on feature branches

gistfile1.txt

Assumptions 

1. We're talking about software that is motivated by a problem that is not a pure technical one: a business problem, a community problem, etc.

2. That we'll be operating the software not just writing it.  In practice this means we're responsible for the software's outcome and that the software will be connected to the network.

Assertion

Given those assumption, software that is instrumented with feature flags  is significantly more operational. 

Reasonable place to start the conversation?

I think what was being debated was a bit more than this. The question was whether you should do feature branches that diverge from master (or whatever you call your main branch) for a significant span of time/commits, and then are merged back in.

One side of the debate says: Yes, feature branches are a good idea. Let's call this the “branch” side.

The other says: You shouldn't do feature branches; instead you should do feature flags that control whether parts of the code are operational. Then you can develop a feature incrementally on master (let's just call it master) and simply turn it on when you're ready. With this approach, you don't need feature branches, and because of the potential for merge conflicts, you should avoid them.

The rebuttal from the branch side is: Feature flags sound like a lot of overhead; are they worth it? Merging can't be that bad.

That's how I see the debate. I'm mostly on the flag side, but I'm trying to frame both sides fairly here and there's definitely a tradeoff. Thoughts?

BTW, I want to make clear that “feature flags” aren't just about user-facing features. The same concept can be adapted to back-end changes—e.g., API or schema changes.

A great example from Kent Beck:

We frequently migrate large amounts of data from one data store to another, to improve performance or reliability. These migrations are an example of succession, because there is no safe way to wave a wand and migrate the data in an instant. The succession we use is:

1. Convert data fetching and mutating to a DataType, an abstraction that hides where the data is stored.
2. Modify the DataType to begin writing the data to the new store as well as the old store.
3. Bulk migrate existing data.
4. Modify the DataType to read from both stores, checking that the same data is fetched and logging any differences.
5. When the results match closely enough, return data from the new store and eliminate the old store.

You could theoretically do this faster as a single step, but it would never work. There is just too much hidden coupling in our system. Something would go wrong with one of the steps, leading to a potentially disastrous situation of lost or corrupted data.

Source: https://www.facebook.com/notes/facebook-engineering/software-design-glossary/10150309412413920

Beck uses this to illustrate the concept of succession: “the art of taking a single conceptual change, breaking it into safe steps, and then finding an order for those steps that optimizes safety, feedback, and efficiency.”

IMO succession is underrated, which is why I'm eager to spread the word here.

My confusion around this discussion is that I have a hard time imagining living without either branches or feature flags, so treating it as an either/or dichotomy feels like dueling strawmen. Also, "branch" appears to be a matter of degree - anything short of RCS-style "I have the lock on this source file" is going to require some kind of merging, so the question just seems to be how much divergence in time and/or code is acceptable.

So I'd frame the discussion as a few linked questions.

First, are feature flags valuable? Are they worth the cost of added cyclomatic complexity etc that they bring?

Second, if you do pervasively use feature flags, should you expect your Mean Time To Merge to be reduced? By how much?

Third, if you buy the above two things, is it reasonable to start to establish policy or social norms that demand an extremely short Mean Time to Merge?

And fourth, what effect does having a short MTTM have on processes and conventions for code review?

First, are feature flags valuable? Are they worth the cost of added cyclomatic complexity etc that they bring?

That was my initial question. I'm happy to assert that feature flags lead to software that is superior to operate, but I wanted to make sure we agreed on that point.

Additionally, and this is where talking about software in the abstract is hard as we aren't that sophisticated an industry yet, I'd assert cyclomatic complexity is only a useful theoretical measure that in practice is often swamped in scale by operational and organizational complexity. (still holding off on getting into the full features flags rationale as I think we've got the common ground there)

Or to say that a different way, the work we do to ensure our software is reliable, resilient, correct, testable, monitorable, secure, usable and efficient is a significant, non-negotiable piece of the complexity of our software, and in that context I find cyclomatic complexity less useful.

Second, if you do pervasively use feature flags, should you expect your Mean Time To Merge to be reduced? By how much?

Third, if you buy the above two things, is it reasonable to start to establish policy or social norms that demand an extremely short Mean Time to Merge?

This is starting to get to the crux of the trade offs point I was making. If you use feature flags as your primary form of branching you should not expect by default that your mean time to merge will be reduced, but if you can assume the a pre-existing norm of frequent commits, then yes your mean time between merges is reduced.

Are there benefits of having a long MTTM? Or can we take that out of the discussion as a design goal?

If you use feature flags as your primary form of branching you should not
expect by default that your mean time to merge will be reduced, but if you
can assume the a pre-existing norm of frequent commits, then yes your mean
time between merges is reduced.

So you're saying that one benefit of doing all development on master is that (if you push frequently) it forces you to merge frequently? With long-lived feature branches, you'd need a policy or social norm to achieve low MTTM.

I can't think of many/any valid reasons to have a long MTTM. It tends to encourage revolutionary rewrites of systems rather than evolutionary changes. Shorter MTTM -- to be hand-wavy -- ensures that the conversations about changes happen in small, frequent, digestible chunks.

I've also never seen a successful policy that had long-lived feature branches merged regularly. At best, I've seen it done daily, but even then if the merge is difficult it gets pushed off for a day or a week. LLFBs encourage the notion that developers are working in their own isolated environment.

Oh! And @jasoncrawford Beck's method for migrating data is old. http://www.amazon.com/Migrating-Legacy-Systems-Interfaces-Incremental/dp/1558603301 Stonebraker wrote a book about it decades ago. :)

Just to suggest some answers to @avibryant's questions:

1. I think feature flags are valuable and worth it, although probably only at a certain level of organizational scale. We're not using them yet at my two-man startup. I would use them in a 100-engineer org. I would probably even use them in a 10-engineer org. (And of course you need them if you want to do A/B tests.)
2. Feature flags don't automatically shorten your MTTM, but they make it possible to shorten your MTTM. That is, if you don't use feature flags, you'll be forced to use feature branches, and that will lengthen your MTTM (or at least, keep it long).
3. I think it's good to show everyone how it makes everyone's lives easier if you have more frequent, smaller merges. Ideally, IMO, this is not done with social pressure, but by letting people deal with the consequences of their own choices. Hopefully after one or two big hairy merges, they learn the lesson.
4. Smaller merges just make code reviews easier, which means they're more likely to get done, happen quickly, and be reasonably thorough.

Martin Fowler says: if something hurts, do it more often.

@jaypalominodb, thanks for the reference!