1-twigflow-intro.md

Twigflow (Rebase)

Twigflow is a workflow for managing a small, end-of-the-line branch or interim fork. It is aimed at solo-developers or small teams (2-5 developers who communicate actively and share clients/workflows/infrastructures).

This workflow is based on the process used by Eileen McNaughton for maintaining Fuzion's fork of civicrm-core. I've based this document on Eileen's approach because she has been fairly successful in maintaining an up-to-date fork.

For sake of clarity, this document describes a very specific flow. There are other flows that are consistent with the metaphors and goals of twigflow, but discussing every possibility clearly could fill a long book, so we pick one and stick with it.

Business Context

Suppose you are a consultant working with a mid-to-large open-source project. The project is maintained in a git repo, and you periodically contribute patches for it. You are positioned as a person "in between": you don't own the main project or git repo; but your clients task you with developing patches for it.

Generally, you want to work with the upstream maintainers to prepare good changes and get them merged; but you also need to provide working builds to the client in a timely way. It is difficult to perfectly align the schedules for upstream, for your client, and for your work. This creates structural lag: a period where you need to use a new patch, but it hasn't been officially approved/released yet.

This is a low-grade form of forking. Forking may not be the desired outcome -- but it is the incidental, interim reality. Twigflow does not make forking easy, but (with moderation) it makes forking more manageable.

Basic Concept

The canonical upstream project develops a master branch and periodically tags a new release (e.g. 4.7.14). Each time there is a new release, you should review/curate the list of patches that you maintain (e.g. resolving conflicts or removing obsolete patches). By combining the upstream release with your list of patches, you produce a new twig-branch (e.g. 4.7.14-dolphin-1) which can be used during the interim period until the next release.

A major goal in twigflow is curation or editorial clarity. Each patch you maintain outside the canonical branch is a liability, and you want a firm grasp on the size and substance of these liabilities. If you are providing support or hosting (to your coworkers or clients), then understanding your patchset will help with:

• Retesting/revalidating after an upgrade. Each commit message hints at something you should test.
• Identifying and understanding conflicts (where one of your patches disagrees with an upstream patch).
• Handling edgy schema changes. (The most difficult patches to maintain are ones which modify schema -- because you need to be conscientious about the upgrade procedure.)

Actively curating the patchset enables you to keep the admin work at a steady, predictable level over the long-term.

Limitations

Twigflow does not enable disorganization or laziness. To maintain a healthy project, you still need to work to align your development plans with other developers in your ecosystem. If your culture or organization don't support this, then your twig may accumulate a large number of patches, making it increasingly hard to maintain. If the twig grows too heavy, it might break off!

Twigflow generally assumes that you support one release at a time (e.g. 4.7.14 with your customizations in 4.7.14-dolphin-1). Historical releases (e.g. 4.7.13) are preserved, but doing concurrent development across multiple (e.g. 4.7.13-dolphin-1 and 4.7.14-dolphin-1) would require extra overhead.

Naming Conventions

We'll need to work with various "remotes" and "branches" in git. For this document, we follow some naming conventions:

• The remote "origin" represents the canonical upstream project. (This is the default if you clone the upstream repo.)
• All other remotes are named to match their Github accounts. For example, the remote alice would correspond to https://github.com/alice/the-project.git. (This is the default if you use the hub CLI.)
• New branches are named after their parent. For example, a branch based on master would be named master-foobar. A branch based on 4.7.14 would be named 4.7.14-foobar.
• Tag and branch names should be unique to avoid ambiguity. For example, if origin/master and alice/master both exist, then the master is ambiguous. These examples never use alice/master.

How To: Setup

Identify the public git repository which represents the official/canonical development. For our purposes, let's suppose it's https://github.com/aquaticalliance/the-project.git. This repository should contain a developmental branch (e.g. master) as well as a tag for a recent release (e.g. 4.7.14).

Identify the person or organization who needs to maintain a twig-branch. There is likely to be a small team of people who work with it -- for example, we might have a team working at the Dolphin Action League, and they appoint Alice as the maintainer of the Dolphins' twig-branch. Coworkers Bob and Carol may also work with the same branch.

Alice should setup a git fork based on aquaticalliance/the-project. Suppose she creates the new fork at https://github.com/dolphins/the-project.git with a new branch 4.7.14-dolphin-1:

git clone https://github.com/aquaticalliance/the-project.git
cd the-project
git remote add dolphins https://github.com/dolphins/the-project.git
git checkout origin/4.7.14 -b 4.7.14-dolphin-1
git push dolphins 4.7.14-dolphin-1

How To: Clone

If Alice has already setup the twig-branch, then Bob can clone it directly:

git clone -o dolphins -b 4.7.14-dolphin-1 https://github.com/dolphins/the-project.git

Alternatively, Carol may have cloned the main repo. She can load the twig-branch as follows:

git clone https://github.com/aquaticalliance/the-project.git
cd the-project
git remote add dolphins https://github.com/dolphins/the-project.git
git fetch dolphins
git checkout 4.7.14-dolphin-1

How To: Develop a new patch (general)

When you prepare a new patch, you need to prepare it on top of some starting-point, e.g.

• dolphins/4.7.14-dolphin-1 (the latest, bleeding-edge code in your fork)
• origin/master (the latest, bleeding-edge canonical code)
• origin/4.7.14 (a recent ancestor shared by origin/master and dolphins/4.7.14-dolphin-1)

Regardless of which starting-point you choose, there will be some risks -- e.g. you might prepare a patch on dolphins/4.7.14-dolphin-1, port it to origin/master, and discover a conflict. You might intuitively think that the starting-point influences this risk, but it actually doesn't matter: if the underlying code has diverged, then you will eventually need to resolve a conflict. (The main way to minimize conflicts is to actively minimize divergence.)

Let's get to it.

Suppose that Alice is a solo-practitioner (or that she has a lot of faith in her coworkers, Bob and Carol). Alice (and Bob and Carol) have permission to push changes directly to dolphins/4.7.14-dolphin-1. Alice (or Bob or Carol) should simply push their revision:

git checkout 4.7.14-dolphin-1
git pull dolphins 4.7.14-dolphin-1
vi css/main.css
git commit css/main.css
git push dolphins 4.7.14-dolphin-1

Note: Framing the code and description of the commit is very helpful in a healthy twigflow. See the section "How To: Develop a new patch (commit design)" for more discussion.

Of course, the patch also needs to be sent upstream to minimize divergence. You can cherry-pick the change and send it upstream:

git checkout origin/master -b master-okbutton
git log origin/4.7.14..dolphins/4.7.14-dolphin-1
## Review the log. Identify "abcd1234" and "ef4321ab" as the important commits.
git cherry-pick abcd1234 ef4321ab
git push alice master-okbutton
hub pull-request

Based on feedback from the code-review process, you may decide that the patch is not good enough. Simply add new commits to the branch.

git checkout master-okbutton
vi css/main.css
git commit css/main.css
git push alice master-okbutton

Pay attention to any new commits. Each of these should be cherry-picked and applied to your twig-branch:

git checkout 4.7.14-dolphin-1
git pull dolphins 4.7.14-dolphin-1
git log master-okbutton
## Review the log. Identify "dcba1234" and "fe4321ab" as the important commits.
git cherry-pick dcba1234 fe4321ab
git push dolphins 4.7.14-dolphin-1

How To: Develop a new patch (commit design)

When preparing a patch, the important thing to remember is: You WILL look at this patch again. Probably multiple times.

• With a bit of skill or luck, the patch is quickly accepted upstream. You'll only look at it once or twice.
• With a bit of misfortune or misguided hackery, the patch will linger and never be suitable for upstream. No one else will take responsibility for it, so you'll be looking at it every month for the next three years.
• If you do this repeatedly, then statistics play a role: some patches will get quickly accepted upstream, and some will linger. It's inevitable, and there's no shame in it. It's just a part of the software business.
• When you look at the patch in the future, you'll want to know what it does and why you should care. You'll want to know if the patch still works.
• When you submit the patch for review upstream, they'll want to know what it does and why they should care. They'll want to know if the patch works.

The best bet is to write a clear commit-message explaining the patch, i.e.

• The first line should summarize the change.
• The message should explain why you care.
• The message should explain what it does.
• The message should explain how to test it. (This could be a list of 5 manual steps. Or, even better, if you've included a unit-test, then just mention the test-name.)
• Don't write a full book. Don't repeat yourself. Use decent grammar.
• If there are supporting materials (such as forum discussions, StackExchange questions, or JIRA issues), then include the URLs.

These properties are simply good, period. They help anyone who does review -- whether that person is future-you or your upstream colleagues. For example, this might be a good message:

ScheduledReminders UI - Move the "From" email field

When defining a scheduled reminder, you set the "From" email address; however, this field is
in a really weird, nonsensical place -- and my brain suffers a seizure everytime I look at it.
The "From" should be next to the email body.

To test this patch:
1. Go to "Administer => Communications => Scheduled Reminders" and make a new one. 
2. Observe: The "From" field should appear right above email message.
3. Play with the "[x] Email" checkbox. See if it works sensibly.
4. Save and reopen. Make sure the "From" address is retained.

See also https://nonce.stackexchange.com/why-dont-fields-make-any-sense-on-scheduled-reminder-screen

A few more tips:

• When patching code, you sometimes find some pre-existing ugliness which you want to clean up. Treat these as two separate patches -- the first patch does some cleanup/rearranging, and the second patch changes the behavior.

How To: Upgrade

Suppose the upstream project has produced a new release, bumping up from 4.7.14 to 4.7.15. You need to build a new branch based on the new release.

git fetch origin
git fetch dolphins
git checkout dolphins/4.7.14-dolphin-1 -b 4.7.15-dolphin-1
git rebase -i origin/4.7.15

The interactive rebase process provides an opportunity to review all the changes that have been accumulated. A few changes you may make:

• If one of your patches has been accepted or replaced upstream, then remove it. There's no need to continue porting it forward.
• If one of your patches has multiple commits (e.g. due to revisions from code-review), then group these together and consider squashing them. It's hard to get your head around all the patches if several of them are basically cleanups of the same thing.

The rebase process may identify conflicts. If these come up, follow the built-in instructions from git to resolve them.

Once you've finished rebasing, take a look back at the commit messages for all the changes:

git log origin/4.7.15..4.7.15-dolphin-1

Assuming you've followed the style-guide, each of these commit message will have notes on how to QA.

Finally, after you've reviewed the new branch, publish it:

git push dolphins 4.7.15-dolphin-1

Question: How many people should collaborate on the same twig-branch?

I'd expect twigflow to work best with solo-practitioners and small teams of 2-5 developers (who have shared clients/workflows/infrastructures and active communication). Large teams or cross-organizational teams are more problematic. A central value of twigflow is the periodic curation, and curation is only possible if one person can wrap her head around each of the changes in the twig-branch... but that doesn't scale well. Moreover, you would expect that larger collaborations would tend to increase divergence (i.e. more commits/SLOC/use-cases/people)... which would lead to more merge-conflicts.

For broader collaboration, you should be using the code-review and release process of the canonical upstream project.

Question: Should I use pull-requests internally (to manage changes within the twig-branch)?

You could probably make this work, but it will require sharper git skills.

For purposes of this guide, I've assumed that you don't want that for a simple reason: twig-branches work best in small teams, and it's easier to trust coworkers in a small team, and it's harder to justify bureaucracy in a small team. That... and it would be harder to present clearly...

Question: Why should we use "git rebase" instead of "git merge"?

It's harder to curate the patch-list in merge-based workflow because merging produces a more complicated history. (Ex: Merging creates merge-commits and conflict-resolution-commits. Ex: There's no opportunity to squash tweaks in a merge-based workflow.)

Question: Why should we use "git rebase" instead of a patch-folder?

A very similar process would be to maintain a folder or text-file with a list of patch-files. This is also pretty reasonable. Trade-offs:

• A patch-folder doesn't require as much git skill.
• A git branch can be cloned/deployed more directly.
• A git branch provides some extra functionality (commit-messages, squashing, diffs, logs).

Question: What happens if upstream rejects my proposal and takes a very different direction?

Eventually, you'll want to upgrade. When you rebase, remove any changes that you made which would conflict -- and figure out how to transition your installations.