Git branching for Agile workflows in Drupal
Agile software development is a popular method used to enable adaptive planning and continuous improvement via flexible response to change. Typically, a product owner will describe a set of clearly defined user stories ("as a site visitor, I want to log in via Facebook so that I can comment on an article") and list them in order of priority. A scrum master will work with a team of developers to deliver these features during short 2-3 week sprints which are followed by a demonstration and review, after which the product owner will either declare the user story complete or ask for corrections. New user stories will be added, the backlog of pending user stories will be reprioritized, the scrum master will choose a set of tickets which can be completed in the upcoming sprint, and the developers will start to work on the next round of improvements. Much has been written on the theory and practice of Agile, including books, online articles, and formal trainings. What I'll cover here is how to apply that theory to branch management in git.
To make this work, it's best to use an issue tracker with a ticket for each user story, and to make all changes to your site on a separate git branch labelled with that ticket number. The changesets should be self-contained, and perform all necessary database changes in code via update hooks, features (Drupal 7), or the configuration management system (Drupal 8). Any further steps necessary to test or deploy the new version should be clearly described in the ticket description.
Once all your updates are in code, you're ready to use git to prepare for and manage your testing and deployment process. You'll need to adopt specific git practices to make this work smoothly. The rest of this document assumes that your workflow has the following needs:
When is this useful?
- You're using an agile workflow, with sprints and regular deployments
- You have a team of multiple developers, a scrum master (project maintainer), and a product owner (client representative)
- You want to do regular releases of whatever tickets are ready
- You review releases (continuous integration, user acceptance testing, change approval board, etc) before deployment
- You occasionally need to do hotfixes for urgent problems directly on the production site
- Before a deployment, you occasionally need to revert from a release branch a ticket that didn't pass review
Drupal best practices
In Drupal 7, the best practice is to export all possible configuration using the Features module, so that it can be tracked and versioned in git. After setting up your site's views, content types, variables, and other configuration, you would use the Features module to export these to code as a series of special modules in sites/all/modules/features/ and from there commit them to git. It's simplest to group these by functionality (for example, you could create one features module which contains all the configuration necessary for the blog section of your site).
After making changes, run drush features-update-all to export the configuration from the database to code. To bring in changes after doing git pull you would run the command drush features-revert-all to import from the version in code to the database. You'll need to return to the Features module to add new components of your site (such as a new view) before they will be included in the set of configuration saved in a features module.
To learn more about exporting the configuration of your Drupal 7 site using features, see the documentation at https://www.drupal.org/docs/7/modules/features/bundling-site-settings-using-features.
In Drupal 8, most configuration is stored in YAML files, making this much simpler.
By default D8 configuration is stored in the public files/ directory under a difficult-to-guess directory name. Wherever possible, add the following to settings.php or settings.local.php to define the config directory, which should be someplace outside the webserver docroot. (This means your website docroot will be a subdirectory of your git root folder, giving you a place above that to store test scripts and other files outside the docroot.)
// Define the sync configuration directory $config_directories[CONFIG_SYNC_DIRECTORY] = '../config/sync';
Once this is done, run the following after making local changes on dev machines: drush config-export
You can now commit and push these changes to your git repository. To import these on test, staging, and production environments, do a git pull and then run the following: drush config-import -y
Like with D7 features, the suggested workflow is to avoid making changes to these files directly on the production website, but instead to deploy by importing from YAML. Changes made directly on prod will be discarded.
You can read more about the basics of configuration management in Drupal 8 here: https://evolvingweb.ca/blog/using-configuration-management-and-git-drupal-8.
It's also possible to use configuration files during site install to spin up a site instance without a database, as described here: https://evolvingweb.ca/blog/restoring-drupal-8-site-configuration-files.
You should also have a convenient way to fetch a copy of the production database from dev & staging environments (eg via drush aliases), and use the stage file proxy module (https://www.drupal.org/project/stage_file_proxy) to avoid having to sync public files from prod back to dev and stage environments until they're needed. Use drush sql-sync --sanitize to reset passwords and email addresses in the user tables when copying the database (note that you can also capture all mail sent from dev & stage by sending it to a utility such as mailhog for local review).
Git branching strategies
To explain my preferred git workflow, I'll start by explaining some other,simpler workflows that can be used by smaller teams. My reference for this is the tutorial from Atlassian: https://www.atlassian.com/git/tutorials/comparing-workflows
I'm assuming knowledge of basic git concepts like branching, committing, and pulling. If these are new to you, I suggest taking the time to read an introductionary tutorial such as https://www.atlassian.com/git/tutorials/setting-up-a-repository.
This is the simplest possible git workflow.
- There are no branches, everyone always commits to master
- Works best when each part of project has only one developer (eg one frontend, one backend)
Feature branches AKA GitHub-style pull requests
Once multiple people are working on a codebase at the same time, it's necessary to use a more sophisticated model. Many open source projects on GitHub work this way, with one person responsible for testing and approving all proposed changes.
- Developers start a new branch to work on each feature, sending a pull request when ready for review
- Developers merge the master branch back into their dev branches on a regular basis during development to keep up with other changes
- Maintainer reviews proposed changes and merges (pulls) feature branches into master when ready
- Maintainer occasionally tags a commit as a named/numbered release
- Works best for simpler projects without a need for external reviewers from outside the dev team
Once your project requires a review process for each set of new features, some of which may depend on others to be completed at the same time, it's necessary to create branches for each release so that these can be tested as a group. It follows that you might need to remove a feature from a release if it isn't accepted during testing, and that you'll occasionnally need to urgently fix the production version of the code without waiting for your usual release cycle.
- Naming convention which allows for dev, feature, hotfix, and release branches, with defined procedures for updating them
- Described by Vincent Driessen in 2010: http://nvie.com/posts/a-successful-git-branching-model/
- Works best for multiple developers using an agile process as described above
Git best practices
Make sure every commit message includes the ticket number, both to create pointers from your issue tracker, and to allow you to find commits related to a given ticket at a later date. You can even set up your CI/automated testing scripts to reject commits without numbers, and create a script in .git/hooks/prepare-commit-msg.sh to insert this automatically in the commit message template, such as this one: https://gist.github.com/mvc1095/00adf7cadac587d056ad798d894460ea#gistcomment-2016202
Set mergeoptions = --no-ff for the master and dev branches in .git/config so that every merge to those branches has a merge commit for later tracking.
Working with branches
List of branches
- master: used only on the production website
- dev/123 or feature/123: used to add a feature or fix a bug, as defined in ticket 123 in your issue tracker (branched off of dev; will be merged back into dev and deleted when resolved)
- dev: used to stage all completed feature branches before they're released (was initially branched off of master)
- release/03: used to stage a set of features (a snapshot of the dev branch) for review and deployment (branched off of dev; will be merged into master and deleted when deployed)
- hotfix/234: used to fix an urgent bug (branched off of master; will be merged back into master and deleted when resolved)
# update local copy of all remotes, removing branches which have been deleted $ git fetch --all --prune Fetching origin # list all local and remote branches # (shows hash and commit subject line, remote tracking branch, and status) $ git branch -vva dev 88c87e0 [origin/dev] Merged branch 'dev/456' into dev dev/234 88c87e2 [origin/dev/234] Fixes footer; refs #234 * hotfix/345 88c87e3 [origin/hotfix/345: ahead 3, behind 1] Urgent homepage fix; refs #345 master 88c87e4 [origin/master: behind 10] Merged branch 'release/02' into master release/03 88c87e6 [origin/release/03] Merged branch 'dev' into release/03 remotes/origin/dev 88c87e0 Merged branch 'dev/345' into dev remotes/origin/dev/123 88c87e1 Fixes sidebar; refs #123 remotes/origin/dev/234 88c87e2 Fixes footer; refs #234 remotes/origin/HEAD -> origin/master remotes/origin/hotfix/345 88c87e7Urgent homepage fix; refs #345 remotes/origin/master 88c87e6 Merged branch 'release/02' into master remotes/origin/release/03 88c87e5 Merged branch 'dev' into release/03
Start working on a feature
git checkout -b dev/123 # create new branch # work happens here git merge dev # update feature branch # more work happens here # can rebase here if desired to clean up commit history git push -u origin dev/123 # push branch to remote for review
Note: rebase when you want to rewrite history (eg remove commits of debugging code); merge when you want to preserve history. You should only rebase a local branch which isn't yet pushed to origin and shared with others! If you're not already familiar with rebasing, ignore this section. If you really want to learn about this, this tutorial is helpful: https://www.atlassian.com/git/tutorials/merging-vs-rebasing
If your goal for rebasing is simply to reduce clutter in your git log, consider using git merge --squash when merging feature branches into dev to combine all changes into one new commit. This is safer than rebasing, and won't cause problems for others.
Review and accept a feature
git checkout -t origin/dev/123 git merge dev git diff dev # show all changes with respect to the dev branch git diff dev --stat # list of changed files, with number of lines added and removed git diff --name-status # list of changed files, tagged as modified, added, or deleted # sync database from prod # testing happens here git checkout dev git merge --no-ff dev/123 # force creation of a merge commit (so you can revert if needed) git branch -d dev/123 # delete local branch git push origin --delete dev/123 # delete remote branch
Prepare release branch
Once enough tickets have been closed and their corresponding feature branches merged into the dev branch, create a release branch and deploy to your staging environment for review.
On local machine
git checkout dev git branch -b release/04 # sync database from prod # perform final local tests of deployment steps git push -u origin release/04
On staging server
# sync database from prod git checkout -t release/04 # run deployment steps
On local machine
git checkout master git merge --no-ff release/04 git tag release-04 git push --tags
On production server
git pull # run deployment steps
Hotfix production site
On local machine
git checkout master git branch -b hotfix/345 # fix up code # testing happens here git checkout master git merge --no-ff hotfix/345 git tag release-04-hotfix-01 git push --tags
On production server
git pull # run deployment steps
Resolving merge conflicts
Your tickets should be small enough in scope that merge conflicts don't happen very often. Here are three things you can do when they come up:
- Run git status to list files with conflicts, then edit each one manually
- Configure a merge tool which can help with three-way merges (see git mergetool --tool-help for supported options on your platform)
- If you get lost halfway through: git merge --abort
There are lots of resources on resolving conflicts online. One introductory tutorial is here: https://githowto.com/resolving_conflicts
Here's an example of working with a three-way merge using vim: http://www.rosipov.com/blog/use-vimdiff-as-git-mergetool/
Removing a feature which didn't pass review
git checkout dev git revert -m 1 88c87e0 # revert the merge which brought in feature branch dev/456 git push git checkout dev/456 git merge dev # update feature branch to include the revert git revert 88c87e1 # revert the revert itself to return to the code that needs further work
A lengthy description of this process is available here: https://www.kernel.org/pub/software/scm/git/docs/howto/revert-a-faulty-merge.html
Finding the commit which made a given change
A common complaint about gitflow is that it fills your git history with merge commits, but in fact git allows you to exclude these with git log --no-merges (or to show only merges with git log --merges).
git log git log --since="2017-01-13" --until="yesterday" git log -3 # shows last 3 commits git log --grep="refs #123" # searches commit messages git log -S"needle" # searches diffs for fixed string "needle" git log -G"ne+dle" # searches diffs for regex "ne+dle"
Managing multiple remotes
You might want to use multiple remotes, for example a local GitLab instance, GitHub for automated integration with Circle CI, and a git instance at your hosting provider (eg Pantheon). You can do this by creating a placeholder remote called "all" with multiple pushurl URLs defined.
[remote "gitlab"] url = email@example.com:projects/clientproject.git fetch = +refs/heads/*:refs/remotes/gitlab/* [remote "pantheon"] url = ssh://firstname.lastname@example.org_hex.drush.in:2222/~/repository.git fetch = +refs/heads/*:refs/remotes/pantheon/* [remote "github"] url = email@example.com:mycompany/clientproject.git fetch = +refs/heads/*:refs/remotes/github/* [remote "all"] url = firstname.lastname@example.org:projects/clientproject.git fetch = +refs/heads/*:refs/remotes/all/* pushurl = email@example.com:projects/clientproject.git pushurl = ssh://firstname.lastname@example.org_hex.drush.in:2222/~/repository.git pushurl = email@example.com:mycompany/clientproject.git
License: CC 4.0 BY SA (Creative Commons Attribution-Share Alike)