The relationship between Work in Progress and Pull-Requests; and why Trunk-based development is a better approach
Poorly managed work in progress (WIP) is a common cause for low performance in development teams. When you have a team juggling too much work, it engages in frequent, expensive context switching which reduces quality and slows delivery. Your team can get demoralised because although they’re working hard, they’re not managing to finish work.
This reduction in quality tends to have a compounding effect on WIP as you will get unexpected bugs, re-work and outages; which in itself causes more WIP. Unplanned work is poison for productivity and predictability.
Managing WIP involves looking at how work flows through your system, trying to make it go quickly, smoothly and predictably. Identifying bottlenecks & constraints allows you to manage them more effectively and focus your efforts on improving them and making sure only the highest priority work goes to them. Any efforts to improve efficiency outside these constraints is waste because it’s the bottlenecks in your system that dictate the pace. With a focus on quality you can reduce the amount of unplanned work which helps you manage WIP.
There is no one true way of working, the best method depends on your team and the context they’re in. Managing WIP, and having a smooth flow of work is not a negotiation if you are serious about managing an effective team.
Pretend you are the lead of a 4 person development team. How your team approaches creating value obviously has a big impact on its ability to deliver it, yet many teams copy other team’s rigid models, rather than thinking about the context they are in and tailoring the process to it.
This is an often-cited, popular way for teams to work.
- Each developer picks up a ticket/story/task
- Developer writes code on a branch. This allows them to work in isolation, focused, and not affect other team members with their own work which may not be completed.
- When the developer believes the work is finished they raise a pull request. When another developer is free, they can take a look at the work and then initiate a feedback cycle with the person who raised the PR until it is of sufficient quality.
- The code is merged, deployed to customers
Often the reasons people adopt this model is it allows a degree of gatekeeping and socialising knowledge. It’s argued that it can help prevent mistakes being shipped and maintain code quality.
For a team of 4 developers, having at least 4 tickets in play at a time feels like high WIP. Chasing 100% “resource” allocation is not a desirable management goal because all systems need a level of slack to cope with unexpected work. If a team is running at full-capacity it is inflexible, and inflexible things tend to break under pressure.
If you raise a PR and none of your colleagues are free, you may be tempted to pick up another ticket. The WIP of the team has now increased to 5. You can easily imagine, and have probably experienced scenarios where this is a slippery slope to climbing WIP. How many times have you heard:
It’s done, it just needs to be reviewed
The work is not done until it is actually delivering value
The feedback cycle in the PR can take some time with asynchronous back-and-forth, so you may end up having to context switch between multiple tickets at a time. In lean terms, the process has caused a bottleneck.
Even with a team of only 4 you can easily end up in a situation where they’re context switching between the story they’re assigned to, but also reviewing lots of other of code. The personal WIP for each team member can climb steeply depending on how back-and-forth the review process is. As the WIP increases for each person, the ability to get code reviewed diminishes.
PRs are often toted as remote friendly due to their asynchronous nature. This is true, but you have to accept then you will have a lot of unfinished work in play, which will raise your team’s WIP. If I were to be managing a team working like this, I think it would be prudent to ensure that you only have 2-3 developers working on stories at a time, giving your system some slack so reviews can happen quicker, control WIP, improve quality and you can deal with unplanned work.
The longer your code lives on a branch, the less integrated you are with the rest of your team. Whilst the 4 of you are described as a team, you are all working on different versions of the code, as individuals.
If you find it hard to get your colleagues to review your changes, this slow integration compounds and teams that adopt this system often have to deal with challenging merge conflicts. In the previous example this way of working can lead to multiple PRs that need merging which can result in multiple merge conflicts having to be resolved. This is waste.
The review back-and-forth can last minutes, hours, days or even weeks. In the worst cases it involves a lot of re-work. Once the developer falls in to the trap of picking up another story whilst their other is in review, WIP can quickly spiral out of control.
Plenty of successful teams ship software like this, it’s not unworkable, but I’m not convinced it’s simple or efficient. I think even when it is managed extremely well it still causes a lot of waste.
What is the most obvious way to reduce WIP?
- Do less work concurrently
- Concentrate on finishing work
- Increase quality to reduce unexpected re-work
So instead of having 4 developers work on 4 tickets, lets instead have 2 developers pairing per ticket.
By having 2 heads work on a problem it liberates us from having to gate-keep the main branch. We should trust that 2 developers on our team can take ownership of a problem from idea to production. Therefore, they can commit their code straight to main. This simplifies the flow of work. We no-longer have a wasteful bottleneck to shipping value.
How many then? 3, 4? Or worse, can it only be put into the main branch if it is blessed by the lead? If you’re a lead making yourself a bottleneck, you’re not doing a good job. You’re not spreading your knowledge, you’re not coaching well enough, you’re increasing bus-factor and you’re hurting the productivity of your team.
By integrating frequently through the day, the chance of merge conflict is low and even when it happens it is typically trivial to fix. Far easier than having to merge a week’s worth of changes back into main.
This is true continuous integration, the team can consistently see what everyone is working on which helps collaboration. No longer will you hear excuses like:
I’d like to refactor the blah blah, but I can’t because I know Alice is working on a PR related to it.
By removing the barriers to changing code and having a tight integration with your team (rather than disparate branches) you can now fearlessly refactor.
It’s an assumption, but not an unfair one that the quality of code should be higher if 2 people are working on a change. High internal-quality makes changing our system simpler and should reduce the chance of unexpected re-work.
If you practice continuous delivery, it means you’ll release smaller changes frequently throughout the day, which reduces risk and makes recovery in the event of an error easier compared to diagnosing a pull request which may have been worked on for a number of days (or weeks!) rather than hours.
The idea of code-review and socialising ideas is now constant and easily available. Developers don’t have to write large amounts of code and then inconvenience another developer, forcing them to context switch to give feedback on your ideas. Instead you have a colleague that you can easily bounce ideas with.
To make sure the socialising of code is spread amongst all 4 developers, we should ensure that we rotate the pairs frequently. We’ll also do a “code review” session every week for a couple of hours where we can talk about our code in broader strokes, in a more holistic way; rather than focusing on an isolated change in a PR.
If the idea of pushing directly to main is uncomfortable for you, it’s likely you’re working on an unsafe system. For TBD to work, you need to engineer your system so that it is safe to work on. You’ll need things like
- Good alerting
- Easy or automatic rollbacks
- An excellent test-suite
- Observability
- “Test in prod”
— Charity Majors
Gatekeeping around PRs and releasing is often reliability-theatre. Nothing can save you from mistakes, they’re inevitable. What you need to focus on is your mean time to recovery.
To commit to main directly, safely & frequently you need an excellent, fast test-suite and a well configured CI-pipeline. Developers need to be disciplined and only commit code they’re happy to put on the main branch.
Every developer is touching mainline, so all features grow in the mainline… which acts as a communication point. With CI, the mainline must always be healthy, so in theory (and often in practice) you can safely release after any commit.
— Martin Fowler
To live with the constraints of wanting to continuously integrate but not make “bad” commits, you need to find ways to break your work down into small, achievable pieces. This discipline makes work more achievable and will reduce risk. By working like this I’d expect our team of 4 devs to be pushing to main at least 8 times a day.
If you do decide to try TBD, be prepared for some friction and growing-pains if you have a team unfamiliar working in this way. A few years back I wrote about how I introduced TBD to a team that was previously familiar with a PR-based approach.It will be a journey of uncovering issues in quality, bottlenecks and collaborating to find ways to improve them.
TBD may expose some uncomfortable shortcomings in your system, and the quality of your code and architecture. I’ve noticed that often a PR flow masks problems like:
- Developers finding it hard to break problems down in to smaller, less risky changes. They have not learned how to work in small batches.
- Low internal quality. If your system suffers from a lot of inappropriate tight coupling you’ll tend to trip over each other’s toes when working together on the main branch.
- Poor/inadequate tests. Lots of teams use the “CI server” as a crutch for poorly written tests that they run slowly and infrequently. Sometimes the system is designed in such a way that important tests are impossible to run locally.
- If you’re pushing every hour, you need to be able to test locally and be confident your change is shippable and won’t affect your colleagues negatively.
This is a journey that many factories (real factories!) have gone through. To work in a lean way, managing WIP requires you to look at how you do work and commit to increase quality from start to finish as a team.
TBD is not niche or unusual. Plenty of big businesses, in well-regulated industries work with pairs of developers committing many times a day to do the main branch. Chromium releases every day and does not use branches.
In this short video, Dave Farley shows a fin-tech company in a well-regulated space working with TBD
Usually these high-performing teams ship these changes extremely frequently to customers, so they can get real feedback quickly and build superior products.
Whilst pairing may seem like an expensive frivolity, the workflow is simpler, more focused and of higher quality. You’ll find that this approach can out-perform the same team working as individuals on branches. There is plenty of evidence to back up this assertion.
— Daniel Terhorst-North
To those unfamiliar with TBD, it may seem risky and even unprofessional, but it forces you to work with more discipline, adopt best practices which will make your development team more resilient and the work simpler.
— Continuous Delivery: Jez Humble & David Farley
Pull-requests work excellently in the open source world where you cant trust external contributors . For your day job though?
— 2017 State of DevOps Report
Think about your context, examine how work flows through your system, set a WIP limit and stick to it. Find ways to improve the flow and your team’s performance will improve.