Skip to content

Instantly share code, notes, and snippets.

@djspiewak
Last active October 22, 2023 04:30
Show Gist options
  • Save djspiewak/8c3c0eadabf16c2fddda796531c3d5ec to your computer and use it in GitHub Desktop.
Save djspiewak/8c3c0eadabf16c2fddda796531c3d5ec to your computer and use it in GitHub Desktop.

Greasing the Skids: Building Remote Teams

In the wake of the virus that-must-not-be-named (which most people misname anyway), it seems like everyone and their cat has posted some sort of opinion or how-to on making remote work, work. This is a good thing! Working remotely, particularly full-time, is hard! I've done it for my entire career (aside from an odd 14 month office period in the middle that we shall not speak of), but more relevantly, for the past two years I've been responsible for building, managing, and enabling an entirely remote team, distributed across nine timezones. Remote teams don't just happen by installing Slack and telling everyone to work on their couch: they require a lot of effort to do correctly and efficiently. But, done right, it can be a massive multiplier on your team efficiency and flexibility.

Here's how we do it. I'm going to attempt to structure this post more towards management than engineering, and so I apologize in advance if I assume terminology or knowledge which isn't entirely universal.

The Importance of Asynchrony

The most important concept to grasp regarding distributed teams is asynchrony. When someone walks up to another team member in an office to ask them a question, it's a synchronous exchange: the second party to the conversation is compelled to stop what they're doing and focus their attention immediately entirely on the first party. When you hold a meeting, you're requiring all of the people in the meeting to synchronize their schedules within that time block. Same concept applies to a phone call.

Synchronous interactions are important for a number of reasons, most of them relating to delivery schedules and dependencies. It's not at all uncommon for one person to not know how to proceed with their work without the input of another person. This first person is then blocked on the second person, and the fastest way out of that mess is a synchronous exchange. In theory, meetings are supposed to serve this purpose on a larger scale.

Unfortunately, synchronous interactions are also immensely costly. Just as a quick example, if you call a meeting between 5 senior developers in the San Francisco office, with a single project manager to "facilitate", and that meeting lasts exactly one hour, the company has spent something on the order of $600 USD just on that one meeting. That's a very expensive way to decide that the widget should be pink rather than mauve. Even synchronous interactions between just two people can be very very costly.

More critically, this cost expands explosively when either one of the parties has to "sync up" with the other one. Say, for example, if two people in different timezones need to have a synchronous interaction in order for one or both of them to get their work done. Neither of them are going to be working on the same schedule, and if the timezones are sufficiently disparate, one of them may be asleep for a large portion of the other's day! In general, it is very common for an American needing a meeting with a European to have to wait an entire work day for the schedules to realign (usually the following morning, North American time). The same thing is, of course, true in reverse, as well as for interactions between Europe/North America and Asia. Even within continents, things can be quite awkward. Three hours between the East and West coast may not sound like much, but when one person is getting a late lunch and the other person is finishing work for the day, it starts to become a serious hassle.

Time is lost as one person cannot proceed forward in their work without a synchronous interaction with another person. Multiply that exponentially by the size of your entirely distributed team, as everyone is waiting for everyone else, and you see how things grind to a halt very quickly.

The solution is to optimize your process and your infrastructure for asynchrony. The most well-known example of this is Slack (and tools like it; we use Discord). Tools like Slack encourage your team to build up a culture of asynchronous interaction: instead of picking up the phone, just send someone a message if you don't need an immediate answer. Counter-intuitively, this means that issues take longer to resolve. The advantage is in the fact that, while issues are holding over unresolved, people are able to continue doing their work as efficiently as possible. It also minimizes forced interruptions in efficient flow, since Person A asking Person B a question doesn't force Person B to immediately drop what they're doing and pivot their full attention to Person A, and thus the inefficiency of context switching is reduced.

Encouraging this kind of asynchronous interaction has a very subtle effect on the entire team: it pushes everyone to structure their schedule and their needs in terms of multiple non-dependent tasks. If Person A is concerned that one of their tasks may end up needing input from Person B, then they often take on a second task that they can switch to while waiting for Person B to answer their Slack message. This is something that people just do very naturally when a pervasive culture of asynchronous interaction is encouraged.

Taking this idea further, you want to create an infrastructure and process around integration of work which is never blocked on anyone else. The most well-known example of this in engineering circles is the "pull request", which is a method by which one person can submit work for approval and integration into the product while simultaneously freeing that person up to move on to other tasks. When one person opens a pull request, they don't have to wait for other people to approve it in order to move on to some other task: they immediately move on, since their work on their previous task is substantially complete (usually). Additionally, approving members of the team (those who will review and ultimately merge the pull request) themselves do not need to interrupt what they are doing in order to give feedback on the work: they can review the pull request at a time which is convenient to their schedule. Again, maximizing asynchrony.

In general, as a manager of a distributed team, you need to treat situations where one person is hard-blocked on another as a priority one problem for you to resolve. NOTHING is more important than unblocking people and making sure they stay unblocked. This is easily where I spend the majority of my effort and time as a manager, and this takes on several forms that aren't just process and infrastructure.

For example, I try to very pro-actively understand what everyone is working on and how it fits into the architecture and the product as a whole. I do this not to keep tabs on everyone (though that is a secondary effect which is largely irrelevant to me), but rather to ensure that I can actively "connect the dots" between people who may be unknowingly working on interrelated things. This happens a lot, by the way. People who are touching related code, or thinking about related problems, need to be brought into contact with each other as soon as possible (and again, asynchronously: I'm not going to "call a meeting" 99% of the time, I'll just point them to each other in the team chat). In a way, this is a form of foresight. You're discovering a situation where two or more people would have become blocked on each other at some point in the future, and you're heading it off now by creating asynchronous coordination, preempting the future synchronous uncoordination.

As a manager of a distributed team, your number one priority is to create asynchrony in all things, proactively and reactively. Encourage that culture, encourage the tooling, and mow down blockers like they are sucking the living soul out of your four year old. Be extremely aggressive, too, in encouraging the development and use of internal tooling which facilitates these things in ways both small and large.

One example of this from a tooling side in our team is the use of semantic versioning and loosely coupled dependencies. We maintain a polyrepo topology of our internal projects for precisely the reason that each repository can be developed entirely independently of the others, so long as inbound and outbound dependencies are strictly declared and versioned. Every component is allowed to move independently, and the current "state" of the system is simply the most recent fully-compatible web of projects. This in turn means that the points at which various people working on unrelated things might incidentally collide are drastically reduced: they aren't even working in the same repository!

This is of course much easier said than done, and we have a lot of tooling built around enabling this kind of loose coupling. For example, we have a "launcher" tool which allows us to select versions of all the various components a la carte, resolve them and their dependencies transitively, and start the application all with a simple, declarative command in a self-contained package that can run in any environment (assuming you have the appropriate credentials and access). This launcher supports explicit eviction for overriding dependencies, allowing features and fixes to land upstream and become immediately visible to anyone who needs them, even before the transitive dependency updates have fully propagated.

Speaking of propagation, we use an automated system to "trickle" dependencies through the project graph without requiring any manual intervention, meaning that projects keep themselves up to date with changes as they arrive. All of this is geared towards enabling fully disconnected, fully asynchronous workflows between each and every member of the team.

There is a lot more behind this story, as well, but hopefully you're starting to get the picture. Invest in tools and process which proactively destroy synchronous blockers between members of your team. It will pay massive dividends, and this is honestly the single most important piece of advice I can possibly give.

Do Things Textually

This relates back to the concept of asynchrony (as does almost everything). Push as much interaction as possible into text. Not just because text is (usually) asynchronous, but also because it can be passively observed by everyone else as they catch up on the public record.

The internet is full of horror stories of people who were the only remote employee in a company that otherwise centralized to a single office. I've had this experience myself on more than one occasion, and it's NOT fun. The classic problem is that people in the office tend to interact (synchronously) with each other via ephemeral media, usually speaking aloud, and those of us outside the office are never privy to what happened or what was decided. The only way for the remote employees to even discover that such interactions happened is to specifically and proactively ask, which just isn't going to happen.

As a manager, you can avert this situation by pushing everyone to interact primarily in textual media. Here again, tools like Slack are immensely helpful. At my company, even though we're entirely distributed, there are three of us who do share a coworking space, and at various other times in our company's history, there have been more. Even when we're all in the same physical space though, we try to eschew verbal communication and instead utilize the team channels in Discord to share information. This makes sure that everyone is on equal footing and everyone has access to all the same information. It also boosts asynchrony, even among co-located persons!

This kind of culture doesn't materialize overnight. However, with the coronavirus threat and the fact that many companies are now forced into a distributed setting, you may find that these things are easier to establish than you might otherwise expect.

Resist the urge to spring for "presence" tools, like having everyone permanently in a group call or something like that. I've tried these sorts of things, and while they can be fun (and are certainly worth doing on a regular basis just for morale!), they aren't a good working solution. Text. Text is the answer.

An underrated benefit of pushing all communication into textual media is the ability to backtrack. In my experience, easily 70-80% of all "blocker" situations (where one person needs input from another person in order to proceed at all) boil down to "person A doesn't remember exactly what person B said about this thing and needs it to be restated". You can avoid that situation in advance by writing things down. Person A can just... scroll up in the chat. Or read the project description in the issue tracker or the document that was shared. The blocker situation then never arises because the person who would have been blocked in a verbal scenario is able to help themselves. This in turn fosters a greater sense of independence, which is an unambiguously good thing for any employee to feel, remote or otherwise.

Meetings

Take a look at your schedule. Look at all the meetings. As an exercise, take a dry erase marker and, at random, cross out half of them. Would the world really come to a crashing halt if you legitimately didn't go to those meetings that you crossed out?

No, no it would not.

Meetings are the crutch of bad management. They really are! It's so easy to call a meeting and feel like you're "doing something". You're getting on top of the problem! You brought the stakeholders together and catalyzed synergy. Or... something.

Most meetings are a complete waste of time. More importantly, they're a synchronous waste of time, and synchronous things are anathema to distributed teams! The peril is that, particularly to management, meetings do not feel like a waste of time when you set them up, and it is very difficult to break out of this trap.

I follow a few rules of thumb whenever I feel the urge to call a meeting:

  • Could this instead be handled by a series of one-on-one conversations? Do I really need to bring all of these people together at one time to just listen to me speak? If they don't need to interact with each other, then maybe they can just interact with me, and everyone's time is saved (except mine, but they're more important than me).
  • Could this be handled by everyone... typing into the team chat? (the answer to this one is almost always "yes"). Verbal communication is a lot faster, of course, and has the advantage of gestures and body language and screensharing and all sorts of other things. But, honestly, most meetings don't need all of that.
  • How much is this worth to the business? Meetings are expensive, as my earlier math showed. Is this meeting worth the price of a round-trip flight to Europe to the business? Most of the time, the answer to this is "no".

After I've gone through that checklist, if I still think we should have a meeting, then I go for it. I look at everyone's free/busy calendars and I pick out a time slot... and I schedule it for half the time that I mentally expect to need. Teach yourself to be concise. Most meetings ramble on needlessly. Get straight to the point and get back to work. You'd be amazed how much you can get done in a 15 minute meeting. I can honestly count on one hand the number of one hour (or more) group meetings that I've scheduled in the past two years.

As a sidebar on this, the ubiquitous "standup" meeting is toxic and horrible. Get rid of it. Create a channel called #standup and have everyone toss their daily status in there, just as a quick sync up and level-set so everyone knows what their teammates are up to. Allow them to add their status at any time during the day, just make sure they post at least once a day. Create a nice format for it. We use the following:

**PREV**: ...
**NEXT**: ...
**BLOCKERS**: (optional)

It doesn't take up any time at all, it doesn't force synchronous interactions, but it still keeps everyone on the same page.

Team Cohesion

We're a social species, and I say this as a full-on introvert who gets very exhausted by constant social interaction. All of us need to interact with people, and when you're trying to build and maintain a team, intrateam culture and interactions are probably the most important "soft factor" to consider. When that entire team is in one physical location, you can use easy tricks like group lunches, team outings, and the omnipresent water cooler to help maintain these kinds of interactions. When everyone is distributed, things get a lot harder.

Having a #memes channel is a good start. Seriously. Encourage people to bullshit random things. It's fun! It's fun and it makes people feel just a bit closer. Encourage humor in the team chat by example. React ironically. Create in-jokes and silly things that only make sense to your team. Be upbeat. It all adds up in a hurry.

More concretely though, set up a meeting. No seriously! Middle managers, this is your time to shine. Set up a meeting in some time slot that works for everyone. It'll be a remote meeting (duh), and there will be no agenda. No topic. No introduction. Optional (but encouraged) attendance.

Encourage everyone to background this meeting. They should log into the Zoom/Hangout/Discord/whatever, then just put themselves on mute and go about their business. Obviously it will be a bit distracting, but that's the point. Have conversations about things. It's okay to discuss projects and work! It's okay to discuss the weather. Or anything. Or just all sit in silence for a while. It helps, trust me.

Everyone who works remotely backgrounds every meeting anyway whether they're supposed to or not. Having a meeting specifically devoted to the celebration of backgrounding is surprisingly liberating, and it creates a space for ad hoc team interaction in a way that almost nothing else can or will. Give it a try! See how well it works for you.

Get Comfortable

As a manager, this is the hardest part: get comfortable with being hands-off. If you're the type of person who needs to see people working in order to be confident that they're getting things done, then that's your problem, not theirs. You need to work on two things: trust, and comprehension. The former is obvious. The latter is a lot more complicated.

When you comprehend to a first-order the work that each member of your team is doing, then it's really not at all difficult to figure out when someone is being productive and when they're being unproductive, at least on a weekly timescale. Identifying that someone is spinning their wheels without ever seeing them in person is a lot easier than most people think, and you as a manager need to get comfortable and confident with this concept.

The problem is that if you don't understand what your team is doing, then you may not be able to piece together whether or not what they did this week was something significant and meaningful. This can be a very unnerving feeling and rightfully so! As a manager, you need to react to this feeling productively. Don't just take it out on your employees by forcing them to check in with you constantly or by looking over their shoulder. This is your problem, not theirs.

Instead, educate yourself. Talk to them. Learn about what they're doing. Make them explain what they're working on, not so that you can check up on them, but so you can really get it. Have them check you. Of course, this also requires a lot of trust, because no employee is going to accurately share the meaning of their work with someone who has been compulsively looking over their shoulder for years. If that's you, then... sorry, you may be out of luck, and you honestly may never be successful as a manager in a distribute team (or at least, not your current one).

If you do have the requisite trust, though, then this kind of experience can be fantastic for both you and your employees. Learn, understand, grow yourself, and develop and appreciation for what your employees are doing and why. You won't regret it.

Summary

If you stuck with me this far, then I congratulate you! There are so many more things I could say, but honestly I need to get back to work. If you take just one thing away from this screed it should be this: optimize for asynchrony. Go back and read that section and pound it into your psyche. Be thinking night and day about how to make it happen for your team. If you can achieve total asynchrony, then your distributed team will be just as productive (and probably a lot more productive!) than the same team in a co-located setting. If you can't achieve asynchrony and you find most of your team interactions are stuck in synchronous land, then you will be constantly bogged down by inefficiency, and in the end, your remote work experiment will fail spectacularly.

@Leammas
Copy link

Leammas commented Mar 16, 2020

What about estimation and retrospective meetings? I mean business guys still require estimates and a team wants a timeplace to say things that bother it aloud. It takes a couple of hours a week.

@mslinn
Copy link

mslinn commented Mar 16, 2020

PRs don't seem to be possible for standalone files, so:

s/askew/eschew/

@djspiewak
Copy link
Author

What about estimation and retrospective meetings? I mean business guys still require estimates and a team wants a timeplace to say things that bother it aloud. It takes a couple of hours a week.

Superb question! My role straddles business and engineering, so from moment to moment I find myself either demanding estimates or protesting giving them. It's good fun.

Estimation doesn't need to be a whole-group activity. Most estimates can be given by individuals, asynchronously. Our project manager generally just direct messages people and asks them about it. When projects cut across multiple people, then it isn't a single thing to be estimated and we break it down entirely into tasks (as issues within an epic) before estimation. In the very rare case where the person who is going to do the work doesn't understand it well enough to estimate, then the feature is insufficiently architected. We spin up a new Discord group with just the relevant stakeholders/experts and start chatting. Often this will lead to some quick one-off meetings amongst that group, but sometimes it can just remain fully textual. This fleshes out the estimation quickly.

Note that we're an R&D organization to a significant extent. A hilarious percentage of the work we've done over the past several years has been of the "no one has ever done this before" variety, so estimates are fun and exciting already. We use a fibonacci point scale which is loosely calibrated to a single person's "work day" per point (i.e. so two people pairing on one thing for a whole day would be 2 points). This is coupled with a certainty label, which is always one of the following:

  • Kinda Sure
  • Not Really Sure
  • No Idea

This allows project management to compose estimates in terms of gaussians rather than pretending they're hard lines in the sand. This surprisingly helps a TON in terms of preserving asynchrony of process and avoiding estimation/retrospective meetings, since things that are uncertain are simply expected to slip. Pinning the value of a point to a work day also helps keep everyone on the same scale, rather than allowing it to float around as some nebulous, scale-less "velocity".

Oh, and we also practice continuous delivery with semi-continuous releases (since our product is installed on customer machines and can't be trivially auto-updated), so our sprint structure is a lot looser and accountability is built around deliverable features rather than hard quantized buckets of tickets, which helps considerably in avoiding scrums/retrospectives/etc.

@djspiewak
Copy link
Author

PRs don't seem to be possible for standalone files, so:

Oh lol fixing.

@Leammas
Copy link

Leammas commented Mar 17, 2020

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment