Lead Developer London 2018

lead-dev-2018.org

Lead dev 2018

Welcome - Meri Williams

• @TheLeadDev #LeadDevLondon
• white coat captioning - @whitecoatcapxg
  • wvnts.co/lduk2018

The Container Operator’s Manual - Alice Goldfuss

• happy pride! it’s like the world cup for people with fashion sense
• who am I?
  • SRE @ github
  • professional worrier with sudo access
• car advert analogy
  • beautiful mountain road,
  • lots of facts and figures
  • very very small print:
    • “laboratory setting”
  • it’s an advert
  • this is how i feel about most container talks
  • “just download this gist, (don’t read it!) and run it”
  • “oh you want to run containers in prod? no problem! prod is just your laptop times 1000”
• we need to examine containers a little more critically than we do today
• my creds
  • I’ve been running containers in prod since early 2015 (docker 0.6)
  • ran dockerized cassandra clusters
  • now, i run the k8s platform behind github

what is a container?

• “docker 101 talk view”
  • host (which is a box)
  • OS (another box)
  • daemon (another box)
  • containers (little boxes)
• this is all lies
• it’s a good way for a human to see containers
• but this is how your computer sees containers:
  • top command
    • python
    • ruby
    • kubelet
  • those python & ruby processes are containers!
  • containers are made of a bunch of linux features stapled together
  • when you build your Dockerfile, you’re making a tarball
  • when you run it, you’re starting a process (just like any other process)
  • this is where those extra features come in
    • namespaces
      • what it can see
    • cgroups
      • what resources it can use
• 4 lessons in this talk

lesson #1: containers have strengths

• they are good when you use them in the right way and play to their strengths
• strength which are: ephemeral, disposable processes
  • ie stateless applications
  • take data in, change it in some way, send it on to somewhere else
  • fantastic for scaling
  • run multiple instances side-by-side, behind load balancer
  • connect to same backend
  • if one dies, nobody cares (ephemeral)
• what do containers give you?
  • they’re portable
    • they don’t depend on dependencies from system
    • easy to iterate
    • easy to upgrade
  • disaster recovery
    • divergent DR processes:
      • jenkins
      • cap
      • some random cron job
      • someone has to sit with you to show you how to deploy their apps
    • if it’s containerised, everything is deployed in the same way
    • also great for production incidents
      • if i get paged, and find the problem isn’t my service, it’s someone else’s service
      • in the old world, you’d have to page them
      • now, you can at least try redeploying their service (because it deploys the same way)
  • testing environments
    • ephemeral spaces with disposable things that you want to clean up afterwards

lesson #2: containers have weaknesses

• stateful apps
  • eg databases
  • “I attended one of those container 101 talks and now i want to containerise all my databases”
  • my response: “are you google?”
    • really: “are you routing billions of requests to thousands of database across hundreds of databases per second?”
    • if not, you don’t need to containerise your databases
  • reasons to try:
    • faster provisioning
    • stability
    • recovery
  • if your current processes take months to add or upscale a database, it can seem really attractive
  • but: you can get all these benefits from a cloud provider and some tooling
    • “omg that will cost $$$$, we’ll build our own”
    • how much do you think building your own is going to cost?
• 2 main ways people do this:
  1. containerised database instances, data goes to some network-backed storage like ceph
     • use something like etcd to handle failure and service-discovery
     • BUT: eventually you’ll be network-bound
  2. store data on host using a mounted volume
     • if one dies, what do you do with the stale volume still on the host?
     • you could redeploy..
       • (this is what we did for cassandra. we had to write a lot of our own tooling)
• maybe: use a cloud provider instead
  • automatic failover
  • scalability
  • read replicas
  • multi-regions
  • everything you need without worrying about:
    • becoming network-bound
    • securing data properly
    • etc

lesson #3: containers need friends

• how will you build your container tarballs?
  • you don’t have to use docker in prod (do you want to? you should read about it)
• orchestration & scheduling
  • resource management
  • placement
• management
  • how will you manage clusters?
  • how will you drain traffic?
  • how will you do automatic failovers?
• networking
  • how will you handle routing, access control, service discovery?
  • there’s a dozen tools that just handle networking in kubernetes
    • linkerd
    • istio
    • envoy
    • calico
    • …
• how do you hook it into your existing infrastructure?
  • deployment: is it the same tooling? how do you support two parallel deployment pipeline?
  • monitoring: what new metrics do you need to be monitoring on your new platform? what counts as a healthy container?
  • provisioning: how do you provision new container hosts? what gets priority? container hosts or legacy hosts?
  • debugging: how are you going to troubleshoot with this new abstraction layer? how will you empower the devs running apps on this platform to debug their own services? logging platform? training?
• gradual rollout. try some things, some fail, try new things
  • minimum 1 year to rollout a new container platform
    • just one service, on a relatively stable platform

lesson #4: containers need headcount

• behind closed doors:
  • “we need containers”
  • “let’s give it to ops”
• containers is a completely new can of worms
• you’ll need a new team
• you’ll need specific skills:
  • ops
    • you need to know where the skeletons are buried
  • deployments
  • tooling
    • you’ll need tooling to glue the platform together
    • you’ll need to test it
  • monitoring
    • someone who knows the right metrics to check when you see weird phenomena
  • kernel engineer
    • analyse kernel crashes
    • understand upstream
  • networking
  • infosec
    • containters have a bad reputation but they’re making great strides
  • internal adoption
    • good relationships with other teams internally
    • you need to manage other teams’ roadmaps to get early beta customers
  • project manager
• each of these skills doesn’t need to be a separate person, but you also need more than one person
  • you need at least 4 people, ideally 6-8
  • less than 4 is death for on call
• empower them to succeed
  • authority
  • budget to try out new cloud instances

should we use containters in prod?

• do you have
  • stateless services?
  • a large, hetorogeneous platform?
  • do you have time, money, people, org support?
  • yes, maybe you should do containers
• do you have
  • a monolith and a few services?
  • a small team with no org support?
  • just want to spite me?
  • maybe don’t do containers?
• do you want containers or a blog post?
• it’s ok not to use containers

First Steps as a Lead - Dan Persa

• why become a lead in the first place?
  • dealing with people problems is much more interesting than dealing with computers
  • i asked a friend for advice who had done it before
    • he said: “dan, don’t do it”, you like coding too much
• each company interprets slightly different way:
  • tech lead?
  • people manager?
  • part of your job is to understand what your role is about
• getting into leadership is not trivial
  • ask for help of people around you
  • your lead, for example
  • head of dept set up a step-by-step process for how to become a lead
  • part of my plan was going through a 3-month transition of intense mentoring
• surprises:
  • how much of my time i had to spend in meetings
    • strategy
    • agile
    • 1:1
    • they’re important
    • i was getting energy drained, couldn’t keep pace
    • i got better in time
    • better at choosing which meetings really mattered
  • career switch
    • this isn’t a promotion, it’s a different skillset
    • becoming a lead isn’t a natural evolution of being a developer
    • the DRY principle is not a good principle with people
• how do you learn new skills really fast?
  • shadowing my mentor
  • having his support helped me have a smooth transition and kept me from feeling lost
  • applying what i read in books
    • not just reading books
    • put it in practice!
  • reflect and remember good and not so good behaviour from other managers in the past
• example: i always hated not feeling i was in an open and safe environment
  • without mutual trust, there couldn’t be an open and safe environment
  • how do you earn trust of team?
    • it’s not easy: trust means opening yourself up to be vulnerable
  • regular 1:1s
  • transparency in decision-making
    • especially around performance evaluation
  • the best solutions to problems come from the team
    • have the team take ownership of the problem
  • feedback loops
    • ask for feedback from team on any occasion
    • healthcheck meetings with the team
• not taking a decision is a decision
  • we are accountable for both the decisions we make and the ones we don’t or postpone
• feedback culture
  • give feedback as close as possible to the moment in time where the action which generated the need to provide feedback happened
  • decouple development feedback from performance evaluation
• why did your company hire you as a lead, rather than as just another software developer in your teams? understand this and unleash your lead superpowers.

It’s Personal - the art of giving and receiving code reviews gracefully

• @alexhillphd
• why code review? they’re good at some hard-to-quantify things:
  • fucntionality
  • readability
  • maintainability
  • scalability
• but it’s also a direct critique of someone and their work
  • defensiveness is a common feeling
• not all comments are equal
  • two axes:
    • low reward - high reward (how important is the comment?)
    • low conflict - high conflict (how likely is recipient to feel defensive?)
  • low reward high conflict:
    • whitespace, typos, arbitrary preferences
    • use a linter instead to resolve these! don’t do it in code review!
  • low conflict:
    • missing tests, bugs, rogue println statements
    • no problem
  • high conflict high reward:
    • design pattern choices
    • over-engineering
    • readability
    • naming choices
    • duplicated logic
    • active collaboration needed
• conflict resolution archetypes:
  • competing
  • collaborating
  • compromising
  • avoiding
  • yielding
  • (dual concerns model)
  • we want to achieve the collaborative approach
    • lower defensiveness
    • raise confidence
• as an organisation;
  • pair program
  • discuss tasks prior to implementation
  • not silo codebases
  • ensure everyone reviews and all code is reviewed
    • not about singling people out
    • seniors don’t get to commit without review
• as a reviewer:
  • raise code by a grade or 2, no more
    • C-> B+, B->A, but not D->A
  • language: use “we” not “you”
    • “you should have re-used this function” -> “we could reuse this function”
  • ask questions
  • give positive feedback
    • “this is a cool library”
    • “this refactor makes a lot of sense”
• as an author:
  • practice the “as if” technique
    • how would i respond if [i were in my best mood, i were<role model>, this code wasn’t mine, the comments were phrased differently]
  • say ‘thank you’
  • annotate your review first
  • solicit feedback with specific questions
• summary
  • anticipate feelings of ownership
  • minimise unnecessary conflict so you can focus on the important ones
  • nudge towards collaboration by reducing defensiveness and raising confidence
• blogpost on website

knowing me, knowing you - growing teams to continuously deliver (pia nilsson)

• @pia_nilsson - engineering manager, spotify
• a journey in combining the lean & XP practices with a focus on the team
• flow in lean: kanban board
  • all about optimising speed from backlog to done
  • find bottlenecks - tech and people
  • increase the flow of work
  • at spotify, we were doing a lot of things, but not finishing many
    • lots in “develop”, not much in “review”, “validate” or “done”
  • we knew we had some bottlenecks
• what did we do?
  • WIP limits
    • improved flow of work, got more stuff done
    • but it also increased friction
      • devs couldn’t pick up a new shiny task
      • had to instead find that key person to get that review done
  • test-driven development and domain-driven design
    • talk about why it makes sense to do tests first
    • teaching sessions on DDD
    • architecture discussions talking about boundaries of a microservice
    • increased flow more
    • but it also increased friction
  • pairing & mobbing as a default
    • added to working agreements - you are expected to pair
    • expected to decrease silos
    • everyone was on board
    • increased flow even more
    • but still increased friction
    • how can we keep the flow but reduce friction?
    • that’s this talk

blockers of flow

blocker #1: I can’t do brainstorming because i haven’t had time to think on my own first

• need to create safe spaces for people to share opinions without worrying about being right
• read up on google project aristotle - “psychological safety”
• “how confident am i that i won’t be blamed or laughed at if i make a mistake or ask a dumb question?”
• “code creme brulee”
  • hard on the surface, soft on the inside
  • we set up to scratch that surface
  • ask the stupid questions
  • speak up about a task, how it makes you feel
  • try to be a bit vulnerable
• training in toxic communications
  • based on studies from Dr Guttmann who studies married couples for 40 years
  • four toxic communicaton patterns:
    • criticism
    • defensiveness
    • stonewalling
    • contempt
  • we introduced “i messages”
    • “this sucks” -> “i think this sucks”
    • reduces defensiveness
  • contempt is often seen in teams with a lot of sarcasm
    • appreciation is the antidote

blocker #2: “I’m interrupted all the time”

• practice active listening
• “yes and”
  • comes from improv theatre
  • build on other’s ideas
  • never say “no” or “but”
• “this unit test does too many things, I’m going to delete it”
  • “yes, and let’s write a new test which is clearer”
  • it’s about hearing a third opinion which you might never hear if you said “no but”
• we’re passionate people
• it’s not great to be interrupted

blocker #3: “we think so differently, it’s just better if we don’t pair together”

• practice: friendly feedback (without blame)
  • conflict is a tragic expression of an unmet need
• non-violent communication cheat sheet:
  • when you [merged to master yesterday], i felt [confused], because i have a need for []. Would you consider [checking in with me first]?
  • key point: don’t leave feelings to implicit interpretation; explicitly talk about them
• turn up the good

blocker #4: “I’m not sure about the purpose of this team or what I’m doing here?”

• if the team is moving really fast, it’s important to know where we’re going and why
• Objectives and Key Results
• can really help if used properly
• in the CD backend team we had these uninspiring objectives
• objectives need to be really inspirational
  • objective: centralise all CI at spotify
• key results: measures that show how far along we are towards our objective
  • key results: increase customer base of platform by 60%
  • note: no mention of how we achieve this
  • that’s up to the team
• despicable design
  • think about our users and how our service could be made worse
    • service window midday
    • slower response times
    • postal reviews
    • stuff like that
  • it was fun and it got us thinking about our users

summary

• how we increased flow while reducing friction
  1. get the talking going
  2. active listening
  3. friendly feedback
  4. answer why
• this isn’t static at all - you can’t do it once and forget about it

how i learned to stop worrying and love meetings (menno van slooten)

• @mennovanslooten - front end chapter lead - backbase
• I’m not sharing this story because i think it is special
  • but because i know it is not
  • and it would have helped me to hear it before i became a lead developer
• i had been a FE dev for >15 years
  • one of my old employers found i was on the market and invited me to become a chapter lead
  • the experience of becoming a lead dev was.. not great
  • i’m hear to explain how i felt being a lead developer as opposed to being a developer
• i loved being a developer
  • i loved programming
  • i loved the way code looks in an editor
  • i loved abstract puzzles
  • i loved learning about some 50-year-algorithm i last heard about in university
  • it was not just satisfying, it was really rewarding. fulfilling.
  • i could open up a browser and show my work to my friends and peers and family and say “i built that!”
  • i worked really hard to get good and stay good.
• but: a nagging voice in the back of my head
  • “how long do you think you can keep doing this?”
  • i had trouble keeping up
  • people much younger, less experienced, who were creating solutions i didn’t understand, to problems i didn’t realise i was having
• it was in this context that my old employer invited me to become a chapter lead
  • it was a huge step in my career, if you can call having the same job for 15 years a career
  • i had no idea what i was signing up for
  • (well, i had some ideas, but none of them turned out to be true)
  • the reality was rather disappointing
    • not nearly as rewarding or fulfilling
    • what did I create today? waht did I do?
    • i wrote more emails than lines of code
    • meeting after meeting which had nothing to do with programming
    • interruptions so bad that my interruptions would get interrupted
• i did something i should have done way before, which was talk to other lead developers
  • almost everyone i talked to was dealing with the same issues
  • you’re not alone, you’re not weird, it’s totally fine
• i came home from that conference determined to get better at being a lead developer, but also to get better at enjoying it
  • and it worked!
  • there’s no four-step programme
• i had to completely let go of programming
  • step away from chasing that feeling of fast feedback
• i had a meeting with my manager where we were asking if someone’s job title matched their performance
  • i found two people in my group were performing way better than their job title suggested
  • i focussed all my email-writing powers into writing to my manager
  • two months later, they were promoted and got a significant raise
  • this blew my mind
  • i had a load of influence and power
  • and i had never really bothered to use it
• this made me reevaluate everything i did
  • interruptions, emails
  • let the developers continue uninterrupted
  • if i sent an email it was for the benefit of the people around me
• with great power, comes great responsibility
• i like helping people
  • not because i’m selfless
  • but because it makes me feel important and valuable

building sustainable teams to handle uncertainty (jenny duckett)

• @jenny_duckett - technical architect, ministry of justice
• think about a time when you were working on an effective team
  • what made you excited about it?
• i was on a team
  • it changed several people over the course of my time there
  • lots of people joining the team were completely new to the team
  • support and improvement work over the 10 product areas we owned
  • we had to shift the focus of the team significantly at one point
• might seem unusual choice for best team ever, given the circumstances
  • an empowered, open team
• i see lots of teams struggling

what can disrupt your team?

• change is often positive!
  • we frequently make deliberate changes about how we’re working in our team
• there’s often uncertainty involved
  • which might be resolved quickly or go on for months and months
• people can feel unvalued
• people leave and join
  • often these are positive changes but it can still be hard
  • the dynamics of personalities and opinions changes whenever people change
• senior leaders move
  • it can be hard for teams to make decisions and handle roadmaps while this is happening
  • it can be hard to revisit decisions that you thought you’d settled
  • it can be hard to make progress
• wider priorities shift
  • can end up feeling frustrated and disempowered
  • why make effort when it might just change anyway?
• teams are re-organised out of existence
  • can make people feel undervalued as individuals or for the work the team has done together
• if you’re putting people through these things more than occasionally, you will have a retention problem
• even things like desk moves (see Lara Hogan’s blog post about this) can have an impact on people
• the way changes make people feel is at least as important as the change itself

what can you do to prepare?

we can prepare for change

• work on yourself first
• help ensure you can sustain through these disruptive changes and be better able to support your team as a result
• if you’re senior, consider the relationships that the teams have worked hard to build together
• don’t do it all yourself
  • you don’t need to review all the PRs
  • let go of the details
  • instead, give people your trust
  • assume positive intent
• if you’re working with other people to lead your team (such as product and delivery managers), work really hard on your relationship with them
• one thing i found really useful was a support network for tech leads within GDS
• set yourself up to lead sustainably

make your team’s work ownable

• define a single clear goal for your team
• we had to stop splitting our efforts and focus on one goal
• the effect was magical
• we agreed as a team what parts were essential by the deadline
• a shared understanding of all this meant the whole team was empowered
• communicate it over and over again
  • it’ll feel like you’re just repeating yourself and you’re boring people
  • but it’s reassuring just to know that the goal hasn’t changed
• you might not be able to pick one single goal
  • but communication is even more important in that case
• give your team the background they need for each piece of work
• add context to your stories
• use story kickoffs
  • don’t get as far as the PR before explaining why you want the work done
• run workshops

empower your team to take ownership

• embrace opportunities for positive change
• become great at integrating new people
  • every new person has a mentor for the first few weeks - often the last person who joined
• recognise that everyone is always learning
  • make it clear that you value learning as part of everyday work
• share understanding of your work in the team
  • write good commit messages - why, not just what
  • comments on tickets and user stories documenting progress
  • when teams make a decision, document the reasons for it
• if you learn to share context frequently, you will be able to keep work flowing during disruptions
• when my team started mobbing, i was amazed how quickly the team made progress and how rarely they were blocked on anything
• we wrote a blog post (written by Emma Beynon )
• teams were able to use mobbing later on to tackle things which were entirely new to them
• taking ownership helps people handle change

support and grow individuals

• use every piece of work to help someone grow
• start with individual needs
• delegate effectively
  • be clear and explicit - tell them that this is what’s happening and why
  • set the scope: what’s the problem they’re trying to solve?
  • who else is going to be involved?
  • when should they bring questions to you?
• teach people to do your job
• grow the next generation of leaders
  • it’s easier and more rewarding to hire less experienced people and grow them
• prepare the team for you moving on even before you know it’s going to happen
• when i came back from a couple of weeks holiday and found my team carried on as normal without me, i had all the feelings
  • sad but proud

show your team where they fit in

• grow people for a resilient team and organisation
• encourage your team to show off their work
• show the team how their work fits into the bigger picture
• don’t over-insulate your team
  • when something gets through your shield, it’ll be very disruptive
  • if there are rumblings around in your organisation, mention it to your team
  • tell them when you’ve said no to your team taking on other work
• a wider view helps people adapt when things change

managing up

• communicate your team’s capacity for change
  • it takes time to become familiar with a new domain
• i frequently had to argue for not restructuring teams wholesale, but come to some accomodation to take on new organisational priorities
• people at the top can’t create a culture of empowerment alone, but they can reinforce or destroy it
• show them why all your team’s work matters
• good communication about change is vital

what does this look like?

• grow sustainable teams for a sustainable organisation
• a team that understands where they fit within the overall strategy can more easily accept changing the focus of that work to support that goal in a different way
• (i recommend reading the whole transcript of this talk (see link at top), there’s way too much value that i missed)

the hardest scaling challenge of all - yourself (christian mccarrick)

• @cmccarrick - SVP of platform engineering - GTL
• start with a story
  • when i was 14, i worked on a farm making peaches
  • long days in the sun, alone, covered in peach fuzz
  • i had a lot of time on my hands
  • i decided there was no way i was doing this for the rest of my life (and got straight As for the first time at school)
  • i realised that there’s only so much that one person can achieve on their own
  • i bought a commodore 64
• i’ve been having conversations iwth hundreds of people
  • “i don’t have enough time”
  • “i am so stressed out”
• what does it mean to scale?
  • my manager quit on a friday; on monday - boom - i was running a team
• i manage over 200 people in 10 offices scattered throughout the world
• there are times when i lose it
  • i get emotionally paralysed and can’t make any progress
• learning to scale yourself is not a one-time task; it’s a lifetime’s effort
  • you should expect your job to change every 6-9 months

1. commmunication

• you have to be good at communicating 1 on 1
• as you grow, you need to get better at geting your point across to larger and larger groups
• become expert at managing up to your managers; across to other teams; and down to your reports
• the “manager readme” - weekly digest
• personal branding and self-promotion
  • in order for your good performance to have any effect on your career, someone needs to notice your good performance
  • we value logic, efficiency, getting the work done
    • to us, that’s all that should matter
    • but often it’s not
• what can i do?
  • take initiatives on high-visibility projects
  • genuinely praise other people
  • have confidence - it’s a positive feedback loop which attracts people to you
• don’t let your weaknesses blind you
  • get honest feedback
  • you need a safe environment
  • get people to expect consequence-free feedback
• most importantly: learn to say “no”
  • without feeling guilty!
  • respectful, but completely unapologetic

2. prioritization and time management

• people are bad at time estimation and time management
• how much time do we have?
  • 168 hours a week
• if you don’t prioritize your life, someone else will do it for you
• write down all the things you need to do (don’t prioritize!)
• but there are things you don’t think about - dark matter tasks
  • people who say “can you help me with this?”
• keep a journal - track all these quick favours
• prioritization:
  • “which one of these is the most important?”
    • “they all have to get done” is not an answer
  • eisenhower method
    • 4 box matrix: urgent/not urgent vs important/not important
    • urgent/important: do
    • not urgent/important: plan
    • not important/urgent: delegate
    • not urgent/not important: eliminate
• multitasking is a scaling anti-pattern
• focus
  • what’s it like being completely focused?
    • relaxed
    • confident
    • lose perception of time
  • how do you recreate that moment more often?
    • get the most done
  • calendar blocking
• meetings OMG
  • makers vs managers schedule
  • delete every recurring meeting you have
  • make your calendar sacrosanct
    • reject double bookings
• email and slack
  • they’re so addictive!
  • “i wish i spent more time on email” - said noone ever
  • turn off notifications
  • schedule short blocks during the day when you quickly check emails
  • schedule longer blocks at the end of the day where you tidy things up
  • your calendar can do your slack statuses for you
  • put response times in your manager readme
    • so people have expectations on response times
• obsess with the things that matter!

3. delegation

• move from “how can i get this done?” to “how can this task/decision/goal get done?”
• not delegating properly is an anti-pattern
• let things go
• delegate decisions!
  • not doing this limits the scaling of your entire team by becoming a bottleneck around decision-making
• leading takes energy
• leading gets lonely
• if athletes get injured, they sit on the bench
  • mental fatigue, burnout, is an issue that affects us more than we might think
• make yourself a priority once in a while. it’s not selfish, it’s necessary
• mental health benefits of exercise

introduction to reasonml (jack lewin)

• @jlewin_ - software developer - sky
• “people who came to the react.js community early are now mostly talking about Reason”

origins

• new syntax for ocaml
• functional programming language
  • 1990s heritage
  • used by financial institutions
• interop with JS

types

• “why can’t i use typescript or flow?”
• with reason, types are baked into the language which gives benefits you can’t get otherwise
  • 100% coverage - everything has a type
  • guaranteed to be sound
  • no runtime type errors
  • pattern matching

JS interop (bucklescript)

• compile reason to (readable!) JS
• access JS in reason
• compile to native
• reasonml-community on github
• create-react-app with reason-scripts
• reprocessing

CI/CD for humans: empathy as the foundation for effective deployments (jackie balzer)

• @jackiebackwards - head of FE development for behance at adobe
• I’m all-in on frontend development
• I’ve never worked in devops
• i did help with the development of the web UI of our CD platform
• what makes a good deploy?
  • as long as it makes it to production and doesn’t set off any alerts, it’s a success
  • what about empathy? what about being compassionate and minimising suffering?
  • a good CI/CD pipeline is the empathetic choice for delivering code to production
• putting code into production is scary
  • it can be the domain of a specialist person with specific skills
  • the first time i wrote a capistrano script to deploy my personal website, i was so baffled by what i’d written, i was afraid to update my site for over a year afterwards
  • a good CI/CD platform does a lot to address the risks & fears around deploying to production
• CI and CD are the practices that enable organizations to frequently and reliably release new features and products
• lots of benefits in terms of reducing risk
• it’s also opinionated which may have impact on how teams do work and require team discussions
• episode I: the FTP menace
  • when i did this in the beginning:
  • svn for version control
    • branches were hard, so we just wrote to master all the time
  • we thought we were “agile” by not being nitpicky about writing tests
  • no process for task management
  • every 2 weeks or so, our one devops engineer would cut a release from our master branch, put it into staging where it would sit for a day or two, then onwards to production
  • we were deploying huge, risky changesets all at once
  • we would attempt to deploy at the lowest-traffic times to compensate
    • we could only deploy after hours
    • don’t do this!
• episode II: attack of the hotfixes!
  • our ops staff would spent days at a time being our menial deploy sherpas
  • we had no way to do a rollback. if we pushed a bug, we just had to resolve it and release another hotfix
• episode III: revenge of the GUIs
  • at some point we switched to git and github
    • branching, code reviews, automated tests
  • jenkins & hubot to allow engineers to deploy changesets to different staging environments
• build nights make deploys more stressful by making things seem higher-stakes than they really are
• it takes someone with a specific lifestyle to be able to hang around the office at all hours
• biggest points of failure
  • how the process affected us and our users
• episode IV: a new hope
  • some of our engineers were sick and tired of our build processes
  • wrote down how deploys should work
  • started to build a new platform
  • solved the probelm in 2 weeks
  • a series of best practices
  • ownership of deployments is empowering
  • development safety:
    • fork-and-branch to allow for consequence-free development playground
    • upstream is locked down and only contains approved code
      • reduces clutter - clean history
    • feature firewall
      • new features are developed incrementally and shipped to production behind flags
      • can do this without fear of negatively impacting user experience
      • our designers can test new features in a real-world environment
    • smaller changesets
      • smaller PRs even for new features
        • feature doesn’t need to be fully complete to be deployed to production
      • less risk
    • peer reviews
      • style checks automated
      • PR peer reviews become higher quality
      • rule: be nice
      • be cogniscant of your colleague’s efforts
    • CI at every step
      • starting from PR
      • if anything fails, you are blocked from moving forward
    • staging and production require manual verification
      • automated tests stop being useful at the point where reality begins
      • human verification of human-made changes intended for human consumption
    • evergreen master
      • master is always deployable
    • safer rollbacks
  • easy interface, easy deploy
    • a good UI is pretty invisible
    • jenkins as an anti-example of this :sick-smiley:
    • moonbeam.io
  • it started as a hackathon, but it’s grown to be a first-class citizen
  • we now use moonbeam to deploy moon

Junior.next() (tara ojo)

• @tara_ojo - software engineer - futurelearn
• think back to your first day in your career
  • full of enthusiasm
  • building something from scratch
• fast forward to today
  • think about that journey
  • everything you’ve learned
• you probably didn’t get to where you are today by yourself
  • you learned from colleagues with loads of experience
  • you probably also learned from colleagues without much experience too
• the way you work with junior developers on a day-to-day basis can have such a big impact on their careers
• before futurelearn, i was on a 12-month grad scheme at M&S
  • we knew we needed to learn how to code
  • but we didn’t know what else we had to work on
• I use the word “junior” but there are other words:
  • intern
  • associate
  • apprentice
  • graduate
  • noob

being a junior

• you could be doing things to support and uplift junior developers in order to get them to the next stage
  • they need to stretch themselves
  • “i could start teaching other people in my team things” - grad developer who felt ready to move on
  • teaching is a good way for a junior to stretch themselves
• if your junior developer is picking up tasks that they know they can do, and doing them quickly and successfully, then they may be sticking to their comfort zone
  • you want them to stretch themselves by taking things a bit beyond this
  • but: avoid going too far into the PANIC zone
  • remember that the boundaries between these zones aren’t fixed
• gather evidence about amazing changes and stretching
  • this will be useful for interviews for roles at a higher level
• being a junior is not something to be ashamed of. it’s not a burden

being a supporter

• invest time
  • teach them
  • pass on your knowledge
• good pairing
  • when i was quite new
  • i was struggling to follow along
  • i fell asleep at one point
  • if that person had given me the keyboard, i would have been much more engaged
  • ask specific open questions
    • “if i run the tests against this code, do you think it’s going to pass?”
• create opportunities
  • living objectives
    • set goals and measure confidence levels in a bunch of different areas
    • check your confidence as you go through
    • it’s a good way to measure your progress
• regular feedback
  • “i feel like i’m not mid-level, i need confirmation from people around me”
  • giving specific and actionable feedback
  • try this structure:
    • do more
    • do less
    • start doing
    • stop doing
    • keep doing

Points Don’t Mean Prizes (adrian howard)

• @adrianh Generalising Specialist - quiet stars
• making a product is easy
  • someone comes up with a “north star” - the problem that needs solving
  • someone else comes up with a journey of how we get there
  • someone splits it up into cards
  • we divide it up into cards and give them to developers
  • we do them all
• this is a lie
• stories come in different sizes
  • some are big, small
  • causes problem with flow - blocked pipelines
• we start doing estimates
  • story points
  • if it’s too big, we cut it up
  • gets us back to our nice list of small tasks
• this is also a lie
• our customers don’t care about our stories
  • when we take a story and cut it up, we often discover the actual value is somewhere other than what we’ve built
  • something’s gotten lost between the north start and the individual stories
• instead of estimating story points first, try these three value-based questions:
  1. can we bin this story (or at least shelve it?)
     • if it’s about building subscriptions, but the first 3 months are free, do we need to build it now?
  2. can we thin it?
     • can we deliver less and still deliver most of the value?
  3. can we split it?
     • into more than one story, which still provide value to the customer?
     • not: database tables
• we still get smaller stories!
  • binning: you can’t get smaller than nothing at all
  • thinning: we’re doing less, making stories smaller
  • split: those stories are smaller
• ask the questions again about the results. and again
• but split based on value, not effort
• book: user story mapping by Jeff Patton
• you have your customer journey, and you map the stories that we need to minimally support that journey
  • that’s release one
• each slice is a release that provides a version of the customer journey
• this is an ongoing activity
• if you bin a story, that tends to push it down to the next release
• if you thin a story, it tends to pull it up to the current release
• if you split a story, it spreads it across releases
• (this is still a lie)
• many organisations don’t let people at the development team level make those decisions about customer value
• things to help people know the goals:
  • OKRs
  • options
  • assumptions
  • hypotheses
  • experiments
• do less together more often to do more

Who Destroyed Three Mile Island? (Nickolas Means)

• @nmeans - VP Engineering - Muve Health
• how does a nuclear reactor generate power?
  • when i was a kid i was given a four-volume set of books called “how things work”
  • my dad often patiently explained to me how things work
  • I distinctly remember turning to these books when a new nuclear reactor came online
  • I turned to the page on reactors
  • the basic mechanics of a nuclear power plant are basically the same as a conventional plant
    • a heat source to heat water
    • high pressure water turns to steam
    • steam expands which pushes a turbine
    • which turns a generator
      • which is where the electricity actually comes from
    • the steam condenses and goes back round the system
  • we’re looking at pressurised water reactors because that’s how Three Mile Island works
    • primary loop: water heated and held at high pressure
    • secondary loop: water turns to heat
    • the two loops don’t mix
    • the primary loop water doesn’t boil
    • or at least…shouldn’t
• three mile island
  • about 10 miles south of the capital of Pennsylvania
  • unit 2 went into operation 1978
  • runs at 90% capacity for 3 months
  • “hot, straight and normal”
• 4 men
  • shift supervisor
  • shift foreman
  • 2 operators
• everything was pretty normal
  • except for a small problem in one of the condensate polishers that the previous shift couldn’t solve
  • 10 hours earlier the swing shift started working on unclogging a clog
  • they tried backwashing but it didn’t work
  • the operators early on developed a backup system where they took air from the pressurised system and used that to break up the resin beads
  • suddenly went quiet (which it shouldn’t with hundreds of tons of water)
  • there was a leaking check valve
  • all eight tanks close simultaneously completely blocking the flow of water
• there is no water to be pumped through the secondary cooling room
  • the main feed water pumps trip offline
  • the turbine and generator trip offline 2 seconds later
  • the palnt’s main safety opens
    • the secondary water is not radioactive at all
    • lets off loads of steam into the night sky
  • the turbine and generator alarms go off
  • the primary water pressure spikes
    • this is by design - without secondary to remove heat, it heats up and so pressure goes up
  • automatic systems kick in here:
  • pressuriser:
    • regulates system pressure
    • measures water level
    • if there’s water in the pressuriser, there’s water in the rest of the system
    • maintains a bubble of steam at the top which absorbs pressure shocks
  • but the pressure is continuing to climb
  • the next thing that happened (and most famously)
    • pilot operated relief valve opens
    • 4 seconds after the turbine and generator trip offline
    • the computer detects that pressure is still climbing
    • so it scrams the reactor
    • neutrons -> uranium fission
    • we control this reaction through cadmium control rods which absorb neutrons gleefully
      • typically raised and lowered to control reaction
    • scram: drop them to the bottom
      • stops the nuclear chain reaction almost immediately
      • but it doesn’t stop heat production quite as quickly
      • secondary fission products are still present - 6.5% heat production
      • in the hours after this, it’s still important to carry heat away from the core
    • control room shows the pilot relief valve has been signalled to close
    • operators are happy - they have procedures for this
      • confident that the system is behaving as it should
    • but: 2 minutes later
      • emergency core cooling system kicks in
      • high pressure values dump water into core
    • they didn’t understand why the system thought it needed more water (because the pressure was still rising)
    • 2.5 minutes later, they switch the high pressure valves off
      • if they hadn’t, there would have been no accident
    • bill zewe wonders about something:
      • the pressure is now dropping but the water level is rising
      • he suspects the pilot relief is stuck open
      • checked the outlet temperature
      • if he had decided to close the block valve, there would have been no accident
    • 6 minutes later:
      • alarm about sump:
        • pilot relief valve has leaked enough to set off this alarm
        • clear signal that there’s a significant leak
        • but they miss it
      • the water in the reactor pool is boiling
      • causes vibrations
      • they have to shut off the primary coolant loop pumps
      • this helps for a little while
      • 30 minute later, the vibration is back
      • it’s now 5:44 in the morning, and a nuclear reactor that was at 90% 2 hours ago has no coolant going round
    • a radiation alarm goes off
      • one of the fuel rods has ruptured
        • uranium is normally encased in zirconium
      • almost certainly, the water level has droped below the top of the core
  • the leadership has come to the plant
    • “they closed the block valve, right?”
    • they finally shut it
    • this would have been the right thing to do 20 minutes into the accident
    • doing it now removed the only source of cooling
    • heat intensified rapidly
    • 8 minutes later, top of core collapsed
  • at 7.20 the radiation alarm at the top of the building goes off
    • maximum yearly allowance in 20 seconds
  • they knew the core had started to melt down
  • they turned on the high pressure injection
  • but turn it off 18 minutes later because they’re concerned about the pressuriser
  • over the next few days they continued to be concerned about radiation leak
    • but the containment did its job
  • there was public concern about a hydrogen explosion
  • on sunday april 1st president jimmy carter visited
  • about 20 tons of melted uranium ended up at the bottom of the reactor vessel
  • a further 10 tons in the middle
  • final cost of cleanup: $1bn
    • still not finished
  • unit 1 is still producing electricity
  • unit 2 won’t be fully cleaned up until unit 1 is shut down
    • currently scheduled 2034
  • sidney dekker: the field guid to understanding “human error”
    • “first stories” and “second stories”
    • the story i told is the first story
      • focuses on humans in story and what they could/should have done differently
      • problem with this: bias
        • hindsight bias
          • if you know the outcome, you exaggerate your own ability to predict and prevent the outcome
          • “I think if i saw water pooling at the bottom of the reactor building, i would have noticed there was a leak”
        • outcome bias
          • given the outcome, you judge decisions based on that
          • eg: turning off the high pressure injectors was obviously stupid
    • how do we get to the second story?
      • work from the participant’s reality
      • assume positive intent
      • everyone involved made the best decisions they could with the knowledge they had
• why did Fred turn off the emergency cooling system?
  • pressurizer
    • he was afraid the water in the pressurizer was going to go solid
    • if you allow the pressurizer to fill with water (no steam bubble) there’s no shock absorbing effect any more
  • this is a smaller problem than the core melting down
  • why is fred so concerned about the pressuriser?
  • they were all naval reactor veterans
    • prime job in a submarine operator was to prevent the pressuriser going solid
    • this makes sense for a submarine reactor: 12 MWth
    • not three mile island: 2.8 GWth
  • heat at shutdown (6.5%):
    • submarine: 0.78 MWth
    • three mile: 185 MWth
  • in a submarine, a water hammer (unabsorbed pressure shock) was the worst-case scenario
• so Fred Schiemann, with a rising pressure, disabled the cooling to keep the system safe.
• why did bill zewe not close the block valve when the temperature was over 200℉?
• the pilot-relief valve was leaking from the day three mile opened
  • if they shut it down every time the temperature went over 200, they’d not generate any electricity
  • they were going to fix it for next refuelling shutdown
• he also figured the pilot-relief valve was closed
  • but the control panel only said what the computer had told it to do, not the actual position
• why did the crew not respond when the sump alarm went off?
  • they never got the alarm!
  • giant panel of lights - each one resembles an alarm
  • on a good day, 40 or 50 lights are lit up
  • there’s no rhyme or reason to their placement
  • the light for reactor vessel pressure is next to the one for a stuck elevator
  • no timeline for lights - when did alarm go off?
  • there is a timeline on the printer
    • but it’s on a 300 baud connection
    • there’s so many alarms that the printer is 2.5 hours behind
• Dr Dekker: human error is never the cause
  • it’s always caused by underlying systems
  • blaming a human prevents us from examining this
  • ask what is responsible, not who
• understand why what people did made sense
  • because if it made sense to them then, it’ll make sense to someone else later
• seek forward accountability, not backward
  • blameless postmortems
    • free people up to candidly share what went wrong
    • so you can really get to the bottom of it
  • the act of telling the story is all the accountability a well-meaning person needs to learn their lesson
  • they’ve already beat themselves up, they don’t need your help with it
• there’s always a second story - look for it
• this talk should really be titled what destroyed three mile island?

go slow to go fast: building strong foundations for leadership (alicia liu)

• (i missed the start of this)
• the higher up you go, the less people will tell you when you get things wrong
• a lot of engineers are promoted into management because they’re good engineers
  • some people are naturally good at both
  • but if you’re good at engineering, it might actually be harder to develop management skills
• people ask me: do you miss coding?
  • no!
  • coding is not about the code. it’s about the achievements
  • my best way of achieving things isn’t coding

legacy code: big rewrite or progressive rejuvenation (uberto barnini)

• @ramtop - functional programming specialist - standard chartered bank
• legacy systems!
• i used to work at banks and big institutions
  • legacy systems are old systems that probably make a lot of money for the company
  • but they’re big and complicated and nobody really knows how they work
• a friend of mine is working on them, and a legacy system is just last year’s system!
  • they’re moving quickly in order to go to market
  • the system became a liability quickly!
• what is a legacy system?
  • an old running system
  • critical for busines
  • hard & expensive to:
    • maintain
    • improve
    • expand
• how do you recognise one?
  • it was a very good design
    • possible too good!
    • people start adding extra features and use cases
    • it becomes overloaded
  • continuous state of emergency
    • i was in a bank, where there were different levels of emergency
      • level 4 wasn’t too important
      • level 1 would involve newspaper headlines
    • there was a project which was consistently at a level 2 emergency
    • I couldn’t really imagine to work on those teams because there was so much pressure
    • but it becomes kind of normal, and nobody really cares
  • we’d like to retire it but… we don’t know who is using it!
    • there was a system that needed to be retired, and they checked everyone using it
    • but: there was a system which had an annual report, and it used the old system, and the report was a legal requirement
• technical debt default
  • when it reaches some point that is so high that you could never ever repay it, that’s really a problem
  • what do we do?

option #1: “there’s no time, just fix it quickly!”

• further degradation: broken windows effect
  • if you have an old building, and somebody breaks a window, and nobody fixes it, the rest of the windows get broken quite quickly
  • there’s a sense that it’s abandoned and nobody cares

option #2: forget the old system! rewrite from scratch!

• problem: the new rewrite left unfinished
  • i worked on a really old system
  • (initial commit was older than the grad on the team!)
  • they had tried to replace it three separate times
• hidden costs of rewrite
  • legacy system complexity is always underestimated
    • because ofsuccess bias
  • all previous knowledge in the code is thrown away and slowly rediscovered
  • new features can delay the release
  • old system is neglected but remains critical
  • data migration is always more painful than expected
• it’s not that rewrites never work, but they’re very risky

option #3: kintsugi

• https://en.wikipedia.org/wiki/Kintsugi
  • fixing things without hiding the fix
  • using lacquer/gold leaf to join broken ceramics back together
• similar to option #2 but it actually works
• progressive rewrite
  • https://www.martinfowler.com/bliki/StranglerApplication.html
  • reduce risk
    • shorter release cycles
• for example: you have a C# app, but you want to migrate to scala+js
  • keep the C# app but add a Newport view onto the new interface
  • replace one feature at a time
  • using the old app as a skeleton
• a progressive rewrite could take longer than what is estimated for a full rewrite
  • but probably much less than the actual time needed

code quality

• we got it completely wrong as an industry what quality is
  • clean code?
  • design patterns?
  • TDD?
  • functional programming?
  • these are tools & techniques. they’re not quality
  • you can’t say “this code is good quality because it has 95% test coverage”
• the only thing that really matters is the time it takes to implement a feature
  • if new features are easy to implement (or bugs are easy to fix), the code is good quality

architecture

build a better hiring process with design thinking (crystal yan)

• @crystalcy - product & UX lead - independent
• i hire designers and product managers
• have you ever felt 😕 while interviewing?
  • .o(“why is this person asking me this question?”)
  • not knowing what the intention behind a question is
• or like 😱 when interviewing someone?
  • not having the tools to run the interview well
  • or give a good impression and make them want to join the team
• Hiring is
  1. frustrating
  2. stressful
  3. not fun
  4. all of the above
• hiring is also expensive
  • if you want to get buy in to improve your hiring processes, talk about time and (opportunity) costs saved
• let’s build a more intentional, inclusive hiring process, using human-centred design
  • less bias, more diversity
  • less frustration, more delight, higher yield
    • how often does the candidate accept an offer we extend?
• our goals
  • cut bias (and improve diversity)
  • set expectations (and increase delight)
  • get to yes (and improve yield rate)
• our methods
  • user interviews, journey mapping, paper prototyping
  • competitive research
    • sat in on existing sessions run by colleagues
    • other dept heads had fairly long presentations
      • that seemed exhausting to the new person
    • i thought: maybe let’s make this more interactive
    • many of these people had not been designers before
    • and main exposure had been to graphic or interior design, not UX
  • new onboarding session
    • i led interactive activities
• design challenge: how might we improve the interview experience?
• conduct research
  • user interviews
  • goal: uncover candidate pain points
  • activity:
    • have new hires pair up (in different roles)
    • ask open-ended questions
    • time-box activity (5-7 minutes)
  • it’s very easy to ask a question that is accidentally misleading
    • don’t ask “what did you like?” but “what did you think? how did that make you feel?”
• map the experience
  • journey maps clarify pain points in the candidate experience
  • example table to amp things out:

• paper prototyping & testing
  • this can be intimidating to people not used to it
  • “I work in HR, I’m not a designer”
  • prototyping isn’t just sketching an interface
    • could be: sketching an email
    • writing a phone script
• ship & iterate
  • focus on quick wins
  • this is how you make a success of it
  • make sure that someone from your hiring team is part of the process
  • lead the way by offering to implement changes for roles you’re hiring for first
• “several different people were constantly emailing me to schedule next steps… it was hard to keep their names and positions stragiht. and if i had a question, who do i go to?”
• clarify roles
  • guides help the candidate navigate the interviewing process
  • hiring managers lead the decision-making process
  • “talent” (recruitment team?) helps candidate navigate candidate journey
  • dept director leads service design
• “it was frustrating that different people kept asking me the same questions over and over again”
• interviewers often didn’t have the resources to interview candidates fairly and effectively
  • interviewer pulled in at the last minute
    • they don’t know where they fit in to the candidate experience
  • we created an interviewer guide
    • what stage are you performing?
    • who has the candidate spoken to before?
    • what are you expected to assess them on?
    • note-taking template
      • help to detail exactly what was said and what candidate answered
      • these notes would be read by someone else
  • people might interpret the way a candidate gives an answer differently
    • based on personality or bias
  • eg: candidate takes a while to get to an answer after rambling for a bit
    • different interviewers interpret this differently
      • the candidate’s a little nervous
      • they’re not a good communicator
      • they’re not good at listening - they didn’t listen to the question
    • your interpretation depends on your own personality
  • we structured it so that interviewer was responsible for writing what was said, and another person for assessing
• review
  • hiring is expensive
  • an intentional, inclusing hiring process is better
    • less bias, more diversity
    • less frustration, more delight, higher yield
  • quick wins: clarify roles, create resources
  • user interviews to uncover candidate pain points
  • journey maps
  • paper prototypes

	application	phone screen	on-site interview
people
touchpoints
emotions

functional programming for everyone (marek rogala)

• @marekrog - CTO - appsilon data science
• imagine a world where:
  • code is perfectly understandable
  • concurrenvcy is a piece of cake
  • testing is a breeze
• this world maybe doesn’t exist, but i think we can get close with functional programming
• learning FP is a great way to grow as an engineer

teaching new tricks - how to enhance the skills of experienced developers (clare sudbery)

• @claresudbery - Lead Consultant Developer - Thoughtworks
• I am a technical lead with thoughtworks
  • software development consultancy
• one of the things i do is introduce new practices to people
  • usually XP - pairing, TDD
  • also: new languages, new ways of doing things, new tools, new skills
• I’m not perfect
  • i make mistakes
  • as a teacher, it’s part of my job to help pupils believe that they can succeed
• let’s stop making other people feel stupid
• let’s stop intellectual elitism
  • the idea that experienced practitioners are supposed to be “clever”
  • you feel you’re supposed to prove to everybody else how clever you are
  • you judge people if you don’t think they are clever enough
  • you end up with this pathology where elite people talk to each other in jargon and nobody understands each other
• let’s talk about empathy
  • teaching should always be inclusive
• strategies:

pairing

• when i first heard about this, i thought it sounded like hell on earth
• why would you want to spend your whole day with someone else
• when i came back into the industry i was insecure, i felt my skills were out of date, i deliberately came back at entry level
• when i was pairing, i learned so much more quickly
• I’m also eternally grateful to all the people who said to me that they were confused

asking questions

• is importantb
  • but the way you do it is important
  • ask simple questions
  • speak clearly
  • make it clear what you do and don’t understand

the workshop

• i led a team of (non-thoughtworks) developers
• my job was to upskill them
• i have run a lot of workshops before
• don’t feel like you’re giving a professional workshop that people would spend thousands of pounds for
• it can be cobbled together
• example: people were confused about async/await in C#
  • i had learned about it previously
  • i spent an hour brushing up on it, reminding myself
  • thinking about what the most useful things to learn would be
  • and where it would be most relevant in our codebase
  • devised an example based on our codebase
  • divided it into work and gave them half an hour to work on it
  • come back, present, split up, work another half hour
  • that’s a good model for a workshop
• you can iterate on it
• it can be a complete shambles at the start
• if you don’t start with “I’m the expert, you’re the learners” then everyone can feel their way through
• have a retro at the end and work out what to improve together
• also, you can google for someone else’s workshop and just run that

group learning - separate from a workshop

• workshop is more structured
• i’ll look for a tutorial, give it a quick look to ensure it’s not completely bonkers
• and go through it together - one screen, one keyboard, rotation timer (you need a big screen for mobbing though!)
• or you can split into pairs
• then do a quick post-it retro at the end

don’t overload

• again, empathy
• pay attention - if you’re going too fast, slow down
• don’t give yourself a learning deadline
  • don’t say “by end of day, we’ll know A,B,C,D,E”
  • because it might just be too much
  • and different people learn at different paces
    • you can get the faster ones to teach the slower ones
    • the best way to consolidate your learning is to explain it to somebody else

lunch & learn

• make learning part of the culture and show that it is celebrated
• regular sessions where you tell each other stuff
• anything! doesn’t have to be technical
  • magic tricks, rubik’s cubes, etc

asking questions (again)

• ask them all the time!
• i want to know the answers!
• i want my team to know the answers!
• i want to show that no question is too stupid
• i don’t like when people use jargon
• people in junior positions say to me: “i’m glad you asked that question, i wanted to know the answer but i thought it was a stupid question”

mobbing

• I had the same first impressions as with pairing:
  • pairing is brilliant, but mobbing seems ridiculous
  • 6 people all on one computer?!
  • but: you’ve got all the knowledge on the codebase in the same place at the same time

deep dives

safety & security

• make an environment where everyone can come to work as themselves

highlight good examples

• when you find them in your codebase, use them and point them out to people

nobody is perfect and nobody knows everything

• retros & feedback help the team own the work that they do
• feedback isn’t a way to tell people what they are doing wrong
  • positive feedback matters
  • feedback is a way to scaffold people and support them

tips for managing a widely distributed team (dirkjan bussink)

• @dbussink - principal engineer - github
• i work for github
  • we’re pretty well-known sticker company
  • i was a senior engineering manager until recently
  • my team has grown from 10 to 30 people
    • i was doing people management, also managing some of the managers, but also involved with the technical discussions
    • it became untenable as we grew
    • we hired some people
    • i’m now focussed on the technical leadership part

why distributed?

• i never realised the way i’d be working today when i graduated
  • internet, video conferencing
  • i never realised that would be viable
• if you as a company want to work with amazing talent you’re not going to find them all in one place any more
• not everyone wants to move
• do you really want to pay to have everyone in san francisco
• not everyone wants to work in a distributed team, some people prefer working in an office

context

• our team:
  • all things i talk about today are things that work for us
  • but may not translate directly to your situation
• today we’re 28 people, 5 different continents (or 7)
• today people work from 11 different countries in the world
  • US coast to coast
  • europe
  • africa
• timezones: biggest difference is 10-11 hours
• people say to me this must be very hard
• I see it as easy mode
  • because of how spread we are, everyone’s in a similar position
  • there’s no advantage to being in any particular location
• the struggle companies have is that they have a big office where most people are
  • there’s a few people scattered around
• one simple thing you can do is: when having a meeting, everyone dials in separately
  • you start adopting conventions
  • eg raise your hand before speaking
• although: sometimes people in the same office might as well not be
  • see slack outage where people joked on twitter that they might have to talk face-to-face
• my first day:
  • i accidentally took staging down
  • someone across the world in the US east coast jumped in and helped
  • this worked because we put conversations somewhere they were public

be explicit about how you will work

• when you work on a distributed team, you have to be much more explicit about how you do things
• psychological safety
  • https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/

get people together regularly

• regular cadence
  • get everyone together regularly
  • you need budget to do this
• use time together well
• what is the time together for? it’s expensive, but also high bandwidth
• so set explicit expectations about what you’re trying to get out of it
• examples:
  • long term strategy
  • team structures - are they working? what should we change?
  • iterate on collaboration processes
    • do we want to hire more people? what’s our strategy here?
• no day-to-day development
• do non-work activities
  • when people are colocated, they might do non-work things together (karaoke? dinner?)
  • try to capture this
  • grab dinner together
  • we did some things to teach coding to kids via https://code.org/
    • we really enjoyed it
    • we got a lot out of it as teachers
  • lightning talks about non-work topics
• tell people that it will be tiring

communication after the get-together

• keep talking
• build these relationships in regular 1:1s
• non work video chats
  • world cup
  • which country is screwing up in local politics today
  • coffee machine chat / water cooler
• i was trying to sign up for a website that had an on-paste disable form handler and i had to hack it to make it work with my password manager and i wanted to complain
  • just complain in chat
  • other people share feelings together
  • another eg: wedding photos
• rubber-ducking with myself in team chat
• be aware of your timezones though
  • don’t always make your meetings a US-friendly time

asynchronous communication

• it’s hard to ad hoc jump into a meeting if you have a 9/10 hour time difference
• talk in public about what you’re doing
• slack has an interesting metric: public/private message ratio
  • you want more public than private
  • in an office, people can jump in if they think they can help
• talk in public channels
  • also: github issues & PRs
• status updates
  • is it useful to have a meeting to do status updates?
  • often people use meetings as a crutch
  • if i ask for people to update their status in a document, it doesn’t happen
  • so i schedule a meeting to force it
• no instant gratification
  • if you have a question, you won’t get a response for at least a workday
  • even if it’s urgent
• you need to work on unblocking others
  • if someone’s waiting on you, you need to get back to them
• don’t block yourself
• over-communicate
  • any blame-free postmortem you’ll find a place where communication broke down
  • by the time you feel you’re communicating too much, you might just be doing enough
• use restraint
  • be aware of how and when you communicate
  • as a manager, I’m fine with people sending me messages any time, but i won’t answer it
  • stop working in the evening
    • this is important!
    • especially if you’re europe, working with US people
    • they’re actively working away
• hardly anything is truly urgent
• “Durch einfach liegen lassen erledigt”
  • (ed’s translation: “finished by simply leaving it be”)
• use timezone spread
  • remember heartbleed? shell shock? people started remediation, then handed over to next timezone

how to run awesome tech internships! (beverley newing)

• @webdevbev - front end developer - oxford computer consultants
• I haven’t had an intern but I’ve had work experience students, and I’ve been an intern myself 3 times
  • 2015: graduated (english & german literature)
  • codefirst:girls internship
  • zooniverse internship
  • oxford computer consultants internship
  • turning permanent at oxford computer consultants at the end
• we talk a lot about diversifying our industry
  • this is a really broad area, and women are just one part of that
  • only ~15% of compsci grads are female
  • if we want to do better, we can’t just hire compsci grads
• when i was at codefirst:girls, i was told there were four options:
  1. bootcamp
     • paid bootcamp like makers academy
       • i couldn’t afford this
  2. graduate schemes
     • often require a STEM degree
  3. apprenticeships
     • most of these requre you don’t have a degree
  4. paid internship
     • this was the only option open for me
• lots of people i met were in my position: not quite ready for a junior position. but how do they get there?
  • i curated the newsletter for codefirst:girls for 7 months
  • we only had one internship come through to us
  • this newsletter goes to 4000 people!
• what is an internship?
  • on-the-job training within a team
  • paid, beginner-suitable opportunities
  • open to anybody regardless of background
• duration
  • I recommend 6 months
  • I did 3 months
    • it takes 1 month to settle in
    • then you’ve only got 2 months left
    • you need to start finding another job
    • applying for jobs but not knowing how to do the tech tests
• salary
  • at least a living wage
    • you are working. you are adding value
  • Oxford is at least as expensive to live in than London
  • you need more than national minimum wage
  • you shouldn’t be too stressed about money
• there is a legal meaning to an internship (link at end)
• structure
• support
  • this is really important to people who are switching careers
    • i gave up a proper contract to do an internship in a new city
    • it was a hell of a gamble!
  • be clear to your intern about what support you will give them
• interacting with an intern
  • be nice in interviews!
  • if you stress someone in an interview, all you learn is how they deal with stress and not the other parts of them
  • it’s important to tell them what they can learn
    • they may not have a compsci background
  • include them in team meetings
    • often they come from other career paths
    • they have lots of skills
    • skills you may not have!
    • eg: during my degree I did some samaritans volunteering and ran active listening workshops
• i run codebar oxford
  • i teach underrepresented people in tech
  • i meet all these people who want to get started and ask me how to do it
  • and I don’t know what to say to them
    • move to london? leave your family behind? do a bootcamp?
    • these are really hard for lots of these people
    • especially mums
• links
  • https://www.wisecampaign.org.uk/
  • https://www.gov.uk/employment-rights-for-interns
  • https://textio.com/
  • https://talent.santander.co.uk/santander-universities-sme-internship-programme

strategies to edit production data (julie qiu)

• @jqiu25 - senior software engineer - spring
• spring - a fashion ecommerce marketplace
  • i lead the catalog team
  • i spend a lot of time thinking about things like how data gets ingested into our system
  • i have spent more times than i care to admit sitting at a SQL prompt wondering how to inject data into production
  • we know it’s not good practice but we do it anyway
• why?
  • internal tools are not available
  • edge cases exist
  • time-sensitive changes are needed
    • often, the SQL prompt is the fastest, easiest solution
• example:

UPDATE products
SET name='julies-product'
WHERE id=1

• I’m writing this query and often I’d get someone to check it over my shoulder
  • but: say it’s friday night and everyone else has gone home
  • and someone is telling me we promised it by monday
  • noone available for review
  • “what’s the worst thing that could happen?”
• I run the query
  • grab some water
  • tell marketing i did it
  • then look at my laptop
  • and realise this is what I actually ran:

UPDATE products
SET name='julies-product'

😱🤯

strategies for safer editing

in order of up-front effort, and benefits received:

develop a review process for manual SQL edits

• how it works
  • we started storing SQL queries in a google spreadsheet
  • checklist of what needs to be done
    1. add a record to the spreadsheet
       • name, date, reviewer
    2. reviewer approves
    3. run the query
       • inside a transaction!
       • if you make a mistake halfway through, it’s easy to roll back
• what’s great
  • easy to implement
    • just start a spreadsheet and send an email
  • audit trail
  • promotes right behaviours
• what’s not so great
  • easy to make mistakes
  • audit trail is at will - room for error
  • difficult to run long and complex logic

What if marketing wanted something more like this?

UPDATE products
SET name='julies-product'
FROM brands
WHERE brands.name='julies-store'
   AND ...

local scripts

• how it works
  1. write a script
     • i do it in python
     • i add an argument parser so it’s reusable
  2. connect to remote database
  3. run the script
     • i like my scripts to have a --dry-run flag
• what’s great
  • reusable
  • easy to manipulate outputs
  • access to common code from your repos
    • reuse logic you’ve already written before
• what’s not so great
  • easy to make mistakes
    • no code review
  • outputs are only available on user’s machine
    • and they go away when you exit terminal
  • network disconnections

How do we go about running really long scripts?

UPDATE products
SET name='julies-product-1'
WHERE id=1

UPDATE products
SET name='julies-product-2'
WHERE id=2

UPDATE products
SET name='julies-product-3'
WHERE id=3
...

• I used to do this by coming into work really early
• but even after all day it was only 25% done
• what if we had a computer to run scripts that was on all the time?

existing server

• how it works
  1. write a script
  2. get the script onto the server
     • deploy?
     • upload?
     • compile first?
     • scp?
  3. ssh and run inside a session

until one day:

• “heyyy jenkins is maybe down? returning 504s on browser, can’t ssh in”
• i suddenly realised that the script i was running on it was using all the CPU and none of the engineers could deploy anything
  • oops
  • fix: reboot the instance

lesson learned.

• what’s great
  • ability to run long-running scripts
  • reliable network connectivity
  • no new infrastructure
• what’s not so great
  • scripts can affect resources on your server
    • cpu, memory
  • not user friendly
    • manual copying
    • ssh
    • start session
    • all just to run a SQL query
  • no persistent audit trail
  • only the engineer that started it has access

task runner

• how it works
  • dedicated jenkins job
  • jenkins job page gives free UI
  • recipe
    1. write script
    2. code review
    3. input arguments to jenkins job and build
• what’s great
  • persistent audit logs from jenkins itself
    • good for tracing what happened to scripts
  • code review and automated tests
  • UI
    • minimised room for error
  • we’ve come a long way here!
• what’s not so great
  • hard to manage credentials
    • this was the main driver for us to look into better solutions
    • we found a dev was still just scping scripts to the server
    • it was because his script had >30 creds to manager
  • environments not clearly separated
  • inputs not verified

we have all these problems: configuration management, logging, separating concerns. haven’t we solved these before?

script runner service

• how it works
  • a new service, architected like the rest of our pipeline applications
  • single UI
  • different script runner service in each environment
  • recipe
    1. write script
    2. code review and run tests
    3. choose environment and run
• what’s great
  • centralized configuration management
  • separation of environments
  • user friendly interface - by far the friendliest
  • we can do more!
    • parallelize and scale our instances
    • preview results
    • up to you to customize!

	raw sql	local scripts	existing server	task runner	script runner
data editing	✓	✓	✓	✓	✓
complex logic		✓	✓	✓	✓
long scripts			✓	✓	✓
testing				✓	✓
code review				✓	✓
audit trail				✓	✓
user interface				✓	✓
separate environment					✓
centralize configs					✓
…and more!					✓

key lessons

• make something usable
  • don’t just think about safety, think about speed and usability
  • that’s why we’re getting behind the SQL prompt in the first place
• invest the effort - it’s worth the cost

goal-setting workshops for managers (melinda seckington)

• @meckington - technical manager - futurelearn
• “where do you see yourself in 5 years time?”
  • i always used to dread this question
  • i felt bad because i felt i “should” have an answer
    • rather than just making it up on the fly like i always did
  • it felt like there was this expectation that someone already has come up with this grand goal
    • things like SMART or GROW are about reaching those goals which are already defined
  • what if someone has no idea?
• thing I started doing with people i manage: explicit goal-setting workshops
  • generate as many ideas as possible
    • broad rather than deep
  • then, later, select which are most important
• format depends on person
  • one catchup or several
  • send questions beforehand
  • leave it up to each person how much prep they do
    • some are happy to get these questions thrown at them
    • others want time to reflect beforehand
  • provide enough time & space to reflect
• four areas & types of questions i ask:
  • values (intrinsic/ideal)
  • future (extrinsic/ideal)
  • current role (extrinsic/now)
  • current skills (intrinsic/now)
  • in this order, because the answers to the early phases can influence answers to later phases
    • eg if they say “i value doing things efficiently” you can ask “do you get to do things efficiently in your current role?”

values

• ask questions to get someone to reflect and think about who they are and how they do things:
  • what are the building blocks that make up you?
  • how would you describe yourself in 5 words?
  • what is your purpose? your passion? your motivation?
  • where do you get your energy from?
• personality tests & quizzes as conversation starters

future

• what do you want in your future?
• think about long-term goals
• ask questions to get people thinking about what their ideal life and career would be:
  • imagine you’re retiring and looking back at your life: what do you want to have achieved?
    • (this is easier to answer than “where do you find yourself in 5 years?”)
  • what does work-life balance look life for you?
  • what does your ideal day look like?
• extra: come up with anti-goals
  • what don’t you want to happen?
  • I never want to be bored
  • i never want to work overtime

how is your current role?

• how have last few weeks/months been?
• ask questions to reflect on current role and company
  • what are the challenges of your role?
  • which of your abilities, interests and/or values are you able to do in your role? which would you like to do more of?
  • what would you miss if you left your role? why?
  • what kinds of people do you most enjoy working with? what work environments

what are your current skills?

• what do you do better than others? what do others see as your strengths?
• are there any gaps in your knowledge or experiences?
• what technical/communication/leadership skills can you develop?

how might this turn out?

• people might be drawn to one particular area, or two
  • the purpose isn’t to have something in all 4 areas
• this is just to get people to reflect and to lead them down that rabbit hole
• if you don’t help people set goals, you end up with the human equivalent of tech debt
  • you wake up one day and don’t really like the person you’ve become
• “which way you want to go depends on where you want to get to…”

how to survive the single page app-ocalypse (jim newbery)

• @froots101 - front-end coach & consultant - tinned fruit
  • based in edinburgh
• I’m a consultant. i aim to help web product teams take their UI engineering to the next level
• i’ve seen a common pattern
• controversial premise: single page apps damage businesses
  • yes, this is a clickbait title, no it’s not all SPAs
• google maps - one of the first if not the first
  • big team, lots of investment
  • application-like experience
  • dynamically change bits of page in-place
  • APIs to get data
  • client-side routing and history
  • static asset bundling
• in reality, a lot of SPAs are much more mundane
  • they are CRUD - Create Read Update Destroy

challenges to survival

• easy to make!
  • if you’ve ever picked up react beginner tutorial you can make a todo list quickly
  • but it’s hard to make good
  • extremely hard to make good, simple, fast
• ⚠ WARNING anecdotes ahead
• it can feel like you’re adding a lot of value quickly in the beginning
• but there’s this mid-stage where it starts to get really hard and feels like a traditional app might have been better
  • you have to push through this to get to the great UX
  • lots of teams never really fulfil that vision
• as developers we naturally blame the tools
  • react, vue, ember, …
  • hype cycle
  • currently vue is the new shiny to react
  • “this time it’ll be different”
  • in extreme cases:
    • backbone -> angular -> react -> vue
    • this is bad for business
    • reinventing user value again and again
    • I’ve done this myself!
• how do we break this cycle?
  • frameworks don’t make apps, teams make apps
  • things to think about before big tech changes
    • is there a product vision?
    • are you all aligned on it?
    • does the team have good comms and psychological safety?
    • skills and capabilities?
    • dev practices
    • what the damned thing actually does
• a rewrite encourages forgetting
• a rework requires learning
  • it’s harder but in the long term much more valuable
• reworking a SPA is really quite hard
  • we’ve been quite monolithic about them
  • only recently have we been looking into ways of breaking them up
  • smaller apps make it easier to find and eliminate waste
  • what’s the simplest way to sustainably add and maintain user value?
  • learn how to live together with your product

using agile techniques to build a more inclusive team (kevin goldsmith)

• @kevingoldsmith - VPE - AstrumU
• this is not a talk about agile
  • before agile, the way we talked about software came from industrial manufacturing processes
  • agile brought us much more people-focussed techniques
  • i’ve developed a bunch of different strategies and frameworks that are inspired by agile
• scenario: first 1:1 with someone
  • someone joins your team, or you move team, or whatever
  • if someone’s remote, i like to do first two 1:1s in person
  • personalise a joint working agreement
    • chance for you to talk about how you approach management
    • chance to discuss what the responsibilities of the role are as you both see them

joint working agreement

• two columns of post-it notes
  • expectations on lead and on report
  • one expectation per post it
  • 10-15 minutes or until you can’t think of any more
• take turns
  • pick up a post it, discuss what it means to you together
  • does this make sense?
  • do we agree? do we disagree?
  • example: “i expect you to be on time to all team meetings”
    • “but on wednesday i do the school run so i’m going to miss standups”
• take a photo
• refer to it in later 1:1s
• revisit periodically
  • see how these expectations change
  • you both get better at your jobs
  • you both have more expectations on each other

mentoring a future lead

• how do you go about this?
• what are the responsibilities of a lead at our company?
• brainstorm this
  • each person writes down all the responsibilities of a lead (one per post it)
  • spend 10-15 minutes or until you can’t think of any more
  • four columns
    • Pat’s responsibility
    • Pat approves
      • it’s a responsibility of mine that i’m going to assign to you
      • i want you to make a plan around this and i want to say yes or no to it
    • Ashley informs
      • i’m going to give these responsibilities to you, you just do them
      • but i want you to tell me when you’ve done it
    • Ashley’s responsibility
      • this is now just your responsibility and i don’t need to hear any more from you
  • take each post it
    • what does it mean?
    • are you ready to take this on?
    • eg hiring
• capture the output
• refer back to it
• revisit every 6 months
• see things move across the board
• “wait a minute, I’m just giving someone else my job”
  • that gives you a chance to expand your skills
• thanks to esther derby for the format
  • i got into senior management because i was so good at seeing the details
  • but my team grew beyond my abilities
  • she showed me this format to help me delegate

scenario: getting someone to talk in 1:1s

• “is it okay if we skip 1:1 this week? there’s some things that need to be delivered”
• “everything’s fine, we don’t need to do it”
• then you force it on week 3
• and they just sit there and say “everything’s good”
  • you can get stuck here
• take a walk
  • guarantees the 1:1 will last a while
  • talk about the match, talk about TV
  • work is a shared experience you both have, you’ll slide into work as a subject naturally
  • toys!
    • something else to focus on
    • you’re not just staring at each other
    • i have a whole series of these things on my instagram
• shut up!
  • wait them out
  • people don’t like silence, so they’ll fill it
  • give them a chance to choose their words and form their thoughts
• “how’s it going?”
  • “good”
  • …
  • i couldn’t handle the silence
  • so i just talked the whole time
• next week i resolved not to say anything
  • “how’s it going?”
  • “good”
  • …
  • …
  • …
  • …
  • …
  • …
  • …
  • …
  • we sat there for what seemed like 2 hours
  • it was maybe 45 seconds
  • he said “i’ve been thinking”
  • and he started to talk!
• i had been robbing him of the opportunity to gather his thoughts

team meetings

• polling - i want your opinion, but i’m the one making the decision
  • give people the freedom to express their opinion but it’s consequence-free
  • you are accountable, not them
  • people have imposter syndrome
  • if you’re not sure if you “should” be part of the group you’re much more likely to hold back
  • take temperature on something
• voting - we’re making the decision together and i will support the group’s decision
  • you absolutely must support and defend that position no matter how much you disagree
  • if your team votes to implement layer 7 routing, you need to defend it to your boss
    • not: we voted and the team said to
    • you’re throwing them under the bus
• be very clear which you are doing
• tell people before they give their opinion
• fist of five
  • fist: i hate this
  • 5 fingers: i love it
  • good for polling or seeing how close the team is to consensus
• roman voting
  • count to three, everyone does thumbs up or down
    • not allowed to be in the middle

scenario: collaborative team meeting agenda

• i thought agendas were too corporate
• shared agenda
  • shared document to capture the agenda and the minutes
  • get agenda items at least 12 hours before meeting
  • the morning of the meeting
    • i groom
    • you wanted 15 minutes for this, we don’t have time, i’ll give you 10
  • a facilitator (you or someone else) runs the agenda
  • person who added the item leads the discussion
  • good for team of 7-10 people
• bigger teams: lean coffee

scenario: more inclusive team meetings

• shut up!
  • positional authority encourages people to conform to your opinion
• set expectations
  • be clear on desired outcomes
• observer role
  • role: watch (not participate)
  • people interrupting each other
  • people repeating ideas without giving credit
  • certain people speaking too much
  • exclusionary language
  • passive or active
    • passive: takes notes
      • at 4.35, pat was talking and ashley interrupted
      • at 4.40, pat said an idea
      • end of meeting: reads back
    • active: when they see it they say it
• rotate roles
  • note-taker
  • facilitator
  • observer
    • frees you from having to participate
• publish the agenda
  • lets everyone prepare mentally and emotionally for the meeting
• remember your role power
  • you set the tone and conduct for your team based on your actions and how you react to others
  • there is the loneliness of leadership
  • one thing I’ve seen with good leaders is they may be good at not interrupting, not stealing ideas
    • but they don’t call other people out
    • we have to say “we don’t do that”
  • it’s hard! you don’t want to not be liked. you don’t want to have a difficult conversation

from the hallway track: prometheus ideas with Chris Evans

• @evnsio - platform team lead, monzo
• note that this bit is very tailored to what I found useful and relevant, so it skips over a lot of detail
• thanos isn’t just a persistent storage engine, it can also be used as a query layer to load balance requests across multiple prometheis (for resilience)
  • deploy thanos as a sidecar to prometheus
  • expose ports for both prometheus direct and thanos
  • (this is easier in kubernetes than in ECS)
• comcast’s trickster is a query optimization layer which does promql-aware caching: eg it caches datapoints and transforms requests for new data into deltas
• service discovery for prometheus in k8s
  • k8s service manifest
  • cluster IPs
    • each pod (task) has its own IP - separate from the host IP
    • IPs are ephemeral
  • service: stable DNS name to talk to that thing
  • config files: configmap
• developers using prometheus locally
  • still a work in progress