Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Lead Developer London 2018

Lead dev 2018

Welcome - Meri Williams

  • @TheLeadDev #LeadDevLondon
  • white coat captioning - @whitecoatcapxg
    • wvnts.co/lduk2018

The Container Operator’s Manual - Alice Goldfuss

  • happy pride! it’s like the world cup for people with fashion sense
  • who am I?
    • SRE @ github
    • professional worrier with sudo access
  • car advert analogy
    • beautiful mountain road,
    • lots of facts and figures
    • very very small print:
      • “laboratory setting”
    • it’s an advert
    • this is how i feel about most container talks
    • “just download this gist, (don’t read it!) and run it”
    • “oh you want to run containers in prod? no problem! prod is just your laptop times 1000”
  • we need to examine containers a little more critically than we do today
  • my creds
    • I’ve been running containers in prod since early 2015 (docker 0.6)
    • ran dockerized cassandra clusters
    • now, i run the k8s platform behind github

what is a container?

  • “docker 101 talk view”
    • host (which is a box)
    • OS (another box)
    • daemon (another box)
    • containers (little boxes)
  • this is all lies
  • it’s a good way for a human to see containers
  • but this is how your computer sees containers:
    • top command
      • python
      • ruby
      • kubelet
    • those python & ruby processes are containers!
    • containers are made of a bunch of linux features stapled together
    • when you build your Dockerfile, you’re making a tarball
    • when you run it, you’re starting a process (just like any other process)
    • this is where those extra features come in
      • namespaces
        • what it can see
      • cgroups
        • what resources it can use
  • 4 lessons in this talk

lesson #1: containers have strengths

  • they are good when you use them in the right way and play to their strengths
  • strength which are: ephemeral, disposable processes
    • ie stateless applications
    • take data in, change it in some way, send it on to somewhere else
    • fantastic for scaling
    • run multiple instances side-by-side, behind load balancer
    • connect to same backend
    • if one dies, nobody cares (ephemeral)
  • what do containers give you?
    • they’re portable
      • they don’t depend on dependencies from system
      • easy to iterate
      • easy to upgrade
    • disaster recovery
      • divergent DR processes:
        • jenkins
        • cap
        • some random cron job
        • someone has to sit with you to show you how to deploy their apps
      • if it’s containerised, everything is deployed in the same way
      • also great for production incidents
        • if i get paged, and find the problem isn’t my service, it’s someone else’s service
        • in the old world, you’d have to page them
        • now, you can at least try redeploying their service (because it deploys the same way)
    • testing environments
      • ephemeral spaces with disposable things that you want to clean up afterwards

lesson #2: containers have weaknesses

  • stateful apps
    • eg databases
    • “I attended one of those container 101 talks and now i want to containerise all my databases”
    • my response: “are you google?”
      • really: “are you routing billions of requests to thousands of database across hundreds of databases per second?”
      • if not, you don’t need to containerise your databases
    • reasons to try:
      • faster provisioning
      • stability
      • recovery
    • if your current processes take months to add or upscale a database, it can seem really attractive
    • but: you can get all these benefits from a cloud provider and some tooling
      • “omg that will cost $$$$, we’ll build our own”
      • how much do you think building your own is going to cost?
  • 2 main ways people do this:
    1. containerised database instances, data goes to some network-backed storage like ceph
      • use something like etcd to handle failure and service-discovery
      • BUT: eventually you’ll be network-bound
    2. store data on host using a mounted volume
      • if one dies, what do you do with the stale volume still on the host?
      • you could redeploy..
        • (this is what we did for cassandra. we had to write a lot of our own tooling)
  • maybe: use a cloud provider instead
    • automatic failover
    • scalability
    • read replicas
    • multi-regions
    • everything you need without worrying about:
      • becoming network-bound
      • securing data properly
      • etc

lesson #3: containers need friends

  • how will you build your container tarballs?
    • you don’t have to use docker in prod (do you want to? you should read about it)
  • orchestration & scheduling
    • resource management
    • placement
  • management
    • how will you manage clusters?
    • how will you drain traffic?
    • how will you do automatic failovers?
  • networking
    • how will you handle routing, access control, service discovery?
    • there’s a dozen tools that just handle networking in kubernetes
      • linkerd
      • istio
      • envoy
      • calico
  • how do you hook it into your existing infrastructure?
    • deployment: is it the same tooling? how do you support two parallel deployment pipeline?
    • monitoring: what new metrics do you need to be monitoring on your new platform? what counts as a healthy container?
    • provisioning: how do you provision new container hosts? what gets priority? container hosts or legacy hosts?
    • debugging: how are you going to troubleshoot with this new abstraction layer? how will you empower the devs running apps on this platform to debug their own services? logging platform? training?
  • gradual rollout. try some things, some fail, try new things
    • minimum 1 year to rollout a new container platform
      • just one service, on a relatively stable platform

lesson #4: containers need headcount

  • behind closed doors:
    • “we need containers”
    • “let’s give it to ops”
  • containers is a completely new can of worms
  • you’ll need a new team
  • you’ll need specific skills:
    • ops
      • you need to know where the skeletons are buried
    • deployments
    • tooling
      • you’ll need tooling to glue the platform together
      • you’ll need to test it
    • monitoring
      • someone who knows the right metrics to check when you see weird phenomena
    • kernel engineer
      • analyse kernel crashes
      • understand upstream
    • networking
    • infosec
      • containters have a bad reputation but they’re making great strides
    • internal adoption
      • good relationships with other teams internally
      • you need to manage other teams’ roadmaps to get early beta customers
    • project manager
  • each of these skills doesn’t need to be a separate person, but you also need more than one person
    • you need at least 4 people, ideally 6-8
    • less than 4 is death for on call
  • empower them to succeed
    • authority
    • budget to try out new cloud instances

should we use containters in prod?

  • do you have
    • stateless services?
    • a large, hetorogeneous platform?
    • do you have time, money, people, org support?
    • yes, maybe you should do containers
  • do you have
    • a monolith and a few services?
    • a small team with no org support?
    • just want to spite me?
    • maybe don’t do containers?
  • do you want containers or a blog post?
  • it’s ok not to use containers

First Steps as a Lead - Dan Persa

  • why become a lead in the first place?
    • dealing with people problems is much more interesting than dealing with computers
    • i asked a friend for advice who had done it before
      • he said: “dan, don’t do it”, you like coding too much
  • each company interprets slightly different way:
    • tech lead?
    • people manager?
    • part of your job is to understand what your role is about
  • getting into leadership is not trivial
    • ask for help of people around you
    • your lead, for example
    • head of dept set up a step-by-step process for how to become a lead
    • part of my plan was going through a 3-month transition of intense mentoring
  • surprises:
    • how much of my time i had to spend in meetings
      • strategy
      • agile
      • 1:1
      • they’re important
      • i was getting energy drained, couldn’t keep pace
      • i got better in time
      • better at choosing which meetings really mattered
    • career switch
      • this isn’t a promotion, it’s a different skillset
      • becoming a lead isn’t a natural evolution of being a developer
      • the DRY principle is not a good principle with people
  • how do you learn new skills really fast?
    • shadowing my mentor
    • having his support helped me have a smooth transition and kept me from feeling lost
    • applying what i read in books
      • not just reading books
      • put it in practice!
    • reflect and remember good and not so good behaviour from other managers in the past
  • example: i always hated not feeling i was in an open and safe environment
    • without mutual trust, there couldn’t be an open and safe environment
    • how do you earn trust of team?
      • it’s not easy: trust means opening yourself up to be vulnerable
    • regular 1:1s
    • transparency in decision-making
      • especially around performance evaluation
    • the best solutions to problems come from the team
      • have the team take ownership of the problem
    • feedback loops
      • ask for feedback from team on any occasion
      • healthcheck meetings with the team
  • not taking a decision is a decision
    • we are accountable for both the decisions we make and the ones we don’t or postpone
  • feedback culture
    • give feedback as close as possible to the moment in time where the action which generated the need to provide feedback happened
    • decouple development feedback from performance evaluation
  • why did your company hire you as a lead, rather than as just another software developer in your teams? understand this and unleash your lead superpowers.

It’s Personal - the art of giving and receiving code reviews gracefully

  • @alexhillphd
  • why code review? they’re good at some hard-to-quantify things:
    • fucntionality
    • readability
    • maintainability
    • scalability
  • but it’s also a direct critique of someone and their work
    • defensiveness is a common feeling
  • not all comments are equal
    • two axes:
      • low reward - high reward (how important is the comment?)
      • low conflict - high conflict (how likely is recipient to feel defensive?)
    • low reward high conflict:
      • whitespace, typos, arbitrary preferences
      • use a linter instead to resolve these! don’t do it in code review!
    • low conflict:
      • missing tests, bugs, rogue println statements
      • no problem
    • high conflict high reward:
      • design pattern choices
      • over-engineering
      • readability
      • naming choices
      • duplicated logic
      • active collaboration needed
  • conflict resolution archetypes:
    • competing
    • collaborating
    • compromising
    • avoiding
    • yielding
    • (dual concerns model)
    • we want to achieve the collaborative approach
      • lower defensiveness
      • raise confidence
  • as an organisation;
    • pair program
    • discuss tasks prior to implementation
    • not silo codebases
    • ensure everyone reviews and all code is reviewed
      • not about singling people out
      • seniors don’t get to commit without review
  • as a reviewer:
    • raise code by a grade or 2, no more
      • C-> B+, B->A, but not D->A
    • language: use “we” not “you”
      • “you should have re-used this function” -> “we could reuse this function”
    • ask questions
    • give positive feedback
      • “this is a cool library”
      • “this refactor makes a lot of sense”
  • as an author:
    • practice the “as if” technique
      • how would i respond if [i were in my best mood, i were<role model>, this code wasn’t mine, the comments were phrased differently]
    • say ‘thank you’
    • annotate your review first
    • solicit feedback with specific questions
  • summary
    • anticipate feelings of ownership
    • minimise unnecessary conflict so you can focus on the important ones
    • nudge towards collaboration by reducing defensiveness and raising confidence
  • blogpost on website

knowing me, knowing you - growing teams to continuously deliver (pia nilsson)

  • @pia_nilsson - engineering manager, spotify
  • a journey in combining the lean & XP practices with a focus on the team
  • flow in lean: kanban board
    • all about optimising speed from backlog to done
    • find bottlenecks - tech and people
    • increase the flow of work
    • at spotify, we were doing a lot of things, but not finishing many
      • lots in “develop”, not much in “review”, “validate” or “done”
    • we knew we had some bottlenecks
  • what did we do?
    • WIP limits
      • improved flow of work, got more stuff done
      • but it also increased friction
        • devs couldn’t pick up a new shiny task
        • had to instead find that key person to get that review done
    • test-driven development and domain-driven design
      • talk about why it makes sense to do tests first
      • teaching sessions on DDD
      • architecture discussions talking about boundaries of a microservice
      • increased flow more
      • but it also increased friction
    • pairing & mobbing as a default
      • added to working agreements - you are expected to pair
      • expected to decrease silos
      • everyone was on board
      • increased flow even more
      • but still increased friction
      • how can we keep the flow but reduce friction?
      • that’s this talk

blockers of flow

blocker #1: I can’t do brainstorming because i haven’t had time to think on my own first

  • need to create safe spaces for people to share opinions without worrying about being right
  • read up on google project aristotle - “psychological safety”
  • “how confident am i that i won’t be blamed or laughed at if i make a mistake or ask a dumb question?”
  • “code creme brulee”
    • hard on the surface, soft on the inside
    • we set up to scratch that surface
    • ask the stupid questions
    • speak up about a task, how it makes you feel
    • try to be a bit vulnerable
  • training in toxic communications
    • based on studies from Dr Guttmann who studies married couples for 40 years
    • four toxic communicaton patterns:
      • criticism
      • defensiveness
      • stonewalling
      • contempt
    • we introduced “i messages”
      • “this sucks” -> “i think this sucks”
      • reduces defensiveness
    • contempt is often seen in teams with a lot of sarcasm
      • appreciation is the antidote

blocker #2: “I’m interrupted all the time”

  • practice active listening
  • “yes and”
    • comes from improv theatre
    • build on other’s ideas
    • never say “no” or “but”
  • “this unit test does too many things, I’m going to delete it”
    • “yes, and let’s write a new test which is clearer”
    • it’s about hearing a third opinion which you might never hear if you said “no but”
  • we’re passionate people
  • it’s not great to be interrupted

blocker #3: “we think so differently, it’s just better if we don’t pair together”

  • practice: friendly feedback (without blame)
    • conflict is a tragic expression of an unmet need
  • non-violent communication cheat sheet:
    • when you [merged to master yesterday], i felt [confused], because i have a need for []. Would you consider [checking in with me first]?
    • key point: don’t leave feelings to implicit interpretation; explicitly talk about them
  • turn up the good

blocker #4: “I’m not sure about the purpose of this team or what I’m doing here?”

  • if the team is moving really fast, it’s important to know where we’re going and why
  • Objectives and Key Results
  • can really help if used properly
  • in the CD backend team we had these uninspiring objectives
  • objectives need to be really inspirational
    • objective: centralise all CI at spotify
  • key results: measures that show how far along we are towards our objective
    • key results: increase customer base of platform by 60%
    • note: no mention of how we achieve this
    • that’s up to the team
  • despicable design
    • think about our users and how our service could be made worse
      • service window midday
      • slower response times
      • postal reviews
      • stuff like that
    • it was fun and it got us thinking about our users

summary

  • how we increased flow while reducing friction
    1. get the talking going
    2. active listening
    3. friendly feedback
    4. answer why
  • this isn’t static at all - you can’t do it once and forget about it

how i learned to stop worrying and love meetings (menno van slooten)

  • @mennovanslooten - front end chapter lead - backbase
  • I’m not sharing this story because i think it is special
    • but because i know it is not
    • and it would have helped me to hear it before i became a lead developer
  • i had been a FE dev for >15 years
    • one of my old employers found i was on the market and invited me to become a chapter lead
    • the experience of becoming a lead dev was.. not great
    • i’m hear to explain how i felt being a lead developer as opposed to being a developer
  • i loved being a developer
    • i loved programming
    • i loved the way code looks in an editor
    • i loved abstract puzzles
    • i loved learning about some 50-year-algorithm i last heard about in university
    • it was not just satisfying, it was really rewarding. fulfilling.
    • i could open up a browser and show my work to my friends and peers and family and say “i built that!”
    • i worked really hard to get good and stay good.
  • but: a nagging voice in the back of my head
    • “how long do you think you can keep doing this?”
    • i had trouble keeping up
    • people much younger, less experienced, who were creating solutions i didn’t understand, to problems i didn’t realise i was having
  • it was in this context that my old employer invited me to become a chapter lead
    • it was a huge step in my career, if you can call having the same job for 15 years a career
    • i had no idea what i was signing up for
    • (well, i had some ideas, but none of them turned out to be true)
    • the reality was rather disappointing
      • not nearly as rewarding or fulfilling
      • what did I create today? waht did I do?
      • i wrote more emails than lines of code
      • meeting after meeting which had nothing to do with programming
      • interruptions so bad that my interruptions would get interrupted
  • i did something i should have done way before, which was talk to other lead developers
    • almost everyone i talked to was dealing with the same issues
    • you’re not alone, you’re not weird, it’s totally fine
  • i came home from that conference determined to get better at being a lead developer, but also to get better at enjoying it
    • and it worked!
    • there’s no four-step programme
  • i had to completely let go of programming
    • step away from chasing that feeling of fast feedback
  • i had a meeting with my manager where we were asking if someone’s job title matched their performance
    • i found two people in my group were performing way better than their job title suggested
    • i focussed all my email-writing powers into writing to my manager
    • two months later, they were promoted and got a significant raise
    • this blew my mind
    • i had a load of influence and power
    • and i had never really bothered to use it
  • this made me reevaluate everything i did
    • interruptions, emails
    • let the developers continue uninterrupted
    • if i sent an email it was for the benefit of the people around me
  • with great power, comes great responsibility
  • i like helping people
    • not because i’m selfless
    • but because it makes me feel important and valuable

building sustainable teams to handle uncertainty (jenny duckett)

  • @jenny_duckett - technical architect, ministry of justice
  • think about a time when you were working on an effective team
    • what made you excited about it?
  • i was on a team
    • it changed several people over the course of my time there
    • lots of people joining the team were completely new to the team
    • support and improvement work over the 10 product areas we owned
    • we had to shift the focus of the team significantly at one point
  • might seem unusual choice for best team ever, given the circumstances
    • an empowered, open team
  • i see lots of teams struggling

what can disrupt your team?

  • change is often positive!
    • we frequently make deliberate changes about how we’re working in our team
  • there’s often uncertainty involved
    • which might be resolved quickly or go on for months and months
  • people can feel unvalued
  • people leave and join
    • often these are positive changes but it can still be hard
    • the dynamics of personalities and opinions changes whenever people change
  • senior leaders move
    • it can be hard for teams to make decisions and handle roadmaps while this is happening
    • it can be hard to revisit decisions that you thought you’d settled
    • it can be hard to make progress
  • wider priorities shift
    • can end up feeling frustrated and disempowered
    • why make effort when it might just change anyway?
  • teams are re-organised out of existence
    • can make people feel undervalued as individuals or for the work the team has done together
  • if you’re putting people through these things more than occasionally, you will have a retention problem
  • even things like desk moves (see Lara Hogan’s blog post about this) can have an impact on people
  • the way changes make people feel is at least as important as the change itself

what can you do to prepare?

we can prepare for change

  • work on yourself first
  • help ensure you can sustain through these disruptive changes and be better able to support your team as a result
  • if you’re senior, consider the relationships that the teams have worked hard to build together
  • don’t do it all yourself
    • you don’t need to review all the PRs
    • let go of the details
    • instead, give people your trust
    • assume positive intent
  • if you’re working with other people to lead your team (such as product and delivery managers), work really hard on your relationship with them
  • one thing i found really useful was a support network for tech leads within GDS
  • set yourself up to lead sustainably

make your team’s work ownable

  • define a single clear goal for your team
  • we had to stop splitting our efforts and focus on one goal
  • the effect was magical
  • we agreed as a team what parts were essential by the deadline
  • a shared understanding of all this meant the whole team was empowered
  • communicate it over and over again
    • it’ll feel like you’re just repeating yourself and you’re boring people
    • but it’s reassuring just to know that the goal hasn’t changed
  • you might not be able to pick one single goal
    • but communication is even more important in that case
  • give your team the background they need for each piece of work
  • add context to your stories
  • use story kickoffs
    • don’t get as far as the PR before explaining why you want the work done
  • run workshops

empower your team to take ownership

  • embrace opportunities for positive change
  • become great at integrating new people
    • every new person has a mentor for the first few weeks - often the last person who joined
  • recognise that everyone is always learning
    • make it clear that you value learning as part of everyday work
  • share understanding of your work in the team
    • write good commit messages - why, not just what
    • comments on tickets and user stories documenting progress
    • when teams make a decision, document the reasons for it
  • if you learn to share context frequently, you will be able to keep work flowing during disruptions
  • when my team started mobbing, i was amazed how quickly the team made progress and how rarely they were blocked on anything
  • we wrote a blog post (written by Emma Beynon )
  • teams were able to use mobbing later on to tackle things which were entirely new to them
  • taking ownership helps people handle change

support and grow individuals

  • use every piece of work to help someone grow
  • start with individual needs
  • delegate effectively
    • be clear and explicit - tell them that this is what’s happening and why
    • set the scope: what’s the problem they’re trying to solve?
    • who else is going to be involved?
    • when should they bring questions to you?
  • teach people to do your job
  • grow the next generation of leaders
    • it’s easier and more rewarding to hire less experienced people and grow them
  • prepare the team for you moving on even before you know it’s going to happen
  • when i came back from a couple of weeks holiday and found my team carried on as normal without me, i had all the feelings
    • sad but proud

show your team where they fit in

  • grow people for a resilient team and organisation
  • encourage your team to show off their work
  • show the team how their work fits into the bigger picture
  • don’t over-insulate your team
    • when something gets through your shield, it’ll be very disruptive
    • if there are rumblings around in your organisation, mention it to your team
    • tell them when you’ve said no to your team taking on other work
  • a wider view helps people adapt when things change

managing up

  • communicate your team’s capacity for change
    • it takes time to become familiar with a new domain
  • i frequently had to argue for not restructuring teams wholesale, but come to some accomodation to take on new organisational priorities
  • people at the top can’t create a culture of empowerment alone, but they can reinforce or destroy it
  • show them why all your team’s work matters
  • good communication about change is vital

what does this look like?

  • grow sustainable teams for a sustainable organisation
  • a team that understands where they fit within the overall strategy can more easily accept changing the focus of that work to support that goal in a different way
  • (i recommend reading the whole transcript of this talk (see link at top), there’s way too much value that i missed)

the hardest scaling challenge of all - yourself (christian mccarrick)

  • @cmccarrick - SVP of platform engineering - GTL
  • start with a story
    • when i was 14, i worked on a farm making peaches
    • long days in the sun, alone, covered in peach fuzz
    • i had a lot of time on my hands
    • i decided there was no way i was doing this for the rest of my life (and got straight As for the first time at school)
    • i realised that there’s only so much that one person can achieve on their own
    • i bought a commodore 64
  • i’ve been having conversations iwth hundreds of people
    • “i don’t have enough time”
    • “i am so stressed out”
  • what does it mean to scale?
    • my manager quit on a friday; on monday - boom - i was running a team
  • i manage over 200 people in 10 offices scattered throughout the world
  • there are times when i lose it
    • i get emotionally paralysed and can’t make any progress
  • learning to scale yourself is not a one-time task; it’s a lifetime’s effort
    • you should expect your job to change every 6-9 months

1. commmunication

  • you have to be good at communicating 1 on 1
  • as you grow, you need to get better at geting your point across to larger and larger groups
  • become expert at managing up to your managers; across to other teams; and down to your reports
  • the “manager readme” - weekly digest
  • personal branding and self-promotion
    • in order for your good performance to have any effect on your career, someone needs to notice your good performance
    • we value logic, efficiency, getting the work done
      • to us, that’s all that should matter
      • but often it’s not
  • what can i do?
    • take initiatives on high-visibility projects
    • genuinely praise other people
    • have confidence - it’s a positive feedback loop which attracts people to you
  • don’t let your weaknesses blind you
    • get honest feedback
    • you need a safe environment
    • get people to expect consequence-free feedback
  • most importantly: learn to say “no”
    • without feeling guilty!
    • respectful, but completely unapologetic

2. prioritization and time management

  • people are bad at time estimation and time management
  • how much time do we have?
    • 168 hours a week
  • if you don’t prioritize your life, someone else will do it for you
  • write down all the things you need to do (don’t prioritize!)
  • but there are things you don’t think about - dark matter tasks
    • people who say “can you help me with this?”
  • keep a journal - track all these quick favours
  • prioritization:
    • “which one of these is the most important?”
      • “they all have to get done” is not an answer
    • eisenhower method
      • 4 box matrix: urgent/not urgent vs important/not important
      • urgent/important: do
      • not urgent/important: plan
      • not important/urgent: delegate
      • not urgent/not important: eliminate
  • multitasking is a scaling anti-pattern
  • focus
    • what’s it like being completely focused?
      • relaxed
      • confident
      • lose perception of time
    • how do you recreate that moment more often?
      • get the most done
    • calendar blocking
  • meetings OMG
    • makers vs managers schedule
    • delete every recurring meeting you have
    • make your calendar sacrosanct
      • reject double bookings
  • email and slack
    • they’re so addictive!
    • “i wish i spent more time on email” - said noone ever
    • turn off notifications
    • schedule short blocks during the day when you quickly check emails
    • schedule longer blocks at the end of the day where you tidy things up
    • your calendar can do your slack statuses for you
    • put response times in your manager readme
      • so people have expectations on response times
  • obsess with the things that matter!

3. delegation

  • move from “how can i get this done?” to “how can this task/decision/goal get done?”
  • not delegating properly is an anti-pattern
  • let things go
  • delegate decisions!
    • not doing this limits the scaling of your entire team by becoming a bottleneck around decision-making
  • leading takes energy
  • leading gets lonely
  • if athletes get injured, they sit on the bench
    • mental fatigue, burnout, is an issue that affects us more than we might think
  • make yourself a priority once in a while. it’s not selfish, it’s necessary
  • mental health benefits of exercise

introduction to reasonml (jack lewin)

  • @jlewin_ - software developer - sky
  • “people who came to the react.js community early are now mostly talking about Reason”

origins

  • new syntax for ocaml
  • functional programming language
    • 1990s heritage
    • used by financial institutions
  • interop with JS

types

  • “why can’t i use typescript or flow?”
  • with reason, types are baked into the language which gives benefits you can’t get otherwise
    • 100% coverage - everything has a type
    • guaranteed to be sound
    • no runtime type errors
    • pattern matching

JS interop (bucklescript)

  • compile reason to (readable!) JS
  • access JS in reason
  • compile to native
  • reasonml-community on github
  • create-react-app with reason-scripts
  • reprocessing

CI/CD for humans: empathy as the foundation for effective deployments (jackie balzer)

  • @jackiebackwards - head of FE development for behance at adobe
  • I’m all-in on frontend development
  • I’ve never worked in devops
  • i did help with the development of the web UI of our CD platform
  • what makes a good deploy?
    • as long as it makes it to production and doesn’t set off any alerts, it’s a success
    • what about empathy? what about being compassionate and minimising suffering?
    • a good CI/CD pipeline is the empathetic choice for delivering code to production
  • putting code into production is scary
    • it can be the domain of a specialist person with specific skills
    • the first time i wrote a capistrano script to deploy my personal website, i was so baffled by what i’d written, i was afraid to update my site for over a year afterwards
    • a good CI/CD platform does a lot to address the risks & fears around deploying to production
  • CI and CD are the practices that enable organizations to frequently and reliably release new features and products
  • lots of benefits in terms of reducing risk
  • it’s also opinionated which may have impact on how teams do work and require team discussions
  • episode I: the FTP menace
    • when i did this in the beginning:
    • svn for version control
      • branches were hard, so we just wrote to master all the time
    • we thought we were “agile” by not being nitpicky about writing tests
    • no process for task management
    • every 2 weeks or so, our one devops engineer would cut a release from our master branch, put it into staging where it would sit for a day or two, then onwards to production
    • we were deploying huge, risky changesets all at once
    • we would attempt to deploy at the lowest-traffic times to compensate
      • we could only deploy after hours
      • don’t do this!
  • episode II: attack of the hotfixes!
    • our ops staff would spent days at a time being our menial deploy sherpas
    • we had no way to do a rollback. if we pushed a bug, we just had to resolve it and release another hotfix
  • episode III: revenge of the GUIs
    • at some point we switched to git and github
      • branching, code reviews, automated tests
    • jenkins & hubot to allow engineers to deploy changesets to different staging environments
  • build nights make deploys more stressful by making things seem higher-stakes than they really are
  • it takes someone with a specific lifestyle to be able to hang around the office at all hours
  • biggest points of failure
    • how the process affected us and our users
  • episode IV: a new hope
    • some of our engineers were sick and tired of our build processes
    • wrote down how deploys should work
    • started to build a new platform
    • solved the probelm in 2 weeks
    • a series of best practices
    • ownership of deployments is empowering
    • development safety:
      • fork-and-branch to allow for consequence-free development playground
      • upstream is locked down and only contains approved code
        • reduces clutter - clean history
      • feature firewall
        • new features are developed incrementally and shipped to production behind flags
        • can do this without fear of negatively impacting user experience
        • our designers can test new features in a real-world environment
      • smaller changesets
        • smaller PRs even for new features
          • feature doesn’t need to be fully complete to be deployed to production
        • less risk
      • peer reviews
        • style checks automated
        • PR peer reviews become higher quality
        • rule: be nice
        • be cogniscant of your colleague’s efforts
      • CI at every step
        • starting from PR
        • if anything fails, you are blocked from moving forward
      • staging and production require manual verification
        • automated tests stop being useful at the point where reality begins
        • human verification of human-made changes intended for human consumption
      • evergreen master
        • master is always deployable
      • safer rollbacks
    • easy interface, easy deploy
      • a good UI is pretty invisible
      • jenkins as an anti-example of this :sick-smiley:
      • moonbeam.io
    • it started as a hackathon, but it’s grown to be a first-class citizen
    • we now use moonbeam to deploy moon

Junior.next() (tara ojo)

  • @tara_ojo - software engineer - futurelearn
  • think back to your first day in your career
    • full of enthusiasm
    • building something from scratch
  • fast forward to today
    • think about that journey
    • everything you’ve learned
  • you probably didn’t get to where you are today by yourself
    • you learned from colleagues with loads of experience
    • you probably also learned from colleagues without much experience too
  • the way you work with junior developers on a day-to-day basis can have such a big impact on their careers
  • before futurelearn, i was on a 12-month grad scheme at M&S
    • we knew we needed to learn how to code
    • but we didn’t know what else we had to work on
  • I use the word “junior” but there are other words:
    • intern
    • associate
    • apprentice
    • graduate
    • noob

being a junior

  • you could be doing things to support and uplift junior developers in order to get them to the next stage
    • they need to stretch themselves
    • “i could start teaching other people in my team things” - grad developer who felt ready to move on
    • teaching is a good way for a junior to stretch themselves
  • if your junior developer is picking up tasks that they know they can do, and doing them quickly and successfully, then they may be sticking to their comfort zone
    • you want them to stretch themselves by taking things a bit beyond this
    • but: avoid going too far into the PANIC zone
    • remember that the boundaries between these zones aren’t fixed
  • gather evidence about amazing changes and stretching
    • this will be useful for interviews for roles at a higher level
  • being a junior is not something to be ashamed of. it’s not a burden

being a supporter

  • invest time
    • teach them
    • pass on your knowledge
  • good pairing
    • when i was quite new
    • i was struggling to follow along
    • i fell asleep at one point
    • if that person had given me the keyboard, i would have been much more engaged
    • ask specific open questions
      • “if i run the tests against this code, do you think it’s going to pass?”
  • create opportunities
    • living objectives
      • set goals and measure confidence levels in a bunch of different areas
      • check your confidence as you go through
      • it’s a good way to measure your progress
  • regular feedback
    • “i feel like i’m not mid-level, i need confirmation from people around me”
    • giving specific and actionable feedback
    • try this structure:
      • do more
      • do less
      • start doing
      • stop doing
      • keep doing

Points Don’t Mean Prizes (adrian howard)

  • @adrianh Generalising Specialist - quiet stars
  • making a product is easy
    • someone comes up with a “north star” - the problem that needs solving
    • someone else comes up with a journey of how we get there
    • someone splits it up into cards
    • we divide it up into cards and give them to developers
    • we do them all
  • this is a lie
  • stories come in different sizes
    • some are big, small
    • causes problem with flow - blocked pipelines
  • we start doing estimates
    • story points
    • if it’s too big, we cut it up
    • gets us back to our nice list of small tasks
  • this is also a lie
  • our customers don’t care about our stories
    • when we take a story and cut it up, we often discover the actual value is somewhere other than what we’ve built
    • something’s gotten lost between the north start and the individual stories
  • instead of estimating story points first, try these three value-based questions:
    1. can we bin this story (or at least shelve it?)
      • if it’s about building subscriptions, but the first 3 months are free, do we need to build it now?
    2. can we thin it?
      • can we deliver less and still deliver most of the value?
    3. can we split it?
      • into more than one story, which still provide value to the customer?
      • not: database tables
  • we still get smaller stories!
    • binning: you can’t get smaller than nothing at all
    • thinning: we’re doing less, making stories smaller
    • split: those stories are smaller
  • ask the questions again about the results. and again
  • but split based on value, not effort
  • book: user story mapping by Jeff Patton
  • you have your customer journey, and you map the stories that we need to minimally support that journey
    • that’s release one
  • each slice is a release that provides a version of the customer journey
  • this is an ongoing activity
  • if you bin a story, that tends to push it down to the next release
  • if you thin a story, it tends to pull it up to the current release
  • if you split a story, it spreads it across releases
  • (this is still a lie)
  • many organisations don’t let people at the development team level make those decisions about customer value
  • things to help people know the goals:
    • OKRs
    • options
    • assumptions
    • hypotheses
    • experiments
  • do less together more often to do more

Who Destroyed Three Mile Island? (Nickolas Means)

  • @nmeans - VP Engineering - Muve Health
  • how does a nuclear reactor generate power?
    • when i was a kid i was given a four-volume set of books called “how things work”
    • my dad often patiently explained to me how things work
    • I distinctly remember turning to these books when a new nuclear reactor came online
    • I turned to the page on reactors
    • the basic mechanics of a nuclear power plant are basically the same as a conventional plant
      • a heat source to heat water
      • high pressure water turns to steam
      • steam expands which pushes a turbine
      • which turns a generator
        • which is where the electricity actually comes from
      • the steam condenses and goes back round the system
    • we’re looking at pressurised water reactors because that’s how Three Mile Island works
      • primary loop: water heated and held at high pressure
      • secondary loop: water turns to heat
      • the two loops don’t mix
      • the primary loop water doesn’t boil
      • or at least…shouldn’t
  • three mile island
    • about 10 miles south of the capital of Pennsylvania
    • unit 2 went into operation 1978
    • runs at 90% capacity for 3 months
    • “hot, straight and normal”
  • 4 men
    • shift supervisor
    • shift foreman
    • 2 operators
  • everything was pretty normal
    • except for a small problem in one of the condensate polishers that the previous shift couldn’t solve
    • 10 hours earlier the swing shift started working on unclogging a clog
    • they tried backwashing but it didn’t work
    • the operators early on developed a backup system where they took air from the pressurised system and used that to break up the resin beads
    • suddenly went quiet (which it shouldn’t with hundreds of tons of water)
    • there was a leaking check valve
    • all eight tanks close simultaneously completely blocking the flow of water
  • there is no water to be pumped through the secondary cooling room
    • the main feed water pumps trip offline
    • the turbine and generator trip offline 2 seconds later
    • the palnt’s main safety opens
      • the secondary water is not radioactive at all
      • lets off loads of steam into the night sky
    • the turbine and generator alarms go off
    • the primary water pressure spikes
      • this is by design - without secondary to remove heat, it heats up and so pressure goes up
    • automatic systems kick in here:
    • pressuriser:
      • regulates system pressure
      • measures water level
      • if there’s water in the pressuriser, there’s water in the rest of the system
      • maintains a bubble of steam at the top which absorbs pressure shocks
    • but the pressure is continuing to climb
    • the next thing that happened (and most famously)
      • pilot operated relief valve opens
      • 4 seconds after the turbine and generator trip offline
      • the computer detects that pressure is still climbing
      • so it scrams the reactor
      • neutrons -> uranium fission
      • we control this reaction through cadmium control rods which absorb neutrons gleefully
        • typically raised and lowered to control reaction
      • scram: drop them to the bottom
        • stops the nuclear chain reaction almost immediately
        • but it doesn’t stop heat production quite as quickly
        • secondary fission products are still present - 6.5% heat production
        • in the hours after this, it’s still important to carry heat away from the core
      • control room shows the pilot relief valve has been signalled to close
      • operators are happy - they have procedures for this
        • confident that the system is behaving as it should
      • but: 2 minutes later
        • emergency core cooling system kicks in
        • high pressure values dump water into core
      • they didn’t understand why the system thought it needed more water (because the pressure was still rising)
      • 2.5 minutes later, they switch the high pressure valves off
        • if they hadn’t, there would have been no accident
      • bill zewe wonders about something:
        • the pressure is now dropping but the water level is rising
        • he suspects the pilot relief is stuck open
        • checked the outlet temperature
        • if he had decided to close the block valve, there would have been no accident
      • 6 minutes later:
        • alarm about sump:
          • pilot relief valve has leaked enough to set off this alarm
          • clear signal that there’s a significant leak
          • but they miss it
        • the water in the reactor pool is boiling
        • causes vibrations
        • they have to shut off the primary coolant loop pumps
        • this helps for a little while
        • 30 minute later, the vibration is back
        • it’s now 5:44 in the morning, and a nuclear reactor that was at 90% 2 hours ago has no coolant going round
      • a radiation alarm goes off
        • one of the fuel rods has ruptured
          • uranium is normally encased in zirconium
        • almost certainly, the water level has droped below the top of the core
    • the leadership has come to the plant
      • “they closed the block valve, right?”
      • they finally shut it
      • this would have been the right thing to do 20 minutes into the accident
      • doing it now removed the only source of cooling
      • heat intensified rapidly
      • 8 minutes later, top of core collapsed
    • at 7.20 the radiation alarm at the top of the building goes off
      • maximum yearly allowance in 20 seconds
    • they knew the core had started to melt down
    • they turned on the high pressure injection
    • but turn it off 18 minutes later because they’re concerned about the pressuriser
    • over the next few days they continued to be concerned about radiation leak
      • but the containment did its job
    • there was public concern about a hydrogen explosion
    • on sunday april 1st president jimmy carter visited
    • about 20 tons of melted uranium ended up at the bottom of the reactor vessel
    • a further 10 tons in the middle
    • final cost of cleanup: $1bn
      • still not finished
    • unit 1 is still producing electricity
    • unit 2 won’t be fully cleaned up until unit 1 is shut down
      • currently scheduled 2034
    • sidney dekker: the field guid to understanding “human error”
      • “first stories” and “second stories”
      • the story i told is the first story
        • focuses on humans in story and what they could/should have done differently
        • problem with this: bias
          • hindsight bias
            • if you know the outcome, you exaggerate your own ability to predict and prevent the outcome
            • “I think if i saw water pooling at the bottom of the reactor building, i would have noticed there was a leak”
          • outcome bias
            • given the outcome, you judge decisions based on that
            • eg: turning off the high pressure injectors was obviously stupid
      • how do we get to the second story?
        • work from the participant’s reality
        • assume positive intent
        • everyone involved made the best decisions they could with the knowledge they had
  • why did Fred turn off the emergency cooling system?
    • pressurizer
      • he was afraid the water in the pressurizer was going to go solid
      • if you allow the pressurizer to fill with water (no steam bubble) there’s no shock absorbing effect any more
    • this is a smaller problem than the core melting down
    • why is fred so concerned about the pressuriser?
    • they were all naval reactor veterans
      • prime job in a submarine operator was to prevent the pressuriser going solid
      • this makes sense for a submarine reactor: 12 MWth
      • not three mile island: 2.8 GWth
    • heat at shutdown (6.5%):
      • submarine: 0.78 MWth
      • three mile: 185 MWth
    • in a submarine, a water hammer (unabsorbed pressure shock) was the worst-case scenario
  • so Fred Schiemann, with a rising pressure, disabled the cooling to keep the system safe.
  • why did bill zewe not close the block valve when the temperature was over 200℉?
  • the pilot-relief valve was leaking from the day three mile opened
    • if they shut it down every time the temperature went over 200, they’d not generate any electricity
    • they were going to fix it for next refuelling shutdown
  • he also figured the pilot-relief valve was closed
    • but the control panel only said what the computer had told it to do, not the actual position
  • why did the crew not respond when the sump alarm went off?
    • they never got the alarm!
    • giant panel of lights - each one resembles an alarm
    • on a good day, 40 or 50 lights are lit up
    • there’s no rhyme or reason to their placement
    • the light for reactor vessel pressure is next to the one for a stuck elevator
    • no timeline for lights - when did alarm go off?
    • there is a timeline on the printer
      • but it’s on a 300 baud connection
      • there’s so many alarms that the printer is 2.5 hours behind
  • Dr Dekker: human error is never the cause
    • it’s always caused by underlying systems
    • blaming a human prevents us from examining this
    • ask what is responsible, not who
  • understand why what people did made sense
    • because if it made sense to them then, it’ll make sense to someone else later
  • seek forward accountability, not backward
    • blameless postmortems
      • free people up to candidly share what went wrong
      • so you can really get to the bottom of it
    • the act of telling the story is all the accountability a well-meaning person needs to learn their lesson
    • they’ve already beat themselves up, they don’t need your help with it
  • there’s always a second story - look for it
  • this talk should really be titled what destroyed three mile island?

go slow to go fast: building strong foundations for leadership (alicia liu)

  • (i missed the start of this)
  • the higher up you go, the less people will tell you when you get things wrong
  • a lot of engineers are promoted into management because they’re good engineers
    • some people are naturally good at both
    • but if you’re good at engineering, it might actually be harder to develop management skills
  • people ask me: do you miss coding?
    • no!
    • coding is not about the code. it’s about the achievements
    • my best way of achieving things isn’t coding

legacy code: big rewrite or progressive rejuvenation (uberto barnini)

  • @ramtop - functional programming specialist - standard chartered bank
  • legacy systems!
  • i used to work at banks and big institutions
    • legacy systems are old systems that probably make a lot of money for the company
    • but they’re big and complicated and nobody really knows how they work
  • a friend of mine is working on them, and a legacy system is just last year’s system!
    • they’re moving quickly in order to go to market
    • the system became a liability quickly!
  • what is a legacy system?
    • an old running system
    • critical for busines
    • hard & expensive to:
      • maintain
      • improve
      • expand
  • how do you recognise one?
    • it was a very good design
      • possible too good!
      • people start adding extra features and use cases
      • it becomes overloaded
    • continuous state of emergency
      • i was in a bank, where there were different levels of emergency
        • level 4 wasn’t too important
        • level 1 would involve newspaper headlines
      • there was a project which was consistently at a level 2 emergency
      • I couldn’t really imagine to work on those teams because there was so much pressure
      • but it becomes kind of normal, and nobody really cares
    • we’d like to retire it but… we don’t know who is using it!
      • there was a system that needed to be retired, and they checked everyone using it
      • but: there was a system which had an annual report, and it used the old system, and the report was a legal requirement
  • technical debt default
    • when it reaches some point that is so high that you could never ever repay it, that’s really a problem
    • what do we do?

option #1: “there’s no time, just fix it quickly!”

  • further degradation: broken windows effect
    • if you have an old building, and somebody breaks a window, and nobody fixes it, the rest of the windows get broken quite quickly
    • there’s a sense that it’s abandoned and nobody cares

option #2: forget the old system! rewrite from scratch!

  • problem: the new rewrite left unfinished
    • i worked on a really old system
    • (initial commit was older than the grad on the team!)
    • they had tried to replace it three separate times
  • hidden costs of rewrite
    • legacy system complexity is always underestimated
      • because ofsuccess bias
    • all previous knowledge in the code is thrown away and slowly rediscovered
    • new features can delay the release
    • old system is neglected but remains critical
    • data migration is always more painful than expected
  • it’s not that rewrites never work, but they’re very risky

option #3: kintsugi

  • https://en.wikipedia.org/wiki/Kintsugi
    • fixing things without hiding the fix
    • using lacquer/gold leaf to join broken ceramics back together
  • similar to option #2 but it actually works
  • progressive rewrite
  • for example: you have a C# app, but you want to migrate to scala+js
    • keep the C# app but add a Newport view onto the new interface
    • replace one feature at a time
    • using the old app as a skeleton
  • a progressive rewrite could take longer than what is estimated for a full rewrite
    • but probably much less than the actual time needed

code quality

  • we got it completely wrong as an industry what quality is
    • clean code?
    • design patterns?
    • TDD?
    • functional programming?
    • these are tools & techniques. they’re not quality
    • you can’t say “this code is good quality because it has 95% test coverage”
  • the only thing that really matters is the time it takes to implement a feature
    • if new features are easy to implement (or bugs are easy to fix), the code is good quality

architecture

build a better hiring process with design thinking (crystal yan)

  • @crystalcy - product & UX lead - independent
  • i hire designers and product managers
  • have you ever felt 😕 while interviewing?
    • .o(“why is this person asking me this question?”)
    • not knowing what the intention behind a question is
  • or like 😱 when interviewing someone?
    • not having the tools to run the interview well
    • or give a good impression and make them want to join the team
  • Hiring is
    1. frustrating
    2. stressful
    3. not fun
    4. all of the above
  • hiring is also expensive
    • if you want to get buy in to improve your hiring processes, talk about time and (opportunity) costs saved
  • let’s build a more intentional, inclusive hiring process, using human-centred design
    • less bias, more diversity
    • less frustration, more delight, higher yield
      • how often does the candidate accept an offer we extend?
  • our goals
    • cut bias (and improve diversity)
    • set expectations (and increase delight)
    • get to yes (and improve yield rate)
  • our methods
    • user interviews, journey mapping, paper prototyping
    • competitive research
      • sat in on existing sessions run by colleagues
      • other dept heads had fairly long presentations
        • that seemed exhausting to the new person
      • i thought: maybe let’s make this more interactive
      • many of these people had not been designers before
      • and main exposure had been to graphic or interior design, not UX
    • new onboarding session
      • i led interactive activities
  • design challenge: how might we improve the interview experience?
  • conduct research
    • user interviews
    • goal: uncover candidate pain points
    • activity:
      • have new hires pair up (in different roles)
      • ask open-ended questions
      • time-box activity (5-7 minutes)
    • it’s very easy to ask a question that is accidentally misleading
      • don’t ask “what did you like?” but “what did you think? how did that make you feel?”
  • map the experience
    • journey maps clarify pain points in the candidate experience
    • example table to amp things out:
  • paper prototyping & testing
    • this can be intimidating to people not used to it
    • “I work in HR, I’m not a designer”
    • prototyping isn’t just sketching an interface
      • could be: sketching an email
      • writing a phone script
  • ship & iterate
    • focus on quick wins
    • this is how you make a success of it
    • make sure that someone from your hiring team is part of the process
    • lead the way by offering to implement changes for roles you’re hiring for first
  • “several different people were constantly emailing me to schedule next steps… it was hard to keep their names and positions stragiht. and if i had a question, who do i go to?”
  • clarify roles
    • guides help the candidate navigate the interviewing process
    • hiring managers lead the decision-making process
    • “talent” (recruitment team?) helps candidate navigate candidate journey
    • dept director leads service design
  • “it was frustrating that different people kept asking me the same questions over and over again”
  • interviewers often didn’t have the resources to interview candidates fairly and effectively
    • interviewer pulled in at the last minute
      • they don’t know where they fit in to the candidate experience
    • we created an interviewer guide
      • what stage are you performing?
      • who has the candidate spoken to before?
      • what are you expected to assess them on?
      • note-taking template
        • help to detail exactly what was said and what candidate answered
        • these notes would be read by someone else
    • people might interpret the way a candidate gives an answer differently
      • based on personality or bias
    • eg: candidate takes a while to get to an answer after rambling for a bit
      • different interviewers interpret this differently
        • the candidate’s a little nervous
        • they’re not a good communicator
        • they’re not good at listening - they didn’t listen to the question
      • your interpretation depends on your own personality
    • we structured it so that interviewer was responsible for writing what was said, and another person for assessing
  • review
    • hiring is expensive
    • an intentional, inclusing hiring process is better
      • less bias, more diversity
      • less frustration, more delight, higher yield
    • quick wins: clarify roles, create resources
    • user interviews to uncover candidate pain points
    • journey maps
    • paper prototypes
applicationphone screenon-site interview
people
touchpoints
emotions

functional programming for everyone (marek rogala)

  • @marekrog - CTO - appsilon data science
  • imagine a world where:
    • code is perfectly understandable
    • concurrenvcy is a piece of cake
    • testing is a breeze
  • this world maybe doesn’t exist, but i think we can get close with functional programming
  • learning FP is a great way to grow as an engineer

teaching new tricks - how to enhance the skills of experienced developers (clare sudbery)

  • @claresudbery - Lead Consultant Developer - Thoughtworks
  • I am a technical lead with thoughtworks
    • software development consultancy
  • one of the things i do is introduce new practices to people
    • usually XP - pairing, TDD
    • also: new languages, new ways of doing things, new tools, new skills
  • I’m not perfect
    • i make mistakes
    • as a teacher, it’s part of my job to help pupils believe that they can succeed
  • let’s stop making other people feel stupid
  • let’s stop intellectual elitism
    • the idea that experienced practitioners are supposed to be “clever”
    • you feel you’re supposed to prove to everybody else how clever you are
    • you judge people if you don’t think they are clever enough
    • you end up with this pathology where elite people talk to each other in jargon and nobody understands each other
  • let’s talk about empathy
    • teaching should always be inclusive
  • strategies:

pairing

  • when i first heard about this, i thought it sounded like hell on earth
  • why would you want to spend your whole day with someone else
  • when i came back into the industry i was insecure, i felt my skills were out of date, i deliberately came back at entry level
  • when i was pairing, i learned so much more quickly
  • I’m also eternally grateful to all the people who said to me that they were confused

asking questions

  • is importantb
    • but the way you do it is important
    • ask simple questions
    • speak clearly
    • make it clear what you do and don’t understand

the workshop

  • i led a team of (non-thoughtworks) developers
  • my job was to upskill them
  • i have run a lot of workshops before
  • don’t feel like you’re giving a professional workshop that people would spend thousands of pounds for
  • it can be cobbled together
  • example: people were confused about async/await in C#
    • i had learned about it previously
    • i spent an hour brushing up on it, reminding myself
    • thinking about what the most useful things to learn would be
    • and where it would be most relevant in our codebase
    • devised an example based on our codebase
    • divided it into work and gave them half an hour to work on it
    • come back, present, split up, work another half hour
    • that’s a good model for a workshop
  • you can iterate on it
  • it can be a complete shambles at the start
  • if you don’t start with “I’m the expert, you’re the learners” then everyone can feel their way through
  • have a retro at the end and work out what to improve together
  • also, you can google for someone else’s workshop and just run that

group learning - separate from a workshop

  • workshop is more structured
  • i’ll look for a tutorial, give it a quick look to ensure it’s not completely bonkers
  • and go through it together - one screen, one keyboard, rotation timer (you need a big screen for mobbing though!)
  • or you can split into pairs
  • then do a quick post-it retro at the end

don’t overload

  • again, empathy
  • pay attention - if you’re going too fast, slow down
  • don’t give yourself a learning deadline
    • don’t say “by end of day, we’ll know A,B,C,D,E”
    • because it might just be too much
    • and different people learn at different paces
      • you can get the faster ones to teach the slower ones
      • the best way to consolidate your learning is to explain it to somebody else

lunch & learn

  • make learning part of the culture and show that it is celebrated
  • regular sessions where you tell each other stuff
  • anything! doesn’t have to be technical
    • magic tricks, rubik’s cubes, etc

asking questions (again)

  • ask them all the time!
  • i want to know the answers!
  • i want my team to know the answers!
  • i want to show that no question is too stupid
  • i don’t like when people use jargon
  • people in junior positions say to me: “i’m glad you asked that question, i wanted to know the answer but i thought it was a stupid question”

mobbing

  • I had the same first impressions as with pairing:
    • pairing is brilliant, but mobbing seems ridiculous
    • 6 people all on one computer?!
    • but: you’ve got all the knowledge on the codebase in the same place at the same time

deep dives

safety & security

  • make an environment where everyone can come to work as themselves

highlight good examples

  • when you find them in your codebase, use them and point them out to people

nobody is perfect and nobody knows everything

  • retros & feedback help the team own the work that they do
  • feedback isn’t a way to tell people what they are doing wrong
    • positive feedback matters
    • feedback is a way to scaffold people and support them

tips for managing a widely distributed team (dirkjan bussink)

  • @dbussink - principal engineer - github
  • i work for github
    • we’re pretty well-known sticker company
    • i was a senior engineering manager until recently
    • my team has grown from 10 to 30 people
      • i was doing people management, also managing some of the managers, but also involved with the technical discussions
      • it became untenable as we grew
      • we hired some people
      • i’m now focussed on the technical leadership part

why distributed?

  • i never realised the way i’d be working today when i graduated
    • internet, video conferencing
    • i never realised that would be viable
  • if you as a company want to work with amazing talent you’re not going to find them all in one place any more
  • not everyone wants to move
  • do you really want to pay to have everyone in san francisco
  • not everyone wants to work in a distributed team, some people prefer working in an office

context

  • our team:
    • all things i talk about today are things that work for us
    • but may not translate directly to your situation
  • today we’re 28 people, 5 different continents (or 7)
  • today people work from 11 different countries in the world
    • US coast to coast
    • europe
    • africa
  • timezones: biggest difference is 10-11 hours
  • people say to me this must be very hard
  • I see it as easy mode
    • because of how spread we are, everyone’s in a similar position
    • there’s no advantage to being in any particular location
  • the struggle companies have is that they have a big office where most people are
    • there’s a few people scattered around
  • one simple thing you can do is: when having a meeting, everyone dials in separately
    • you start adopting conventions
    • eg raise your hand before speaking
  • although: sometimes people in the same office might as well not be
    • see slack outage where people joked on twitter that they might have to talk face-to-face
  • my first day:
    • i accidentally took staging down
    • someone across the world in the US east coast jumped in and helped
    • this worked because we put conversations somewhere they were public

be explicit about how you will work

get people together regularly

  • regular cadence
    • get everyone together regularly
    • you need budget to do this
  • use time together well
  • what is the time together for? it’s expensive, but also high bandwidth
  • so set explicit expectations about what you’re trying to get out of it
  • examples:
    • long term strategy
    • team structures - are they working? what should we change?
    • iterate on collaboration processes
      • do we want to hire more people? what’s our strategy here?
  • no day-to-day development
  • do non-work activities
    • when people are colocated, they might do non-work things together (karaoke? dinner?)
    • try to capture this
    • grab dinner together
    • we did some things to teach coding to kids via https://code.org/
      • we really enjoyed it
      • we got a lot out of it as teachers
    • lightning talks about non-work topics
  • tell people that it will be tiring

communication after the get-together

  • keep talking
  • build these relationships in regular 1:1s
  • non work video chats
    • world cup
    • which country is screwing up in local politics today
    • coffee machine chat / water cooler
  • i was trying to sign up for a website that had an on-paste disable form handler and i had to hack it to make it work with my password manager and i wanted to complain
    • just complain in chat
    • other people share feelings together
    • another eg: wedding photos
  • rubber-ducking with myself in team chat
  • be aware of your timezones though
    • don’t always make your meetings a US-friendly time

asynchronous communication

  • it’s hard to ad hoc jump into a meeting if you have a 9/10 hour time difference
  • talk in public about what you’re doing
  • slack has an interesting metric: public/private message ratio
    • you want more public than private
    • in an office, people can jump in if they think they can help
  • talk in public channels
    • also: github issues & PRs
  • status updates
    • is it useful to have a meeting to do status updates?
    • often people use meetings as a crutch
    • if i ask for people to update their status in a document, it doesn’t happen
    • so i schedule a meeting to force it
  • no instant gratification
    • if you have a question, you won’t get a response for at least a workday
    • even if it’s urgent
  • you need to work on unblocking others
    • if someone’s waiting on you, you need to get back to them
  • don’t block yourself
  • over-communicate
    • any blame-free postmortem you’ll find a place where communication broke down
    • by the time you feel you’re communicating too much, you might just be doing enough
  • use restraint
    • be aware of how and when you communicate
    • as a manager, I’m fine with people sending me messages any time, but i won’t answer it
    • stop working in the evening
      • this is important!
      • especially if you’re europe, working with US people
      • they’re actively working away
  • hardly anything is truly urgent
  • “Durch einfach liegen lassen erledigt”
    • (ed’s translation: “finished by simply leaving it be”)
  • use timezone spread
    • remember heartbleed? shell shock? people started remediation, then handed over to next timezone

how to run awesome tech internships! (beverley newing)

  • @webdevbev - front end developer - oxford computer consultants
  • I haven’t had an intern but I’ve had work experience students, and I’ve been an intern myself 3 times
    • 2015: graduated (english & german literature)
    • codefirst:girls internship
    • zooniverse internship
    • oxford computer consultants internship
    • turning permanent at oxford computer consultants at the end
  • we talk a lot about diversifying our industry
    • this is a really broad area, and women are just one part of that
    • only ~15% of compsci grads are female
    • if we want to do better, we can’t just hire compsci grads
  • when i was at codefirst:girls, i was told there were four options:
    1. bootcamp
      • paid bootcamp like makers academy
        • i couldn’t afford this
    2. graduate schemes
      • often require a STEM degree
    3. apprenticeships
      • most of these requre you don’t have a degree
    4. paid internship
      • this was the only option open for me
  • lots of people i met were in my position: not quite ready for a junior position. but how do they get there?
    • i curated the newsletter for codefirst:girls for 7 months
    • we only had one internship come through to us
    • this newsletter goes to 4000 people!
  • what is an internship?
    • on-the-job training within a team
    • paid, beginner-suitable opportunities
    • open to anybody regardless of background
  • duration
    • I recommend 6 months
    • I did 3 months
      • it takes 1 month to settle in
      • then you’ve only got 2 months left
      • you need to start finding another job
      • applying for jobs but not knowing how to do the tech tests
  • salary
    • at least a living wage
      • you are working. you are adding value
    • Oxford is at least as expensive to live in than London
    • you need more than national minimum wage
    • you shouldn’t be too stressed about money
  • there is a legal meaning to an internship (link at end)
  • structure
  • support
    • this is really important to people who are switching careers
      • i gave up a proper contract to do an internship in a new city
      • it was a hell of a gamble!
    • be clear to your intern about what support you will give them
  • interacting with an intern
    • be nice in interviews!
    • if you stress someone in an interview, all you learn is how they deal with stress and not the other parts of them
    • it’s important to tell them what they can learn
      • they may not have a compsci background
    • include them in team meetings
      • often they come from other career paths
      • they have lots of skills
      • skills you may not have!
      • eg: during my degree I did some samaritans volunteering and ran active listening workshops
  • i run codebar oxford
    • i teach underrepresented people in tech
    • i meet all these people who want to get started and ask me how to do it
    • and I don’t know what to say to them
      • move to london? leave your family behind? do a bootcamp?
      • these are really hard for lots of these people
      • especially mums
  • links

strategies to edit production data (julie qiu)

  • @jqiu25 - senior software engineer - spring
  • spring - a fashion ecommerce marketplace
    • i lead the catalog team
    • i spend a lot of time thinking about things like how data gets ingested into our system
    • i have spent more times than i care to admit sitting at a SQL prompt wondering how to inject data into production
    • we know it’s not good practice but we do it anyway
  • why?
    • internal tools are not available
    • edge cases exist
    • time-sensitive changes are needed
      • often, the SQL prompt is the fastest, easiest solution
  • example:
UPDATE products
SET name='julies-product'
WHERE id=1
  • I’m writing this query and often I’d get someone to check it over my shoulder
    • but: say it’s friday night and everyone else has gone home
    • and someone is telling me we promised it by monday
    • noone available for review
    • “what’s the worst thing that could happen?”
  • I run the query
    • grab some water
    • tell marketing i did it
    • then look at my laptop
    • and realise this is what I actually ran:
UPDATE products
SET name='julies-product'

😱🤯

strategies for safer editing

in order of up-front effort, and benefits received:

develop a review process for manual SQL edits

  • how it works
    • we started storing SQL queries in a google spreadsheet
    • checklist of what needs to be done
      1. add a record to the spreadsheet
        • name, date, reviewer
      2. reviewer approves
      3. run the query
        • inside a transaction!
        • if you make a mistake halfway through, it’s easy to roll back
  • what’s great
    • easy to implement
      • just start a spreadsheet and send an email
    • audit trail
    • promotes right behaviours
  • what’s not so great
    • easy to make mistakes
    • audit trail is at will - room for error
    • difficult to run long and complex logic

What if marketing wanted something more like this?

UPDATE products
SET name='julies-product'
FROM brands
WHERE brands.name='julies-store'
   AND ...

local scripts

  • how it works
    1. write a script
      • i do it in python
      • i add an argument parser so it’s reusable
    2. connect to remote database
    3. run the script
      • i like my scripts to have a --dry-run flag
  • what’s great
    • reusable
    • easy to manipulate outputs
    • access to common code from your repos
      • reuse logic you’ve already written before
  • what’s not so great
    • easy to make mistakes
      • no code review
    • outputs are only available on user’s machine
      • and they go away when you exit terminal
    • network disconnections

How do we go about running really long scripts?

UPDATE products
SET name='julies-product-1'
WHERE id=1

UPDATE products
SET name='julies-product-2'
WHERE id=2

UPDATE products
SET name='julies-product-3'
WHERE id=3
...
  • I used to do this by coming into work really early
  • but even after all day it was only 25% done
  • what if we had a computer to run scripts that was on all the time?

existing server

  • how it works
    1. write a script
    2. get the script onto the server
      • deploy?
      • upload?
      • compile first?
      • scp?
    3. ssh and run inside a session

until one day:

  • “heyyy jenkins is maybe down? returning 504s on browser, can’t ssh in”
  • i suddenly realised that the script i was running on it was using all the CPU and none of the engineers could deploy anything
    • oops
    • fix: reboot the instance

lesson learned.

  • what’s great
    • ability to run long-running scripts
    • reliable network connectivity
    • no new infrastructure
  • what’s not so great
    • scripts can affect resources on your server
      • cpu, memory
    • not user friendly
      • manual copying
      • ssh
      • start session
      • all just to run a SQL query
    • no persistent audit trail
    • only the engineer that started it has access

task runner

  • how it works
    • dedicated jenkins job
    • jenkins job page gives free UI
    • recipe
      1. write script
      2. code review
      3. input arguments to jenkins job and build
  • what’s great
    • persistent audit logs from jenkins itself
      • good for tracing what happened to scripts
    • code review and automated tests
    • UI
      • minimised room for error
    • we’ve come a long way here!
  • what’s not so great
    • hard to manage credentials
      • this was the main driver for us to look into better solutions
      • we found a dev was still just scping scripts to the server
      • it was because his script had >30 creds to manager
    • environments not clearly separated
    • inputs not verified

we have all these problems: configuration management, logging, separating concerns. haven’t we solved these before?

script runner service

  • how it works
    • a new service, architected like the rest of our pipeline applications
    • single UI
    • different script runner service in each environment
    • recipe
      1. write script
      2. code review and run tests
      3. choose environment and run
  • what’s great
    • centralized configuration management
    • separation of environments
    • user friendly interface - by far the friendliest
    • we can do more!
      • parallelize and scale our instances
      • preview results
      • up to you to customize!
raw sqllocal scriptsexisting servertask runnerscript runner
data editing
complex logic
long scripts
testing
code review
audit trail
user interface
separate environment
centralize configs
…and more!

key lessons

  • make something usable
    • don’t just think about safety, think about speed and usability
    • that’s why we’re getting behind the SQL prompt in the first place
  • invest the effort - it’s worth the cost

goal-setting workshops for managers (melinda seckington)

  • @meckington - technical manager - futurelearn
  • “where do you see yourself in 5 years time?”
    • i always used to dread this question
    • i felt bad because i felt i “should” have an answer
      • rather than just making it up on the fly like i always did
    • it felt like there was this expectation that someone already has come up with this grand goal
      • things like SMART or GROW are about reaching those goals which are already defined
    • what if someone has no idea?
  • thing I started doing with people i manage: explicit goal-setting workshops
    • generate as many ideas as possible
      • broad rather than deep
    • then, later, select which are most important
  • format depends on person
    • one catchup or several
    • send questions beforehand
    • leave it up to each person how much prep they do
      • some are happy to get these questions thrown at them
      • others want time to reflect beforehand
    • provide enough time & space to reflect
  • four areas & types of questions i ask:
    • values (intrinsic/ideal)
    • future (extrinsic/ideal)
    • current role (extrinsic/now)
    • current skills (intrinsic/now)
    • in this order, because the answers to the early phases can influence answers to later phases
      • eg if they say “i value doing things efficiently” you can ask “do you get to do things efficiently in your current role?”

values

  • ask questions to get someone to reflect and think about who they are and how they do things:
    • what are the building blocks that make up you?
    • how would you describe yourself in 5 words?
    • what is your purpose? your passion? your motivation?
    • where do you get your energy from?
  • personality tests & quizzes as conversation starters

future

  • what do you want in your future?
  • think about long-term goals
  • ask questions to get people thinking about what their ideal life and career would be:
    • imagine you’re retiring and looking back at your life: what do you want to have achieved?
      • (this is easier to answer than “where do you find yourself in 5 years?”)
    • what does work-life balance look life for you?
    • what does your ideal day look like?
  • extra: come up with anti-goals
    • what don’t you want to happen?
    • I never want to be bored
    • i never want to work overtime

how is your current role?

  • how have last few weeks/months been?
  • ask questions to reflect on current role and company
    • what are the challenges of your role?
    • which of your abilities, interests and/or values are you able to do in your role? which would you like to do more of?
    • what would you miss if you left your role? why?
    • what kinds of people do you most enjoy working with? what work environments

what are your current skills?

  • what do you do better than others? what do others see as your strengths?
  • are there any gaps in your knowledge or experiences?
  • what technical/communication/leadership skills can you develop?

how might this turn out?

  • people might be drawn to one particular area, or two
    • the purpose isn’t to have something in all 4 areas
  • this is just to get people to reflect and to lead them down that rabbit hole
  • if you don’t help people set goals, you end up with the human equivalent of tech debt
    • you wake up one day and don’t really like the person you’ve become
  • “which way you want to go depends on where you want to get to…”

how to survive the single page app-ocalypse (jim newbery)

  • @froots101 - front-end coach & consultant - tinned fruit
    • based in edinburgh
  • I’m a consultant. i aim to help web product teams take their UI engineering to the next level
  • i’ve seen a common pattern
  • controversial premise: single page apps damage businesses
    • yes, this is a clickbait title, no it’s not all SPAs
  • google maps - one of the first if not the first
    • big team, lots of investment
    • application-like experience
    • dynamically change bits of page in-place
    • APIs to get data
    • client-side routing and history
    • static asset bundling
  • in reality, a lot of SPAs are much more mundane
    • they are CRUD - Create Read Update Destroy

challenges to survival

  • easy to make!
    • if you’ve ever picked up react beginner tutorial you can make a todo list quickly
    • but it’s hard to make good
    • extremely hard to make good, simple, fast
  • WARNING anecdotes ahead
  • it can feel like you’re adding a lot of value quickly in the beginning
  • but there’s this mid-stage where it starts to get really hard and feels like a traditional app might have been better
    • you have to push through this to get to the great UX
    • lots of teams never really fulfil that vision
  • as developers we naturally blame the tools
    • react, vue, ember, …
    • hype cycle
    • currently vue is the new shiny to react
    • “this time it’ll be different”
    • in extreme cases:
      • backbone -> angular -> react -> vue
      • this is bad for business
      • reinventing user value again and again
      • I’ve done this myself!
  • how do we break this cycle?
    • frameworks don’t make apps, teams make apps
    • things to think about before big tech changes
      • is there a product vision?
      • are you all aligned on it?
      • does the team have good comms and psychological safety?
      • skills and capabilities?
      • dev practices
      • what the damned thing actually does
  • a rewrite encourages forgetting
  • a rework requires learning
    • it’s harder but in the long term much more valuable
  • reworking a SPA is really quite hard
    • we’ve been quite monolithic about them
    • only recently have we been looking into ways of breaking them up
    • smaller apps make it easier to find and eliminate waste
    • what’s the simplest way to sustainably add and maintain user value?
    • learn how to live together with your product

using agile techniques to build a more inclusive team (kevin goldsmith)

  • @kevingoldsmith - VPE - AstrumU
  • this is not a talk about agile
    • before agile, the way we talked about software came from industrial manufacturing processes
    • agile brought us much more people-focussed techniques
    • i’ve developed a bunch of different strategies and frameworks that are inspired by agile
  • scenario: first 1:1 with someone
    • someone joins your team, or you move team, or whatever
    • if someone’s remote, i like to do first two 1:1s in person
    • personalise a joint working agreement
      • chance for you to talk about how you approach management
      • chance to discuss what the responsibilities of the role are as you both see them

joint working agreement

  • two columns of post-it notes
    • expectations on lead and on report
    • one expectation per post it
    • 10-15 minutes or until you can’t think of any more
  • take turns
    • pick up a post it, discuss what it means to you together
    • does this make sense?
    • do we agree? do we disagree?
    • example: “i expect you to be on time to all team meetings”
      • “but on wednesday i do the school run so i’m going to miss standups”
  • take a photo
  • refer to it in later 1:1s
  • revisit periodically
    • see how these expectations change
    • you both get better at your jobs
    • you both have more expectations on each other

mentoring a future lead

  • how do you go about this?
  • what are the responsibilities of a lead at our company?
  • brainstorm this
    • each person writes down all the responsibilities of a lead (one per post it)
    • spend 10-15 minutes or until you can’t think of any more
    • four columns
      • Pat’s responsibility
      • Pat approves
        • it’s a responsibility of mine that i’m going to assign to you
        • i want you to make a plan around this and i want to say yes or no to it
      • Ashley informs
        • i’m going to give these responsibilities to you, you just do them
        • but i want you to tell me when you’ve done it
      • Ashley’s responsibility
        • this is now just your responsibility and i don’t need to hear any more from you
    • take each post it
      • what does it mean?
      • are you ready to take this on?
      • eg hiring
  • capture the output
  • refer back to it
  • revisit every 6 months
  • see things move across the board
  • “wait a minute, I’m just giving someone else my job”
    • that gives you a chance to expand your skills
  • thanks to esther derby for the format
    • i got into senior management because i was so good at seeing the details
    • but my team grew beyond my abilities
    • she showed me this format to help me delegate

scenario: getting someone to talk in 1:1s

  • “is it okay if we skip 1:1 this week? there’s some things that need to be delivered”
  • “everything’s fine, we don’t need to do it”
  • then you force it on week 3
  • and they just sit there and say “everything’s good”
    • you can get stuck here
  • take a walk
    • guarantees the 1:1 will last a while
    • talk about the match, talk about TV
    • work is a shared experience you both have, you’ll slide into work as a subject naturally
    • toys!
      • something else to focus on
      • you’re not just staring at each other
      • i have a whole series of these things on my instagram
  • shut up!
    • wait them out
    • people don’t like silence, so they’ll fill it
    • give them a chance to choose their words and form their thoughts
  • “how’s it going?”
    • “good”
    • i couldn’t handle the silence
    • so i just talked the whole time
  • next week i resolved not to say anything
    • “how’s it going?”
    • “good”
    • we sat there for what seemed like 2 hours
    • it was maybe 45 seconds
    • he said “i’ve been thinking”
    • and he started to talk!
  • i had been robbing him of the opportunity to gather his thoughts

team meetings

  • polling - i want your opinion, but i’m the one making the decision
    • give people the freedom to express their opinion but it’s consequence-free
    • you are accountable, not them
    • people have imposter syndrome
    • if you’re not sure if you “should” be part of the group you’re much more likely to hold back
    • take temperature on something
  • voting - we’re making the decision together and i will support the group’s decision
    • you absolutely must support and defend that position no matter how much you disagree
    • if your team votes to implement layer 7 routing, you need to defend it to your boss
      • not: we voted and the team said to
      • you’re throwing them under the bus
  • be very clear which you are doing
  • tell people before they give their opinion
  • fist of five
    • fist: i hate this
    • 5 fingers: i love it
    • good for polling or seeing how close the team is to consensus
  • roman voting
    • count to three, everyone does thumbs up or down
      • not allowed to be in the middle

scenario: collaborative team meeting agenda

  • i thought agendas were too corporate
  • shared agenda
    • shared document to capture the agenda and the minutes
    • get agenda items at least 12 hours before meeting
    • the morning of the meeting
      • i groom
      • you wanted 15 minutes for this, we don’t have time, i’ll give you 10
    • a facilitator (you or someone else) runs the agenda
    • person who added the item leads the discussion
    • good for team of 7-10 people
  • bigger teams: lean coffee

scenario: more inclusive team meetings

  • shut up!
    • positional authority encourages people to conform to your opinion
  • set expectations
    • be clear on desired outcomes
  • observer role
    • role: watch (not participate)
    • people interrupting each other
    • people repeating ideas without giving credit
    • certain people speaking too much
    • exclusionary language
    • passive or active
      • passive: takes notes
        • at 4.35, pat was talking and ashley interrupted
        • at 4.40, pat said an idea
        • end of meeting: reads back
      • active: when they see it they say it
  • rotate roles
    • note-taker
    • facilitator
    • observer
      • frees you from having to participate
  • publish the agenda
    • lets everyone prepare mentally and emotionally for the meeting
  • remember your role power
    • you set the tone and conduct for your team based on your actions and how you react to others
    • there is the loneliness of leadership
    • one thing I’ve seen with good leaders is they may be good at not interrupting, not stealing ideas
      • but they don’t call other people out
      • we have to say “we don’t do that”
    • it’s hard! you don’t want to not be liked. you don’t want to have a difficult conversation

from the hallway track: prometheus ideas with Chris Evans

  • @evnsio - platform team lead, monzo
  • note that this bit is very tailored to what I found useful and relevant, so it skips over a lot of detail
  • thanos isn’t just a persistent storage engine, it can also be used as a query layer to load balance requests across multiple prometheis (for resilience)
    • deploy thanos as a sidecar to prometheus
    • expose ports for both prometheus direct and thanos
    • (this is easier in kubernetes than in ECS)
  • comcast’s trickster is a query optimization layer which does promql-aware caching: eg it caches datapoints and transforms requests for new data into deltas
  • service discovery for prometheus in k8s
    • k8s service manifest
    • cluster IPs
      • each pod (task) has its own IP - separate from the host IP
      • IPs are ephemeral
    • service: stable DNS name to talk to that thing
    • config files: configmap
  • developers using prometheus locally
    • still a work in progress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment