Science-based Development

science-based_development.md

Science-based Development

Summary

Development, like science, is a messy endeavor. It’s near impossible to control all the variables. Testing code is a noble goal, but it’s very easy to test the wrong thing. Many experienced scientists are fooled by their senses and biases. To account for that they crafted and refined a Scientific Method. That method has been stress tested by centuries of experimentation, discovery and peer-review. It probably wouldn’t hurt to try to apply it to development and see what it can offer.

Steps

1. Inquire
2. Hypothesize
3. Disprove
4. Reflect
5. Publish
6. Think Different

1 - Inquire

Let your mind loose of the tight corset of implementation. It doesn't mean you have to forget reality. The goal is to solve real problems. Observing the world around should prompt you to wonder: "What if it worked better?".

2 - Hypothesize

Explain in how you could solve a problem. "To make X better we could do Y". A hypothesis is not a test, all you need do is formulate a possible solution to a problem.

3 - Disprove

In Science, you don't prove a hypothesis is correct, you prove it isn't incorrect. Learn how to falsify your hypothesis and you will become a better tester first, and a better developer in the long run.

4 - Reflect

What happened? Did your hypothesis hold up to reality? Or did it blow up in a thick dust of incorrect assumptions? This is a crucial phase where you have to be honest with yourself. You have to be curious enough to poke the very corpse of your beautiful idea. This is rarely a fun thing to do but it can become fun once your realize how much you learn here.

5 - Publish

This is an odd one, it doesn’t really fit our idea of software development. Does it? But what if, once in a while — when one of your assumptions is shot through experimentation — what if you wrote something about that publicly. Allowing yourself to reflect on your process, letting peers get a peak at it and maybe learn from it, and allow you to document your progress.

6 - Think Different

Resilience matters here. At the core, developers — like designers — are people who like to make things. Sadly, the solution of the problem we’re trying to solve is not always the one we imagined. It often takes time, error, and iteration for us to find a truly elegant solution. The child-like ability for untammed imagination is what sets apart the people who will try once or twice and give up, from the people who will relentlessly strive to improve their work.

It’s about craftsmanship, not talent.

Eric had good feedback on experimental setup. It could be interesting in the section on experimentation to teach developers about clinical trial techniques and dangers: bias, double-blinding, controls, contamination, margin of error, statistical significance.

Recent ideas:

• where's the science in computer science? Is it just math? Or is it logic?
• can we objectively tell bad practices apart from good ones? Not just for us but for a majority of developers?
• is there clinical research in development practices? In the psychology of teams, projects and work? Of course there is.
• what is science-based medicine? Isn't all medicine based on science? Not these days.
• what does it mean to be based in science? Can I wear a cape? Sure, but it won't help you. Science is a mindset and a process.
• Process. Mention Rands on Process at Apple.
• science isn't frozen, it's a body of knowledge that morphs it's shape to espouse truth
• what is a consensus? Brian Dunning on SkepticBlog
• what is logical fallacy? DHH vs Corey Haines example & whatisyourlogicalfallacy.com

Inspirations:

http://www.sciencebasedmedicine.org/index.php/about-science-based-medicine/

Great example A/B testing as Science-based development:
http://danwin.com/2013/01/infinite-scroll-fail-etsy/

Make a claim, try to disprove it, adjust accordingly.

Design -> Develop -> Measure

or

Design -> Develop -> Measure small feature and repeat

Dan McKinley from Etsy calls this "Continuous Experimentation" for lack of a better word. I'd like to call this Science-based Development.

Code School example

Like everybody and their grandma, we finally decided to roll out third-party authentication on our site last fall. Since most of our customers are developers, we used GitHub. Letting people sign up with GitHub sounds like a no-brainer:

• everybody hates signing up for a new service
• everybody loves GitHub
• everybody will love signing up to our service with GitHub

Bulletproof?

Except humans always project the most desirable outcomes. More importantly, we're terrible estimators [citation needed], especially the people who think they're good estimators [citation needed].

So when we planned to build this feature at the end of our big effort to redesign Code School last summer, Adam Rensel and I were like: sure this won't take any time.

Well....

• to sign someone up we need an email address
• what happens if their email address already exists on Code School
• what happens if they already had an account on Code School
• what happens if GitHub is down and they can't sign in

Some of these things we can chose to care about or not, but they'll surely affect the experience of our users in some way. All we wanted was to make things easier for them, but it turns out we've also made things harder on ourselves, and as a results of that complexity, we often need to make their experience less simple and less easy.

Is it worth it? After 6 months, 8% of our users use GitHub to sign in to the site. Sounds good to me.

But would we have made that effort at the last minute of a major release had we known it would only impact 8% of our users? Maybe not.

Could we have known it would only help 8% of our users? At first glance, it doesn't seem so:

• it's possible no users at the time cared about using GitHub to sign in but found it useful and switched to that because passwords are hard (use 1Password)
• it's possible no users at the time cared about using GitHub to sign in but we managed to get tons of new people to sign up through GitHub