Evidence based software development

idea.md

I am considering starting up a community-funded project that is aimed at exploring software development methodologies and practices through the lens of evidence-based scientific studies. The purpose of this project would be to design, conduct, and report on real world experiments that attempt to answer questions that are relevant to everyday programmers.

This project would be subscriber-supported, but would follow a completely open publication model.

Subscribers would pay a small fee (maybe $5 to $10/month) and get all of the following benefits:

• A monthly progress report summarizing all interesting activity on the project
• Opportunities to read early drafts of articles
• Opportunities to participate in pilot sessions for studies
• Access to a members-only mailing list to discuss research activities with other supporters and with the project's maintainers.
• Ability to respond to surveys and participate in interviews / discussions to help guide the direction of the research project.

In addition, their funding would support the following benefits for the software development community as a whole:

• To the extent it is possible to do so, all research materials will be released in their final form under open content licenses. This includes data from our experiments, articles about our research, and the source for any programs we build to support our research or conduct our experiments.

• Because the research itself will be community-funded, it will focus on the kinds of things that most software developers care about. Our methods will be rigorous, but the questions we answer will be focused on practical topics, not purely academic curiousities.

• We'll try hard to make our work highly accessible and independently verifiable, so that others can build on top of it or test our ideas for themselves.

While the funding we receive will ultimately determine how much time, resources, and effort we put into this project, we're pretty well equipped to get things started from day 1:

• I have been programming professionally for about a decade now, working actively in Ruby since 2004. I've worked extensively on open source projects in Ruby, written some books and hundreds of articles, and taught dozens of classes on how to get better at programming.

• My wife has been doing scientific research in experimental psychology for almost as long as I have been programming professionally, and while she does not come from a computer science background, she has plenty of experience in data analysis, statistical programming, experiment design, and scientific thinking. For those interested in academic credentials, she has a PhD in experimental psych. and is an Associate Research Scientist at Yale. ;-)

As of right now, this is just an idea, but I'd love to know what you think of it, and whether you'd be willing to become a paying supporter if and when we decide to do this. Please feel free to share your thoughts in the comments below!

What are the "questions that are relevant to everyday programmers."

That's definitely very broadly scoped :-)

I think that the things that affect all programmers are things like testing methodologies (how much, when, what kinds), dealing with defects (and regressions), risks / rewards surrounding dependencies, issues of code complexity and how they relate to maintainability, release management, etc.

I realize that I'm still talking in broad generalities, but the general idea is that I'd want to target the kinds of questions that are somewhat fundamental to software development, particularly the ones that have a lot of FUD and/or hype around them.

A big part of this project would be figuring out where to start, though.

You should talk to Alistair Cockburn.