I’m assuming a lot of people in the audience haven’t studied statistics, but because this is Rubyconf, plenty of you know the principles of test-driven-development (TDD). If you haven’t studied statistics before, it follows the same principle as TDD.
In TDD, you demonstrate that your code is correct in two steps. First, assume your code is wrong. Second, try to disprove that assumption. The first step is when you write the test so that it fails. The second step is to change your application code so that the test passes.
In statistics we do the same thing. We first assume the opposite of what we want to prove. If we want to show that a drug treats a disease, we first assume that this drug has no effect. That’s what the placebo group is for. The placebo group is the “red” portion of “red-green refactoring.” The group that’s treated with the drug is (hopefully) the “green” portion of “red-green” factoring.
A statistical test will never PROVE that t