Note: this content is reposted from my old Google Plus blog, which disappeared when Google took Plus down. It was originally published on 2016-05-16. My views and the way I express them may have evolved in the meantime, and I have likely revisited this investigation in a somewhat more rigorous manner since.
So here I was idly looking at Twitter, when Scott Nickell innocently poked me, regarding one more instance of the old "cost of defects" chestnut:
"I can't tell if this "Systems Sciences Institute at IBM" thing is a new study, or just the same-old."
I was feeling lazy, so I encouraged Scott to apply the usual Leprechaun hunting process: "Here's how you could tell: Google exact phrase for a portion of the article citing it, then note the publication dates of hits." (Try it yourself…)
Scott replied after a few minutes: "Well, at a quick glance, I traced it as far as a blog post from about 2008. That's enough to make me confident it's nothing new."
But somehow I felt we shouldn't stop there. I laid off Twitter for a moment and had a quick look at Google Scholar.
Notice anything? Strangely enough, the "Systems Science Insitute" is only ever cited for one "result": the aforementioned bogus numbers about cost of defects.
My curiosity piqued, I tried looking for any contemporary evidence of the existence of this "Systems Science Insitute" at IBM, and could find none. The IBM web site's search box returns zero hits for that name, for instance.
I was eventually able to track down, in a 2009 obituary for the IBM Systems Journal, some evidence for the existence of something called "Systems Research Institute" at IBM.
Meanwhile, Scott helpfully prodded me into looking at result #5 on the Google Scholar list, which mentions those results as being "summarized in Pressman 1992". I know that book - I've run into it a lot, so I own an ebook copy now: "Software Engineering, a Practitioner's Approach".
Looking it up, Pressman cites IBM as follows: "Implementing Software Inspections. course notes, IBM Systems Sciences Institute, IBM Corporation, 1981"
Wait a minute: course notes?
What's worse, here's how Pressman introduces the data on cost of defects (emphasis mine): "To illustrate the cost impact of early error detection, we consider a series of relative costs that are based on actual cost data collected for large software projects [IBM81]."
Pressman adds in a footnote: "Although these data are more than 20 years old, they remain applicable in a modern context." Apparently many people in 2016 still believe with Pressman that 35 year old data are still relevant to a context that has seen such upheavals as the personal computer and the Internet.
But the thing that sticks with me is "course notes". This is essentially an admission that this so-called data was recalled from memory (and quite possibly poorly recalled, as the Systems Research/Sciences approximation suggests).
So here we have the telephone game again - some IBM instructor gave a course in 1981, Pressman wrote up numbers "based on" the numbers from that course a few years later, everyone else quoted Pressman as gospel and most of them deleted the somewhat inconvenient "course notes". It became "a report from IBM", and appears as such for instance in the book "Agile Testing" by my colleagues Lisa Crispin and Janet Gregory.
Citing "reports" from non-existent "institutes" isn't even the worst offense to common sense committed on a routine basis in my profession - it's just the latest example to make me want to crawl into a hole.
Not for the first time, I get this feeling that everyone in this profession is making it up as they go along, and the entire edifice of "software engineering" (as a supposed academic discipline) is the Emperor's brand new clothes.
Maybe we all need to become little kids again before it can get any better?