The IBM Systems Science Institute

ibm-systems-science-institute.md

The IBM Systems Science Institute

Rubric: Software Engineering : Factual Claims : Defect Cost Increase : Pressman Ratios

Context

Background: I have been researching quantity and quality of empirical evidence underlying claims in software engineering. What do we know, and how well-established is that? See in particular https://leanpub.com/leprechauns which concludes that the answer is in (too) many cases "not much, and poor".

This applies in particular to the "Defect Cost Increase" claim, which is poorly supported by evidence. The claim states that the longer a defect stays undiscovered after being introduced in a software system's artefacts, the more expensive it is to correct.

The "Pressman Ratios" are a specific quantitative assessment of this claimed effect:

"Assume that an error uncovered during design will cost 1.0 monetary unit to correct. Relative to this cost, the same error uncovered just before testing commences will cost 6.5 units; during testing, 15 units; and after release, between 60 and 100 units."

We can compress this to "1:6.5:15:60-100", for ease of distinguishing with other variants of the same claim that differ only in the precise ratios.

The reference cited by Pressman is initially formatted as follows:

[IBM81] "Implementing Software Inspections," course notes, IBM Systems Sciences Institute, IBM Corporation, 1981

Evidence for the existence of the Institute

Evidence that the Systems Sciences Institute even existed is scant and hard to come by. A search of the IBM corporate web site yields nothing, for instance.

Using Google Scholar with search terms "IBM Systems Sciences Institute" (Scholar) yields a short list of people who seem to have been affiliated with the organization:

• Claude Wesley Burrill Burrill
• Arnold Oral Allen Allen Allen2
• Janice Richmond Lourie Lourie
• Thomas M. Yokoyama Yokoyama
• Paul R. Melichar Melichar

I am using middle names and middle initials where available to be allow specific Google searches and disambiguations, the usual citation style only has the last name in full, e.g. "A. O. Allen".

What was the IBM Systems Science Institute ?

Quoting Claude W. Burrill:

The Systems Science Institute is an educational organization and is a unit of IBM. We offer a variety of advanced courses for management and for professionals in the computer field.

The Insitute was, in essence, an internal training program for employees at IBM.

Link to the IBM Systems Research Institute

Confusingly, several biographies of the people listed above associate them with the "IBM Systems Research Institute" (for instance, Burrill's obituary). A New York Times article describes it as follows:

(...) virtually all I.B.M. employees receive some kind of company-financed education or training each year beyond basic job training. The education might range from attendance at special lectures to full-fledged courses, inside or outside corporate facilities. Despite its other programs, I.B.M. believes it needs a graduate-level school for its own use. The Systems Research Institute, founded in 1960, is the closest thing to an academic center at I.B.M.. (...) But the Systems Research Institute is not an academic place. ''It's all business,'' said its associate director, Joseph E. Flanagan.

The IBM Systems Research Institute was based in the state of New York, with offices in Manhattan and Thornwood. (Giant, Origins, Thornwood)

Location, history and details

The IBM Systems Science Institute was based in Los Angeles, at 3550 Wilshire Boulevard. (Address).

It started operations as early as 1967 (according to an IBM ad referring to its post-1982 name, see next paragraph). Started

It ceased to operate under that name before 1982, according to a different ad. (Rename) It became the "Information Systems Management Institute".

Incidence on the evaluation of the Defect Cost Increase claim

Another article will deal in more detail with the credibility of the Pressman Ratios, and how they are widely but generally inappropriately cited throughout the literature on software engineering.

The main findings of the present entry on the "IBM Systems Science Institute" are as follows:

• the Institute was a corporate training program, not a research body; as such it is inappropriate to cite the source of the ratios as "an IBM study" or "a study by the IBM Systems Science Institute", in the total absence of any claim that the Institute was the primary source;
• the original project data, if any exist, are not more recent than 1981, and probably older; and could be as old as 1967.

References

Hi! Hope you don't mind me chiming in - I found your note while trying to run down the same exact IBM paper and failing. Found this paper instead, which is addressing the same basic question: is this "defect cost increase" thing completely apocryphal? Short answer: seems to be, and recent studies don't show it.

https://arxiv.org/pdf/1609.04886.pdf

@trdrake Not at all, it's such a relief to know I'm not the only one wondering :) Thanks for the link!

Here's the follow-up: https://gist.github.com/Morendil/6d664bf990c17ea0f88f9bb0cc403c64

There's a real paper from a similar timeframe that does show this effect of exponential growth in cost of correction over time.

That paper is Boehm, Barry W., "Software Engineering", IEEE Transactions on Computers, December 1976, pp. 1226-1241, vol. 25
DOI Bookmark: 10.1109/TC.1976.1674590, https://www.computer.org/csdl/journal/tc/1976/12/01674590/13rRUzp02n5 At the time Barry W. Boehm was at the TRW Systems and Energy Group.

It says "Fig. 3 shows a summary of current experience at IBM[4], GTE[5], and TRW on the relative cost of correcting software errors as a function of the phase in which they are corrected." and indeed figure 3 (page 1228) shows exponential growth. It only shows averages or ranges for each data source, and that's a legitimate critique. That said, it does show them for multiple companies, and then presents a trend line that plausibly follows from the data provided. Boehm has a good reputation, I expect that this really was a reasonable observation from real data.

It's legitimate to question whether or not that is still true. Computer "science" is notorious for having almost no science - experiments are almost non-existent. I would love to see this & many other experiments conducted to see what's true today.

Yep, that's one of the ur-papers that you get to if you chase citations around long enough, the "telephone game" pattern I explore at length in Leprechauns.

But it has its problems - some of the data was obtained from student projects Boehm was supervising, for instance. Another problem is that you still have to check the papers cited by Boehm to follow the "chain of data custody" - Boehm's towering reputation notwithstanding.

To pick just one (I have a more thorough survey in the book) the source of the GTE data is a paper by Daly (paywalled by the usual suspects, but Sci-Hub is your friend), the only relevant portions of which are shown below:

The development cost required to detect an error by reading code is approximately equal to 25 percent of that required to detect the same error via machine debugging. This statistic was gathered from two projects, a 111 000 instructions project and a 12000 instructions project.

As you can see from the figure, the categories of interest are labeled differently. Daly was comparing "reading code" to "machine debugging", not phases. Also importantly, Daly was talking about the cost incurred to find a bug, not the time to fix one. And in the 1970s the primary cost of "machine debugging" would have been the cost of running the computer, which tended to dwarf the labor costs of the human programmers.

Anyway, it's not clear on what basis it was decided that "Eyeball" corresponds to "Requirements", "Prototype system" to "Development test", and "Post cutover" to "Operation" - other than Boehm wanting to shoehorn the data into his model. (And what does "eyeball" even mean in this context ? Not clear.)

Finally, it's somewhat obvious that Boehm made up the error bars around the log-graph data point which stands for "fifteen" - why would he do that, other than to lend an air of statistical legitimacy to what was in fact a wet-finger-in-the-air approximation from Daly? The intellectually honest thing to do here isn't to draw fake error bars, it's just to plot the damn points, quote the small paragraph above from Daly, and warn the reader about the approximate nature of the equivalence being made. Yes, that adds half a page to the paper, but that's how scientific publications are supposed to be, long and painstakingly detailed and rigorous.

But Boehm's paper is not science, it's a marketing brochure, selling a new-fangled concept called "Software Engineering". So it has to be short and convincing, and scientific accuracy is a secondary concern.

Thank you for this analysis and much appreciated source links. I just spent about half an hour trying to track down the original cost ratio chart I am seeing everywhere, finally got to here by searching, in DuckDuckGo (not google) '"IBM System Science Institute Relative Cost of Fixing Defects" original'.

@ecmonsen Very glad this was helpful !