terryjbates/java_shop_politics.txt

## java_shop_politics.txt

1
 Michael O. ChurchRants, essays, and diatribes.APRIL 13, 2012 BY MICHAELOCHURCHJava Shop PoliticsOnce, I was at a company that was considering (and eventually did so) moving its infrastructure over to Java, and there was a discussionabout the danger of “Java Shop Politics”. It would seem strange to any non-programmer that a company’s choice of programminglanguage would alter the political environment– these languages are just tools, right? Well, no. In this case, almost all of us knew exactlywhat was being talked about. Most software engineers have direct experience with Java Shop Politics, and it has a distinct and unpleasantflavor.

When Unix and C were developed, they were designed by people who had already experienced firsthand the evils of Big Software, ormonolithic systems comprised of hundreds of thousands of lines of code without attention paid to modularity, that had often swelled tothe point where no one understood the whole system. (In the 1970s, these monoliths were often written in assembly language, in whichmacroscopically incomprehensible code is not hard to create.) The idea behind the Unix environment, as a reaction to this, was toencourage people to write small programs and build larger systems using simple communication structures like pipes and files. AlthoughC is a strictly compiled language and no Lisp-style REPL existed for it, C programs were intended to be small enough that the Unixoperating system was an acceptable REPL. This was not so far from the functional programming vision, and far more practical in its time.The idea behind both is to write small programs (functional “building blocks”) that are easy to reason about, and build more complexsystems out of them, while retaining the ability to piecewise debug simple components in event of failure. This style of developmentworks extremely well, because it encourages people to build tools for general use, rather than massive projects that, if they fail, render
almost all of the effort put into the project useless. As more code is written, this leads to the growth of generally useful libraries andexecutable utilities. In the big-program model of development, for a contrast, it leads to increasing complexity within one program, whichcan easily make whole-program comprehension impossible, and make decay (in the absence of forward refactoring) inevitable, regardlessof programming language or engineer talent.

I adhere to this small-program mentality. I’m not saying “don’t be ambitious”. Be ambitious, but build large systems while keepingindividual modules small. If a file is too big to read (for full comprehension) in one sitting, break it up. If “a program” is too big to read ina week, then it should be respected (even if it runs as one executable) as a system, and systems are harder to manage than single modules.While it is harder, up front, to develop in a modular style, the quality of the product is substantially higher, and this saves time in the long run.

Many software managers, unfortunately, like Big Projects. They like huge systems with names like cool-sounding names like “Rain Man”and “Dexter” that swell to a hundred thousand lines of code and provide features no one asked for. They like having the name ofsomething Big, something their bosses might have heard of, in their story. Big Projects also provide a lot more in the way of managerialcontrol. A manager can’t control the “chaotic”, often as-needed, growth of small-program development, whereas directing a Big Project isrelatively straightforward. Make it Object-Oriented, use this set of analytic tools, complete at least 20 of the 34 feature-request tickets in thequeue (it doesn’t matter which 20, just do 20) and have it done by Friday.

Beginning around 1990 was a pernicious attempt by software managers to commoditize programming talent. Enter C++, designed withunrelated intent, but a language that made big-program development seem palatable for C. C’s not high-level enough for most applicationdevelopment in 2012, but C++ is not the solution (except in very specific cases from numeric programming where templates allow very-fast code to be written once for a range of numeric types, with no runtime performance overhead) either. For over 90 percent of theapplications that use it, it’s the wrong language. Here’s the metaphor I like to use: assembly coding is infantry: as fine-grained as onecould want to be, but slow (to write), lumbering along at 3.1 miles per hour. C is a tank. It’s robust, it’s powerful, and it’s extremelyimpressive and well-designed, but it moves at ground speeds: about 50 miles per hour. That’s what it’s designed to do. Languages likePython and Ocaml are airplanes– very fast, from a development perspective, but not fine-grained at all. C++ exists because someone had afever dream in which these two classes of vehicles got mixed up and thought, “I’m going to put wings on a fucking tank”. The drag andawkwardness imposed by the wings made it terrible as a tank, but it doesn’t fly well either. Java was invented after a few horrible tank-plane crashes, the realization being that the things are too powerful and fly too fast. It’s a similarly ridiculous “tank-icopter”. It’s not asfast as the tank-plane, and few people enjoy flying them, but it’s less likely to kill people.

It’s not that C++ or Java, as languages, are evil. They’re not. Languages are just tools. Java was designed to run in embedded systems likeautomatic coffee pots and cable-TV “set top boxes”, so closures were cut for time in the first release, because these use cases don’t requirehigh-level programming features. I haven’t map-reduced a toaster cluster for years. Contrary to popular history, Java wasn’t designed
with an ideological, “enterprise” distrust of the programmer. That trend, in the Java community, came later, when the 1990s attempt tocommoditize programming talent (a dismal FailureFactory) co-opted the language.

Java and C++ became languages in which “object-oriented programming” was the norm. The problem is that what is currently called OOPis nothing like Alan Kay’s vision. His inspiration was the cell, which hides (encapsulates) immense mechanical complexity behind asimpler interface of chemical and electrical signals. The idea was that, when one needs complexity, simpler interfaces are invaluable, and thatcomplex systems generally should have comprehensible interfaces. Kay was not saying, “go out there and create giant objects” or “useobject-oriented programming everywhere”. He was attempting to provide tools for dealing with complexity when it becomes inevitable.Unfortunately, a generation of software managers took “object-oriented” magic and immodularity as virtues. This is similar to the“waterfall” software methodology, named by a person clearly stating it was the worst idea, and yet taken by suits as having been“recommended by some smart guy” once it was given a name.

What’s wrong with Big Project development? First, it encourages reliance on internal vaporware. Important work can be delayed “untilMagneto is done”. When Magneto is done, that will solve all our problems. This works for managers seeking to allocate blame for slowprogress– that fucking Magneto team can never get their shit done on time– but it’s a really bad way to structure software. In the small-programmodel, useful software is being continually released, and even an initiative that fails will provide useful tools. In the big-program arena,the project is either delivered in toto or not at all. What if half the Magneto team quits? What if the project fails for other reasons? ThenMagneto est perdu. It’s a huge technical risk.

[ETA: when I wrote this essay, I was unaware that a company named Magneto existed. “Magneto”, in this essay, was a potential name fora large project in a hypothetical software company. There is absolutely no connection between these project names used here and realcompanies. Consider names like “Cindarella” and “Visigoth” to be gensyms.]

Second, under a big-program regime, people are trackable, because most programmers are “on” a single program. This is also somethingthat managerial dinosaurs love, because it provides implicit time-tracking. Mark is on Cindarella, which has a headcount of 3. Sally is onVisigoth, which has a headcount of 5. Alan is on the project that used to be called 4:20 until Corporate said that name wasn’t okay and thatnow can’t decide if it wants to be called 4:21 or BikeShed.

What this kills, however, is extra-hierarchical collaboration– the lifeblood of a decent company, despite managerial objections. In a small-program software environment, people frequently help out other teams and contribute to a number of efforts. A person might be involvedin more than 20 programs, and take ownership of quite a few, in a year. Those programs end up having general company-wide use, andthat creates a lot of cross-hierarchical relationships. MBA dinosaurs, alas, hate cross-hierarchical, unmetered, work. That kind of work isimpossible to measure, and a tightly-connected company makes it hard to fire people. On the other hand, if a programmer only works onone Big Project at a time, and has no other interaction with other teams, it’s much easier to “reduce headcount”.

The third problem with big-program methodology is that it’s inefficient. Six months can be spent “on-boarding” an engineer into thecomplexities of the massive Big Project he’s been assigned to work on. In light of the average job lasting two to three years, that’s justintolerable. The on-boarding problem also restricts internal mobility. If someone is a bad fit for his first Big Project, moving him to anothermeans he could spend up to a year just on-boarding, accomplishing zilch. Transfers become unacceptable, from a business perspective, sopeople who don’t fit well with their first projects are Just Fucked.

Why is the on-boarding problem so severe? Big Projects, like large Java programs in general, tend to turn into shitty, ad-hoc domain-specific languages (DSLs). Greenspun’s Tenth Rule sets in, because programmers tend to compensate for underpowered tools by addingpower, but in hasty ways, to the ones they have. They end up developing a terminology that no one outside of them understands. Theybecome systems where to understand any of it requires understanding all of it. This means that people hired to modify or expand theseBig Projects have to spend months understanding the existing system, whose intricacies are only known to a few people in the company,before they can accomplish anything.

The fourth problem with the big-program methodology is the titular Java Shop Politics. In a small-program development environment,engineers write programs. Plural. Several. An engineer can be judged based on whether he or she writes high-quality software that isuseful to other people in the organization, and this knowledge (talent discovery) is redundant throughout the organization because theengineer is continually writing good code for a wide array of people. What this means is that technology companies can have thelightweight political environment to which they claim to aspire, in which a person’s clout is a product of (visible) contribution.

On the other hand, in a big-program shop, an engineer only works on one Project, and that project is often a full-time effort for manypeople. Most people in the company– especially not managers– have no idea whether an individual engineer is contributingappropriately. If John is chugging away at 5 LoC per day on Lorax, is that because he sucks, because the team failed to on-board him, orbecause Lorax is a badly structured project? In a small-program environment where John could establish himself, such a question could beanswered. The good programmers and bad projects (and vice versa) could be objectively discovered. In a big-program world, none of thatwill ever be known. The person with the most clout on Lorax, usually the technical lead, gets to make that assessment. Naturally, he’sgoing to choose the theory that benefits him, not John. He’ll never admit that Lorax is badly designed or, worse yet, was a mistake in thefirst place. So under the bus John goes.

In general, it is this fog of war that creates “office politics”. When no one knows who the good and bad contributors are, there are majorincentives toward social manipulation. Eventually, these manipulations take more of peoples’ emotional energy than the actual work, andthe quality of the latter declines. This is not limited to technology; it’s the norm in white-collar environments. Small-program developmentis an antidote. Large-program development accelerates (but does not necessarily cause) the poison.

The solution to this is simple: Don’t become a Java Shop. I’m not saying that Java or the Java Virtual Machine (JVM) is evil. Far from it, Ithink Clojure and Scala (which run on the JVM) are excellent languages, and the JVM itself is a great piece of software. Writing anoccasional (small) Java or C++ program won’t destroy a company, obviously. On the other hand, fully becoming a Java Shop (where thevast majority of development is done on large Java programs, and where success relies on understanding defective “design patterns” andflawed “best practices” instead of being able to code) will. There is no avoidance of this; politically speaking, Java Shops go straight down.The quality of engineering, predictably, follows.

The company I discussed earlier, which was one of this country’s most promising startups before this happened, did become a Java Shop,despite furious protest from existing talent. Within weeks, the politics of the organization became toxic. There was an “old team” thatadhered to the Unix philosophy, and a “new team” that was all-Java. (No Scala or Clojure; Scala was flirted with and had seriousmanagerial support at first, but they killed their Scala efforts for being “too close” to the old team.) This old/new cleavage ruptured thecompany, and led to an epic talent bleed– a small company, it lost several engineers in a month. Java Shop Politics had arrived, and therewas no turning back.Here’s what that looks like. First, it’s not that Java (or C++) code is inherently evil. Any decent software engineer will have a passingcompetency in half a dozen languages (or more) by my age, and these languages are worth knowing. Some Clojure and Scala programsrequire classes to be written in Java for performance. That’s fine. Where a company starts to slide is when it becomes clear that “the realcode” is to be written in Java (or, as at Google, C++) and when, around the same time, big-program methodologies become the norm. Thismakes the company “feel” managerially simpler, because headcount and project efforts can be tracked, but it also means that the companyhas lost touch with the ability to assess or direct individual contribution. Worse yet, because big-program methodologies and immodularprojects are usually defective, the usual result is that people with bad tastes thrive. Then the company becomes less like a softwareenterprise and more like a typical corporation, in which all the important decisions are made by the wrong people and decline becomes inevitable.

This entry was posted in Programming and tagged C++, java, java shop, java shop politics, office politics, software engineering, unix.Bookmark the permalink.36 thoughts on “Java Shop Politics”

	1
	Michael O. ChurchRants, essays, and diatribes.APRIL 13, 2012 BY MICHAELOCHURCHJava Shop PoliticsOnce, I was at a company that was considering (and eventually did so) moving its infrastructure over to Java, and there was a discussionabout the danger of “Java Shop Politics”. It would seem strange to any non-programmer that a company’s choice of programminglanguage would alter the political environment– these languages are just tools, right? Well, no. In this case, almost all of us knew exactlywhat was being talked about. Most software engineers have direct experience with Java Shop Politics, and it has a distinct and unpleasantflavor.

	When Unix and C were developed, they were designed by people who had already experienced firsthand the evils of Big Software, ormonolithic systems comprised of hundreds of thousands of lines of code without attention paid to modularity, that had often swelled tothe point where no one understood the whole system. (In the 1970s, these monoliths were often written in assembly language, in whichmacroscopically incomprehensible code is not hard to create.) The idea behind the Unix environment, as a reaction to this, was toencourage people to write small programs and build larger systems using simple communication structures like pipes and files. AlthoughC is a strictly compiled language and no Lisp-style REPL existed for it, C programs were intended to be small enough that the Unixoperating system was an acceptable REPL. This was not so far from the functional programming vision, and far more practical in its time.The idea behind both is to write small programs (functional “building blocks”) that are easy to reason about, and build more complexsystems out of them, while retaining the ability to piecewise debug simple components in event of failure. This style of developmentworks extremely well, because it encourages people to build tools for general use, rather than massive projects that, if they fail, render
	almost all of the effort put into the project useless. As more code is written, this leads to the growth of generally useful libraries andexecutable utilities. In the big-program model of development, for a contrast, it leads to increasing complexity within one program, whichcan easily make whole-program comprehension impossible, and make decay (in the absence of forward refactoring) inevitable, regardlessof programming language or engineer talent.

	I adhere to this small-program mentality. I’m not saying “don’t be ambitious”. Be ambitious, but build large systems while keepingindividual modules small. If a file is too big to read (for full comprehension) in one sitting, break it up. If “a program” is too big to read ina week, then it should be respected (even if it runs as one executable) as a system, and systems are harder to manage than single modules.While it is harder, up front, to develop in a modular style, the quality of the product is substantially higher, and this saves time in the long run.

	Many software managers, unfortunately, like Big Projects. They like huge systems with names like cool-sounding names like “Rain Man”and “Dexter” that swell to a hundred thousand lines of code and provide features no one asked for. They like having the name ofsomething Big, something their bosses might have heard of, in their story. Big Projects also provide a lot more in the way of managerialcontrol. A manager can’t control the “chaotic”, often as-needed, growth of small-program development, whereas directing a Big Project isrelatively straightforward. Make it Object-Oriented, use this set of analytic tools, complete at least 20 of the 34 feature-request tickets in thequeue (it doesn’t matter which 20, just do 20) and have it done by Friday.

	Beginning around 1990 was a pernicious attempt by software managers to commoditize programming talent. Enter C++, designed withunrelated intent, but a language that made big-program development seem palatable for C. C’s not high-level enough for most applicationdevelopment in 2012, but C++ is not the solution (except in very specific cases from numeric programming where templates allow very-fast code to be written once for a range of numeric types, with no runtime performance overhead) either. For over 90 percent of theapplications that use it, it’s the wrong language. Here’s the metaphor I like to use: assembly coding is infantry: as fine-grained as onecould want to be, but slow (to write), lumbering along at 3.1 miles per hour. C is a tank. It’s robust, it’s powerful, and it’s extremelyimpressive and well-designed, but it moves at ground speeds: about 50 miles per hour. That’s what it’s designed to do. Languages likePython and Ocaml are airplanes– very fast, from a development perspective, but not fine-grained at all. C++ exists because someone had afever dream in which these two classes of vehicles got mixed up and thought, “I’m going to put wings on a fucking tank”. The drag andawkwardness imposed by the wings made it terrible as a tank, but it doesn’t fly well either. Java was invented after a few horrible tank-plane crashes, the realization being that the things are too powerful and fly too fast. It’s a similarly ridiculous “tank-icopter”. It’s not asfast as the tank-plane, and few people enjoy flying them, but it’s less likely to kill people.

	It’s not that C++ or Java, as languages, are evil. They’re not. Languages are just tools. Java was designed to run in embedded systems likeautomatic coffee pots and cable-TV “set top boxes”, so closures were cut for time in the first release, because these use cases don’t requirehigh-level programming features. I haven’t map-reduced a toaster cluster for years. Contrary to popular history, Java wasn’t designed
	with an ideological, “enterprise” distrust of the programmer. That trend, in the Java community, came later, when the 1990s attempt tocommoditize programming talent (a dismal FailureFactory) co-opted the language.

	Java and C++ became languages in which “object-oriented programming” was the norm. The problem is that what is currently called OOPis nothing like Alan Kay’s vision. His inspiration was the cell, which hides (encapsulates) immense mechanical complexity behind asimpler interface of chemical and electrical signals. The idea was that, when one needs complexity, simpler interfaces are invaluable, and thatcomplex systems generally should have comprehensible interfaces. Kay was not saying, “go out there and create giant objects” or “useobject-oriented programming everywhere”. He was attempting to provide tools for dealing with complexity when it becomes inevitable.Unfortunately, a generation of software managers took “object-oriented” magic and immodularity as virtues. This is similar to the“waterfall” software methodology, named by a person clearly stating it was the worst idea, and yet taken by suits as having been“recommended by some smart guy” once it was given a name.

	What’s wrong with Big Project development? First, it encourages reliance on internal vaporware. Important work can be delayed “untilMagneto is done”. When Magneto is done, that will solve all our problems. This works for managers seeking to allocate blame for slowprogress– that fucking Magneto team can never get their shit done on time– but it’s a really bad way to structure software. In the small-programmodel, useful software is being continually released, and even an initiative that fails will provide useful tools. In the big-program arena,the project is either delivered in toto or not at all. What if half the Magneto team quits? What if the project fails for other reasons? ThenMagneto est perdu. It’s a huge technical risk.

	[ETA: when I wrote this essay, I was unaware that a company named Magneto existed. “Magneto”, in this essay, was a potential name fora large project in a hypothetical software company. There is absolutely no connection between these project names used here and realcompanies. Consider names like “Cindarella” and “Visigoth” to be gensyms.]

	Second, under a big-program regime, people are trackable, because most programmers are “on” a single program. This is also somethingthat managerial dinosaurs love, because it provides implicit time-tracking. Mark is on Cindarella, which has a headcount of 3. Sally is onVisigoth, which has a headcount of 5. Alan is on the project that used to be called 4:20 until Corporate said that name wasn’t okay and thatnow can’t decide if it wants to be called 4:21 or BikeShed.

	What this kills, however, is extra-hierarchical collaboration– the lifeblood of a decent company, despite managerial objections. In a small-program software environment, people frequently help out other teams and contribute to a number of efforts. A person might be involvedin more than 20 programs, and take ownership of quite a few, in a year. Those programs end up having general company-wide use, andthat creates a lot of cross-hierarchical relationships. MBA dinosaurs, alas, hate cross-hierarchical, unmetered, work. That kind of work isimpossible to measure, and a tightly-connected company makes it hard to fire people. On the other hand, if a programmer only works onone Big Project at a time, and has no other interaction with other teams, it’s much easier to “reduce headcount”.

	The third problem with big-program methodology is that it’s inefficient. Six months can be spent “on-boarding” an engineer into thecomplexities of the massive Big Project he’s been assigned to work on. In light of the average job lasting two to three years, that’s justintolerable. The on-boarding problem also restricts internal mobility. If someone is a bad fit for his first Big Project, moving him to anothermeans he could spend up to a year just on-boarding, accomplishing zilch. Transfers become unacceptable, from a business perspective, sopeople who don’t fit well with their first projects are Just Fucked.

	Why is the on-boarding problem so severe? Big Projects, like large Java programs in general, tend to turn into shitty, ad-hoc domain-specific languages (DSLs). Greenspun’s Tenth Rule sets in, because programmers tend to compensate for underpowered tools by addingpower, but in hasty ways, to the ones they have. They end up developing a terminology that no one outside of them understands. Theybecome systems where to understand any of it requires understanding all of it. This means that people hired to modify or expand theseBig Projects have to spend months understanding the existing system, whose intricacies are only known to a few people in the company,before they can accomplish anything.

	The fourth problem with the big-program methodology is the titular Java Shop Politics. In a small-program development environment,engineers write programs. Plural. Several. An engineer can be judged based on whether he or she writes high-quality software that isuseful to other people in the organization, and this knowledge (talent discovery) is redundant throughout the organization because theengineer is continually writing good code for a wide array of people. What this means is that technology companies can have thelightweight political environment to which they claim to aspire, in which a person’s clout is a product of (visible) contribution.

	On the other hand, in a big-program shop, an engineer only works on one Project, and that project is often a full-time effort for manypeople. Most people in the company– especially not managers– have no idea whether an individual engineer is contributingappropriately. If John is chugging away at 5 LoC per day on Lorax, is that because he sucks, because the team failed to on-board him, orbecause Lorax is a badly structured project? In a small-program environment where John could establish himself, such a question could beanswered. The good programmers and bad projects (and vice versa) could be objectively discovered. In a big-program world, none of thatwill ever be known. The person with the most clout on Lorax, usually the technical lead, gets to make that assessment. Naturally, he’sgoing to choose the theory that benefits him, not John. He’ll never admit that Lorax is badly designed or, worse yet, was a mistake in thefirst place. So under the bus John goes.

	In general, it is this fog of war that creates “office politics”. When no one knows who the good and bad contributors are, there are majorincentives toward social manipulation. Eventually, these manipulations take more of peoples’ emotional energy than the actual work, andthe quality of the latter declines. This is not limited to technology; it’s the norm in white-collar environments. Small-program developmentis an antidote. Large-program development accelerates (but does not necessarily cause) the poison.

	The solution to this is simple: Don’t become a Java Shop. I’m not saying that Java or the Java Virtual Machine (JVM) is evil. Far from it, Ithink Clojure and Scala (which run on the JVM) are excellent languages, and the JVM itself is a great piece of software. Writing anoccasional (small) Java or C++ program won’t destroy a company, obviously. On the other hand, fully becoming a Java Shop (where thevast majority of development is done on large Java programs, and where success relies on understanding defective “design patterns” andflawed “best practices” instead of being able to code) will. There is no avoidance of this; politically speaking, Java Shops go straight down.The quality of engineering, predictably, follows.

	The company I discussed earlier, which was one of this country’s most promising startups before this happened, did become a Java Shop,despite furious protest from existing talent. Within weeks, the politics of the organization became toxic. There was an “old team” thatadhered to the Unix philosophy, and a “new team” that was all-Java. (No Scala or Clojure; Scala was flirted with and had seriousmanagerial support at first, but they killed their Scala efforts for being “too close” to the old team.) This old/new cleavage ruptured thecompany, and led to an epic talent bleed– a small company, it lost several engineers in a month. Java Shop Politics had arrived, and therewas no turning back.Here’s what that looks like. First, it’s not that Java (or C++) code is inherently evil. Any decent software engineer will have a passingcompetency in half a dozen languages (or more) by my age, and these languages are worth knowing. Some Clojure and Scala programsrequire classes to be written in Java for performance. That’s fine. Where a company starts to slide is when it becomes clear that “the realcode” is to be written in Java (or, as at Google, C++) and when, around the same time, big-program methodologies become the norm. Thismakes the company “feel” managerially simpler, because headcount and project efforts can be tracked, but it also means that the companyhas lost touch with the ability to assess or direct individual contribution. Worse yet, because big-program methodologies and immodularprojects are usually defective, the usual result is that people with bad tastes thrive. Then the company becomes less like a softwareenterprise and more like a typical corporation, in which all the important decisions are made by the wrong people and decline becomes inevitable.

	This entry was posted in Programming and tagged C++, java, java shop, java shop politics, office politics, software engineering, unix.Bookmark the permalink.36 thoughts on “Java Shop Politics”