What Killed Waterfall Could Kill Agile.

  • Download Gist
What Killed Waterfall could Kill Agile.textile
Textile

ganked from unreadable scribd doc here: http://cleancoder.posterous.com/what-killed-waterfall-could-kill-agile

What Killed Waterfall could Kill Agile.

Robert C. Martin
20 Nov, 2010

In 1970 a software engineer named Dr. Winston W. Royce wrote a seminal paper entitled Managing the Development of Large Software Systems. This paper described the software process that Royce felt was appropriate for large-scale systems. As a designer for the Aerospace industry, he was uniquely qualified.

He began the paper by setting up a straw-man process to knock down. He described this naïve process as “grandiose”. He depicted it with a simple diagram on an early page of his paper. Then the paper methodically tears this “grandiose” process apart. In the end, Royce proposed a far more nuanced and insightful approach, leaving the reader to giggle at the silliness of the “grandiose” model.

Royce’s paper was an instant hit. It was cited in many other papers, including several very important process documents in the early ’70s. One of the most influential of these was DOD2167, the document that described the software development process for the American Department of Defense. Royce was acclaimed, and became known as the father of the DOD process.

There was just one problem. The process that DOD2167 adopted was Royce’s straw man! Apparently the authors of DOD2167 did not actually read Royce’s paper; because they adopted the “grandiose”, naïve process that Royce’s paper had derided. To his great chagrin, Dr. Winston W. Royce became known as the father of the waterfall.

Though Royce railed and fought against it, the snowball was in motion. It kept on growing as it rolled down the mountains of software companies and industrial countries. Year by year the waterfall gained in popularity leaving it’s father to wonder about the justice of the universe and whether there was intelligent life on Earth.

By the middle of the 1990s, the waterfall process dominated the world of software. The field of Software Engineering was defined by it; and by the catalog of analysis and design documents that Architects, Designers, and Analysts were expected to produce. Coding was a detail — the least important part of the process. If you wrote your documents well, and drew all the necessary diagrams, then you were doing it right. You were an engineer. The code could be left to the to the unwashed minions in the cellar.

This attitude created a schism in the technical community. There were the elite Architects, Designers, and System Analysts who did the real engineering by satisfying the first two phases of the waterfall. And then there were the grunts who actually had to make everything work in the final phase. When the project got behind schedule, it was the grunts who worked overtime. When the project failed, it was the grunts who bore the blame.

This was a great deal for the elite Architects, Designers, and Analysts! Who wouldn’t want a job like that? You have the authority to specify everything, and none of the responsibility to actually make it work. You get to command a high-salary, the respect of your peers, and the envy of the masses; and there’s almost no way you can fail. When bad things happen, you can always blame it on the programmers. Yes, this is a bit of an exaggeration; but only just a bit. The attitudes of elitism were very real. Those who could analyze and design were considered too valuable to waste on mere coding. Code became the orphaned child of Software Engineering. By 1998 the cracks were already appearing in the waterfall edifice. Programmers everywhere were starting to reject the elitism of the Architects, Analysts, and Designers. They started complaining about the rigidity and weight of the waterfall mantle they were forced to wear. Beedle, Devos, Schwaber, and Sutherland had published their seminal paper on Scrum, and Kent Beck had created a movement around eXtreme Programming (XP).

Scrum broke the waterfall apart. Rather than spending months or years creating reams of documents through a series of sequential phases, Scrum suggested that a team of developers should work in short 30-day1 cycles to implement features.

1 Nowadays two weeks is more common than 30 days. Scrum and XP teams have found that too much can go wrong in a month.

Scrum suggested that the development team should decide when and if a document was necessary. Scrum took the emphasis away from documents and artifacts, and put it squarely on getting features working, and on the decision making power of the team. Scrum broke the monopoly on authority held by the elites and put it in the hands of the development team.

XP took this a step farther by increasing the emphasis on the act of programming, and by declaring code to be important. XP is the integration of Scrum with a set of engineering disciplines. Those engineering disciplines have a huge effect! Scrum is a day-by-day process. It provides a framework that describes what your day will be like, but it says nothing at all about how you should work in the hours and minutes of that day. XP is a minute-by-minute process. The engineering disciplines of XP fill your day. XP provides guidance for the creation of each line of code. It provides a framework within which coding and design decision can be made.

Scrum is a subjective process: The team rules. There is no objective measure of success, or of quality, or even of completion. It is up to the team to define these things. The engineering disciplines of XP add some objective measures to Scrum. XP defines design and code quality, and provides guidance on how to achieve it. XP defines the meaning of done, and how done-ness can be measured.

By 2001 the software community was a-buzz with these revolutionary ideas. The Agile Manifesto had been written, and had become the centerpiece behind an energetic, enthusiastic, and growing movement. The very definition of Software Engineering was being challenged, and that challenge was succeeding.

The elitism engendered by Waterfall was being attacked. Neither Scrum nor XP had any role for an elitist who assumed authority without taking responsibility. In Scrum, the rule of the team is stronger than the rule of an Architect or Designer. Contributions are welcome, but not necessarily followed.

XP took this even farther. If you wanted to contribute to an XP team, you were welcome; but you took explicit responsibility for your contributions. For Architects and Designers that meant they wrote some code. For Analysts, that meant they wrote tests. Everyone on the team had the responsibility to make things work.

For a while it looked like Software Engineering elitism was dying; that authority and responsibility had been aligned once and for all. But elitism is a tough thing to kill. Whenever you rob Peter to pay Paul, you can be sure that Paul’s outrage will be less than Peter’s.

Both XP and Scrum defined the role of a coach. It is the coach’s responsibility to defend the process. The coach reminds everyone of their commitment to their disciplines and to the process. When the schedule looms, and customers are angry, it is the coach who reminds the team that the best way to meet the schedule and calm the managers is to hold to their disciplines. In Scrum, this role was called “Scrum Master”.

XP defined the role of coach quite informally. The role would float between members of the team. One month it would be Joe, the next it would be Jane. It was not a title, and it conferred no authority. There were no decisions to be made, and no power of enforcement granted. The coach had the responsibility to remind, not to command.

In Scrum, something different happened…

The very first Certified Scrum Master course was taught at the Object Mentor offices in Vernon Hills, Illinois. Ken Schwaber called me up and asked if I had a training room he could use. I told him I would be more than pleased to host his course. He kindly allowed many of the people who worked for me at the time to attend for free, and to become CSMs.

Frankly, I thought the idea was a bit silly. I didn’t think thousands of people would be lining up to get their certifications. But I had not considered the lure of elitism. It didn’t occur to me that this special training course, coupled to the term Certified Scrum Master, would become a wedge to break the alignment between authority and responsibility.

Who was it who lined up to take the CSM courses? Was it Scrum team members who wanted to help their teams? Was it programmers and testers? Yes, there were certainly some CSMs who came from existing teams. But the vast majority of CSMs have a project management background. In essence they have added CSM to the PMBOK. They have become CSMs so that they have the authority to manage Scrum teams.

This was never the intent. The role of the coach was to act as a gentle reminder of process and discipline. The coach was never supposed to manage the project or the schedule! Indeed, these two roles were supposed to be adversarial! It is the project manager’s role to remind the team about the schedule and to encourage them to change something so that the schedule can be met. It is the coach’s role to remind the team to hold to the process.

True XP coaches are not project managers nor are they team leaders. They do not lead the team to success, and cannot claim credit for that success. Indeed, the role is considered optional because mature teams will probably not need frequent reminders. But CSMs often assume the role of team leader. They are viewed as a critical component of the team; without whom the team cannot function. In XP a team without a coach is no big deal; but a scrum team without a scrum master is an oxymoron.

Indeed, the role of Scrum Master is considered so important, that it requires certification to obtain. If your Scrum team does not have a Certified Scrum Master, then something must be wrong with you.

When a Scrum team succeeds, it is the CSM who steps forward to receive the award (on behalf of the team, of course). But what happens when a Scrum team fails? Is it the CSM who steps forward and falls on his sword? Does the CSM take the lashes and protect the team?

The elitism is back, and it’s growing. More courses with certifications are available, and even more are envisioned. Other training companies are offering their own certifications. After all, the lure of elitism is a great moneymaker. The snowball is rolling down the mountain, and getting bigger with each turn.

And when the revolution comes… ?

I can only hope that when Scrum goes down it doesn’t take the whole Agile movement with it

Thank you for making it work without Flash ;-)

Well, this is what's bound to happen when you start to name things.

"Agile" is doomed too... it's been since when the name "Agile" started to pop up in conferences and on resumès. When you invent a label it comes with a cost: everyone can use it. And guess who likes labels a lot? Usually someone who has poor knowledge of the subject but needs some handles because he has to make decisions about it (hint: management).

If you can code, you're a programmer. If you're the best coder in your team, you're the team leader. It's really as simple as this. Nobody who actually is a team leader needs labels - the only ones who can give you real authority are your teammates... the ones who (when the time comes) will have to trust you and submit to your authority.

In practice, "elitism" can kill any process. It's not the fault of Scrum if the process is not run correctly, and when something else comes and replaces Scrum, also that process will be ruined by those who manage it to hell. Authority without responsibility is something straight from Dilbert, and we all know the success rate of those projects. :-)

I can agree that a lot of people are cashing in on the buzzwords to make more money (you can hardly blame anyone for that). And this inevitably creates elitism. The role of management is to ensure egos and attitudes remain in check. Passing out awards to the PM/CSM/Lead is a management problem, not a process problem.

I like this idea of Scrum elites "only if you make it exactly as we say it will work".
What about inspect & adapt? We are not using Scrum for a long time now. Moved to custom process based on Kanban. Whoever is doing Scrum exactly as learned on the course is ... doing it wrong.

Formalism and an abundance of buzzwords is why I never liked scrum in the first place. Agile is about being adaptive, no one formalised system can ever meet this requirement.

The article's grasp of the abuses of Scrum is as flawless as its grasp of the mysterious career of "Waterfall - The Process That Never Was." But there's no need to be so tribal about this. Of course there are drones collecting merit badges in the world, and the hazard of feeding those animals made "Certification" a bit controversial in Scrum circles from the outset. But since the actual training is about facilitation not leading, empowerment not glory, adapting and leaning the process, and maturing the team so the formal role can wither away, it's more in the line of a (perhaps Quixotic) battle against the abuses the article catalogs.

I think you're spot on about how what really matters is the team, not anyone who would take credit for the team.

But the picture you paint of Scrum and Scrum Masters doesn't apply to my company. I work for a very humongous online retailer and here the "coach" role you describe in XP is played by managers and team members and project managers. Project managers often wear the hat of "Scrum Master" but for us that simply means they manage the Scrum artifacts and do the day-to-day work of keeping everyone apprised of progress and schedules and dependencies. They also unblock the team by helping to solve problems with internal and external dependencies, and so on. When the team succeeds or fails the entire team succeeds or fails. In my nearly 4 years here I have never seen a Scrum Master or project manager or even manager take credit for the success of a project.

In short: Just because you use Scrum doesn't mean you're going to have some elitist role that takes the credit or works against the best interests of the team. You might have this, but then again any process is vulnerable to arrogance and corruption, even XP.

it would be nice if these comments were on bob's OP.

Good point, JonKernPA: re-commented over there.

Software development is almost entirely dependent on people. Required technical skilled to do the work is the property of people we hire. People want to be commensurate for the skills they have. Three options:

  • Elite rules: risk returning back to command and control
  • People who do the work rule: risk is that technical devs with low people skills are in leading position.
  • No body rules: risk is companies want some person to be in charge

chill out guys
you are blinded by the red mist if you do not see that we all have different skills and levels of that skill
if you have a perfect cross functional team then all roles of a scrum team can be the developers/testers etc (except the prod owner)
in what walk of life do you not find controls or rules? preferably these are soft, made to be broken, but there to hug and support if you go off track.
certification: what does this help with? stopping any old numpty saying he is a surgeon!!!! "yeah i'll do your brain surgery , step out back"
its just that experience should back the certs and you should choose your roles based on the knowledge needed.
the only scrum guide is from the scrum.org
if you do not follow that you are not doing scrum, it does not mean you dont have a better idea, just that you are not doing scrum
as i say, take a chill pill, play nicely, communicate freely, diverge to converge and all agree on what it is you really want.
laterz dudes

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.