Imagine a Process Regression

I was recently out to dinner with some friends that are fellow techies, and we were talking about software development methodologies. One of them, who works in a shop that practices pretty true-to-form Scrum, was talking about how management via the product owner was starting to put pressure on them to produce certain, lower point estimates during the planning poker game. He said that it struck the younger team members who’d only worked there as extremely strange and that he personally felt that the powers that be were sort of trying to steer them to a more “waterfall-ish” style. I had a random thought and, rudely, burst out laughing during his anecdote which was, in no way, intended to be funny.

The thing that I had thought of, I explained to my confused friends, was how incredibly weird and counter-intuitive moving to a waterfall development methodology would seem to his teammates who had come into the software development world knowing only Scrum. It’d be like coming home one day and telling your family that you were through buying soap because from now on all you needed was some lye, some elbow grease and the river on the other side of town. Of course, there would (and has been) culture shock going from Waterfall to Agile, but that’s mostly born of a kind of “that’s an impossible pipe dream” mentality. Going the other way has more of a “you want to do what…. and why?” vibe.

Think of this. You’ve never known anything but a disciplined agile approach to software. You have an automated build and deployment setup. Every time you check in code, the build executes, unit tests are run, and the code is pushed to a test environment for exploratory QA and validation of user stories. Checkins are frequent, collaboration is ubiquitous and refactoring and cleaning the code is something you do on a regular basis. Once every other week, you ship the code and it’s no big deal. The (business) day after shipping, you and the team collaboratively plan what you’re going to do for the next two weeks, trusting that someone is prioritizing the work you wind up doing toward a larger technical product/vision. Your users see regular progress and give you regular feedback, creating a tight communication loop and eliminating surprises. And, if the users change their mind about the features they want or don’t like what you’ve done over the last two weeks, you simply spend the next two weeks steering methodically back in the right direction. But don’t worry, that’s all about to change!

Forget about all of that bi-weekly shipping nonsense. We’re going to ship once, as far as we’re concerned. A second shipment is by no means a given because frankly, the first one is rather likely to fail and anger the users. So, let’s not concern ourselves with anything but the first shipment. For the first month or so, we’re going to gather the requirements for the project. We’ll skip through a meadow, plucking them off of the beautiful flora and placing them in our baskets. That’ll be a good month. At the end of that month, we’ll tell our users that they’d better be sure that our endless array of diagrams and thousands of statements about what the software “shall” do matches exactly what they want because once we move onto the next phase, there’s no going back or changing their minds without either expensive change requests or lawyers.

Next up comes the design phase. This is the part where we spend a month figuring out how to write all of the code that we’re going to write over the next several months, but without actually writing that code. While this may sound nuts, the reason that we have to do this is because the labor is going to be divided up in such a way that everyone will kind of go off into their own silo and write code for months prior to an extended “integration phase” where our architect, Dr. Frankenstein, assembles the various individually created pieces into a reasonable attempt at a piece of software and then attempts to breath life into it. In order for that task to be as minimally gruesome as possible, a lot of planning is needed.

The next phase is implementation. Implementation phase is where we start coding and the previous two months of ‘work’ become completely irrelevant. Everyone codes in their silos until integration time. Some attempts are made at early integrations, but those are abandoned when it’s discovered that they are time consuming and the project managers are worried about the fact that they’re already behind schedule. Everyone will then pray that somehow we make up the time that we’ve gotten behind at integration time, but, who are we kidding? That’s not going to happen. The next phase will ideally be the testing phase, but we won’t have time for that. Instead, we’ll start the death march phase, where everyone works 14 hour days, 7 days a week up to the deadline.

Then, my friend, we ship. Tada! Ooh, that wasn’t supposed to happen. Heh, heh, weird. I’m sure the rest of it will be fine if I just — ooh, yikes. What do you mean this screen doesn’t look at all like what you were picturing? Heh, heh, maybe we can reschedule this demo. This is the phase where the thing we chucked over the wall, behind schedule and with a minimum of testing, blows up in our faces. After another month or so of death march following the release, we manage to wrangle the software into a state where, while the source code looks hideous, it’s at least stable and no one is screaming at us anymore.

There, doesn’t that sound better than the one where you’re shipping software every two weeks and not working 90 hour weeks to make up for the fact that you’re not actually very good at predicting the future? But of course, I kid. It was just an interesting exercise for me to think of how the world would seem going from agile to waterfall. Agile methodologies are inherently developer-centric when done correctly. Developers like quick feedback, routine shipments, and process automation because this makes for sustainable, incremental progress. The main complaining I hear from waterfall developers isn’t that agile doesn’t sound good — it’s that it sounds too good to be true. Usually it’s things like “we couldn’t do agile because we’re a shop that does _______” where the blank is some excuse that isn’t actually reasonable unless the company is shipping Mars Rovers. So if you’re already doing agile — you’ve already seen how well it works — going back would seem certifiably crazy. And, you know what? I like that there’s a generation of developers out there that think Waterfall is just a bed time story that the old timers use to scare the newbies into behaving.

  • http://genehughson.wordpress.com/ Gene Hughson

    The question that first comes to mind is who is pushing for this – IT management? Business management?

    I’d find the second case very worrying because it would tell me that the customer isn’t getting something they want. Not that a waterfall process is likely to give them what they want, but in process as in development, people tend to talk in terms of what they believe are solutions rather than elaborating on problems.

  • http://www.daedtech.com/blog Erik Dietrich

    I got the sense that it was some kind of matter of bureaucratic optimization for the purpose of a mid-level manager’s bonus, but that and anything else I’d say would really be hearsay.

    Point taken, though. The business leaning on the developers to change their process clearly wants… something.

  • http://www.schmonz.com/ Amitai Schlair

    I imagine I’d be in a position to like this post a lot better if it required me to do some imagining. But I like how realistic it is!

  • Christopher Dewan

    I work at a company that “is shipping Mars Rovers”. Waterfall still doesn’t work, you still end up with a shitty product that is full of bugs and errata (which is just a bug we know about but choose not to fix). They try “agile” but it’s really just scrummerfall. Waterfall doesn’t work, agile seems like too high a risk model for space work, but Waterfall has failed, there has to be a 3rd way for “must work” software.

  • Peter Morlion

    I used to work in a team that did Scrum very well and it worked. I switched jobs (because you can’t stay at one company forever, at least I can’t) and now it’s in between, but with too much focus on process and analysis in my opinion. Some decisions or practices do seem crazy but I have a hard time communicating that Agile/Scrum would be better. But that also means managers need to accept less control, which they aren’t very willing to do. It’s also a matter of trust. Trusting the teams/developers that they too want to deliver great software.

  • http://freeside.co/ Rob Fletcher

    It seems to frequently happen when priorities are driven by marketing or budgeting schedules. e.g. “we need a big bang release to coordinate with our marketing push” or “we need a plan to justify our budget for the next financial year”.

    To my mind this is what’s missing in the description of an “agile” process above. Agile is (or should be) about more than just how the developers behave. It should be a whole-organization approach. “trusting that someone is prioritizing the work you wind up doing toward a larger technical product/vision” is just developers acting in a bubble that sooner or later is going to conflict with the business at large.

  • http://www.daedtech.com/blog Erik Dietrich

    That’s a good insight because based on my (again, limited) knowledge of that organization, I think it is an isolated pocket of agile in a much more bureaucratic world. If the devs “go agile” but the organization doesn’t buy in or understand, there will be friction points.

    My anecdote was really just from the hypothetical junior developer’s position though. I think a good manager will create a bubble in which the developers can be productive. I don’t mean keeping them in the dark about how the company works, but rather making it so that they can focus on building software without organizational chaos or politics interfering.

  • http://www.daedtech.com/blog Erik Dietrich

    You’ve gone from a disciplined agile process to a “traditional waterfall” death march style place? Ouch.

  • http://www.daedtech.com/blog Erik Dietrich

    This is one situation in which I’ve never worked, so I’d be interested to read about or experience the third way you mention. I’ve always assumed that if the business truly has unlimited capacity to define requirements and enforce that they won’t change, then it’s theoretically possible to make a waterfall approach work (with waterfall’s big weakness, in my book, being that it hinges on the quixotic notion that you can really predict and control future unknowns with enough analysis).

    Sounds like even with complete control it’s not a given and the approach doesn’t work. Agile does indeed seem tough in your situation becuase it seems to carry and overriding philosophy of “let’s start out with our best shot quickly, see what can be improved and go from there.” I’m wondering if there aren’t advancements coming in the arenas of design by contract and even programming correctness proofs that might address your situation.

  • http://www.daedtech.com/blog Erik Dietrich

    I really do think there’s a lot of unintentional sabotage from companies, and especially people, hanging onto traditional management models. My team follows Scrum pretty well and we’re also in the sweet spot for team size, and what I’ve observed is that the role of “project manager” is pretty much unneeded. Project managers, when you parse out the organizational politics, pretty much plan project timelines and report status. But with good ALM tooling and Scrum, no status reporting is necessary, and the planning is accomplished via backlog grooming and product owner prioritization.

    So I agree with your take that PMs and line managers are hesitant to relinquish control. But I think that the reason goes beyond a trust issue alone and creates a bit of an existential crisis for the PM who thinks “well, if this works, I guess I’m pretty expendable now aren’t I?”

  • http://www.schmonz.com/ Amitai Schlair

    I’ve been on a team that implemented TDD, then Scrum, got reliably and predictably excellent results, changed hands, and was being passive-aggressively allowed to keep doing what it’s doing, except when it isn’t. When it wasn’t, I mean. Because this isn’t something that’s happening to me right now, definitely.

    Unless I can change the direction (and thus far I’ve failed), the destination of our new journey will be much more like waterfall than like anything Agile. If I stick around to continue resisting, it’ll take a while; if don’t not, it won’t.

    It’s a strange irony that while good Agile teams can respond adaptively to nearly any kind of change, practicing Agile well is extremely brittle (and often breaks permanently) when team composition changes.

    You mentioned somewhere you were ruminating on taking a crack at Agile coaching, right? I’ve just started doing that and am finding it awesome. Hoping to find a way to write about it.

  • http://www.daedtech.com/blog Erik Dietrich

    This makes me wonder if some kind of self-referential process, in the Godel sense, wouldn’t be an interesting concept to philosophize about. Is there a software development process that responds to external stimulus by improving (or at least not falling apart)?

    Anyway, I’m intrigued by the concept of agile coaching, but I have little idea as to what it entails. I see a general future for myself in free agency where I’d sign on with small development shops in a contract capacity to make their team of junior or home-grown devs more effective. Certainly agile coaching seems in line with that eventual end. I have your feed in my RSS reader, but I must admit that I don’t the reader as often as I used to, so please feel free to send me a link if you do write about it.