DaedTech

Stories about Software

By

Agile Methodologies or Agile Software?

Over the last couple of months, I’ve been doing mostly management-y things, so I haven’t had a lot of trade craft driven motivations to pick Pluralsight videos to watch while jogging. In other words, I’m not coming up to speed on any language, framework, or methodology, so I’m engaging in undirected learning and observation. (I’m also shamelessly scouring other authors’ courses for ways to make my own courses better). That led me to watch this course about Agile fundamentals.

As I was watching and jogging, I started thinking about Agile Manifesto and the 14 years that have passed since its conception. “Agile” is undeniably here to stay and probably approaching “industry standard.” It’s become so commonplace, in fact, that it is an industry unto itself, containing training courses, conferences, seminars, certifications — the works. And this cottage industry around “getting Agile” has sparked a good bit of consternation and, frequently, derision. We as an industry, critics might say, got so good at planning poker and daily standups that we forgot about the relatively minor detail of making software. Martin Fowler coined a term, “flaccid Scrum” to describe this phenomenon wherein a team follows all of the mechanics of some Agile methodology (presumably Scrum) to the letter and still produces crap.

It’s no mystery how something like this could happen. You’ve probably seen it. The most common culprit is some “Waterfall” shop that decides it wants to “get Agile.” So the solution is to go out and get the coaches, the certifiers, the process experts, and the whole crew to teach everyone how to do all of the ceremonies. A few weeks or months, some hands on training, some seminars, and now the place is Agile. But, what actually happens is that they just do the same basic thing they’d been doing, more or less, but with an artificially reduced cycle time. Instead of shipping software every other year with a painful integration period, they now ship software quarterly, with the same painful integration period. They’re just doing waterfall on a scale of an eighth of the previous size. But with daily standups and retrospectives.

There may be somewhat more nuance to it in places, but it’s a common theme, this focus on the process instead of the “practices.” In fact, it’s so common that I believe the Software Craftsmanship Manifesto and subsequent movement was mainly a rallying cry to say, “hey, remember that stuff in Agile about TDD and pair programming and whatnot…? Instead of figuring out how to dominate Scrum until you’re its master, let’s do that stuff.” So, the Agile movement is born and essentially says, “let’s adopt short feedback cycles and good development practices, and here are some frameworks for that” and what eventually results is the next generation of software process fetishism (following on the heels of the “Rational Unified Process” and “CMM”).

That all played through my head pretty quickly, and what I really started to contemplate was “why?” Why did this happen? It’s not as if the original signatories of the manifesto were focused on process at the exclusion of practices by a long shot. So how did we get to the point where the practices became a second class citizen? And then, the beginnings of a hypothesis occurred to me, and so exists this post.

The Agile Manifesto starts off with “We are uncovering better
ways
of developing software…” (emphasis mine). The frameworks for this type of development were and are referred to as “Agile Methodologies.” Subtly but very clearly, the thing we’re talking about here — Agile — is a process. Here were a bunch of guys who got together and said, “we’ve dumped a lot of the formalism and had good results and here’s how,” and, perversely, the only key phrase most of the industry heard was “here’s how.” So when the early adopted success became too impressive too ignore, the big boys with their big, IBM-ish processes brought in Agile Process People to say, “here’s a 600 page slide deck on exactly how to replace your formal, buttoned-up waterfall process with this new, somehow-eerily-similar, buttoned-up Agile process.” After all, companies that have historically tended to favor waterfall approaches tend to view software development as a mashup of building construction and assembly line pipelining, so their failure could only, possibly be caused by a poorly engineered process. They needed the software equivalent of an industrial engineer (a process coach) to come in and show them where to place the various machines and mindless drones in their employ responsible for the software. Clearly, the problem was doing design documents instead of writing story cards and putting Fibonacci numbers on them.

The Software Craftsmanship movement, I believe, stands as evidence to support what I’m saying here. It removes the emphasis altogether from process and places it, in very opinionated fashion, on the characteristics of the actual software: “not only working software, but also well-crafted software.” (emphasis theirs) I can’t speak exactly to what drove the creation of this document, but I suspect it was at least partially driven by the obsession with process instead of with actually writing software.

MolotovCocktail

All of this leads me to wonder about something very idly. What if the Agile Manifesto, instead of talking about “uncovering better ways,” had spoken to the idea of “let’s create agile software?” In other words, forget about the process of doing this altogether, and let’s simply focus on the properties of the software… namely, that it’s agile. What if it had established a definition that agile software is software that should be able to be deployed within, say, a day? It’s software that anyone on the team can change without fear. It’s software that’s so readable that new team members can understand it almost immediately. And so on.

I think there’s a deep appeal to this concept. After all, one of the most annoying things to me and probably to a lot of you is having someone tell me how to solve a problem instead of what their problem is, when asking for help. And, really, software development methodologies/processes are perhaps the ultimate example of this. Do a requirements phase first, then a design phase, then an implementation phase, etc. Or, these days, write what the users want on story cards, have a grooming session with the product owner, convene the team for planning poker, etc. In both cases, what the person giving the direction is really saying is, “hey, I want you to produce software that caters to my needs,” but instead of saying that and specifying those needs, they’re telling you exactly how to operate. What if they just said, “it should be possible to issue changes to the software with the press of a button, it needs to be easy for new team members to come on board, I need to be able to have new features implemented without major architectural upheaval, etc?” In other words, what if they said, “I need agile software, and you figure out how to do that?”

I can’t possibly criticize the message that came out of that meeting of the minds and gave birth to the Agile Manifesto. These were people that bucked entrenched industry trends, gained traction, and pulled it off with incredible success. They changed how we conceive of software development and clearly for the better. And it’s entirely possible that any different phrasing would have made the message either too radical or too banal for widespread adoption. But I can’t help but wonder… what if the call had been for agile software rather than agile methods.

  • Andy Bailey

    This struct so many chords with me!

    I revisited the links posted before I commented, printed the Manifesto for Software Craftsmanship and stuck it up on our Scrum Board.

    The reason I did this is because I want to remind members of my team that Agile Development is not simply about crushing the Backlog but doing so without sacrificing quality. Sadly this is something which often falls by the wayside, many developers are prepared to accept mediocre as the way to produce results quickly.

    A transition from “Waterfall” to more modern software development practices has to, as a priority, throw out most of the entrenched practices which evolve in the former. Without that the same entrenched, often stifling, practices simply get merged into the latter. I know that this is so obvious that you didn’t need to point it out however there is an enormous personal cost to the management team who make the decisions. They will be resistant to change if they cannot see any benefit and won’t see any benefit until they can understand the processes involved, in terms they can understand. Their mindset will be “lets keep the old but dress it up in the new” because that is easier for them to understand. This is where all the coaching, training and “Agile” seminars should be directed, the majority of software developers will be engaged from the outset: it gives them the environment they need to flourish.

    • I think it’s the business itself that pushes us into churning out code quickly rather than worry about quality. I’ve seen it happen so many times.

      Most of the time they don’t even understand what we do.
      The management sees as a cost rather than the engine, which is what it should be.

    • The shift in mindset is difficult for a lot of people in management roles, I think. It’s a migration from command and control to trusting knowledge workers, which would be difficult anyway, but especially because there’s a lot of uncertainty in change, and command and control types tend to react to change by exerting even more control.

  • Jordan Whiteley

    The agile software angle is truth. Don’t test, write tests. Don’t deploy write deployments scripts. Don’t update the database, write a migration script to update the database. Don’t populate the database with testing data, write a seed method to load it up on a successful migration. Don’t build software like it’s a static object. Build a living breathing moving self sustaining ecosystem.

    • Exactly. I’ve described this myself in posts, if memory serves. Instead of documenting or creating methods, automate/implement.

  • Geoff Mazeroff

    Part of your thesis reminded me of several discussions about how where I work (FDA-regulated) used to operate: Follow 21 CFR 820 and you will magically produce working (and compliant) products. Now we treat those regs as the low-bar. We strive to build quality products (one of which is a software component) and because we do, we inherently meet the FDA’s acceptable minimum. In short, you can’t just follow a recipe and expect the right answer to come out of the oven.

    I used to be more active in the software craftsmanship movement and found myself getting too wrapped up in it as an identity claim. It seems like we know there’s something better, and we cast around looking for gurus to hand down the wisdom we so desperately seek. I don’t quite agree, however, with your argument (please correct me if I misinterpreted it), that process is the problem. At some level we have general guidance about how to develop, maintain, and reason about software. My opinion is that if you marry yourself to process hoping it will fix what ails you, a healthy long-term outcome is not what results.

    Part of me wonders if were in this collective binary search for the truth: The pendulum swings from structure to complete freedom, org charts to flat management. Where will we go next?

    Thanks for sharing your insights.

    • It’s not that I think process is the problem or unnecessary. It’s more that I think software groups should define their own processes with their goals in mind. In other words, whether it’s waterfall, Scrum, RUP, whatever, there’s usually some management structure saying, “we want you to make good software and here are the steps you’re going to follow in order to do that.” What I’m saying is that they should just describe their goals (good, “agile” software) and let the group figure out what process it needs to meet those goals.

      I do think you’re right about swinging pendulums and over-compensation as a general norm. It’s probably human nature to get tired of heavy process and just throw process out altogether and attempt to go with the polar opposite approach.

    • Paul

      I agree, the agile process isn’t the problem so long as one remembers that the engineering of the software is an integral part of it. I blame Scrum for planting the idea in peoples heads that one can have “agility” in isolation of appropriate technical practices and there is now whole industry devoted to peddling that particular brand of snake oil. When you look at successful dev shops they are by and large doing what Kent Beck set out in the seminal “Extreme Programming Explained”. The problem is that one cannot approach a blue chip company with the pitch “hey man were going to get you guys doing extreme programming” so most agile consultancies pitch on the Scrum ticket. Next thing you know someone in the client company reads up about Scrum and thinks they get it … when they dont.

  • Kostas Triantafyllou

    I think this is a very well written post and the only thing missing from it is the management perspective. Software development costs money therefore budget restrictions apply to the whole process. Sometimes from a technical viewpoint we tend to miss this point and concentrate more on aspects that relate to improving our work and this is always welcome. But we are not just doing software to satisfy ourselves. Somebody pays for it. Personally i don’t care about agile or not, i do care about the quality (character) of people working together and setting some guidelines on doing things. This works better because you get to tweak the process according to your budget.

    • Ethan Moe

      In my observation, the inter-relatedness of quality software with the rest of the business is ignored by management. Because the dev team was not given the extra three months to craft the software correctly, the company needs 500 people in the tech support call center instead of 250… which leads to budget constraints… which lead to shorter development times, which leads to poorer software… which leads to more phone monkeys…

      I’ve worked at companies trying to dig themselves out of that hole. It takes years. Some never make it.

    • When you refer to management, are you talking about the comparably large overhead structure at most medium sized to large companies? The reason I ask is that if you were working in a shop that was one small Scrum team working on a public-facing product, “the management perspective” would be essentially non-existent, or at least non-distinguishable from the team’s. A good manager there would simply say, “do what the customers want and let me know how I can make your job easier.”

      It sounds as though you’re talking about situations where there are more degrees of separation between the development team and the software stakeholders…?

      • Kostas Triantafyllou

        well i think budget restrictions apply to all projects. Even the one you describe has some funding either from the owner of the company or the first customers (maybe both). The actual budget comes as a consequent of project or product estimation in your case. If this is flawed (and this can happen for several reasons) then so will be the budget. In the worst case scenario you will run out of money and you ll have to take shortcuts to succeed or just fail.
        I think this better describes “the management perspective” and i wasn’t talking about good or bad management – this is another story.

        • Oh okay, I think that clarifies it for me. When I think of the idea of “agile software” I think about giving someone who’s managing the purse strings (and thus priority decisions) as many options as possible. With software that’s easy to deploy, easy to modify, and easy to understand, there are a lot more options than with software that doesn’t have those properties. You can ramp staff up or down, you can quickly re-prioritize features based on market response, you can easily cut scope (or add it), etc.

  • BobC

    I think that there are two characteristics we want in the software we are developing.
    While the concept of “agile” (flexible, responsive, adaptable) development was an outcome of the shortcomings of “waterfall” development, it is not enough to have “agile” software.

    By “agile” software, I mean software that is readily and easily altered when needed. This is the flexibility of the software’s design and coding that makes it adaptable in response to new conditions, whether those conditions be a new requirement from the user, or a migration to a new database, or a change in staff.

    In addition to the characteristic of agility, our software should be “robust”. Robust individuals remain healthy, even in the presence of stressors in their environment. Robust software continues to work properly in conditions that do not fall into a narrow set of assumptions. Too often, the expeditious route is to assume perfect behaviour by users and a system environment that never falters.

    • I think that (robustness) is certainly another way tp describe good software. I wasn’t intending to define the canonical way to describe it — just musing that it might have produced less “hucksterism” in the industry to describe the software rather than the process.

  • Tom McCollough

    I don’t think it needs to be an either-or situation. I think that Agile refers to a small set of values for a software development process. In particular the values cover the question of lightness, at a time when we were realizing that heaviness wasn’t working. I don’t think the Agile Manifesto claims to cover every possible value one could have for software, nor for a software development process.

    • Agreed. There’s certainly no reason that the team can’t also be agile in its approach (and it seems a worthy first-class goal). And I wasn’t posting anything intending to be a criticism by any stretch — just musing.

  • Ivan Viteri

    Great article! Totally agree, but I would take it a step further and ask if we are building good software that really solves business problems (or whatever its objectives are).

    I have seen great software that is not aligned with the actual need or process and becomes great technology that nobody wants to useā€¦

    • Andy Bailey

      Great software that has no market is still a failure, not of the dev team(s) but of the management that called it into being.
      There are also very good Open Source projects out there that suffered the same problem and either died or are not maintained any longer.
      Either way these are just as tragic and avoidable as producing a substandard product.

    • Interestingly, after a quick read back through both manifestos, I see that the idea of a market for their software (i.e. the business) is referred to only obliquely. There is talk of “adding value” but I think it’s taken as axiomatic that improving the quality of the software is a commercial win. They might say that commercial viability is beyond their scope, and I don’t know that I could fault them, but I do think it’s worth considering that software isn’t really an end but a means to an end. In other words, I agree with you that there’s something deeply quixotic about writing an awesomely crafted piece of software that helps no one do anything.

  • Lyndsey Padget

    This article embodies my exact struggle as a “software engineer/architect reluctantly turned project manager”. I believe in agile the way I believe in going to the gym – you only get gains out of it if you spend your time efficiently, doing the things that work for you (not the guy next to you). Agile has been turned into a meaningless buzzword in most cases, causing engineers to become jaded about its original purpose. On the other hand, I regularly solicit their feedback about how “the processes” (yeah, I said it) are working (or not), because many folks are quick to complain in private but hesitant to collaborate openly. An essential way I build trust among engineers – as well as preserve my own sanity – is to keep being an engineer. The moment I stop coding alongside my group is the moment I lose sight of the struggles they face in building well-crafted software… or at least, the software someone wanted us to build.

    • I was on a call yesterday where I heard someone say (paraphrased from memory) “developers like to complain that they’re not being heard, but don’t actually want to be heard.” I think it was a tongue in cheek comment, but it underscores the point that you raise, which is that it’s important to get feedback and look to improve steadily.

      I think that your approach with continuing to write some code is the new way forward in technical leadership. I’ve formed the opinion that technical leadership should happen by example and not by org chart.