Automation and the Art of Software Maintenance
Editorial Note: I originally wrote this post for the SubMain blog. You can check out the original here, at their site. While you’re there, check out CodeIt.Right for automating your code review process.
I have long since cast my lot with the software industry. But, if I were going to make a commercial to convince others to follow suit, I can imagine what it would look like. I’d probably feature cool-looking, clear whiteboards, engaged people, and frenetic design of the future. And a robot or two. Come help us build the technology of tomorrow.
Of course, you might later accuse me of bait and switch. You entered a bootcamp, ready to build the technology of tomorrow. Three years later, you found yourself on safari in a legacy code jungle, trying to wrangle some Sharepoint plugin. Erik, you lied to me.
So, let me inoculate myself against that particular accusation. With a career in software, you will certainly get to work on some cool things. But you will also find yourself doing the decidedly less glamorous task of software maintenance. You may as well prepare yourself for that now.
The Conceptual Difference: Build vs Maintain
From the software developer’s perspective, this distinction might evoke various contrasts. Fun versus boring. Satisfying versus annoying. New problem versus solved problem. My stuff versus that of some guy named Steve that apparently worked here 8 years ago. You get the idea.
But let’s zoom out a bit. For a broader perspective, consider the difference as it pertains to a business.
Build mode (green field) means a push toward new capability. Usually, the business will regard construction of this capability as a project with a calculated return on investment (ROI). To put it more plainly, “we’re going to spend $500,000 building this thing that we expect to make/save us $1.5 million by next year.”
Maintenance mode, on the other hand, presents the business with a cost center. They’ve now made their investment and (at least partially) realized return on it. The maintenance team just hangs around to prevent backslides. For instance, should maintenance problems crop up, you may lose customers or efficiency.
Plan of a Attack: Build vs Maintain
Because the business regards these activities differently, it will attack them differently. And, while I can’t speak to every conceivable situation, my consulting work has shown me wide variety. So I can speak to general trends.
In green field mode, the business tends to regard the work as an investment. So, while management might dislike overruns and unexpected costs, they will tend to tolerate them more. Commonly, you see a “this will pay off later” mentality.
On the maintenance side of things, you tend to see far less forgiveness. Certainly, all parties forgive unexpected problems a lot less easily. They view all of it as a burden.
This difference in attitude translates to the planning as well. Green field projects justifiably command full time people for the duration of the project. Maintenance mode tends to get you familiar with the curious term “half of a person.” By this, I mean you hear things like “we’re done with the Sigma project, but someone needs to keep the lights on. That’ll be half of Alice.” The business grudgingly allocates part time duty to maintenance tasks.
Why? Well maintenance tends to arise out of reactive scenarios.
Reactive Mode and the Value of Automation
Maintenance mode in software will have some planned activities, particularly if it needs scheduled maintenance. But most maintenance programmers find themselves in a reactive, “wait and see” kind of situation. They have little to do on the project in question until an outage happens, someone discovers a bug, or a customer requests a new feature. Then, they spring into action.
Business folks tend to hate this sort of situation. After all, you need to plan for this stuff, but you might have someone sitting around doing nothing. It is from this fundamental conundrum that “half people” and “quarter people” arise. Maintenance programmers usually have other stuff to juggle along with maintaining “Sigma.”
Because of this double duty, the business doubles down on pressure to minimize maintenance. After all, not only does it create cost, but it takes the people away from other, profit-driven things that they could otherwise do.
So how do we, as programmers, and we, as software shops, best deal with this? We make maintenance as turnkey as possible by automating as much as possible. Oh, and you should automate this stuff during green field time, when management is willing to invest. If you tell them it means less maintenance cost, they’ll probably bite.
Automate the Test Suite
First up for automation candidates, think of the codebase’s test suite. Hopefully, you’ve followed my advice and built this during green field mode. But, if not, it’s never too late to start.
Think of how time consuming a job QA has. If manually running the software and conducting experiments constitutes the entirety of your test strategy, you’ll find yourself hosed at maintenance time. With “half a person” allocated, no one has time for that. Without an automated suite, then, testing falls by the wayside, making your changes to a production system even more risky.
You need to automate a robust test suite that lets you know if you have broken anything. This becomes even more critical when you consider that most maintenance programmers haven’t touched the code they modify in a long time, if ever.
Automate Code Reviews
If I were to pick a one-two punch for code quality, that would involve unit tests and code review. Therefore, just as you should automate your test suite, you should automate your code review as well.
If you think testing goes by the wayside in an under-staffed, cost-center model, you can forget about peer review altogether. During the course of my travels, I’ve rarely seen code review continue into maintenance mode, except in regulated industries.
Automated code review tools exist, and they don’t require even “half a person.” An automated code review tool serves its role without consuming bandwidth. And, it provides maintenance programmers operating in high risk scenarios with a modicum of comfort and safety net.
Automate Production Monitoring
For my last maintenance mode automation tip of the post, I’ll suggest that you automate production monitoring capabilities. This covers a fair bit of ground, so I’ll generalize by saying these include anything that keeps your finger on the pulse of your system’s production behavior.
You have logging, no doubt, but do you monitor the logs? Do you keep track of system outages and system load? If you roll software to production, do you have a system of checks in place to know if something smells fishy?
You want to make the answer to these questions, “yes.” And you want to make the answer “yes” without you needing to go in and manually check. Automate various means of monitoring your production software and providing yourself with alerts. This will reduced maintenance costs across the board.
Automate Anything You Can
I’ve listed some automation examples that come to mind as the most critical, based on my experience. But, really, you should automate anything around the maintenance process that you can.
Now, you might think to yourself, “we’re programmers, we should automate everything.” Well, that subject could make for a whole post in and of itself, but I’ll speak briefly to the distinction. Build mode usually involves creating something from nothing on a large scale. While you can automate the scaffolding around this activity, you’ll struggle to automate a significant amount of the process.
But that ratio gets much better during maintenance time. So the cost center nature of maintenance, combined with the higher possible automation percentage, makes it a rich target. Indeed, I would argue that strategic automation defines the art of maintenance.