Maintenance is a necessary evil. You're never going to get the product perfect. So you'll always be called upon to fix bugs. Typical large I.T. shops spend an extraordinary amount of effort on "maintenance." Estimates range from 10% of labor budget to 70%. In the past our maintenance budgets have been in the 50% range. That seemed obscene to me so we checked into the actual work being done and learned a lot as we've struggled to reduce the amount of time and money we spend on maintenance. When thinking through how to reduce maintenance expenditure, I recommend consideration of the following:
Use great prototypes to narrow the "gap of misunderstanding" between you and your customer regarding scope before you start development. The practice we're trying hard to implement is having Interaction Designers 6-8 weeks ahead of development before ever entering a Cycle/Milestone/Sprint/Release (or whatever you want to call it). So our "agility" comes from working back and forth with the customer on high fidelity prototypes. This substantially reduces the number of times a customer comes back after you're already done, asking for some new feature they thought "was going to be included from the very beginning."
Test, test, test! Don't under-invest in your QA team or in automation. We made the mistake of under-investing for a long time and felt the pain. We've staffed a high quality QA team with many engineers who could easily be developers in our shop. We find many, many bugs before our customers do. We can do much better, but the improvements in QA have materially decreased the amount we spend on maintenance.
Think Quality. As good as your QA team is you can't make them responsible for quality. They're responsible for offering information. They're not responsible for quality. Your project engineers (developer, infrastructure, networking, desktop, database, etc.) are responsible for quality. Create that culture.
Seperate enhancements from bug fixing. We found, as with most companies, that much of the work being billed as "maintenance" wasn't maintenance at all, but actually "enhancements." I think I may have heard an audible, commiserative sigh. You developers know precisely what I'm talking about:
"Well, sure it's a bug! You promised me the ability to report and this is a report I need!"
"It's only a few hours, huh? I don't want to go back for approval. Just do it this once, wontcha?"
"Um. Hey Fred. Uh... Listen... I know you're working on a new project, but I was in the neighborhood and happened to have these tickets to the playoffs..."
"Agile development," though waking up and maturing to the realities of the enterprise, pushes the developer into the position of working continually to please the customer. It's hard to draw the line between bugs and enhancements. But drawing the line is critical. Which leads us to...
Implement good governance. Customers should not "shoulder tap" developers to fix bugs OR to make enhancements. Under any circumstance. Ever. Some kind of project manager, program manager, development manager, QA manager or otherwise dispassionate party should create a buffer between the customer (who has genuine needs) and the developer (who, bless his heart, only wants to serve the customer) to make sure that each bug listed is truly a bug. I've had more than one conversation with an executive who, when I mentioned that many enhancements were slipping through into maintenance, asked me why I was allowing his people to do that without his authorization. Learn to say no--gently. Help the customer understand why enhancements are different than bugs and why they should be handled differently. Similary, help the customer understand when it is or is not ok to choose NOT to fix a bug. Not every bug needs to be fixed!
Simplify bureaucracy. In our shop customers have historically tried to stuff enhancements through maintenance because the bureaucracy to do a new set of enhancements was so painful that they couldn't bear to even think about going through the process. We can't do that to our customers! Simplify bureaucracy as much as possible so you become a joy to work with. We have a long way to go on this score.
Allow (Force) customers to prioritize. I.T. should not be saying no. You should give the customer(s) information about what resources you have available, and they should (collectively) make decisions about what to do with the resource. You can cajole, influence, or merely suggest, but the customers should be making these calls. This year, we allowed our customers to re-direct maintenance budgets to new projects. In other words, for every hour saved on maintenance we are allowing that hour to be spent on new project work. Our maintenance expenditures continue to drop!!
How are you dealing with maintenance in your shops?
Change Control Policies:
ReplyDeleteMight I add to the “Implement Good Governance” and “Allow (Force) customers to priorities”, that a strong change control policy, driven by the customer, will assist in the reduction of the “Maintenance Monkey”. I have found that customers often do not understand what they are asking for and that if you drive them to a change control board change requests that where once considered critical the customer will not even make it to a change request, if they are forced to defend the request to the board. The change control board will also reduce the number of request as they begin to see the impact and risk assessments of change requests and realize the true scope of what they are asking for.
I think IT shops get into a habit of not giving the customer enough credit for the ability to understand. This mistake is due to OUR faults in speaking in tech languages rather than in business languages. I agree that a good project manger, program manger, etc. is vital and should be on the CCB in an advisory position, but the CCB voting members should be customers with decision making authority.
[Joel: Couldn't agree more. Thank you!]
Excellent suggestion. Given your customer base at the LDS church. How do you gently say "no" to your general authority customers?
ReplyDelete[Joel: General authorities are reasonable and understanding. Admittedly, people sometimes tiptoe around them when it's not necessary, but that's due to a respect for the office. General Authorities, like executives, desire and expect people to push back appropriately, to deliver bad news when needed and to offer accurate, timely information.]
Great practices and reminders. Some of us are not blessed with large IT shops though and must fill the role of project manager, QA, and developer all at the same time. We have 3 in our small shop who's customer base is both the company we work for (about 200 some employees) and the customers of the company (in this particular case the company is a government entity).
ReplyDeleteThese circumstances may make a great excuse to not follow many of the procedures and processes you've mentioned here, but I feel that it makes them even more dire. It just means that we have to make sure we fill those roles and make sure they are individually and separately managed even though they are the same person fulfilling them.
It is so much more necessary that we prototype and clarify in the design phase because of limitations on our time once that phase is complete. It is so important that each step does not get overlooked just because the people performing them are the same people.
Maintenance still needs to happen, but hopefully it can be reduced by making sure that much of it is prevented along the way. As more "products" are produced for the customers that means less time for producing more products and more time for maintaining each new product. It is in our best interest to make sure that the "maintenance creep" is as slow as possible if we don't want to become overwhelmed in a tidal wave of bug fixes and enhancement requests.
We maintain about 125-150 web sites with a team of 14. We average about 70 requests per week--most of which are small. We also had the problem with people “shoulder tapping” the developers office as you put it. It seemed like nothing was being completed.
ReplyDeleteWe've since changed things around and have two strong project managers who filter all requests. All request are submitted through our intranet and come to the project managers as enhancements. They then decide if the project is a bug or an enhancement or if it is a large request that needs a more formal project process. We still get people calling the project managers directly with urgent issues but we have successfully buffered the developers and our productivity has dramatically increased.
I have always viewed this as part of the process. We actually utilize two systems for tracking. One system is for features and enhancements and the other is for defects and maintenance. What we are discovering is that it is extremely important to give the proper attention to both systems. We have noticed that users are becoming more and more aware of IT habits and our response process. If we procrastinate or don't give proper attention to one list we begin to notice submissions with a "creative" description to make the other list that is getting our attention. I think that overall there isn't quite the gap that separated "techies" from what was once considered casual users.
ReplyDeleteI think personal technologies have created a population that is very aware of how to get results from their staff.
At the same time though, it has also pushed IT staffers to move a little more quickly on projects before a "knowledgeable" end user comes along to circumvent your processes, create a "work around" or "solve" the problem themselves. When these situations arise, developers begin to lose control over the proper function their products and may end up never getting it back. I have seen many forums that are riddled with submission that say, "I was having problem (A) and I found a site that said to do (B) and now the application doesn't work at all."
I have to admit, I think I am guilty of that too.
It literally warms my heart to see that these issues that are huge problems in almost every IT organization around the globe, are being addressed within the Church. This problem is so common that it would have been naive to assume that it did not occur or simply look past it. I congratulate you.
ReplyDeleteWith all the enormous projects you handle (NewFamilySearch, for example) I just have to comment on the support section of your division. As a non-techie person at a stake Family History Center who has needed support more than once, I am greatly appreciative of the telephone help and the people working there.
ReplyDeleteSpeaking of "Think Quality," I was highly surprised to find that in your own .../about-joel page here on your webpage, you are not following the Church's style guide to capitalize the leading "T" in The Church of Jesus Christ of Latter-day Saints.
ReplyDelete[Joel: Nice catch. The beauty of having a community to help out!! We'll get it fixed. Thank you!]
I worked at Church Headquarters from 1983-85 during the COBOL days. In graduate school and another programming job before working for the Church, I learned that maintenance was far more expense than new development. I believe they quoted 70% back in those days as well.
ReplyDeleteAnd observing the number of bugs in some of the programming products from Church HQ within the last 5 years, I'd say that CHQ abandoned the rigorous review process it used in the 80's before putting new programs into production.
It's ashamed we forget lessons learned and have to relearn them.