technical debt – Jim Grey

If you want your software to keep producing, be prepared to do some dirty work

By Jim Grey (about)

A woman named Verna built the house I live in. She landscaped it nicely; a sprawling flowerbed stretches in front of my front door and picture windows. Every spring, I eagerly await Verna’s spring color: yellow daffodils, purple hyacinths, red tulips, and finally the giant pink peonies.

I’ve added a few things: lilies, mums, lavender, coreopsis, phlox. I love phlox! But my eagerness to keep adding color petered out pretty quickly because it turns out I hate digging in the dirt.

I don’t much enjoy any of the other routine garden maintenance, either. Mulching. Deadheading. Dividing overgrown plants. Weeding – oh god, the weeding. Does it make me lazy that I just spray my weeds with Roundup and move on?

I just want to enjoy the flowers. But this ninth spring I’ve lived in my home, a few of my plants didn’t come back as strong. A couple didn’t come back at all.

So I asked my mom. She’s the gardener in our family. “When was the last time you fertilized?” she said. “Um, never,” I said. “Ah,” she said.

It turns out that you can’t just ignore the soil, or the plants themselves for that matter. Things growing in it year after year uses up all the nutrients, and crowded plants compete with each other for what little is there. “I’m surprised your flowers didn’t stop coming back a few years ago,” Mom said.

I did some serious fertilizing this season. I also separated some overgrown hostas and moved some of Verna’s plants so they had some elbow room. Not fun, but necessary.

♦ ♦ ♦

I once worked for a software company whose flagship product sold briskly. Version 1.0 was five years in the past, and since then we’d added lots of new features so the product could continue to lead the market. And now here came the head of Product Management asking for more new features

Dan, a quiet fellow, graying at the temples, led Development. “Well, yes, we can add all those features,” Dan said, adjusting his glasses. “This one will take six months. That one will take four. This other one, well, I think that’ll take a year.”

The Product Manager was dumbfounded. “Features of similar scope took far less time in the past, and you had fewer developers then. What gives?”

Dan looked up at the Product Manager kindly, and drew a breath. “Well, we’ve been under such pressure to quickly add features to this product that we’ve not focused enough on its overall design. We’ve also made no time to keep our underlying architecture up to date. These are things I’ve been pointing out all along the way. But we’ve just bolted features on wherever we thought we could get away with it. Now, to add any one of the features you’ve requested, we basically have to unbolt three or four other features, and blend the code all together. And we have to write complicated bridge code to do modern things with our aging architecture, and when that doesn’t work we will have to upgrade some parts of it and test the product well to make sure everything still works. It’s a slow process. And it’s just going to get slower and slower the longer we keep going like this.”

That the product’s design had become cancerous and the underlying architecture had gone out of date were not considered a crisis –- but not being able to rapidly add new features sure was. It focused the company’s entire attention. Their response was to code up a “next generation” product from scratch, which was a disastrous idea for a whole bunch of reasons beyond the point of this story. When the dot-com bubble burst in 2001-2002, they had not yet successfully launched the next-generation product, and they still couldn’t add features to the old product fast enough. Revenue fell precipitously. Quarterly layoffs began, but it was not enough to keep the wolves from the door. That once-promising company was sold; the company that bought it outsourced development to China.

More recently I went to work for another promising software company. They had been in business for about a decade and had sold their software to a number of very large companies. But in the couple years before I’d been hired, the pace of new feature delivery had slowed to a crawl. Adding new features had become increasingly difficult and always broke existing features. As a result, it took longer and longer to test the product, but even then, major bugs were still being delivered to customers. Meanwhile, younger, more nimble competitors were stealing business away from us. As the rate of new revenue decreased, support costs skyrocketed. It was unsustainable, and that company, too, had to sell itself to another company to avoid collapse.

It was much the same story: the company had focused entirely on rapid new-feature delivery and not enough on ongoing design and architecture. After a decade, their soil had gone infertile and the code had become tangled. Nothing new would grow.

♦ ♦ ♦

Software as a garden: to be able to grow more software, to be able to grow revenue with it, you have to keep the soil fertile and give the roots room. The problem is, gardening projects are a hard sell. These are things like refactoring older parts of the code that no longer serve efficiently, or upgrading or replacing outdated parts of the architecture, or redesigning subsystems that work fine today but can’t adapt to things the company wants to do in the future. When you tell executives you need to do these things, what they hear is that they can’t have new features while you do it. New features fuel growing companies.

But if you don’t tend your garden, sooner or later it will stop producing.

Giving testers less to do

By Jim Grey (about)

I hear legends of companies who hire nothing but programmers in their test departments, and rely almost entirely on code-based tests in their software development methodologies. I’ve never seen such a shop in person. Out here in the Midwest, most testing involves humans directly exercising user interfaces.

Eric Jacobson recently wondered aloud on his blog whether testers are simply too busy finding bugs through the UI to move into more technical or programmatic testing. My typical experience has been that Development doesn’t deliver software to QA that is solid enough that testers didn’t have to spend the bulk of their time making sure the UI and the immediate interface with the database are working.

For years my schtick was to join a company when it is transitioning from small to mid-sized, when it was feeling crushed by quality problems caused by mounting technical and defect debt. They had focused on getting to market fast but had grown a tangled mess of code. My response was always to grow the QA team, primarily by hiring automation engineers to build out large automated regression suites.

After doing that at a couple companies, I lost interest in the strategy. It was expensive, it took too much time, and it never moved the quality needle enough. It just seems absurd to me now to prop up years of thin development practices with more post-development testing, especially given that automated tests in QA generally work through the UI and are therefore brittle and slow.

It’s like when my home’s crawl space used to flood after each heavy rain. A company that specialized in drying out crawl spaces recommended $6,000 in a French drain, multiple sump pumps, and encapsulation to move the water out and keep the moisture from seeping up into the house. But a buddy of mine who builds houses said, “You’ve got a negative grade around your foundation. Buy $300 in topsoil and a couple cases of beer. Invite all your friends over and issue them shovels. Fix the grading and you’ll keep the water from getting in.” I went with the topsoil and the friends. I also put in one sump pump, just in case. It runs pretty much only when the rain is torrential.

In case my admittedly imperfect metaphor isn’t obvious, the graded topsoil is the unit testing, and the sump pump is the lightweight QA automation solution. Let’s try preventing the bugs from getting in as much as we can, shall we? But let’s still check for the odd and extreme cases that are bound to get by.

This is the hierarchy of testing similar to the one Mike Kelly recommends in this blog post. (He’s building on the work of Brian Marick, by the way, who gives a framework for testing in this blog post.) Mike recommends building on the order of thousands of automated unit and component tests, hundreds of automated business-logic tests, and tens of UI-level automated tests. The unit tests run fast and frequently. The inherently slow UI automated tests run far less often.

That’s what I resolved to try after I lost my will to build huge automated regression suites in QA. I deliberately took a QA leadership role with a company transitioning out of its startup phase. The product wasn’t yet too large and too saddled with technical and defect debt; I felt like we could make up the lost ground. After some encouragement from me, the fellow who ran engineering began insisting that developers write automated unit tests for all new code. We started building each release into an environment where developers could perform rudimentary testing on it themselves with realistic data. Their goal was to make sure all the happy paths work, so that when my testers get in there they are not immediately stymied by obvious critical bugs.

Before we started to make this transition, I quietly started tracking a simple little metric. I counted the defects QA found, plus the defects created in the release that were found in production, and divided by the number of development hours in the release. The metric was a little mushy because I was working with estimated and not actual hours, and because I was having to make judgment calls about which defects in production were caused by the release and which were latent bugs. But it’s hard to ignore the order-of-magnitude improvement we got on this metric. We were tracking to about .3 defects per development hour in the couple releases before we made these changes, and within two releases we dropped to about .05-.09 defects per development hour and held steady.

This had incredible impact. Initial quality went way up in QA, meaning initial quality went way up in production. Just adding these two steps was like flipping a switch not only on many of the quality challenges we faced, but also on the amount of chaos and churn we experienced as an overall engineering team.

A side benefit was that developers seemed happier. It wasn’t that writing tests made them happy – it didn’t. They would rather have built more new stuff. But delivering better code into QA meant that they spent less time in the fix-test cycle and were interrupted by way fewer production crises. They told me that finally they could focus.

The reason why I titled this post as I did – and it’s meant to be tongue in cheek, by the way – is because my strategy means hiring more developers and fewer testers. But the benefit to testers is that they get to do far more interesting work, going deeper, thinking more creatively, and exploring more technical kinds of testing.

I can’t imagine ever moving to an all-code testing strategy. All automated testing can do is repeat series of actions. Skilled human testers can cope with complexity and adapt to change, gain and synthesize knowledge and apply it to their testing, know from experience where the product is likely to be broken, and explore the system creatively. The kinds of products I’ve always delivered and am likely to keep delivering will always need that.