By Jim Grey (about)
It’s a common mistake among new testers: test it all, every time. But the weight of all that checking soon crushes the tester, and s/he starts looking for ways to test less without missing anything important.
And so begins the journey of understanding risk likelihood and impact: how likely is a thing to be broken, and how bad is it when it is. Smart testers prioritize likelihood and impact, and test in priority order. That way, should time run out, only low risk and low impact areas of the product remain untested. Heck, you might even skip tests that are ranked low enough. Maybe you should skip those tests, as they’re likely to find bugs nobody cares about. A radical thought!
But how to rank risk and impact? This reminds me of an old joke.
There was an engineer who kept the big machine on the shop floor running faithfully for 30 years. After he retired, the machine promptly broke down. Nobody could get it running again. In desperation, the company called the engineer and implored him to come back and fix it.
The retired engineer returned, albeit reluctantly. He spent a day looking the machine over. Then he called everybody together and marked an X in chalk on a particular component. “Replace this, and the machine will work again.” Glory be, he was right! “Send us an invoice,” the boss said.
And the engineer did: for $10,000. “Ten thousand dollars!” the boss cried. “You need to justify that!” The engineer said he’d send an itemized invoice. Here’s how it read:
One chalk mark: $1
Knowing where to put it: $9,999
Testing for risk and impact means knowing where to put it — that is, knowing where to go to find the most serious bugs. You get good at that by building these three mental models:
Code
What impact on the rest of the software will these code changes have? In other words, what is likely not to work as desired after these code changes are made?
This means you have to learn how is the product is designed and built. That doesn’t mean you necessarily have to be able to read code, although it doesn’t hurt. You just have to pay attention as you test the product and listen to the developers’ explanations of the product’s technical details. You will know you’re building this model when you articulate how you think the product is built to a developer and they say something like, “Yeah. Those aren’t exactly the words I’d use, but they’re accurate enough.”
The code mental model helps you assess risk likelihood. “That part of the product is a little brittle, and every time something interacts with it, things are broken,” or, “I know we designed that function to handle a certain throughput, but what we’re contemplating is 10 times that, and so I’m concerned it’ll fold under the pressure.”
Customer
What parts of the product, when not working as desired, will be a problem for the customer or user? How severe a problem will it be?
To build this model, form good relationships with your support and implementation teams. You might even do rotations through support from time to time, and review customer-reported problems and seek clarity from support on how difficult they were for customers.
The customer mental model helps you assess impact. “If we ship this bug, customers are going to scream,” or, “I think support can talk customers around this bug,” or, “Customers are probably not going to even notice this bug.”
Business
What parts of the product, when not working as desired, put the company’s revenue or reputation at risk, or interrupts smooth and efficient company operations? How severe a problem will it be?
The business mental model gets at how your company makes money and grows the business. This is often the hardest mental model to build, but to the extent you build it, you can make much more nuanced test coverage decisions. To start, you can form an understanding of the kinds of product issues that get customers to call the CEO and threaten to cancel or sue. You can come to understand the kinds of problems that place heavy burden on the support and implementation teams, or would cost the company money in terms of time taken away from revenue-generating activity or services given for free to help regain an angry customer’s trust.
Come to understand which customers, especially the most lucrative ones, are up for renewal soon, and which are unhappy with your company and why.
The business mental model helps you further assess impact. “If this doesn’t perform well, customers are going to quit us,” or, “Bugs in this part of the product always flood us with calls and disrupt our ability to deliver more software.”
Jim Grey 