Category: Testing

Categories
Career Testing

How to hire an entry-level tester

By Jim Grey (about)

Where I live and work, it’s difficult to find experienced developers and testers. Demand outpaces supply. So one strategy I’ve been using to grow my testing team is to hire people who have never tested.

Obviously, that means they need to be taught how to test. It so happens I like doing that! But this means I have to be able to identify non-traditional candidates who are apt to be good students.

I look to my company’s customer-service team for candidates, as those people know the software and understand the customer. But that’s not enough; I need to know whether they can be taught to test.

Faces

Fortunately, I’ve figured out four key traits someone needs to be able to learn to test and to enjoy it. The new tester must have a knack for troubleshooting, which means they are curious and persistent, can create mental models of the system they’re examining, and can develop and investigate theories of why things are broken.

Be curious:  A tester needs to notice when something is wrong and feel compelled to investigate. “Wait, what? Why did the software do that? I’d better dig in and find out.”

Be persistent:  But sometimes the investigations take a while. A tester needs to keep feeling compelled, even when the investigation stretches on. (Bonus points if they recognize when they’ve reached a point of diminishing returns, and choose a different avenue of exploration.) 

Create mental models:  Building a mental model of a system, even if it’s incomplete or partially inaccurate, helps a tester orient themselves to a problem and generate ideas on how to work through it.

Develop a theory of error:  Thinking about how the current problem is creating or exposing a flaw in the system informs the steps taken to prove or disprove the theory — and isolate bugs.

A person who’s not curious won’t troubleshoot. A person who’s not persistent won’t keep going when the troubleshooting gets rough. Troubleshooting doesn’t get very far if you can’t see how the thing you’re troubleshooting is a system and imagine reasons why it’s not working.

So when I interview testers, I ask this question, which gets at all of these things:

Think of a time you had a problem with software or electronic equipment when you tried to solve the problem. It might have been trouble with your home network, some difficulty upgrading your PC or your phone to a new OS, a misbehaving printer, or anything that went wrong with technology you use. Tell me how you assessed the problem and tried to figure out what was wrong, and what steps you took to try to fix the problem.

I ask candidates to tell me two or three stories in answer to this question. Many people can come up with one story, but someone who enjoys, or at least doesn’t mind, getting to the bottom of technology problems will have more than one story.

I’m discouraged when I hear these things in answers:

  • Trying nothing beyond obvious troubleshooting steps, such as rebooting
  • Giving up after a couple steps: “I tried a couple things and then just called support”
  • Trying things randomly in hopes one of them works
  • Fear of trying things because they might break things worse

I’m excited when I hear these things in answers:

  • Developing a theory of how the system works, including applying knowledge from other systems — “I wonder if this thing I’m trying to figure out is like that other thing I know very well”
  • Some logical, systematic method for exploring the problem, generally starting from the simplest, most probable causes and moving stepwise through successively less likely and more complicated possible causes
  • Research, usually via Internet search but also by asking someone who might know
  • Evidence that they learned something about how the system worked, and felt some satisfaction in that

It’s fine if the candidate didn’t successfully resolve the problem. What I’m listening for is whether they systematically eliminate possibilities and move toward finding root cause.

Good answers can be mundane. One candidate told me about sitting down one night to watch Netflix, but his movie kept failing to stream. He was used to experiencing a little of that on Friday and Saturday night; he knew Netflix drew a lot of Internet traffic on those nights. But because it was a weeknight he wondered whether the problem might be within his home. He knew that rebooting the Roku usually solved connectivity problems, but it didn’t work this time. So he went over to his AT&T Uverse modem to check out its blinking lights. “I don’t know much about those lights,” he said, “but I do know that if most of them aren’t blinking, especially the INTERNET light, there’s a problem.” Sure enough, the INTERNET light was out. So he tried rebooting the modem, but the INTERNET light wouldn’t come back on. He found his modem’s manual, but it didn’t cover the situation. He tried a couple Google searches for this condition, but found no useful information. Out of ideas, he called AT&T, which reported intermittent outages in his area.

The candidate’s answer showed that he created a mental model of how Netflix and the Internet work and tried the most likely causes first. I especially liked that he confidently used his incomplete technical knowledge to narrow down the problem. And I didn’t mind it a bit that he ended up calling for help — he’d first done all he reasonably could.

The rest of my interview questions look to gauge technical experience and fit for the company and the team. It’s pretty standard stuff. But without good answers to the troubleshooting question, the candidate goes no further.

So look for these four traits. They give you trainable team members.

Thanks to Rick Grey for his valuable feedback that helped shape this post.

Categories
Quality Testing

Fast failure recovery lets you take more risk and increase speed

By Jim Grey (about)

I can just imagine the “oh no” moment that rippled through Feedly last week when they learned that their big new release for iOS 9 crashed on launch for iOS 7 and 8 users. Their first response: add a hastily-written warning to the App Store description.

IMG_4066

Their second response: fix the bug, fast. It was ready the next day.

IMG_4069

Apparently, Apple has a way for developers to rush the very occasional critical fix into the App Store, sidestepping the normal, weeks-long approval queue.

Apple’s App Store fast track provides a safety net, and I don’t blame them for using it. Because Feedly could recover fast, hardly anybody will remember this gaffe, and it won’t cost the company very much.

I actually feel slightly bad talking about it, because it keeps the memory of this bug alive. But it illustrates such a simple equation: the faster you can recover from failure, the less perfect your product has to be when you ship it, and therefore the less expensive it is to build and support, and the faster your company can move.

I’m not advocating for delivering buggy product on purpose. Follow good development practices and test for important risks to deliver the best software you can as fast as you can. But when you inevitably deliver a bad bug, being set up to recover fast means you can deliver without worry.

I remember when the best way to get software to users was to mail it to them on CDs. A bug of this magnitude was a much bigger deal then. It was harder to warn users of the problem, and lots slower and more expensive to correct it. In that world, Feedly’s bug could have damaged their reputation for a long time. So back then it made sense to test more thoroughly — and therefore spend more time and money before releasing.

But today, in a world of Web software and 24-hour emergency App Store turnaround, you’ll deliver faster and with less expense when you set yourself up for fast failure recovery. Continuous integration and continuous delivery are usually a part of that strategy.

Categories
Quality Testing

You can’t test it all

By Jim Grey (about)

That new build of software you are about to test? It’s a haystack with some unknown number of needles (bugs) in it.

Hay rolls 3
Have fun finding all the needles!

As a tester, you might think your job is to find all the needles. But how do you do that when you don’t know how many needles are in there? What if there are a lot of needles in there? You’ll never have time to find them all.

You need a plan. You want to find the showstopper bugs right away, and then find as many other bugs that people will care about within the time you have. And then when they come breathing down your neck to stop already and ship, you want to be able to tell them just what badness still might lurk in the code. Give them a reason to think it over.

You do that through assessing risk and targeting test coverage. To assess risk, ask yourself some questions:

  • How stable is the code that was changed? What interactions within the software might these changes break? You’re trying to figure out how likely it is you’ll find bugs.
  • If stuff around these code changes is broken, how much could it hurt the user? How much could it hurt your company? You’re trying to figure out the impact of any bugs you might find.

Risk is the product of likelihood and impact. Test for the highest risk bugs first, working down through the risks. Test more deeply for the bigger risks, more lightly for the smaller ones.

Let’s say they want to ship before you’ve tested through the risks you think people will care about. You can then talk about the risks you haven’t checked for yet, and ask if they’re okay with shipping like that. Do you see the mind shift here? You’re not saying you haven’t run all of your tests yet, which sounds an awful lot like you can’t keep up. Instead, you’re saying that the code might not be ready yet, and here are the specific things you’d like to still check for. It puts you in a much stronger position to get that extra time — and makes it the boss’s decision about what to do next.

Ultimately, it’s best if your developers can and will take great care to not deliver so many needles. That’s always the best case. Click here to read more about it.

Categories
Quality Testing

The tester’s three mental models

By Jim Grey (about)

It’s a common mistake among new testers: test it all, every time. But the weight of all that checking soon crushes the tester, and s/he starts looking for ways to test less without missing anything important.

And so begins the journey of understanding risk likelihood and impact: how likely is a thing to be broken, and how bad is it when it is. Smart testers prioritize likelihood and impact, and test in priority order. That way, should time run out, only low risk and low impact areas of the product remain untested. Heck, you might even skip tests that are ranked low enough. Maybe you should skip those tests, as they’re likely to find bugs nobody cares about. A radical thought!

But how to rank risk and impact? This reminds me of an old joke.

Ice cream board
I wish all chalk marks could be about ice cream.

There was an engineer who kept the big machine on the shop floor running faithfully for 30 years. After he retired, the machine promptly broke down. Nobody could get it running again. In desperation, the company called the engineer and implored him to come back and fix it.

The retired engineer returned, albeit reluctantly. He spent a day looking the machine over. Then he called everybody together and marked an X in chalk on a particular component. “Replace this, and the machine will work again.” Glory be, he was right! “Send us an invoice,” the boss said.

And the engineer did: for $10,000. “Ten thousand dollars!” the boss cried. “You need to justify that!” The engineer said he’d send an itemized invoice. Here’s how it read:

One chalk mark: $1

Knowing where to put it: $9,999

Testing for risk and impact means knowing where to put it — that is, knowing where to go to find the most serious bugs. You get good at that by building these three mental models:

Code

What impact on the rest of the software will these code changes have? In other words, what is likely not to work as desired after these code changes are made?

This means you have to learn how is the product is designed and built. That doesn’t mean you necessarily have to be able to read code, although it doesn’t hurt. You just have to pay attention as you test the product and listen to the developers’ explanations of the product’s technical details. You will know you’re building this model when you articulate how you think the product is built to a developer and they say something like, “Yeah. Those aren’t exactly the words I’d use, but they’re accurate enough.”

The code mental model helps you assess risk likelihood. “That part of the product is a little brittle, and every time something interacts with it, things are broken,” or, “I know we designed that function to handle a certain throughput, but what we’re contemplating is 10 times that, and so I’m concerned it’ll fold under the pressure.”

Customer

What parts of the product, when not working as desired, will be a problem for the customer or user? How severe a problem will it be?

To build this model, form good relationships with your support and implementation teams. You might even do rotations through support from time to time, and review customer-reported problems and seek clarity from support on how difficult they were for customers.

The customer mental model helps you assess impact. “If we ship this bug, customers are going to scream,” or, “I think support can talk customers around this bug,” or, “Customers are probably not going to even notice this bug.”

Business

What parts of the product, when not working as desired, put the company’s revenue or reputation at risk, or interrupts smooth and efficient company operations? How severe a problem will it be?

The business mental model gets at how your company makes money and grows the business. This is often the hardest mental model to build, but to the extent you build it, you can make much more nuanced test coverage decisions. To start, you can form an understanding of the kinds of product issues that get customers to call the CEO and threaten to cancel or sue. You can come to understand the kinds of problems that place heavy burden on the support and implementation teams, or would cost the company money in terms of time taken away from revenue-generating activity or services given for free to help regain an angry customer’s trust.

Come to understand which customers, especially the most lucrative ones, are up for renewal soon, and which are unhappy with your company and why.

The business mental model helps you further assess impact. “If this doesn’t perform well, customers are going to quit us,” or, “Bugs in this part of the product always flood us with calls and disrupt our ability to deliver more software.”

Categories
Career Quality Testing

Endangered species: Managers and Directors of Quality Assurance

By Jim Grey (about)

I lost my job a month ago. The software company where I worked cut expenses to fit revenue. That’s business.

I was Director of Quality Assurance. I built and led a team of functional testers, software engineers (writing code-based tests), and technical writers, and developed the strategies and processes the company used to test and deliver its products. It was a role my whole career had prepared me for, and I was glad to have it.

But now I’m working my network, looking for my next gig. It’s been fun. I’ve spoken to a few dozen CTOs, VPs of Engineering, and even CEOs so far in my search, usually from startups and young companies.

I’ve delivered so much software using waterfall and “scrum-fall” (the developers sprint, and testers try to keep up, but final testing still happens after developers are done) at various well-established companies that I thought that full-on agile was a myth, at least here in Indiana.

Rosie
Endangered.

But the startup leaders tell me they’re all in: it’s agile, continuous integration, and continuous delivery all the way. And while all of them have or will soon need testers, they are all hedging on hiring specialized managers for them.

One CTO, a fellow I worked with several years ago, told a typical story: “I have a few scrum teams, each with one tester. But I can’t imagine ever hiring somebody to run QA as I don’t consider it to be a separate function. Testing is part of the scrum team. I’d have to think QA management jobs will eventually dry up everywhere. I’m sorry; I’m sure that’s not welcome news for you.”

When waterfall ruled the world, it made sense to have a QA department with its own leadership hierarchy. After all, the testers truly were a separate team with their own schedules, methodologies, and needs for care and feeding. But in agile, testers feel fully part of their scrum team rather than part of the team of people who test.

I admit to some anxiety over this evolution, entirely because I’m unexpectedly on the job market after several years of being Manager and Director of QA. But I think these changes are good for quality software delivery.

And these changes are creating opportunity, because there are things a good QA leader knows how to do that a CTO or VP of Engineering probably doesn’t. I networked my way into another startup CTO’s office just to introduce myself and ended up walking out with a short-term consulting job. It turns out that he’s ready to hire his first tester, but wants guidance on how testing should integrate into his team and how the tester should approach the work to find the most important bugs early. So I’m testing alongside his developers for now to get a feel for their context, and will develop a strategy and process his team can follow after I’m gone.

In my conversations, over and over CTOs are telling me they need people to coach and guide on functional testing strategy and technique, or on building test automation and performance testing practices. One VP of Engineering I’ve talked to had an unusual take: that I should be able to help him build and manage his end-to-end software delivery process, including engineering and testing, because he sees process as key to predictable and reliable software quality.

I certainly have experience in all of these areas. I imagine many Managers and Directors of QA do.

I don’t want to count my chickens before they hatch, but I feel pretty good that at least one of these threads will lead to a full-time job soon. And with it, I’ll step out of the waterfall/scrum-fall past and into a present I previously didn’t know existed.

Categories
Quality Testing

Myths of test automation – debunked!

By Jim Grey (about)

I wrote a post last year criticizing test automation when it’s used to cover for piles of technical debt and poor development practices. But I still think there’s a place for automation in post-development testing. There are two keys to using it well: knowing what it’s good at, and counting the costs. Without those keys it’s easy to fall prey to several myths of test automation. I aim to debunk them here.

Myth: Automation is cheap and easy

It is seductive to think that just by recording your manual tests you can build a comprehensive regression-test suite. But it never seems to really work that way. Every time I’ve used record and playback, the resulting scripts wouldn’t perfectly execute the test, and I’ve had to write custom code to make it work.

St. Paul's Episcopal Church

What I’ve found is that it takes 3 to 10 times longer to automate one test than to execute it manually. And then, especially for automation that exercises the UI, the tests can be brittle: you have to keep modifying scripts to keep them running as the system under test changes.

I’ve done straight record and playback. I’ve created automated modules that can be arranged into specific checks. I’ve led a team that created tests on a keyword-driven framework. And I currently lead a team that writes code that directly exercises a product’s API. The amount of maintenance has decreased with each successive approach.

A side note: given the cost of automating one test, can you see that you want to automate only what you are going to run over and over again, because otherwise the investment doesn’t pay?

Myth: Automation can test anything, and is as good as human testing

Automation is really good at repeating sets of actions, performing calculations, iterating over many data sets, addressing APIs, and doing database reads and writes. I love to automate these things, because humans executing them over and over is a waste of their potential.

This gets at a whole philosophical discussion about what testing is. I think that running predetermined scripts, whether automated or not, is just checking, as in, “Let me check whether clicking Save actually saves the record.” This subset of testing just evaluates the software based on predefined criteria that were determined in the past, presumably based on the state of the software and/or its specification or set of user stories as they were then.

The rest of testing involves human testers experimenting and learning, evaluating the software in its context now. This is critical work if for no other reason than the software and its context (environment, hardware, related software, customer needs, business needs, and so on) changes. An exploring human can find critical problems that no automated test can.

I want human testers to be free to test creatively and deeply. I love automated checks because they take this boring, repetitive work away from humans so they have more time to explore.

Myth: When the automation passes, you can ship!

It’s seductive to think that if testing is automated, that passing automation is some sort of Seal of Approval that takes out all the risk. It’s as if “tested” is a final destination, an assurance that all bets are covered, a promise that nothing will go wrong with the software.

But automation is only as good as its coverage. And if nobody outside your automation team understands what the automation covers, saying “the automation passed” has no fixed meaning.

It’s hard to overcome this myth, but to the extent I have, it’s because as an automation lead and manager I’ve required engineers to write detailed coverage statements into each test. I’ve then aggregated them into broad, brief coverage statements over all of the parts of the software under test. Then I’ve shared that information — sometimes in meetings with PowerPoint decks, always in a central repository that others can access and to which I can link in an email when I inevitably need to explain why passing automation isn’t enough. Keeping this myth at bay takes constant upkeep and frequent reminders.

Myth: Automation is always ready to go

Hope Rescue Mission

“Hey, we want to upgrade to the next version of the database in the sandbox environment. Can you run the automation against that and see what happens?”

My answer: “Let’s assume I can even run the automation in sandbox. If it passes, what do you think you will know about the software?” The answer almost always involves feelings: “Well, I’ll feel like things are basically okay.” See “When the automation passes, you can ship!” above.

Automation is software, full of tradeoffs aimed at meeting a set of implicit and explicit goals. Unless one of those goals was “must be able to run against any environment,” it probably won’t run in sandbox. The automation might count on particular test data existing (or not existing). It might not clean up after itself, leaving lots of data behind, and that might not be welcome in the target environment. It might depend on a particular configuration of the product and its environment that isn’t present.

Even in the environment the automation usually runs in, it might not be ready to go at a moment’s notice. Another goal would need to be, “must be able to run at any time.” There are often setup tasks to perform before the automation can run: a reset of the database the automation uses, or the execution of scripts that seed data that the automation needs.

Myth: Just running the automation is enough

When I run automated tests, part of me secretly hopes they all pass. That’s because when there’s a failure, I have to comb through the automation logs to find what happened, figure out what the automation was doing when it failed, and log into the software myself and try to recreate the problem manually. Sometimes the automation finds just the tip of a bug iceberg and I spend hours exploring to fully understand the problem. Some portion of the time, the failure is a bug in the automation that must be fixed. When it’s a legitimate product bug, then I have to write the bug in the bug tracker.

I am endlessly amused by how often I’ve had to explain that just running the automation isn’t the end of it: that if there are any failures, the automation doesn’t automatically generate bug reports. The standard response is some variation of “What? …ohhhhhh,” as it dawns on them. So far, thankfully, it has always dawned on them.

Myth: Automated tests can make up for years of bad development practices

I’ve just got to restate my point from my older post on this subject. If your development team doesn’t follow good practices such as writing lots of automated unit tests (to achieve about 80% code coverage), code reviews, paired testing, or test-driven development, automation from QA is not going to fix it. You can’t test in quality — you have to build it in.

If you’re sitting on a messy legacy codebase, one where your test team plays whack-a-mole with bugs every time you make changes to it, you are far, far better served investing in the code itself. Refactor, and write piles of automated unit tests.

You want on the order of magnitude of thousands of automated unit tests, hundreds of automated business-rule tests (which hopefully directly exercise an API, rather than exercising a UI, for resiliency and maintainability), and tens of automated checks to make sure the UI is functioning.

I’ll belabor this point: Invest in better code and better development practices first. When you deliver better quality to QA, you’ll keep the cost of testing as low as possible and more easily and reliably deliver better quality to your customers and users.

Categories
Quality Testing

When test automation is nothing more than turdpolishing

By Jim Grey (about)

I used to think that writing a fat suite of automated regression tests was the way to hold the line on software quality release over release. But after 12 years of pursuing that goal at various companies, I’ve given up. It was always doomed to fail.

In part, it’s because I’ve always had to automate tests through a UI. When I did straight record-and-playback automation, the tests were enormously fragile. Even when I designed the tests as reusable modules, and even when I worked with a keyword-driven framework, the tests were still pretty fragile. My automation teams always ended up spending more time maintaining the test suite than building new tests. It’s tedious and expensive to keep UI-level test automation running.

http://xkcd.com/844/
http://xkcd.com/844/

But the bigger reason is that I’ve made a fundamental shift in how I think about software quality. Namely, you can’t test in quality – you have to build it in. Once code reaches the test team, it’s garbage in, garbage out. The test team can’t polish a turd.

Writing an enormous pile of automated tests through the UI? Turdpolishing.

I’ve worked in some places where turdpolishing was the best that could be done. Company leadership couldn’t bear the thought of spending the time and money necessary to pay down years of technical debt, and hoped that building out a big pile of automated tests would hold the line on quality well enough. I’ve led the effort at a couple companies to do just that. We never developed the breadth and depth of coverage necessary to prevent every critical bug from reaching customers, but the automation did find some bugs and that made company leadership feel better. So I guess the automation had some value.

But if you want to deliver real value, you have to improve the quality of the code that reaches your test team. Even if the software you’re building is sitting on a mountain of technical debt, better new code can be delivered to the test team starting today. I’m a big believer in unit testing. If your software development team writes meaningful unit tests for all new code that cover 60, 70, 80 percent of the code, you will see initial code quality skyrocket. Other practices such as continuous integration, pair programming, test-driven development, and even good old code reviews can really help, too.

But whatever you do, don’t expect your software test team to be a magic filter through which working software passes. You will always be disappointed.

Categories
Quality Testing

Giving testers less to do

By Jim Grey (about)

I hear legends of companies who hire nothing but programmers in their test departments, and rely almost entirely on code-based tests in their software development methodologies. I’ve never seen such a shop in person. Out here in the Midwest, most testing involves humans directly exercising user interfaces.

Eric Jacobson recently wondered aloud on his blog whether testers are simply too busy finding bugs through the UI to move into more technical or programmatic testing. My typical experience has been that Development doesn’t deliver software to QA that is solid enough that testers didn’t have to spend the bulk of their time making sure the UI and the immediate interface with the database are working.

For years my schtick was to join a company when it is transitioning from small to mid-sized, when it was feeling crushed by quality problems caused by mounting technical and defect debt. They had focused on getting to market fast but had grown a tangled mess of code. My response was always to grow the QA team, primarily by hiring automation engineers to build out large automated regression suites.

After doing that at a couple companies, I lost interest in the strategy. It was expensive, it took too much time, and it never moved the quality needle enough. It just seems absurd to me now to prop up years of thin development practices with more post-development testing, especially given that automated tests in QA generally work through the UI and are therefore brittle and slow.

flood
The view from my deck after a particularly heavy rain. You’d better believe the sump pump was running.

It’s like when my home’s crawl space used to flood after each heavy rain. A company that specialized in drying out crawl spaces recommended $6,000 in a French drain, multiple sump pumps, and encapsulation to move the water out and keep the moisture from seeping up into the house. But a buddy of mine who builds houses said, “You’ve got a negative grade around your foundation. Buy $300 in topsoil and a couple cases of beer. Invite all your friends over and issue them shovels. Fix the grading and you’ll keep the water from getting in.” I went with the topsoil and the friends. I also put in one sump pump, just in case. It runs pretty much only when the rain is torrential.

In case my admittedly imperfect metaphor isn’t obvious, the graded topsoil is the unit testing, and the sump pump is the lightweight QA automation solution. Let’s try preventing the bugs from getting in as much as we can, shall we? But let’s still check for the odd and extreme cases that are bound to get by.

This is the hierarchy of testing similar to the one Mike Kelly recommends in this blog post. (He’s building on the work of Brian Marick, by the way, who gives a framework for testing in this blog post.) Mike recommends building on the order of thousands of automated unit and component tests, hundreds of automated business-logic tests, and tens of UI-level automated tests. The unit tests run fast and frequently. The inherently slow UI automated tests run far less often.

That’s what I resolved to try after I lost my will to build huge automated regression suites in QA. I deliberately took a QA leadership role with a company transitioning out of its startup phase. The product wasn’t yet too large and too saddled with technical and defect debt; I felt like we could make up the lost ground. After some encouragement from me, the fellow who ran engineering began insisting that developers write automated unit tests for all new code. We started building each release into an environment where developers could perform rudimentary testing on it themselves with realistic data. Their goal was to make sure all the happy paths work, so that when my testers get in there they are not immediately stymied by obvious critical bugs.

Before we started to make this transition, I quietly started tracking a simple little metric. I counted the defects QA found, plus the defects created in the release that were found in production, and divided by the number of development hours in the release. The metric was a little mushy because I was working with estimated and not actual hours, and because I was having to make judgment calls about which defects in production were caused by the release and which were latent bugs. But it’s hard to ignore the order-of-magnitude improvement we got on this metric. We were tracking to about .3 defects per development hour in the couple releases before we made these changes, and within two releases we dropped to about .05-.09 defects per development hour and held steady.

This had incredible impact. Initial quality went way up in QA, meaning initial quality went way up in production. Just adding these two steps was like flipping a switch not only on many of the quality challenges we faced, but also on the amount of chaos and churn we experienced as an overall engineering team.

A side benefit was that developers seemed happier. It wasn’t that writing tests made them happy – it didn’t. They would rather have built more new stuff. But delivering better code into QA meant that they spent less time in the fix-test cycle and were interrupted by way fewer production crises. They told me that finally they could focus.

The reason why I titled this post as I did – and it’s meant to be tongue in cheek, by the way – is because my strategy means hiring more developers and fewer testers. But the benefit to testers is that they get to do far more interesting work, going deeper, thinking more creatively, and exploring more technical kinds of testing.

I can’t imagine ever moving to an all-code testing strategy. All automated testing can do is repeat series of actions. Skilled human testers can cope with complexity and adapt to change, gain and synthesize knowledge and apply it to their testing, know from experience where the product is likely to be broken, and explore the system creatively. The kinds of products I’ve always delivered and am likely to keep delivering will always need that.