automation – Jim Grey

By Jim Grey (about)

I enjoy the Signal v. Noise blog from Basecamp. Recently they published a post about the human touch in customer service. Read it here. Its thesis, in short: automating any part of support is akin to saying the company doesn’t care.

I am charmed by the work ethos Signal v. Noise espouses: work at a sustainable pace on things that are interesting to you, and get plenty of rest. It’s very appealing. But sometimes I think they take the contrarian position as if it’s their brand, and this is one of those times.

There’s nothing wrong with a good robot providing routine customer service. As long as the experience is good, and can seamlessly hand off to a human when things go off script, it’s a win.

That’s because so much of customer service work can be rote and repetitive. One of my career stops was at a company called MOBI, which provided mobile-device management services to large companies. One customer was a global manufacturer that had something like 10,000 mobile phones deployed to its executives. MOBI helped them manage cost by keeping each phone on the least-expensive plan for its typical usage, and also provided a help desk with the promise of answering a request in 30 seconds and staying on the case until it was solved. (Try getting that from your carrier.)

But it was too expensive for MOBI to hire a bunch of new customer-service reps every time they took on a new customer. It cut the profit margin too thin.

Moreover, a large percentage of the customer-service work was simply boring and repetitive. After you’ve done a handful of password resets, equipment orders, and plan changes, you can do them in your sleep.

MOBI has devised a number of robots (“Mobots,” they call them) to take away the drudgery.

If a MOBI customer user jumps on a chat, Mobot Audrey answers. She can handle many of the common, repetitive tasks. The minute the customer asks something she can’t handle, she seamlessly transfers the chat to a human, who handles the case from there.

I had lunch a few weeks ago with MOBI’s CTO, my former boss, who told me that the customer-service team was apprehensive at first, but is happy now because they are working on things that tap into their deep knowledge of the product, and that require them to solve problems creatively. As a result, they are more engaged with their work.

This resonates with me from another perspective. I was on the team that built the call-center software that Medicare customer-service reps use nationwide. I led the test team. Our contract required us to do a “full” regression test on every release — meaning we had to run thousands of written test cases. It took almost 24 person-weeks to execute those tests! This was about 15 years ago when automation tools weren’t great. But I hired a couple engineers to automate those tests anyway, and they finished after a few months. It took just 0.6 person-weeks to execute those tests forever after. My functional testers were free to spend more time performing targeted, complex, and exploratory tests, using their experience to find critical defects the automation couldn’t find. It was much more interesting and meaningful work for them.

But back to customer-service robots: we all remember how bad the early ones were. That Medicare customer-service center had an early voice-response robot for callers, and it was awful. It didn’t understand callers right most of the time, it took a long time to navigate, and once navigated the caller found that the robot couldn’t help them and they needed to speak to a human anyway. Callers hated it.

But the technologies have begun to mature. Some of them claim to be able to learn, even, although I’m skeptical that it’s true, full-on AI or ML. At least it’s possible to create a good experience now. MOBI’s Mobots have done it; their user testing bears it out: most users have no idea that they are interacting with a Mobot.

So why wouldn’t a company like MOBI manage support costs and increase employee happiness by automating away the boring, repetitive tasks? I’m for it.

By Jim Grey (about)

I wrote a post last year criticizing test automation when it’s used to cover for piles of technical debt and poor development practices. But I still think there’s a place for automation in post-development testing. There are two keys to using it well: knowing what it’s good at, and counting the costs. Without those keys it’s easy to fall prey to several myths of test automation. I aim to debunk them here.

Myth: Automation is cheap and easy

It is seductive to think that just by recording your manual tests you can build a comprehensive regression-test suite. But it never seems to really work that way. Every time I’ve used record and playback, the resulting scripts wouldn’t perfectly execute the test, and I’ve had to write custom code to make it work.

What I’ve found is that it takes 3 to 10 times longer to automate one test than to execute it manually. And then, especially for automation that exercises the UI, the tests can be brittle: you have to keep modifying scripts to keep them running as the system under test changes.

I’ve done straight record and playback. I’ve created automated modules that can be arranged into specific checks. I’ve led a team that created tests on a keyword-driven framework. And I currently lead a team that writes code that directly exercises a product’s API. The amount of maintenance has decreased with each successive approach.

A side note: given the cost of automating one test, can you see that you want to automate only what you are going to run over and over again, because otherwise the investment doesn’t pay?

Myth: Automation can test anything, and is as good as human testing

Automation is really good at repeating sets of actions, performing calculations, iterating over many data sets, addressing APIs, and doing database reads and writes. I love to automate these things, because humans executing them over and over is a waste of their potential.

This gets at a whole philosophical discussion about what testing is. I think that running predetermined scripts, whether automated or not, is just checking, as in, “Let me check whether clicking Save actually saves the record.” This subset of testing just evaluates the software based on predefined criteria that were determined in the past, presumably based on the state of the software and/or its specification or set of user stories as they were then.

The rest of testing involves human testers experimenting and learning, evaluating the software in its context now. This is critical work if for no other reason than the software and its context (environment, hardware, related software, customer needs, business needs, and so on) changes. An exploring human can find critical problems that no automated test can.

I want human testers to be free to test creatively and deeply. I love automated checks because they take this boring, repetitive work away from humans so they have more time to explore.

Myth: When the automation passes, you can ship!

It’s seductive to think that if testing is automated, that passing automation is some sort of Seal of Approval that takes out all the risk. It’s as if “tested” is a final destination, an assurance that all bets are covered, a promise that nothing will go wrong with the software.

But automation is only as good as its coverage. And if nobody outside your automation team understands what the automation covers, saying “the automation passed” has no fixed meaning.

It’s hard to overcome this myth, but to the extent I have, it’s because as an automation lead and manager I’ve required engineers to write detailed coverage statements into each test. I’ve then aggregated them into broad, brief coverage statements over all of the parts of the software under test. Then I’ve shared that information — sometimes in meetings with PowerPoint decks, always in a central repository that others can access and to which I can link in an email when I inevitably need to explain why passing automation isn’t enough. Keeping this myth at bay takes constant upkeep and frequent reminders.

Myth: Automation is always ready to go

“Hey, we want to upgrade to the next version of the database in the sandbox environment. Can you run the automation against that and see what happens?”

My answer: “Let’s assume I can even run the automation in sandbox. If it passes, what do you think you will know about the software?” The answer almost always involves feelings: “Well, I’ll feel like things are basically okay.” See “When the automation passes, you can ship!” above.

Automation is software, full of tradeoffs aimed at meeting a set of implicit and explicit goals. Unless one of those goals was “must be able to run against any environment,” it probably won’t run in sandbox. The automation might count on particular test data existing (or not existing). It might not clean up after itself, leaving lots of data behind, and that might not be welcome in the target environment. It might depend on a particular configuration of the product and its environment that isn’t present.

Even in the environment the automation usually runs in, it might not be ready to go at a moment’s notice. Another goal would need to be, “must be able to run at any time.” There are often setup tasks to perform before the automation can run: a reset of the database the automation uses, or the execution of scripts that seed data that the automation needs.

Myth: Just running the automation is enough

When I run automated tests, part of me secretly hopes they all pass. That’s because when there’s a failure, I have to comb through the automation logs to find what happened, figure out what the automation was doing when it failed, and log into the software myself and try to recreate the problem manually. Sometimes the automation finds just the tip of a bug iceberg and I spend hours exploring to fully understand the problem. Some portion of the time, the failure is a bug in the automation that must be fixed. When it’s a legitimate product bug, then I have to write the bug in the bug tracker.

I am endlessly amused by how often I’ve had to explain that just running the automation isn’t the end of it: that if there are any failures, the automation doesn’t automatically generate bug reports. The standard response is some variation of “What? …ohhhhhh,” as it dawns on them. So far, thankfully, it has always dawned on them.

Myth: Automated tests can make up for years of bad development practices

I’ve just got to restate my point from my older post on this subject. If your development team doesn’t follow good practices such as writing lots of automated unit tests (to achieve about 80% code coverage), code reviews, paired testing, or test-driven development, automation from QA is not going to fix it. You can’t test in quality — you have to build it in.

If you’re sitting on a messy legacy codebase, one where your test team plays whack-a-mole with bugs every time you make changes to it, you are far, far better served investing in the code itself. Refactor, and write piles of automated unit tests.

You want on the order of magnitude of thousands of automated unit tests, hundreds of automated business-rule tests (which hopefully directly exercise an API, rather than exercising a UI, for resiliency and maintainability), and tens of automated checks to make sure the UI is functioning.

I’ll belabor this point: Invest in better code and better development practices first. When you deliver better quality to QA, you’ll keep the cost of testing as low as possible and more easily and reliably deliver better quality to your customers and users.