Programming is the number one quality assurance activity.
Or maybe it’s design. Or maybe it’s writing good user stories. Or maybe it’s having good ideas for things to build in the first place. Or maybe it’s paying down technical debt.
But it sure as hell isn’t testing.
When I talk to engineering leaders at small software development shops and they find out I make my living in testing, many of them admit that they don’t have any testers yet. They’re sheepish about it. It’s as if I’ll be offended!
I tell them to rock on, and to delay hiring that first tester for as long as they can.
And then I ask them about the quality challenges they have. Too many bugs? Won’t scale? Bogs down under load? Don’t fall prey to the gut reaction “oh my god we need to test,” as if testers are a magic filter through which perfect software passes.
Because if you respond by hiring testers, you’re likely to end up with testing theater. Your testers will do testery things and find bugs, sometimes even good ones, bugs that let you sleep better at night.
But they can’t fix your quality problems. Only your developers can do that. Instead of letting your developers do whatever it is they do and hope a tester can find everything that’s wrong, challenge your developers to get better.
I ask them these questions:
Do your developers have the skills needed to build the software you’re asking of them? If not, help them build those skills or, gulp, replace them with developers who have them.
Are you following good development practices? Test-driven development, pairing, and code reviews. Not only do they promote solid code, they help create a culture of quality among your developers.
Is your team writing lots of automated unit tests and and acceptance tests? (By acceptance tests, I mean thin functional tests at the API or controller level.) Do they run on every commit? This traps basic problems as soon as they’re introduced.
Do you have a well-functioning delivery methodology, at the right scale for your organization? If you’re two developers, that might be a kanban board drawn on a whiteboard. If you’re ten developers, you might use a tool like Pivotal Tracker and have a couple defined but light rules and ceremonies. If you’re 100 developers, it might be some scaled agile methodology like SAFe, backed up with a tool like JIRA, guided by a program management office. Whatever it is, are you following it and is work flowing smoothly through the delivery pipeline?
Are you giving your developers the time they need to do thoughtful work? To design resilient software that performs? To architect it for long-term growth?
Do all of these things before you hire your first tester. Your developers will give you a level of quality that makes it hard for testers to find low-level bugs. Testers will then have time to find more interesting and valuable bugs because basic stuff is seldom broken. This makes testing a lot less expensive, by the way. You need way fewer testers when you deliver software to them where core functionality works.
And then your testers, instead of feverishly testing the basics, can contribute at a higher level. Testers bring a different mindset to your teams, one of critical thinking about how the software is deployed and used. When you harness that mindset, you can improve your designs and your architecture before you write that first line of code.
I had lunch recently with a fellow I worked for several years ago. He gave me my last job as a technical writer and my first job as a software tester. He’s currently leading product development at a different software company. “Times have changed,” he said. “I don’t have any technical writers anymore. These days, I want the UX to be good enough that documentation isn’t needed.”
A few days later I had a drink with another former boss. I managed testers and technical writers while I worked for him. Since then, he’s started his own consulting company. “Technical writing is dying off,” he said. “It’s all about clean, engaging UX now. I have talked to more than a hundred startup and small software companies as I’ve built my business. Almost none of them have technical writers, and almost all of them have UX designers.”
Smart tech writers see the writing on the wall.
It’s clear: companies are leaning into good user-interface design and stepping away from online Help systems and printed/PDF documentation.
It’s a relief. Nobody wants to have to read something to learn how to use a software product. When usage falls right to hand, users are happier and more likely to use the product more. Good UX really can reduce the documentation burden.
My last company had mighty good UI design. One technical writer worked for me, and she kept up with about ten developers. In past companies with lesser UIs, I needed one writer for every four or five developers. It took more words to explain those products’ cumbersome usage.
Admittedly, the latent technical writer in me wants to holler, “Users need instructions for even the best designed products!” Some interactions are incredibly hard to design well enough to forego usage instructions. And users will always need usage reminders for seldom-executed tasks.
But the writing is on the wall. If you’re not finding fewer technical writing job openings yet, you will soon. Fortunately, your skills transfer to other jobs in software development organizations. You will need to build some new skills for many of these jobs, but you might be able to land that first new job without them and build them as you work.
Tester or quality assurance engineer
Testers explore software systems looking for bugs. In some shops, they write and execute detailed test cases; in others, they explore based on high-level test charters. The goal is to report, usually by writing bug reports, on the level of quality the developers have delivered so that decisions can be made about when to ship.
Technical writers routinely find product bugs. At my last company, my lone writer routinely found really important bugs. The VP of Engineering even noticed: “She finds outstanding bugs,” he said. “It must be because she thinks like a user.”
When I shifted into testing 15 years ago, I designed my tests in much the same way I designed online help: by first asking what tasks users would perform in the system. Then I set about making sure those tasks worked. You can do this too. You’ll have plenty more to learn about testing, but this is a great place to start.
Existing skills you will use: Creating and articulating mental models of software systems. Exploring software to discover how it works. A basic understanding of how software is built and delivered. Writing.
But build these skills: Database skills, especially writing queries. Light coding or scripting, to help you automate tests. System administration, to help you better understand and manipulate the environments the software lives in. (Don’t be daunted. I’ve seen many writers surprise themselves when they quickly start to learn these things. It’s as if they soaked up technical abilities by osmosis, just by being around lots of people who have them.)
Product owner, product manager, or business analyst
All of these jobs involve understanding customer needs and translating them into user stories, specifications, or requirements that the developers and testers use to build software. They may involve building a vision for what a product needs to be to meet its market’s needs. These people usually work closely with developers and testers to make sure the vision is realized, and to resolve implementation challenges.
Existing skills you will use: Understanding of customer needs. Creating and articulating mental models of software systems (though, in this case, often systems that have not been built yet). Writing. A basic understanding of how software is built and delivered.
But build these skills: Negotiation, as you might need to manage expectations of customers, the development team, and sometimes even management. Estimation and project management, as you might have to participate in sizing work and projecting delivery dates.
Various UX roles
The UX field includes a number of jobs that, together, create the way the software works and feels for the user. Typical titles include UX Designer, Information Architect, Visual Designer, Interaction Designer, and Content Specialist. These roles involve work such as creating wireframes of screens, designing user workflows, performing usability testing of prototypes, interviewing users and sometimes even shadowing them as they work, writing field labels and error messages, and choosing typography.
This might seem like a real stretch for a technical writer. But my experience is that writers often have innate insight into bad UX: if it’s hard to write about, then it’s hard to use. I find that technical writers can often extemporaneously evaluate product usability and give very useful ideas on how to improve UX.
Existing skills you will use: Interviewing. Understanding of customer needs. Creating and articulating mental models of software systems. Writing.
But build these skills: This depends heavily on the role, but: Design, in general. Graphic design. Usability testing. Prototyping.
These jobs crackle with career growth. But if you’d rather stay true to writing, you can shift into marketing communications, instructional design, or even good old-fashioned business writing (policies and procedures, disaster recovery plans, and the like). My town’s biggest employer is a pharmaceutical manufacturer; lots of software writers here have shifted into validation writing, which is an FDA compliance activity. You might even be able to move into writing technology articles and books; I’ve done a little of that. And some software technical writing jobs will likely always remain in regulated industries, and on government contracts, and for highly technical products.
Nostalgia for my former technical-writing career makes this a sad passage for me. But I think this trend toward effective UX is better for the user, and gives writers good paths for growth.
I wrote a post last year criticizing test automation when it’s used to cover for piles of technical debt and poor development practices. But I still think there’s a place for automation in post-development testing. There are two keys to using it well: knowing what it’s good at, and counting the costs. Without those keys it’s easy to fall prey to several myths of test automation. I aim to debunk them here.
Myth: Automation is cheap and easy
It is seductive to think that just by recording your manual tests you can build a comprehensive regression-test suite. But it never seems to really work that way. Every time I’ve used record and playback, the resulting scripts wouldn’t perfectly execute the test, and I’ve had to write custom code to make it work.
What I’ve found is that it takes 3 to 10 times longer to automate one test than to execute it manually. And then, especially for automation that exercises the UI, the tests can be brittle: you have to keep modifying scripts to keep them running as the system under test changes.
I’ve done straight record and playback. I’ve created automated modules that can be arranged into specific checks. I’ve led a team that created tests on a keyword-driven framework. And I currently lead a team that writes code that directly exercises a product’s API. The amount of maintenance has decreased with each successive approach.
A side note: given the cost of automating one test, can you see that you want to automate only what you are going to run over and over again, because otherwise the investment doesn’t pay?
Myth: Automation can test anything, and is as good as human testing
Automation is really good at repeating sets of actions, performing calculations, iterating over many data sets, addressing APIs, and doing database reads and writes. I love to automate these things, because humans executing them over and over is a waste of their potential.
This gets at a whole philosophical discussion about what testing is. I think that running predetermined scripts, whether automated or not, is just checking, as in, “Let me check whether clicking Save actually saves the record.”This subset of testing just evaluates the software based on predefined criteria that were determined in the past, presumably based on the state of the software and/or its specification or set of user stories as they were then.
The rest of testing involves human testers experimenting and learning, evaluating the software in its context now. This is critical work if for no other reason than the software and its context (environment, hardware, related software, customer needs, business needs, and so on) changes. An exploring human can find critical problems that no automated test can.
I want human testers to be free to test creatively and deeply. I love automated checks because they take this boring, repetitive work away from humans so they have more time to explore.
Myth: When the automation passes, you can ship!
It’s seductive to think that if testing is automated, that passing automation is some sort of Seal of Approval that takes out all the risk. It’s as if “tested” is a final destination, an assurance that all bets are covered, a promise that nothing will go wrong with the software.
But automation is only as good as its coverage. And if nobody outside your automation team understands what the automation covers, saying “the automation passed” has no fixed meaning.
It’s hard to overcome this myth, but to the extent I have, it’s because as an automation lead and manager I’ve required engineers to write detailed coverage statements into each test. I’ve then aggregated them into broad, brief coverage statements over all of the parts of the software under test. Then I’ve shared that information — sometimes in meetings with PowerPoint decks, always in a central repository that others can access and to which I can link in an email when I inevitably need to explain why passing automation isn’t enough. Keeping this myth at bay takes constant upkeep and frequent reminders.
Myth: Automation is always ready to go
“Hey, we want to upgrade to the next version of the database in the sandbox environment. Can you run the automation against that and see what happens?”
My answer: “Let’s assume I can even run the automation in sandbox. If it passes, what do you think you will know about the software?” The answer almost always involves feelings: “Well, I’ll feel like things are basically okay.” See “When the automation passes, you can ship!” above.
Automation is software, full of tradeoffs aimed at meeting a set of implicit and explicit goals. Unless one of those goals was “must be able to run against any environment,” it probably won’t run in sandbox. The automation might count on particular test data existing (or not existing). It might not clean up after itself, leaving lots of data behind, and that might not be welcome in the target environment. It might depend on a particular configuration of the product and its environment that isn’t present.
Even in the environment the automation usually runs in, it might not be ready to go at a moment’s notice. Another goal would need to be, “must be able to run at any time.” There are often setup tasks to perform before the automation can run: a reset of the database the automation uses, or the execution of scripts that seed data that the automation needs.
Myth: Just running the automation is enough
When I run automated tests, part of me secretly hopes they all pass. That’s because when there’s a failure, I have to comb through the automation logs to find what happened, figure out what the automation was doing when it failed, and log into the software myself and try to recreate the problem manually. Sometimes the automation finds just the tip of a bug iceberg and I spend hours exploring to fully understand the problem. Some portion of the time, the failure is a bug in the automation that must be fixed. When it’s a legitimate product bug, then I have to write the bug in the bug tracker.
I am endlessly amused by how often I’ve had to explain that just running the automation isn’t the end of it: that if there are any failures, the automation doesn’t automatically generate bug reports. The standard response is some variation of “What? …ohhhhhh,” as it dawns on them. So far, thankfully, it has always dawned on them.
Myth: Automated tests can make up for years of bad development practices
I’ve just got to restate my point from my older post on this subject. If your development team doesn’t follow good practices such as writing lots of automated unit tests (to achieve about 80% code coverage), code reviews, paired testing, or test-driven development, automation from QA is not going to fix it. You can’t test in quality — you have to build it in.
If you’re sitting on a messy legacy codebase, one where your test team plays whack-a-mole with bugs every time you make changes to it, you are far, far better served investing in the code itself. Refactor, and write piles of automated unit tests.
You want on the order of magnitude of thousands of automated unit tests, hundreds of automated business-rule tests (which hopefully directly exercise an API, rather than exercising a UI, for resiliency and maintainability), and tens of automated checks to make sure the UI is functioning.
I’ll belabor this point: Invest in better code and better development practices first. When you deliver better quality to QA, you’ll keep the cost of testing as low as possible and more easily and reliably deliver better quality to your customers and users.
Tomorrow it will have been 25 years since I started my career in the software industry.
It might seem odd that I remember the day only until you know that I started work on Monday, July 3, 1989, making my second day a paid holiday. The office was nearly deserted on my first day. My boss regretted not having me start on July 5 so he could have had an extra-long weekend too.
I graduated from that tough engineering school hoping to find work as a programmer. Jobs were hard to come by that year, so when a software company wanted to hire me as a technical writer I was thrilled just to work. And then it turned out I had a real knack for explaining software to people. I did it for twelve years, including a brief stint in technology publishing and five years managing writers.
I then returned to my technical roots, testing software and managing software testers. I learned to write automated functional and performance tests – code that tests code – and it has taken me places in my career that I could never have imagined.
My office at one of my career stops
I’ve worked for eight companies in 25 years. The longest I’ve stayed anywhere is five years. I left one company in which I was a poor fit after just 14 months. I’ve moved on voluntarily seven times, was laid off once, and was fired and un-fired once (which is quite a story; read it here). Changing jobs this often isn’t unusual in this industry and has given me rich experience I couldn’t have gained by staying with one company all this time.
I’ve worked on software that managed telephone networks, helped media buyers place advertising, helped manufacturers manage their business, run Medicare call centers, helped small banks make more money, enabled very large companies to more effectively market their products, and gave various medical verticals insight so they can improve their operations and their business.
Some of these companies were private and others were public; so far, I’ve liked private companies better. Some of them made lots of money, some of them had good and bad years, and one of them folded. Some of them were well run and others had cheats and liars at the helm. Some were very difficult places to work, but those were crucibles in which I learned the most. Others have brought successes beyond anything I could have hoped for a quarter century ago.
I did, however, hope for a good, long run in this industry, and I got it. But I’m also having a hard time envisioning another 25 years. It’s not just because I’d be 71 then. I really like to work, and – right now at least – I plan to do so for as long as I am able. But I’m starting to have trouble imagining what mountains I might yet climb in this career. Maybe that’s part of reaching middle age – indeed, many of my similarly aged colleagues, some with careers far beyond mine, have gone into other lines of work. I’m still having a lot of fun making software, though. I currently manage six software testers, one test-automation and performance-test developer, and one technical writer. I get to bring all of my experience to bear, and encourage my teams to reach and grow. I don’t want to stop just yet.
… If this sounds familiar, it’s because it’s an update of an eariler post. Cross-posted to my personal blog, Down the Road.
“Our product is a Web app,” the CEO said. “Why wouldn’t it work flawlessly on an iPad?”
The answer, of course, was that we didn’t build it with the iPad in mind. When we got it to work on our PCs on Chrome, we declared it done. But some guy at one customer site decided to run it on his iPad, and he was upset because it didn’t work “at all.” Support escalated the case to Engineering, where one of the testers poked around the product on her personal iPad. She found that she could log in, navigate, and enter data on any screen. “It was a pain. I’d rather do it on my laptop any day. But it did work,” she said.
So we were puzzled by our user’s bad experience. And then we learned that he had an original iPad. Our tester had a fourth-generation iPad – on more modern hardware and the latest version of iOS. Unless we wanted to shop for an old iPad on eBay, we were never going to get to the bottom of this complaint. The user wasn’t thrilled when we told him that we couldn’t support his old tablet.
Crushed under the weight of platform support
If you don’t tell your users what they can use to access and run your product, they will run it on whatever they want. When it doesn’t work, they’ll demand you support it. This will crush you. You need to set some limits.
This isn’t meant to be done by fiat. You need to think it through: What are your customers likely to want to use? You need to be there. How will they access your product – devices, operating systems, browsers? If your product will be installed on their equipment, what hardware (or virtual servers), operating systems, and databases are they likely to use? If your product interfaces with other products, what versions will your users want to use?
What you’re doing is limiting the edge cases, like that user’s original iPad. You’re also limiting the list to what you can meaningfully test.
Creating a matrix of supported platforms
Create a spreadsheet that says your product runs on and with these versions of these other products. If you support several versions of your product, the list of supported platforms will probably vary a little version to version.
Give the spreadsheet to support and sales and say, “Use this to set expectations. If they want to use something that’s not on this list, try to talk them into sticking to the matrix. If they insist on going their own way, tell them we might not help them if they have problems.”
This isn’t a once-and-done job. New versions of your supported platforms will be released. Your users will want to use entirely new products with your product. You’ll have to decide when and whether to add them to your matrix. You’ll need to remove old platforms from the matrix sometimes. You need to keep gathering information and keep updating the matrix.
You can’t support it all: why I hate Android
I used to have a Palm Pre (right). I loved it, but it was on nobody’s platform matrix! My iPhone 5 (left) is on everybody’s matrix.
Still stinging from the customer who had the bad iPad experience, the CEO asked, “Do we have any idea how the mobile Web site behaves on Android? It needs to run on Android, too. It needs to run on any device they have in their hands.”
The problem with Android is that there are eleventeen base versions in active use, each of which a vendor can customize for their devices. And the devices themselves run the processing-power and screen-resolution gamut. When I explained that I’d need to buy a hundred devices and hire twenty testers to test them all, the CEO became less excited about Android.
You can limit the scope of Android, and any other platform with a lot of permutations, by deciding which permutations you’ll support and testing only those. As I write this, the 2.3.x (Gingerbread) versions of Android are the most widely used. You can buy a couple of popular devices on that version and test just those.
The only challenge is that where there are lots of permutations, you’ll have plenty of users on unsupported permutations who aren’t going to be happy.
This reminds me of the browser interoperability problems from several years ago, and why companies usually limited their Web apps to work only on Internet Explorer. Browsers are better now, and it’s more common for Web apps to work on any modern browser. Here’s hoping Android settles down as browsers have.
You can’t test it all: Supported vs. Certified
The items on your supported platforms matrix needs to be things with which your product actually works. As much as possible, design your product up front to work with them. But if you’ve just created a supported platforms matrix for a product that’s been around a while, the best you can do is test your product with those supported platforms to find out where the bugs are. The knee-jerk approach is to do a “full” (whatever that means) regression test on every platform.
One place I worked, our supported platforms matrix grew huge. When we printed it, we needed a jeweler’s loupe to read it. For each version of our product, there were fifty or more supported platforms. I couldn’t afford the army of testers I’d need to do “full” regression passes on them all.
But all that testing really wasn’t necessary. You can look at what’s new in the latest version of a supported platform and make a pretty good assessment of the risk to your product, and size your testing accordingly. For example, a product I used to work on ran on the Oracle and SQL Server databases. When Oracle 11g Release 2 (11.2.0.1) came out, it offered enough new stuff that we weren’t sure how it would run with our product. So in our next release, we did our normal release testing, including regression, on that database. We also ran some special tests that exercised our most complicated SQL queries on it. We found and fixed some bugs that way.
When Oracle issued 11.2.0.2, we looked at what was new in that release and saw no reason to believe that any of it would create new bugs in our product. Our experience was that if a database version was going to throw bugs in our product, those bugs would be foundational and would appear right away under even light testing. We had a bug-fix release coming up, so we installed 11.2.0.2 on that test server and tested the bug fixes on it. The release sailed through testing, so we shipped it and added 11.2.0.2 to the supported platforms matrix.
Platforms that got the so-called “full” test treatment were listed as “Certified” on the matrix. What we told our customers was that we had “performed a certification test” on the platform, had fixed the bugs we found, and were highly confident of a good experience on that platform.
Platforms that got light testing were listed as “Supported” on the matrix. Sometimes our analysis showed low enough risk that we did no testing and still called the platform Supported. We told our customers that we expected that they would have a good experience on that platform.
The bottom line for both Certified and Supported was that if the customer experienced a problem, we’d take their call and resolve their issue.
The bigger bottom line is that you want a manageable list of platforms with and on which you believe your users will have good experiences using your product, a list that won’t kill you to keep up with. Through understanding what your customers want, limiting the list to the most common and important platforms, and varying the depth of testing based on risk, you can keep up.
I have been astonished in my life by how few problems are truly unsolvable. I have also noticed that, most of the time, when a problem ends up not being solved it is for one of two reasons: people deny the problem, or they won’t work together on the solution.
There are any number of ways to make software. All of them involve programming, of course; it’s the core activity. But there are other jobs to perform, and the bigger the software being developed, the more people you need to perform them. It becomes necessary to organize the people and form work processes. The ways to do that range from loose and chaotic to structured but flexible to ponderous and formal. They all have their strengths and weaknesses; they all can work. I’ve made software in all of these situations.
Even a gravel road will get you there.
One of my past employers sequestered its programmers in a room to hack code together as fast as they could. When they were done, a couple people would play with the product for a week or two in hopes of finding some bugs, which the programmers might or might not fix. Then we shipped it to five customers, who upon installing it invariably found that it failed spectacularly. When we fixed all the problems they found, we chose five more customers. We repeated this process until the software stopped failing, and then announced to the rest of our customers that it was ready.
At the other end of the scale, another past employer had such a heavy software development process that following it felt like slipping into a straitjacket. There were reams of documentation to write and keep with layers of approval at each step. I led the test team. We had to take a screen shot of every test step, print it, and put it in a folder. At the end of a project, we had boxes and boxes and boxes of printouts, hundreds of pounds worth, that we would send to an offsite storage facility. This was in case our only customer, which happened to be the U.S. government, wanted to audit us.
Both companies are still in business.
A third past employer carefully hired smart people and trusted them to know what to do. Because they hired well, this worked for a long time. But as the company grew, this approach became more and more difficult to manage. Product quality became a problem. A big part of the software was an accounting package, and in one fateful release the general ledger simply would not balance. This affected every customer, and to make a long story short, it took the company almost a year to fix it. Several customers quit us.
Too often it takes great pain to drive important change. This pain made us face that we needed more structure to deliver successfully. So we hired a consultant to guide us toward a better software development process. He showed us a way with just enough checks and balances that we could have greater confidence in our work without being too bogged down. We also hired a consultant to help the management team (of which I was a part) work more collaboratively. She taught us how to lead the company through this transition.
It worked. In the very first product release under the new process, quality problems fell off dramatically and we delivered on time for the first time anybody could remember.
The more important hire was the second consultant. The new methodology was good, but it wasn’t magic, and it might not have worked for us if we had not done a good job helping the team through the changes. You see, a few people didn’t understand why the old way couldn’t still work and still others thought our new process wasn’t going far enough. Our task was to overcome the resistance and get everybody singing from the same sheet of music, and it was the hardest thing we did. But we pulled it off.
It’s a rare organization that faces its failings and works to change. It’s a rarer organization still that drives change collaboratively.
Jim Grey 