Programming is the number one quality assurance activity.
Or maybe it’s design. Or maybe it’s writing good user stories. Or maybe it’s having good ideas for things to build in the first place. Or maybe it’s paying down technical debt.
But it sure as hell isn’t testing.
When I talk to engineering leaders at small software development shops and they find out I make my living in testing, many of them admit that they don’t have any testers yet. They’re sheepish about it. It’s as if I’ll be offended!
I tell them to rock on, and to delay hiring that first tester for as long as they can.
And then I ask them about the quality challenges they have. Too many bugs? Won’t scale? Bogs down under load? Don’t fall prey to the gut reaction “oh my god we need to test,” as if testers are a magic filter through which perfect software passes.
Because if you respond by hiring testers, you’re likely to end up with testing theater. Your testers will do testery things and find bugs, sometimes even good ones, bugs that let you sleep better at night.
But they can’t fix your quality problems. Only your developers can do that. Instead of letting your developers do whatever it is they do and hope a tester can find everything that’s wrong, challenge your developers to get better.
I ask them these questions:
Do your developers have the skills needed to build the software you’re asking of them? If not, help them build those skills or, gulp, replace them with developers who have them.
Are you following good development practices? Test-driven development, pairing, and code reviews. Not only do they promote solid code, they help create a culture of quality among your developers.
Is your team writing lots of automated unit tests and and acceptance tests? (By acceptance tests, I mean thin functional tests at the API or controller level.) Do they run on every commit? This traps basic problems as soon as they’re introduced.
Do you have a well-functioning delivery methodology, at the right scale for your organization? If you’re two developers, that might be a kanban board drawn on a whiteboard. If you’re ten developers, you might use a tool like Pivotal Tracker and have a couple defined but light rules and ceremonies. If you’re 100 developers, it might be some scaled agile methodology like SAFe, backed up with a tool like JIRA, guided by a program management office. Whatever it is, are you following it and is work flowing smoothly through the delivery pipeline?
Are you giving your developers the time they need to do thoughtful work? To design resilient software that performs? To architect it for long-term growth?
Do all of these things before you hire your first tester. Your developers will give you a level of quality that makes it hard for testers to find low-level bugs. Testers will then have time to find more interesting and valuable bugs because basic stuff is seldom broken. This makes testing a lot less expensive, by the way. You need way fewer testers when you deliver software to them where core functionality works.
And then your testers, instead of feverishly testing the basics, can contribute at a higher level. Testers bring a different mindset to your teams, one of critical thinking about how the software is deployed and used. When you harness that mindset, you can improve your designs and your architecture before you write that first line of code.
I lost my job a month ago. The software company where I worked cut expenses to fit revenue. That’s business.
I was Director of Quality Assurance. I built and led a team of functional testers, software engineers (writing code-based tests), and technical writers, and developed the strategies and processes the company used to test and deliver its products. It was a role my whole career had prepared me for, and I was glad to have it.
But now I’m working my network, looking for my next gig. It’s been fun. I’ve spoken to a few dozen CTOs, VPs of Engineering, and even CEOs so far in my search, usually from startups and young companies.
I’ve delivered so much software using waterfall and “scrum-fall” (the developers sprint, and testers try to keep up, but final testing still happens after developers are done) at various well-established companies that I thought that full-on agile was a myth, at least here in Indiana.
But the startup leaders tell me they’re all in: it’s agile, continuous integration, and continuous delivery all the way. And while all of them have or will soon need testers, they are all hedging on hiring specialized managers for them.
One CTO, a fellow I worked with several years ago, told a typical story: “I have a few scrum teams, each with one tester. But I can’t imagine ever hiring somebody to run QA as I don’t consider it to be a separate function. Testing is part of the scrum team. I’d have to think QA management jobs will eventually dry up everywhere. I’m sorry; I’m sure that’s not welcome news for you.”
When waterfall ruled the world, it made sense to have a QA department with its own leadership hierarchy. After all, the testers truly were a separate team with their own schedules, methodologies, and needs for care and feeding. But in agile, testers feel fully part of their scrum team rather than part of the team of people who test.
I admit to some anxiety over this evolution, entirely because I’m unexpectedly on the job market after several years of being Manager and Director of QA. But I think these changes are good for quality software delivery.
And these changes are creating opportunity, because there are things a good QA leader knows how to do that a CTO or VP of Engineering probably doesn’t. I networked my way into another startup CTO’s office just to introduce myself and ended up walking out with a short-term consulting job. It turns out that he’s ready to hire his first tester, but wants guidance on how testing should integrate into his team and how the tester should approach the work to find the most important bugs early. So I’m testing alongside his developers for now to get a feel for their context, and will develop a strategy and process his team can follow after I’m gone.
In my conversations, over and over CTOs are telling me they need people to coach and guide on functional testing strategy and technique, or on building test automation and performance testing practices. One VP of Engineering I’ve talked to had an unusual take: that I should be able to help him build and manage his end-to-end software delivery process, including engineering and testing, because he sees process as key to predictable and reliable software quality.
I certainly have experience in all of these areas. I imagine many Managers and Directors of QA do.
I don’t want to count my chickens before they hatch, but I feel pretty good that at least one of these threads will lead to a full-time job soon. And with it, I’ll step out of the waterfall/scrum-fall past and into a present I previously didn’t know existed.
I used to think that writing a fat suite of automated regression tests was the way to hold the line on software quality release over release. But after 12 years of pursuing that goal at various companies, I’ve given up. It was always doomed to fail.
In part, it’s because I’ve always had to automate tests through a UI. When I did straight record-and-playback automation, the tests were enormously fragile. Even when I designed the tests as reusable modules, and even when I worked with a keyword-driven framework, the tests were still pretty fragile. My automation teams always ended up spending more time maintaining the test suite than building new tests. It’s tedious and expensive to keep UI-level test automation running.
But the bigger reason is that I’ve made a fundamental shift in how I think about software quality. Namely, you can’t test in quality – you have to build it in. Once code reaches the test team, it’s garbage in, garbage out. The test team can’t polish a turd.
Writing an enormous pile of automated tests through the UI? Turdpolishing.
I’ve worked in some places where turdpolishing was the best that could be done. Company leadership couldn’t bear the thought of spending the time and money necessary to pay down years of technical debt, and hoped that building out a big pile of automated tests would hold the line on quality well enough. I’ve led the effort at a couple companies to do just that. We never developed the breadth and depth of coverage necessary to prevent every critical bug from reaching customers, but the automation did find some bugs and that made company leadership feel better. So I guess the automation had some value.
But if you want to deliver real value, you have to improve the quality of the code that reaches your test team. Even if the software you’re building is sitting on a mountain of technical debt, better new code can be delivered to the test team starting today. I’m a big believer in unit testing. If your software development team writes meaningful unit tests for all new code that cover 60, 70, 80 percent of the code, you will see initial code quality skyrocket. Other practices such as continuous integration, pair programming, test-driven development, and even good old code reviews can really help, too.
But whatever you do, don’t expect your software test team to be a magic filter through which working software passes. You will always be disappointed.
“Our product is a Web app,” the CEO said. “Why wouldn’t it work flawlessly on an iPad?”
The answer, of course, was that we didn’t build it with the iPad in mind. When we got it to work on our PCs on Chrome, we declared it done. But some guy at one customer site decided to run it on his iPad, and he was upset because it didn’t work “at all.” Support escalated the case to Engineering, where one of the testers poked around the product on her personal iPad. She found that she could log in, navigate, and enter data on any screen. “It was a pain. I’d rather do it on my laptop any day. But it did work,” she said.
So we were puzzled by our user’s bad experience. And then we learned that he had an original iPad. Our tester had a fourth-generation iPad – on more modern hardware and the latest version of iOS. Unless we wanted to shop for an old iPad on eBay, we were never going to get to the bottom of this complaint. The user wasn’t thrilled when we told him that we couldn’t support his old tablet.
Crushed under the weight of platform support
If you don’t tell your users what they can use to access and run your product, they will run it on whatever they want. When it doesn’t work, they’ll demand you support it. This will crush you. You need to set some limits.
This isn’t meant to be done by fiat. You need to think it through: What are your customers likely to want to use? You need to be there. How will they access your product – devices, operating systems, browsers? If your product will be installed on their equipment, what hardware (or virtual servers), operating systems, and databases are they likely to use? If your product interfaces with other products, what versions will your users want to use?
What you’re doing is limiting the edge cases, like that user’s original iPad. You’re also limiting the list to what you can meaningfully test.
Creating a matrix of supported platforms
Create a spreadsheet that says your product runs on and with these versions of these other products. If you support several versions of your product, the list of supported platforms will probably vary a little version to version.
Give the spreadsheet to support and sales and say, “Use this to set expectations. If they want to use something that’s not on this list, try to talk them into sticking to the matrix. If they insist on going their own way, tell them we might not help them if they have problems.”
This isn’t a once-and-done job. New versions of your supported platforms will be released. Your users will want to use entirely new products with your product. You’ll have to decide when and whether to add them to your matrix. You’ll need to remove old platforms from the matrix sometimes. You need to keep gathering information and keep updating the matrix.
You can’t support it all: why I hate Android
Still stinging from the customer who had the bad iPad experience, the CEO asked, “Do we have any idea how the mobile Web site behaves on Android? It needs to run on Android, too. It needs to run on any device they have in their hands.”
The problem with Android is that there are eleventeen base versions in active use, each of which a vendor can customize for their devices. And the devices themselves run the processing-power and screen-resolution gamut. When I explained that I’d need to buy a hundred devices and hire twenty testers to test them all, the CEO became less excited about Android.
You can limit the scope of Android, and any other platform with a lot of permutations, by deciding which permutations you’ll support and testing only those. As I write this, the 2.3.x (Gingerbread) versions of Android are the most widely used. You can buy a couple of popular devices on that version and test just those.
The only challenge is that where there are lots of permutations, you’ll have plenty of users on unsupported permutations who aren’t going to be happy.
This reminds me of the browser interoperability problems from several years ago, and why companies usually limited their Web apps to work only on Internet Explorer. Browsers are better now, and it’s more common for Web apps to work on any modern browser. Here’s hoping Android settles down as browsers have.
You can’t test it all: Supported vs. Certified
The items on your supported platforms matrix needs to be things with which your product actually works. As much as possible, design your product up front to work with them. But if you’ve just created a supported platforms matrix for a product that’s been around a while, the best you can do is test your product with those supported platforms to find out where the bugs are. The knee-jerk approach is to do a “full” (whatever that means) regression test on every platform.
One place I worked, our supported platforms matrix grew huge. When we printed it, we needed a jeweler’s loupe to read it. For each version of our product, there were fifty or more supported platforms. I couldn’t afford the army of testers I’d need to do “full” regression passes on them all.
But all that testing really wasn’t necessary. You can look at what’s new in the latest version of a supported platform and make a pretty good assessment of the risk to your product, and size your testing accordingly. For example, a product I used to work on ran on the Oracle and SQL Server databases. When Oracle 11g Release 2 (188.8.131.52) came out, it offered enough new stuff that we weren’t sure how it would run with our product. So in our next release, we did our normal release testing, including regression, on that database. We also ran some special tests that exercised our most complicated SQL queries on it. We found and fixed some bugs that way.
When Oracle issued 184.108.40.206, we looked at what was new in that release and saw no reason to believe that any of it would create new bugs in our product. Our experience was that if a database version was going to throw bugs in our product, those bugs would be foundational and would appear right away under even light testing. We had a bug-fix release coming up, so we installed 220.127.116.11 on that test server and tested the bug fixes on it. The release sailed through testing, so we shipped it and added 18.104.22.168 to the supported platforms matrix.
Platforms that got the so-called “full” test treatment were listed as “Certified” on the matrix. What we told our customers was that we had “performed a certification test” on the platform, had fixed the bugs we found, and were highly confident of a good experience on that platform.
Platforms that got light testing were listed as “Supported” on the matrix. Sometimes our analysis showed low enough risk that we did no testing and still called the platform Supported. We told our customers that we expected that they would have a good experience on that platform.
The bottom line for both Certified and Supported was that if the customer experienced a problem, we’d take their call and resolve their issue.
The bigger bottom line is that you want a manageable list of platforms with and on which you believe your users will have good experiences using your product, a list that won’t kill you to keep up with. Through understanding what your customers want, limiting the list to the most common and important platforms, and varying the depth of testing based on risk, you can keep up.