Category: Quality

Categories
Project Management Quality

Ship Mode: Recognizing when it’s time to stop polishing your work and just ship it

By Jim Grey (about)

Every creative project has three phases: make the thing, polish the thing, deliver the thing. The polish step is where you remove errors through testing, editing, inspection, or other review. The deliver step is where you put your work in people’s hands.

It’s so easy to get stuck on the polish step. You keep checking until you’re sure it’s perfect! This is all about fear. When your work is in the world, they can judge it. They can even ignore it. We want to avoid how bad that feels.

Also, perfection is expensive. You can spend as much time rooting out every tiny flaw as you did making the thing. Those tiny flaws will be embarrassing. But they won’t really hurt anything, and they won’t keep anyone from clicking Buy Now. Crucially, eliminating every last minor flaw keeps you from working on new projects that create new value.

When you’ve applied reasonable polish, when you feel the fear of rejection, it’s time to enter Ship Mode.

In Ship Mode, you single-mindedly do the tasks that put the work in people’s hands. You’re not looking for problems anymore. You choose to think of your work as a finished product. You might notice an error while you’re in Ship Mode, but unless it’s truly egregious, you keep shipping.

My new book was good enough to ship — but it’s not perfect

I write in my spare time, personal stories and essays. I’ve done it for years. I recently self-published a book of some of my earliest work. (Scroll to the bottom for more details!) I did the whole job: writing, editing, creating the print-ready and e-book files, and (now) marketing. I saved money doing it all myself, but I’m skilled in only some of these tasks. Also, there came a point where I’d looked at my book so much I had become blind to it.

I’m a recovering perfectionist and I’m mildly OCD (officially diagnosed). It was hard for me to learn to let go and enter Ship Mode. But I’m glad I learned it many years ago, or I would still be making myself nuts perfecting my book. I have a day job. There’s only so much time to work on side projects. If I polished this one to perfection, it would not be available for several more weeks yet.

Curse you, Page 50!

I saw it only in the last step of the submission process to Amazon: this ungraceful flow on page 50. No publishing company would allow a paragraph on one page to spill three words onto the next right before an illustration.

When I saw it I gritted my teeth. I probably said a four-letter word. But not only is this problem not egregious, but most readers won’t even recognize it as a problem. Nobody will demand a refund because of it. I clicked the Approve button to finish the submission. That’s Ship Mode!

I edited every story as I assembled the book, and then made two proofreading passes. But when my author copy arrived I found two typos in five minutes. How frustrating!

But I will be shocked if you find something really messed up, like garbled sentences or missing paragraphs. During the polish phase I sweated out everything that would have seriously damaged your experience with the book.

And then I got on with shipping it so you could read it and, I hope, enjoy it.

Ship Mode! Because there’s a point past which polish doesn’t pay.

My new book, A Place to Start: Stories and Essays from Down the Road, collects early work from my personal blog. I had just come through a great difficulty in my life, a real setback that left me reeling. Through writing my stories I was able to make sense of what had happened so I could move on. My book captures those stories and shows a way to find your way through hard times to a better future.

Here’s how you can get my new book:

Categories
Quality The Business of Software

Love your salespeople because they make your paychecks not bounce

By Jim Grey (about)

He was a salesman at the company where I had gone to work after graduating from engineering school. And he routinely made my life, and the lives of the other engineers with which I worked, crazy difficult as he made wild promises during the sales cycle that we then had to fulfill in our product on impossible deadlines.

It was the classic tension between sales and engineering. There should have been a class on it when I was in school. I might have called it “Sales Douchebaggery 101” but perhaps the topic would better have been wrapped into a larger course on life as a working engineer. Cost constraints out the wazoo. Adding features to a product under pressure without doing necessary redesign of existing features so they scale and perform. Being bogged down by the avalanche of customer support that follows releasing the resultant buggy product. And all the while, salespeople promising crazy stuff to prospects to get them to sign on the line. Sales happily wrote checks, if you will, that engineers then had to figure out how to cash.

This particular salesman was the king of signing us up to build stuff on impossible deadlines. He seemed to take special glee at doing it. Cynically, we all grumbled to each other that it was about making his commission check as fat as possible. One of the engineers delivered this classic line: “He’s in sales. That means he’s coin operated.” I’ve gotten a lot of mileage out of that one.

LincolnMarkA

And then one day he showed up at work in a brand new Lincoln Mark VIII. It was a stunning car in a pearly white. Lincoln called it “white opalescent,” and it was impossibly deep, almost liquid. I expected that if I touched the car, my fingers would ooze into the body and come back wet. It was mesmerizing.

Yet I was angry and jealous. Here was this guy blithely, happily making engineer lives miserable while piling up a crap ton of dough. I wasn’t going to make that kind of money anytime soon; no new Lincolns were in my future. But this wasn’t really about financial injustice. No, this was about him never suffering any consequences of his actions while living a lifestyle that reinforced his bad behavior. Such bullshit.

That was in the early 1990s. Since then I’ve worked for many other companies, successful and not, and have learned a few key things about sales. First and foremost: celebrate every sale, because they keep your paychecks from bouncing. I’ve lived through that and never want to again.

But second, that company had a lot to learn about product marketing and sales. The sales team was left to fend entirely for itself, with only a vaguely identified target market, no coherent story to tell about the product line, and nothing that generated qualified leads for them to pursue. They earned every closed deal from the ground up: cold calls, relationship building, and then doing what they had to do — including making stuff up — to get a yes so they could meet their quotas. It had to be brutal for them. So no wonder this guy was so giddy every time he closed a sale. He worked his butt off for it.

And third, and most importantly, back then I was content to grumble and that didn’t help my company at all. Learning how to estimate, negotiate scope, determine minimum viable product, set and manage expectations, and manage projects has been much more effective for me and for the places where I’ve worked. It’s helped sales teams know that Engineering is in their corner helping them succeed — and helped them better understand what Engineering is capable of delivering so they make fewer bad promises.

LincolnMarkB

Today I don’t sweat that fellow his Lincoln. Seeing this one in a restaurant parking lot recently in sort of sad condition reminded me of him. I looked him up on LinkedIn. He’s still selling in that industry, for a company we all derided then as being the last place you’d want to work. Looks like his career had a similar trajectory to this Lincoln: still going, but not looking all that great. Fortunately, I’ve grown up enough to have empathy for him.

I regularly take photos of old parked cars and write about them for Curbside Classic, a site that tells these old cars’ stories. This post is heavily based on a post I wrote there a couple years ago; read it here.

Categories
Quality

Programming is the number one quality assurance activity

By Jim Grey (about)

Programming is the number one quality assurance activity.

Or maybe it’s design. Or maybe it’s writing good user stories. Or maybe it’s having good ideas for things to build in the first place. Or maybe it’s paying down technical debt.

Test

But it sure as hell isn’t testing.

When I talk to engineering leaders at small software development shops and they find out I make my living in testing, many of them admit that they don’t have any testers yet. They’re sheepish about it. It’s as if I’ll be offended!

I tell them to rock on, and to delay hiring that first tester for as long as they can.

And then I ask them about the quality challenges they have. Too many bugs? Won’t scale? Bogs down under load? Don’t fall prey to the gut reaction “oh my god we need to test,” as if testers are a magic filter through which perfect software passes.

Because if you respond by hiring testers, you’re likely to end up with testing theater. Your testers will do testery things and find bugs, sometimes even good ones, bugs that let you sleep better at night.

But they can’t fix your quality problems. Only your developers can do that. Instead of letting your developers do whatever it is they do and hope a tester can find everything that’s wrong, challenge your developers to get better.

I ask them these questions:

Do your developers have the skills needed to build the software you’re asking of them? If not, help them build those skills or, gulp, replace them with developers who have them.

Are you following good development practices? Test-driven development, pairing, and code reviews. Not only do they promote solid code, they help create a culture of quality among your developers.

Is your team writing lots of automated unit tests and and acceptance tests? (By acceptance tests, I mean thin functional tests at the API or controller level.) Do they run on every commit? This traps basic problems as soon as they’re introduced.

Do you have a well-functioning delivery methodology, at the right scale for your organization? If you’re two developers, that might be a kanban board drawn on a whiteboard. If you’re ten developers, you might use a tool like Pivotal Tracker and have a couple defined but light rules and ceremonies. If you’re 100 developers, it might be some scaled agile methodology like SAFe, backed up with a tool like JIRA, guided by a program management office. Whatever it is, are you following it and is work flowing smoothly through the delivery pipeline?

Are you giving your developers the time they need to do thoughtful work? To design resilient software that performs? To architect it for long-term growth?

Do all of these things before you hire your first tester. Your developers will give you a level of quality that makes it hard for testers to find low-level bugs. Testers will then have time to find more interesting and valuable bugs because basic stuff is seldom broken. This makes testing a lot less expensive, by the way. You need way fewer testers when you deliver software to them where core functionality works.

And then your testers, instead of feverishly testing the basics, can contribute at a higher level. Testers bring a different mindset to your teams, one of critical thinking about how the software is deployed and used. When you harness that mindset, you can improve your designs and your architecture before you write that first line of code.

Categories
Quality The Business of Software

If you want your software to keep producing, be prepared to do some dirty work

By Jim Grey (about)

A woman named Verna built the house I live in. She landscaped it nicely; a sprawling flowerbed stretches in front of my front door and picture windows. Every spring, I eagerly await Verna’s spring color: yellow daffodils, purple hyacinths, red tulips, and finally the giant pink peonies.

Home

I’ve added a few things: lilies, mums, lavender, coreopsis, phlox. I love phlox! But my eagerness to keep adding color petered out pretty quickly because it turns out I hate digging in the dirt.

I don’t much enjoy any of the other routine garden maintenance, either. Mulching. Deadheading. Dividing overgrown plants. Weeding – oh god, the weeding. Does it make me lazy that I just spray my weeds with Roundup and move on?

I just want to enjoy the flowers. But this ninth spring I’ve lived in my home, a few of my plants didn’t come back as strong. A couple didn’t come back at all.

So I asked my mom. She’s the gardener in our family. “When was the last time you fertilized?” she said. “Um, never,” I said. “Ah,” she said.

It turns out that you can’t just ignore the soil, or the plants themselves for that matter. Things growing in it year after year uses up all the nutrients, and crowded plants compete with each other for what little is there. “I’m surprised your flowers didn’t stop coming back a few years ago,” Mom said.

Daffodils

I did some serious fertilizing this season. I also separated some overgrown hostas and moved some of Verna’s plants so they had some elbow room. Not fun, but necessary.

♦ ♦ ♦

I once worked for a software company whose flagship product sold briskly. Version 1.0 was five years in the past, and since then we’d added lots of new features so the product could continue to lead the market. And now here came the head of Product Management asking for more new features

Dan, a quiet fellow, graying at the temples, led Development. “Well, yes, we can add all those features,” Dan said, adjusting his glasses. “This one will take six months. That one will take four. This other one, well, I think that’ll take a year.”

The Product Manager was dumbfounded. “Features of similar scope took far less time in the past, and you had fewer developers then. What gives?”

Dan looked up at the Product Manager kindly, and drew a breath. “Well, we’ve been under such pressure to quickly add features to this product that we’ve not focused enough on its overall design. We’ve also made no time to keep our underlying architecture up to date. These are things I’ve been pointing out all along the way. But we’ve just bolted features on wherever we thought we could get away with it. Now, to add any one of the features you’ve requested, we basically have to unbolt three or four other features, and blend the code all together. And we have to write complicated bridge code to do modern things with our aging architecture, and when that doesn’t work we will have to upgrade some parts of it and test the product well to make sure everything still works. It’s a slow process. And it’s just going to get slower and slower the longer we keep going like this.”

That the product’s design had become cancerous and the underlying architecture had gone out of date were not considered a crisis –- but not being able to rapidly add new features sure was. It focused the company’s entire attention. Their response was to code up a “next generation” product from scratch, which was a disastrous idea for a whole bunch of reasons beyond the point of this story. When the dot-com bubble burst in 2001-2002, they had not yet successfully launched the next-generation product, and they still couldn’t add features to the old product fast enough. Revenue fell precipitously. Quarterly layoffs began, but it was not enough to keep the wolves from the door. That once-promising company was sold; the company that bought it outsourced development to China.

More recently I went to work for another promising software company. They had been in business for about a decade and had sold their software to a number of very large companies. But in the couple years before I’d been hired, the pace of new feature delivery had slowed to a crawl. Adding new features had become increasingly difficult and always broke existing features. As a result, it took longer and longer to test the product, but even then, major bugs were still being delivered to customers. Meanwhile, younger, more nimble competitors were stealing business away from us. As the rate of new revenue decreased, support costs skyrocketed. It was unsustainable, and that company, too, had to sell itself to another company to avoid collapse.

It was much the same story: the company had focused entirely on rapid new-feature delivery and not enough on ongoing design and architecture. After a decade, their soil had gone infertile and the code had become tangled. Nothing new would grow.

♦ ♦ ♦

Software as a garden: to be able to grow more software, to be able to grow revenue with it, you have to keep the soil fertile and give the roots room. The problem is, gardening projects are a hard sell. These are things like refactoring older parts of the code that no longer serve efficiently, or upgrading or replacing outdated parts of the architecture, or redesigning subsystems that work fine today but can’t adapt to things the company wants to do in the future. When you tell executives you need to do these things, what they hear is that they can’t have new features while you do it. New features fuel growing companies.

But if you don’t tend your garden, sooner or later it will stop producing.

Categories
Quality The Business of Software

Don’t piss off your users by suddenly changing your UI

By Jim Grey (about)

Delivering software on the Web is great. Especially with continuous delivery, we can deliver changes large and small anytime we want. And then we can get quick feedback from our users and the market, adjust the software accordingly, and push those updates fast, too. It’s utopia and the Holy Grail rolled into one!

Except that users are not very excited when we change things. They want software to stay as it is. Well, mostly: they want us to fix the bugs that affect them, of course, or even to add this or that little feature. But please, they plead, don’t make it work differently than it does now.

Meanwhile, we face various pressures. Markets shift; new needs emerge and old needs become less important. Technologies shift; old frameworks become outdated, new frameworks enable us to keep pace. Today everything has to not only work on mobile, but feel native to mobile — and all run on a single codebase. This is shifting product direction across our industry.

That’s the backdrop against which WordPress, the content engine behind one out of every four Web sites, rolled out a new editor last week. It was part of a complete rewrite of all of WordPress.com. Their old technologies just couldn’t stretch to where the world was moving. So they threw it out and started from scratch. Their new editor is fast — fast! — and works fluidly, while looking great both in my browser and on my phone.

2015-11-22_2157
Spanking new editor in my browser…

But boy, were users pissed. Pissed! Check the WordPress.com forum: 19 pages of complaints and counting. Sometimes, I swear, users wouldn’t be happy if you sent them gold bars, because they preferred the silver bars you used to send them. But very often users have a point: they’ve gotten into the swing of your software, and now you’ve changed it and they have to learn it all again. Worse, maybe now something they used to be able to do isn’t there anymore, or if it is, they can’t find it.

IMG_4117
…and on my phone

For the record, I was the first commenter on that thread, because I experienced some of those frustrations. I tried to be kind, but several features I use either went missing or were accessed in a way that I couldn’t easily discover. Argh! And I wished the editing space were wider; it felt awfully cramped. I wasn’t alone in any of these complaints.

I wanted to just edit a post. I didn’t want to learn a new interface. But I found that there’s no way to just revert to the previous editor. It is simply gone.

I understand what drives changes like this and know that this is a monumental achievement this is for WordPress. Still, because I’m a heavy WordPress user, more than anything else I feel frustrated. The new editor breaks all of my usage flows, and I’m having to rediscover everything. I didn’t want this.

It’s the same, by the way, with Microsoft Office’s ribbon, which replaced an older menu structure way back in Office 2007. That’s forever ago in software terms. Yet there are still features I can tell you exactly how to find in those old menus, but I have to Google where they are on the ribbon.

Users don’t give a rip about your business or the future of technology. They use your product to accomplish a thing. As long as they can consistently and easily accomplish that thing, they stay happy. Many users learn your product’s nuances and become quite adept with them. When you suddenly change the UI and all of their flows are interrupted, of course they’re frustrated.

So what would happen if you followed Basecamp’s model? Their software helps companies small and large manage projects. Last month, they released Basecamp 3, a ground-up rewrite — yet they received not a single complaint from existing users. That’s because Basecamp 2, and for that matter Basecamp 1, remain fully active. Existing users can upgrade if they want, or stay put if they don’t. There are compelling reasons to move to Basecamp 3. But if you’re a happy Basecamp 1 or 2 user, those products will be there, fully supported, for as long as you want to use them.

Maybe your company can’t do that. But what can you do to ease the transition for your users, so they can stay fully productive? Think this through. It’s more important than any technology or implementation decision you make.

Fortunately, WordPress does, for some reason, still provide back-door access to an even older editor.

2015-11-22_2158
Outdated but highly functional classic editor

I don’t care that this is based on outdated technology: it’s fully featured, and I know how to make it sing. I cut my blogging teeth on this editor when I started my personal blog in 2007. I’ve written over a thousand posts in it. I hope it never goes away.

Categories
Quality Testing

Fast failure recovery lets you take more risk and increase speed

By Jim Grey (about)

I can just imagine the “oh no” moment that rippled through Feedly last week when they learned that their big new release for iOS 9 crashed on launch for iOS 7 and 8 users. Their first response: add a hastily-written warning to the App Store description.

IMG_4066

Their second response: fix the bug, fast. It was ready the next day.

IMG_4069

Apparently, Apple has a way for developers to rush the very occasional critical fix into the App Store, sidestepping the normal, weeks-long approval queue.

Apple’s App Store fast track provides a safety net, and I don’t blame them for using it. Because Feedly could recover fast, hardly anybody will remember this gaffe, and it won’t cost the company very much.

I actually feel slightly bad talking about it, because it keeps the memory of this bug alive. But it illustrates such a simple equation: the faster you can recover from failure, the less perfect your product has to be when you ship it, and therefore the less expensive it is to build and support, and the faster your company can move.

I’m not advocating for delivering buggy product on purpose. Follow good development practices and test for important risks to deliver the best software you can as fast as you can. But when you inevitably deliver a bad bug, being set up to recover fast means you can deliver without worry.

I remember when the best way to get software to users was to mail it to them on CDs. A bug of this magnitude was a much bigger deal then. It was harder to warn users of the problem, and lots slower and more expensive to correct it. In that world, Feedly’s bug could have damaged their reputation for a long time. So back then it made sense to test more thoroughly — and therefore spend more time and money before releasing.

But today, in a world of Web software and 24-hour emergency App Store turnaround, you’ll deliver faster and with less expense when you set yourself up for fast failure recovery. Continuous integration and continuous delivery are usually a part of that strategy.

Categories
Quality Testing

You can’t test it all

By Jim Grey (about)

That new build of software you are about to test? It’s a haystack with some unknown number of needles (bugs) in it.

Hay rolls 3
Have fun finding all the needles!

As a tester, you might think your job is to find all the needles. But how do you do that when you don’t know how many needles are in there? What if there are a lot of needles in there? You’ll never have time to find them all.

You need a plan. You want to find the showstopper bugs right away, and then find as many other bugs that people will care about within the time you have. And then when they come breathing down your neck to stop already and ship, you want to be able to tell them just what badness still might lurk in the code. Give them a reason to think it over.

You do that through assessing risk and targeting test coverage. To assess risk, ask yourself some questions:

  • How stable is the code that was changed? What interactions within the software might these changes break? You’re trying to figure out how likely it is you’ll find bugs.
  • If stuff around these code changes is broken, how much could it hurt the user? How much could it hurt your company? You’re trying to figure out the impact of any bugs you might find.

Risk is the product of likelihood and impact. Test for the highest risk bugs first, working down through the risks. Test more deeply for the bigger risks, more lightly for the smaller ones.

Let’s say they want to ship before you’ve tested through the risks you think people will care about. You can then talk about the risks you haven’t checked for yet, and ask if they’re okay with shipping like that. Do you see the mind shift here? You’re not saying you haven’t run all of your tests yet, which sounds an awful lot like you can’t keep up. Instead, you’re saying that the code might not be ready yet, and here are the specific things you’d like to still check for. It puts you in a much stronger position to get that extra time — and makes it the boss’s decision about what to do next.

Ultimately, it’s best if your developers can and will take great care to not deliver so many needles. That’s always the best case. Click here to read more about it.

Categories
Quality Testing

The tester’s three mental models

By Jim Grey (about)

It’s a common mistake among new testers: test it all, every time. But the weight of all that checking soon crushes the tester, and s/he starts looking for ways to test less without missing anything important.

And so begins the journey of understanding risk likelihood and impact: how likely is a thing to be broken, and how bad is it when it is. Smart testers prioritize likelihood and impact, and test in priority order. That way, should time run out, only low risk and low impact areas of the product remain untested. Heck, you might even skip tests that are ranked low enough. Maybe you should skip those tests, as they’re likely to find bugs nobody cares about. A radical thought!

But how to rank risk and impact? This reminds me of an old joke.

Ice cream board
I wish all chalk marks could be about ice cream.

There was an engineer who kept the big machine on the shop floor running faithfully for 30 years. After he retired, the machine promptly broke down. Nobody could get it running again. In desperation, the company called the engineer and implored him to come back and fix it.

The retired engineer returned, albeit reluctantly. He spent a day looking the machine over. Then he called everybody together and marked an X in chalk on a particular component. “Replace this, and the machine will work again.” Glory be, he was right! “Send us an invoice,” the boss said.

And the engineer did: for $10,000. “Ten thousand dollars!” the boss cried. “You need to justify that!” The engineer said he’d send an itemized invoice. Here’s how it read:

One chalk mark: $1

Knowing where to put it: $9,999

Testing for risk and impact means knowing where to put it — that is, knowing where to go to find the most serious bugs. You get good at that by building these three mental models:

Code

What impact on the rest of the software will these code changes have? In other words, what is likely not to work as desired after these code changes are made?

This means you have to learn how is the product is designed and built. That doesn’t mean you necessarily have to be able to read code, although it doesn’t hurt. You just have to pay attention as you test the product and listen to the developers’ explanations of the product’s technical details. You will know you’re building this model when you articulate how you think the product is built to a developer and they say something like, “Yeah. Those aren’t exactly the words I’d use, but they’re accurate enough.”

The code mental model helps you assess risk likelihood. “That part of the product is a little brittle, and every time something interacts with it, things are broken,” or, “I know we designed that function to handle a certain throughput, but what we’re contemplating is 10 times that, and so I’m concerned it’ll fold under the pressure.”

Customer

What parts of the product, when not working as desired, will be a problem for the customer or user? How severe a problem will it be?

To build this model, form good relationships with your support and implementation teams. You might even do rotations through support from time to time, and review customer-reported problems and seek clarity from support on how difficult they were for customers.

The customer mental model helps you assess impact. “If we ship this bug, customers are going to scream,” or, “I think support can talk customers around this bug,” or, “Customers are probably not going to even notice this bug.”

Business

What parts of the product, when not working as desired, put the company’s revenue or reputation at risk, or interrupts smooth and efficient company operations? How severe a problem will it be?

The business mental model gets at how your company makes money and grows the business. This is often the hardest mental model to build, but to the extent you build it, you can make much more nuanced test coverage decisions. To start, you can form an understanding of the kinds of product issues that get customers to call the CEO and threaten to cancel or sue. You can come to understand the kinds of problems that place heavy burden on the support and implementation teams, or would cost the company money in terms of time taken away from revenue-generating activity or services given for free to help regain an angry customer’s trust.

Come to understand which customers, especially the most lucrative ones, are up for renewal soon, and which are unhappy with your company and why.

The business mental model helps you further assess impact. “If this doesn’t perform well, customers are going to quit us,” or, “Bugs in this part of the product always flood us with calls and disrupt our ability to deliver more software.”

Categories
Quality The Business of Software

It’s remarkable that the “Great Glitch of July 8” doesn’t happen more often

By Jim Grey (about)

It was never a coordinated “cyber-attack,” as several news outlets speculated.

GlitchForbes

It was simple coincidence that several separate systems failed on the same day, last Wednesday, July 8: the trading system at the New York Stock Exchange, many systems at United Airlines, and the Web site of The Wall Street Journal.

Technology fails all the time. You just don’t usually recognize it. Have you ever noticed a page on a site loading unusually slowly? Or have you ever been unceremoniously logged out? I’m sure that as long as the screen finished loading, or that you were able to successfully log right back in, you shrugged it off and moved on. It might have been random Internet gremlins or lousy Wi-Fi. But it could also have been a failure in the service. Perhaps monitoring software noticed it and quietly performed a restart. Or maybe a few minutes of high drama unfolded in some technical operations center somewhere as technicians righted the situation.

But why do such systems fail? Several reasons:

GlitchNBC

Legacy systems patched and updated for so many years that the code has become sclerotic. Big, old companies like United Airlines are bursting with old systems. I wouldn’t be surprised if some part of their reservation system involves a mainframe! Systems like these have been repaired and extended for years upon years, and by now none of the original programmers and technicians still work there. The code has become difficult to restabilize after any change. It’s prohibitively expensive to build a new system from scratch, and even if you could afford it, you’d just introduce a whole host of new problems anyway.

System integrations and data migrations gone wrong. Company A buys Company B. There’s a lot of overlap in the technologies they use, so they integrate them or migrate the data from one to the other. In any such project, a thousand edge cases lurk that, when triggered, can cause failure. Even the most crack project team will miss some. There’s never time and money to find them all anyway. Missed edge cases are just ticking time bombs.

GlitchCNN

Poor original engineering. Because software engineering is still a nascent discipline, we’re still figuring out how best to do it. Every methodology has challenges and limitations. Smart engineers do the best they can to design a system that will work well, but are always limited by time and money. Sometimes revenue pressure leads engineers to favor fast over good. And even then, it’s very hard to imagine all the demands that will be placed on a system over time.

One of my past employers had a Web service that pumped customers’ backend-system data into our database. It was fast and reliable until we sold the product to a customer that wanted to blast in 10 years of historic data. We’d never done that before, nobody checked with the engineers first, and sure enough it made the Web service fall right onto its face. All of our customers experienced an outage.

Good old-fashioned hardware failure. United blamed its July 8 outage on a failed router. Some years ago, squirrels brought down NASDAQ by chewing through some power lines. These things happen, and most companies hedge against it with redundant hardware. But even then, sometimes a failure gets through.

Imperfect failure planning. Almost every company has failure plans in place. Most of them use as much automated failure recovery as they can. But there are just situations that evade even the best plans and the best automation.

Perfect technology is a myth. Occasional failure is certain.

Categories
Career Quality Testing

Endangered species: Managers and Directors of Quality Assurance

By Jim Grey (about)

I lost my job a month ago. The software company where I worked cut expenses to fit revenue. That’s business.

I was Director of Quality Assurance. I built and led a team of functional testers, software engineers (writing code-based tests), and technical writers, and developed the strategies and processes the company used to test and deliver its products. It was a role my whole career had prepared me for, and I was glad to have it.

But now I’m working my network, looking for my next gig. It’s been fun. I’ve spoken to a few dozen CTOs, VPs of Engineering, and even CEOs so far in my search, usually from startups and young companies.

I’ve delivered so much software using waterfall and “scrum-fall” (the developers sprint, and testers try to keep up, but final testing still happens after developers are done) at various well-established companies that I thought that full-on agile was a myth, at least here in Indiana.

Rosie
Endangered.

But the startup leaders tell me they’re all in: it’s agile, continuous integration, and continuous delivery all the way. And while all of them have or will soon need testers, they are all hedging on hiring specialized managers for them.

One CTO, a fellow I worked with several years ago, told a typical story: “I have a few scrum teams, each with one tester. But I can’t imagine ever hiring somebody to run QA as I don’t consider it to be a separate function. Testing is part of the scrum team. I’d have to think QA management jobs will eventually dry up everywhere. I’m sorry; I’m sure that’s not welcome news for you.”

When waterfall ruled the world, it made sense to have a QA department with its own leadership hierarchy. After all, the testers truly were a separate team with their own schedules, methodologies, and needs for care and feeding. But in agile, testers feel fully part of their scrum team rather than part of the team of people who test.

I admit to some anxiety over this evolution, entirely because I’m unexpectedly on the job market after several years of being Manager and Director of QA. But I think these changes are good for quality software delivery.

And these changes are creating opportunity, because there are things a good QA leader knows how to do that a CTO or VP of Engineering probably doesn’t. I networked my way into another startup CTO’s office just to introduce myself and ended up walking out with a short-term consulting job. It turns out that he’s ready to hire his first tester, but wants guidance on how testing should integrate into his team and how the tester should approach the work to find the most important bugs early. So I’m testing alongside his developers for now to get a feel for their context, and will develop a strategy and process his team can follow after I’m gone.

In my conversations, over and over CTOs are telling me they need people to coach and guide on functional testing strategy and technique, or on building test automation and performance testing practices. One VP of Engineering I’ve talked to had an unusual take: that I should be able to help him build and manage his end-to-end software delivery process, including engineering and testing, because he sees process as key to predictable and reliable software quality.

I certainly have experience in all of these areas. I imagine many Managers and Directors of QA do.

I don’t want to count my chickens before they hatch, but I feel pretty good that at least one of these threads will lead to a full-time job soon. And with it, I’ll step out of the waterfall/scrum-fall past and into a present I previously didn’t know existed.