By Jim Grey (about)
I have been astonished in my life by how few problems are truly unsolvable. I have also noticed that, most of the time, when a problem ends up not being solved it is for one of two reasons: people deny the problem, or they won’t work together on the solution.
There are any number of ways to make software. All of them involve programming, of course; it’s the core activity. But there are other jobs to perform, and the bigger the software being developed, the more people you need to perform them. It becomes necessary to organize the people and form work processes. The ways to do that range from loose and chaotic to structured but flexible to ponderous and formal. They all have their strengths and weaknesses; they all can work. I’ve made software in all of these situations.
One of my past employers sequestered its programmers in a room to hack code together as fast as they could. When they were done, a couple people would play with the product for a week or two in hopes of finding some bugs, which the programmers might or might not fix. Then we shipped it to five customers, who upon installing it invariably found that it failed spectacularly. When we fixed all the problems they found, we chose five more customers. We repeated this process until the software stopped failing, and then announced to the rest of our customers that it was ready.
At the other end of the scale, another past employer had such a heavy software development process that following it felt like slipping into a straitjacket. There were reams of documentation to write and keep with layers of approval at each step. I led the test team. We had to take a screen shot of every test step, print it, and put it in a folder. At the end of a project, we had boxes and boxes and boxes of printouts, hundreds of pounds worth, that we would send to an offsite storage facility. This was in case our only customer, which happened to be the U.S. government, wanted to audit us.
Both companies are still in business.
A third past employer carefully hired smart people and trusted them to know what to do. Because they hired well, this worked for a long time. But as the company grew, this approach became more and more difficult to manage. Product quality became a problem. A big part of the software was an accounting package, and in one fateful release the general ledger simply would not balance. This affected every customer, and to make a long story short, it took the company almost a year to fix it. Several customers quit us.
Too often it takes great pain to drive important change. This pain made us face that we needed more structure to deliver successfully. So we hired a consultant to guide us toward a better software development process. He showed us a way with just enough checks and balances that we could have greater confidence in our work without being too bogged down. We also hired a consultant to help the management team (of which I was a part) work more collaboratively. She taught us how to lead the company through this transition.
It worked. In the very first product release under the new process, quality problems fell off dramatically and we delivered on time for the first time anybody could remember.
The more important hire was the second consultant. The new methodology was good, but it wasn’t magic, and it might not have worked for us if we had not done a good job helping the team through the changes. You see, a few people didn’t understand why the old way couldn’t still work and still others thought our new process wasn’t going far enough. Our task was to overcome the resistance and get everybody singing from the same sheet of music, and it was the hardest thing we did. But we pulled it off.
It’s a rare organization that faces its failings and works to change. It’s a rarer organization still that drives change collaboratively.
2 replies on “Even a mediocre plan will work if everybody follows it”
Something I find interesting is that, as company size gets larger, it gets a lot tougher to identify the problems that most need to be solved. Each one of the groups that serves the broader goals of the business (marketing, training, documentation, BAs, support, sales, testing, HR, etc.) has their own Biggest Problem to deal with, and as every function becomes more specialized it gets harder and harder to have a clear view of the best way to move the organization forward.
Put another way: every team can’t have a perfect situation, so which teams have to accept which problems so that the company overall can be most successful?
You framed the core problem of “unsolvable problems” as a case of people denying the problem or failing to work together on the solution. I think that’s accurate, and I’d add that when groups can’t see past their own Biggest Problem then it’s easy to deny another team’s Biggest Problem, and from there it’s easy to not work together on solutions. (Why work on solving that other team’s problem? We’ve got big pain right here!)
The end result is a systemic problem for the company which goes unidentified. It’s seldom solved by accident, or when just one or two of the groups gets their Biggest Problem addressed. (And, as I think I first read in a Jerry Weinberg book, when you solve your number one problem number two gets a promotion.)
I think that’s what makes your last story an interesting example of the way of an inflection point can break this type of deadlock. With a new software methodology on the table, the game changed: the Biggest Problem of yesterday (for groups, not the company) was replaced with a big pile of uncertainty, a *shared* uncertainty across many groups. That can make it easier to work together on solutions for shared system goals, which is what it sounds like happened in your case.
I need to write a post here someday about perspective. There’s a great cover from New Yorker magazine that shows the world from the vantage point of 9th Avenue. Beyond the Hudson River, things get really sketchy.
We all do some version of that in our companies. Our perspective on which problem is most important depends a lot on where we sit in the org.
I’ve yet to figure out how to crack this nut in large companies, which I think is why I tend to leave them. It’s also why I think large companies tend to do things like hire armies of monkey testers to run through boring regression scripts. Upper management can’t see the problems that the smart testers do, and even if they did, they think they have bigger problems. So they end up throwing money at testing. It’s a boon to the tester who wants job security over exciting work.