But breakthrough innovation is not smooth. It stutters. It is often overweightand poorly groomed, with dark circles under its eyes from overwork. It comes into the world stumbling and falling, only later to gain Olympic level prowess. When we first see it, it looks like crap.

This poses a problem for organizations who seek to innovate. While they need they need to deliver for customers, investors and employees, there is a constant tension between optimization and innovation. Don’t optimize and you won’t keep up. Don’t innovate and you’ll become irrelevant. You have to do both and it’s not always pretty.

Portrait of an Innovator

Linus Pauling was an innovator of historic proportions. He is widely acknowledged as the father of molecular biology, combining biology, chemistry and physics in ways that nobody imagined possible before he came along. It’s tough to imagine modern medicine without his work.

One of his earliest and greatest breakthroughs was the discovery of the structure of hemoglobin. His work on x-ray diffraction led him to propose an unlikely helical structure that turned out to be exactly on the mark. That was the first time the mysteries of a protein of that level of complexity had been unravelled.

He was the most celebrated scientist of his day and a shoe-in for the Nobel Prize, which he first won in 1954. When he set his sights on DNA, everybody expected him to win the race. Everybody was wrong.

It was, in fact, two unknown postgraduates at Cambridge named Watson and Crick who utilized x-ray diffraction analysis to discover the helical structure of DNA. Pauling, not surprisingly, was convinced that DNA was a protein, which it was not. The young upstarts, unencumbered by previous accomplishments, had less to blind them.

Testing and Learning

The example of Watson and Crick is not uncommon. As I’ve noted before, breakthrough innovations often come from the young and unestablished. From Isaac Newton to Albert Einstein to the leaders of modern political revolutions, paradigm shifts tend to come from below, not above.

However, they also usually fail. Even those that are successful usually hit the skids a few times before they ultimately prevail (as Mark Pincus, the founder of Zynga bravely points out in this article). You don’t make an omelet without breaking a few eggs.

Venture capitalists understand this. They expect that most of their investments will go bust and expect to gain great overall returns on the few that make it. Established corporations are starting to get it too, many of whom have instituted test and learn programs.

The reason is that prediction is difficult, for a variety of reasons. The only thing we do know is that it is likely to be different from the past. So if we want to be successful, we have to learn to accept and deal with failure.

The Difference Between Strategy and Innovation

One of the major difficulties regarding innovation is that it’s often confused with strategy. That’s a huge mistake.

Strategy is a high level function, which concerns the overall direction of the company and therefore necessitates considerable involvement of senior management as well as, at times, high-priced consultants. The outcome is a plan that puts relative strength against relative weakness or opportunity.

Innovation has nothing to do with overall direction. It involves building a portfolio and hedging bets. You go into it knowing that most of what you do won’t work out. You continually fail, dust yourself off, pick yourself up and try again. Keep at it and you will succeed eventually. To survive is to thrive.

Think about it and you can see the problem. Once you have too much involvement by senior management or high-priced consultants, you’re chances of innovating are drastically reduced. The only way to fail your way to success is to fail cheaply.

Innovation is a Dirty, Messy Business

That’s what brings us to the truth about innovation. It’s a big word, but it’s a dirty, messy business. While it may conjure up visions of Steve Jobs wowing the crowds at Macworld, the reality is confusion, late nights and petty squabbles over minutiae. We glamourize the 1% inspiration, but forget about the 99% perspiration. After all, perspiration stinks.

That’s why so few companies can innovate well. It is so obviously important that it cries out for heavy senior-level involvement. However that diminishes the chances of success because it hinders the ability to fail cheaply. Big egos don’t admit mistakes easily. Hefty consultancy fees don’t allow for uncertainty.

There are some viable solutions. 3M has had decades of success with its 15% rule. P&G has learned to outsourced failure with its connect and develop program. Innocentive has created a marketplace for messiness and corporate innovation labs, which focus on testing and learning, are coming into vogue.

Any successful innovation effort begins with the realization that the path to innovation is not a straight line, but a drunkard’s walk.

Greg Satell is a U.S.-based independent media analyst.You can read his blog entries at http://www.digitaltonto.com