So it’s not surprising that the path to innovation is shrouded in mystery, something only to be undertaken by those dazzling intelligence, incredible daring and a certain flair.

However, in truth, there is an astonishing amount of agreement among experts on how to go about it. From scientists, to economists to entrepreneurs, the same themes repeat themselves with almost metronomic regularity. So while the endpoint of any innovative process is impossible to determine, the path to innovation is well paved.

Purpose

The distinction between scientific discovery and innovation is an important one. Science is often serendipitous. Penicillin, x-rays and scores of other scientific phenomena have been discovered by chance, in the the process of searching for something else. Sometimes, profound things simply pop up in strange places.

Innovation, however, is not science nor is it strictly discovery (although that’s important too). It is, in the words of Brian Arthur, the harnessing of phenomena for a specific purpose. So when Alexander Fleming found one of his bacteria cultures was contaminated with a fungus, he made a scientific discovery. Purposing the penicillin mold as an antibiotic was an innovation.

Therefore, the first step to creating innovation is to have a clear objective. That could be to produce a faster computer chip, organize a more efficient distribution system or write more beneficial trade legislation. The main thing is that it has to solve a problem. If it doesn’t, it’s not innovation.


Variation and selection

Great innovations don’t arise from grand plans, but through trail and error. As I noted in an earlier post about how technology evolves, that means that evolution tends to follow a Darwinian path, namely variation and selection, rather than strategic planning. As Thomas Edison once said, it’s 1% inspiration and 99% perspiration.

This has long been taken on faith in innovation circles, but researchers Lu Hong and Scott Page actually put the idea to the test. In this paper, they describe how a diverse group of “dumb” agents can outperform a group of smarter, but more similar agents in solving complex problems.

However, merely having a diverse group of ideas isn’t enough. They need to be tested. You have to see which ones work and which ones don’t. That’s the selection part. There is a catch, though. People shouldn’t get killed for ideas that die. As a senior executive at Google once proudly told me over coffee, “We learn from every failure.”


Combination

No idea lives on it’s own, but succeeds in a particular context. It is through combining technologies that we create the new and exciting.

One famous historical example is the discovery of genetics. In 1865, when Gregor Mendel published his groundbreaking study of inheritance of characteristics in pea plants, it went nowhere. It took nearly a half century before the concept was combined with Darwin’s natural selection to unleash a torrent of innovations in the medical and scientific worlds.

A more recent example is the Apple ecosystem. There were plenty of digital music players around when Steve Jobs launched the i-Pod, but he combined his player with i-Tunes, which made content both more accessible and palatable to music companies. He then threw new products into the mix – the i-Phone and i-Pad – creating more combinations and greater value.

An important thing to understand about Apple is that they are not a great technology company. In fact, their R&D budget is small by industry standards and they generate few new discoveries. Their enormous success stems more from how they combine technologies to create the world’s most innovative products.


Minimize switching costs

One often overlooked aspect of innovation is that it asks people to change their habits, something they are loathe to do. Anybody involved in technology for a sustained period of time understands the paradox: technology and competition move much faster than consumers do.

Tim Kastelle, a professor who studies innovation points out that switching costs can be a major obstacle to innovation. He gives the salient example of the Princeton University library, which created it’s own classification system just a few years before the Library of Congress launched their standard.

Nevertheless, it took Princeton 120 years to make the switch, even though they recognized the one they had created was inferior. At any given time, the burdens of switching trumped the benefits of a better system.

So, when you want to launch an innovative product, even if it is far superior to existing technology, it’s important to understand that you need to ease the burden of switching. There will be barriers – both real and cognitive. So those obstacles need to be lowered, through early announcements and education, and clear benefits need to be sold through.


Sustainable failure

Finally, and perhaps most importantly, it is crucial to understand that if you seek to innovate you will fail – repeatedly. So rather than swing for the fences every time, you have to be able to sustain failure.

That means that you can’t marry your ideas. You start small. See what gains traction. Be prepared to reverse direction once you realize you are headed down a blind alley. You have unlimited bites at the apple, as long as you don’t try to swallow it whole.

And that’s the great thing about innovation. It never has to end, but is a path that you can continue down for a lifetime. If you’re lucky, others will carry the torch for you even beyond that. We do, as Issac Newton remarked, stand on the shoulders of giants and if we are true to our purpose, others can stand on ours.

In that sense there are no limits, except of course, the ones we build for ourselves.

Greg Satell is a blogger and a consultant at the Americal online media Digital Tonto. You can read his blog entries at http://www.digitaltonto.com