If You Want to Favor Diversisty on the Internet, Increase Friction (+ Insights from Genetic Algorithms)
Posted by Bob Warfield on October 15, 2007
Tim O’Reilly writes about similarities between Facebook, Quant Funds, and the Techmeme Leaderboard. He says there is a danger of self-fulfilling prophecies when feedback mechanisms emphasize herd behaviour, and that this can ultimately interfere with the wisdom of crowds, which requires a diverse set of independently acting individuals to achieve this effect.
This is exactly what I’ve been writing about in my series on punctuated equilibrium. It is ironic that much of the promise held for the Internet has revolved around its ability to empower those who were previously unable to make a difference. The dark side is that the Internet is one of the most ruthlessly efficient means of allocating almost all the wealth to a few yet discovered. Why? Because it eliminates friction and in so doing reinforces herdlike behaviour.
We’ve all seen the articles about various long tail phenomena:
- The top 84 out of 5000+ Facebook apps have 87% of the usage.
- The top Google blogs show a similar long tailed curve.
- Google controls 40% of online advertising.
The list goes on. And it is normal in the course of human affairs for consolidation to happen around just a few areas. That’s always been the case, the Internet is just radically more efficient at running the Darwinian processes that lead to the extinction of diversity.
How does this happen?
Think of competition. Initially, there are many different approaches to a market. Some are better than others, and everyone in the market starts copying the better features and trying to be more like the better players. A couple of things happen. First, features start getting attributed to the better players no matter who has them, which only favors the better player more. The ultimate example is if a company name gets turned into a verb, “I’ll Google her company and see what they do.” Second, there are a set of flaws in the early competitors. Sometimes these flaws are very hard to weed out. Enter the Fast Follower. These folks take the good and leave the bad. Pretty soon, all the good features and almost none of the bad are concentrated in just a few of the players. Others that try to copy that formula are pointless–unless they bring something new to the table, they are non-starters.
This all fits perfectly well into the evolutionary model. “Punctuated Equilibrium” is when the initial set of brand new features come to a market, perhaps even creating a new market. That’s the Big Bang that brings that market into existence. The rest is just competitive forces refining the original ideas until a few emerge.
I’ve seen this firsthand playing with genetic algorithms on a couple of projects. These algorithms mimic the evolutionary process in order to solve a problem of some sort. They work well for problems that are too complex to be solved directly. Genetic Algorithms start out with a randomly generated population that has “features”. Let’s say we wanted to create on that evolves a stock trading system. Each individual has a set of rules (“the features”) that determine the trading behaviour. Perhaps there are 16 rules that vote to buy, sell, or hold. We randomly generate a set of “traders” that have different random sets of rules. We evaluate how well those organisms perform against historical stock market data and assign a score to each one.
Next, we evolve the next generation. To do so, we randomly generate some organisms. It works best if there aren’t too many randoms. Perhaps just 1/4 to 1/3. Then we take the top 1/4 or 1/3 best scoring organisms from the last generation and copy them over unchanged. Now we fill up the middle in a very clever way that is analagous to sexual reproduction. We pick pairs of organisms randomly, but skewed by score. Organisms that scored higher are more likely to be chosen–they’re the beautiful people. The ugly ducks may still be chosen, but much less likely. From the pair, we randomly pick rules until we have another 16 rules to make a new organism. This is the comingling of chromosomes. Research shows this is the most important part of the process.
Can you see why this is so much like what happens on the Internet as sites compete in a market for “fitness”. Maybe fitness is as simple as a slot on a leaderboard. You can see the feature comingling at work, although it tends not to be as random because we can measure and sense why some features are better. We keep the original market leaders in play until the new guys do better. What I can tell you is that if you run such genetic algorithm experiments, they lead to the same long tails. Just a few organisms own the shop and it happens amazingly fast.
What does it take to help diversity succeed? In the real world, it has taken dinosaur-killing events that were very violent. But there are some less violent alternatives to learn from. Islands have been repositories for independent evolution that has bred diversity. Witness the unique animals on each continent. You can do the same with genetic algorithms by creating multiple islands that don’t inter-breed, and it works well as a way to keep diversity up over longer periods.
This happens on the Internet too. The “islands” on the net are those areas outside leaderboards or other measures of fitness and scrutiny. There are interesting memes and products brewing out there. Their question is whether they have to move back into the mainstream to thrive. Look in these areas of increased friction for diversity. If there is a bubble brewing as some say, the survivors will be the ones at the top of the long tail graphs and the ones off the edge of the map on islands in the Internet.
VC’s and entrepreneurs focused on “new model” for Internet startups take heed: you’re herding together in a way that will make you vulnerable to these effects.
Widget makers are unhappy, while Performers, such as bloggers, prosper. Why? Because Performers are well equipped to create punctuated equilibrium over and over again, while Widgets are flashes in the pan that rapidly consolidate at the top.