Who Hides Data From Search Engines And Why?
Posted by Bob Warfield on April 11, 2008
Fred Wilson just did an interesting post about delicious and its traffic after being acquired by Yahoo. On the face of it, looking at Compete or other stats, it appeared delicious had gone into decline after being acquired. Reality is a bit different as Joshu wrote back to Fred:
We continue to grow normally.
Unique users is not a good measure of our growth, though.
Much of our traffic is through the firefox and other browser extensions, which is not measured by these systems.
Additionally, we cut off search indexing several months ago, which also hurts the UU numbers.
It is fascinating to consider just how much of the web is not measurable even today due to such things as running traffic through browser extensions. What I really found interesting was the last line, though.
Why would delicious cut of search indexing?
As one commenter on the thread pointed out:
I would think Joshua would be delighted if the “funny-video” tag was the first search result at Google for the search team “funny video”… it would mean both greater distribution and influence.
Evidently not. There is no good cost reason to turn away web crawlers. For a property like delicious, crawling have to represent a tiny fraction of their traffic. It seems to me the reasons would have to be strategic.
For the conspiracy theorists out there, consider this. Perhaps Yahoo is doing this for a lot of their valuable properties and only letting Yahoo search engines index the data. This means a Yahoo search can find delicious posts but Google can’t.
It seemed a great theory, but alas I could not verify it at all. I tried half a dozen of the most popular posts on delicious and could not find them listed on searches for the title on either Google or Yahoo search.
So I’m stumped. It would be fascinating to see a list of sites that exclude search engines sorted by popularity. Even more fascinating would be understanding why they exclude the searchers.
Update: It seems that Yahoo does do something special with delicious, but maybe it isn’t all in production yet based on my limited testing. See TechCrunch for more.
Don’t you think this is a tacky way to compete? By limiting availability of search results? Search result integrity is essential, and here we have companies outright trying to sabotage the search results of their competition. This will get worse if MSFT has anything to say: they don’t believe in a fair fight!