Tech & VC 18 Aug 2006 05:12 pm
Could Digg Ever Rival a Search Engine?
I wonder if Digg could ever build up enough posts and acquire new posts at a fast enough rate that it could compete with a specialty search engine, like Technorati?
As long as users are posting articles by hand, Digg will never have enough articles to rival any search engine, but if a crawler could be created to automatically post articles to Digg with a decent summary and title (which would require some NLP, and AI), then perhaps Digg could be a search engine where instead of pagerank as a ranking, the ranking would be popularity based on votes. That’s all that pagerank is now anyway, a proxy for popularity. So, why not cut out the proxy?
Of course, the crawler would make or break the implementation. If the fresh diggs posted by the crawler were total garbage, nothing would ever get traction because any half-decent post would drown in the noise, and digg would crumble. But if the crawler could discern some interestingness algorithmically (see last post for more on that topic) and post only interesting articles, then the signal to noise ratio could be high enough that really great posts could get traction in voting. The secret sauce in the crawler could make digg comprehensive without drowning users in a sea of adsense leeches and spam.
10 Responses to “Could Digg Ever Rival a Search Engine?”
on 20 Aug 2006 at 5:16 am 1.fred said …
that was where we were going with delicious before it was sold to Yahoo!
i think that’s an obvious thing to do with an index of URLs that is user generated
try searching the web here
http://del.icio.us/search/?
on 21 Aug 2006 at 6:35 pm 2.Miles Shapiro said …
The Democracizing of Interestingness is an interesting idea in and of itself. But it is kind of like baseball statistics. The further back in history the more radical the variance, and the statistics are more notable because of the variance– if and only if you factor in the steroid cheating. I think the Sabermaticians refer to it as “reversion to the mean.” I think our democracy of interestingness has gone through a similar process. Britney Spears is interesting to the whole nation at large. So is Jessica Simpson and Linsay Lohan and everyone else you see online at the grocery store. They’re intesting according to this democratic means of determining interest. It’s worse in our country’s politics (see: present day Oval Office). Mediocrity reigns via tyranny of the majority. Because these websites are by similar standards still in their inception, the variance in the “interest quotient” is then still quite wide, but time takes its toll on mountains and statistics I guess.
on 21 Aug 2006 at 6:49 pm 3.Andrew Parker said …
Terrific comment Miles. Would you then expect that as Flickr grows larger that its interestingness will also revert to the mean? I suspect not, and whatever they do to prevent that problem is what would need to be leveraged here too.
on 03 Sep 2006 at 8:46 am 4.The Gong Show :: Netscape’s Digg-like Search Implementation said …
[...] A few weeks ago, I wrote a post wondering if Digg could ever be a competitive and comprehensive search engine. Jason Calacanis has taken the first step by incorporating Netscape News results on top of Google results in the Netscape search page. It’s worth a look, but it isn’t even close to being a Google replacement. I still wonder if a bot that automatically crawls and adds stories to Netscape would be able to take this search implementation to the next level. Good luck Jason; I’ll be watching to see in what directions this project evolves. [...]
on 03 Oct 2006 at 9:02 am 5.Adam said …
delicious was sold to Yahoo? I guess I missed that headline!
:(
on 03 Oct 2006 at 10:47 am 6.Mike said …
What in the world makes you compair a search engine with a website?
Its apples and oranges
on 03 Oct 2006 at 1:37 pm 7.Andrew Parker said …
Looks like this post got dugg, and I have been officially defamed in the comments of the digg post. Brutal. To set the record straight for anyone else coming from digg. I’m not a VC, though I do work in VC. Furthermore, I don’t necessarily think this is going to happen; it’s a thought experiment. Ponderings… Anyway, glad to know my hosting can survive being dugg.
on 03 Oct 2006 at 5:42 pm 8.Digg at The Gong Show said …
[...] My post on whether or not Digg could be leveraged as a search engine was the target of the Digg. It’s not a particularly good post. It obviously made the front page because it’s a sensational idea to the Digg community. I don’t think more than 20% of the commenters over at Digg actually read it, considering the contents of their comments. [...]
on 04 Oct 2006 at 5:25 am 9.EveryDigg » Blog Archive » Could Digg Ever Rival A Search Engine? said …
[...] VC Andrew Parker debates whether or not digg could ever compete with specialty search engines, if it could gather enough meaningful articles at a rapid pace…read more | digg story [...]
on 26 Oct 2006 at 10:50 pm 10.Mexico501 » Blog Archive » Could Digg Ever Rival A Search Engine? said …
[...] Page Summary: So, why not cut out the proxy?Of course, the crawler would make or break the implementation. If the fresh diggs posted by the crawler were total garbage, nothing would ever get traction because any half-decent post would drown in the noise, and digg would crumble. The secret sauce in the crawler could make digg comprehensive without drowning users in a sea of adsense leeches and spam. Would you then expect that as Flickr grows larger that its interestingness will also revert to the mean.read more | digg story [...]