MovableBlog: Grub Distributed Search Engine

Nuance 2.0

March 19, 2003

Grub, an open-source decentralized search tool for the Internet. From the docs:

By having websites crawl their own content, and having volunteers donate their bandwidth and clock cycle resources, it decreases bandwidth consumption across the Internet dramatically, allows for pre-processing on the resulting data, and ultimately improves search results sent to end users."

Emphasis mine. Somehow I don't think bandwidth will be decreased dramatically, but rather merely redistributed. It also apparently respects robots.txt. Mark Pilgrim has yet to pronounce that it is a unwanted robot from hell and/or include it in his robots.txt. [first link via Scripting News]

Posted by Richard at 11:03