by Mike Banks Valentine copyright
© 2003
Oh, for the simplicity of the good old days of 1995!
Where all a web site owner had to do was put important
keywords about her web site in the metatags of the HTML
code of the front page of her site with a clear description
and title to rank well in the search engines and make
it easy for web searchers to find her.
What made it become so difficult to gain and maintain
a bit of internet visibility? It got crowded on the
web. Crowds made it more difficult to get noticed -
so instead of simply including proper metatags, web
site owners had to begin to make themselves stand out
to get noticed. Many did that by resorting to trickery,
like keyword stuffing and link farms. Since then, it
has become increasingly difficult to get a web site
indexed and ranked well at the major search engines.
Strategies have evolved that seem to work, then mergers
and partnerships change and so do all of the rules.
It never ends and the confusion grows with each passing
day.
It seemed that paid inclusion and pay-per-click were
the only ways to get your site listed anywhere besides
Google lately, let alone ranked well.
LookSmart has come up with a brilliant idea to grow
their own index that, if it catches on, will offer huge
benefits to both LookSmart and to the participants in
the program. That seems to be the elusive win-win situation
that all but Google have been looking for. It's called
distributed web crawling and gives web site owners a
hand in growing the Looksmart index as well as the ability
to include their own site in the results!
This win-win situation requires participants to download
the "Grub" distributed crawler client to their
PC and allow it to run when the computer is otherwise
idle. The software to run a LookSmart Grub web crawler
is available at:
http://www.grub.org
It is based on open source software and is made available
to all to distribute and even modify! Since installing
"Grubby" on my own PC three days ago, I've
personally contributed over 100,000 URL's to the LookSmart
database!
It's fascinating to watch and the GUI is both entertaining
and attractive. It can run invisibly when minimized
in the system tray or you can sit and watch as the URL's
fly by during a crawl from within two available views
on your PC desktop.
I love the concept and plan to continue helping LookSmart
grow their database. I've also taken advantage of the
option to have "Grubby" crawl both my own
sites and those of clients by including a small text
file with a digital ID in the root directory of each
server I have access to.
There are no drawbacks here, just benefits. I help "Grubby"
to crawl the web for LookSmart and Grubby helps me to
get my own sites listed there by agreeing to crawl them
regularly.
No payment involved other than the donated bandwidth
and the PC computing power that would be running a silly
screen saver otherwise. They even provide a cool screensaver
of their own that simulates the crawling process with
colorful orbs that represent the sites you are crawling
to help weave the web! When you visit the front page
at the URL above, you see the latest statistics on your
contribution to crawling the web.
They've even introduced a measure of competition into
the program by showing top contributors and ranking
them. Sunday the stats at the top of the page read,
"323 clients running - crawling 24,355,058 URLs
in the last 24 hours." It will be interesting and
very telling to see if that number grows significantly
or very soon because that will be some measure of the
success of the program over time.
This process and the software all seem to harken back
to friendlier times on the web where community mattered
more than profit. This could even threaten the mighty
Google if it spreads, since we all love to participate
in things that interest us and LookSmart has a really
interesting little Grub helping to grow the web from
my home office and yours.