Google, Wherefore Art Thou Google? Sites Abandoned by Googlebot! © August 30, 2004
As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to
site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.
Two clients that hired me for recent work saw their rankings shoot to
top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve
same instant ranking boost overnight.
Those clients that do succeed in this way are often thrilled with
results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.
WHY DO NEW SITES SUFFER?
What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in
form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet
site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all
titles, tags & page text, created a complete site map and placed links to all these resources on
front page.
I know that
site is not being crawled because Google's cached copy of
front page shows it before we did
work four months ago, without
new links and without title tags. We've submitted
site by hand, (manually) once a month for three months via
Google Add URL page. http://www.google.com/addurl.html When
hand submission failed to get it re-indexed for four months, we submitted
sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!
Part of
problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to
site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in
URL. This is to avoid crashing
server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1