Google, Wherefore Art Thou Google? Sites Abandoned by Googlebot! © August 30, 2004
As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.
Two clients that hired me for recent work saw their rankings shoot to top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve same instant ranking boost overnight.
Those clients that do succeed in this way are often thrilled with results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.
WHY DO NEW SITES SUFFER?
What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all titles, tags & page text, created a complete site map and placed links to all these resources on front page.
I know that site is not being crawled because Google's cached copy of front page shows it before we did work four months ago, without new links and without title tags. We've submitted site by hand, (manually) once a month for three months via Google Add URL page. http://www.google.com/addurl.html When hand submission failed to get it re-indexed for four months, we submitted sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!
Part of problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in URL. This is to avoid crashing server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1