Here is what could happen if your server returns incorrect code for “page not found”. A forum user reports that Google has ignored hist custom 404 pages and it now says he has 2.6 million pages, which he don’t.

What’s more disturbing is the fact that Google has dropped his rankings to the level that even when he type in his own domain name, he still only rank 20th or so. On Yahoo!, the site rank #1 for hundreds of keywords though.

This reminds me of the same problem I have found on one of the WordPress plug-ins I use on my blogs. The plugin failed to work for new version of WP and after fixing the code, I successfully had it displayed the right pages.

The problem is, the HTTP header still returned 404 to the browser, although the page has been displayed correctly. This is kind of like having custom 404 pages for every pages that use the plugin.

Fortunately, I submit Google Sitemaps for this domain, of which quickly I could identify this problem and fix the code again.

Here are what I suggest if you have similar problems:

1. Submit sitemap file to Google Sitemaps so you can track how Google think about your domain pages. The information often is pretty useful and quite surprising. For example, using another plugin that tracks outgoing traffic using Google Analytics makes Google crawls <domain>/outgoing/ directory, which never exist. I will take a look at this later if I have time. It doesn’t affect anything so far so I keep it in a lower priority.
2. Make sure that the server returns correct response status code in the header. The tool I use is a Firefox extension called Live HTTP Headers. This is one of the extensions that I think is a must for every SEO out there.

Examining the headers should immediately reveal the culprit for this kind of issue. On one of my new domains, I have Google robot visited the page several times but failed to index the pages. I fix the 404 code, add more inbound links and most of the pages are in the index almost overnight. A coincidence? Maybe, but I would think otherwise. Site structure and function obviously matter too.

[SEO Chat]