Unwanted Web Pages

As web sites change and grow, there will be times when some pages are no longer needed. If you just delete the page, however, search engines may still be sending traffic to it. This will result in visitors getting an error message that will probably cause them to move on to another web site. You lose the visitor and the potential sale.

I’ve previously discussed using a 301 redirect to send visitors to a different page. But, there are other options.

Let’s say a product has been discontinued and you no longer need the page describing that product. But, that page is still getting a significant amount of traffic from search engines. If you put a 301 redirect to the replacement product you might confuse people—they won’t understand why they clicked on a link for one product and they are on a page discussing another product.

What I like to do is replace the old page with a new one at the same URL. The new page explains what has happened. It features the old product in such a way that visitors immediately see they have found what they are looking for. But, it also explains the old product is discontinued and it offers recommendations for alternatives. Never give visitors the unexpected. Meet their expectations and then direct them along the course you’d like them to take.

Another alternative, that is a technique you can use in conjunction with the above, is to use the robots.txt file. It allows you to tell Google, Yahoo and the other search engines that you’d like a page (or entire folder) removed from their index.

To remove a page from Google include the following in your robots.txt file:

User-agent: Googlebot
Disallow: /foldername/page.html

To remove all files of a specific file type (for example, .gif) from Google, put the following in your robots.txt file:

User-agent: Googlebot
Disallow: /*.gif$

For Yahoo replace “Googlebot” with “Slurp”.

If you’d like to tell all search engines to remove a page from their index, replace “Googlebot” with an asterick (*). For example:

User-agent: *
Disallow: /foldername/filename.html

Or to have them ignore a folder

User-agent: *
Disallow: /foldername/