HIDING? Yes, you heard me right, I said hiding from search engines! Let's take a look at a few of the ways you might do that without meaning to do so. Hide and seek! Peekaboo!

Secure Server

Search engines do have the ability to spider secure server hosted pages, but often these pages require either that a visitor fill out a form or log-in with a password and user name before being allowed past a certain point. If any page requires filling out of forms or passwords to reach, search engine robots will simply leave. They can't log in because they can't fill out forms, leave email addresses or enter passwords.

I was contacted by a webmaster for a 4500 page ecommerce web site. He wondered why search engines were ignoring such a large site. I asked the URL of the site and visited the home page. I noted that upon loading, there was an immediate passing of the URL http://anybusiness.com site to a secure httpS://anybusiness.com page. This has two immediate faults that may be a problem - the forwarding method and different server. If the instant forward is by javascript, bad news.

First, search engines often either penalize or downgrade sites that use immediate URL forwarding, especially from a home page. URL forwarding suggests doorway pages (a search engine no-no) or affiliate URL's forwarding to an affiliate program site, or the worst of all scenarios, cloaking software on your server. You may not be doing any of these things, but the robots don't know, don't care, and don't index your site, plain and simple.

Secondly, secure servers are very often a separate web site, meaning that the secure server is actually a different machine and is an entirely different site from the non-secure server site unless your site is hosted on a dedicated server on it's own IP address, security certificate at the same domain. This can happen when secure shopping carts are hosted by a third party host so that a small ecommerce site needn't purchase a security certificate or set up complex shopping carts.

For example, if your shopping cart is hosted by Yahoo stores or other application service providers (ASP's), pages hosted in the shopping cart don't reside on your domain and can't be recognized as pages on YOUR site unless you also host your domain with the same company. Unfortunately, many shopping cart ASP's use dynamic IP addresses (IP address is different each time you visit) and use database generated dynamic pages.

The process of serving dynamic pages is not the problem. The problem is simply that the URL of those pages contains several characters that either stop or severely curtail search engine spiders. Question marks (?) are the biggest culprit, followed by ampersands (&), equal signs (=) percent symbols (%) and plus signs (+) in the URL's of dynamic pages.

These symbols serve as alarm bells to the spiders and either turn them away entirely or dramatically slow the indexing of your pages. This is stated simply in the Google Information for Webmasters page.

"Reasons your site may not be included.

"Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index."

Just because your site is dynamically generated, creating long URL's full of question marks, equal signs and ampersands like
www.domain.com/category.asp?ct=this+%28that+other%29&l=thing doesn't mean you are in search engine limbo. There are simple solutions available for your webmaster. Here are a couple of articles explaining an elegant solution called "mod_rewrite".

You can read about that technique if technically inclined:

http://alistapart.com/articles/urls/
http://alistapart.com/articles/succeed/

This technique is simply creating a set of instructions for your web server to present URL's in a different form that replaces those "bad" question marks and ampersands with slash marks (/) instead. The method will require that your webmaster is a bit more technically savvy than most home business CEO's who created their own web site. Some hosts will help here by simply turning on the "mod_rewrite" for shared hosting clients.

Don't play hide and seek with the search engines! Tell them EXACTLY where to find every page on your site and if there's any question that they will find every page on your site, give them a map.

A Site Map

Hard code those dynamic URL's for most categories within the categories of different sections of your web site into your comprehensive site map. As long as those dynamic links (even those that include ?=+%& symbols) are hard coded into a site map, the spiders will follow them. Clearly those 4500 pages mentioned earlier would be too much for a site map listing. But the main category pages could be provided for the engines.

I visited the site map page of the webmaster mentioned above and saw fourteen pages listed on the site map. That explains why they have fourteen pages, not 4500, indexed by Google.

How to find out how many pages of your site are indexed? Go to Google search and type "allinurl:www.domain.com" without the quotes. Replace "domain" in the above example with your own domain name. This query operator will return a list of every page of your site. Look in the blue bar across the top of the Google results page and you'll see the number of pages indexed at your site! That should do it. Get indexed and stop playing hide-and-seek!