Tuesday, March 1, 2011

Robots.txt is CaSe SeNsItIvE :(

I discovered today that robots.txt is Case Sensitive. Here is a good post describing options. http://www.slicksurface.com/blog/2007-04/be-careful-robotstxt-is-case-sensitive

So the long and short of it is that you can get better exclusions of your URLs if you also use meta tags on pages that you do not want indexed. However there is no sure fire way to make sure that a page never gets indexed (except of course to remove it from your website :)