Robots Exclusion Protocol: now with even more flexibility “Details Last updated Februar 17, 2012. Webmasters — Google Developers Playing with the X-Robots-Tag HTTP header “Details On moz.com>learn>seo>robotstxt How to stop search engines from crawling the whole website? “Details Apache HTTP Server Version 2.2 httpd.apache.org>docs>2.2>mod Header set X-Robots-Tag “noindex, noarchive, nosnippet, nofollow, noimageindex, noodp” www.intelligentpositioning.com/blog/2009/08/x-robots-tag-control-google-indexing-via-http-headers/ http://noarchive.net/xrobots/