robots.txt FileTo exclude all Robots From The Entire Site
This article is from our archive. It is kept as it was first written, so some steps or versions may be out of date.
If you don't want to let your sites get indexed on search engines, you can disable access via robots.txt file.
Create a file named robots.txt and upload it to your root directory of domain.
User-agent: *
Disallow: /
To exclude all robots from certain directories robots.txt:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/