robots.txt FileTo exclude all Robots From The Entire Site

If you don’t want to let your sites get indexed on search engines, you can disable access via robots.txt file.

Create a file named robots.txt and upload it to your root directory of domain.

User-agent: *
Disallow: /

To exclude all robots from certain directories robots.txt:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/