If you don’t want to let your sites get indexed on search engines, you can disable access via robots.txt file.
Create a file named robots.txt and upload it to your root directory of domain.
User-agent: *
Disallow: /
To exclude all robots from certain directories robots.txt:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
