Setting your Robots.txt file for your SuperSite 2

What is a Robots.txt file?

The Robot Exclusion Standard (Robots Exclusion Protocol or robots.txt protocol) is a convention to prevent search engines and web robots from accessing all or part of a website which is otherwise publicly viewable. It can be used in conjunction with Sitemaps, a robot inclusion standard for websites.

A Robots.txt file is an easy way for webmasters to inform search engines and web robots about pages on their sites that need to be excluded from indexing/crawling. The Robots.txt file must be placed in the root of the website heirarchy, for example yourdomainname.com/robots.txt. This text file should contain the instructions in a specific format mentioned below. Search engines and web robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist, they assume that the web owner wishes to provide no specific instructions.

The above instructions tell the search engines and web robots to exclude https://Your-SuperSite2-Domain-Name.com/admin from indexing/crawling.

You can also provide instructions regarding the Sitemap of your SuperSite 2:

To Save/Edit your Robots.txt file for your SuperSite 2

1. Login to your SuperSite 2 Admin Area. See details

2. Under the My Languages section, click the Manage Site button individually besides the language for which you want to save/edit the Robots.txt file.

   Note

   You may set a Robots.txt file for each Live Site configured in your SuperSite 2.

   Additional Information

   Live SuperSite 2

3. Click the Edit Content button.

4. Click the Edit link besides the Robots.txt file.

5. Place the instructions in the Your Content textbox.

6. Click the Save Changes button.