Editing your robots.txt file
The robots.txt file is used to tell search engine crawlers how to crawl your website, and which pages of your website they should show. Using this file correctly can help boost your SEO by reducing the time it takes for search engines to search through and find the important parts of your website. Additionally, it also helps the crawlers to ignore duplicate parts of your website which could currently be hurting your rankings on search engines.
To customise the settings and contents of your robots.txt file select from the menu. Find the ‘Robots.txt File’ section:
In this section you can choose if you want the file to be generated automatically by the platform or manually by you.
If you select automatic generation you can also choose whether or not search engines should crawl and list your website at all:
If you select manual generation the section will change so you can input the contents of the file:
Warning!
Be careful when manually configuring your robots.txt file as mistakes can lead to large portions of your website being unable to be accessed by search engines.
Once you are finished editing your file select save changes.