A robots.txt file tells search engine crawlers which parts of your website they are allowed or not allowed to crawl.
It can be useful for controlling crawler access to specific pages or directories, but it does not directly improve your search rankings.
What does robots.txt do?
A robots.txt file contains rules for automated crawlers, such as Googlebot.
For example:
User-agent: *
Disallow: /private/This tells crawlers that follow the rule not to crawl URLs under /private/.
You can also allow crawlers to access your entire website:
User-agent: * Allow: /Important: robots.txt does not remove pages from Google
Blocking a URL in robots.txt does not guarantee that the URL will be removed from Google's search results.
If you want a page to be excluded from search results, other methods such as a noindex directive may be more appropriate.
When should you modify robots.txt?
Most websites don't need to change their robots.txt file.
You may want to customize it if you need to:
- Control which parts of your site crawlers can access.
- Prevent crawlers from accessing certain directories or resources.
- Add specific crawler rules.
- Manage access for certain search or AI crawlers.
Be careful when adding Disallow rules. Blocking an important page or resource can affect how search engines crawl your website.
How to add robots.txt to a Super site
Uploading files to your Super site is a Pro feature.
If you're on the Pro plan:
- Create a plain text file named
robots.txt. - Add your crawler rules to the file.
- Upload the file to your site's root directory.
- Your file should then be available at:
https://yourdomain.com/robots.txtMake sure the filename is exactly robots.txt.
Example:
A basic robots.txt file that allows all crawlers to access your site looks like:
User-agent: * Allow: /If you want to block a specific directory:
User-agent: * Disallow: /private/Only add rules that you understand and actually need.
Robots.txt and SEO
A correctly configured robots.txt file can help search engines crawl your website efficiently. However, changing your robots.txt file isn't a general SEO optimization technique.
If your goal is to have your pages appear in Google Search, make sure you aren't accidentally blocking important pages or resources.
After making changes, you can check your file by visiting:
https://yourdomain.com/robots.txtThe key takeaway
Use robots.txt to control crawler access, not as a way to improve rankings. Only block URLs or directories when you have a specific reason to prevent crawlers from accessing them.
Note: Uploading a custom robots.txt file requires a Super Pro plan.