Robots.txt and SEO - The Ultimate Guide from the Experts

Your “robots.txt” file, one of the most important files within your website, is a file that lets search engine crawlers know if they should crawl a web page or leave it alone.

One of the main aims of a robots.txt file is to ensure your website servers are not overloaded with search engine bots crawling your website at all hours of the day, but did you know that there are also SEO-related benefits that come with a properly optimised robots file?

Keep reading this guide to understand more about robots.txt, the best practices, and how this helps your website’s SEO efforts.

What Does the Robots.txt File Do?

A common misconception is that a robots.txt file prevents a page from getting indexed by search engines.

That’s not true.

Google says, “If other pages point to your page with descriptive text, Google could still index the URL without visiting the page”. If Google indexes a page disallowed by robots.txt, it won’t have a description because they weren’t allowed to visit the page.

When you want to prevent a page from being indexed, use the noindex tag. Alternatively, lock the page with a password to prevent Google from indexing private information.

The default assumption is that crawlers can crawl, index and rank all of the pages on your website unless they are blocked with a disallow directive. So, if a robots.txt file doesn’t exist or isn’t accessible, search engines will crawl, index and rank every single page they can find on your website as if there were not any restrictions in place.

Robots.txt Syntax and Formatting

Robots exclusion standard (aka. Robots.txt) should follow a set of rules to be valid and used by search engine crawlers.

The structure of a robots.txt file includes:

• Group: Names the user-agent and directives that it must follow. A robots.txt file can have as many groups as you want, but most user-agents will only follow the first groups that apply to them.
• User-Agent: The identification of a crawler. Naming the user-agent in the robots.txt the user-agent the directives set for them. For example, you can name Googlebot as one user-agent and name the Pinterest bot as another.
• Directives: These are the instructions that each user-agent named in the same group has to follow.
• XML Sitemap: It’s possible and common for robots.txt files to mention the sitemap because it makes it easy for search engine crawlers to find the file.

Crawlers usually process groups from top to bottom, but Googlebot and Bingbot will default for the most specific rules as they are usually less restrictive.

User agents can only follow one group, and you should avoid having contradictory directives for the same user agent. If a group of directives targets a user-agent more than once, they will likely ignore it, following only the first group they found on the robots.txt.

You can also use the robots exclusion file on subdomains (for example, www.mydomain.com/robots.txt or blog.mydomain.com/robots.txt) or non-standard ports ( mydomain.com:8181/robots.txt).

The main rules of the robots.txt file:

• Must be UTF-8 encoded.
• Must be named “robots.txt”.
• Must be located on the root of the domain.
• It will only be valid for the same protocol (HTTP or HTTPS) and subdomain (for example, www or non-www) where it’s located.
• Should use only relative paths (except for the sitemap).
• Must have only one directive per line.
• Directives are case-sensitive.
• Comments start with # and don’t get read by crawlers.

Robots.txt Directives

Different search engine crawlers, no matter whether it is Googlebot or Bingbot, for example, will follow each directive relevant to them within a robots.txt file to ensure they understand which pages on a website they are allowed to navigate to. This allows them to understand which pages on a website can be crawled, indexed, and ranked.

It is worth noting that not all crawlers support the same directives or even interpret the syntax of a directive in the same way.

Googlebot is one of those user agents that doesn’t support all directives. But before explaining those in detail, let’s first see the list of all directives:

• Sitemap.
• Disallow.
• Allow.
• Crawl-delay.
• Noindex.
• Nofollow.

All directives, except the sitemap, support wildcards from RegEx for the entire string, prefix, or suffix. The directives should start with a slash (“/”) when referring to a page and finish with “/” when referring to a directory.

Sitemap Directive

The sitemap directive shows the URL where the XML sitemap of a website is, thus making it easier for crawlers to find them. This directive is supported both inside and outside of groups. Unless you have a specific sitemap for a particular bot, it’s better to declare it at the beginning of the robots.txt, so all crawlers can use it.

As mentioned in our guide to XML Sitemaps, they aren’t mandatory, and if you’ve already submitted it on Google Search Console this directive can be redundant. However, declaring the XML sitemap doesn’t hurt you and makes it easier for other user agents like Bingbot to find it.

Example showing usage for Sitemap directive:

Sitemap: https://mydomain.com/sitemap.xml
User-agent: *
Disallow: /admin

Disallow Directive

The disallow directive tells crawlers they aren’t allowed to visit the URL or matching expression (when using RegEx). This is the directive you would be using more frequently in your robots.txt file as, by default, there aren’t limitations on pages bots can visit.

In the example below the disallow directive is not allowing bots to crawl the admin pages on a WordPress site, and it would look like this:

User-agent: *
Disallow: /wp-admin/

Allow Directive

The allow directive tells crawlers they can visit and crawl a URL or matching RegEx. This rule is mainly used to overwrite a disallow directive when you want bots to crawl a page from a blocked directory.

An example could be allowing crawlers to visit the login page but not all the admin pages of a WordPress site.

User-agent: *
Disallow: /admin/
Allow: /admin/login

Crawl-Delay Directive

The crawl-delay directive limits how frequently crawlers visit URLs to avoid overloading servers. Not all crawlers support this directive, and they can interpret the number of the crawl-delay differently.

Example:

User-agent: *
Crawl-delay: 1

Noindex Directive

The noindex directive in robots.txt prevents URLs from getting indexed. However, Google ended support for it in 2019 as they never documented it.

One of the main reasons for this is staging websites. Naturally, whilst a website is being built in a staging environment, almost all of the pages are noindexed to ensure Google and other search engines leave them alone during the build. Once a website goes live, it’s entirely possible for someone to forget to edit the robots.txt file and leave sections of the website set to noindex.

Nofollow Directive

The nofollow directive tells crawlers to not follow links in a URL. This is similar to what the nofollow tag does, but instead of doing it for a link, it applies to every URL in the page. Google doesn’t support this directive as they announced in 2019 (same announcement as noindex above).

Supported Robots.txt Directives by Google Crawlers

Googlebot only supports the following robots.txt directives:

• User-agent.
• Disallow.
• Allow.
• Sitemap (when mentioned outside a group).

How Google Interprets Robots.txt Directives

Google has extensive documentation about how its crawlers interpret directives from robots.txt files.

Robots.txt Example Rules

Disallow Crawling a Directory

User-agent: *
Disallow: /grandma-recipes/

Block Access to a Single Crawler

User-agent: annoying-bot
Disallow: /

Disallow Crawling of a Single Page

User-agent: *
Disallow: /best-grandma-cookies

Block all images on your site from Google Images

User-agent: Googlebot-Image
Disallow: /

Block a Specific Image from Google Images

User-agent: Googlebot-Image
Disallow: /images/cookies.jpg

Disallow Crawling a Specific File Type

User-agent: *
Disallow: /*.pdf$

Best Practices

Using Regex to Simplify Directives

The robots.txt file supports the use of RegEx. This will make declaring the instructions in the file simpler because you can group instructions into one expression instead of writing one directive for each URL.

Use Each User Agent Only Once

Most crawlers read the robots.txt from top to bottom and follow the first applicable group for their user agent.

Be Specific with Directives

Being specific in the robots.txt pays off and prevents unintentional consequences of bots not crawling essential sections on your site.

FAQs

Do you need to use robots.txt?

Robots.txt isn’t a ranking factor or a requirement for having good organic results, and most websites won’t notice any difference if they don’t use one.

What are the limitations of a robots.txt file?

The main limitations of robots.txt are the following:

• Not all search engine crawlers support the same directives.
• Different crawlers interpret syntax in the file differently.
• Disallowed pages can still be indexed.
• Badly behaved bots can ignore the instructions of robots.txt.

What happens if you don’t use robots.txt?

If you don’t use a robots.txt file, crawlers will interpret that there aren’t limitations on the pages they can visit and index from your site.