Interactive: robots.txt

Basic exclusion

Lesson URL:

Copy Lesson URL

Teacher: Will Critchlow

You can view this introductory lesson for free. To view the whole module you need to sign up for a paid account. Alternatively you can try out our three demo modules by signing up for a free demo account.

There are a variety of ways to control the behavior of search engine crawlers. You can learn more about the alternatives in our technical SEO module. Robots.txt is a plain-text file found in the root of a domain (e.g. www.example.com/robots.txt). It is a widely-acknowledged standard and allows webmasters to control all kinds of automated consumption of their site - not only by search engines.

In addition to reading about the protocol, robots.txt is one of the more accessible areas of SEO since you can access any site's robots.txt. Once you have completed this module, you will find value in making sure you understand the robots.txt files of some large sites (for example Google and Amazon).

What you will learn in this module:

How to block all robots from certain areas of your site
How to restrict your robots.txt instructions to apply only to certain robots
How to override exclusion directives to allow access to certain areas of your site despite exclusion rules
Use wildcards to apply your rules to whole swathes of your site
Other robots.txt syntax such as sitemap file directives

The most common use-case for robots.txt is to block robots from accessing specific pages. The simplest version applies the rule to all robots with a line saying User-agent: *. Subsequent lines contain specific exclusions that work cumulatively, so the code below blocks robots from accessing /secret.html.

Add another rule to block access to /secret2.html in addition to /secret.html.

Clue Show answer

Add another “disallow:” directive that blocks robots from /secret2.html

We hope you enjoyed this free introductory lesson. To view the whole module you need to sign up for a paid account. Alternatively you can try out our three demo modules by signing up for a free demo account.

Not yet completed

Exclude directories

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Allow specific paths

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Restrict to specific user agents

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Add multiple blocks

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Use more specific user-agents

10m

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Basic wildcards

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Block certain parameters

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Match whole filenames

10m

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Add an XML sitemap

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.

Add a video sitemap

Teacher: Will Critchlow

To view this lesson you need to sign up for a paid account. Alternatively, you can learn more about DistilledU first, or try our three demo modules by signing up for a free demo account.