robots.txt meaning, in plain words

robots.txt tells search robots what they may crawl. What RFC 9309 defines, why it is not access control, and why listing paths in it can backfire on you.

Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated

robots.txt is a file that tells search robots whether they may crawl. RFC 9309, an IETF standard, defines this mechanism as the Robots Exclusion Protocol. It is the means by which an operator states how access to their own content should be handled.

How does robots.txt work?

It is like hanging a sign at a shop entrance that says, "Staff only, please." People who see the sign turn back, but the door is not locked.

According to RFC 9309, the file that holds these rules is placed at the top level of the site, with the name robots.txt in all lower case. The character encoding is UTF-8. Programs that crawl are asked to respect the rules written in this file.

What matters here is that they are only asked to, and are not forced. RFC 9309 itself says that these rules are not a form of access authorization.

The behavior of crawlers is organized on the crawler glossary page. The free URL check reads your public page, robots.txt, DNS and encryption information.

Why does robots.txt matter?

It matters because it is easy to make the mistake of trying to protect a page you do not want in search with robots.txt alone. As a security consideration, RFC 9309 says that this mechanism is not a substitute for valid security measures.

It also says that listing a path in robots.txt makes that path public and easier to find. If you line up the locations of an admin screen or of non-public documents here, it works in effect as a notice of where they are.

For targets whose access you want to control, RFC 9309 recommends a mechanism that checks the communication on the spot, such as HTTP authentication. IPA's checkpoints also include an item that asks whether files that should not be public have been made public.

Related terms

  • Crawler: a program that visits sites automatically and collects information.
  • Scraping: pulling large amounts of information from a site with a program.

Next step

Open your own robots.txt and see whether the names of places that should be non-public are listed there. To check whether files that must not be public can be opened from outside, contact us. The free URL check does not test for that. We take questions about how to protect them through a free consultation.

Frequently asked questions

If I write a page in robots.txt, can I hide it from people who should not see it?

No. RFC 9309 says that listing a path in robots.txt makes it public and easier to find. For access you want to control, it recommends using a proper security measure such as HTTP authentication.

Where do I put robots.txt?

Under the definition in RFC 9309, it goes at the top level of the site, with the name robots.txt in all lower case. The character encoding is UTF-8. A file placed anywhere else is not in the place this definition sets.

Frequently asked questions

If I write a page in robots.txt, can I hide it from people who should not see it?

No. RFC 9309 says that listing a path in robots.txt makes it public and easier to find. For access you want to control, it recommends using a proper security measure such as HTTP authentication.

Where do I put robots.txt?

Under the definition in RFC 9309, it goes at the top level of the site, with the name robots.txt in all lower case. The character encoding is UTF-8. A file placed anywhere else is not in the place this definition sets.

See what an attacker can see, today

The free URL check returns a grade in under a minute. For costs and next steps, book a free consultation.