Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
A web crawler is a program that automatically visits sites and collects information. RFC 9309 from the IETF calls a crawler an automated client, and explains that search engines use crawlers to follow links and build an index.
How does a web crawler work?
Imagine a librarian who walks the shelves from one end to the other, copying each book's title into a catalog. A crawler does this on the internet, automatically. It opens a page, finds the links on it, moves to the next page, and repeats.
The information collected this way becomes the material for search results. Your site can be found in search because it was visited and its content was registered.
The side being visited can describe in a file called robots.txt which range may be visited. RFC 9309 explains that crawlers are requested to honor these rules. In other words, it stays a request and is not a mechanism that forces compliance. RFC 9309 is explicit that this file is not a means of access authorization; information you do not want shown needs a separate safeguard such as authentication.
The details of how to write it are collected on the robots.txt term page. How your site appears can be viewed in the free URL check.
Why do web crawlers matter?
Being visited is itself a movement that you need in order to be found by search. At the same time, not every crawler belongs to a search engine.
OWASP's document organizes the automated act of collecting information and using it elsewhere as scraping. It lists crawler, and collecting for AI training, among the aliases. When the purpose differs, so does the effect on the site.
The first thing an operator should confirm is whether pages not meant to be shown are public. If information behind a login, or a file that should be private, can be opened from outside, anyone can read it, not only a crawler.
Related terms
- robots.txt: a file that tells search robots whether they may visit
- Web scraping: pulling large amounts of information out of a site with a program
Frequently asked questions
Are all crawlers run by search engines?
No. RFC 9309 says search engines use crawlers to follow links and build an index. OWASP, on the other hand, lists the name crawler as one alias for an automated act of collecting information. Purposes are not uniform, so you need a way to tell who is visiting.
Can a site refuse a crawler's visits?
A file named robots.txt can tell crawlers which range they may visit. RFC 9309 states, though, that this mechanism is not a means of access authorization. Whether it is followed depends on the crawler, so information you do not want shown needs another safeguard.
Next step
If some information would cause trouble when visited, protect it with login and permission checks and not with robots.txt. The idea of holding down access volume is in rate limiting. To check the range visible from outside, use the free URL check. To talk about your policy, contact us.
Frequently asked questions
Are all crawlers run by search engines?
No. RFC 9309 says search engines use crawlers to follow links and build an index. OWASP, on the other hand, lists the name crawler as one alias for an automated act of collecting information. Purposes are not uniform, so you need a way to tell who is visiting.
Can a site refuse a crawler's visits?
A file named robots.txt can tell crawlers which range they may visit. RFC 9309 states, though, that this mechanism is not a means of access authorization. Whether it is followed depends on the crawler, so information you do not want shown needs another safeguard.