Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
Web scraping is automatically extracting a large amount of a site's information with a program. The OWASP document organizes the automated act of collecting an application's content or data in order to use it elsewhere as scraping.
How does web scraping work?
It is close to putting every page of a library book through a copier instead of copying it by hand, one book at a time. It is faster than a person looking, and the same work continues without rest.
According to the OWASP document, what is collected can be content visible from outside, or data returned as the result of processing. Some collection happens without logging in, and some uses fake accounts or taken-over accounts. Pages, APIs and parameters may be tried in turn, and information is pulled from the responses that come back.
The collection can run continuously or at fixed intervals. The document also touches on the possibility that the structure and behavior of a system can be read from what was collected.
This OWASP page itself does not list countermeasures, and leaves them to other guidance. The problem of an API returning too many items is covered in excessive data exposure. The range visible from outside can also be checked with the free URL check.
Why does web scraping matter?
Member-only information, price lists and lists a company gathered on its own are the value of a site. If they are carried off whole in a short time, the value of the product can drop, or the data can be used for another purpose. The OWASP document gives abuse of proprietary information and use for price manipulation as examples.
As an operator, first confirm whether the items shown on screen match the items the API returns. Next, look in the records at whether heavy access has continued over a short time. Rate limiting and bot management come after that.
Note that robots.txt, which tells visitors whether they may crawl, is not a mechanism that makes anyone comply. RFC 9309 says these rules are not a means of access authorization.
Related terms
- Excessive data exposure: a flaw where responses carry more items than needed
- Rate limiting: a mechanism that holds down the number of accesses accepted in a given time
Frequently asked questions
Are scraping and crawling the same thing?
They are close but not the same. A crawler is an automated program that follows links to visit pages. Scraping means extracting the information you need from what was collected and using it. The OWASP document lists crawler as one alias of scraping.
If a page needs login, will it not be scraped?
You cannot say that. The OWASP document says fake accounts or taken-over accounts may be used. Login is only one entrance, so the volume of access afterward and the items that can be retrieved also need review.
Next step
List the APIs on your site that return member information or prices. The entry points visible from outside can be checked with the free URL check. To review which items each one returns, contact us.
Frequently asked questions
Are scraping and crawling the same thing?
They are close but not the same. A crawler is an automated program that follows links to visit pages. Scraping means extracting the information you need from what was collected and using it. The OWASP document lists crawler as one alias of scraping.
If a page needs login, will it not be scraped?
You cannot say that. The OWASP document says fake accounts or taken-over accounts may be used. Login is only one entrance, so the volume of access afterward and the items that can be retrieved also need review.