Subject areaChannels and tools
Also known asspiderweb crawler
Crawlers and robots.txt: A crawler (also called a spider, the program that travels across websites by following links) reads pages to build the index of a search engine or to feed an artificial intelligence system.
The robots.txt file is a text document that sits in the root of the website (/robots.txt) and tells crawlers which folders they may visit and which they may not.
The protocol (the Robots Exclusion Protocol) dates back to 1994 and was set as an IETF standard, the body that maintains the technical rules of the internet, with RFC 9309 in 2022. It works on trust: well-behaved crawlers follow the rules, others ignore them. Protecting content therefore requires access with credentials, and the file serves to govern the traffic of those who cooperate.
A page forbidden to crawlers can still appear in results, without a description, if other sites link to it: to keep it out of the index a different instruction, written in the page itself, is used. The file also states where the sitemap is, and the llms.txt and robots.txt files have different jobs.
Source: IETF, RFC 9309, Robots Exclusion Protocol, settembre 2022..
An example
The case that follows is constructed, because the term does not yet appear in the articles. A company that makes chemical plants publishes an area reserved for distributors and a section of open catalogues. In the /robots.txt file it asks crawlers not to visit the /reserved/ folder and gives the address of the sitemap. Well-behaved crawlers read the file before every visit and skip the folder; to keep it truly closed, however, a password is needed.
Enrico Giubertoni. Licensed under CC BY-NC 4.0: you may quote, translate and adapt this entry with the author’s name and a link, stating any changes. For commercial use, write to info@enricogiubertoni.com
Spotted an inaccuracy or something that needs updating? Write to info@enricogiubertoni.com and help make this glossary a better place.
What are your business challenges?
Tell us about your priorities and the objectives you want to reach.
You will receive a free, targeted answer within one working day.