robots.txt

Meaning

robots.txt is a text file at the root of a website that gives crawlers instructions about which areas they may or may not visit. Google's documentation says it is used mainly to manage crawler traffic and is not a mechanism for keeping a page out of Google; for that, use noindex or password protection. One wrong line can block an entire site, so review it with care, especially after a redesign. Blocking a page also stops crawlers from seeing its noindex tag. See also: Noindex, Crawler, and the guide Technical SEO Basics: Speed, Mobile, Crawling and Security.