Crawler

Meaning

A crawler, also called a bot or spider, is an automated program that visits web pages by following links so a search engine can discover and read them. Google's crawler is Googlebot, which has a smartphone version that handles most requests and a desktop version, and it checks robots.txt before fetching a URL. Crawlers cannot sign in, so login-only pages stay hidden, and slow or erroring servers make them give up. A clear structure with internal links helps them reach what matters. See also: Index, robots.txt, and the guide How Search Engines Crawl, Index and Rank Pages.