How Search Engines Crawl, Index and Rank Pages

Key takeaways

• Google Search works in three stages, crawling, indexing and serving results, and not every page makes it through each stage.
• Being crawled does not guarantee being indexed; noindex, duplicates and thin content are common reasons for exclusion.
• Search engines do not publish an exact ranking formula, so be sceptical of anyone who claims to know precisely how much each factor counts.
• Link every important page from another page, keep a clean XML sitemap and check robots.txt carefully.
• When a page does not appear, work through the stages in order: can it be found, is it stored, is it the best answer?

What are the three stages behind every search result?

When you type a query into Google or another search engine, the results appear in a fraction of a second. That speed hides a long process that happened earlier. Search engines work in three broad stages: crawling, indexing and ranking. Knowing what each stage does helps you understand why a page may not appear, and what you can reasonably do about it.

What is crawling?

Crawling is discovery. Search engines use automated programs, often called crawlers, bots or spiders, that move from page to page by following links. A crawler starts with pages it already knows, reads them, notes the links on them and then visits those linked pages in turn. In this way it gradually explores the web.

A new page is usually found in one of a few ways.

• Links from known pages: if a page the crawler already knows links to your new page, it can follow that link.
• Sitemaps: an XML sitemap is a file that lists the important URLs on your site. You can submit it through the search engine's free webmaster tool.
• Direct requests: webmaster tools often let you ask for a specific URL to be inspected.

Crawlers do not have unlimited time for every site. Large or slow websites may be visited less thoroughly, and pages buried many clicks deep can be discovered late. A clear structure with sensible internal links helps crawlers reach what matters.

What can block crawling

• A robots.txt file that tells crawlers not to visit certain areas. It is useful for private sections, but a mistake can block important pages.
• Pages that need a login, because crawlers cannot sign in.
• Links that exist only inside scripts or forms that a crawler cannot follow.
• Server errors or very slow responses, which make a crawler give up for that visit.

What is indexing?

After a page is crawled, the search engine tries to understand it. It reads the text, the headings, the images and their descriptions, the links and other signals such as the language and the page's apparent purpose. Useful information is stored in a very large database called the index. You can imagine the index as a library catalogue: it is not the web itself, only an organised record of what the search engine has learned about pages.

Being crawled does not guarantee being indexed. A page may be left out for reasons such as these.

• It carries a "noindex" instruction, which tells the search engine not to store it.
• It is very similar to another page, so the engine chooses one version as the main one. This is called canonicalisation, and the preferred page is the canonical URL.
• Its content is thin, unhelpful or low in quality.
• It returns an error or redirects elsewhere.

Duplicate content is a common issue on Indian shopping and listing sites, where the same product may be reachable through several URLs with different filters. Telling the search engine which URL is the main one avoids confusion.

What is ranking?

When a person searches, the search engine looks through its index for pages that could answer the query and orders them. This ordering is ranking. The systems involved are complex and change over time, and search engines do not publish an exact formula. It is therefore wise to be sceptical of anyone who claims to know precisely how much each factor counts.

What search engines do say, in general terms, is that they try to understand the meaning of the query, the relevance of the page, the quality and trustworthiness of the content, the usability of the page and the context of the searcher, such as location and language. Someone searching for "chemist near me" in Pune needs a different result from someone in Chennai, even though the words are identical.

The three stages at a glance

Why do results differ between people?

Two people searching the same words can see different results. Several things cause this.

• Location: local intent is strong for services and shops.
• Language and region settings: results for Google India can differ from results in other countries.
• Device: mobile and desktop experiences can differ.
• Search history and settings: these can influence what is shown in some cases.

This is why checking your own rankings by searching alone can be misleading. Webmaster tools give a more reliable picture of how your pages appear on average.

The results page is more than ten links

A modern results page can include many elements besides the classic blue links: featured answers, image results, video results, maps, local business panels, shopping listings and suggested questions. The search engine chooses what to show based on what the query seems to want. This is one reason understanding search intent, which is the goal behind a query, matters as much as understanding keywords. You will find more on this in the guide on content and search intent.

What can you do at each stage?

To help crawling

• Link to every important page from at least one other page on your site.
• Keep a clean XML sitemap and submit it.
• Check robots.txt carefully so you do not block useful sections.
• Fix broken links and server errors.

To help indexing

• Write original pages with a clear purpose.
• Use one preferred URL for each piece of content.
• Avoid accidental noindex instructions, especially after a website redesign.
• Use descriptive titles and headings so the topic is obvious.

To help ranking

• Answer the searcher's question completely and honestly.
• Present information clearly, with good structure and readable language.
• Make the page work well on mobile phones.
• Earn genuine mentions and links by being worth citing.

How can you check what the search engine sees?

The free tools offered by search engines are the best place to start. A URL inspection feature can tell you whether a page is known, whether it can be indexed and when it was last crawled. The Page indexing report lists pages that were excluded and give reasons. Searching for your site name together with a distinctive phrase from a page can also show whether it appears at all, although this is only a rough check.

Patience and realistic expectations

New pages and changes are not reflected instantly. Crawling, indexing and ranking all take time, and the time varies by site and by topic. If a page does not appear after a few days, avoid making panicked changes. Check for technical blocks first, make sure the page is linked and listed in your sitemap, and then give the system a reasonable time. Repeated rewriting without a diagnosis usually wastes effort.

In short

Search engines find pages by crawling, store what they understand in an index and then rank pages for each query using many signals they do not fully disclose. You cannot control the process, but you can remove obstacles, make your pages clear and useful, and check your progress with official tools. When something does not appear, work through the stages in order: can it be found, is it stored, and is it the best answer for the query?

Sources and further reading

• Google Search Central: How Google Search works: describes the three stages of crawling, indexing and serving results, states that indexing is not guaranteed and that Google does not accept payment to crawl or rank a site higher.
• Google Search Central: Introduction to robots.txt: states that robots.txt tells crawlers which URLs they can access and is not a mechanism for keeping a page out of Google.
• Google Search Central: Sitemaps overview: explains when a sitemap helps, describes a small site as about 500 pages or fewer and states a sitemap does not guarantee crawling or indexing.
• Google Search Central: Googlebot: describes the smartphone and desktop crawlers and that both honour the same robots.txt rules.

Frequently asked questions