How Search Engines Work: Crawling, Indexing and Ranking Explained

Typing a question into a search box and getting answers in a fraction of a second feels instant, but it rests on three separate processes that happen well before the query is typed: crawling, indexing and ranking. Google lays out this three-stage model in its own developer documentation, and the same basic framework underlies other search engines as well.

Crawling: finding pages on the web

Crawling is the discovery phase. According to Google's developer guide, automated programs called crawlers — Google's is named Googlebot — find new or updated pages mainly through three routes: revisiting pages they already know about, following hyperlinks from one page to another, and reading sitemap files that site owners submit directly to the search engine.

Google's documentation notes that Googlebot comes in a mobile and a desktop variant, and that most crawl requests now come from the mobile crawler because Google primarily indexes the mobile version of a page. Crawlers also render pages using a recent version of the Chrome browser so they can execute JavaScript, since many sites load content dynamically rather than serving it as plain HTML.

Crawlers do not fetch everything indiscriminately. Google's guide states that Googlebot's crawl requests are throttled algorithmically, with most sites typically accessed no more than about once every few seconds on average, and that Googlebot respects the rules website owners set in a robots.txt file. Fetches are also capped in size: up to 2MB for most file types and 64MB for PDFs, after which Googlebot stops downloading and works with whatever portion it already retrieved.

Indexing: organising what was found

Once a page is fetched, the search engine has to decide what it is about and whether it is worth storing. Google's documentation describes this as analysing text, images and video content, reading HTML elements such as title tags and image alt attributes, and applying canonicalization — grouping near-duplicate pages and selecting one representative version to show in results.

Crucially, Google states that crawling a page does not guarantee it will be indexed: pages can be excluded for low content quality, a robots meta tag that blocks indexing, or technical problems with the site. This is the technical basis for the 'Crawled – currently not indexed' status site owners sometimes see in Google's tools.

Under the hood, the storage structure search engines use is typically an inverted index. As explained in a technical overview published by GeeksforGeeks, an inverted index flips the usual document-to-word relationship: instead of listing which words are in which document, it lists, for every word, which documents contain it. Building one involves breaking text into individual terms (tokenizing), removing common filler words, and normalising word variants before recording where each term appears. This structure lets a search engine jump straight to the relevant documents for a query instead of scanning the entire web line by line, which the GeeksforGeeks overview describes as impractically slow once a collection of documents grows large.

Ranking: ordering results for a query

Ranking happens at the moment someone searches. Google's documentation says that of all the indexed pages that match a query, the system orders them by relevance, and that relevance is determined by what it describes as 'hundreds of factors,' which can include signals such as the user's location, language and device. Google also states explicitly that it does not accept payment to rank organic pages higher and that ranking is done programmatically.

One of the earliest and best-documented ranking signals is PageRank, created by Google co-founders Larry Page and Sergey Brin at Stanford University in 1997 and first filed as a patent in January 1998, according to a history compiled by SEO analytics firm Ahrefs. PageRank treats each hyperlink to a page as a vote of confidence, with the value of each vote weighted by the authority of the linking page. The original formula included a damping factor of 0.85, meant to model the declining likelihood that a user keeps clicking through links the further they are from a starting page.

Ahrefs' overview notes that PageRank is no longer a standalone score — public access through Google's toolbar ended in 2016 — but Google's own Gary Illyes has said links still feed into broader authority signals the company groups under E-E-A-T (experience, expertise, authoritativeness, trustworthiness). Google has also moved toward a reasonable-surfer model that weighs some links more heavily than others based on how likely a real visitor would be to click them, rather than splitting authority evenly across every link on a page.

Why pages can be crawled but still invisible in search

The three stages explain a common source of confusion for website owners: a page being crawled does not mean it is indexed, and being indexed does not guarantee a high ranking. Google's guide is direct about this limitation, stating that it cannot guarantee a given page will be crawled, indexed, or served, even when a site follows Google's published guidelines.