Every page that ranks on Google had to go through two separate steps first: crawling and indexing. Confusing the two is one of the most common reasons site owners misdiagnose why their content “isn’t ranking” — when the real problem is it was never indexed in the first place.
Table of Contents
Crawling vs. Indexing vs. Ranking
These are three distinct stages, and a page must pass through all of them in order:
- Crawling — Googlebot discovers your page by following a link to it, or by reading it from your XML sitemap.
- Indexing — Google analyzes the page’s content and stores it in its massive index, essentially a library catalog of the entire web.
- Ranking — When someone searches, Google’s algorithm decides which indexed pages best match the query and in what order to display them.
A page can be crawled but not indexed (if it’s low quality or blocked). It can be indexed but rank poorly. Getting indexed is necessary but not sufficient — see our guide on whether you need to submit your site to Google for the discovery side of this process.
How Googlebot Crawls a Website
Googlebot works through a queue of URLs, prioritized by factors like site authority, update frequency, and crawl budget. On each visit it:
- Requests the page and downloads its HTML (and increasingly, renders JavaScript)
- Extracts all links on the page to add to its crawl queue
- Checks your
robots.txtfile for any crawling restrictions - Reads your XML sitemap for a prioritized list of URLs
Mobile-First Indexing
Since 2020, Google predominantly uses the mobile version of your site’s content for indexing and ranking, using a mobile crawler by default. In practice this means: if content, links, or structured data are missing or hidden on your mobile layout compared to desktop, Google may simply never see them. Checking that your mobile and desktop versions carry equivalent content is worth doing even if most of your traffic happens to be desktop.
What Determines Crawl Budget
Larger or lower-authority sites need to think about crawl budget — the number of pages Google is willing to crawl within a given timeframe. Key factors:
- Site speed — faster servers get crawled more thoroughly
- Internal linking — well-linked pages get crawled more often; see our internal linking guide
- Duplicate or thin content — wastes crawl budget on low-value pages
- Sitemap accuracy — a clean, updated XML sitemap helps Google prioritize correctly
For most small to mid-sized sites, crawl budget isn’t a binding constraint — it becomes relevant mainly once a site reaches the tens of thousands of pages, or has significant technical waste like faceted navigation generating near-duplicate URLs.
Common Reasons Pages Don’t Get Indexed
- Accidental
noindextag left on the page - Page blocked in
robots.txt - No internal links pointing to the page (an “orphan” page)
- Duplicate or very thin content Google considers low value
- Canonical tag pointing to a different URL
You can check indexing status for any URL using Search Console’s URL Inspection tool, or the Pages report under the Indexing section. For the full technical reference, see Google’s crawling and indexing documentation.
Reading the Pages Report in Search Console
The Pages report groups your URLs into indexed and non-indexed buckets, each with a specific reason. A few of the most common non-indexed reasons and what they typically mean:
- “Discovered – currently not indexed” — Google knows the URL exists but hasn’t crawled it yet, often a crawl budget or low-priority signal
- “Crawled – currently not indexed” — Google looked at the page and decided against indexing it, usually a content quality signal worth investigating
- “Excluded by noindex tag” — intentional or accidental; worth double-checking it’s not a mistake
- “Duplicate without user-selected canonical” — Google found near-identical content elsewhere and picked a different canonical than you may have intended
Frequently Asked Questions
Crawling and indexing is the first stage of a bigger process — see our SEO Basics guide for the complete picture.
Want a technical crawl audit of your own site? Web Solution Zone’s SEO team can identify exactly what’s blocking your pages from being indexed.
