How Search Engines Work: Crawling, Indexing and Ranking

What happens after you publish a new web page? How does Google discover it, understand it and decide whether to show it? Publishing a page does not automatically put it in search results.

Think of a library. Someone first has to find the book, then catalogue it, and finally a librarian picks the right one for each reader’s question. Search engines follow the same three stages, and once you understand them, most SEO basics start to make sense.

Quick answer: Search engines work in three stages. They crawl the web to find pages, index those pages by analysing and storing them, and then rank the most relevant results when someone searches.

What Is a Search Engine?

A search engine uses automated systems to discover, process, organise and retrieve information from the web. It has several main parts:

• Web crawlers: programs that find and fetch pages
• A search index: a huge database of page information
• Ranking systems: systems that choose and order results
• Search result pages: what you see after you search
• Query interpretation: working out what you meant

It’s far more than an alphabetical list of websites. Google’s own explanation of how Search works describes these same three stages.

Stage 1: Crawling

Crawling means fetching web content to discover and understand pages.

How search engines discover URLs

• Links from pages they already know
• XML sitemaps
• URLs they’ve found before
• References from other websites
• New URLs found during routine crawling

Google’s main crawler is called Googlebot. It doesn’t have one fixed list of sites, so it keeps finding new pages through these routes.

What happens during a crawl?

A crawler may:

1. Discover a URL.
2. Request the page.
3. Process the server’s response.
4. Fetch associated resources, such as CSS and scripts.
5. Render the content where needed.
6. Extract links and other information.

Why a page might not be crawled

• Few or no internal links point to it.
• Robots.txt blocks it.
• The server returns errors.
• Network problems get in the way.
• The page needs a login.
• Crawl scheduling means it hasn’t been reached yet.

Pro tip: Link to every new page from at least one relevant, established page on your site. This gives crawlers a clear path, and it helps your visitors too.

Stage 2: Indexing

Indexing means analysing a page and potentially storing information about it. A search engine looks at:

• Text content
• Titles and headings
• Images and other media
• Which version is the canonical one
• Duplicate or near-duplicate URLs
• Language and regional signals
• Indexing directives, such as noindex
• Accessibility and quality

A crawled page is not always indexed, and that often surprises people.

Crawling vs. indexing

These are simplified descriptions, because real search systems and reports are more nuanced.

How Google chooses the canonical page

Many pages exist in several versions, such as with and without tracking codes. Google groups similar pages together and picks one as the canonical, which is the version it may show in results. Your canonical tags, redirects, sitemap and internal links all act as signals, although Google can still choose differently.

Do you need to worry about crawl budget?

Most small businesses don’t.

Fact: Google’s crawl budget guide is aimed at very large sites, such as those with 1 million or more unique pages changing moderately, or 10,000 or more changing daily.

If your website has a few hundred pages, spend your time on page quality and internal links instead.

Stage 3: Ranking and Serving Results

When someone searches, the engine reviews potentially relevant indexed content and picks results using many signals:

• Relevance to the search
• Content usefulness
• Meaning and context
• Language
• Location, where relevant
• Device and search context
• Page-level signals
• Spam detection
• Search features suited to the query

Fact: There’s no single public formula or guaranteed position. Google’s ranking systems guide describes the systems it uses, and they change over time.

Why two people can see different results

Results aren’t one fixed list, so your own check can mislead you. Differences can come from:

• Location: "plumber" means something different in Cork and in Dublin.
• Language: results follow your language settings.
• Device: mobile and desktop layouts differ.
• Past searches: some personalisation can apply.
• Time: the index and results change continuously.

That’s why Search Console’s average position is more useful than searching for yourself once. If you need a neutral look, try a private browser window, but treat it as a rough check only.

Where do AI answers fit in?

Google’s AI features, such as AI Overviews, draw on its existing systems and index. Google states that a page must be indexed and eligible to show a snippet to appear as a supporting link, and it has no extra technical requirements. So the same three stages still matter. Our AI search optimisation service explains how we approach this.

What Happens When Someone Searches?

Take this search: "best hardwood floor refinishing contractor near me". In broad terms:

1. Google interprets both the service and the local intent.
2. It considers relevant pages and local results.
3. It uses available location and relevance signals.
4. It presents results and features that suit the search.
5. The person compares options and chooses one.

This example is an illustration, not a precise description of Google’s internal sequence. It does show why the same search looks different in Dublin and Cork, and why local SEO needs its own approach.

Why Doesn’t Every Page Appear in Google?

• It hasn’t been discovered.
• It’s inaccessible.
• It’s blocked from indexing.
• Its content duplicates another URL.
• It doesn’t offer enough distinct value.
• It isn’t relevant enough for the search.
• It’s indexed but rarely selected for the searches you’re checking.

How to investigate:

1. Open Google Search Console.
2. Paste the page’s address into the URL Inspection tool.
3. Read the indexing status and the reason given.
4. Check the page indexing report for wider patterns.

Our technical SEO guide explains how to fix what you find.

What Is the Difference Between Indexing and Ranking?

What Are Search Engine Ranking Factors?

Google uses many signals and systems, not one checklist. They include:

• Content relevance: does the page match the search?
• Helpfulness and originality: does it add real value?
• Links: do other sites reference it?
• Usability and page experience: does it work well?
• Language and location: does it suit this person?
• Spam policies: does it follow the rules?
• Context: what does this person want right now?

Be sceptical of anyone who gives exact percentages for these. Google doesn’t publish ranking weights.

How to Help Search Engines Discover Your Pages

• Create logical navigation.
• Link important pages internally.
• Publish useful, accessible content.
• Submit an appropriate sitemap.
• Check HTTP responses.
• Avoid accidental indexing restrictions.
• Keep canonical signals consistent.
• Fix broken links.

Common Myths About How Search Works

Related Guides and Services

• Choose what to target with the keyword research guide.
• Optimise each page using the on-page SEO guide.
• Learn how outside links help discovery in the off-page SEO guide.
• Prefer expert help? Our technical SEO service finds and fixes indexing problems.
• Read more answers in our SEO FAQ.

FAQs

Your Next Step

Search site:yourwebsite.com in Google for a rough look at what’s indexed. If important pages are missing, read our technical SEO guide, then work through the SEO checklist. You can also ask us for a free review.