Robots.txt, sitemaps, crawl budget and the indexation checks that decide whether you're eligible to rank at all.
Before a page can rank, be snippet-worthy, or be cited in an AI answer, it has to clear two technical gates: it must be crawlable — reachable by the engine — and indexable — allowed and worthy of being stored. Get these wrong and nothing downstream matters, because a page the engine cannot reach or chooses not to store is invisible however good it is. This is the practical machinery of crawlability and indexation: robots.txt, sitemaps, crawl budget, and the indexation checks that decide whether you are eligible to appear at all.
Crawlability and indexability are distinct gates, and confusing them is a common source of trouble. Crawlability is reachability: can the engine’s crawler discover and fetch the page — is it linked, not blocked, served without errors. Indexability is being allowed and worthy: once fetched, is the page permitted to be indexed (no directive blocks it) and worth storing (not thin, duplicate, or low-value). A page can be perfectly crawlable yet not indexed, because indexation adds a quality-and-permission judgment on top of reachability.
Holding the distinction clear is what makes diagnosis possible. If a page is not being crawled, the problem is reachability — linking, robots rules, or errors. If it is crawled but not indexed, the problem is permission or worth — a directive, duplication, or thin content. The two gates fail differently and are fixed differently, so knowing which one a page is stuck at is the first step. Crawlability is reachability; indexability is being allowed and worthy — two separate hurdles a page must clear to be eligible.
The robots.txt file, at your site’s root, is the primary tool for controlling which crawlers can access which parts of your site. It lists directives — which user-agents (crawlers) are allowed or disallowed from which paths — and compliant crawlers obey it. This lets you steer crawling: blocking crawlers from low-value or sensitive areas (internal search results, admin paths, endless parameter combinations) so they spend their effort on the pages that matter, and keeping the crawlable surface clean.
The power of robots.txt is also its danger: a misplaced disallow can accidentally block important pages or whole sections from being crawled, silently removing them from eligibility. This is one of the most common and damaging technical mistakes, because the pages simply vanish from results with no obvious error. The discipline is to use robots.txt deliberately — blocking genuine junk to focus crawling, while ensuring nothing important is disallowed — and to check it whenever visibility drops unexpectedly, since it is a frequent culprit.
An XML sitemap is a file listing the URLs you want engines to know about, handed to them as an explicit map of your important pages. It helps crawlers discover pages — especially on large sites or for pages not well-linked internally — and signals which URLs you consider canonical and worth indexing. Submitting a sitemap through Search Console, and referencing it in robots.txt, ensures engines have a clear inventory of your pages to crawl, complementing discovery through links.
A sitemap is a guide, not a guarantee — listing a URL does not force it to be crawled or indexed — but it materially helps discovery and signals intent. The discipline is to keep the sitemap clean and current: including your canonical, indexable pages and excluding junk, redirects, and non-canonical URLs, so it accurately represents what you want indexed. Together with clean internal linking, a good sitemap ensures engines can discover your important pages, which is the first requirement of the crawl gate.
Crawl budget is the amount of crawling an engine will do on your site in a given period, shaped by your site’s size, health, authority, and update frequency. For most sites it is ample and not a constraint; it matters chiefly for very large sites, where crawlers may not reach every page, so how the budget is spent determines what gets crawled. Wasting budget on low-value URLs — duplicate parameters, faceted-navigation combinations, soft-404s — means fewer important pages get fetched.
The practical work, for sites large enough to care, is to spend crawl budget efficiently: block or prune junk URLs so crawlers focus on valuable pages, fix errors and slow responses that waste fetches, and keep the important pages well-linked and healthy. This is where robots.txt, clean architecture, and good performance converge on crawl efficiency. Crawl budget rewards a clean, healthy, well-organized site by concentrating crawling on the pages that matter, which is the practical lever at the crawl gate for large sites.
Crawlability is reachability; indexability is being allowed and worthy. A page must clear both before it can rank, win a snippet, or be cited — and each gate fails, and is fixed, differently.
Indexation is the second gate: once crawled, is the page permitted and worth storing in the index. Permission is a matter of directives — a noindex tag or equivalent tells engines not to index the page, and its accidental presence is a common reason pages vanish. Worth is a matter of quality — engines choose not to index thin, duplicate, or low-value pages even when permitted, because storing them adds nothing. Quality gates the index: thin and duplicate pages get dropped.
This means being indexed requires both clearing the permission check (no noindex, correct canonicalization) and offering genuine value (substantial, unique, useful content). A page that is crawlable but thin or duplicative may be crawled and then not stored; a page with an accidental noindex is excluded regardless of quality. The indexation gate is where reachability meets permission and worth, and passing it requires attending to both — ensuring nothing blocks indexing and that the page genuinely deserves a place in the index.
A status you will see in Search Console — “Crawled — currently not indexed” — is worth understanding, because it is the index gate speaking. It means the engine fetched the page but chose not to store it: the page was crawled, so reachability is fine, but it was not judged worth indexing. This usually signals a quality or duplication problem — the page is thin, too similar to others, or simply not distinctive or valuable enough to earn a place in the index.
The fix for “Crawled — currently not indexed” is therefore about worth, not access: making the page genuinely valuable, substantial, and distinct, or consolidating it if it duplicates other content. Chasing crawl or robots fixes for this status misses the point, since the page was crawled fine. Understanding this status is a practical diagnostic skill: it tells you the page cleared the crawl gate but failed the worth check at the index gate, pointing you squarely at content quality as the thing to fix.
The same crawl and index gates decide whether AI engines can retrieve and cite you. DUNkē tracks whether you’re actually surfaced across eight AI engines — so you can confirm your technical foundation is translating into presence in the answers.
For crawlability and indexation, Search Console — not a rank tracker — is your source of truth. It reports directly from the engine: which pages are indexed, which are excluded and why, crawl statistics, and the exact status of individual URLs through the URL Inspection tool. A rank tracker tells you positions for pages that are already indexed; it cannot tell you whether a page is crawled, indexed, or blocked. For the technical gates, you need the engine’s own reporting, which Search Console provides.
The discipline is to use Search Console as the authoritative view of your technical health: monitoring the index coverage report for excluded pages and reasons, inspecting individual URLs to see their crawl and index status and how they render, and watching crawl stats for problems. This is where you diagnose why a page is not appearing — is it crawled, is it indexed, is it blocked — with data from the engine itself. Treating Search Console as the source of truth for the gates is fundamental technical practice.
The common mistakes cluster at each gate. At the crawl gate: accidentally disallowing important pages in robots.txt, orphaning pages with no internal links, serving errors or slow responses, and blocking resources crawlers need to render. At the index gate: leaving a noindex on pages that should be indexed, letting thin or duplicate pages proliferate, and misconfiguring canonical tags so the wrong version is indexed or pages are canonicalized away. Each mistake silently caps eligibility.
These mistakes are damaging precisely because they are quiet — the pages simply do not appear, with no error the user sees. The remedy is regular technical hygiene: checking robots.txt and noindex directives, ensuring important pages are linked and rendering, watching the index coverage report for unexpected exclusions, and getting canonicalization right. Because these gates are foundational, a mistake here can undermine all other work, which is why disciplined attention to crawlability and indexation is the technical baseline every site needs.
noindex tags and correct canonicalization; consolidate thin or duplicate pages.Crawlability and indexation are foundational for AI visibility, not just traditional search, because AI engines must discover and access content before they can retrieve and cite it. An engine can only cite what it can crawl and has in its sources; a page blocked or unindexed is as invisible to an AI engine as to a traditional one. The same gates that decide whether you can rank decide whether you are eligible to be retrieved and cited — reachability and worth underlie AI answers too.
There is an AI-specific dimension, covered in a companion piece: AI engines use their own crawlers (GPTBot, ClaudeBot, PerplexityBot, and others), and your robots.txt controls their access specifically, so being crawlable to them is a distinct check. But the principle is the same — being reachable and worthy is the prerequisite for being used in answers. Crawlability and indexation are the technical foundation of all visibility, traditional and AI alike, which is why they are the first thing to get right.
A robots.txt file works through directives grouped by user-agent. A User-agent line names the crawler the rules apply to (or * for all), followed by Disallow lines specifying paths crawlers should not fetch, and optionally Allow lines carving out exceptions within a disallowed path. Wildcards let you match patterns — for example, blocking URLs containing certain parameters. This lets you target rules precisely: blocking specific crawlers, specific paths, or specific URL patterns while permitting the rest.
The precision is powerful but demands care, because the rules are interpreted literally and a broad or misplaced Disallow can block far more than intended. The discipline is to write directives deliberately, test them against your important URLs to confirm they are not accidentally blocked, and keep the file simple enough to reason about. Understanding how the directives work — user-agents, disallow, allow, wildcards — is what lets you control crawling accurately rather than blocking by accident, which is the difference between robots.txt as a precise tool and as a source of silent damage.
A subtle but critical distinction: blocking a page in robots.txt is not the same as keeping it out of the index, and confusing the two causes real problems. Robots.txt stops crawlers from fetching a page; a noindex directive (in the page’s HTML or headers) tells engines not to index it. Crucially, if you block a page in robots.txt, the engine cannot fetch it — and therefore cannot see a noindex on it — so a blocked page can sometimes still appear in results without its content, because the engine knows of it but cannot read its directive.
The practical rule is: to keep a page out of the index, use noindex and allow it to be crawled so the engine can see the directive; to save crawl budget on truly irrelevant paths, use robots.txt. Do not block in robots.txt a page you are trying to deindex, because the block prevents the engine from seeing the noindex. Understanding this distinction — crawl control versus index control — prevents a common, confusing mistake and is essential to controlling indexation correctly.
Canonicalization is how you tell engines which version of duplicate or similar pages is the primary one to index. When content is accessible at multiple URLs — parameter variations, print versions, similar pages — a canonical tag declares the preferred URL, consolidating signals onto it and preventing the duplicates from diluting or competing. Getting canonicalization right ensures the version you want is indexed and that near-duplicate pages do not fragment your visibility or waste the index gate on redundant copies.
The common canonicalization mistakes are damaging: pointing canonicals at the wrong URL (canonicalizing your important page away), inconsistent canonicals, or failing to canonicalize genuine duplicates, letting them compete. The discipline is to declare accurate canonicals that point to the true primary version, keep them consistent, and use them to consolidate duplicates. Canonicalization is a core part of the index gate, because it governs which of your similar URLs is stored and credited — getting it wrong can quietly bury the page you actually wanted indexed.
For large sites — those with many thousands or millions of URLs — crawl budget becomes a genuine constraint worth managing deliberately, because crawlers may not reach and refresh every page. The levers are efficiency and prioritization: pruning or blocking low-value URLs (endless facets, parameters, near-duplicates) so budget concentrates on valuable pages; keeping the site fast and error-free so fetches are not wasted; and ensuring important pages are well-linked and appear in a clean sitemap so they are prioritized for crawling.
The goal is to make every crawl count toward your important, changing pages rather than being spent on junk. Monitoring crawl stats in Search Console reveals how budget is being used and whether important pages are being crawled and refreshed adequately. On large sites, crawl-budget management is a real technical discipline that determines how much of your site gets discovered and kept current; on smaller sites it rarely binds, but understanding it clarifies why a clean, efficient, well-organized site crawls better regardless of size.
A sitemap delivers the most value when it is kept clean and current. Best practice is to include only your canonical, indexable URLs — excluding redirects, non-canonical variants, blocked pages, and junk — so the sitemap accurately signals what you want indexed. Keep it up to date as pages are added or removed, since a stale sitemap listing dead or non-canonical URLs sends mixed signals. For very large sites, sitemaps have size limits, so multiple sitemaps referenced by a sitemap index file are used to cover all URLs cleanly.
The sitemap should be submitted through Search Console and referenced in robots.txt so engines find it reliably. Treating the sitemap as an accurate, maintained inventory of your important pages — rather than a dumping ground of every URL — is what makes it useful for discovery and for signaling your canonical set. Good sitemap hygiene complements clean internal linking and correct canonicalization, together ensuring engines can discover and correctly identify the pages you want crawled and indexed.
When an important page will not index, a systematic diagnosis using Search Console isolates the cause. Inspect the URL: is it discovered and crawled, or not even found? If not found, the issue is discovery — linking or sitemap. If crawled but not indexed, check for a noindex directive or a canonical pointing elsewhere — permission problems that exclude it. If permitted but still “Crawled — currently not indexed,” the issue is worth — the page is thin, duplicative, or low-value and needs to be made genuinely substantial and distinct.
This bottom-up diagnosis — discovery, then permission, then worth — pinpoints the actual gate the page is failing, so you apply the right fix rather than guessing. A discovery problem needs links; a permission problem needs directive or canonical fixes; a worth problem needs content improvement. Using the URL Inspection tool and coverage report to walk this diagnosis is a core technical skill, turning “this page won’t rank” into a precise identification of why it is not even indexed — the necessary first step before ranking is even possible.
Crawlability and indexation are not set-and-forget — they need ongoing monitoring, because sites change and problems creep in. A deployment can introduce a stray noindex or a bad robots.txt rule; a migration can break links or canonicals; content growth can spawn duplicate or thin pages. Monitoring the index coverage report for unexpected drops or exclusions, watching crawl stats, and spot-checking important URLs catches these problems before they quietly erode visibility over weeks or months.
The discipline is to treat technical health as something you watch continuously, not audit once. Sudden changes in indexed page counts, spikes in exclusions, or crawl errors are signals to investigate, often revealing a directive or configuration problem introduced by a change. Because the crawl and index gates are foundational, catching regressions here quickly prevents them from undermining everything downstream. Monitoring crawlability over time — with Search Console as the source of truth — is how you keep the technical foundation sound as your site evolves.
Crawlability and indexation are the two technical gates a page must clear to be eligible for any visibility: crawlability is reachability — can the engine discover and fetch the page — controlled through robots.txt, sitemaps, internal links, and crawl budget; indexability is being allowed and worthy — permitted by directives and valuable enough to store, since quality gates the index and thin or duplicate pages get dropped. The two gates fail and are fixed differently, so diagnosing which one a page is stuck at is the essential first step.
Search Console, not a rank tracker, is your source of truth for these gates, reporting the engine’s own view of what is crawled, indexed, and excluded. The common mistakes — accidental disallows, stray noindex tags, thin or duplicate pages, broken canonicalization — are quiet but damaging, which is why disciplined hygiene here is the technical baseline. And because AI engines must discover and access content before citing it, these gates underlie AI visibility too. Clear them, and everything else has a foundation to build on.
“Before a page can rank or be cited, it has to be reachable and worth storing. Crawlability is reachability; indexability is being allowed and worthy — two gates, and most technical trouble is one of them, quietly closed.” The Age’X Research Team
Get a free GEO audit — the same analysis behind every article here.