Technical SEO for large sites

Below about fifty thousand URLs, technical SEO is a checklist. Above a million, it becomes a resource-allocation problem: search engines will not fetch everything you publish, so the question stops being "is this page correct" and becomes "is this page worth the crawl".

Crawl budget is real at this size

The usual objection is that crawl budget is a myth. For most sites it is. For a large catalogue with faceted navigation it is the constraint that determines whether new pages get indexed this week or next quarter.

You can tell it is binding when log files show crawlers spending the majority of their fetches on parameter URLs, pagination and redirects rather than on pages you want ranked.

Faceted navigation

Facets are where crawl budget goes to die. Three filters with ten values each produce a thousand combinations per category, and most sites expose all of them as crawlable links.

The workable policy:

• Pick the facet combinations with genuine search demand and make those indexable, with unique titles and copy.
• Everything else gets noindex plus a nofollow on the link, or is rendered so it produces no crawlable URL at all.
• Enforce the rule in code, then monitor it — facet governance decays the moment someone adds a filter.

Rendering

If content arrives via JavaScript, assume it is fetched late and sometimes not at all. Compare the raw and rendered DOM for each template. The gap between them is your risk.

The failure that costs the most is subtle: a template that renders fine in a browser but drops its description or its internal links under a crawler's rendering budget.

Log files

Analytics tells you about people. Logs tell you about crawlers, and only logs answer:

• Which templates get fetched, how often, and how that changed
• Where fetches are wasted — redirect chains, 404s, parameter noise
• Whether a new section is being discovered at all

Working the backlog

Group findings by template, not by URL. One fix on a product template closes eight thousand issues; one fix on one URL closes one. Prioritise by traffic at risk rather than by severity label — a "medium" issue on your highest-earning template outranks a "critical" on a page nobody visits.

Then verify. Re-crawl after deployment and confirm the issue is closed, because a meaningful share of fixes do not survive contact with the release process.