What is llms.txt and why does it matter?

llms.txt is like robots.txt but for AI crawlers. It specifies which content to use for training versus real-time retrieval. As AI search tools become a larger share of discovery, controlling how they access and use your content becomes as important as controlling how Googlebot crawls it.

How is AI search different from traditional SEO?

Traditional SEO optimises for clicks from blue links. AI search optimises for inclusion in generated answers. You can rank first on Google and be absent from ChatGPT's response to the same query. The overlap exists but is smaller than the marketing suggests: AI models reward semantic clarity and entity consistency, not just crawlable structure and backlinks.

What structured data should every site have?

At minimum: Organisation, Product (where applicable), Article or BlogPosting, and BreadcrumbList. These form the baseline that feeds both traditional rich results and AI answer engines. Beyond that, add FAQPage for Q&A content, HowTo for guides, and Review or AggregateRating where you have genuine reviews.

Technical SEO Checklist for 2026

TL;DR

Technical SEO fundamentals remain essential, but generative AI search adds a distinct optimisation layer. Core requirements include crawlability, canonical integrity, Core Web Vitals performance, and Schema.org markup. Key additions for 2026 include machine discovery endpoints (/llms.txt), semantic HTML structuring for RAG parsers, cross-platform entity consistency, and passage-level answer placement. Auditing both traditional indexation and AI retrieval ensures comprehensive organic visibility.

Technical SEO hasn't changed as much as the discourse suggests. The fundamentals still apply. What's new is the layer of AI-specific requirements on top.

Google Search Central pipeline architecture illustrating Googlebot URL discovery, JavaScript headless Chromium rendering, and index ingestion — Official Google Search Central architecture: Googlebot discovers URLs, renders JavaScript via headless Chromium, and delivers parsed DOM signals into the search index. Source: Google Search Central.

Core Technical Foundations

These remain non-negotiable:

Crawlability: XML sitemaps, robots.txt, no orphan pages
Indexability: Proper canonicals, no accidental noindex, clean parameter handling
Performance: Core Web Vitals under thresholds, mobile-first rendering
Structured data: Organisation, Product, Article, BreadcrumbList as baseline

Google Search structured data and Schema.org JSON-LD architecture diagram illustrating entity disambiguation and Knowledge Graph extraction — Structured data schema architecture from Google Search Central, showing how JSON-LD feeds entity recognition and rich search integrations. Source: Google Search Central.

AI-Search Additions

New items that didn't exist two years ago:

llms.txt: Like robots.txt but for AI crawlers. Specifies which content to use for training vs. real-time retrieval
Semantic HTML: AI models parse meaning from structure. Use <article>, <section>, <nav> properly
Entity consistency: Ensure brand name, product names, and key personnel are mentioned identically across pages
Citation-friendly content: Clear authorship, dates, and factual statements that AI models can confidently attribute

The Audit Process

Run your standard technical audit first. Then add the AI-specific layer. The two aren't competing. They're complementary.

Most enterprise sites I audit have solid technical SEO but miss the AI layer entirely. That's the opportunity.

Frequently Asked Questions

llms.txt is like robots.txt but for AI crawlers. It specifies which content to use for training versus real-time retrieval. As AI search tools become a larger share of discovery, controlling how they access and use your content becomes as important as controlling how Googlebot crawls it.
Traditional SEO optimises for clicks from blue links. AI search optimises for inclusion in generated answers. You can rank first on Google and be absent from ChatGPT's response to the same query. The overlap exists but is smaller than the marketing suggests: AI models reward semantic clarity and entity consistency, not just crawlable structure and backlinks.
At minimum: Organisation, Product (where applicable), Article or BlogPosting, and BreadcrumbList. These form the baseline that feeds both traditional rich results and AI answer engines. Beyond that, add FAQPage for Q&A content, HowTo for guides, and Review or AggregateRating where you have genuine reviews.

Core Technical Foundations

AI-Search Additions

The Audit Process

Frequently Asked Questions

Related Reading

Google I/O AI Search: Predictions vs. Reality (The Empirical Verification)

Why Brands Need a Non-Branded SEO Strategy: Surviving SERP Collisions

Get Search Notes in Your Inbox.

Tell Me What You Need to Win.