If you’ve been paying attention to AEO services and LLM SEO services discussions lately, you’ve probably seen “llms.txt” mentioned. Maybe you’ve looked it up and found a mix of enthusiastic claims and technical explanations that weren’t totally clear.

This guide is the version that makes sense to an actual human.

We’ll cover what llms.txt actually is, where the idea came from, how it differs from robots.txt, which AI systems use it, and how to create one step by step. By the end of this, you’ll have everything you need to set it up yourself today.

What llms.txt Actually Is and Where the Idea Came From

llms.txt is a plain text file that you place in the root directory of your website, the same place you’d put robots.txt or sitemap.xml that provides structured information about your website to AI systems and large language models.

The idea was proposed by Jeremy Howard, founder of fast.ai, in late 2024. The premise was simple: websites communicate rules and information to web crawlers through files like robots.txt and sitemaps, but those formats were designed for traditional search engines. As AI systems started crawling and indexing web content to build their training data and retrieval indexes, there was no standard way for website owners to communicate with them specifically.

llms.txt is the proposed solution. It’s a Markdown-formatted file that provides AI crawlers with a clear, structured summary of what your site contains, your most important pages, and any guidance you want to offer on how your content should be understood or prioritised.

The format is still technically a proposal; it hasn’t been officially adopted by any major standard body. But major AI platforms, including Anthropic, and a growing number of companies, have already started implementing it. It’s gaining traction fast enough that adding one to your site now is genuinely useful, not just theoretical.

How LLMs Use This File When Crawling and Indexing Your Website

When an AI system crawls your website, whether to build training data, to populate a search index, or to retrieve content for a real-time query, it has to make decisions about what to read, what to prioritise, and what to skip.

Without any guidance from you, the crawler makes those decisions algorithmically. It might prioritise your homepage and your most-linked pages. It might miss your most important content if it’s poorly linked. It might index pages you’d rather it didn’t, like internal tools or outdated content.

LLMs.txt gives you a way to guide those decisions. A well-written LLMs.txt file:

  • Tells the AI what your website is about in clear, direct language
  • Points to your most important pages with direct URLs and brief descriptions
  • Signals which sections of your site are most relevant for specific query types
  • Can indicate which pages or sections you’d prefer the AI not to focus on

Think of it as writing a briefing document for an AI that’s about to read your entire website. Instead of letting it figure everything out on its own, you give it the executive summary first.

In terms of technical implementation: when a compatible AI crawler visits your site, it checks for llms.txt at yourwebsite.com/llms.txt before diving into the rest of the content. If the file exists, it reads it. If not, it proceeds without guidance.

llms.txt vs robots.txt: What’s the Actual Difference?

This is the question that comes up most often, so let’s clear it up properly.

robots.txt is a file that tells web crawlers, such as Google’s crawler, Bing’s crawler, and many others, which pages they are and are not allowed to access. It’s a gatekeeper file. You use it to block crawlers from certain directories, prevent indexing of duplicate content, and protect pages that shouldn’t be publicly accessible to search engines.

Robots.txt is about permission: “You can go here; you can’t go there.”

llms.txt is about guidance and communication: “Here’s what we are, here’s what’s most important, and here’s how to understand our content.”

You’re not blocking or allowing anything with llms.txt. You’re providing context and prioritisation signals to AI systems specifically. A crawler that ignores llms.txt can still access all your content. The file just helps cooperative crawlers do a better job of understanding what matters.

Another key difference: robots.txt is a mature, universally supported standard that every major web crawler respects. llms.txt is newer and currently adopted by a subset of AI systems. Compliance is voluntary and growing, not universal.

Does robots.txt affect LLMs? Yes, you can block AI crawlers like OpenAI’s GPTBot in your robots.txt, and they’re supposed to respect that. But robots.txt doesn’t tell AI systems anything about your content; it only controls access. LLMs.txt does the positive communication that robots.txt doesn’t.

You need both for different purposes.

Google Support for llms.txt in 2026 Explained

This is where we need to be honest about the current state of adoption.

As of 2026, Google has not publicly confirmed that its crawlers or AI systems explicitly parse llms.txt as part of their indexing or AI Overview generation process. Google has its own established mechanisms for understanding website content, sitemaps, structured data, and its own crawlers.

However, Google has also not indicated that it ignores or opposes the format. Several sources have noted that Google’s systems will read any plain-text file in a website’s root directory, which means the content of your llms.txt is at minimum accessible to Google’s crawlers even if it’s not being parsed in a structured way.

Anthropic, the company behind Claude, has officially implemented support for llms.txt on its own documentation site and has indicated that Claude’s retrieval systems can use the file for guidance. OpenAI has shown interest in the format. Perplexity has not made official statements but is known to experiment with crawler guidance mechanisms.

The practical answer: adding an llms.txt file gives you documented, verified benefits for AI systems that support it today and puts you ahead of the curve for the systems that will likely support it in the near future. The cost of adding one is about 30 minutes of your time. The cost of not having one is any opportunity cost from AI crawlers that are looking for it and not finding it.

Step-by-Step: How to Create and Upload Your llms.txt File

This is genuinely simple. No coding required. Here’s the exact process.

Step 1: Open a plain text editor

Use Notepad (Windows), TextEdit in plain text mode (Mac), or any code editor like VS Code. Do not use Microsoft Word, as it adds hidden formatting that will break the file. The file needs to be plain text with Markdown formatting.

Step 2: Write your llms.txt content

The format follows a simple Markdown structure. Here’s the basic template:

# [Your Business or Website Name]

> [One to three sentence description of what your website is and who it serves]

## Key pages

– [Page Title](URL): Brief description of what this page covers and why it’s important

– [Page Title](URL): Brief description

– [Page Title](URL): Brief description

## About us

[Two to four sentences about your organization, your expertise, and what makes your content credible]

## Optional: Content focus areas

– [Topic 1]: Brief description of your coverage of this topic

– [Topic 2]: Brief description

Fill this in with your actual information. Be specific and factual. This is the briefing document for any AI that crawls your site. Write it as if you’re explaining your website to a smart stranger who has 30 seconds to understand what you do and where to look.

Step 3: Save the file as ‘llms.txt’.

Save the file with exactly this name: llms.txt, all lowercase, no spaces. The file extension is ‘.’. txt.

Step 4: Upload it to your website’s root directory

The root directory is the top level of your website’s file system, the same place where your robots.txt file lives. If you’re using cPanel hosting, this is usually the public_html folder. If you’re on WordPress with FTP access, it’s the folder that contains your wp-config.php file.

For WordPress users without direct FTP access: use a plugin like “File Manager” (available in the WordPress plugin directory) to upload the file directly from your WordPress dashboard.

Step 5: Verify it’s accessible

Open a browser and navigate to yourwebsite.com/llms.txt. You should see the plain text content of your file. If you see a 404 error, the file isn’t in the right location.

Step 6: Update it regularly

Your llms.txt isn’t a set-and-forget file. When you add major new content sections, update the key pages list. When your business description changes, update the about section. A stale llms.txt is better than none, but a current one is better still.

What to Include in Your llms.txt and What to Leave Out

Include:

  • Your most important pages are the ones you most want AI systems to find and understand. For an agency, this means your service pages, your pillar content, and your main FAQ or about pages.
  • A clear description of your area of expertise. Be specific. “We provide marketing services” is useless. ” We specialise in answer engine optimisation and LLM citation strategy for B2B businesses,” tells an AI exactly what you’re about.
  • Any content that directly answers the questions your target audience is asking AI tools. If you have a guide that covers a topic comprehensively, list it here.
  • Your publication name and author information if your site publishes content as a media property.

Leave out:

  • Pages you wouldn’t want AI systems citing include internal tools, staging pages, thin or low-quality content, privacy policies, and legal boilerplate.
  • Extremely long lists of every page of your site. The value of llms.txt is prioritisation, not comprehensiveness. A shorter, focused file is more useful to an AI than an exhaustive one.
  • Any information that’s confidential, internal, or not appropriate for public AI consumption.
  • Marketing copy or promotional language. Write it like documentation, not an ad.

Real Examples from Brands Already Using llms.txt

The adoption of llms.txt has been growing through 2025 and into 2026. Here are some real-world implementation patterns observed across early adopters:

Anthropic (anthropic.com/llms.txt): Anthropic’s own implementation is a useful reference. Their llms.txt includes a clear description of the organisation, links to their key documentation sections with brief descriptions, and structured guidance for AI systems crawling their technical docs. It’s a clean, practical implementation that follows the basic format closely.

Technical documentation sites: Software companies and developer tool providers have been early adopters, often using llms.txt to point AI systems to their API documentation, getting-started guides, and key reference pages, the content that developers are most likely to ask AI tools about.

Marketing agencies and consultancies: A growing number of agencies have started implementing llms.txt to list their service pages, case studies, and key educational content. For a business whose expertise is visible thinking and content strategy, having a well-written LLMs.txt is both a practical step and a signal to sophisticated clients that you understand the space.

News and media publishers: Some publishers have used llms.txt to list their key content categories and their most-cited articles, essentially giving AI tools a roadmap of their most authoritative content by topic.

The common thread across all these implementations: the most effective llms.txt files are specific, well-organised, and focused on genuinely important pages rather than trying to list everything.