What is llms.txt and Why Your Website Needs One

TL;DR

llms.txt is a Markdown file placed at your website root (example.com/llms.txt) that tells AI search engines what your site is about. It includes your organization name, one-paragraph description, and categorized links to products, resources, and contact info. Unlike robots.txt which controls crawler access, llms.txt provides context. Less than 5% of websites have one (Originality.ai), making it an early-adopter advantage. Create it in under 30 minutes: write a clear description, list key pages with URLs, deploy to your root directory. Two honest caveats: no major AI engine has publicly committed to reading llms.txt, and the file is worthless if a CDN bot rule is returning 403 to crawlers your robots.txt permits. Generate it from your existing content rather than maintaining it by hand, because a hand-maintained file goes stale within a quarter.

The llms.txt file is a plain-text file placed at the root of a website that provides AI systems and large language models with a structured, human-readable summary of the site's purpose, products, and key resources. The llms.txt standard was proposed by Jeremy Howard (co-founder of fast.ai) in 2024 as a complement to robots.txt, which tells crawlers what they can access but not what the site is about. While robots.txt controls permissions, llms.txt provides context. Fewer than 5% of websites have implemented an llms.txt file, according to analysis from Originality.ai. As AI search engines like ChatGPT, Perplexity, and Google AI Overviews become significant traffic sources (AI-referred traffic grew 527% year-over-year per BrightEdge), the llms.txt file is becoming an essential component of technical GEO (Generative Engine Optimization) strategy.

How Does llms.txt Differ from robots.txt?

The robots.txt file and the llms.txt file serve fundamentally different purposes. Robots.txt is a decades-old standard that tells web crawlers which pages they are allowed or disallowed from accessing. Robots.txt uses directives like "User-agent" and "Allow" or "Disallow" to control crawler behavior. The llms.txt file does not control access. Instead, llms.txt provides AI systems with a structured overview of what the website contains, what the organization does, and where the most important content lives. Think of robots.txt as the bouncer at the door (who gets in?) and llms.txt as the concierge in the lobby (here is what we offer and where to find it). Both files live at the root of the website, and both are important for AI search visibility, but they address different needs.

Three root files, three different questions

Adding a sitemap makes the division clearer, and the distinction matters because these files are frequently treated as substitutes.

The dependency runs one way and is worth stating plainly: an llms.txt file cannot help a crawler that is being blocked. If a WAF or CDN bot rule returns 403 to ClaudeBot while robots.txt reads Allow: /, adding llms.txt accomplishes exactly nothing. Verify access first, with a per-bot request rather than a browser visit:

curl -sI -A "ClaudeBot/1.0" https://yourdomain.com/llms.txt

Read the status code. A 403 there means the work described in the rest of this article is premature.

What is the llms.txt File Format?

The llms.txt file follows a simple Markdown-based format that is both human-readable and machine-parseable. The file begins with an H1 heading containing the site or organization name, followed by a blockquote with a one-paragraph description. Subsequent sections use H2 headings to organize content into categories. Each section contains a list of links with optional descriptions. The format is intentionally simple to encourage adoption.

The basic structure of an llms.txt file looks like this:

# Organization Name

> A one-paragraph description of the organization, its products,
> and its primary purpose. This description should be concise
> and informative, written for an AI system that needs to
> understand what this website is about.

Products and Services

- Product Name: Brief description of the product
- Service Name: Brief description of the service

Resources

- Blog: Articles about the main topic area
- Documentation: Technical documentation and API reference

Company

- About: Company background and team
- Contact: Contact information

Contact

- Website: https://example.com
- GitHub: https://github.com/example

The llms.txt specification also supports a companion file called llms-full.txt, which includes expanded descriptions for every link. The llms-full.txt file provides AI systems with deeper context when they need more detail than the summary version offers.

The description paragraph is the whole file

If you spend thirty minutes on llms.txt, spend twenty on the blockquote. It is the one part a model is most likely to consume verbatim, and it is where almost every implementation goes wrong in the same way: it describes a category instead of an entity.

Weak: "We are a leading provider of innovative marketing solutions that help businesses grow." Nothing here distinguishes the organization from ten thousand others. There is no product, no mechanism, no audience, and no checkable claim. A model reading this learns that you exist.

Strong: name what the thing is, what it does mechanically, and who it serves. Ours reads:

Echloe is an AI-agent-powered digital marketing platform that automates the full organic growth pipeline: keyword discovery, content creation, multi-channel distribution, audience engagement, GEO/SEO optimization, and conversion attribution.

Three properties make that usable. It names a category ("digital marketing platform") a model can slot into an answer about categories. It enumerates specific capabilities rather than adjectives, so it can be matched against a narrow question like "what tool handles conversion attribution for organic content". And it contains no unfalsifiable superlatives, which matters because "leading" and "innovative" are the words a model has learned to treat as marketing rather than fact.

Write it in third person using your actual name, not "we". The passage travels away from your site, and "we" has no referent once extracted, which is the same self-containment problem that governs citable body copy.

A real file, and what to notice about it

Ours lives at echloe.io/llms.txt and is about 8 KB across four sections: Product, Blog, Company, and contact details. The Blog section carries all 24 published articles, each as a link with its meta description.

Two decisions in there are worth stealing, and one gap is worth naming.

The blog section is generated, not written. Each entry's title, URL, and description come from the article's own frontmatter at deploy time. That is the difference between a file that stays accurate and one that describes your site as it was two quarters ago. More on this below, because it is the single highest-leverage implementation choice.

Contact is a section, not a footnote. Listing your domain, email, and platform profiles does the same entity-resolution work as the sameAs property in JSON-LD Organization schema: it tells a model that the LinkedIn page, the GitHub organization, and the website are one entity rather than three. Cross-platform mentions only accumulate into authority once something connects them.

We do not publish llms-full.txt. The specification supports it, this article recommends it, and we have not shipped one. It is listed here rather than quietly omitted because a post recommending a practice its own author skipped should say so. The reason is honest and unglamorous: with 24 articles the summary file already carries a full description per entry, so a full variant would mostly duplicate it. That calculus changes at a few hundred pages.

What Are the Benefits of Having an llms.txt File?

The benefits of implementing an llms.txt file span three categories: discoverability, authority, and efficiency. For discoverability, llms.txt gives AI crawlers a structured roadmap of website content that might otherwise require extensive crawling to discover. Research from Princeton University's GEO group indicates that AI systems preferentially cite content they can easily categorize and attribute to a specific entity. For authority, the llms.txt file establishes topical expertise by explicitly listing the organization's areas of knowledge and linking to supporting content. For efficiency, AI crawlers have limited crawl budgets, and llms.txt ensures the most important pages are identified immediately rather than relying on the crawler to find them through link discovery. Gartner projects that traditional search volume will decline 25% by 2026 as users shift to AI search, making early llms.txt adoption a competitive advantage.

What the evidence actually supports

The case for llms.txt is reasonable and it is weaker than most articles on the subject imply, including the earlier version of this one. Being straight about that is more useful than another confident claim.

No major AI engine has publicly committed to reading llms.txt. OpenAI, Anthropic, Google, and Perplexity have not documented it as a supported convention the way search engines documented sitemap.xml. Adoption is real on the publishing side and largely unconfirmed on the consuming side. Anyone telling you GPTBot parses your llms.txt is inferring, not citing.

Crawler-budget arguments are strongest for large sites. If a crawler can reach your whole site in a hundred requests, a roadmap saves it little. Our own logs recorded 172 AI crawler visits in a week across a 24-article site, which is ample coverage without any prioritization help. The efficiency argument scales with page count, and for a small site it is close to zero.

Correlation claims deserve scrutiny. Reports that sites adding llms.txt saw referral increases within 60 days are worth reading, and worth reading carefully: sites that adopt a brand-new convention are also sites actively working on AI visibility, publishing more, and fixing crawler access in the same period. That is a textbook confound, and none of the available reports control for it.

So why do it? Because the honest case does not need the inflated one. It costs half an hour. The downside is zero. The description paragraph forces you to state what your organization is in one falsifiable sentence, which is useful even if no crawler ever reads it. And if the convention does get adopted, the file is already there. That is a good bet at this price, which is a different claim from "llms.txt increases your traffic".

Our own position, for calibration: we publish llms.txt, our crawler access is confirmed at 172 visits a week with a 0% error rate, and our measured citation rate across Gemini, OpenAI, and Anthropic is zero against a fixed set of 54 test queries. llms.txt is not what stands between a young domain and citations.

How Do You Create an llms.txt File Step by Step?

Creating an llms.txt file involves five steps that can be completed in under 30 minutes.

Step 1: Write the header and description. Start with the organization name as an H1 heading. Write a one-paragraph blockquote that clearly describes what the organization does, what products or services it offers, and who it serves. This description is the most important part of the file because AI systems use it to understand the entity behind the website.

Step 2: List products and services. Create an H2 section called "Products and Services" (or a similar label). List each product or service as a Markdown link with a brief description. Use the actual URL where users can learn more about each item.

Step 3: List key resources. Create an H2 section for resources including the blog, documentation, API reference, case studies, or any other content that demonstrates expertise. These links help AI systems discover the content most likely to contain citable information.

Step 4: Add company and contact information. Include sections for company pages and contact details. List the website URL, social media profiles, GitHub repositories, and any other platforms where the organization has a presence. These links function similarly to sameAs properties in JSON-LD schema, helping AI systems connect the organization across platforms.

Step 5: Deploy the file. Place the llms.txt file at the root of the website so it is accessible at https://yourdomain.com/llms.txt. If creating a companion llms-full.txt file with expanded descriptions, place that file at the root as well. Verify both files are accessible by visiting the URLs in a browser.

Step 6: Generate it, so it cannot go stale

The five steps above produce a correct file. They also produce a file that will be wrong within a quarter, because every new article, renamed product, and retired page is a manual edit nobody remembers to make. A stale llms.txt is worse than none: it points a model at 404s and describes an offering you no longer sell.

The fix is to build it from the content you already maintain. Our own file is regenerated on every deploy: the header and description are fixed text, and the Blog section is assembled by reading the title, slug, and description from each article's frontmatter. In shell, the core of it is unremarkable:

for f in content/blog/*.md; do
  TITLE=$(grep -m1 '^title:' "$f" | sed 's/^title: *"\?//; s/"$//')
  SLUG=$(grep -m1 '^slug:'  "$f" | sed 's/^slug: *"\?//; s/"$//')
  DESC=$(grep -m1 '^description:' "$f" | sed 's/^description: *"\?//; s/"$//')
  echo "- ${TITLE}: ${DESC}" >> public/llms.txt
done

Any language does this in a dozen lines. The property that matters is not the implementation, it is that the frontmatter is the single source of truth for the article's title, URL, and description, so the same values appear in the page, the sitemap, and llms.txt without anyone reconciling three copies by hand.

Two failure modes to guard against, both of which we have hit:

A hand-edited generated file gets silently overwritten. Once generation is in place, editing the output is a change that vanishes on the next deploy. Edit the generator or the frontmatter, never the artifact. If a section genuinely needs to be hand-written, keep it in the generator's fixed header rather than patching the output.

Committing the file is not deploying it. A generation step that commits public/llms.txt and pushes proves nothing about what is live. Verify against the deployed URL with a cache-buster, because a CDN will happily serve you a stale copy and make a broken pipeline look fine:

curl -s "https://yourdomain.com/llms.txt?cb=$(date +%s)" | head -5

Which pages belong in it

Selectivity is the point, so the harder question is what to leave out. Include pages that are citable or that establish what you are: the homepage, product pages, documentation, and articles carrying original data or a definitive treatment of a topic.

Leave out anything a model cannot quote or that misrepresents your focus. Login and signup pages render nothing citable. Pricing pages date fast and a stale price is a wrong answer with your name on it. Tag and category archives are navigation rather than content. And thin posts are worse than absent: promoting a 400-word page tells a model that this is what you consider representative.

For a blog beyond roughly fifty posts, list the strongest twenty rather than all of them. The file is a recommendation, and a recommendation that includes everything is not one.

How Do AI Crawlers Use llms.txt in Practice?

AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity AI) access llms.txt when crawling a website for the first time or when refreshing their understanding of a domain. The llms.txt file provides these crawlers with immediate context about the organization and directs them to the most important content. While the llms.txt standard is not yet universally adopted by all AI systems, its adoption is growing as more AI companies recognize the value of structured site descriptions. According to analysis from Search Engine Journal, websites that implemented llms.txt alongside proper AI crawler configuration in robots.txt saw measurable increases in AI search referral traffic within 60 days of deployment.

How to check whether anything reads yours

You do not have to take anyone's word for it, including ours. Server logs record every request for /llms.txt with its user agent, so grep for the path and read which bots appear:

grep '/llms.txt' access.log | awk '{print $12}' | sort | uniq -c | sort -rn

Adjust the field number for your log format. Three outcomes, three conclusions. Regular fetches from AI crawler user agents mean the convention is being consumed on your site. Fetches only from browsers and generic tools mean you are looking at human curiosity and scrapers. No fetches at all, while your other pages are being crawled normally, is a real answer too: nothing is reading it yet, and the file is a cheap option on that changing rather than a live channel.

Run the same check on /robots.txt as a control. If robots.txt is fetched frequently and llms.txt never, that gap is the measurement, and it is more informative than any published adoption estimate.

What Common Mistakes Should Be Avoided?

Common mistakes when creating an llms.txt file include writing vague descriptions that do not clearly identify what the organization does, listing too many links without meaningful descriptions, using relative URLs instead of absolute URLs, including broken links that return 404 errors, and failing to update the file when products or resources change. The llms.txt file should be treated as a living document that is updated whenever significant content is added to the website. Echloe provides a free llms.txt generator at echloe.io that automatically creates properly formatted llms.txt and llms-full.txt files based on website analysis, ensuring the output follows the specification and includes all relevant content.

The mistakes that are specific to this file

The list above covers the obvious errors. Four more are specific to llms.txt and each has a concrete symptom.

Served with the wrong content type. Some hosts serve unknown extensions as application/octet-stream, which prompts a download rather than a read. curl -sI https://yourdomain.com/llms.txt | grep -i content-type should show text/plain or text/markdown.

Placed at the wrong host. Root files are per-host, so example.com/llms.txt does nothing for blog.example.com. A site whose content lives on a subdomain needs a file there too, which is the same trap that catches robots.txt.

Behind the bot rule that blocks everything else. If your CDN challenges unfamiliar user agents, /llms.txt is challenged along with every other path. The file existing and the file being readable by a crawler are separate facts, and only the per-bot curl distinguishes them.

Written for a search engine instead of a model. Keyword-stuffed descriptions and repeated brand names read as spam to a system whose job is producing a trustworthy answer. Write plain declarative prose. The audience is a reader that will paraphrase you to someone else.

FAQ

Where should I place my llms.txt file?

Place llms.txt at the root of your website domain, accessible at https://yourdomain.com/llms.txt. This location mirrors where robots.txt lives. AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity AI) check this location when crawling your site. Do not place llms.txt in a subdirectory or rename it. AI systems expect the exact filename at the root. You can verify deployment by visiting the URL directly in a browser, and verify it is readable by a crawler with curl -sI -A "ClaudeBot/1.0", which is a different check: a CDN bot rule can return 403 to a crawler on a URL that loads fine for you. Note also that root files are per-host, so a blog on a subdomain needs its own copy.

What is the difference between llms.txt and llms-full.txt?

llms.txt provides a concise summary with product names, categories, and brief descriptions. llms-full.txt includes expanded descriptions for every link, offering deeper context when AI systems need more detail. Both use the same Markdown format and both live at the website root. Most sites should start with llms.txt only, and many should stop there: if your summary file already carries a real description for every entry, a full variant mostly duplicates it. We publish llms.txt and not llms-full.txt for exactly that reason across 24 articles. The calculus changes at a few hundred pages, where the summary has to become a list of links and the expanded file is where the descriptions go.

How often should I update my llms.txt file?

Never by hand, which is the better answer than a frequency. Generate the file from your existing content at deploy time, reading titles, URLs, and descriptions from the frontmatter you already maintain, and the update question disappears. A hand-maintained file goes stale within a quarter because every new article and retired page is an edit nobody remembers, and a stale file is worse than none since it points models at 404s and describes an offering you no longer sell. If generation is genuinely not an option, review quarterly and treat any product launch or site restructure as a trigger. One caution once generation is in place: editing the generated file directly is a change that vanishes on the next deploy, so edit the generator or the frontmatter instead.

Does llms.txt actually improve AI search visibility?

The honest answer is that the evidence is thinner than most articles claim. No major AI engine has publicly documented llms.txt as a supported convention, so anyone asserting that GPTBot parses yours is inferring rather than citing. Reports of referral increases within 60 days of adoption do not control for an obvious confound: sites adopting a new convention are also sites actively fixing crawler access and publishing more in the same window. The crawl-budget argument is real but scales with size, and for a small site it is close to zero, since our own 24-article site receives 172 crawler visits a week without needing prioritization help. The case that survives scrutiny is different and still sound: it costs half an hour, the downside is zero, writing the description forces you to state what your organization is in one falsifiable sentence, and if the convention is adopted you are already there. For calibration, we publish one, our crawler access is confirmed, and our measured citation rate across three engines is zero.

Can I tell whether any AI system is actually reading my llms.txt?

Yes, from your server logs, and it is worth doing rather than assuming. Grep the access log for requests to /llms.txt and tally the user agents. Frequent fetches from AI crawler user agents mean the convention is being consumed on your site. Fetches only from browsers and generic HTTP clients mean you are seeing human curiosity and scrapers. No fetches at all, while your other pages are crawled normally, is also a real answer: nothing reads it yet, and the file is a cheap option rather than a live channel. Use /robots.txt as a control, since a large gap between the two request counts is more informative than any published adoption estimate.

Should llms.txt list every page on my site?

No, and doing so removes the reason the file exists. A sitemap is the exhaustive, unranked inventory of your URLs; llms.txt is a selective, editorial recommendation of what represents you. A file that lists everything is not a recommendation. Include the homepage, product pages, documentation, and articles with original data or definitive treatment of a topic. Leave out login and signup pages, which contain nothing quotable, pricing pages, which date fast and turn a stale number into a wrong answer with your name on it, tag and category archives, which are navigation rather than content, and thin posts, which are worse than absent because promoting a 400-word page tells a model that is what you consider representative. Past roughly fifty blog posts, list the strongest twenty.

Does llms.txt replace robots.txt, a sitemap, or schema markup?

No. The four answer different questions and none substitutes for another. Robots.txt says what a crawler may fetch. A sitemap says what exists. llms.txt says what is worth reading first and why. JSON-LD schema says what entities and relationships a specific page describes. The dependency order matters for sequencing your work: access first, since llms.txt cannot help a crawler receiving a 403, then structure and schema on the pages that carry your content, then llms.txt as the cheap editorial layer on top. Adding llms.txt while a WAF blocks ClaudeBot is effort spent in the wrong place.