What is LLMS.txt and should you care?

By Tuukka Tähtinen

October 30th, 2025

With the rise of LLM based searching in recent years, you may have had the thought: Can we take advantage of this and how?

So far LLM based searches have been depending on processing web pages into readable data to provide results for users and we have been lacking the tools to help instruct LLMs on where to look.

However, following this proposal by Jeremy Howard of Answer.AI, in the near future we may be able to do just that!

What is llms.txt?

The llms.txt file is a proposed standard for a Large Language Model (LLM) readable markdown file to allow quicker and more precise processing of your website contents. The file is set to include key information about your site and its contents, including a title and description of the site in general, and sections with lists of pages within the site. Sections could include "Services", "Blog", "Case studies" and so on.

Lists inside these sections would include:

• The title of the page

• The URL of the page

• An optional description for the page.

The expected use case outlined in the proposal is that the llms.txt file would be used during inference only, and not for training data. The additional descriptions attributed to each page would in return give LLM agents an index for quickly determining the relevant pages to process to procure an accurate answer for the user.

Effectively this would be similar to a sitemap providing information to crawlers, but with the addition of also describing the contents of the pages.

LLM Readable Markdown files

In addition to implementing a markdown file index of pages, it is also proposed that markdown versions of those pages on the same path with .md appended would be implemented.

A fine idea.

Instead of requiring LLM agents to first convert the HTML page into a readable format and then processing the data, risking the content getting misinterpreted along the way, we would provide a markdown page for them to process. Including these markdown pages alongside your usual pages shouldn’t cause any issues with traditional search engines, as file paths ending with .md won't be indexed by search engine crawlers by default, so no worries there.

But there are some user experience concerns.

At this time there is no proposed standardised way of directing the agents on how to direct users to the HTML page the content would be on, instead of the markdown page the LLM would be reading. This would lead users to view the markdown formatted page, leading to a worse viewing experience and leave them to find the original page on their own, and as a result, reducing traffic to that page.

The llms-full.txt file

If you have read the proposal, you may have spotted a reference to a llms-ctx-full.txt file mentioned in relation to the FastHTML project following the proposed standard. This project is maintained by the same company the llms.txt proposal is made by.

There is so far no proposed standard for what a full llms text file should contain, how it should be laid out or how to properly direct the LLM reading it to cite the correct pages in relation to the content to users making the search queries. And so again there is the risk of LLMs linking users to this file instead of the source page for the resulting information.

However, the implied use for a llms-full.txt file is to include the content of your site in one single markdown file for LLMs, removing the need to parse through the entire site.

In their use case they are using it to include their entire documentation for more efficient processing by LLM agents. This may be a good solution for some use cases, like searching through the documentation of a product, giving the LLM all of the relevant information in one handy packet. What it necessarily isn’t good for, is bringing in user traffic to a website as the LLM will provide users with the needed information, thus removing the need to visit the site.

This does raise the question: How does the llms-full.txt file work together with the llms.txt file? Or does it do so at all?

At this current time it doesn’t seem like it does.

From the perspective of an LLM provider, if an llms-full.txt file is available, why bother parsing through the URLs listed in the llms.txt file instead of just using the information that is already available to them? The benefits are skewed towards the LLM provider and you should consider carefully if you believe that helping LLM agents read your entire site from a single file can help you create traffic or business after a query has been answered without ever having to visit the site.

Should you use either

So should you implement an llms.txt or an llms-full.txt file in your website?

You could, but there's no pressure to.

There is no urgent need to implement either as currently there is no indication that any LLM search agents use these files or even look for them. Anthropic does host both the llms.txt and llms-full.txt file on the documentation site for ClaudeAI, but has not stated on whether Claude supports the files or not. As a result we'll have to assume LLM agents are still utilizing search results affected by traditional SEO practices.

At the current state of the proposal it seems like you should choose just one based on your use case. Either you implement the llms.txt file to provide an index for LLMs to help direct users to your site, or you implement the llms-full.txt file to include all relevant information on your site. For a documentation site it could make sense to include it if LLMs begin using them. For a regular site I would consider using a regular llms.txt file instead.

It is worth to note that so far no LLM agent has claimed to be using either of these files so, at the end of the day, this is all just speculation and subject to change. It may still be smart to begin looking into the required changes you may have to make to begin implementing these accommodations in the future once support for the proposal or something similar gets rolled out by LLM agents. While we can't predict the future, it does seem inevitable that something similar would be implemented as it would be useful for the LLM providers.

Moving forward

The proposal was initially put up in September of 2024, and new suggestions are being brought up on the GitHub issues for the proposal, such as updating the file name to .md to match the content format. These suggestions could lead to the standard being updated in the future to help align with pre-existing standards such as the sitemap files.

It's an interesting development towards normalizing AI-powered browsing of the internet, but it is unlikely to reduce the need for traditional SEO practices, and will mostly serve as an improvement to the LLM agent search experience.

However the proposal does seem a little slow to update and will likely need some more time to get off the ground, so we'll have to keep an eye on it to see what might change come 2026 and if it gets adopted.