What is LLMs.txt and should you care?

By Tuukka Tähtinen

October 30th, 2025

With the rise of LLM based searching in recent years, you may have had the thought: Can we take advantage of this and how?

So far LLM based searches have been depending on processing web pages into readable data to provide results for users and we have been lacking the tools to help instruct LLMs on where to look.

However, following this proposal by Jeremy Howard of Answer.AI, in the near future we may be able to do just that!

What is llms.txt?

The llms.txt file is a proposed standard for a Large Language Model (LLM) readable markdown file to allow quicker and more precise processing of your website contents. The file is set to include key information about your site and its contents, including a title and description of the site in general, and sections with lists of pages within the site. Sections could include "Services", "Blog", "Case studies" and so on.

Lists inside these sections would include:

• The title of the page

• The URL of the page

• An optional description for the page.

The expected use case outlined in the proposal is that the llms.txt file would be used during inference only, and not for training data. The additional descriptions attributed to each page would in return give LLM agents an index for quickly determining the relevant pages to process to procure an accurate answer for the user.

Effectively this would be similar to a sitemap providing information to crawlers, but with the addition of also describing the contents of the pages.

The llms.txt file would contain information for the pages under the directory it's is hosted in. In other words, an llms.txt file at the root of the website would contain information for the entire site, while an llms.txt file hosted at /docs, for example, would contain information for the pages under the /docs path.

LLM Readable Markdown files

In addition to implementing a markdown file index of pages, it is also proposed that markdown versions of those pages on the same path with .md appended would be implemented.

A fine idea.

Instead of requiring LLM agents to first convert the HTML page into a readable format and then processing the data, risking the content getting misinterpreted along the way, we would provide a markdown page for them to process. Including these markdown pages alongside your usual pages shouldn’t cause any issues with traditional search engines, as file paths ending with .md won't be indexed by search engine crawlers by default, so no worries there. Additionally, link relations should be included on the original page to point to the llms.txt file with information regarding the page, along with a link relation pointing to the markdown version of the page.

But there are some user experience concerns.

At this time there is no proposed standardized way of directing the agents on how to direct users to the HTML page the content would be on, instead of the markdown page the LLM would be reading. This would lead users to view the markdown formatted page, leading to a worse viewing experience and leave them to find the original page on their own, and as a result, reducing traffic to that page.

Should you implement the practice

So should you implement llms.txt files in your website?

You could, but there's no pressure to.

There is no urgent need to implement them s currently there is no indication that any LLM search agents use these files or even look for them. Anthropic does host both the llms.txt and the (now dropped from the proposal) llms-full.txt file on the documentation site for ClaudeAI, but has not stated on whether Claude supports the files or not. As a result we'll have to assume LLM agents are still utilizing search results affected by traditional SEO practices.

For a documentation site it could make sense to include it if LLMs begin using them.

It is worth to note that so far no LLM agent has claimed to be using these files so, at the end of the day, this is all just speculation and subject to change. It may still be smart to begin looking into the required changes you may have to make to begin implementing these accommodations in the future once support for the proposal or something similar gets rolled out by LLM agents. While we can't predict the future, it does seem inevitable that something similar would be implemented as it would be useful for the LLM providers.

Moving forward

The proposal was initially put up in September of 2024, and new suggestions are being brought up on the GitHub issues for the proposal, such as updating the file name to .md to match the content format. These suggestions could lead to the standard being updated in the future to help align with pre-existing standards such as the sitemap files.

It's an interesting development towards normalizing AI-powered browsing of the internet, but it is unlikely to reduce the need for traditional SEO practices, and will mostly serve as an improvement to the LLM agent search experience.

However, while the proposal did get an update dropping the llms-full.txt file specification in 2026, the proposal does seem a little slow to update and will likely need some more time to get off the ground, so we'll have to keep an eye on it to see what might change come 2027 and if it gets adopted.