The /llms.txt file – llms-txt

What it is

The /llms.txt file is a proposed standard for providing concise, structured information to help Large Language Models (LLMs) understand and interact with a website's content. It is designed for website owners, developers, and organizations who want to make their online information more accessible to LLMs, particularly for use cases like software documentation, e-commerce, education, and personal websites.

Main Features

Core Specification

• A Markdown file located at the root path /llms.txt (or optionally in a subpath)
• Uses a specific, consistent structure that is both human and LLM-readable
• Designed to be processed by both classical programming techniques (parsers, regex) and LLMs

Content Structure

• A required H1 heading with the project or site name
• A blockquote containing a short summary of the project
• Optional detailed information in paragraphs or lists
• Sections delimited by H2 headers containing curated lists of URLs to further detail
• An optional Optional section for secondary information that can be skipped if context is limited

Complementary Proposal

• Suggests that websites provide clean Markdown (.md) versions of their HTML pages at the same URL with a .md extension appended

How it works

For LLM Inference

When a user asks an LLM a question that requires context from a website, the LLM or its supporting tools can fetch the /llms.txt file. This file provides a curated overview and direct links to the most relevant Markdown documentation, allowing the LLM to efficiently gather necessary information without exceeding its context window by processing entire websites.

For Software Documentation

A project like FastHTML uses its /llms.txt file to point to key documentation, such as a quick start guide and an HTMX reference. Tools like llms_txt2ctx can then automatically expand this file into comprehensive context files (e.g., llms-ctx.txt) that bundle this information for direct use with an LLM.

For Website Integration

Website owners create an /llms.txt file following the specified format. They can use various available integrations (e.g., plugins for VitePress, Docusaurus, or Drupal) to automate the generation of this file and the accompanying Markdown versions of their web pages.

Key Points

• Addresses the critical limitation of LLM context windows being too small to process entire websites.
• Complements existing web standards like robots.txt and sitemap.xml but serves a different, inference-focused purpose.
• The information is primarily intended for use during inference (when a user is seeking assistance) rather than for model training.
• The format is versatile and can be applied to many domains, from software APIs to e-commerce product descriptions and educational resources.
• The proposal is community-driven and open for input and discussion.

Additional Details

• Availability: The specification is informal and hosted on a GitHub repository for public discussion.
• Integrations & Tools: Several tools exist to help implement the standard:
  • llms_txt2ctx: A CLI and Python module for parsing files and generating LLM context.
  • Plugins for popular static site generators like VitePress and Docusaurus.
  • Support for Drupal sites via the Drupal LLM Support module.
  • JavaScript and PHP libraries are also available.
• Directories: Websites that have implemented /llms.txt can be listed in community directories such as llmstxt.site and directory.llmstxt.cloud.
• Community: A Discord channel is available for sharing implementation experiences and best practices.