Skip to content

About

Generate llms.txt for any docs site or repository: build llms.txt and llms-full.txt from a directory of Markdown or HTML or from a sitemap URL, validate existing files against the llmstxt.org convention, optionally check every link is live, and print the link tag. Standard library only, deterministic, CLI plus GitHub Action and pre-commit hook.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

llms-txt-gen: generate llms.txt for any docs site or repository

Generate llms.txt for any docs site or repository: point llms-txt-gen at a directory of Markdown or HTML, or at a sitemap URL, and it writes an llms.txt (and, with --full, an llms-full.txt) that follows the convention published at llmstxt.org. It also checks an existing file against that convention and prints the HTML tag that tells crawlers where the file lives. Standard library only, no model in the loop, deterministic output, runs in CI.

CI License: MIT Python 3.11+

Who it is for: maintainers of documentation sites and open-source repositories who want their docs to be read well by AI assistants and coding agents, and who want the file rebuilt in CI rather than kept by hand. Project site: https://basitalisandhu.github.io/llms-txt-gen/.

Why llms.txt

Language models and the tools built on them read documentation the same way a hurried person does: they land on a page, skim, and give up when the navigation, cookie banners and scripts crowd out the text. llms.txt is a plain Markdown file at the root of a site (/llms.txt) that says, in a few hundred words, what the project is and which pages matter, with one line per page: a name, a link and a short note. A model that reads it first spends its limited context on the right pages. llms-full.txt goes one step further and inlines the content of those pages so a tool can load the whole documentation set in one request. Keeping either file by hand drifts out of date within weeks; this tool regenerates them from the source files or the live sitemap on every build.

Quickstart

pipx install git+https://github.com/basitalisandhu/llms-txt-gen      # see "Install" below

llms-txt-gen generate docs/ --out llms.txt --base-url https://docs.example.com
llms-txt-gen generate https://docs.example.com/sitemap.xml --out llms.txt --full
llms-txt-gen check llms.txt
llms-txt-gen serve-snippet

Output for the bundled examples/docs/ (committed as examples/llms.txt), from a clone of this repository:

$ llms-txt-gen generate examples/docs --stdout --base-url https://acme.example.com/docs --strip-ext --optional 'changelog*'
# Acme Widgets

> Acme Widgets is a small HTTP service for creating and shipping widgets. These pages are the example input for llms-txt-gen; run llms-txt-gen generate examples/docs --stdout to see the output.

## Guides

- [Authentication](https://acme.example.com/docs/guides/authentication): Every request carries a short-lived token in the Authorization header. Tokens expire after one hour and can be rotated at any time.
- [Quickstart](https://acme.example.com/docs/guides/quickstart): Create your first widget in five minutes.

## Reference

- [HTTP API](https://acme.example.com/docs/reference/api): Every endpoint, with request and response examples.
- [Command line](https://acme.example.com/docs/reference/cli): Flags and subcommands of the acme command.

## Optional

- [Changelog](https://acme.example.com/docs/changelog): All notable changes to Acme Widgets, newest first.

Without --base-url the links stay relative (guides/quickstart.md), and without --optional the changelog is listed under ## Docs. llms-txt-gen check examples/llms.txt then reports ok and one info line: 3 section(s), 5 link(s).

Install

Container image (linux/amd64 and linux/arm64), published to GitHub Packages on every release. Mount the docs or repository at /work; relative paths, including --out, resolve from there:

docker run --rm -v "$PWD:/work" ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 generate docs --out llms.txt --full
docker run --rm -v "$PWD:/work:ro" ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 check llms.txt --strict

The image runs as uid 1000, so the mounted directory must be writable by that user for generate. Each image is signed with cosign (keyless) and has a build provenance attestation and an SPDX SBOM (attached to the GitHub Release). To verify:

cosign verify ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 \
  --certificate-identity-regexp '^https://github.com/basitalisandhu/llms-txt-gen/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
gh attestation verify oci://ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 --owner basitalisandhu

Python package: requires Python 3.11 or newer and nothing else. PyPI publication is pending, so install from the repository:

pipx install git+https://github.com/basitalisandhu/llms-txt-gen                      # isolated CLI install
uvx --from git+https://github.com/basitalisandhu/llms-txt-gen llms-txt-gen --help    # run without installing
pip install git+https://github.com/basitalisandhu/llms-txt-gen                       # into the current environment
git clone https://github.com/basitalisandhu/llms-txt-gen && cd llms-txt-gen && uv sync   # for development

Once published to PyPI:

pip install llms-txt-gen

The other short forms work then too: pipx install llms-txt-gen, uvx llms-txt-gen --help.

Commands

Command What it does
llms-txt-gen generate SOURCE --out llms.txt Build llms.txt from a directory of .md, .mdx, .markdown, .html or .htm files, or from a sitemap URL (sitemap.xml or a sitemap index). --full also writes llms-full.txt; --stdout prints instead of writing.
llms-txt-gen check FILE-OR-URL Validate against the convention: H1 title first, optional blockquote summary, H2 sections of - [name](url): notes lines, no deeper headings, Optional last, no duplicate links, and for local files no broken relative links. --format json, --strict (warnings fail), --fetch (request every http(s) link, see below). Exit 1 on errors, 2 on usage errors.
llms-txt-gen serve-snippet Print <link rel="alternate" type="text/markdown" href="/llms.txt" title="llms.txt"> for the site <head>. --full adds the llms-full.txt tag, --header prints HTTP Link: headers instead, --href sets the location.

generate options

Option Meaning
--title, --summary, --body TEXT Override the H1, the blockquote and add free paragraphs after it. Defaults come from the root README.md or index.md (title and first paragraph); for a sitemap the host name is the title.
--base-url URL Prefix for links generated from a directory, so guides/start.md becomes https://docs.example.com/guides/start.md.
--strip-ext Drop .md and .html from generated links and turn index pages into their folder URL.
--include GLOB, --exclude GLOB Restrict or skip files (repeatable). node_modules, .git, virtual environments and build directories are always skipped.
--optional GLOB URLs or section names matching the glob go to the conventional ## Optional section, which models may skip when context is short.
--nav MKDOCS_YML Take section names, page order and link names from the nav list of an MkDocs mkdocs.yml (directory sources only; SOURCE is the MkDocs docs_dir). See "MkDocs navigation" below.
--max-pages N Page cap when reading a sitemap (default 200). Sitemap indexes are followed, at most 50 sitemaps are read.
--timeout SECONDS Per-request timeout for sitemap and page fetches (default 10). Responses over 5 MB are rejected.

How the content is chosen

For local Markdown pages, front matter llms: false excludes the page from both outputs, including a root README or index. Front matter llms-section: Name overrides the folder-derived section; an empty value keeps the default section.

  • Title: front matter title:, else the first # Heading (or Setext underline heading), else the HTML <h1>, else <title> with a trailing | Site or - Site removed, else the file name humanised.
  • Note after the link: front matter description:, else the first prose paragraph after the title (code blocks, lists, quotes and tables are skipped), else the HTML <meta name="description">, else the first <p>. Truncated to 200 characters at a sentence or word boundary.
  • Section: the first path segment, humanised (getting-started/ becomes ## Getting started). Files at the root go under ## Docs, which is rendered first. A folder's index.md is listed first in its section. The root README.md or index.md supplies the document title and summary and is not listed.
  • Order: sections alphabetical (Docs first, Optional last), pages in file order. Two runs over the same input give byte-identical output.
  • llms-full.txt: the title and summary, then every page as # Title, Source: url and its content (Markdown verbatim without front matter, HTML reduced to text with navigation, scripts and styles removed).

MkDocs navigation: generate --nav

MkDocs projects already say how their docs are organised in mkdocs.yml. With --nav, those choices are used instead of folder names and file order:

llms-txt-gen generate docs --nav mkdocs.yml --out docs/llms.txt --base-url https://docs.example.com --strip-ext
  • A top-level group (- User Guide: followed by a nested list) becomes ## User Guide; deeper groups stay in their top-level section. Pages listed at the top level go to ## Docs.
  • Sections and pages follow the nav order. A title written in nav (- License: about/license.md) becomes the link name.
  • Pages that are not in nav keep today's rules and come after the nav pages. A page in a folder that nav maps to a section (user-guide/extra.md when nav lists other user-guide/ pages under "User Guide") joins that section.
  • Nav paths are relative to the docs folder, so SOURCE must be that folder (usually docs/). External links in nav are skipped.

Only the nav: block is read, with a small parser in the standard library. It understands block lists nested by indentation, - path.md, - Title: path.md, - Title: groups, quoted values and # comments. Flow style (nav: [a.md, b.md]), anchors and multi-line values are refused with the file and line number rather than guessed at.

Link liveness: check --fetch

A hand-kept llms.txt goes stale when pages move. check --fetch requests each distinct http or https link once: HEAD first, then GET if HEAD gets a status of 400 or above (some servers refuse HEAD). The body is never read. A final status of 400 or above, a network error, more than 5 redirects or a redirect to a non-HTTP URL is reported as E007 on the link's line. --max-links N (default 100) caps the requests; links beyond the cap are counted in an I002 line. --timeout applies to each request. Relative links are not requested; for a local file they are checked on disk as before (E006).

$ llms-txt-gen check llms.txt --fetch
llms.txt: problems found
  info    I001 llms.txt: 1 section(s), 2 link(s)
  error   E007 llms.txt:8: link returned HTTP 404: https://llmstxt.org/no-such-page

Network use

generate with a URL fetches the sitemap and each page it lists; check with a URL fetches that file, and check --fetch requests the links in it. Nothing else touches the network. Only http and https URLs are accepted, every request has a timeout, bodies are capped at 5 MB (also after gzip decompression), the page count is capped by --max-pages, and the User-Agent identifies the tool. Pages that fail to fetch are skipped with a message on stderr and do not fail the run.

Checks

Code Level Meaning
E001 error No H1 title, or an empty one. The first content line must be # Name.
E002 error Content or a section before the H1.
E003 error A heading deeper than H2. The convention allows only the H1 title and H2 section names.
E004 error More than one H1.
E005 error A list item in a section that is not - [name](url) or - [name](url): notes.
E006 error A relative link that does not resolve next to the file (local files only).
E007 error With --fetch: an http(s) link returned 400 or above, could not be reached, or redirected more than 5 times or to a non-HTTP URL.
W001 warning No blockquote summary after the title.
W002 warning No H2 sections.
W003 warning A section with no links.
W004 warning The same URL linked twice.
W005 warning Optional is not the last section.
W007 warning Duplicate section name.
W008 warning The summary is shorter than ten characters.
W009 warning Prose inside a section; sections should contain only link lines.
I001 info Section and link counts.
I002 info With --fetch: how many links were not requested because of --max-links.

CI usage

Regenerate on every change to the docs and fail the build when the committed file is stale or invalid:

llms-txt-gen generate docs/ --out llms.txt --base-url https://docs.example.com
git diff --exit-code llms.txt
llms-txt-gen check llms.txt --strict

GitHub Action

name: llms-txt
on:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  llms-txt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: basitalisandhu/[email protected]      # pin a release tag
        with:
          source: docs
          out: llms.txt
          base-url: https://docs.example.com
          full: "true"
      - run: git diff --exit-code llms.txt llms-full.txt || echo "llms.txt changed; commit it"

Inputs: source (directory or sitemap URL), out, full, base-url, strip-ext, optional (space-separated globs), nav (path to mkdocs.yml), max-pages, timeout, check (run check --strict on the result, default true), python-version, version (PyPI version; empty installs the action's own checkout). Outputs: out, links, sections. See action.yml.

pre-commit

Validate llms.txt before every commit:

repos:
  - repo: https://github.com/basitalisandhu/llms-txt-gen
    rev: v0.4.0
    hooks:
      - id: llms-txt-check

Troubleshooting

Real messages and what to do about them. Every error goes to stderr and exits 2; findings in a checked file exit 1.

Message Cause and fix
ERROR: Package 'llms-txt-gen' requires a different Python: 3.10.12 not in '>=3.11' The tool needs Python 3.11 or newer. Use pipx install --python python3.12 ..., uvx, or the container image.
error: docs is not a directory or a URL SOURCE must be an existing folder, or a URL starting with http:// or https://. Run the command from the repository root or pass an absolute path.
error: no Markdown or HTML files found under ... Only .md, .mdx, .markdown, .html and .htm files are read. Hidden folders, node_modules, build, dist and virtual environments are always skipped; check your --include and --exclude globs and any llms: false front matter.
error: llms.txt is UTF-16 text; save it as UTF-8 and run again Windows PowerShell 5 writes UTF-16 with >. Use Set-Content -Encoding utf8, or regenerate the file with generate --out.
error: llms.txt is not UTF-8 text (byte 0xe9 at offset 5); save it as UTF-8 and run again The file was saved in a legacy encoding such as Windows-1252. Re-save it as UTF-8 in your editor.
error: cannot write llms.txt: Permission denied (container) The image runs as uid 1000. Mount a folder that user can write to, or run with --user "$(id -u):$(id -g)".
error: cannot write out: Is a directory --out names a file, not a folder: use --out out/llms.txt.
error: sitemap is not well-formed XML: ... The URL did not return a sitemap; often it is an HTML 404 page or a login wall. Open the URL in a browser, and try /sitemap_index.xml or the location named in robots.txt.
error: the sitemap yielded no pages The sitemap parsed but listed no <url><loc>. Point at the child sitemap, or raise --max-pages if it is set very low.
skipped https://...: HTTP Error 403: Forbidden A page refused the request. The run continues without it; some hosts block unknown user agents.
E006 broken local link on a generated file The links are relative to the docs folder but the file was written elsewhere. Write --out inside the docs folder, or pass --base-url so the links are absolute.
note: nav lists guide/a.md, which is not under . With --nav, SOURCE must be the MkDocs docs_dir (usually docs/), because nav paths are relative to it.
error: mkdocs.yml:4: 'nav:' must be followed by a block list ... The nav parser reads block lists only. Rewrite nav: [a.md, b.md] as one - a.md item per line.

Frequently asked questions

What goes in llms.txt? An H1 with the project name, a one-paragraph blockquote summary, optionally a few paragraphs of context without headings, then H2 sections each holding a list of - [page name](url): one line about the page. A section named Optional holds pages a model may skip. That is the whole convention; llms-txt-gen check enforces it and the README of llmstxt.org documents it.

Does it need network access? Only when you pass a sitemap URL to generate or a URL to check. Directory mode and file mode never make a request, which is why they are safe in CI and in pre-commit hooks.

Can I keep the file by hand and only check it? Yes. check is independent of generate. Many projects write the summary by hand and generate only the link lists; use --title, --summary and --body to keep the prose fixed while the lists are regenerated.

How do I tell crawlers the file exists? Serve it at /llms.txt, and add the tag from serve-snippet to the site <head> (or the Link: header from serve-snippet --header). The tag is a discoverability hint, not part of the convention, so leaving it out does not make the file invalid.

Roadmap

  • Read Docusaurus sidebars and the Sphinx toctree for section names and order, as --nav does for MkDocs.
  • A diff command that shows which links were added or removed since the committed file.

Contributing

Issues and pull requests are welcome; the starter list is in docs/good-first-issues.md. Run make check (ruff and pytest) before opening a pull request; see CONTRIBUTING.md. Security problems: see SECURITY.md.

Sibling projects

More tools by the same author: https://github.com/basitalisandhu

Licence

MIT, see LICENSE. Copyright 2026 Muhammad Basit Ali.

About

Generate llms.txt for any docs site or repository: build llms.txt and llms-full.txt from a directory of Markdown or HTML or from a sitemap URL, validate existing files against the llmstxt.org convention, optionally check every link is live, and print the link tag. Standard library only, deterministic, CLI plus GitHub Action and pre-commit hook.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages