Generate llms.txt for any docs site or repository: point llms-txt-gen at a directory of Markdown or HTML, or at a sitemap URL, and it writes an llms.txt (and, with --full, an llms-full.txt) that follows the convention published at llmstxt.org. It also checks an existing file against that convention and prints the HTML tag that tells crawlers where the file lives. Standard library only, no model in the loop, deterministic output, runs in CI.
Who it is for: maintainers of documentation sites and open-source repositories who want their docs to be read well by AI assistants and coding agents, and who want the file rebuilt in CI rather than kept by hand. Project site: https://basitalisandhu.github.io/llms-txt-gen/.
Language models and the tools built on them read documentation the same way a hurried person does: they land on a page, skim, and give up when the navigation, cookie banners and scripts crowd out the text. llms.txt is a plain Markdown file at the root of a site (/llms.txt) that says, in a few hundred words, what the project is and which pages matter, with one line per page: a name, a link and a short note. A model that reads it first spends its limited context on the right pages. llms-full.txt goes one step further and inlines the content of those pages so a tool can load the whole documentation set in one request. Keeping either file by hand drifts out of date within weeks; this tool regenerates them from the source files or the live sitemap on every build.
pipx install git+https://github.com/basitalisandhu/llms-txt-gen # see "Install" below
llms-txt-gen generate docs/ --out llms.txt --base-url https://docs.example.com
llms-txt-gen generate https://docs.example.com/sitemap.xml --out llms.txt --full
llms-txt-gen check llms.txt
llms-txt-gen serve-snippetOutput for the bundled examples/docs/ (committed as examples/llms.txt), from a clone of this repository:
$ llms-txt-gen generate examples/docs --stdout --base-url https://acme.example.com/docs --strip-ext --optional 'changelog*'
# Acme Widgets
> Acme Widgets is a small HTTP service for creating and shipping widgets. These pages are the example input for llms-txt-gen; run llms-txt-gen generate examples/docs --stdout to see the output.
## Guides
- [Authentication](https://acme.example.com/docs/guides/authentication): Every request carries a short-lived token in the Authorization header. Tokens expire after one hour and can be rotated at any time.
- [Quickstart](https://acme.example.com/docs/guides/quickstart): Create your first widget in five minutes.
## Reference
- [HTTP API](https://acme.example.com/docs/reference/api): Every endpoint, with request and response examples.
- [Command line](https://acme.example.com/docs/reference/cli): Flags and subcommands of the acme command.
## Optional
- [Changelog](https://acme.example.com/docs/changelog): All notable changes to Acme Widgets, newest first.
Without --base-url the links stay relative (guides/quickstart.md), and without --optional the changelog is listed under ## Docs. llms-txt-gen check examples/llms.txt then reports ok and one info line: 3 section(s), 5 link(s).
Container image (linux/amd64 and linux/arm64), published to GitHub Packages on every release. Mount the docs or repository at /work; relative paths, including --out, resolve from there:
docker run --rm -v "$PWD:/work" ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 generate docs --out llms.txt --full
docker run --rm -v "$PWD:/work:ro" ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 check llms.txt --strictThe image runs as uid 1000, so the mounted directory must be writable by that user for generate. Each image is signed with cosign (keyless) and has a build provenance attestation and an SPDX SBOM (attached to the GitHub Release). To verify:
cosign verify ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 \
--certificate-identity-regexp '^https://github.com/basitalisandhu/llms-txt-gen/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
gh attestation verify oci://ghcr.io/basitalisandhu/llms-txt-gen:0.4.0 --owner basitalisandhuPython package: requires Python 3.11 or newer and nothing else. PyPI publication is pending, so install from the repository:
pipx install git+https://github.com/basitalisandhu/llms-txt-gen # isolated CLI install
uvx --from git+https://github.com/basitalisandhu/llms-txt-gen llms-txt-gen --help # run without installing
pip install git+https://github.com/basitalisandhu/llms-txt-gen # into the current environment
git clone https://github.com/basitalisandhu/llms-txt-gen && cd llms-txt-gen && uv sync # for developmentOnce published to PyPI:
pip install llms-txt-genThe other short forms work then too: pipx install llms-txt-gen, uvx llms-txt-gen --help.
| Command | What it does |
|---|---|
llms-txt-gen generate SOURCE --out llms.txt |
Build llms.txt from a directory of .md, .mdx, .markdown, .html or .htm files, or from a sitemap URL (sitemap.xml or a sitemap index). --full also writes llms-full.txt; --stdout prints instead of writing. |
llms-txt-gen check FILE-OR-URL |
Validate against the convention: H1 title first, optional blockquote summary, H2 sections of - [name](url): notes lines, no deeper headings, Optional last, no duplicate links, and for local files no broken relative links. --format json, --strict (warnings fail), --fetch (request every http(s) link, see below). Exit 1 on errors, 2 on usage errors. |
llms-txt-gen serve-snippet |
Print <link rel="alternate" type="text/markdown" href="/llms.txt" title="llms.txt"> for the site <head>. --full adds the llms-full.txt tag, --header prints HTTP Link: headers instead, --href sets the location. |
| Option | Meaning |
|---|---|
--title, --summary, --body TEXT |
Override the H1, the blockquote and add free paragraphs after it. Defaults come from the root README.md or index.md (title and first paragraph); for a sitemap the host name is the title. |
--base-url URL |
Prefix for links generated from a directory, so guides/start.md becomes https://docs.example.com/guides/start.md. |
--strip-ext |
Drop .md and .html from generated links and turn index pages into their folder URL. |
--include GLOB, --exclude GLOB |
Restrict or skip files (repeatable). node_modules, .git, virtual environments and build directories are always skipped. |
--optional GLOB |
URLs or section names matching the glob go to the conventional ## Optional section, which models may skip when context is short. |
--nav MKDOCS_YML |
Take section names, page order and link names from the nav list of an MkDocs mkdocs.yml (directory sources only; SOURCE is the MkDocs docs_dir). See "MkDocs navigation" below. |
--max-pages N |
Page cap when reading a sitemap (default 200). Sitemap indexes are followed, at most 50 sitemaps are read. |
--timeout SECONDS |
Per-request timeout for sitemap and page fetches (default 10). Responses over 5 MB are rejected. |
For local Markdown pages, front matter llms: false excludes the page from both
outputs, including a root README or index. Front matter llms-section: Name
overrides the folder-derived section; an empty value keeps the default section.
- Title: front matter
title:, else the first# Heading(or Setext underline heading), else the HTML<h1>, else<title>with a trailing| Siteor- Siteremoved, else the file name humanised. - Note after the link: front matter
description:, else the first prose paragraph after the title (code blocks, lists, quotes and tables are skipped), else the HTML<meta name="description">, else the first<p>. Truncated to 200 characters at a sentence or word boundary. - Section: the first path segment, humanised (
getting-started/becomes## Getting started). Files at the root go under## Docs, which is rendered first. A folder'sindex.mdis listed first in its section. The rootREADME.mdorindex.mdsupplies the document title and summary and is not listed. - Order: sections alphabetical (
Docsfirst,Optionallast), pages in file order. Two runs over the same input give byte-identical output. llms-full.txt: the title and summary, then every page as# Title,Source: urland its content (Markdown verbatim without front matter, HTML reduced to text with navigation, scripts and styles removed).
MkDocs projects already say how their docs are organised in mkdocs.yml. With --nav, those choices are used instead of folder names and file order:
llms-txt-gen generate docs --nav mkdocs.yml --out docs/llms.txt --base-url https://docs.example.com --strip-ext- A top-level group (
- User Guide:followed by a nested list) becomes## User Guide; deeper groups stay in their top-level section. Pages listed at the top level go to## Docs. - Sections and pages follow the nav order. A title written in nav (
- License: about/license.md) becomes the link name. - Pages that are not in nav keep today's rules and come after the nav pages. A page in a folder that nav maps to a section (
user-guide/extra.mdwhen nav lists otheruser-guide/pages under "User Guide") joins that section. - Nav paths are relative to the docs folder, so SOURCE must be that folder (usually
docs/). External links in nav are skipped.
Only the nav: block is read, with a small parser in the standard library. It understands block lists nested by indentation, - path.md, - Title: path.md, - Title: groups, quoted values and # comments. Flow style (nav: [a.md, b.md]), anchors and multi-line values are refused with the file and line number rather than guessed at.
A hand-kept llms.txt goes stale when pages move. check --fetch requests each distinct http or https link once: HEAD first, then GET if HEAD gets a status of 400 or above (some servers refuse HEAD). The body is never read. A final status of 400 or above, a network error, more than 5 redirects or a redirect to a non-HTTP URL is reported as E007 on the link's line. --max-links N (default 100) caps the requests; links beyond the cap are counted in an I002 line. --timeout applies to each request. Relative links are not requested; for a local file they are checked on disk as before (E006).
$ llms-txt-gen check llms.txt --fetch
llms.txt: problems found
info I001 llms.txt: 1 section(s), 2 link(s)
error E007 llms.txt:8: link returned HTTP 404: https://llmstxt.org/no-such-page
generate with a URL fetches the sitemap and each page it lists; check with a URL fetches that file, and check --fetch requests the links in it. Nothing else touches the network. Only http and https URLs are accepted, every request has a timeout, bodies are capped at 5 MB (also after gzip decompression), the page count is capped by --max-pages, and the User-Agent identifies the tool. Pages that fail to fetch are skipped with a message on stderr and do not fail the run.
| Code | Level | Meaning |
|---|---|---|
| E001 | error | No H1 title, or an empty one. The first content line must be # Name. |
| E002 | error | Content or a section before the H1. |
| E003 | error | A heading deeper than H2. The convention allows only the H1 title and H2 section names. |
| E004 | error | More than one H1. |
| E005 | error | A list item in a section that is not - [name](url) or - [name](url): notes. |
| E006 | error | A relative link that does not resolve next to the file (local files only). |
| E007 | error | With --fetch: an http(s) link returned 400 or above, could not be reached, or redirected more than 5 times or to a non-HTTP URL. |
| W001 | warning | No blockquote summary after the title. |
| W002 | warning | No H2 sections. |
| W003 | warning | A section with no links. |
| W004 | warning | The same URL linked twice. |
| W005 | warning | Optional is not the last section. |
| W007 | warning | Duplicate section name. |
| W008 | warning | The summary is shorter than ten characters. |
| W009 | warning | Prose inside a section; sections should contain only link lines. |
| I001 | info | Section and link counts. |
| I002 | info | With --fetch: how many links were not requested because of --max-links. |
Regenerate on every change to the docs and fail the build when the committed file is stale or invalid:
llms-txt-gen generate docs/ --out llms.txt --base-url https://docs.example.com
git diff --exit-code llms.txt
llms-txt-gen check llms.txt --strictname: llms-txt
on:
push:
branches: [main]
permissions:
contents: read
jobs:
llms-txt:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: basitalisandhu/[email protected] # pin a release tag
with:
source: docs
out: llms.txt
base-url: https://docs.example.com
full: "true"
- run: git diff --exit-code llms.txt llms-full.txt || echo "llms.txt changed; commit it"Inputs: source (directory or sitemap URL), out, full, base-url, strip-ext, optional (space-separated globs), nav (path to mkdocs.yml), max-pages, timeout, check (run check --strict on the result, default true), python-version, version (PyPI version; empty installs the action's own checkout). Outputs: out, links, sections. See action.yml.
Validate llms.txt before every commit:
repos:
- repo: https://github.com/basitalisandhu/llms-txt-gen
rev: v0.4.0
hooks:
- id: llms-txt-checkReal messages and what to do about them. Every error goes to stderr and exits 2; findings in a checked file exit 1.
| Message | Cause and fix |
|---|---|
ERROR: Package 'llms-txt-gen' requires a different Python: 3.10.12 not in '>=3.11' |
The tool needs Python 3.11 or newer. Use pipx install --python python3.12 ..., uvx, or the container image. |
error: docs is not a directory or a URL |
SOURCE must be an existing folder, or a URL starting with http:// or https://. Run the command from the repository root or pass an absolute path. |
error: no Markdown or HTML files found under ... |
Only .md, .mdx, .markdown, .html and .htm files are read. Hidden folders, node_modules, build, dist and virtual environments are always skipped; check your --include and --exclude globs and any llms: false front matter. |
error: llms.txt is UTF-16 text; save it as UTF-8 and run again |
Windows PowerShell 5 writes UTF-16 with >. Use Set-Content -Encoding utf8, or regenerate the file with generate --out. |
error: llms.txt is not UTF-8 text (byte 0xe9 at offset 5); save it as UTF-8 and run again |
The file was saved in a legacy encoding such as Windows-1252. Re-save it as UTF-8 in your editor. |
error: cannot write llms.txt: Permission denied (container) |
The image runs as uid 1000. Mount a folder that user can write to, or run with --user "$(id -u):$(id -g)". |
error: cannot write out: Is a directory |
--out names a file, not a folder: use --out out/llms.txt. |
error: sitemap is not well-formed XML: ... |
The URL did not return a sitemap; often it is an HTML 404 page or a login wall. Open the URL in a browser, and try /sitemap_index.xml or the location named in robots.txt. |
error: the sitemap yielded no pages |
The sitemap parsed but listed no <url><loc>. Point at the child sitemap, or raise --max-pages if it is set very low. |
skipped https://...: HTTP Error 403: Forbidden |
A page refused the request. The run continues without it; some hosts block unknown user agents. |
E006 broken local link on a generated file |
The links are relative to the docs folder but the file was written elsewhere. Write --out inside the docs folder, or pass --base-url so the links are absolute. |
note: nav lists guide/a.md, which is not under . |
With --nav, SOURCE must be the MkDocs docs_dir (usually docs/), because nav paths are relative to it. |
error: mkdocs.yml:4: 'nav:' must be followed by a block list ... |
The nav parser reads block lists only. Rewrite nav: [a.md, b.md] as one - a.md item per line. |
What goes in llms.txt?
An H1 with the project name, a one-paragraph blockquote summary, optionally a few paragraphs of context without headings, then H2 sections each holding a list of - [page name](url): one line about the page. A section named Optional holds pages a model may skip. That is the whole convention; llms-txt-gen check enforces it and the README of llmstxt.org documents it.
Does it need network access?
Only when you pass a sitemap URL to generate or a URL to check. Directory mode and file mode never make a request, which is why they are safe in CI and in pre-commit hooks.
Can I keep the file by hand and only check it?
Yes. check is independent of generate. Many projects write the summary by hand and generate only the link lists; use --title, --summary and --body to keep the prose fixed while the lists are regenerated.
How do I tell crawlers the file exists?
Serve it at /llms.txt, and add the tag from serve-snippet to the site <head> (or the Link: header from serve-snippet --header). The tag is a discoverability hint, not part of the convention, so leaving it out does not make the file invalid.
- Read Docusaurus sidebars and the Sphinx toctree for section names and order, as
--navdoes for MkDocs. - A
diffcommand that shows which links were added or removed since the committed file.
Issues and pull requests are welcome; the starter list is in docs/good-first-issues.md. Run make check (ruff and pytest) before opening a pull request; see CONTRIBUTING.md. Security problems: see SECURITY.md.
More tools by the same author: https://github.com/basitalisandhu
- agent-threat-model: describe an agent system in YAML, get a STRIDE and OWASP Agentic threat model.
- agent-config-audit: audit AI agent configuration files for security risks.
- mcp-server-template: secure MCP server template in TypeScript and Python.
- awesome-agent-security: curated list of AI agent security tools, papers and datasets.
MIT, see LICENSE. Copyright 2026 Muhammad Basit Ali.