llms.txt is an increasingly adopted standard of navigation for agents.
- See https://llmstxt.org
- People are asking this to become well-known but there is currently little movement.
- all links in llms.txt or sitemap should be agent-friendly
This specification is still in active development, but is getting a lot of adoption.
See https://modelcontextprotocol.io for more info.
For discoverability, there is no defined standard yet, but there are several competing ideas, which you could already adopt:
Allow any MCP client to let the user authenticate. If you have protected resources in your app, a set of practices specified these listed documents allows MCP clients (and other protocols) to use your tools.
- test via https://mcp.agent-friendly.com
- see https://github.com/janwilmake/universal-mcp-oauth
- see https://github.com/janwilmake/simplerauth-provider
The OpenAPI Specification allows describing your entire server functionality in a discoverable static file. Here are some best practices to creating a great discoverable OpenAPI.
API calls are often part of a bigger sequence of operations. Besides describing this in documentation and your OpenAPI, there's also the Arazzo Specification which allows you to define these sequences in a standardized way.
Besides this, it makes sense to instruct agents on next steps in a sequence in text responses, including in non OK responses that would otherwise be a dead end. This reduces the likelihood that an agent gets stuck, just like you would normally design for when creating a website. This can be done at the REST endpoint level since it also helps programmers. However, some people may prefer this for MCP.
There isn't one dominant strategy yet to make agents able to pay for your products, so I'll create an overview here for competing standards.
The HTML Spec specifies you can refer to alternate representations of documents using <link rel="alternate" type="your-mime-type">
Example:
<link rel="alternate" type="text/markdown" href="docs.md" title="Docs" />By adding a "text/markdown", "text/plain", or "application/json" representation to your HTML documents, you can instruct programmatic access to more token-dense and more machine-readable versions of the same document.
Browser requests normally have text/html at the start of their accept header they send when requesting web-pages. Agents don't, yet often recieve the same HTML back on a specific address.
By responding with a more machine-friendly representation of the same document if the accept header prioritizes it (see how this works here) we make it easier for agents to retrieve the content they're looking for.
Does NOT serve an 'im not a robot' page
If you're blocking robots but allowing humans
Coming soon
The following is an overview of the packages in this repo, and what their purpose is.
- getassetmanifest - builds a
manifest.jsonfile compatible with cloudflare workers - llmstxt-generate - generates a llms.txt file from your
manifest.json. Useful to make a llms.txt for static websites hosted on cloudflare workers.