Potato is a free, self-hosted annotation platform for NLP, Agentic, GenAI, and qualitative research. Annotate text, audio, video, images, documents, agent traces, and more — or run a full qualitative data analysis (QDA) workflow with a living codebook, memos, and cases. Configured entirely through YAML. No coding required.
Try the live demo on HuggingFace Spaces — no installation needed. More at www.potatoannotator.com.
pip install potato-annotation
# The examples/ folder ships with the source repo (see "run from source" below).
# After a PyPI install, clone the repo for the examples, or point `potato start`
# at your own config (see docs/quick-start.md).
potato start examples/classification/single-choice/config.yaml -p 8000Or run from source (recommended to get the examples/):
git clone https://github.com/davidjurgens/potato.git
cd potato && pip install -r requirements.txt
python potato/flask_server.py start examples/classification/single-choice/config.yaml -p 8000Open http://localhost:8000 and start annotating. Browse the examples/ directory for ready-to-use templates.
Potato handles the full spectrum of annotation tasks — from traditional NLP labeling to evaluating the latest AI agent systems, to interpretive qualitative analysis.
The tables below are a representative sample, not a complete list. Schemes and data types compose freely, custom layouts and raw HTML let you build interfaces beyond these, and new schema types can be added. If you don't see your task here, it's likely still possible.
| Modality | Capabilities |
|---|---|
| Text | Classification, span labeling, entity linking, coreference, pairwise comparison (docs) |
| Agent Traces | Step-by-step evaluation of LLM agents, tool calls, ReAct chains, and multi-agent systems (docs) |
| Web Agents | Screenshot-based review with SVG click/scroll overlays, or live browsing with automatic trace recording (docs) |
| RAG Pipelines | Retrieval relevance, answer faithfulness, citation accuracy, hallucination detection |
| Audio | Waveform visualization, segment labeling, ELAN-style tiered annotation (docs) |
| Video | Frame-by-frame labeling, temporal segments, playback sync (docs) |
| Images | Bounding boxes, polygons, landmarks, classification (docs) |
| Dialogue | Turn-level annotation, conversation trees, interactive chat evaluation |
| Documents | PDF, Word, Markdown, code, and spreadsheets with coordinate mapping (docs) |
| Scheme | Use Case |
|---|---|
| Radio / Checkbox / Likert | Classification, multi-label, rating scales |
| Span annotation | NER, highlighting, hallucination marking |
| Pairwise comparison | A/B testing, best-worst scaling |
| Per-step ratings | Evaluate individual agent actions or dialogue turns |
| Free text | Open-ended responses with validation |
| Triage | Rapid accept/reject/skip curation (docs) |
| Conditional logic | Adaptive forms that respond to prior answers (docs) |
Potato provides purpose-built tooling for evaluating AI agents at every level of granularity.
Import traces from any major agent framework with the built-in converter:
python -m potato.trace_converter --input traces.json --input-format openai --output data.jsonlSupported formats: OpenAI, Anthropic/Claude, ReAct, LangChain, LangFuse, WebArena, SWE-bench, OpenTelemetry, CrewAI/AutoGen/LangGraph, MCP, Aider, Claude Code, ATIF, SWE-Agent, and Web Agent. Auto-detection is available with --auto-detect.
| Level | What You Annotate | Example |
|---|---|---|
| Trajectory | Overall task success, efficiency, safety | "Did the agent complete the task?" |
| Step | Individual action correctness, reasoning quality | Per-turn Likert ratings on each agent step |
| Span | Specific text segments within agent output | Highlight hallucinated claims, factual errors |
| Comparison | Side-by-side A/B agent evaluation | "Which agent performed better?" |
An interactive viewer for GUI agent traces — navigate step-by-step through screenshots with SVG overlays showing clicks, bounding boxes, mouse paths, and scroll actions. Annotators rate each step with inline controls while a filmstrip bar provides quick navigation.
| Example | What It Evaluates |
|---|---|
| agent-trace-evaluation | Text agent traces with MAST error taxonomy + hallucination spans |
| visual-agent-evaluation | GUI agents with screenshot grounding accuracy |
| agent-comparison | Side-by-side A/B agent comparison |
| rag-evaluation | RAG retrieval relevance and citation accuracy |
| openai-evaluation | OpenAI Chat API traces with tool calls |
| anthropic-evaluation | Claude messages with tool_use blocks |
| swebench-evaluation | Coding agents with patch correctness ratings |
| multi-agent-evaluation | Multi-agent coordination (CrewAI, AutoGen, LangGraph) |
| web-agent-review | Pre-recorded web traces with step-by-step overlay viewer |
| web-agent-creation | Live web browsing with automatic trace recording |
Potato isn't only for label-and-aggregate tasks — it also supports interpretive qualitative research, the kind of work done in tools like NVivo, ATLAS.ti, or MAXQDA, fully self-hosted and free.
| Capability | Description |
|---|---|
| Living codebook | The codebook is an evolving markdown document of rules, definitions, examples, and rationales — not just a label list. Edit it in a full-page document view or inline while coding, with versioning, diff, and restore; semantic edits can re-flag affected excerpts for review (docs) |
| In-vivo coding | Create codes directly from a highlighted passage, in the participant's own words (example) |
| Memos | Attach analytic notes to excerpts, codes, or the whole project as your interpretation develops (docs) |
| Cases | Group instances into units of analysis — participants, interviews, documents, sites — for case-based comparison (docs) |
| Search | Full-text search across your corpus and annotations to find, revisit, and code recurring patterns (docs) |
| Codebook distillation | Turn the human-authored codebook into an LLM prompt for AI-assisted coding |
Enable it with qda_mode, which sensibly cascades these features on; see the QDA Mode guide and the runnable qda-mode-example.
Integrate any LLM provider to pre-annotate instances and suggest labels. Annotators review and correct — dramatically faster than labeling from scratch.
Supported backends: OpenAI, Anthropic, Ollama, vLLM, Gemini, HuggingFace, OpenRouter
Potato reorders your annotation queue based on model uncertainty so annotators label the most informative instances first. Supports uncertainty sampling, BADGE, BALD, diversity, and hybrid strategies (docs).
A human-LLM collaborative workflow where the system learns from annotator feedback and progressively transitions to autonomous LLM labeling as agreement improves (docs).
An LLM-powered sidebar where annotators can ask questions about difficult instances. The AI provides guidance informed by your task description and annotation guidelines — helping annotators think through decisions without auto-labeling (docs).
| Feature | Description |
|---|---|
| Attention checks | Automatically inserted known-answer items to verify engagement |
| Gold standards | Track annotator accuracy against expert labels |
| Inter-annotator agreement | Krippendorff's alpha (general) and Cohen's kappa (step-level agent evaluation) |
| Training phase | Practice annotations with feedback before the real task |
| Behavioral tracking | Timing, click patterns, and annotation change history |
| Workflow | Description |
|---|---|
| Multi-annotator | Multiple annotators per item with overlap control and agreement metrics |
| Adjudication | Expert review of annotator disagreements to produce gold labels (docs) |
| Solo mode | Human-LLM collaboration with progressive automation (docs) |
| Crowdsourcing | Prolific and MTurk integration with platform-specific auth (docs) |
| Triage | Rapid accept/reject/skip for data curation (docs) |
Close the loop from production traces to graded, regression-gated evaluation:
| Capability | Description |
|---|---|
| Capture | Instrument any agent with the @traceable tracing SDK, or POST traces to the ingestion webhook |
| Automate | Rules (filter → sample → actions) route incoming traces to queues, datasets, evaluators, or webhooks |
| Curate | Versioned datasets & experiments + semantic search/slices to find what to review |
| Evaluate | Programmatic evaluators (trajectory match, tool-use, LLM-judge, heuristics) + a side-by-side model arena |
| Gate | Run evals in pytest and fail CI on score-threshold regressions |
| Calibrate | LLM-judge ↔ human alignment with auto-calibration from human corrections; judges categorical, span, and free-text outputs |
Potato supports multiple authentication methods, from passwordless quick-start to enterprise SSO:
| Method | Use Case |
|---|---|
| In-memory | Local development, quick studies |
| Password + file persistence | Team annotation with shared credential files (docs) |
| Database | Production deployments with SQLite or PostgreSQL (docs) |
| OAuth / SSO | Google, GitHub, or institutional OIDC login (docs) |
| Clerk | Managed authentication via Clerk.com (docs) |
| Passwordless | Low-stakes tasks where ease of access matters (docs) |
Passwords are hashed with per-user PBKDF2-SHA256 salts. Admins can reset passwords via CLI (potato reset-password) or REST API. Self-service token-based reset is also available.
Ready-to-use templates organized by type in examples/:
| Category | Examples |
|---|---|
| Classification | Radio, checkbox, Likert, slider, pairwise comparison |
| Span | NER, span linking, coreference, entity linking |
| Agent Traces | LLM agents, web agents, RAG, multi-agent, code agents |
| Audio | Waveform annotation, classification, ELAN-style tiered |
| Video | Frame-level labeling, temporal segments |
| Image | Bounding boxes, PDF/document annotation |
| Advanced | Solo mode, adjudication, quality control, conditional logic |
| QDA | Qualitative analysis: living codebook, in-vivo coding, memos, cases |
| AI-Assisted | LLM suggestions, Ollama integration |
| Custom Layouts | Content moderation, dialogue QA, medical review |
Try Potato in your browser — no installation. A growing catalog of one-click demo Spaces covers classification, span/NER, agent-trace evaluation, multimodal, QDA, and more:
- 🤗 Flagship demo — agent trace evaluation
- 📋 Full demo catalog & collection — every annotation type as a live Space
- 🚀 Deploy your own —
build_space.py+deploy_space.pyfrom a single manifest
The Potato Showcase contains annotation projects from published research — sentiment analysis, dialogue evaluation, summarization, and more.
Potato has two complementary doc sites: potatoannotator.com/docs for guides, tutorials, and higher-level walkthroughs, and Read the Docs for the complete, version-matched technical reference (every config option, the full HTTP API, and internals). The links below point to the guide pages.
# Run tests
pytest tests/ -v
# By category
pytest tests/unit/ -v # Unit tests (fast)
pytest tests/server/ -v # Integration tests
pytest tests/selenium/ -v # Browser tests
# With coverage
pytest --cov=potato --cov-report=html- Issues: GitHub Issues
- Questions: [email protected]
- Docs: potatoannotator.readthedocs.io
Potato is free software, licensed under the GNU General Public License v3.0 or later (GPLv3+). You are free to use, study, modify, and redistribute it — including for commercial purposes — provided that any distributed derivative works are also licensed under the GPLv3+ and made available with their source code. See the LICENSE file for the full terms.
If you use Potato in your research, please cite the Potato 2.0 paper (ACL 2026 System Demonstrations):
@inproceedings{jurgens-etal-2026-potato,
title = "Potato 2.0: A Comprehensive Annotation Platform with {AI}-in-the-Loop Support",
author = "Jurgens, David and
Chen, Michael and
Iyer, Lina",
editor = "Durrett, Greg and
Jian, Ping",
booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.acl-demo.37/",
pages = "374--386",
ISBN = "979-8-89176-392-0",
}To reference the original Potato release, cite the Potato 1.0 paper (EMNLP 2022 System Demonstrations):
@inproceedings{pei-etal-2022-potato,
title = "{POTATO}: The Portable Text Annotation Tool",
author = "Pei, Jiaxin and
Ananthasubramaniam, Aparna and
Wang, Xingyao and
Zhou, Naitian and
Dedeloudis, Apostolos and
Sargent, Jackson and
Jurgens, David",
editor = "Che, Wanxiang and
Shutova, Ekaterina",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = dec,
year = "2022",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-demos.33/",
doi = "10.18653/v1/2022.emnlp-demos.33",
pages = "327--337",
}