Annotation Showcase

Named entity recognition for adverse drug events from patient-reported experiences, based on the CADEC corpus (Karimi et al., 2015). Annotates drugs, adverse effects, symptoms, diseases, and findings from colloquial health forum posts with mapping to medical vocabularies (SNOMED-CT, MedDRA).

Biomedical NLP

AMI Meeting Multi-Tier Annotation

Multi-tier ELAN-style annotation of multi-party meeting recordings. Annotators segment speaker turns, head gestures, and focus of attention on parallel timeline tiers, then classify dialogue acts and topic segments. Based on the AMI Meeting Corpus.

video_annotationradio

Discourse Analysis

Analysis of Clinical Text: Disorder Identification and Normalization

Identify disorder mentions and their attributes in clinical discharge summaries, based on SemEval-2015 Task 14 (Elhadad et al.). Annotators mark disorder spans, body locations, severity indicators, and classify the assertion status of each disorder.

Aspect-Based Sentiment Analysis

Identification of aspect terms in review text with sentiment polarity classification for each aspect. Based on SemEval-2016 Task 5 (ABSA).

Aspect-Based Sentiment Analysis (Original ABSA)

Identify aspect terms in review text and classify their sentiment polarity, based on SemEval-2014 Task 4 (Pontiki et al.). Annotators highlight aspect terms and assign sentiment labels across restaurant and laptop review domains.

Biomedical Entity Linking (MedMentions)

Entity mention detection and UMLS concept linking for biomedical text based on MedMentions. Annotators identify biomedical entity mentions in PubMed abstracts and link them to UMLS Concept Unique Identifiers (CUIs), supporting large-scale biomedical knowledge base construction and clinical NLP.

radiospantext

Biomedical NLP

Biomedical Named Entity Recognition (JNLPBA)

Named entity recognition for biomedical text based on the JNLPBA shared task. Annotate entities including proteins, DNA, RNA, cell lines, and cell types following BioNLP community standards.

span

Biomedical

BioNLP 2011 - Gene Regulation Event Extraction

Biomedical event extraction for gene regulation, based on the BioNLP 2011 Shared Task (Kim et al., ACL Workshop 2011). Annotators identify biological entities and mark regulatory events such as gene expression, transcription, and protein catabolism in scientific abstracts.

event_annotationspan

Audio Annotation30

DISPLACE 2024 - Speaker and Language Diarization

Speaker and language diarization in multilingual conversational audio. Annotators mark speaker turn boundaries, identify speakers, and label the language of each segment in conversational environments (Kundu et al., INTERSPEECH 2024).

radiospan

Speech Processing

Sound Event Detection

Temporal sound event annotation with strong labels following DCASE Challenge protocols.

spanmultiselect

Speaker Diarization

Identify and label different speakers in audio recordings with timestamp-based segment annotation.

Speech

ToBI Prosodic Annotation

Multi-tier prosodic annotation following the Tones and Break Indices (ToBI) framework. Annotators label pitch accents, phrase accents, boundary tones, and break indices on speech utterances, producing a layered prosodic transcription aligned to the audio timeline (Silverman et al., Speech Communication 1992).

spanradiotext

Prosody

Acoustic Scene Classification

Classify audio recordings by acoustic environment following the TUT/DCASE dataset format.

radiolikert

Audio Transcription Review

Review and correct automatic speech recognition transcriptions with waveform visualization.

likertmultiselectradio+1

Audio-Visual Sentiment Analysis

Rate sentiment in speech segments following CMU-MOSI and CMU-MOSEI multimodal annotation protocols.

likertradio

AudioHate - Audio Hate Speech Detection

Audio hate speech detection with explanations. Annotators classify audio clips for hate speech presence, identify target groups, and note acoustic indicators such as tone, emphasis, and prosody (Guo et al., SIGDIAL 2024).

radiomultiselect

Speech Processing

Image Annotation41

Breakfast Actions Segmentation

Fine-grained temporal action segmentation of breakfast preparation activities. Annotators label sequences of cooking actions like 'take cup', 'pour milk', 'stir'.

textvideo_annotation

EPIC-KITCHENS Egocentric Action Annotation

radiotextvideo_annotation

Annotate fine-grained actions in egocentric kitchen videos with verb-noun pairs. Identify cooking actions from a first-person perspective.

FineGym Dataset: Fine-Grained Gymnastics Action Recognition

FineGym is a hierarchical video dataset (CVPR 2020) for fine-grained gymnastics action recognition: events, sets, and 99-530 element classes annotated with millisecond-level temporal boundaries. See the label taxonomy and a runnable Potato config to annotate or extend it.

radiovideo_annotation

FineSports: Fine-Grained Basketball Action Dataset

multiselectvideo_annotation

FineSports is a CVPR 2024 dataset of 10,000 NBA videos labeled with 52 fine-grained action types under 12 ball-handler categories. This Potato config reproduces its bounding-box and hierarchical action labeling task.

Harmony4D Human Interaction Tracking

multiselectvideo_annotation

Close-range human interaction tracking and annotation. Annotators track multiple people during close physical interactions (dancing, martial arts, collaborative tasks) with bounding boxes and interaction labels.

How2Sign Sign Language Multi-Tier Annotation

video_annotationradiotext

Multi-tier ELAN-style annotation of continuous American Sign Language videos. Annotators segment sign glosses, mark mouthing patterns, classify sign handedness, and provide English translations aligned to video timelines. Based on the How2Sign large-scale multimodal ASL dataset.

Sign Language

MSAD Multi-Scenario Anomaly Detection

Video anomaly detection across multiple scenarios. Annotators watch surveillance-style videos and mark temporal segments containing anomalous events, classifying the anomaly type.

video_annotationradio

ADE20K: Scene Parsing and Semantic Segmentation Dataset

ADE20K is a scene parsing dataset from MIT CSAIL (Zhou et al., CVPR 2017) with dense pixel labels for objects, stuff, and parts. This Potato config reproduces its 150-class segmentation annotation workflow.

multiselectradio

Video Annotation28

ActivityNet Captions: Dense Video Captioning Dataset

ActivityNet Captions pairs 20k untrimmed videos with 100k temporally localized sentence descriptions for dense-captioning research. This Potato config reproduces the segment-and-describe workflow.

video_annotationtext

ActivityNet: Temporal Action Localization Benchmark

ActivityNet is a large-scale video benchmark for human activity understanding, with untrimmed YouTube videos labeled across 200 daily activities. This Potato config reproduces its temporal localization annotation: marking start and end times of activity instances.

video_annotation

AVA: Atomic Visual Actions Dataset

multiselectvideo_annotation

AVA spatio-temporally localizes 80 atomic actions on people in movie clips, labeled at 1 Hz keyframes. This Potato config reproduces the box-and-action labeling task for video clips.

Charades Indoor Activity Segmentation

Multi-label temporal activity segmentation in indoor home videos. Annotators identify action instances using compositional verb-object labels (e.g., 'opening door', 'sitting on chair') with precise temporal boundaries.

video_annotation

Charades-STA Temporal Grounding

Ground natural language descriptions to video segments. Given a sentence describing an action, identify the exact temporal boundaries where that action occurs.

radiovideo_annotation

Clinical TempEval - Temporal Information Extraction from Clinical Notes

Extraction of temporal information from clinical text, identifying time expressions, event mentions, and their temporal relations. Based on SemEval-2016 Task 12 (Clinical TempEval).

DiDeMo Moment Retrieval

Localizing natural language descriptions to specific video moments. Given a text query, annotators identify the corresponding temporal segment in the video.

radiovideo_annotation

Ego4D: Egocentric Video Episodic Memory Annotation

video_annotationtextradio

Annotate egocentric (first-person) video for episodic memory tasks including activity segmentation, hand state tracking, natural language query generation, and scene narration. Supports temporal segment annotation with multiple label tiers for the Ego4D benchmark.

Egocentric Vision

Comparison Tasks3

Best-Worst Scaling

MaxDiff annotation where annotators select the best and worst items from a set for relative comparison.

best-worst

Ranking Task

Drag-and-drop ranking interface to order items from best to worst.

ranking

SWE-Bench+ Patch Screening

radiomultiselectpairwise+1

Screen and compare model-generated patches against gold patches for SWE-Bench+ instances. Annotators evaluate correctness, identify specific issues, and compare model vs. gold solutions side-by-side.

Software Engineering

Preference Learning29

Interpretable Semantic Textual Similarity

Fine-grained semantic similarity assessment between sentence pairs with span alignment, combining chunk-level annotation with graded similarity scoring. Based on SemEval-2016 Task 2.

spanlikert

SaGA Gesture-Speech Alignment Multi-Tier Annotation

video_annotationradiotext

Multi-tier ELAN-style annotation of co-speech gestures and their alignment with spoken language. Annotators segment gesture phases and types on parallel timeline tiers, classify handedness and spatial reference frames, and transcribe concurrent speech. Based on the SaGA corpus.

Gesture Studies

AlpacaEval: Instruction-Following Preference Evaluation

Pairwise preference annotation for instruction-following language models. Annotators compare two model responses side by side, select their preferred response, indicate preference strength, and rate individual response quality across diverse instruction categories.

pairwiseradiotext

AI Evaluation

AlpacaFarm Preference Simulation

Natural Language Processing

Simulate human preferences for instruction-following responses. Create preference data for efficient RLHF research and LLM evaluation.

likertradio

Arena Hard Auto - LLM Pairwise Evaluation

Pairwise evaluation of LLM responses on challenging prompts from the Arena Hard benchmark (Li et al., arXiv 2024). Annotators compare two responses on a continuous scale and rate question difficulty.

pairwiselikert

BeaverTails Safety Preference

Annotate AI responses for safety across multiple harm categories. Identify unsafe content and rate response quality for building safer AI systems.

multiselectradio

AI Safety

Chatbot Arena: Pairwise LLM Preference Evaluation

Chatbot Arena collects human pairwise preference votes between anonymous LLM responses to rank models with a Bradley-Terry leaderboard. This Potato config reproduces the pairwise and best-worst scaling judgment task.

bwspairwise

CodePRM Code Process Reward

Process reward annotation for step-by-step code generation with execution feedback. Annotators verify each incremental code generation step against problem requirements, check syntactic validity, and provide feedback informed by test execution results.

radiotext

Code Generation

Surveys57

ESA: Error Span Annotation for Machine Translation

Error span annotation for machine translation output. Annotators identify error spans in translations, classify error types (accuracy, fluency, terminology, style), and rate severity.

spanradiolikert

LongEval: Faithfulness Evaluation for Long-Form Summarization

LongEval is the EACL 2023 protocol for human evaluation of faithfulness in long-form summaries (Krishna et al.). This Potato config reproduces its fine-grained, clause-level faithfulness judgments against source documents.

spanradiolikert

News Headline Emotion Roles (GoodNewsEveryone)

Annotate emotions in news headlines with semantic roles. Based on Bostan et al., LREC 2020. Identify emotion, experiencer, cause, target, and textual cue.

likertradiospan

NLI with Explanations (e-SNLI)

Natural language inference with human explanations. Based on e-SNLI (Camburu et al., NeurIPS 2018). Classify entailment/contradiction/neutral and provide natural language justifications.

likertradiospan

RT-2 - Robotic Action Annotation

Robotic manipulation task evaluation and action segmentation based on RT-2 (Brohan et al., CoRL 2023). Annotators evaluate task success, describe actions, rate execution quality, and segment video into action phases.

radiotextlikert+1

Robotics

SciFact: Scientific Claim Verification Dataset

SciFact pairs 1,409 expert-written scientific claims with 5,183 research abstracts labeled SUPPORTS, REFUTES, or NOINFO plus rationale sentences. The Potato config below reproduces the annotation task.

likertradiospan

AgentBoard Progress Scoring

Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress, categorizing the agent environment, and noting partial progress observations.

multiselectlikertradio+1

Agentic AI

AnnoMI Counselling Dialogue Annotation

Annotation of motivational interviewing counselling dialogues based on the AnnoMI dataset. Annotators label therapist and client utterances for MI techniques (open questions, reflections, affirmations) and client change talk (sustain talk, change talk), with quality ratings for therapeutic interactions.

radiomultiselectlikert

Evaluation Tasks34

Code Review Annotation (CodeReviewer)

Annotation of code review activities based on the CodeReviewer benchmark. Annotators identify issues in code diffs, classify defect types, assign severity levels, make review decisions, and provide natural language review comments, supporting research in automated code review and software engineering.

spanradiotext

Software Engineering

EA-MT - Entity-Aware Machine Translation

Entity-aware machine translation evaluation requiring annotators to identify entity spans, classify translation errors, and provide corrected translations. Based on SemEval-2025 Task 2.

spanradiotext

FAVA: Fine-grained Hallucination Annotations for Faithful Generation

Fine-grained hallucination span annotation. Annotators identify hallucinated spans in LLM output and classify hallucination types (entity error, relation error, contradicted, invented, subjective, unverifiable). Based on the FAVA framework for fine-grained faithfulness evaluation.

MathDial: Math Tutoring Dialogue Dataset

MathDial is a dataset of 2,861 one-to-one math tutoring dialogues grounded in GSM8K word problems, annotated with a four-category teacher-move taxonomy. This Potato config reproduces those teacher-move labels.

radiomultiselectspan

#HashtagWars - Learning a Sense of Humor

Humor ranking of tweets submitted to Comedy Central's @midnight #HashtagWars, classifying comedic quality. Based on SemEval-2017 Task 6.

radio

AgentRewardBench Trajectory Scoring

Evaluate web agent trajectories by rating step-level quality across multiple dimensions, judging overall success, and identifying where automatic evaluators disagree with human judgment.

multirateradiotext

Agentic AI

AndroidWorld: Mobile Agent Task Evaluation

Evaluation of autonomous agents performing tasks in dynamic Android environments. Annotators assess task completion, identify interaction types and touch gestures, evaluate UI understanding, and describe mobile-specific issues across diverse Android apps.

radiomultiselecttext

Mobile Agents

ArgSciChat Scientific Argumentation Dialogue

Annotation of argumentative dialogues about scientific papers based on the ArgSciChat dataset. Annotators label dialogue turns for argument components (claim, evidence, rebuttal) and assess argument quality dimensions such as clarity, relevance, and persuasiveness.

multiselectradio