This repo hosts Human Detectors (People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text), a dataset of expert annotations of human-written and AI-generated articles. Expert annotations include a decision (Human-written or AI-generated), confidence score, and explanation. We also include the output of many automatic detectors.
Authors: Jenna Russell, Marzena Karpinska, Mohit Iyyer
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.
- Clone the repository:
git clone https://github.com/yourusername/human_detectors.git
cd human_detectors- Install dependencies:
pip install -r requirements.txtThe detect.py script can be used to detect AI-generated text. It supports various input formats and can process multiple texts in batch.
python detect.py [options]Options:
--text TEXT: Direct text input to analyze--file FILE: Input file containing text(s) to analyze (supports CSV, JSON, Excel, text files)--model MODEL: Model to use for detection (default: gpt-4o-2024-08-06)--key_file KEY_FILE: File containing API key--api_key API_KEY: Direct API key input--no_explain: Disable explanation generation--no_guide: Disable detection guide--output OUTPUT: Output file path for results (CSV)
Examples:
# Detect a single text
python detect.py --text "Your text here" --model gpt-4o-2024-08-06 --key_file key.txt
# Process multiple texts from CSV
python detect.py --file texts.csv --output results.csv --key_file key.txt
# Process without explanations
python detect.py --file texts.json --no_explain --key_file key.txtThe evade.py script can be used to test evasion techniques on AI-generated text. It supports batch processing and can test detection on the evaded text.
python evade.py [options]Options:
--text TEXT: Direct text input to evade detection--file FILE: Input file containing text(s) to evade (supports CSV, JSON, Excel, text files)--model MODEL: Model to use for evasion (default: gpt-4o-2024-08-06)--key_file KEY_FILE: File containing API key--api_key API_KEY: Direct API key input--publication PUBLICATION: Target publication style (required)--examples EXAMPLES: File containing example texts--test_detection: Test detection after evasion--output OUTPUT: Output file path for results (CSV, default: ./evaded_results.csv)
Examples:
# Evade detection for a single text
python evade.py --text "Your text here" --publication "New York Times" --key_file key.txt
# Process multiple texts from CSV
python evade.py --file texts.csv --publication "The Guardian" --test_detection --key_file key.txt
# Process with example texts
python evade.py --file texts.json --publication "Reuters" --examples examples.txt --output results.csv --key_file key.txtAll prompts used to generate the articles (including paraphrasing and humanization) can be found in the prompts folder.
Each expert annotator annotated a total of 300 articles (150 human-written and 150 machine-generated). Annotation data is stored in a JSON file, formatted as follows:
* `generation_model`: (str) - the name of the model which generated the article
* `prompt_id`: (int) - the unique prompt id
* `title`: (str) - original title of the article
* `sub-title`: (str) - original subtitle of the article
* `author`: (str) - author of the original human article
* `source`: (str) - source of the original human article
* `issue`: (str) - publication date of the original human article
* `section`: (str) - thematic section of the original human article
* `link`: (str) - link to the original article
* `article`: (str) - the article used for annotation (human or model-generated depending on the value in `generation_model`)
* `id`: (int) - unique id
* `ground_truth`: (str) - gold label (either "Human-written" or "Machine-generated")
* `pangram`: (str) - detailed output of the pangram detector
* `pangram_humanizers`: (str) - detailed output of the pangram humanizers detector
* `gptzero`: (str) - detailed output of the gptzero detector
* `e5_lora`: (int) - output of the e5_lora detector
* `RADAR`: (int) - output of the radar detector
* `binoculars_lowfpr`: (str) - output of the binoculars detector set to prioritize low fpr
* `bincoulars_accuracy`: (str) - output of the binoculars detector set to prioritize accuracy
* `annotator_1`: (dict) - annotations done by annotator 1
* `guess`: (str) - label assigned by annotator 1
* `confidence`: (int) - self-assessed confidence of annotator 1 (5-point scale)
* `comment`: (str) - justification for their label submitted by annotator 1
* `annotator_2`: (dict) - annotations done by annotator 2
* `guess`: (str) - label assigned by annotator 2
* `confidence`: (int) - self-assessed confidence of annotator 2 (5-point scale)
* `comment`: (str) - justification for their label submitted by annotator 2
* `annotator_3`: (dict) - annotations done by annotator 3
* `guess`: (str) - label assigned by annotator 3
* `confidence`: (int) - self-assessed confidence of annotator 3 (5-point scale)
* `comment`: (str) - justification for their label submitted by annotator 3
* `annotator_4`: (dict) - annotations done by annotator 4
* `guess`: (str) - label assigned by annotator 4
* `confidence`: (int) - self-assessed confidence of annotator 4 (5-point scale)
* `comment`: (str) - justification for their label submitted by annotator 4
* `annotator_5`: (dict) - annotations done by annotator 5
* `guess`: (str) - label assigned by annotator 5
* `confidence`: (int) - self-assessed confidence of annotator 5 (5-point scale)
* `comment`: (str) - justification for their label submitted by annotator 5
* `expert_majority_vote`: (str) - result of the majority vote (at least 3 out of 5 agree)Example entry
{"0":
{
"generation_model":"gpt-4o",
"prompt_id":1,
"title":"Louisiana schools won't display Ten Commandments before November as lawsuit plays out",
"sub-title":"Louisiana has agreed to delay implementing a requirement that the Ten Commandments be placed in all of the state\u2019s public school classrooms, at least until November.",
"author":"Kevin McGill",
"source":"Associated Press",
"issue":"7/19/24",
"section":"Education",
"link":"https://apnews.com/search?q=Louisiana+schools+won%E2%80%99t+display+Ten+Commandments+before+November+as+lawsuit+plays+out#nt=navsearch",
"article":"Louisiana won't take official steps to implement a law requiring the Ten ...",
"id":1,
"ground_truth":"Human-written",
"pangram":"pangram_output",
"pangram_humanizers":"pangram_humanizers_output",
"gptzero":"gptzero_output",
"e5_lora":0.0420721397,
"RADAR":0.8631911874,
"binoculars_lowfpr":"Most likely human-generated",
"binoculars_accuracy":"Most likely human-generated",
"annotator_1":
{
"guess":"Human-Generated",
"confidence":4,
"comment":"The article overall is very factual and looks quite well-researched. It reads like a standard news story..."
},
"annotator_2":
{
"guess":"Human-Generated",
"confidence":4,
"comment":"I don't see any of the typical words used by AI. Also, the sentences are longer and more complex than..."
},
"annotator_3":
{
"guess":"Human-Generated",
"confidence":4,
"comment":"Some of the phrasing sounds slightly awkward (highlighted), and there're places where the punctuation is off."},
"annotator_4":
{
"guess":"Human-Generated",
"confidence":5,
"comment":"The article appears human-written. While many of the sentences are long, they're packed with information..."},
"annotator_5":
{
"guess":"Human-Generated",
"confidence":5,
"comment":"Although the author tries to report only facts, the final sentence..."
},
"expert_majority_vote": "Human-written"
},
}If you use this dataset, please cite it as follows:
@misc{russell2025peoplefrequentlyusechatgpt,
title={People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text},
author={Jenna Russell and Marzena Karpinska and Mohit Iyyer},
year={2025},
eprint={2501.15654},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.15654},
}