VeriFastScore is a fast and efficient factuality evaluation tool that jointly extracts and verifies fine-grained factual claims from long-form LLM-generated responses, conditioned on evidence collated from google search results retrieved using SERPER API.
This repository packages VeriFastScore as a pip-installable Python package with a command-line interface (verifastscore), simplifying usage and deployment.
Coming soon: Dynamic Batching to further imrpove speedups and add support for multi-GPU usage
VeriFastScore supports both CPU and GPU setups. Please follow the appropriate instructions below:
To install VeriFastScore for CPU-only usage:
pip install verifastscore
python3 -m spacy download en_core_web_sm
If you want GPU acceleration with FlashAttention, you must install compatible versions of torch and flash-attn manually before installing verifastscore.
# Install CUDA-enabled PyTorch (adjust for your CUDA version)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
# Install flash-attn (manually, requires build tools and matching CUDA)
pip install flash-attn --no-build-isolation
# Then install verifastscore
pip install verifastscore
# Finally, download the spaCy model
python3 -m spacy download en_core_web_sm
git clone https://github.com/rishanthrajendhran/verifastscore.git
cd verifastscore
pip install -e .
python3 -m spacy download en_core_web_sm
VeriFastScore retrieves external evidence using Serper.dev. You'll need to set your API key in the environment:
export SERPER_KEY_PRIVATE="your-key-here"Add to ~/.bashrc or ~/.zshrc for permanence:
echo 'export SERPER_KEY_PRIVATE="your-key-here"' >> ~/.bashrc
source ~/.bashrc
set SERPER_KEY_PRIVATE=your-key-here$env:SERPER_KEY_PRIVATE="your-key-here"Once installed, use the CLI tool:
verifastscore --input_file responses.jsonlOr with custom arguments:
verifastscore \
--input_file responses.jsonl \
--data_dir ./data \
--output_dir ./outputs \
--model_name rishanthrajendhran/VeriFastScore \
--search_res_num 10
| Argument | Type | Default | Description |
|---|---|---|---|
--input_file | str | required | Input file (.jsonl) in --data_dir. |
--data_dir | str | ./data | Directory for input files. |
--output_dir | str | ./data | Where to write outputs. |
--cache_dir | str | ./data/cache | Directory to store SERPER search cache. |
--model_name | str | rishanthrajendhran/VeriFastScore | Hugging Face model name or local path. The default value is the path to the trained VeriFastScore model on HuggingFace. Do not change this path unless you want to test your own model. |
--search_res_num | int | 10 | Evidence snippets per sentence. |
The input must be a .jsonl file with the following structure:
{
["id": "Optional_string_ID"]
"response": "The capital of France is Paris.",
[...]
}
Place the file in the --data_dir directory.
A sample input file can be found here
./outputs/evidence_*.jsonl– Response with retrieved evidence./outputs/model_output/decomposition_verification_*.jsonl– Claim-level factuality labels./outputs/time/verifastscore_time_*.pkl– Timing breakdown
Example output:
"claim_verification_result": [
{"claim": "Paris is the capital of France.", "verification_result": "supported"},
{"claim": "France is in South America.", "verification_result": "unsupported"}
]
Console output will include average VeriFastScore, timing per instance, and per stage.
To prepare the Python environment:
- Clone the repository and create the conda environment:
conda env create -f environment.yml conda activate verifastscore - Download spaCy English tokenizer:
python3 -m spacy download en_core_web_sm - Install PyTorch (choose the version that matches your CUDA setup):
pip install torch torchvision torchaudio - Install FlashAttention:
pip install flash-attn --no-build-isolation
@misc{rajendhran2025verifastscorespeedinglongformfactuality,
title={VeriFastScore: Speeding up long-form factuality evaluation},
author={Rishanth Rajendhran and Amir Zadeh and Matthew Sarte and Chuan Li and Mohit Iyyer},
year={2025},
eprint={2505.16973},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.16973},
}
This project is licensed under the Apache 2.0 License.