Better rankings from fewer votes
compere is a Python library and FastAPI service for pairwise-comparison ranking. A multi-armed bandit picks which pair to compare next; Elo turns the verdicts into a leaderboard you can actually read.
pip install compere | # | entity | elo | n |
|---|---|---|---|
| 1 | variant-b | 1642 | 31 |
| 2 | variant-d | 1578 | 28 |
| 3 | variant-a | 1503 | 26 |
| 4 | variant-c | 1441 | 24 |
| 5 | variant-e | 1336 | 23 |
✓ 5 entities ranked from 132 pairwise votes · UCB-selected
What is compere?
compere is a pairwise-comparison ranking engine written in Python. You show people (or a model) two things and ask “which is better?”. compere decides which pair to ask about next with a UCB1 multi-armed bandit, then converts the verdicts into Elo ratings — a single, sortable leaderboard. Run it as a FastAPI service or import it as a library.
Register entities
Add the things you want to rank — model outputs, headlines, designs, catalog items. Anything you can express as an entity with an id.
Vote on the pairs it picks
Ask /mab/next_comparison for the
next matchup. UCB1 targets uncertain pairs so you spend votes where they change the ranking.
Read the leaderboard
Every verdict updates Elo ratings (K = 32, initial 1500). GET /ratings
returns entities ordered by rating — a plain function of the votes.
The problems compere solves
Ranking from human judgement is deceptively hard. compere targets the four places it usually goes wrong.
Rating scales are noisy
The problem
Ask people to score items 1–5 and you get anchoring, drift, and different internal scales per rater. The numbers look precise but rank things badly.
How compere helps
Pairwise “A or B?” is a far easier judgement. compere aggregates those binary verdicts into a single Elo ranking that reflects real preference, not scale bias.
Exhaustive comparison is quadratic
The problem
Comparing every pair is n(n-1)/2 judgements. For 50 items that is 1,225 comparisons — most of them between an obvious winner and an obvious loser.
How compere helps
The UCB1 bandit spends votes on the pairs whose outcome is genuinely uncertain and skips the settled ones, so you converge on a good ranking sub-quadratically.
Elo alone leaves out the hard part
The problem
A 30-line Elo gist gives you the rating update. It says nothing about which pair to ask about next — the actual bottleneck in a comparison study.
How compere helps
compere pairs UCB selection with Elo rating in one package, plus persistence, an HTTP API, and a leaderboard. Selection and rating ship together.
You can’t tell when to stop
The problem
Collect too few votes and the ranking is unstable; too many and you burn annotator time. Most tools give you no signal either way.
How compere helps
Because ratings are a transparent function of votes, you can watch the leaderboard stabilise and apply an explicit stopping rule instead of guessing.
Install, run, vote
Same engine, two entry points. Run the FastAPI service for a frontend to hit, or import the modules directly for offline studies and notebooks.
# 1. run the service (SQLite by default)
compere --port 8090
# 2. register the things you want to rank
curl -X POST localhost:8090/entities/ \
-H "Content-Type: application/json" \
-d '{"name": "headline-a"}'
# 3. ask UCB which pair to compare next
curl localhost:8090/mab/next_comparison
{"entity1_id": 3, "entity2_id": 7}
# 4. record the verdict
curl -X POST localhost:8090/comparisons/ \
-d '{"entity1_id":3,"entity2_id":7,"selected_entity_id":3}'
# 5. read the leaderboard
curl localhost:8090/ratings# no server needed — use it as a library
from compere.modules import entity, comparison, rating
a = entity.create_entity(name="design-a")
b = entity.create_entity(name="design-b")
# record a pairwise verdict
comparison.create_comparison(
entity1_id=a.id,
entity2_id=b.id,
selected_entity_id=a.id,
)
# Elo ratings, sorted high to low
for r in rating.get_ratings():
print(r.name, r.rating)
# design-a 1516.0
# design-b 1484.0
Interactive OpenAPI docs are served at /docs.
The full HTTP surface is eleven endpoints.
Everything a comparison study needs
Selection, rating, storage, and an HTTP surface — as one cohesive package, not a pile of scripts.
Pair selection
Deciding which pair to ask about next — the part most Elo libraries leave to you.
UCB1 pair selection
A multi-armed bandit picks the next pair to compare, concentrating votes on uncertain matchups instead of asking about obvious ones. Exploration constant c = 1.414 by default.
Learn more →Similarity pairing (alternative)
A second strategy pairs similar entities instead of UCB-optimal ones, exposed at /comparisons/next. Swap the selection policy without touching the rating layer.
Learn more →Rating & ranking
Turning binary verdicts into an interpretable, sortable leaderboard.
Elo rating updates
The same Elo math you know from chess. K = 32 and initial rating 1500 by default, both configurable. Nothing fancier is claimed — you can explain any rating change in two sentences.
Learn more →Sortable leaderboard
GET /ratings returns entities ordered by Elo. The board is a pure function of the votes it received — replayable and auditable, not a black box.
Learn more →Service & library
Run it as a FastAPI server or import it as a plain Python library.
FastAPI HTTP service
Eleven endpoints for entities, comparisons, pair selection, ratings, and health. Interactive OpenAPI docs are served at /docs the moment you run compere.
Learn more →Use it as a library
Skip the server entirely. Import create_entity, create_comparison, and get_ratings from compere.modules for offline studies and notebooks.
Learn more →Persistence & operations
Storage, auth, and deployment knobs — sensible defaults, off unless you need them.
SQLite or PostgreSQL
SQLAlchemy persistence: SQLite by default (sqlite:///./compere.db), PostgreSQL via DATABASE_URL when you outgrow it. Same code on either.
Learn more →Auth, CORS & rate limits
Optional JWT auth (AUTH_ENABLED), configurable CORS origins, and request rate limiting — all off by default so the happy path stays simple.
Learn more →Docker-ready
A Dockerfile and docker-compose.yml ship in the repository. Container handles entities, comparisons, pair selection, and ratings as one cohesive service.
Learn more →Fits your stack
One engine, several ways in. Ship a service, embed a library, or spin up a container.
From the blog
Field notes on the parts of pairwise ranking that bite in production.
Stopping rules: when have you compared enough?
There is no universal answer, but there are three honest tests. We walk through each, with the queries you can run against the compere API.
Read post →Reading the Elo output without lying
Compere ships Elo, not Bradley-Terry. The numbers it produces are easy to misread. A field guide to what an Elo gap actually means.
Read post →Why 50 pairwise votes beat 500 ratings
Rating scales drift, anchor, and lie. Pairwise comparisons survive all three. Here is the math we use, and where it actually breaks down.
Read post →Explore the docs
Every part of compere, one click away — from the algorithms to the use cases and honest comparisons.
Features
UCB selection, Elo ratings, the leaderboard, and the full 11-endpoint HTTP API.
Learn more →How it works
The UCB1 and Elo formulas, their defaults, and the request flow end to end.
Learn more →Quickstart
Install, run the server, register entities, vote, and read the ratings back.
Learn more →Use cases
RLHF preference data, eval leaderboards, A/B content ranking, and taste-graph catalogs.
Learn more →Comparisons
Honest write-ups of compere versus plain Elo libraries and AHP-style ranking SaaS.
Learn more →FAQ
Straight answers on algorithms, install, databases, and whether compere fits.
Learn more →Glossary
Plain-language definitions of every ranking term compere uses.
Learn more →Blog
Field notes on the parts of pairwise ranking that bite in production.
Learn more →About
What compere is, what it is not, and who builds it at Skelf-Research.
Learn more →Start ranking with fewer votes
compere is open source (MIT). Install it, register your entities, and let UCB decide what to ask next.
Have a ranking problem in mind?
Tell us what you're comparing and how many votes you can realistically collect.