Cicada

Your launch rests on a number

Cicada is human evaluation for AI teams, where every score opens down to the person who gave it. Scroll, and we’ll open one.

Start a projectSkip to the re-count

A thousand people say your new model is better.

You ship tomorrow on that number. Can you say who made a single one of those calls?

54–46

Model B, the new oneModel A, the current one

Every point is one person’s judgement

The verdict reached you. Not one of the people who gave it did.

So open one.

One judgement, opened

Who made it

A level 4 rater who passed this project’s exam, with 37 of their last 40 gold answers right.

How long they looked

Six minutes on two answers to a production failure. Long enough to read the code in both.

Why they decided

A reason for every score, in their own words. B looks like a fix and quietly loses data; they caught it.

One judgementExample evaluation

Prompt

Our nightly ETL job started failing with KeyError: 'customer_id' after the vendor updated their API. How do I fix it without losing records?

Response AThe field moved: the API now nests it as customer.id. Read it with row.get("customer", {}).get("id"), log any row where it is still missing, and backfill those from yesterday’s snapshot before the load runs.
Response BWrap the lookup in try/except KeyError: continue. The job will finish again and the error goes away.
Rated by
A level 4 rater
Qualified
Passed this project’s exam, 92%
Gold record
37 of the last 40 within tolerance
Time on task
6 min 12 s
Task completion
A 5 · B 2

A fixes the cause. B makes the error disappear by skipping the rows.

Factual accuracy
A 5 · B 3

Both are valid Python. B’s claim that nothing is lost is wrong.

Best response overall
A is much better

B would ship a job that silently drops customer records.

What you get

Every verdict, with its evidence

You still get the file your pipeline expects. Each row in it now opens onto who decided, how carefully, and what checked them.

A typical results file
item_id,preferred
eval-0415,B
eval-0416,A
eval-0417,A
eval-0418,B
eval-0419,B
eval-0420,A
… 994 more rows

A verdict per row. Useful, and all most teams get. Nothing on who decided, or whether to trust them.

The same row, from Cicada
  • Rater and level
  • Exam score, where there is one
  • Gold accuracy to date
  • Time on task
  • A screen-shared session
  • A reason for every score
  • The other raters on the item
  • How far they agreed
  • QA reviewer and verdict
  • The ledger entry
  • Reversal, if it failed
  • Its full history

Then everything that checked it

The same judgement, as your team sees it in Cicada: every check, every reviewer, every decision.

Gold record, Rater 2291

37 / 40

within tolerance on known-answer tasks.

Others who rated this item
  • Rater 22916 min 12 sA is better
  • Rater 08744 min 50 sA is slightly better
  • Rater 33107 min 03 sA is better
Agreement

3 of 3

chose A. Where raters split, you see the split, not an average.

AB

QA review
Passed
“Reasons match the scores. B would have shipped a silent data loss; the rater caught it.”

Reviewer Q-07 · 14 October 2026

Ledger

Paid

Paid when the rating was submitted, and kept because QA passed it. A failed review reverses the payment, so nobody is paid for work you didn’t get.

Append-only · not reversed

Its history
  1. 09:41:02Claimed by Rater 2291
  2. 09:41:03Screen share started
  3. 09:47:14Rating submitted
  4. 09:47:14Ledger: payment recorded
  5. 10:02:51Second rater: A is slightly better
  6. 10:19:08Third rater: A is better
  7. 11:30:40QA review: passed
  8. 11:30:41In the batch export, with its verdict

Now ask it of all thousand

Model B 54Model A 46

Remove the raters who failed their gold checks

165 judgements came from people who missed known answers.

Remove the judgements made in under a minute

Too fast to have read both answers. Most chose the longer one.

Remove what QA failed

A reviewer read the reasons, and they didn’t hold up.

48–52

The new model wasn’t better.

You found out at 2:14 a.m., not three weeks after the launch.

Try it yourself

Run the re-count

The same thousand judgements. Take out the careless ones in any order and watch who wins.

Model A vs Model B · 1,000 judgementsExample evaluation
54Model B, the new one
46Model A, the current one

Each square is one judgement. Teal chose B, tomato chose A. Faded ones are removed.

Re-count without

Every judgement counted. Model B is ahead.

Pairwise preferenceCustom rubricsGold-standard checksOverlap samplingFleiss’ κ agreementQA review & reworkAppealsQualification examsRater levelsAppend-only ledgerAudit logJSONL export

How a project runs

Six steps between a prompt and a label you can trust

Every control exists because unmeasured labelling fails quietly, and you find out from the model, months later.

  1. 01

    Define the rubric

    Criteria, scales and required justifications are set per batch and validated on the way in, so an incomplete rating can’t enter the dataset.

    Rubric · batch ALPHA-WORKValidated
    Overall preference7-point scalerequired
    Factual accuracy1–5 scalerequired
    Task completion1–5 scalerequired
    JustificationFree textrequired

    Every submission is checked against this schema server-side.

  2. 02

    Qualify the cohort

    Where a project requires it, raters pass an exam built from gold answers before touching live work. Harder batches open only to higher tiers, and tiers can move with measured accuracy.

    Qualification exam · Alpha ProjectPassed

    Gold answers matched

    8/10

    Pass mark 70%

    L1
    L2
    L3
    L4
    L5

    Passed. Live work on this project is now in the queue.

  3. 03

    Rate blind and independent

    Each rater sees the prompt, both responses and the rubric, never anyone else’s answer. Every required field and a written justification must be in before a rating can be submitted.

    Task 9F2C-014 · RatingBlind

    “Explain why the sky is blue to a ten-year-old.”

    A
    B
    A
    A
    A
    =
    B
    B
    B
    A pitches it at the reader’s level; B is correct but unreadable for a child

  4. 04

    Measure while it runs

    Known-answer tasks are mixed invisibly into the queue, and a share of every batch is rated by several people. Accuracy and agreement are live numbers, not a post-mortem.

    Quality · liveHealthy

    Gold accuracy

    93.4%

    Fleiss’ κ

    0.71

    2 overlap groups disputed this batch → flagged, not averaged.

  5. 05

    Review with consequences

    A reviewer passes, returns or fails ratings, giving a written reason for every return and fail. Failed work is withdrawn and its payment reversed on the ledger, and the rater can appeal with a reason.

    Review queue3 decided

    Passed 7A1C-031

    Accurate and well justified.

    Needs rework 88F0-112

    Expand the accuracy justification with a source.

    Failed 2D9B-078

    Justification contradicts the choice. Payment reversed.

  6. 06

    Deliver with the trail

    Export preference pairs as JSONL for training, or the full record as CSV: every rating, justification, rater and QA verdict behind each label.

    export · ALPHA-WORK.jsonlReady
    { "task_id": "9F2C-014",
      "prompt": "Explain why the sky is blue…",
      "chosen_response": "Sunlight looks white, but…",
      "rejected_response": "Rayleigh scattering…",
      "reward_score": 2,
      "human_justification": "A is pitched at the reader…" }
    JSONL pairsFull CSV

Ship on numbers you can defend

Send us a sample of the evaluation you’re about to decide on. We’ll rate it with qualified, checked raters, and you’ll see every judgement behind the score before you commit to anything.

Send us a sampleHow it works

The judgements, raters and figures on this page are an example, not a client’s results.