Criteria for Low-Quality Tasks on Loft 2.
These criteria are designed to avoid submitting a task with very low quality and divided in two
groups: “Basic-Level” and “Cumulative” mistakes.
Group 1: Basic-Level Mistakes
These are mistakes that can be noticed while giving the task a quick scroll-through:
● Skipping a task that should not be skipped
● Pre-filled speaker turns not corrected.
○ Pre filled speaker turns are around 1 minute long. If more than one pre-filled speaker
turn is not adequately corrected, it’s counted as a basic-level mistake.
● 30-second rule not followed for the entire or most of the task.
○ If the task has continuous speech that lasts for several minutes, even one 30-second
speaker turn incorrectly created can impact the entire task and create a lot of work for
the reviewer, in this case, a task will be considered as low-quality of work.
○ If there are less than 5 speaker turns (not continuous) that violate the 30-second rule,
it will be tolerated.
● Missing annotations - noise and music annotations not present, but required in the task
○ No annotations at all (would be a contract-terminator reason)
○ Some annotations created, but insufficiently - will be counted as a cumulative mistake
● Annotations not used properly
○ One long annotation created for the entire task, but there is no noise/music audible for
that duration of time
○ Many smaller noise annotations created, but noise is present for the entire task
■ This means that all small annotations should be removed and one large
annotation created - very time consuming
○ If annotations are used, but are incorrectly placed - will be counted as a cumulative
mistake
● Missing speaker turns - Speech not transcribed
○ More than 10 seconds of missing speech
■ These 10 seconds can be cumulative, meaning 2 seconds at the beginning of
the task, 5 seconds in the middle, and 3 seconds in the end of the task would
add up to a missing speech of 10 seconds.
● Missing speaker labels - All of the speech transcribed under one speaker label with more
speakers clearly heard.
● PII not marked
○ PII not marked where required throughout the task - (more than 10 times at least)
○ Some PII marked, others not - Cumulative mistake
Group 2: Cumulative Mistakes
This group focuses on the number of mistakes rather than the type of mistakes. If there are
more than 10 mistakes per minute (for tasks shorter than 5 minutes), or more than 50 mistakes
in the first 5 minutes of the task (for longer tasks) - this will drastically decrease the quality rate
of the relevant task, thus the transcriber’s overall quality score.
What is counted as a single mistake?
● The start time of a segment is incorrect and needs adjustment
○ 100ms buffer rule not respected - speech either cut off at the beginning, or buffer too
large.
● The end time of a segment is incorrect and needs adjustment
○ 100ms buffer rule not respected - speech either cut off at the end, or buffer too large.
● A segment has one 0.5s mistake
○ Each 0.5s mistake in a segment is counted as one mistake. If a segment has 3
mistakes of this kind, it will be counted as 3 mistakes.
● Speaker labels mistake
○ A speaker turn is attributed to the wrong speaker
● Missing speaker turn
○ Each speaker turn that should be created, but it’s not is counted as a single mistake
● Incorrect annotation
○ Added or missing noise/music/laughter/applause annotations - each annotation
missing or added - single mistake.
○ Incorrect start/end time of annotation - single mistake
● Incorrect punctuation usage
○ Each time a rule from the guidelines regarding punctuation is not followed is counted
as a single mistake
● Incorrect number (number/date/time/currency) formatting
○ Each number incorrectly formatted - single mistake
● Violation of any other rule in the guidelines - single mistake
● Blank space at ends or beginnings of speaker turns
○ Each blank space - single mistake
● Incorrect transcription - Each incorrect transcription is counted as a single mistake.
