Noise Excerpt Intro Wbur

INTRODUCTION
Two Kinds of Error
I magine that four teams of friends have gone to a shooting arcade.

Each team consists of five people; they share one rifle, and each
person fires one shot. Figure 1 shows their results.
In an ideal world, every shot would hit the bull’s‑eye.
Team A Team B
Team C Team D
Figure 1: Four teams
Noise_HCtextF1.indd 3 3/17/21 10:26:41 AM
introduction
That is nearly the case for Team A. The team’s shots are tightly
clustered around the bull’s‑eye, close to a perfect pattern.
We call Team B biased because its shots are systematically off
target. As the figure illustrates, the consistency of the bias supports a
prediction. If one of the team’s members were to take another shot,
we would bet on its landing in the same area as the first five. The con‑
sistency of the bias also invites a causal explanation: perhaps the
gunsight on the team’s rifle was bent.
We call Team C noisy because its shots are widely scattered. There
is no obvious bias, because the impacts are roughly centered on the
bull’s‑eye. If one of the team’s members took another shot, we would
know very little about where it is likely to hit. Furthermore, no inter‑
esting hypothesis comes to mind to explain the results of Team C.
We know that its members are poor shots. We do not know why they
are so noisy.
Team D is both biased and noisy. Like Team B, its shots are sys‑
tematically off target; like Team C, its shots are widely scattered.
But this is not a book about target shooting. Our topic is human
error. Bias and noise — systematic deviation and random scatter —
are different components of error. The targets illustrate the difference.
The shooting range is a metaphor for what can go wrong in
human judgment, especially in the diverse decisions that people
make on behalf of organizations. In these situations, we will find the
two types of error illustrated in figure 1. Some judgments are biased;
they are systematically off target. Other judgments are noisy, as peo‑
ple who are expected to agree end up at very different points around
the target. Many organizations, unfortunately, are afflicted by both
bias and noise.
Figure 2 illustrates an important difference between bias and
noise. It shows what you would see at the shooting range if you were
shown only the backs of the targets at which the teams were shoot‑
ing, without any indication of the bull’s‑eye they were aiming at.
From the back of the target, you cannot tell whether Team A or
Team B is closer to the bull’s‑eye. But you can tell at a glance that
t w o k i n d s o f e rr o r
Team A Team B
Team C Team D
Figure 2: Looking at the back of the target
Teams C and D are noisy and that Teams A and B are not. Indeed,
you know just as much about scatter as you did in figure 1. A general
property of noise is that you can recognize and measure it while
knowing nothing about the target or bias.
The general property of noise just mentioned is essential for our
purposes in this book, because many of our conclusions are drawn
from judgments whose true answer is unknown or even unknowable.
When physicians offer different diagnoses for the same patient, we
can study their disagreement without knowing what ails the patient.
When film executives estimate the market for a movie, we can study
the variability of their answers without knowing how much the film
eventually made or even if it was produced at all. We don’t need to
know who is right to measure how much the judgments of the same
case vary. All we have to do to measure noise is look at the back of the
target.
To understand error in judgment, we must understand both bias
and noise. Sometimes, as we will see, noise is the more important
problem. But in public conversations about human error and in
introduction
organizations all over the world, noise is rarely recognized. Bias is the
star of the show. Noise is a bit player, usually offstage. The topic of
bias has been discussed in thousands of scientific articles and dozens
of popular books, few of which even mention the issue of noise. This
book is our attempt to redress the balance.
In r eal-world decisions, the amount of noise is often scandalously
high. Here are a few examples of the alarming amount of noise in
situations in which accuracy matters:
• Medicine is noisy. Faced with the same patient, different doctors

make different judgments about whether patients have skin
cancer, breast cancer, heart disease, tuberculosis, pneumonia,
depression, and a host of other conditions. Noise is especially
high in psychiatry, where subjective judgment is obviously
important. However, considerable noise is also found in areas
where it might not be expected, such as in the reading of X‑rays.
• Child custody decisions are noisy. Case managers in child
protection agencies must assess whether children are at risk of
abuse and, if so, whether to place them in foster care. The
system is noisy, given that some managers are much more likely
than others to send a child to foster care. Years later, more of
the unlucky children who have been assigned to foster care by
these heavy-handed managers have poor life outcomes: higher
delinquency rates, higher teen birth rates, and lower earnings.
• Forecasts are noisy. Professional forecasters offer highly variable

predictions about likely sales of a new product, likely growth in
the unemployment rate, the likelihood of bankruptcy for troubled
companies, and just about everything else. Not only do they
disagree with each other, but they also disagree with themselves.
For example, when the same software developers were asked on
two separate days to estimate the completion time for the same
task, the hours they projected differed by 71%, on average.
• Asylum decisions are noisy. Whether an asylum seeker will be
admitted into the United States depends on something like a
t w o k i n d s o f e rr o r
lottery. A study of cases that were randomly allotted to different

judges found that one judge admitted 5% of applicants, while
another admitted 88%. The title of the study says it all: “Refugee
Roulette.” (We are going to see a lot of roulette.)
• Personnel decisions are noisy. Interviewers of job candidates
make widely different assessments of the same people.
Performance ratings of the same employees are also highly
variable and depend more on the person doing the assessment
than on the performance being assessed.
• Bail decisions are noisy. Whether an accused person will be
granted bail or instead sent to jail pending trial depends partly
on the identity of the judge who ends up hearing the case. Some
judges are far more lenient than others. Judges also differ
markedly in their assessment of which defendants present the
highest risk of flight or reoffending.
• Forensic science is noisy. We have been trained to think of
fingerprint identification as infallible. But fingerprint examiners
sometimes differ in deciding whether a print found at a crime
scene matches that of a suspect. Not only do experts disagree,
but the same experts sometimes make inconsistent decisions
when presented with the same print on different occasions.
Similar variability has been documented in other forensic
science disciplines, even DNA analysis.
• Decisions to grant patents are noisy. The authors of a leading
study on patent applications emphasize the noise involved:
“Whether the patent office grants or rejects a patent is
significantly related to the happenstance of which examiner is
assigned the application.” This variability is obviously
troublesome from the standpoint of equity.
All these noisy situations are the tip of a large iceberg. Wherever
you look at human judgments, you are likely to find noise. To improve
the quality of our judgments, we need to overcome noise as well
as bias.

Noise Excerpt Intro Wbur

Uploaded by

Noise Excerpt Intro Wbur

Uploaded by

INTRODUCTION

Two Kinds of Error

I magine that four teams of friends have gone to a shooting arcade.

• Medicine is noisy. Faced with the same patient, different doctors

• Forecasts are noisy. Professional forecasters offer highly variable

lottery. A study of cases that were randomly allotted to different

You might also like