N O V E M B E R 2 0 17

© KTSDESIGN/Getty Images

Risk

Controlling machine-learning
algorithms and their biases
Myths aside, artificial intelligence is as prone to bias as the human kind. The good news is that the biases in
algorithms can also be diagnosed and treated.

Tobias Baer and Vishnu Kamalnath

Companies are moving quickly to apply machine is even older, having been expressed by pioneering
learning to business decision making. New mathematicians in the early 19th century. It has
programs are constantly being launched, setting come into its own in the past two decades, with
complex algorithms to work on large, frequently the advent of powerful computers, the Internet,
refreshed data sets. The speed at which this is taking and mass-scale digitization of information. In the
place attests to the attractiveness of the technology, domain of artificial intelligence, machine learning
but the lack of experience creates real risks. increasingly refers to computer-aided decision
Algorithmic bias is one of the biggest risks because it making based on statistical algorithms generating
compromises the very purpose of machine learning. data-driven insights (see sidebar, “Machine learning:
This often-overlooked defect can trigger costly The principal approach to realizing the promise of
errors and, left unchecked, can pull projects and artificial intelligence”).
organizations in entirely wrong directions. Effective
efforts to confront this problem at the outset will Among its most visible uses is in predictive modeling.
repay handsomely, allowing the true potential This has wide and familiar business applications,
of machine learning to be realized most efficiently. from automated customer recommendations
to credit-approval processes. Machine learning
Machine learning has been in scientific use for magnifies the power of predictive models through
more than half a century as a term describing great computational force. To create a functioning
programmable pattern recognition. The concept statistical algorithm by means of a logistic

Controlling machine-learning algorithms and their biases 1

 Machine learning: The principal approach to realizing
the promise of artificial intelligence
Artificial intelligence is the science and engineering model can predict outcomes for new observations.
of automated problem solving. The object is to The form of data used to predict outcomes
generate solutions by using computers to mimic can be structured or unstructured, whether or
the cognitive functions associated with deliberative not supervised learning is applied. However,
thought, including perception, reasoning, and learning. unstructured data can be processed directly only
through machine learning; when more traditional
Machine learning is the most prevalent means by techniques such as regression are used, the
which the potential of artificial intelligence is being data scientist must first aggregate unstructured data
exploited. The term refers to the ability of computers into structured data based on business rules
to detect patterns in large data sets through the or independent analyses and procedures.
application of algorithms. In addition to uncovering
potentially powerful insights in the data, computers Deep learning is the most advanced technique for
can be programmed to train themselves to make predictive modeling. It connects software-based
data-driven predictions. calculators to form a complex artificial “neural
network,” often 50 or more layers deep. The simplest
Predictive modeling, also called supervised predictive-modeling techniques are regression
learning, is a machine-learning approach that builds modeling and simple decision trees. More advanced
pattern-recognition models using sample data with techniques include random forests (a more complex
known attributes and outcomes (labeled “training and sensitive decision-tree model) and support
data”). Working from the known patterns, the vector machines (for sophisticated data classification).

regression, for example, missing variables must defaults may be in the offing, unaccounted for in the
be replaced by assumed numeric values (a process statistical risk model of the lending institution. With
called imputation). Machine-learning algorithms access to the right data and guidance by subject-
are often constructed to interpret “missing” as a matter experts, predictive machine-learning models
possible value and then proceed to develop the best could find the hidden patterns in the data and
prediction for cases where the value is missing. correct for such spikes.
Machine learning is able to manage vast amounts of
data and detect many more complex patterns within The persistence of bias
them, often attaining superior predictive power. In automated business processes, machine-
learning algorithms make decisions faster than
In credit scoring, for example, customers with a long human decision makers and at a fraction of the
history of maintaining loans without delinquency cost. Machine learning also promises to improve
are generally determined to be of low risk. But decision quality, due to the purported absence of
what if the mortgages these customers have been human biases. Human decision makers might, for
maintaining were for years supported by substantial example, be prone to giving extra weight to their
tax benefits that are set to expire? A spike in personal experiences. This is a form of bias known

2 Controlling machine-learning algorithms and their biases

as anchoring, one of many that can affect business Predicting behavior: ‘Winner takes all’
decisions. Availability bias is another. This is a Machine learning can perpetuate and even amplify
mental shortcut (heuristic) by which people make behavioral biases. By design, a social-media site
familiar assumptions when faced with decisions. filtering news based on user preferences reinforces
The assumptions will have served adequately in natural confirmation bias in readers. The site may
the past but could be unmerited in new situations. even be systematically preventing perspectives
Confirmation bias is the tendency to select evidence from being challenged with contradictory evidence.
that supports preconceived beliefs, while loss- The self-fulfilling prophecy is a related by-product
aversion bias imposes undue conservatism on of algorithms. Financially sound companies can
decision-making processes. run afoul of banks’ scoring algorithms and find
themselves without access to working capital. If they
Machine learning is being used in many decisions are unable to sway credit officers with factual logic,
with business implications, such as loan approvals a liquidity crunch could wipe out an entire class of
in banking, and with personal implications, such as businesses. These examples reveal a certain “winner
diagnostic decisions in hospital emergency rooms. takes all” outcome that affects those machine-
The benefits of removing harmful biases from such learning algorithms designed to replicate human
decisions are obvious and highly desirable, whether decision making.
they come in financial, medical, or some other form.
Data limitations
Some machine learning is designed to emulate Machine learning can reveal valuable insights in
the mechanics of the human brain, such as deep complex data sets, but data anomalies and errors
learning, with its artificial neural networks. If can lead algorithms astray. Just as a traumatic
biases affect human intelligence, then what about childhood accident can cause lasting behavioral
artificial intelligence? Are the machines biased? distortion in adults, so can unrepresentative events
The answer, of course, is yes, for some basic reasons. cause machine-learning algorithms to go off course.
First, machine-learning algorithms are prone to Should a series of extraordinary weather events or
incorporating the biases of their human creators. fraudulent actions trigger spikes in default rates,
Algorithms can formalize biased parameters created for example, credit scorecards could brand a region
by sales forces or loan officers, for example. Where as “high risk” despite the absence of a permanent
machine learning predicts behavioral outcomes, structural cause. In such cases, inadequate
the necessary reliance on historical criteria will algorithms will perpetuate bias unless corrective
reinforce past biases, including stability bias. This is action is taken.
the tendency to discount the possibility of significant
change—for example, through substitution effects Companies seeking to overcome biases with
created by innovation. The severity of this bias statistical decision-making processes may find
can be magnified by machine-learning algorithms that the data scientists supervising their machine-
that must assume things will more or less continue learning algorithms are subject to these same
as before in order to operate. Another basic bias- biases. Stability biases, for example, may cause
generating factor is incomplete data. Every machine- data scientists to prefer the same data that human
learning algorithm operates wholly within the decision makers have been using to predict outcomes.
world defined by the data that were used to calibrate Cost and time pressures, meanwhile, could deter
it. Limitations in the data set will bias outcomes, them from collecting other types of data that harbor
sometimes severely. the true drivers of the outcomes to be predicted.

Controlling machine-learning algorithms and their biases 3

 The problem of stability bias First, users of machine-learning algorithms need
Stability bias—the tendency toward inertia in an to understand an algorithm’s shortcomings and
uncertain environment—is actually a significant refrain from asking questions whose answers will
problem for machine-learning algorithms. be invalidated by algorithmic bias. Using a machine-
Predictive models operate on patterns detected in learning model is more like driving a car than riding
historical data. If the same patterns cease to exist, an elevator. To get from point A to point B, users cannot
then the model would be akin to an old railroad simply push a button; they must first learn operating
timetable—valuable for historians but not useful procedures, rules of the road, and safety practices.
for traveling in the here and now. It is frustratingly
difficult to shape machine-learning algorithms to Second, data scientists developing the algorithms
recognize a pattern that is not present in the data, must shape data samples in such a way that biases
even one that human analysts know is likely to are minimized. This step is a vital and complex
manifest at some point. To bridge the gap between part of the process and worthy of much deeper
available evidence and self-evident reality, synthetic consideration than can be provided in this short
data points can be created. Since machine-learning article. For the moment, let us remark that available
algorithms try to capture patterns at a very detailed historical data are often inadequate for this purpose,
level, however, every attribute of each synthetic data and fresh, unbiased data must be generated through
point would have to be crafted with utmost care. a controlled experiment.

In 2007, an economist with an inkling that Finally, executives should know when to use and
credit-card defaults and home prices were linked when not to use machine-learning algorithms. They
would have been unable to build a predictive must understand the true values involved in the
model showing this relationship, since it had not trade-off: algorithms offer speed and convenience,
yet appeared in the data. The relationship was while manually crafted models, such as decision
revealed, precipitously, only when the financial trees or logistic regression—or for that matter even
crisis hit and housing prices began to fall. If certain human decision making—are approaches that have
data limitations are permitted to govern modeling more flexibility and transparency.
choices, seriously flawed algorithms can result.
Models will be unable to recognize obviously real What’s in your black box?
but unexpected changes. Some US mortgage models From a user’s standpoint, machine-learning
designed before the financial crisis could not algorithms are black boxes. They offer quick and
mathematically accept negative changes in home easy solutions to those who know little or nothing
prices. Until negative interest rates appeared in of their inner workings. They should be applied
the real world, they were statistically unrecognized with discretion, but knowing enough to exercise
and no machine-learning algorithm in the world discretion takes effort. Business users seeking
could have predicted their appearance. to avoid harmful applications of algorithms are a
little like consumers seeking to eat healthy food.
Addressing bias in machine-learning algorithms Health-conscious consumers must study literature
As described in a previous article in McKinsey on on nutrition and read labels in order to avoid
Risk,1 companies can take measures to eliminate excess calories, harmful additives, or dangerous
bias or protect against its damaging effects in allergens. Executives and practitioners will
human decision making. Similar countermeasures likewise have to study the algorithms at the core
can protect against algorithmic bias. Three filters of their business and the problems they are
are of prime importance. designed to resolve.

4 Controlling machine-learning algorithms and their biases

They will then be able to understand monitoring promotional strategy. In one way or the other, sound
reports on the algorithms, ask the right questions, business judgment therefore is indispensable.
and challenge assumptions.
Squeezing bias out of the development sample
In credit scoring, for example, built-in stability Tests can ensure that unwanted biases of past
bias prevents machine-learning algorithms from human decision makers, such as gender biases, for
accounting for certain rapid behavioral shifts in example, have not been inadvertently baked into
applicants. These can occur if applicants recognize machine-learning algorithms. Here a challenge lies
the patterns that are being punished by models. in adjusting the data such that the biases disappear.
Salespeople have been known to observe the decision
patterns embedded in algorithms and then coach One of the most dangerous myths about machine
applicants by reverse-engineering the behaviors that learning is that it needs no ongoing human
will maximize the odds of approval. intervention. Business users would do better to view
the application of machine-learning algorithms like
A subject that frequently arises as a predictor of the creation and tending of a garden. Much human
risk in this context is loan tenor. Riskier customers oversight is needed. Experts with deep machine-
generally prefer longer loan tenors, in recognition learning knowledge and good business judgment are
of potential difficulties in repayment. Many low-risk like experienced gardeners, carefully nurturing the
customers, by contrast, aim to minimize interest plants to encourage their organic growth. The data
expense by choosing shorter tenors. A machine- scientist knows that in machine learning the answers
learning algorithm would jump on such a pattern, can be useful only if we ask the right questions.
penalizing applications for longer tenors with a
higher risk estimate. Soon salespeople would nudge In countering harmful biases, data scientists seek
risky applicants into the approval range of the to strengthen machine-learning algorithms
credit score by advising them to choose the shortest where it most matters. Training a machine-learning
possible tenor. Burdened by an exceptionally high algorithm is a bit like building muscle mass.
monthly installment (due to the short tenor), many Fitness trainers take great pains in teaching their
of these applicants will ultimately default, causing a clients the proper form of each exercise so that
spike in credit losses. only targeted muscles are worked. If the hips are
engaged in a motion designed to build up biceps,
Astute observers can thus extract from the black for example, the effectiveness of the exercise
box the variables with the greatest influence will be much reduced. By using stratified sampling
on an algorithm’s predictions. Business users and optimized observation weights, data scientists
should recognize that in this case loan tenor was ensure that the algorithm is most powerful for
an influential predictor. They can either remove those decisions in which the business impact of a
the variable from the algorithm or put in place a prediction error is the greatest. This cannot be done
safeguard to prevent a behavioral shift. Should automatically, even by advanced machine-learning
business users fail to recognize these shifts, banks algorithms such as boosting (an algorithm designed
might be able to identify them indirectly, by to reduce algorithmic bias). Advanced algorithms
monitoring the distribution of monthly applications can correct for a statistically defined concept of
by loan tenor. The challenge here is to establish error, but they cannot distinguish errors with high
whether a marked shift is due to a deliberate change business impact from those of negligible importance.
in behavior by applicants or to other factors, Another example of the many statistical techniques
such as changes in economic conditions or a bank’s data scientists can deploy to protect algorithms

Controlling machine-learning algorithms and their biases 5

 from biases is the careful analysis of missing values. To defeat fraud, organizations need to deploy
By determining whether the values are missing algorithms that adjust instantaneously, the
systematically, data scientists are introducing moment the defrauders change their tactics.
“hindsight bias.” This use of bias to fight bias allows
the algorithm to peek beyond its data-determined ƒƒ What insights do we have? The superiority of
limitations to the correct answer. The data scientists the handcrafted model depends on the business
can then decide whether and how to address insights embedded in it by the data scientist.
the missing values or whether the sample structure If an organization possesses no insights, then
needs to be adjusted. the problem solving will have to be guided by
the data. At this point, a machine-learning
Deciding when to use machine-learning algorithms algorithm might be preferred for its speed and
An organization considering using an algorithm convenience. However, rather than blindly
on a business problem should be making an trusting an algorithm, an organization in this
explicit choice based on the cost-benefit trade- situation could decide that it is better to bring
off. A machine-learning algorithm will be fast in a consultant to help develop value-adding
and convenient, but more familiar, traditional business insights.
decision-making processes will be easier to build
for a particular purpose and will also be more ƒƒ Which problems are worth solving? One of
transparent. Traditional approaches include human the promises of machine learning is that it can
decision making or handcrafted models such as address problems that were once unrecognized
decision trees or logistic-regression models—the or thought to be too costly to solve with a
analytic workhorses used for decades in business handcrafted model. Decision making on these
and the public sector to assign probabilities to outcomes. problems has been heretofore random or
The best data scientists can even use machine- unconscious. When reconsidering such problems,
learning algorithms to enhance the power of hand- organizations should identify those with significant
crafted models. They have been able to build advanced bottom-line business impact and then assign
logistic-regression models with predictive power their best data scientists to work on them.
approaching that of a machine-learning algorithm.
In addition to these considerations, companies
Three questions can be considered when deciding to implementing large-scale machine-learning
use machine-learning algorithms: programs should make appropriate organizational
and cultural changes to support them. Everyone
ƒƒ How soon do we need the solution? within the scope of the programs should understand
The time factor is often of prime importance and trust the machine-learning models—only then
in solving business problems. The optimal will maximum impact be achieved.
statistical model may be obsolete by the time it
is completed. When the business environment Implementation: Standards,
is changing fast, a machine-learning algorithm validation, knowledge
developed overnight could far outperform a How would a business go about implementing these
superior traditional model that is months in recommendations? The practical application and
the making. For this reason, machine-learning debiasing of machine-learning algorithms should be
algorithms are preferred for combating fraud. governed by a conscious and eventually systematic
Defrauders typically act quickly to circumvent process throughout the organization. While not
the latest detection mechanisms they encounter. as stringent and formal, the approach is related to

6 Controlling machine-learning algorithms and their biases

mature model development and validation processes ƒƒ A culture for continuous knowledge
by which large institutions are gaining strategic development. Institutions should invest in
control of model proliferation and risk. Three building developing and disseminating knowledge on
blocks are critically important for implementation: data science and business applications. Machine-
learning applications should be continuously
ƒƒ Business-based standards for machine-learning monitored for new insights and best practices,
approvals. A template should be developed in order to create a culture of knowledge
for model documentation, standardizing the enhancement and to keep people informed of
process for the intake of modeling requests. It both the difficulties and successes that come
should include the business context and prompt with using such applications.
requesters with specific questions on business
impact, data, and cost-benefit trade-offs. The Creating a conscious, standards-based system for
process should require active user participation developing machine-learning algorithms will involve
in the drive to find the most suitable solution leaders in many judgment-based decisions. For this
to the business problem (note that passive check- reason, debiasing techniques should be deployed to
lists or guidelines, by comparison, tend to be maximize outcomes. An effective technique in this
ignored). The model’s key parameters should be context is a “premortem” exercise designed to pinpoint
defined, including a standard set of analyses to the limitations of a proposed model and help executives
be run on the raw data inputs, the processed sample, judge the business risks involved in a new algorithm.
and the modeling outputs. The model should be
challenged in a discussion with business users.

ƒƒ Professional validation of machine-learning Sometimes lost in the hype surrounding machine

algorithms. An explicit process is needed learning is the fact that artificial intelligence is
for validating and approving machine-learning as prone to bias as the real thing it emulates. The
algorithms. Depending on the industry and good news is that biases can be understood and
business context—especially the economic managed—if we are honest about them. We cannot
implication of errors—it may not have to be afford to believe in the myth of machine-perfected
as stringent as the formal validation of banks’ intelligence. Very real limitations to machine learning
risk models by internal validation teams and must be constantly addressed by humans. For
regulators. However, the process should businesses, this means the creation of incremental,
establish validation standards and an ongoing insights-based value with the aid of well-monitored
monitoring program for the new model. The machines. That is a realistic algorithm for achieving
standards should account for the characteristics machine-learning impact.
of machine-learning models, such as automatic
updates of the algorithm whenever fresh data 1 Tobias Baer, Sven Heiligtag, and Hamid Samandari, “The
are captured. This is an area where most banks business logic in debiasing,” May 2017, [Link].
still need to develop appropriate validation and Tobias Baer is a partner in McKinsey’s Taipei office, and
monitoring standards. If algorithms are updated Vishnu Kamalnath is an analytics specialist in the North
weekly, for example, validation routines must be American Knowledge Center in Waltham, Massachusetts.
completed in hours and days rather than weeks
and months. Yet it is also extremely important to Copyright © 2017 McKinsey & Company.
put in place controls that alert users to potential All rights reserved.
sudden or creeping bias in fresh data.

Controlling machine-learning algorithms and their biases 7