Bandit Algorithms

Tor Lattimore and Csaba Szepesvári

This is the (free) online edition. The content is the same as the print edition, published
by Cambridge University Press, except that minor typos are corrected here. There are
also font and other typographical differences that mean the page numbers do not
match between the versions.
 Contents

Contents page ii

Part I Bandits, Probability and Concentration 7

1 Introduction 8
1.1 The Language of Bandits 10
1.2 Applications 13
1.3 Notes 16
1.4 Bibliographic Remarks 16

2 Foundations of Probability ( ) 18
2.1 Probability Spaces and Random Elements 18
2.2 σ-Algebras and Knowledge 26
2.3 Conditional Probabilities 28
2.4 Independence 29
2.5 Integration and Expectation 30
2.6 Conditional Expectation 33
2.7 Notes 37
2.8 Bibliographic Remarks 41
2.9 Exercises 42

3 Stochastic Processes and Markov Chains ( ) 46

3.1 Stochastic Processes 47
3.2 Markov Chains 48
3.3 Martingales and Stopping Times 49
3.4 Notes 52
3.5 Bibliographic Remarks 53
3.6 Exercises 54

4 Stochastic Bandits 56
4.1 Core Assumptions 56
4.2 The Learning Objective 57
4.3 Knowledge and Environment Classes 57
 CONTENTS iii

4.4 The Regret 60

4.5 Decomposing the Regret 62
4.6 The Canonical Bandit Model ( ) 63
4.7 The Canonical Bandit Model for Uncountable Action Sets ( ) 66
4.8 Notes 66
4.9 Bibliographical Remarks 68
4.10 Exercises 69

5 Concentration of Measure 73
5.1 Tail Probabilities 73
5.2 The Inequalities of Markov and Chebyshev 74
5.3 The Cramér-Chernoff Method and Subgaussian Random Variables 76
5.4 Notes 78
5.5 Bibliographical Remarks 80
5.6 Exercises 80

Part II Stochastic Bandits with Finitely Many Arms 88

6 The Explore-Then-Commit Algorithm 90

6.1 Algorithm and Regret Analysis 90
6.2 Notes 94
6.3 Bibliographical Remarks 94
6.4 Exercises 95

7 The Upper Confidence Bound Algorithm 101

7.1 The Optimism Principle 101
7.2 Notes 109
7.3 Bibliographical Remarks 110
7.4 Exercises 110

8 The Upper Confidence Bound Algorithm: Asymptotic Optimality 116

8.1 Asymptotically Optimal UCB 116
8.2 Notes 119
8.3 Bibliographic Remarks 119
8.4 Exercises 120

9 The Upper Confidence Bound Algorithm: Minimax Optimality ( ) 122

9.1 The MOSS Algorithm 122
9.2 Two Problems 126
9.3 Notes 127
9.4 Bibliographic Remarks 128
9.5 Exercises 128

10 The Upper Confidence Bound Algorithm: Bernoulli Noise ( ) 132

 CONTENTS iv

10.1 Concentration for Sums of Bernoulli Random Variables 132

10.2 The KL-UCB Algorithm 136
10.3 Notes 139
10.4 Bibliographic Remarks 140
10.5 Exercises 140

Part III Adversarial Bandits with Finitely Many Arms 143

11 The Exp3 Algorithm 147

11.1 Adversarial Bandit Environments 147
11.2 Importance-Weighted Estimators 149
11.3 The Exp3 Algorithm 151
11.4 Regret Analysis 152
11.5 Notes 156
11.6 Bibliographic Remarks 159
11.7 Exercises 159

12 The Exp3-IX Algorithm 164

12.1 The Exp3-IX Algorithm 164
12.2 Regret Analysis 166
12.3 Notes 170
12.4 Bibliographic Remarks 171
12.5 Exercises 171

Part IV Lower Bounds for Bandits with Finitely Many Arms 176

13 Lower Bounds: Basic Ideas 179

13.1 Main Ideas Underlying Minimax Lower Bounds 180
13.2 Notes 182
13.3 Bibliographic Remarks 183
13.4 Exercises 183

14 Foundations of Information Theory ( ) 185

14.1 Entropy and Optimal Coding 185
14.2 Relative Entropy 187
14.3 Notes 190
14.4 Bibliographic Remarks 193
14.5 Exercises 193

15 Minimax Lower Bounds 197

15.1 Relative Entropy Between Bandits 197
15.2 Minimax Lower Bounds 198
15.3 Notes 200
15.4 Bibliographic Remarks 202
 CONTENTS v

15.5 Exercises 202

16 Instance-Dependent Lower Bounds 205

16.1 Asymptotic Bounds 206
16.2 Finite-Time Bounds 208
16.3 Notes 209
16.4 Bibliographic Remarks 210
16.5 Exercises 210

17 High-Probability Lower Bounds 214

17.1 Stochastic Bandits 215
17.2 Adversarial Bandits 217
17.3 Notes 219
17.4 Bibliographic Remarks 219
17.5 Exercises 220

Part V Contextual and Linear Bandits 221

18 Contextual Bandits 223

18.1 Contextual Bandits: One Bandit per Context 223
18.2 Bandits with Expert Advice 225
18.3 Exp4 228
18.4 Regret Analysis 228
18.5 Notes 230
18.6 Bibliographic Remarks 232
18.7 Exercises 233

19 Stochastic Linear Bandits 237

19.1 Stochastic Contextual Bandits 237
19.2 Stochastic Linear Bandits 239
19.3 Regret Analysis 242
19.4 Notes 244
19.5 Bibliographic Remarks 246
19.6 Exercises 247

20 Confidence Bounds for Least Squares Estimators 253

20.1 Martingales and the Method of Mixtures 255
20.2 Notes 260
20.3 Bibliographic Remarks 261
20.4 Exercises 261

21 Optimal Design for Least Squares Estimators 266

21.1 The Kiefer–Wolfowitz Theorem 266
21.2 Notes 269
 CONTENTS vi

21.3 Bibliographic Remarks 270

21.4 Exercises 270

22 Stochastic Linear Bandits with Finitely Many Arms 272

22.1 Notes 273
22.2 Bibliographic Remarks 274
22.3 Exercises 274

23 Stochastic Linear Bandits with Sparsity 276

23.1 Sparse Linear Stochastic Bandits 276
23.2 Elimination on the Hypercube 277
23.3 Online to Confidence Set Conversion 281
23.4 Sparse Online Linear Prediction 284
23.5 Notes 285
23.6 Bibliographical Remarks 286
23.7 Exercises 286

24 Minimax Lower Bounds for Stochastic Linear Bandits 287

24.1 Hypercube 288
24.2 Unit Ball 289
24.3 Sparse Parameter Vectors 291
24.4 Misspecified Models 292
24.5 Notes 294
24.6 Bibliographic Remarks 294
24.7 Exercises 294

25 Asymptotic Lower Bounds for Stochastic Linear Bandits 295

25.1 An Asymptotic Lower Bound for Fixed Action Sets 295
25.2 Clouds Looming for Optimism 299
25.3 Notes 301
25.4 Bibliographic Remarks 301
25.5 Exercises 301

Part VI Adversarial Linear Bandits 303

26 Foundations of Convex Analysis ( ) 305

26.1 Convex Sets and Functions 305
26.2 Jensen’s Inequality 307
26.3 Bregman Divergence 307
26.4 Legendre Functions 309
26.5 Optimisation 311
26.6 Projections 312
26.7 Notes 313
26.8 Bibliographic Remarks 313
 CONTENTS vii

26.9 Exercises 313

27 Exp3 for Adversarial Linear Bandits 317

27.1 Exponential Weights for Linear Bandits 317
27.2 Regret Analysis 319
27.3 Continuous Exponential Weights 320
27.4 Notes 322
27.5 Bibliographic Remarks 323
27.6 Exercises 324

28 Follow-the-Regularised-Leader and Mirror Descent 326

28.1 Online Linear Optimisation 326
28.2 Regret Analysis 330
28.3 Application to Linear Bandits 334
28.4 Linear Bandits on the Unit Ball 336
28.5 Notes 339
28.6 Bibliographic Remarks 342
28.7 Exercises 343

29 The Relation between Adversarial and Stochastic Linear Bandits 349

29.1 Unified View 349
29.2 Reducing Stochastic Linear Bandits to Adversarial Linear Bandits 350
29.3 Stochastic Linear Bandits with Parameter Noise 351
29.4 Contextual Linear Bandits 353
29.5 Notes 354
29.6 Bibliographic Remarks 355
29.7 Exercises 355

Part VII Other Topics 357

30 Combinatorial Bandits 361

30.1 Notation and Assumptions 362
30.2 Applications 362
30.3 Bandit Feedback 363
30.4 Semi-bandit Feedback and Mirror Descent 364
30.5 Follow-the-Perturbed-Leader 366
30.6 Notes 371
30.7 Bibliographic Remarks 373
30.8 Exercises 374

31 Non-stationary Bandits 377

31.1 Adversarial Bandits 377
31.2 Stochastic Bandits 380
31.3 Notes 382
 CONTENTS viii

31.4 Bibliographic Remarks 384

31.5 Exercises 385

32 Ranking 387
32.1 Click Models 388
32.2 Policy 391
32.3 Regret Analysis 393
32.4 Notes 397
32.5 Bibliographic Remarks 399
32.6 Exercises 400

33 Pure Exploration 402

33.1 Simple Regret 402
33.2 Best-Arm Identification with a Fixed Confidence 404
33.3 Best-Arm Identification with a Budget 411
33.4 Notes 412
33.5 Bibliographical Remarks 414
33.6 Exercises 416

34 Foundations of Bayesian Learning 420

34.1 Statistical Decision Theory and Bayesian Learning 420
34.2 Bayesian Learning and the Posterior Distribution 421
34.3 Conjugate Pairs, Conjugate Priors and the Exponential Family 425
34.4 The Bayesian Bandit Environment 429
34.5 Posterior Distributions in Bandits 430
34.6 Bayesian Regret 431
34.7 Notes 432
34.8 Bibliographic Remarks 435
34.9 Exercises 435

35 Bayesian Bandits 439

35.1 Bayesian Optimal Regret for k-Armed Stochastic Bandits 439
35.2 Optimal Stopping ( ) 440
35.3 One-armed Bayesian Bandits 442
35.4 Gittins Index 446
35.5 Computing the Gittins Index 452
35.6 Notes 453
35.7 Bibliographical Remarks 455
35.8 Exercises 456

36 Thompson Sampling 459

36.1 Finite-Armed Bandits 460
36.2 Frequentist Analysis 461
36.3 Linear Bandits 465
 CONTENTS ix

36.4 Information Theoretic Analysis 467

36.5 Notes 470
36.6 Bibliographic Remarks 473
36.7 Exercises 474

Part VIII Beyond Bandits 477

37 Partial Monitoring 478

37.1 Finite Adversarial Partial Monitoring Problems 479
37.2 The Structure of Partial Monitoring 482
37.3 Classification of Finite Adversarial Partial Monitoring 486
37.4 Lower Bounds 487
37.5 Policy and Upper Bounds 491
37.6 Proof of Theorem 37.16 496
37.7 Proof of Theorem 37.17 497
37.8 Proof of the Classification Theorem 502
37.9 Notes 502
37.10 Bibliographical Remarks 506
37.11 Exercises 507

38 Markov Decision Processes 511

38.1 Problem Set-Up 511
38.2 Optimal Policies and the Bellman Optimality Equation 515
38.3 Finding an Optimal Policy ( ) 518
38.4 Learning in Markov Decision Processes 521
38.5 Upper Confidence Bounds for Reinforcement Learning 522
38.6 Proof of Upper Bound 525
38.7 Proof of Lower Bound 528
38.8 Notes 531
38.9 Bibliographical Remarks 535
38.10 Exercises 537

Bibliography 548

Index 582
Preface

Multi-armed bandits have now been studied for nearly a century. While research
in the beginning was quite meandering, there is now a large community publishing
hundreds of articles every year. Bandit algorithms are also finding their way into
practical applications in industry, especially in on-line platforms where data is
readily available and automation is the only way to scale.
We had hoped to write a comprehensive book, but the literature is now so vast
that many topics have been excluded. In the end we settled on the more modest
goal of equipping our readers with enough expertise to explore the specialised
literature by themselves, and to adapt existing algorithms to their applications.
This latter point is important. Problems in theory are all alike; every application is
different. A practitioner seeking to apply a bandit algorithm needs to understand
which assumptions in the theory are important and how to modify the algorithm
when the assumptions change. We hope this book can provide that understanding.
What is covered in the book is covered in some depth. The focus is on the
mathematical analysis of algorithms for bandit problems, but this is not a
traditional mathematics book, where lemmas are followed by proofs, theorems
and more lemmas. We worked hard to include guiding principles for designing
algorithms and intuition for their analysis. Many algorithms are accompanied by
empirical demonstrations that further aid intuition.
We expect our readers to be familiar with basic analysis and calculus and
some linear algebra. The book uses the notation of measure-theoretic probability
theory, but does not rely on any deep results. A dedicated chapter is included to
introduce the notation and provide intuitions for the basic results we need. This
chapter is unusual for an introduction to measure theory in that it emphasises the
reasons to use σ-algebras beyond the standard technical justifications. We hope
this will convince the reader that measure theory is an important and intuitive
tool. Some chapters use techniques from information theory and convex analysis,
and we devote a short chapter to each.
Most chapters are short and should be readable in an afternoon or presented in
a single lecture. Some components of the book contain content that is not really
about bandits. These can be skipped by knowledgeable readers, or otherwise
referred to when necessary. They are marked with a ( ) because ‘Skippy the
 Preface 2

Kangaroo’ skips things.1 The same mark is used for those parts that contain
useful, but perhaps overly specific information for the first-time reader. Later
parts will not build on these chapters in any substantial way. Most chapters end
with a list of notes and exercises. These are intended to deepen intuition and
highlight the connections between various subsections and the literature. There
is a table of notation at the end of this preface.

Thanks
We’re indebted to our many collaborators and feel privileged that there are
too many of you to name. The University of Alberta, Indiana University and
DeepMind have all provided outstanding work environments and supported the
completion of this book. The book has benefited enormously from the proofreading
efforts of a large number of our friends and colleagues. We are sorry for all the
mistakes introduced after your hard work. Alphabetically, they are: Aaditya
Ramdas, Abbas Mehrabian, Aditya Gopalan, Ambuj Tewari, András György,
Arnoud den Boer, Branislav Kveton, Brendan Patch, Chao Tao, Chao Qin,
Christoph Dann, Claire Vernade, Emilie Kaufmann, Eugene Ji, Gellért Weisz,
Gergely Neu, Johannes Kirschner, Julian Zimmert, Kwang-Sung Jun, Lalit Jain,
Laurent Orseau, Marcus Hutter, Michal Valko, Omar Rivasplata, Pierre Menard,
Ramana Kumar, Roman Pogodin, Ronald Ortner, Ronan Fruit, Ruihao Zhu,
Shuai Li, Toshiyuki Tanaka, Wei Chen, Yoan Russac, Yufei Yi and Zhu Xiaohu.
We are especially grateful to Gábor Balázs and Wouter Koolen, who both read
almost the entire book. Thanks to Lauren Cowels and Cambridge University
Press for providing free books for our proofreaders, tolerating the delays and
for supporting a freely available PDF version. Réka Szepesvári is responsible for
converting some of our primary school figures to their current glory. Last of all,
our families have endured endless weekends of editing and multiple false promises
of ‘done by Christmas’. Rosina and Beáta, it really is done now!

1 Taking inspiration from Tor’s grandfather-in-law, John Dillon [Anderson et al., 1977].
Notation

Some sections are marked with special symbols, which are listed and described
below.

This symbol is a note. Usually this is a remark that is slightly tangential to

the topic at hand.

A warning to the reader.

Something important.

An experiment.

Nomenclature and Conventions

A sequence (an )∞ n=1 is increasing if an+1 ≥ an for all n ≥ 1 and
decreasing if an+1 ≤ an . When the inequalities are strict, we say strictly
increasing/decreasing. The same terminology holds for functions. We will
not be dogmatic about what is the range of argmin/argmax. Sometimes they
return sets, sometimes arbitrary elements of those sets and, where stated, specific
elements of those sets. We will be specific when it is non-obvious/matters. The
infimum of the empty set is inf ∅ = ∞ and the supremum is sup ∅ = −∞. The
P Q
empty sum is i∈∅ ai = 0 and the empty product is i∈∅ ai = 1.

Landau Notation
We make frequent use of the Bachmann–Landau notation. Both were nineteenth
century mathematicians who could have never expected their notation to be
adopted so enthusiastically by computer scientists. Given functions f, g : N →
 Notation 4

[0, ∞), define

f (n)
f (n) = O(g(n)) ⇔ lim sup < ∞,
n→∞ g(n)
f (n)
f (n) = o(g(n)) ⇔ lim = 0,
n→∞ g(n)

f (n)
f (n) = Ω(g(n)) ⇔ lim inf > 0,
n→∞ g(n)

f (n)
f (n) = ω(g(n)) ⇔ lim inf = ∞,
n→∞ g(n)

f (n) = Θ(g(n)) ⇔ f (n) = O(g(n)) and f (n) = Ω(g(n)) .

We make use of the (Bachmann–)Landau notation in two contexts. First, in
proofs where limiting arguments are made, we sometimes write lower-order terms
√ √
using Landau notation. For example, we might write that f (n) = n + o( n), by
√
which we mean that limn→∞ f (n)/ n = 1. In this case we use the mathematical
definitions as envisaged by Bachmann and Landau. The second usage is to
informally describe a result without the clutter of uninteresting constants. For
better or worse, this usage is often a little √
imprecise. For example, we will often
write expressions of the form: Rn = O(m dn). Almost always what is meant
by this is that there exists a universal constant c > 0 (a constant√that does
not depend on either of the quantities involved) such that Rn ≤ cm dn for all
(reasonable) choices of m, d and n. In this context we are careful not to use Landau
notation to hide large lower-order terms. For example, if f (x) = x2 + 10100 x, we
will not write f (x) = O(x2 ), although this would be true.

Bandits
At action in round t
k number of arms/actions
n time horizon
Xt reward in round t
Yt loss in round t
π a policy
ν a bandit
µi mean reward of arm i
Sets
∅ empty set
N, N+ natural numbers, N = {0, 1, 2, . . .} and N+ = N \ {0}
R real numbers
R̄ R ∪ {−∞, ∞}
[n] {1, 2, 3, . . . , n − 1, n}
2A the power set of set A (the set of all subsets of A)
S∞
A∗ set of finite sequences over A, A∗ = i=0 Ai
B2d d-dimensional unit ball, {x ∈ Rd : kxk2 ≤ 1}
Pd probability simplex, {x ∈ [0, 1]d+1 : kxk1 = 1}
 Notation 5

P(A) set of distributions over set A

B(A) Borel σ-algebra on A
[x, y] convex hull of vectors or real values x and y

Functions, Operators and Operations

|A| the cardinality (number of elements) of the finite set A
(x)+ max(x, 0)
a mod b remainder when natural number a is divided by b
bxc, dxe floor and ceiling functions of x
dom(f ) domain of function f
E expectation
V variance
Supp support of distribution or random variable
∇f (x) gradient of f at x
∇v f (x) directional derivative of f at x in direction v
∇2 f (x) Hessian of f at x
∨, ∧ maximum
Rx and minimum, a∨b = max(a, b) and a∧b = min(a, b)
erf(x) √2 exp(−y 2
)dy
π 0
erfc(x) 1 − erf(x) R∞
Γ(z) Gamma function, Γ(z) = 0 xz−1 exp(−x)dx
φA (x) support function φA (x) = supy∈A hx, yi
f ∗
(y) convex conjugate, f ∗ (y) = supx∈A hx, yi − f (x)
n
k binomial coefficient
argmaxx f (x) maximiser or maximisers of f
argminx f (x) minimiser or minimisers of f
Iφ indicator function: converts Boolean φ into binary
IB indicator of set B
D(P, Q) Relative entropy between probability distributions P and Q
d(p, q) Relative entropy between B(p) and B(q)

Linear Algebra
e1 , . . . , e d standard basis vectors of the d-dimensional Euclidean space
0, 1 vectors whose elements are all zeros and all ones, respectively
det(A) determinant of matrix A
trace(A) trace of matrix A
im(A) image of matrix A
ker(A) kernel of matrix A
span(v1 , . . . , vd ) span of vectors v1 , . . . , vd
λmin (G) minimum eigenvalue of matrix G
P
hx, yi inner product, hx, yi = i xi yi
kxkp p-norm of vector x
kxk2G x> Gx for positive definite G ∈ Rd×d and x ∈ Rd
 Notation 6

≺,  Loewner partial order of positive semidefinite matrices: A  B

(A ≺ B) if B − A is positive semidefinite (respectively, definite).
Distributions
N (µ, σ 2 ) Normal distribution with mean µ and variance σ 2
B(p) Bernoulli distribution with mean p
U(a, b) uniform distribution supported on [a, b]
Beta(α, β) Beta distribution with parameters α, β > 0
δx Dirac distribution with point mass at x

Topological
cl(A) closure of set A
int(A) interior of set A
∂A boundary of a set A, ∂A = cl(A) \ int(A)
co(A) convex hull of A
aff(A) affine hull of A
ri(A) relative interior of A
Part I
Bandits, Probability and
Concentration
1 Introduction

Bandit problems were introduced by William R. Thompson in an article

published in 1933 in Biometrika. Thompson was interested in medical
trials and the cruelty of running a trial blindly, without adapting the
treatment allocations on the fly as the drug appears more or less effective.
The name comes from the 1950s, when
Frederick Mosteller and Robert Bush decided
to study animal learning and ran trials on
mice and then on humans. The mice faced
the dilemma of choosing to go left or right
after starting in the bottom of a T-shaped
maze, not knowing each time at which end
they would find food. To study a similar
learning setting in humans, a ‘two-armed
bandit’ machine was commissioned where
humans could choose to pull either the left or
Figure 1.1 Mouse learning a T-maze.
the right arm of the machine, each giving a
random pay-off with the distribution of pay-
offs for each arm unknown to the human player. The machine was called a
‘two-armed bandit’ in homage to the one-armed bandit, an old-fashioned name
for a lever-operated slot machine (‘bandit’ because they steal your money).
There are many reasons to care about bandit problems. Decision-making with
uncertainty is a challenge we all face, and bandits provide a simple model of
this dilemma. Bandit problems also have practical applications. We already
mentioned clinical trial design, which researchers have used to motivate their
work for 80 years. We can’t point to an example where bandits have actually
been used in clinical trials, but adaptive experimental design is gaining popularity
and is actively encouraged by the US Food and Drug Administration, with the
justification that not doing so can lead to the withholding of effective drugs until
long after a positive effect has been established.
While clinical trials are an important application for the future, there are
applications where bandit algorithms are already in use. Major tech companies
use bandit algorithms for configuring web interfaces, where applications include
news recommendation, dynamic pricing and ad placement. A bandit algorithm
 Introduction 9

plays a role in Monte Carlo Tree Search, an algorithm made famous by the recent
success of AlphaGo.
Finally, the mathematical formulation of bandit problems leads to a rich
structure with connections to other branches of mathematics. In writing this
book (and previous papers), we have read books on convex analysis/optimisation,
Brownian motion, probability theory, concentration analysis, statistics, differential
geometry, information theory, Markov chains, computational complexity and more.
What fun!
A combination of all these factors has led to an enormous growth in research
over the last two decades. Google Scholar reports less than 1000, then 2700 and
7000 papers when searching for the phrase ‘bandit algorithm’ for the periods of
2001–5, 2006–10, and 2011–15, respectively, and the trend just seems to have
strengthened since then, with 5600 papers coming up for the period of 2016 to
the middle of 2018. Even if these numbers are somewhat overblown, they are
indicative of a rapidly growing field. This could be a fashion, or maybe there is
something interesting happening here. We think that the latter is true.

A Classical Dilemma
Imagine you are playing a two-armed bandit machine and you already pulled
each lever five times, resulting in the following pay-offs (in dollars):

Round 1 2 3 4 5 6 7 8 9 10

left 0 10 0 0 10

right 10 0 0 0 0

The left arm appears to be doing slightly better. The

average pay-off for this arm is $4, while the average for the Figure 1.2 Two-
right arm is only $2. Let’s say you have 10 more trials (pulls) armed bandit
altogether. What is your strategy? Will you keep pulling
the left arm, ignoring the right? Or would you attribute the poor performance of
the right arm to bad luck and try it a few more times? How many more times?
This illustrates one of the main interests in bandit problems. They capture the
fundamental dilemma a learner faces when choosing between uncertain options.
Should one explore an option that looks inferior or exploit by going with the
option that looks best currently? Finding the right balance between exploration
and exploitation is at the heart of all bandit problems.
 1.1 The Language of Bandits 10

1.1 The Language of Bandits

A bandit problem is a sequential game between a learner and an environment.

The game is played over n rounds, where n is a positive natural number called
the horizon. In each round t ∈ [n], the learner first chooses an action At from a
given set A, and the environment then reveals a reward Xt ∈ R.

In the literature, actions are often also called ‘arms’. We talk about k-armed
bandits when the number of actions is k, and about multi-armed bandits
when the number of arms is at least two and the actual number is immaterial
to the discussion. If there are multi-armed bandits, there are also one-armed
bandits, which are really two-armed bandits where the pay-off of one of the
arms is a known fixed deterministic number.

Of course the learner cannot peek into the future when choosing their
actions, which means that At should only depend on the history Ht−1 =
(A1 , X1 , . . . , At−1 , Xt−1 ). A policy is a mapping from histories to actions: A
learner adopts a policy to interact with an environment. An environment is a
mapping from history sequences ending in actions to rewards. Both the learner
and the environment may randomise their decisions, but this detail is not so
important for now. The most common objective of the learner is to choose actions
that lead to the largest possible cumulative reward over all n rounds, which is
Pn
t=1 Xt .
The fundamental challenge in bandit problems is that the environment is
unknown to the learner. All the learner knows is that the true environment
lies in some set E called the environment class. Most of this book is about
designing policies for different kinds of environment classes, though in some cases
the framework is extended to include side observations as well as actions and
rewards.
The next question is how to evaluate a learner. We discuss several performance
measures throughout the book, but most of our efforts are devoted to
understanding the regret. There are several ways to define this quantity. To avoid
getting bogged down in details, we start with a somewhat informal definition.

Definition 1.1. The regret of the learner relative to a policy π (not necessarily
that followed by the learner) is the difference between the total expected reward
using policy π for n rounds and the total expected reward collected by the learner
over n rounds. The regret relative to a set of policies Π is the maximum regret
relative to any policy π ∈ Π in the set.

The set Π is often called the competitor class. Another way of saying all this
is that the regret measures the performance of the learner relative to the best
policy in the competitor class. We usually measure the regret relative to a set of
policies Π that is large enough to include the optimal policy for all environments
 1.1 The Language of Bandits 11

in E. In this case, the regret measures the loss suffered by the learner relative to
the optimal policy.
Example 1.2. Suppose the action set is A = {1, 2, . . . , k}. An environment is
called a stochastic Bernoulli bandit if the reward Xt ∈ {0, 1} is binary valued
and there exists a vector µ ∈ [0, 1]k such that the probability that Xt = 1 given
the learner chose action At = a is µa . The class of stochastic Bernoulli bandits is
the set of all such bandits, which are characterised by their mean vectors. If you
knew the mean vector associated with the environment, then the optimal policy
is to play the fixed action a∗ = argmaxa∈A µa . This means that for this problem
the natural competitor class is the set of k constant polices Π = {π1 , . . . , πk },
where πi chooses action i in every round. The regret over n rounds becomes
" n #
X
Rn = n max µa − E Xt ,
a∈A
t=1

where the expectation is with respect to the randomness in the environment and
policy. The first term in this expression is the maximum expected reward using
any policy. The second term is the expected reward collected by the learner.
For a fixed policy and competitor class, the regret depends on the environment.
The environments where the regret is large are those where the learner is behaving
worse. Of course the ideal case is that the regret be small for all environments.
The worst-case regret is the maximum regret over all possible environments.
One of the core questions in the study of bandits is to understand the growth
rate of the regret as n grows. A good learner achieves sublinear regret. Letting Rn
denote the regret over n rounds, this means that Rn = o(n) or equivalently that
limn→∞ Rn /n = 0. Of course one can ask for more. Under what circumstances is
√
Rn = O( n) or Rn = O(log(n))? And what are the leading constants? How does
the regret depend on the specific environment in which the learner finds itself?
We will discover eventually that for the environment class in Example 1.2, the
√
worst-case regret for any policy is at least Ω( n) and that there exist policies for
√
which Rn = O( n).

A large environment class corresponds to less knowledge by the learner. A

large competitor class means the regret is a more demanding criteria. Some
care is sometimes required to choose these sets appropriately so that (a)
guarantees on the regret are meaningful and (b) there exist policies that
make the regret small.

The framework is general enough to model almost anything by using a rich

enough environment class. This cannot be bad, but with too much generality it
becomes impossible to say much. For this reason, we usually restrict our attention
to certain kinds of environment classes and competitor classes.
A simple problem setting is that of stochastic stationary bandits. In this
 1.1 The Language of Bandits 12

case the environment is restricted to generate the reward in response to each action
from a distribution that is specific to that action and independent of the previous
action choices and rewards. The environment class in Example 1.2 satisfies these
conditions, but there are many alternatives. For example, the rewards could follow
a Gaussian distribution rather than Bernoulli. This relatively mild difference does
not change the nature of the problem in a significant way. A more drastic change
is to assume the action set A is a subset of Rd and that the mean reward for
choosing some action a ∈ A follows a linear model, Xt = ha, θi + ηt for θ ∈ Rd
and ηt a standard Gaussian (zero mean, unit variance). The unknown quantity
in this case is θ, and the environment class corresponds to its possible values
(E = Rd ).
For some applications, the assumption that the rewards are stochastic and
stationary may be too restrictive. The world mostly appears deterministic, even
if it is hard to predict and often chaotic looking. Of course, stochasticity has
been enormously successful in explaining patterns in data, and this may be
sufficient reason to keep it as the modelling assumption. But what if the stochastic
assumptions fail to hold? What if they are violated for a single round? Or just for
one action, at some rounds? Will our best algorithms suddenly perform poorly?
Or will the algorithms developed be robust to smaller or larger deviations from
the modelling assumptions?
An extreme idea is to drop all assumptions on how the rewards are generated,
except that they are chosen without knowledge of the learner’s actions and lie
in a bounded set. If these are the only assumptions, we get what is called the
setting of adversarial bandits. The trick to say something meaningful in this
setting is to restrict the competitor class. The learner is not expected to find
the best sequence of actions, which may be like finding a needle in a haystack.
Instead, we usually choose Π to be the set of constant policies and demand that
the learner is not much worse than any of these. By defining the regret in this
way, the stationarity assumption is transported into the definition of regret rather
than constraining the environment.
Of course there are all shades of grey between these two extremes. Sometimes
we consider the case where the rewards are stochastic, but not stationary. Or
one may analyse the robustness of an algorithm for stochastic bandits to small
adversarial perturbations. Another idea is to isolate exactly which properties of
the stochastic assumption are really exploited by a policy designed for stochastic
bandits. This kind of inverse analysis can help explain the strong performance of
policies when facing environments that clearly violate the assumptions they were
designed for.

1.1.1 Other Learning Objectives

We already mentioned that the regret can be defined in several ways, each
capturing slightly different aspects of the behaviour of a policy. Because the
regret depends on the environment, it becomes a multi-objective criterion: ideally,
 1.2 Applications 13

we want to keep the regret small across all possible environments. One way to
convert a multi-objective criterion into a single number is to take averages. This
corresponds to the Bayesian viewpoint where the objective is to minimise the
average cumulative regret with respect to a prior on the environment class.
Maximising the sum of rewards is not always the objective. Sometimes the
learner just wants to find a near-optimal policy after n rounds, but the actual
rewards accumulated over those rounds are unimportant. We will see examples
of this shortly.

1.1.2 Limitations of the Bandit Framework

One of the distinguishing features of all bandit problems studied in this book
is that the learner never needs to plan for the future. More precisely, we will
invariably make the assumption that the learner’s available choices and rewards
tomorrow are not affected by their decisions today. Problems that do require
this kind of long-term planning fall into the realm of reinforcement learning,
which is the topic of the final chapter. Another limitation of the bandit framework
is the assumption that the learner observes the reward in every round. The setting
where the reward is not observed is called partial monitoring and is the topic
of Chapter 37. Finally, often, the environment itself consists of strategic agents,
which the learner needs to take into account. This problem is studied in game
theory and would need a book on its own.

1.2 Applications

After this short preview, and as an appetiser before the hard work, we briefly
describe the formalisations of a variety of applications.

A/B Testing
The designers of a company website are trying to decide whether the ‘buy it now’
button should be placed at the top of the product page or at the bottom. In
the old days, they would commit to a trial of each version by splitting incoming
users into two groups of 10 000. Each group would be shown a different version
of the site, and a statistician would examine the data at the end to decide which
version was better. One problem with this approach is the non-adaptivity of the
test. For example, if the effect size is large, then the trial could be stopped early.
One way to apply bandits to this problem is to view the two versions of the
site as actions. Each time t a user makes a request, a bandit algorithm is used
to choose an action At ∈ A = {SiteA, SiteB}, and the reward is Xt = 1 if the
user purchases the product and Xt = 0 otherwise.
 1.2 Applications 14

In traditional A/B testing, the objective of the statistician is to decide which

website is better. When using a bandit algorithm, there is no need to end
the trial. The algorithm automatically decides when one version of the site
should be shown more often than another. Even if the real objective is to
identify the best site, then adaptivity or early stopping can be added to the
A/B process using techniques from bandit theory. While this is not the focus
of this book, some of the basic ideas are explained in Chapter 33.

Advert Placement
In advert placement, each round corresponds to a user visiting a website, and
the set of actions A is the set of all available adverts. One could treat this as
a standard multi-armed bandit problem, where in each round a policy chooses
At ∈ A, and the reward is Xt = 1 if the user clicked on the advert and Xt = 0
otherwise. This might work for specialised websites where the adverts are all
likely to be appropriate. But for a company like Amazon, the advertising should
be targeted. A user that recently purchased rock-climbing shoes is much more
likely to buy a harness than another user. Clearly an algorithm should take this
into account.
The standard way to incorporate this additional knowledge is to use the
information about the user as context. In its simplest formulation, this might
mean clustering users and implementing a separate bandit algorithm for each
cluster. Much of this book is devoted to the question of how to use side information
to improve the performance of a learner.
This is a good place to emphasise that the world is messy. The set of available
adverts is changing from round to round. The feedback from the user can be
delayed for many rounds. Finally, the real objective is rarely just to maximise
clicks. Other metrics such as user satisfaction, diversity, freshness and fairness,
just to mention a few, are important too. These are the kinds of issues that make
implementing bandit algorithms in the real world a challenge. This book will not
address all these issues in detail. Instead we focus on the foundations and hope
this provides enough understanding that you can invent solutions for whatever
peculiar challenges arise in your problem.

Recommendation Services
Netflix has to decide which movies to place most prominently in your ‘Browse’
page. Like in advert placement, users arrive at the page sequentially, and the
reward can be measured as some function of (a) whether or not you watched a
movie and (b) whether or not you rated it positively. There are many challenges.
First of all, Netflix shows a long list of movies, so the set of possible actions
is combinatorially large. Second, each user watches relatively few movies, and
individual users are different. This suggests approaches such as low-rank matrix
factorisation (a popular approach in ‘collaborative filtering’). But notice this is
 1.2 Applications 15

not an offline problem. The learning algorithm gets to choose what users see and
this affects the data. If the users are never recommended the AlphaGo movie,
then few users will watch it, and the amount of data about this film will be
scarce.

Network Routing
Another problem with an interesting structure is network routing, where the
learner tries to direct internet traffic through the shortest path on a network. In
each round the learner receives the start/end destinations for a packet of data.
The set of actions is the set of all paths starting and ending at the appropriate
points on some known graph. The feedback in this case is the time it takes for
the packet to be received at its destination, and the reward is the negation of
this value. Again the action set is combinatorially large. Even relatively small
graphs have an enormous number of paths. The routing problem can obviously
be applied to more physical networks such as transportation systems used in
operations research.

Dynamic Pricing
In dynamic pricing, a company is trying to automatically optimise the price of
some product. Users arrive sequentially, and the learner sets the price. The user
will only purchase the product if the price is lower than their valuation. What
makes this problem interesting is (a) the learner never actually observes the
valuation of the product, only the binary signal that the price was too low/too
high, and (b) there is a monotonicity structure in the pricing. If a user purchased
an item priced at $10, then they would surely purchase it for $5, but whether or
not it would sell when priced at $11 is uncertain. Also, the set of possible actions
is close to continuous.

Waiting Problems
Every day you travel to work, either by bus or by walking. Once you get on the
bus, the trip only takes 5 minutes, but the timetable is unreliable, and the bus
arrival time is unknown and stochastic. Sometimes the bus doesn’t come at all.
Walking, on the other hand, takes 30 minutes along a beautiful river away from
the road. The problem is to devise a policy for choosing how long to wait at
the bus stop before giving up and walking to minimise the time to get to your
workplace. Walk too soon, and you miss the bus and gain little information. But
waiting too long also comes at a price.
While waiting for a bus is not a problem we all face, there are other applications
of this setting. For example, deciding the amount of inactivity required before
putting a hard drive into sleep mode or powering off a car engine at traffic lights.
The statistical part of the waiting problem concerns estimating the cumulative
distribution function of the bus arrival times from data. The twist is that the
data is censored on the days you chose to walk before the bus arrived, which
is a problem analysed in the subfield of statistics called survival analysis. The
 1.3 Notes 16

interplay between the statistical estimation problem and the challenge of balancing
exploration and exploitation is what makes this and the other problems studied
in this book interesting.

Resource Allocation
A large part of operations research is focussed on designing strategies for allocating
scarce resources. When the dynamics of demand or supply are uncertain, the
problem has elements reminiscent of a bandit problem. Allocating too few
resources reveals only partial information about the true demand, but allocating
too many resources is wasteful. Of course, resource allocation is broad, and many
problems exhibit structure that is not typical of bandit problems, like the need
for long-term planning.

Tree Search
The UCT algorithm is a tree search algorithm commonly used in perfect-
information game-playing algorithms. The idea is to iteratively build a search
tree where in each iteration the algorithm takes three steps: (1) chooses a path
from the root to a leaf; (2) expands the leaf (if possible); (3) performs a Monte
Carlo roll-out to the end of the game. The contribution of a bandit algorithm is in
selecting the path from the root to the leaves. At each node in the tree, a bandit
algorithm is used to select the child based on the series of rewards observed
through that node so far. The resulting algorithm can be analysed theoretically,
but more importantly has demonstrated outstanding empirical performance in
game-playing problems.

1.3 Notes

1 The reader may find it odd that at one point we identified environments with
maps from histories to rewards, while we used the language that a learner
‘adopts a policy’ (a map from histories to actions). The reason is part historical
and part because policies and their design are at the center of the book, while
the environment strategies will mostly be kept fixed (and relatively simple).
On this note, strategy is also a word that sometimes used interchangeably with
policy.

1.4 Bibliographic Remarks

As we mentioned in the very beginning, the first paper on bandits was by

Thompson [1933]. The experimentation on mice and humans that led to the
name comes from the paper by Bush and Mosteller [1953]. Much credit for the
popularisation of the field must go to famous mathematician and statistician,
Herbert Robbins, whose name appears on many of the works that we reference,
 1.4 Bibliographic Remarks 17

with the earliest being: [Robbins, 1952]. Another early pioneer is Herman Chernoff,
who wrote papers with titles like ‘Sequential Decisions in the Control of a
Spaceship’ [Bather and Chernoff, 1967].
Besides these seminal papers, there are already a number of books on bandits
that may serve as useful additional reading. The most recent (and also most
related) is by Bubeck and Cesa-Bianchi [2012] and is freely available online. This is
an excellent book and is warmly recommended. The main difference between their
book and ours is that (a) we have the benefit of seven years of additional research
in a fast-moving field and (b) our longer page limit permits more depth. Another
relatively recent book is Prediction, Learning and Games by Cesa-Bianchi and
Lugosi [2006]. This is a wonderful book, and quite comprehensive. But its scope
is ‘all of’ online learning, which is so broad that bandits are not covered in great
depth. We should mention there is also a recent book on bandits by Slivkins
[2019]. Conveniently it covers some topics not covered in this book (notably
Lipschitz bandits and bandits with knapsacks). The reverse is also true, which
should not be surprising since our book is currently 400 pages longer. There are
also four books on sequential design and multi-armed bandits in the Bayesian
setting, which we will address only a little. These are based on relatively old
material, but are still useful references for this line of work and are well worth
reading [Chernoff, 1959, Berry and Fristedt, 1985, Presman and Sonin, 1990,
Gittins et al., 2011].
Without trying to be exhaustive, here are a few articles applying bandit
algorithms; a recent survey is by Bouneffouf and Rish [2019]. The papers
themselves will contain more useful pointers to the vast literature. We mentioned
AlphaGo already [Silver et al., 2016]. The tree search algorithm that drives its
search uses a bandit algorithm at each node [Kocsis and Szepesvári, 2006]. Le et al.
[2014] apply bandits to wireless monitoring, where the problem is challenging
due to the large action space. Lei et al. [2017] design specialised contextual
bandit algorithms for just-in-time adaptive interventions in mobile health: in
the typical application the user is prompted with the intention of inducing a
long-term beneficial behavioural change. See also the article by Greenewald et al.
[2017]. Rafferty et al. [2018] apply Thompson sampling to educational software
and note the trade-off between knowledge and reward. Sadly, by 2015, bandit
algorithms still have not been used in clinical trials, as explicitly mentioned
by Villar et al. [2015]. Microsoft offers a ‘Decision Service’ that uses bandit
algorithms to automate decision-making [Agarwal et al., 2016].
2 Foundations of Probability ( )

This chapter covers the fundamental concepts of measure-theoretic probability,

on which the remainder of this book relies. Readers familiar with this topic can
safely skip the chapter, but perhaps a brief reading would yield some refreshing
perspectives. Measure-theoretic probability is often viewed as a necessary evil,
to be used when a demand for rigour combined with continuous spaces breaks
the simple approach we know and love from high school. We claim that measure-
theoretic probability offers more than annoying technical machinery. In this
chapter we attempt to prove this by providing a non-standard introduction.
Rather than a long list of definitions, we demonstrate the intuitive power of
the notation and tools. For those readers with little prior experience in measure
theory this chapter will no doubt be a challenging read. We think the investment
is worth the effort, but a great deal of the book can be read without it, provided
one is willing to take certain results on faith.

2.1 Probability Spaces and Random Elements

The thrill of gambling comes from the fact that the bet is placed on future
outcomes that are uncertain at the time of the gamble. A central question in
gambling is the fair value of a game. This can be difficult to answer for all but
the simplest games. As an illustrative example, imagine the following moderately
complex game: I throw a dice. If the result is four, I throw two more dice; otherwise
I throw one dice only. Looking at each newly thrown dice (one or two), I repeat
the same, for a total of three rounds. Afterwards, I pay you the sum of the values
on the faces of the dice. How much are you willing to pay to play this game with
me?
Many examples of practical interest exhibit a complex random interdependency
between outcomes. The cornerstone of modern probability as proposed by
Kolmogorov aims to remove this complexity by separating the randomness from
the mechanism that produces the outcome.
Instead of rolling the dice one by one, imagine that sufficiently many dice were
rolled before the game has even started. For our game we need to roll seven
dice, because this is the maximum number that might be required (one in the
first round, two in the second round and four in the third round. See Fig. 2.1).
 2.1 Probability Spaces and Random Elements 19

X1 := throw()

No Yes
X1 = 4?

X21 := throw() X21 := throw() X22 := throw()

Figure 2.1 The initial phase of a gambling game with a random number of dice rolls.
Depending on the outcome of a dice roll, one or two dice are rolled for a total of three
rounds. The number of dice used will then be random in the range of three to seven.

After all the dice are rolled, the game can be emulated by ordering the dice and
revealing the outcomes sequentially. Then the value of the first dice in the chosen
ordering is the outcome of the dice in the first round. If we see a four, we look at
the next two dice in the ordering; otherwise we look at the single next dice.
By taking this approach, we get a simple calculus for the probabilities of all
kinds of events. Rather than directly calculating the likelihood of each pay-off,
we first consider the probability of any single outcome of the dice. Since there
are seven dice, the set of all possible outcomes is Ω = {1, . . . , 6}7 . Because
all outcomes are equally probable, the probability of any ω ∈ Ω is (1/6)7 . The
probability of the game pay-off taking value v can then be evaluated by calculating
the total probability assigned to all those outcomes ω ∈ Ω that would result
in the value of v. In principle, this is trivial to do thanks to the separation of
everything that is probabilistic from the rest. The set Ω is called the outcome
space, and its elements are the outcomes. Fig. 2.2 illustrates this idea. Random
outcomes are generated on the left, while on the right, various mechanisms are
used to arrive at values; some of these values may be observed and some not.
There will be much benefit from being a little more formal about how we
come up with the value of our artificial game. For this, note that the process by
which the game gets its value is a function X that maps Ω to the reals (simply,
X : Ω → R). We find it ironic that functions of this type (from the outcome
space to subsets of the reals) are called random variables. They are neither
random nor variables in a programming language sense. The randomness is in
the argument that X is acting on, producing randomly changing results. Later
we will put a little more structure on random variables, but for now it suffices to
think of them as maps from the outcome space to the reals.
 2.1 Probability Spaces and Random Elements 20

Outcomes

Randomising device
Mechanisms
all randomness

Figure 2.2 A key idea in probability theory is the separation of sources of randomness
from game mechanisms. A mechanism creates values from the elementary random
outcomes, some of which are visible for observers, while others may remain hidden.

We follow the standard convention in probability theory where random

variables are denoted by capital letters. Be warned that capital letters are
also used for other purposes as demanded by different conventions.

Pick some number v ∈ N. What is the probability of seeing X = v? As

described above, this probability is (1/6)7 times the size of the set X −1 (v) =
{ω ∈ Ω : X(ω) = v}. The set X −1 (v) is called the preimage of v under X. More
generally, the probability that X takes its value in some set A ⊆ N is given by
(1/6)7 times the cardinality of X −1 (A) = {ω ∈ Ω : X(ω) ∈ A}, where we have
overloaded the definition of X −1 to set-valued inputs.
Notice in the previous paragraph we only needed probabilities assigned to
subsets of Ω, regardless of the question asked. To make this a bit more general,
let us introduce a map P that assigns probabilities to certain subsets of Ω. The
intuitive meaning of P is as follows. Random outcomes are generated in Ω. The
probability that an outcome falls into a set A ⊂ Ω is P (A). If A is not in the
domain of P, then there is no answer to the question of the probability of the
outcome falling in A. But let’s postpone the discussion of why P should be
restricted to only certain subsets of Ω later. In the above example with the dice,
the set of subsets in the domain of P is not restricted and, in particular, for any
subset A ⊆ Ω, P (A) = (1/6)7 |A|.


The probability of seeing X taking the value of v is thus P X −1 (v) . To
minimise clutter, the more readable notation for this is P (X = v). But
always
keep in mind that this familiar form is just a shorthand for P X −1 (v) . More
generally, we also use

P (predicate(U, V, . . . )) = P ({ω ∈ Ω : predicate(U (ω), V (ω), . . . ) is true})

 2.1 Probability Spaces and Random Elements 21

with any predicate (an expression evaluating to true or false) where U, V, . . . are
functions with domain Ω.
What properties should P satisfy? Since Ω is the set of all possible outcomes,
it seems reasonable to expect that P is defined for Ω and P(Ω) = 1 and since ∅
contains no outcomes, P(∅) = 0 is also expected to hold. Furthermore, probabilities
should be non-negative so P(A) ≥ 0 for any A ⊂ Ω on which P is defined. Let
Ac = Ω \ A be the complement of A. Then we should expect that P is defined
for A exactly when it is defined for Ac and P(Ac ) = 1 − P(A) (negation rule).
Finally, if A, B are disjoint so that A ∩ B = ∅ and P(A), P(B) and P(A ∪ B) are
all defined, then P(A ∪ B) = P(A) + P(B). This is called the finite additivity
property.
Let F be the set of subsets of Ω on which P is defined. It would seem silly if
A ∈ F and Ac ∈ / F, since P(Ac ) could simply be defined by P(Ac ) = 1 − P(A).
Similarly, if P is defined on disjoint sets A and B, then it makes sense if A∪B ∈ F.
We will also require the additivity property to hold (i) regardless of whether
the sets are disjoint and (ii) even for countably infinitely many sets. If {Ai }i
is a collection of sets and Ai ∈ F for all i ∈ N, then ∪i Ai ∈ F, and if these
P
sets are pairwise disjoint, P(∪i Ai ) = i P(Ai ). A set of subsets that satisfies all
these properties is called a σ-algebra, which is pronounced ‘sigma-algebra’ and
sometimes also called a σ-field (see Note 1).
Definition 2.1 (σ-algebra and probability measures). A set F ⊆ 2Ω is a σ-
algebra if Ω ∈ F and Ac ∈ F for all A ∈ F and ∪i Ai ∈ F for all {Ai }i with
Ai ∈ F for all i ∈ N. That is, it should include the whole outcome space and
be closed under complementation and countable unions. A function P : F → R
is a probability measure if P(Ω) = 1 and for all A ∈ F, P(A) ≥ 0 and
P
P(Ac ) = 1 − P(A) and P(∪i Ai ) = i P(Ai ) for all countable collections of disjoint
sets {Ai }i with Ai ∈ F for all i. If F is a σ-algebra and G ⊂ F is also a σ-algebra,
then we say G is a sub-σ-algebra of F. If P is a measure defined on F, then
the restriction of P to G is a measure P|G on G defined by P|G (A) = P(A) for
all A ∈ G.
At this stage, the reader may rightly wonder about why we introduced the notion
of sub-σ-algebras. The answer should become clear quite soon. The elements
of F are called measurable sets. They are measurable in the sense that P
assigns values to them. The pair (Ω, F) alone is called a measurable space,
while the triplet (Ω, F, P) is called a probability space. If the condition that
P(Ω) = 1 is lifted, then P is called a measure. If the condition that P(A) ≥ 0
is also lifted, then P is called a signed measure. For measures and signed
measures, it would be unusual to use the symbol P, which is mostly reserved for
probabilities. Probability measures are also called probability distributions,
or just distributions.
Random variables lead to new probability

measures. In particular, in the
example above PX (A) = P X −1 (A)
is a probability measure defined for all the
subsets A of R for which P X −1 (A) is defined. More generally, for a random
 2.1 Probability Spaces and Random Elements 22

variable X, the probability measure PX is called the law of X, or the push-

forward measure of P under X.

The significance of the push-forward measure PX is that any probabilistic

question concerning X can be answered from the knowledge of PX alone.
Even Ω and the details of the map X are not needed. This is often used as
an excuse to not even mention the underlying probability space (Ω, F, P).

If we keep X fixed but change P (for example, by switching to loaded dice),

then the measure induced by X changes. We will often use arguments that do
exactly this, especially when proving lower bounds on the limits of how well
bandit algorithms can perform.
The astute reader would have noticed that we skipped over some details.
Measures are defined as functions from a σ-algebra to R, so if we want to call
PX a measure, then its domain {A ⊂ R : X −1 (A) ∈ F} better be a σ-algebra.
This holds in great generality. You will show in Exercise 2.3 that for functions
X : Ω → X with X arbitrary, the collection {A ⊂ X : X −1 (A) ∈ F} is a
σ-algebra.
It will be useful to generalise our example a little by allowing X to take on
values in sets other than the reals. For example, the range could be vectors
or abstract objects like sequences. Let (Ω, F) be a measurable space, X be an
arbitrary set and G ⊆ 2X . A function X : Ω → X is called an F /G-measurable
map if X −1 (A) ∈ F for all A ∈ G. Note that G need not be a σ-algebra.
When F and G are obvious from the context, X is called a measurable map.
What are the typical choices for G? When X is real-valued, it is usual to let
G = {(a, b) : a < b with a, b ∈ R} be the set of all open intervals. The reader can
verify that if X is F/G-measurable, then it is also F/σ(G)-measurable, where
σ(G) is the smallest σ-algebra that contains G. This smallest σ-algebra can be
shown to exist. Furthermore, it contains exactly those sets A that are in every
σ-algebra that contains G (see Exercise 2.5). When G is the set of open intervals,
σ(G) is usually denoted by B or B(R) and is called the Borel σ-algebra of R.
This definition is extended to Rk by replacing open intervals with open rectangles
Qk
of the form i=1 (ai , bi ), where a < b ∈ Rk . If G is the set of all such open
rectangles, then σ(G) is the Borel σ-algebra: B(Rk ). More generally, the Borel
σ-algebra of a topological space X is the σ-algebra generated by the open sets of
X.

Definition 2.2 (Random variables and elements). A random variable

(random vector) on measurable space (Ω, F) is a F/B(R)-measurable function
X : Ω → R (respectively F/B(Rk )-measurable function X : Ω → Rk ). A random
element between measurable spaces (Ω, F) and (X , G) is a F/G-measurable
function X : Ω → X .

Thus, random vectors are random elements where the range space is
 2.1 Probability Spaces and Random Elements 23

(Rk , B(Rk )), and random vectors are random variables when k = 1. Random
elements generalise random variables and vectors to functions that do not
take values in Rk . The push-forward measure (or law) can be defined for
any random element. Furthermore, random variables and vectors work nicely
together. If X1 , . . . , Xk are k random variables on the same domain (Ω, F),
then X(ω) = (X1 (ω), . . . , Xk (ω)) is an Rk -valued random vector, and vice versa
(Exercise 2.2). Multiple random variables X1 , . . . , Xk from the same measurable
space can thus be viewed as a random vector X = (X1 , . . . , Xk ).
Given a map X : Ω → X between measurable spaces (Ω, F) and (X , G), we let
σ(X) = {X −1 (A) : A ∈ G} be the σ-algebra generated by X. The map X is
F/G-measurable if and only if σ(X) ⊆ F. By checking the definitions one can
show that σ(X) is a sub-σ-algebra of F and in fact is the smallest sub-σ-algebra
for which X is measurable. If G = σ(A) itself is generated by a set system
A ⊂ 2X , then to check the F/G-measurability of X, it suffices to check whether
X −1 (A) = {X −1 (A) : A ∈ A} is a subset of F. The reason this is sufficient is
because σ(X −1 (A)) = X −1 (σ(A)), and by definition the latter is σ(X). In fact,
to check whether a map is measurable, either one uses the composition rule or
checks X −1 (A) ⊂ F for a ‘generator’ A of G.
Random elements can be combined to produce new random elements by
composition. One can show that if f is F/G-measurable and g is G/H-measurable
for σ-algebras F, G and H over appropriate spaces, then their composition g ◦ f
is F/H-measurable (Exercise 2.1). This is used most often for Borel functions,
which is a special name for B(Rm )/B(Rn )-measurable functions from Rm to
Rn . These functions are also called Borel measurable. The reader will find it
pleasing that all familiar functions are Borel. First and foremost, all continuous
functions are Borel, which includes elementary operations such as addition and
multiplication. Continuity is far from essential, however. In fact one is hard-
pressed to construct a function that is not Borel. This means the usual operations
are ‘safe’ when working with random variables.

Indicator Functions
Given an arbitrary set Ω and A ⊆ Ω, the indicator function of A is
IA : Ω → {0, 1} given by
(
1 , if ω ∈ A ;
IA (ω) =
0 , otherwise .

Sometimes A has a complicated description, and it becomes convenient to abuse

notation by writing I {ω ∈ A} instead of IA (ω). Similarly, we will often write
I {predicate(X, Y, . . .)} to mean the indicator function of the subset of Ω on
which the predicate is true. It is easy to check that an indicator function IA is a
random variable on (Ω, F) if and only if A is measurable: A ∈ F.
 2.1 Probability Spaces and Random Elements 24

Why So Complicated?
You may be wondering why we did not define P on the power set of Ω, which
is equivalent to declaring that all sets are measurable. In many cases this is a
perfectly reasonable thing to do, including the example game where nothing
prevents us from defining F = 2Ω . However, beyond this example, there are two
justifications not to have F = 2Ω , the first technical and the second conceptual.
The technical reason is highlighted by the following surprising theorem
according to which there does not exist a uniform probability distribution on
Ω = [0, 1] if F is chosen to be the power set of Ω (a uniform probability distribution
over [0, 1], if existed, would have the property of assigning its length to every
interval). In other words, if you want to be able to define the uniform measure,
then F cannot be too large. By contrast, the uniform measure can be defined
over the Borel σ-algebra, though proving this is not elementary.
Theorem 2.3. Let Ω = [0, 1], and F be the power set of Ω. Then there does not
exist a measure P on (Ω, F) such that P([a, b]) = b − a for all 0 ≤ a ≤ b ≤ 1.
The main conceptual reason of why not to have F = 2Ω is because then we
can use σ-algebras to represent information. This is especially useful in the study
of bandits where the learner is interacting with an environment and is slowly
gaining knowledge. One useful way to represent this is by using a sequence of
nested σ-algebras, as we explain in the next section. One might also be worried
that the Borel σ-algebra does not contain enough measurable sets. Rest assured
that this is not a problem and you will not easily find a non-measurable set. For
completeness, an example of a non-measurable set will still be given in the notes,
along with a little more discussion on this topic.
A second technical reason to prefer the measure-theoretic approach to
probabilities is that this approach allows for the unification of distributions
on discrete spaces and densities on continuous ones (the uninitiated reader will
find the definitions of these later). This unification can be necessary when dealing
with random variables that combine elements of both, e.g. a random variable
that is zero with probability 1/2 and otherwise behaves like a standard Gaussian.
Random variables like this give rise to so-called “mixed continuous and discrete
distributions”, which seem to require special treatment in a naive approach
to probabilities, yet dealing with random variables like these are nothing but
ordinary under the measure-theoretic approach.

From Laws to Probability Spaces and Random Variables

A big ‘conspiracy’ in probability theory is that probability spaces are seldom
mentioned in theorem statements, despite the fact that a measure cannot be
defined without one. Statements are instead given in terms of random elements
and constraints on their joint probabilities. For example, suppose that X and Y
are random variables such that
|A ∩ [6]| |B ∩ [2]|
P (X ∈ A, Y ∈ B) = · for all A, B ∈ B(R) , (2.1)
6 2
 2.1 Probability Spaces and Random Elements 25

which represents the joint distribution for the values of a dice (X ∈ [6]) and coin
(Y ∈ [2]). The formula describes some constraints on the probabilistic interactions
between the outputs of X and Y , but says nothing about their domain. In a way,
the domain is an unimportant detail. Nevertheless, one must ask whether or not
an appropriate domain exists at all. More generally, one may ask whether an
appropriate probability space exists given some constraints on the joint law of a
collection X1 , . . . , Xk of random variables. For this to make sense, the constraints
should not contradict each other, which means there is a probability measure
µ on B(Rk ) such that µ satisfies the postulated constraints. But then we can
choose Ω = Rk , F = B(Rk ), P = µ and Xi : Ω → R to be the ith coordinate
map: Xi (ω) = ωi . The push-forward of P under X = (X1 , . . . , Xk ) is µ, which by
definition is compatible with the constraints.
A more specific question is whether for a particular set of constraints on the
joint law there exists a measure µ compatible with the constraints. Very often the
constraints are specified for elements of the cartesian product of finitely many
σ-algebras, like in Eq. (2.1). If (Ω1 , F1 ), . . . , (Ωn , Fn ) are measurable spaces, then
the cartesian product of F1 , . . . Fn is
F1 × · · · × Fn = {A1 × · · · × An : A1 ∈ F1 , . . . , An ∈ Fn } ⊆ 2Ω1 ×···×Ωn .
Elements of this set are known as measurable rectangles in Ω1 × · · · × Ωn .
Theorem 2.4 (Carathéodory’s extension theorem). Let (Ω1 , F1 ), . . . , (Ωn , Fn )
be measurable spaces and µ̄ : F1 × · · · × Fn → [0, 1] be a function such that
(a) µ̄(Ω1 × · · · × Ωn ) = 1; and
P∞
k=1 Ak ) =
(b) µ̄(∪∞ k=1 µ̄(Ak ) for all sequences of disjoint sets with Ak ∈
F1 × · · · × Fn .
Let Ω = Ω1 × · · · × Ωn and F = σ(F1 × · · · × Fn ). Then there exists a unique
probability measure µ on (Ω, F) such that µ agrees with µ̄ on F1 × · · · × Fn .
The theorem is applied by letting Ωk = R and Fk = B(R). Then the values of
a measure on all cartesian products uniquely determines its value everywhere.

It is not true that F1 ×F2 = σ(F1 ×F2 ). Take, for example, F1 = F2 = 2{1,2} .
Then, |F1 × F2 | = 1 + 3 × 3 = 10 (because ∅ × X = ∅), while, since
F1 × F2 includes the singletons of 2{1,2}×{1,2} , σ(F1 × F2 ) = 2{1,2}×{1,2} .
Hence, six sets are missing from F1 × F2 . For example, {(1, 1), (2, 2)} ∈
σ(F1 × F2 ) \ F1 × F2 .

The σ-algebra σ(F1 × · · · × Fn ) is called the product σ-algebra of (Fk )k∈[n]

and is also denoted by F1 ⊗ · · · ⊗ Fn . The product operation turns out to be
associative: (F1 ⊗ F2 ) ⊗ F3 = F1 ⊗ (F2 ⊗ F3 ), which justifies writing F1 ⊗ F2 ⊗ F3 .
As it turns out, things work out well again with Borel σ-algebras: for p, q ∈ N+ ,
B(Rp+q ) = B(Rp ) ⊗ B(Rq ). Needless to say, the same holds when there are more
 2.2 σ-Algebras and Knowledge 26

than two terms in the product. The n-fold product σ-algebra of F is denoted by
F ⊗n .

2.2 σ-algebras and knowledge

One of the conceptual advantages of measure-theoretic probability is the

relationship between σ-algebras and the intuitive idea of ‘knowledge’. Although
the relationship is useful and intuitive, it is regrettably not quite perfect. Let
(Ω, F), (X , G) and (Y, H) be measurable spaces and X : Ω → X and Y : Ω → Y
be random elements. Having observed the value of X (‘knowing X’), one might
wonder what this entails about the value of Y . Even more simplistically, under
what circumstances can the value of Y be determined exactly having observed X?
The situation is illustrated in Fig. 2.3. As it turns out, with some restrictions, the
answer can be given in terms of the σ-algebras generated by X and Y . Except

X
(Ω, F) (X , G)

f
Y
(Y, H)

Figure 2.3 The factorisation problem asks whether there exists a (measurable) function
f that makes the diagram commute.

for a technical assumption on (Y, H), the following result shows that Y is a
measurable function of X if and only if Y is σ(X)/H-measurable. The technical
assumption mentioned requires (Y, H) to be a Borel space, which is true of all
probability spaces considered in this book, including (Rk , B(Rk )). We leave the
exact definition of Borel spaces to the next chapter.
Lemma 2.5 (Factorisation lemma). Assume that (Y, H) is a Borel space. Then Y
is σ(X)-measurable (σ(Y ) ⊆ σ(X)) if and only if there exists a G/H-measurable
map f : X → Y such that Y = f ◦ X.
In this sense σ(X) contains all the information that can be extracted from X
via measurable functions. This is not the same as saying that Y can be deduced
from X if and only if Y is σ(X)-measurable because the set of X → Y maps
can be much larger than the set of G/H-measurable functions. When G is coarse,
there are not many G/H-measurable functions with the extreme case occurring
when G = {X , ∅}. In cases like this, the intuition that σ(X) captures all there
is to know about X is not true anymore (Exercise 2.6). The issue is that σ(X)
does not only depend on X, but also on the σ-algebra of (X , G) and that if G is
coarse-grained, then σ(X) can also be coarse-grained and not many functions
will be σ(X)-measurable. If X is a random variable, then by definition X = R
 2.2 σ-Algebras and Knowledge 27

and G = B(R), which is relatively fine-grained, and the requirement that f

be measurable is less restrictive. Nevertheless, even in the nicest setting where
Ω = X = Y = R and F = G = H = B(R), it can still occur that Y = f ◦ X for
some non-measurable f . In other words, all the information about Y exists in X
but cannot be extracted in a measurable way. These problems only occur when
X maps measurable sets in Ω to non-measurable sets in X . Fortunately, while
such random variables exist, they are never encountered in applications, which
provides the final justification for thinking of σ(X) as containing all that there is
to know about any random variable X that one may ever expect to encounter.

Filtrations
In the study of bandits and other online settings, information is revealed to the
learner sequentially. Let X1 , . . . , Xn be a collection of random variables on a
common measurable space (Ω, F). We imagine a learner is sequentially observing
the values of these random variables. First X1 , then X2 and so on. The learner
needs to make a prediction, or act, based on the available observations. Say, a
prediction or an act must produce a real-valued response. Then, having observed
.
X1:t = (X1 , . . . , Xt ), the set of maps f ◦ X1:t where f : Rt → R is Borel, captures
all the possible ways the learner can respond. By Lemma 2.5, this set contains
exactly the σ(X1:t )/B(R)-measurable maps. Thus, if we need to reason about
the set of Ω → R maps available after observing X1:t , it suffices to concentrate
on the σ-algebra Ft = σ(X1:t ). Conveniently, Ft is independent of the space of
possible responses, and being a subset of F, it also hides details about the range
space of X1:t . It is easy to check that F0 ⊆ F1 ⊆ F2 ⊆ · · · ⊆ Fn ⊆ F, which
means that more and more functions are becoming Ft -measurable as t increases,
which corresponds to increasing knowledge (note that F0 = {∅, Ω}, and the set
of F0 -measurable functions is the set of constant functions on Ω).
Bringing these a little further, we will often find it useful to talk about increasing
sequences of σ-algebras without constructing them in terms of random variables
as above. Given a measurable space (Ω, F), a filtration is a sequence (Ft )nt=0 of
sub-σ-algebras of F where Ft ⊆ Ft+1 for all t < n. We also allow n = ∞, and in
this case we define
∞
!
[
F∞ = σ Ft
t=0

to be the smallest σ-algebra containing the union of all Ft . Filtrations can also
be defined in continuous time, but we have no need for that here. A sequence
of random variables (Xt )nt=1 is adapted to filtration F = (Ft )nt=0 if Xt is Ft -
measurable for each t. We also say in this case that (Xt )t is F-adapted. The
same nomenclature applies if n is infinite. Finally, (Xt )t is F-predictable if Xt
is Ft−1 -measurable for each t ∈ [n]. Intuitively we may think of an F-predictable
process X = (Xt )t as one that has the property that Xt can be known (or
‘predicted’) based on Ft−1 , while a F-adapted process is one that has the property
that Xt can be known based on Ft only. Since Ft−1 ⊆ Ft , a predictable process
 2.3 Conditional Probabilities 28

is also adapted. A filtered probability space is the tuple (Ω, F, F, P), where
(Ω, F, P) is a probability space and F = (Ft )t is filtration of F.

2.3 Conditional Probabilities

Conditional probabilities are introduced so that we can talk about how

probabilities should be updated when one gains some partial knowledge about a
random outcome. Let (Ω, F, P) be a probability space, and let A, B ∈ F be such
that P (B) > 0. The conditional probability P (A | B) of A given B is defined
as
P (A ∩ B)
P (A | B) = .
P (B)
We can think about the outcome ω ∈ Ω as the result of throwing a many-sided
dice. The question asked is the probability that the dice landed so that ω ∈ A
given that it landed with ω ∈ B. The meaning of the condition ω ∈ B is that we
focus on dice rolls when ω ∈ B is true. All dice rolls when ω ∈ B does not hold
are discarded. Intuitively, what should matter in the conditional probability of A
given B is how large the portion of A is that lies in B, and this is indeed what
the definition means.

The importance of conditional probabilities is that they define a calculus of

how probabilities are to be updated in the presence of extra information.

The probability P (A | B) is also called the a posteriori (‘after the fact’)

probability of A given B. The a priori probability is P (A). Note that P (A | B) is
defined for every A ∈ F as long as P (B) > 0. In fact, A 7→ P (A | B) is a probability
measure over the measure space (Ω, F) called the a posteriori probability measure
given B (see Exercise 2.7). In a way the temporal characteristics attached to
the words ‘a posteriori’ and ‘a priori’ can be a bit misleading. Probabilities are
concerned with predictions. They express the degrees of uncertainty one assigns
to future events. The conditional probability of A given B is a prediction of
certain properties of the outcome of the random experiment that results in ω
given a certain condition. Everything is related to a future hypothetical outcome.
Once the dice is rolled, ω gets fixed, and either ω ∈ A, B or not. There is no
uncertainty left: predictions are trivial after an experiment is done.
Bayes rule states that provided events A, B ∈ F both occur with positive
probability,
P (B | A) P (A)
P (A | B) = . (2.2)
P (B)
Bayes rule is useful because it allows one to obtain P (A | B) based on information
about the quantities on the right-hand side. Remarkably, this happens to be
 2.4 Independence 29

the case quite often, explaining why this simple formula has quite a status in
probability and statistics. Exercise 2.8 asks the reader to verify this law.

2.4 Independence

Independence is another basic concept of probability that relates to

knowledge/information. In its simplest form, independence is a relation that
holds between events on a probability space (Ω, F, P). Two events A, B ∈ F are
independent if

P (A ∩ B) = P (A) P (B) . (2.3)

How is this related to knowledge? Assuming that P (B) > 0, dividing both sides
by P (B) and using the definition of conditional probability, we get that the above
is equivalent to

P (A | B) = P (A) . (2.4)

Of course, we also have that if P (A) > 0, (2.3) is equivalent to P (B | A) = P (B).

Both of the latter relations express that A and B are independent if the probability
assigned to A (or B) remains the same regardless of whether it is known that B
(respectively, A) occurred.
We hope our readers will find the definition of independence in terms of a ‘lack
of influence’ to be sensible. The reason not to use Eq. (2.4) as the definition is
mostly for the sake of convenience. If we started with (2.4), we would need to
separately discuss the case of P (B) = 0, which would be cumbersome. A second
reason is that (2.4) suggests an asymmetric relationship, but intuitively we expect
independence to be symmetric.
Uncertain outcomes are often generated part by part with no interaction
between the processes, which naturally leads to an independence structure (think
of rolling multiple dice with no interactions between the rolls). Once we discover
some independence structure, calculations with probabilities can be immensely
simplified. In fact, independence is often used as a way of constructing probability
measures of interest (cf. Eq. (2.1), Theorem 2.4 and Exercise 2.9). Independence
can also appear serendipitously in the sense that a probability space may hold
many more independent events than its construction may suggest (Exercise 2.10).

You should always carefully judge whether assumptions about independence

are really justified. This is part of the modelling and hence is not
mathematical in nature. Instead you have to think about the physical
process being modelled.

A collection of events G ⊂ F is said to be pairwise independent if any two

distinct elements of G are independent of each other. The events in G are said
 2.5 Integration and Expectation 30

to be mutually independent if for any n > 0 integer and A1 , . . . , An distinct

Qn
elements of G, P (A1 ∩ · · · ∩ An ) = i=1 P (Ai ). This is a stronger restriction
than pairwise independence. In the case of mutually independent events, the
knowledge of joint occurrence of any finitely many events from the collection will
not change our prediction of whether some other event in the collection happens.
But this may not be the case when the events are only pairwise independent
(Exercise 2.10). Two collections of events G1 , G2 are said to be independent of
each other if for any A ∈ G1 and B ∈ G2 it holds that A and B are independent.
This definition is often applied to σ-algebras.
When the σ-algebras are induced by random variables, this leads to the
definition of independence between random variables. Two random
variables X and Y are independent if σ(X) and σ(Y ) are independent of each
other. The notions of pairwise and mutual independence can also be naturally
extended to apply to collections of random variables. All these concepts can be
and are in fact extended to random elements.
The default meaning of independence when multiple events or random variables
are involved is mutual independence.

When we say that X1 , . . . , Xn are independent random variables, we mean

that they are mutually independent. Independence is always relative to
some probability measure, even when a probability measure is not explicitly
mentioned. In such cases the identity of the probability measure should be
clear from the context.

2.5 Integration and Expectation

A key quantity in probability theory is the expectation of a random variable. Fix

a probability space (Ω, F, P) and random variable X : Ω → R. The expectation X
is often denoted by E [X]. This notation unfortunately obscures the dependence
on the measure P. When the underlying measure is not obvious from context, we
write EP to indicate the expectation with respect to P. Mathematically, we define
the expected value of X as its Lebesgue integral with respect to P:
Z
E [X] = X(ω) dP(ω) .
Ω
R
The right-hand side is also often abbreviated to X dP. The integral on the
right-hand side is constructed to satisfy the following two key properties:

(a) The integral of indicators is the probability of the underlying

R event. If X(ω) =
I {ω ∈ A} is an indicator function for some A ∈ F, then XdP = P (A).
(b) Integrals are linear. For all random variables X1 , X2 and reals α1 , α2 such
 2.5 Integration and Expectation 31

R R R
that X1 dP and X2 dP are defined, (α1 X1 + α2 X2 )dP is defined and
satisfies
Z Z Z
(α1 X1 + α2 X2 ) dP = α1 X1 dP + α2 X2 dP . (2.5)
Ω Ω Ω
Pn
These two properties together tell us that whenever X(ω) = i=1 αi I {ω ∈ Ai }
for some n, αi ∈ R and Ai ∈ F, i = 1, . . . , n, then
Z X
XdP = αi P (Ai ) . (2.6)
Ω i

Functions of the form X are called simple functions.

In defining the Lebesgue integral of some random variable X, we use (2.6) as
the definition of the integral when X is a simple function. The next step is to
extend the definition to non-negative random variables. Let X : Ω → [0, ∞) be
measurable. The idea is to approximate X from below using simple functions
and take the largest value that can be obtained this way:
Z Z 
XdP = sup h dP : h is simple and 0 ≤ h ≤ X . (2.7)
Ω Ω

The meaning of U ≤ V for random variables U, V is that U (ω) ≤ V (ω) for all
ω ∈ Ω. The supremum on the right-hand side could be infinite, in which case we
say the integral of X is not defined. Whenever the integral of X is defined, we
say that X is integrable or, if the identity of the measure P is unclear, that X
is integrable with respect to P.
Integrals for arbitrary random variables are defined by decomposing the
random variable into positive and negative parts. Let X : Ω → R be any
measurable function. Then define X + (ω) = X(ω)I {X(ω) > 0} and X − (ω) =
−X(ω)I {X(ω) < 0} so that X(ω) = X + (ω) − X − (ω). Now X + and X − are
both non-negative random variables called the positive and negative parts of
X. Provided that both X + and X − are integrable, we define
Z Z Z
XdP = X + dP − X − dP .
Ω Ω Ω

Note that X is integrable if and only if the non-negative-valued random variable

|X| is integrable (Exercise 2.12).

None of what we have done depends on P being a probability measure. The

definitions hold for any measure, though for signed measures it is necessary to
split Ω into disjoint measurable sets on which the measure is positive/negative,
an operation that is possible by the Hahn decomposition theorem. We
will never need signed measures in this book, however.

A particularly interesting case is when Ω = R is the real line, F = B(R) is

the Borel σ-algebra and the measure is the Lebesgue measure λ, which is the
 2.5 Integration and Expectation 32

unique measure on B(R) such that λ((a, b)) = b − a for any a ≤ b. In this scenario,
if f : R → R is a Borel-measurable function, then we can write the Lebesgue
integral of f with respect to the Lebesgue measure as
Z
f dλ .
R

Perhaps unsurprisingly, this almost always coincides

R∞ with the improper Riemann
integral of f , which is normally written as −∞ f (x)dx. Precisely, if |f | is both
Lebesgue integrable and Riemann integrable, then the integrals are equal.

There exist functions that are Riemann integrable and not Lebesgue
integrable, and also the other way around (although examples of the former
are more exotic than the latter).

The Lebesgue measure and its relation to Riemann integration is mentioned

because when it comes to actually calculating the value of an expectation or
integral, this is often reduced to calculating integrals over the real line with
respect to the Lebesgue measure. The calculation is then performed by evaluating
the Riemann integral, thereby circumventing the need to rederive the integral
of many elementary functions. Integrals (and thus expectations) have a number
of important properties. By far the most important is their linearity, which was
postulated above as the second property in (2.5). To practice using the notation
with expectations, we restate the first half of this property. In fact, the statement
is slightly more general than what we demanded for integrals above.

Proposition 2.6. Let (Xi )i be a (possibly infinite) sequence of random variables

on the same probability space and assume that E [Xi ] exists for all i and
P P
furthermore that X = i Xi and E [ i |Xi |] also exist. Then
X
E [X] = E [Xi ] .
i

This exchange of expectations and summation is the source of much magic

in probability theory because it holds even if Xi are not independent. This
means that (unlike probabilities) we can very often decouple the expectations of
dependent random variables, which often proves extremely useful (a collection
of random variables is dependent if they are not independent). You will prove
Proposition 2.6 in Exercise 2.14. The other requirement for linearity is that if
c ∈ R is a constant, then E [cX] = c E [X] (Exercise 2.15).
Another important statement is concerned with independent random variables.

Proposition 2.7. If X and Y are independent, then E [XY ] = E [X] E [Y ].

In general E [XY ] =
6 E [X] E [Y ] (Exercise 2.18). Finally, an important simple
result connects expectations of non-negative random variables to their tail
probabilities.
 2.6 Conditional Expectation 33

Proposition 2.8. If X ≥ 0 is a non-negative random variable, then

Z ∞
E [X] = P (X > x) dx .
0

The integrand in Proposition 2.8 is called the tail probability function

x 7→ P (X > x) of X. This is also known as the complementary cumulative
distribution function of X. The cumulative distribution function (CDF) of
X is defined as x 7→ P (X ≤ x) and is usually denoted by FX . These functions
are defined for all random variables, not just non-negative ones. One can check
that FX : R → [0, 1] is increasing, right continuous and limx→−∞ FX (x) = 0 and
limx→∞ FX (x) = 1. The CDF of a random variable captures every aspect of the
probability measure PX induced by X, while still being just a function on the real
line, a property that makes it a little more human friendly than PX . One can also
generalise CDFs to random vectors: if X is an Rk -valued random vector, then its
CDF is defined as the FX : Rk → [0, 1] function that satisfies FX (x) = P (X ≤ x),
where, in line with our conventions, X ≤ x means that all components of X are
less than or equal to the respective component of x. The pushforward PX of a
random element is an alternative way to summarise the distribution of X. In
particular, for any real-valued, f : X → R measurable function,
Z
E [f (X)] = f (x)dPX (x)
X

provided that either the right-hand side, or the left-hand side exist.

2.6 Conditional Expectation

Conditional expectation allows us to talk about the expectation of a random

variable given the value of another random variable, or more generally, given
some σ-algebra.

Example 2.9. Let (Ω, F, P) model the outcomes of an unloaded dice: Ω = [6],
F = 2Ω and P(A) = |A|/6. Define two random variables X and Y by
Y (ω) = I {ω > 3} and X(ω) = ω. Suppose we are interested in the expectation
of X given a specific value of Y . Arguing intuitively, we might notice that Y = 1
means that the unobserved X must be either 4, 5 or 6, and that each of these
outcomes is equally likely, and so the expectation of X given Y = 1 should
be (4 + 5 + 6)/3 = 5. Similarly, the expectation of X given Y = 0 should be
(1 + 2 + 3)/3 = 2. If we want a concise summary, we can just write that ‘the
expectation of X given Y ’ is 5Y + 2(1 − Y ). Notice how this is a random variable
itself.

The notation for this conditional expectation is E [X | Y ]. Using this notation,

in Example 2.9 we can concisely write E [X | Y ] = 5Y + 2(1 − Y ). A little more
generally, if X : Ω → X and Y : Ω → Y with X , Y ⊂ R and |X |, |Y| < ∞, then
 2.6 Conditional Expectation 34

E[X | Y ] : Ω → R is the random variable given by E[X | Y ](ω) = E[X | Y = Y (ω)],

where
X X x P (X = x, Y = y)
E[X | Y = y] = x P (X = x | Y = y) = . (2.8)
P (Y = y)
x∈X x∈X

This is undefined when P(Y = y) = 0 so that E[X | Y ](ω) is undefined on the

measure zero set {ω : P(Y = Y (ω)) = 0}.
Eq. (2.8) does not generalise to continuous random variables because P (Y = y)
in the denominator might be zero for all y. For example, let Y be a random
variable taking values on [0, 1] according to a uniform distribution and X ∈ {0, 1}
be Bernoulli with biasRY . This means that the joint measure on X and Y is
P (X = 1, Y ∈ [p, q]) = p xdx for 0 ≤ p < q ≤ 1. Intuitively it seems like E[X | Y ]
q

should be equal to Y , but how to define it? The mean of a Bernoulli random
variable is equal to its bias so the definition of conditional probability shows that
for 0 ≤ p < q ≤ 1,

E[X = 1 | Y ∈ [p, q]] = P (X = 1 | Y ∈ [p, q])

P (X = 1, Y ∈ [p, q])
=
P (Y ∈ [p, q])
q − p2
2
=
2(q − p)
p+q
= .
2
This calculation is not well defined when p = q because P (Y ∈ [p, p]) = 0.
Nevertheless, letting q = p + ε for ε > 0 and taking the limit as ε tends to zero
seems like a reasonable way to argue that P (X = 1 | Y = p) = p. Unfortunately
this approach does not generalise to abstract spaces because there is no canonical
way of taking limits towards a set of measure zero, and different choices lead to
different answers.
Instead we use Eq. (2.8) as the starting point for an abstract definition of
conditional expectation as a random variable satisfying two requirements. First,
from Eq. (2.8) we see that E[X | Y ](ω) should only depend on Y (ω) and so
should be measurable with respect to σ(Y ). The second requirement is called the
‘averaging property’. For measurable A ⊆ Y, Eq. (2.8) shows that
X
E[IY −1 (A) E[X | Y ]] = P (Y = y) E[X | Y = y]
y∈A
XX
= x P (X = x, Y = y)
y∈A x∈X

= E[IY −1 (A) X] .

This can be viewed as putting a set of linear constraints on E[X | Y ] with one
constraint for each measurable A ⊆ Y. By treating E[X | Y ] as an unknown
σ(Y )-measurable random variable, we can attempt to solve this linear system. As
 2.6 Conditional Expectation 35

it turns out, this can always be done: the linear constraints and the measurability
restriction on E [X | Y ] completely determine E[X | Y ] except for a set of measure
zero. Notice that both conditions only depend on σ(Y ) ⊆ F. The abstract
definition of conditional expectation takes these properties as the definition and
replaces the role of Y with a sub-σ-algebra.
Definition 2.10 (Conditional expectation). Let (Ω, F, P) be a probability space
and X : Ω → R be random variable and H be a sub-σ-algebra of F. The
conditional expectation of X given H is denoted by E[X | H] and defined to be
any H-measurable random variable on Ω such that for all H ∈ H,
Z Z
E[X | H]dP = XdP . (2.9)
H H

Given a random variable Y , the conditional expectation of X given Y is

E [X | Y ] = E [X | σ(Y )].
Theorem 2.11. Given any probability space (Ω, F, P), a sub-σ-algebra H of F
and a P-integrable random variable X : Ω → R, there exists an H-measurable
function f : Ω → R that satisfies (2.9). Further, any two H-measurable functions
f1 , f2 : Ω → R that satisfy (2.9) are equal with probability one: P(f1 = f2 ) = 1.
When random variables X and Y agree with P-probability one, we say they
are P-almost surely equal, which is often abbreviated to ‘X = Y P-a.s.’, or
‘X = Y a.s.’ when the measure is clear from context. A related useful notion is
the concept of null sets: U ∈ F is a null set of P, or a P-null set if P(U ) = 0.
Thus, X = Y P-a.s. if and only if X = Y agree except on a P-null set.

The reader may find it odd that E[X | Y ] is a random variable on Ω rather
than the range of Y . Lemma 2.5 and the fact that E[X | σ(Y )] is σ(Y )-
measurable shows there exists a measurable function f : (R, B(R)) →
(R, B(R)) such that E[X | σ(Y )](ω) = (f ◦ Y )(ω) (see Fig. 2.4). In this sense
E[X | Y ](ω) only depends on Y (ω), and occasionally we write E[X | Y ](y).

(Ω, F)

Y E[X | Y ]

(R, B(R)) (R, B(R))

f

Figure 2.4 Factorisation of conditional expectation. When there is no confusion, we

occasionally write E[X | Y ](y) in place of f (y).

Returning to Example 2.9, we see that E [X | Y ] = E [X | σ(Y )] and σ(Y ) =

 2.6 Conditional Expectation 36

{{1, 2, 3}, {4, 5, 6}, ∅, Ω}. Denote this set-system by H for brevity. The condition
that E[X | H] is H-measurable can only be satisfied if E[X | H](ω) is constant on
{1, 2, 3} and {4, 5, 6}. Then (2.9) immediately implies that
(
2, if ω ∈ {1, 2, 3} ;
E [X | H] (ω) =
5, if ω ∈ {4, 5, 6} .
While the definition of conditional expectations given above is non-constructive
and E[X | H] is uniquely defined only up to events of P-measure zero, none of
this should be of a significant concern. First, we will rarely need closed-form
expressions for conditional expectations, but we rather need how they relate to
other expectations, conditional or not. This is also the reason why it should not
be concerning that they are only determined up to zero probability events: usually,
conditional expectations appear in other expectations or in statements that are
concerned with how probable some event is, making the difference between the
different ‘versions’ of conditional expectations disappear.
We close the section by summarising some additional important properties of
conditional expectations. These follow from the definition directly, and the reader
is invited to prove them in Exercise 2.20.
Theorem 2.12. Let (Ω, F, P) be a probability space, G, G1 , G2 ⊂ F be sub-σ-
algebras of F and X, Y integrable random variables on (Ω, F, P). The following
hold true:

1 If X ≥ 0, then E [X | G] ≥ 0 almost surely.

2 E [1 | G] = 1 almost surely.
3 E [X + Y | G] = E [X | G] + E [Y | G] almost surely.
4 E [XY | G] = Y E [X | G] almost surely if E [XY ] exists and Y is G-measurable.
5 If G1 ⊂ G2 , then E [X | G1 ] = E [E [X | G2 ] | G1 ] almost surely.
6 If σ(X) is independent of G2 given G1 , then E [X | σ(G1 ∪ G2 )] = E [X | G1 ]
almost surely.
7 If G = {∅, Ω} is the trivial σ-algebra, then E [X | G] = E [X] almost surely.
Properties 1 and 2 are self-explanatory. Property 3 generalises the linearity of
expectation. Property 4 shows that a measurable quantity can be pulled outside
of a conditional expectation and corresponds to the property that for constants
c, E [cX] = cE [X]. Property 5 is called the tower rule or the law of total
expectations. It says that the fineness of E[X | G2 ] is obliterated when taking the
conditional expectation with respect to G1 . Property 6 relates independence and
conditional expectations, and it says that conditioning on independent quantities
does not give further information on expectations. Here, the two event systems A
and B are said to be conditionally independent of each other given a σ-algebra
F if for all A ∈ A and B ∈ B, P (A ∩ B | F) = P (A | F) P (B | F) holds almost
surely. We also often say that A is conditionally independent of B given F, but
of course, this relation is symmetric. This property is often applied with random
variables: X is said to be conditionally independent of Y given Z, if σ(X) is
 2.7 Notes 37

conditionally independent of σ(Y ) given σ(Z). In this case, E [X | Y, Z] = E [X | Z]

holds almost surely. Property 7 states that conditioning on no information gives
the same expectation as not conditioning at all.

The above list of abstract properties will be used over and over again. We
encourage the reader to study the list carefully and convince yourself that
all items are intuitive. Playing around with discrete random variables can
be invaluable for this. Eventually it will all become second nature.

2.7 Notes

1 The Greek letter σ is often used by mathematicians in association with

countable infinities. Hence the term σ-algebra (and σ-field). Note that countable
additivity is often called σ-additivity. The requirement that additivity should
hold for systems of countably infinitely many sets is made so that probabilities
of (interesting) limiting events are guaranteed to exist.
2 Measure theory is concerned with measurable spaces, measures and with
their properties. An obvious distinction between probability theory and measure
theory is that in probability theory, one is (mostly) concerned with probability
measures. But the distinction does not stop here. In probability theory, the
emphasis is on the probability measures and their relations to each other. The
measurable spaces are there in the background, but are viewed as part of the
technical toolkit rather than the topic of main interest. Also, in probability
theory, independence is often at the center of attention, while independence is
not a property measure-theorists care much about.
3 In our toy example, instead of Ω = [6]7 , we could have chosen Ω = [6]8
(considering rolling eight dice instead of seven, one dice never used). There are
many other possibilities. We can consider coin flips instead of dice rolls (think
about how this could be done). To make this easy, we could use weighted coins
(for example, a coin that lands on heads with probability 1/6), but we don’t
actually need weighted coins (this may be a little tricky to see). The main
point is that there are many ways to emulate one randomisation device by
using another. The difference between these is the set Ω. What makes a choice
of Ω viable is if we can emulate the game mechanism on the top of Ω so that
in the end the probability of seeing any particular value remains the same. But
the main point is that the choice of Ω is far from unique. The same is true for
the way we calculate the value of the game! For example, the dice could be
reordered, if we stay with the first construction. This was noted already, but it
cannot be repeated frequently enough: the biggest conspiracy in all probability
theory is that we first make a big fuss about introducing Ω, and then it turns
out that the actual construction of Ω does not matter.
 2.7 Notes 38

4 All Riemann-integrable functions on a bounded domain are Lebesgue integrable.

Difficulties
R∞ only arise when taking improper integrals. A standard example
is 0 sin(x)dx
x , which is an improper
R Riemann integrable function, but is
not Lebesgue integrable because (0,∞) | sin(x)/x|dx = ∞. The situation is
analogous to the difference between conditionally and absolutely convergent
series, with the Lebesgue integral only defined in the latter case.
5 Can you think of a set that is not Borel measurable? Such sets exist, but do not
arise naturally in applications. The classic example is the Vitali set, which is
formed by taking the quotient group G = R/Q and then applying the axiom
of choice to choose a representative in [0, 1] from each equivalence class in G.
Non-measurable functions are so unusual that you do not have to worry much
about whether or not functions X : R → R are measurable. With only a few
exceptions, questions of measurability arising in this book are not related to
the fine details of the Borel σ-algebra. Much more frequently they are related
to filtrations and the notion of knowledge available having observed certain
random elements.
6 There is a lot to say about why the sum, or the product of random variables
are also random variables. Or why inf n Xn , supn Xn , lim inf n Xn , lim supn Xn
are measurable when Xn are. The key point is to show that the composition of
measurable maps is a measurable map and that continuous maps are measurable
and then apply these results (Exercise 2.1). For lim supn Xn , just rewrite it as
limm→∞ supn≥m Xn ; note that supn≥m Xn is decreasing (we take suprema of
smaller sets as m increases), hence lim supn Xn = inf m supn≥m Xn , reducing
the question to studying inf n Xn and supn Xn . Finally, for inf n Xn note that
it suffices if {ω : inf n Xn ≥ t} is measurable for any t real. Now, inf n Xn ≥ t
if and only if Xn ≥ t for all n. Hence, {ω : inf n Xn ≥ t} = ∩n {ω : Xn ≥ t},
which is a countable intersection of measurable sets, hence measurable (this
latter follows by the elementary identity (∩i Ai )c = ∪i Aci ).
7 The factorisation lemma, Lemma 2.5, is attributed to Joseph Doob and Eugene
Dynkin. The lemma sneakily uses the properties of real numbers (think about
why), which is another reason why what we said about σ-algebras containing
all information is not entirely true. The lemma has extensions to more general
random elements [Taraldsen, 2018, for example]. The key requirement in a
way is that the σ-algebra associated with the range space of Y should be rich
enough.
8 We did not talk about basic results like Lebesgue’s dominated/monotone
convergence theorems, Fatou’s lemma or Jensen’s inequality. We will definitely
use the last of these, which is explained in a dedicated chapter on convexity
(Chapter 26). The other results can be found in the texts we cite. They are
concerned with infinite sequences of random variables and conditions under
which their limits can be interchanged with Lebesgue integrals. In this book
we rarely encounter problems related to such sequences and hope you forgive
us on the few occasions they are necessary (the reason is simply because we
 2.7 Notes 39

mostly focus on finite time results or take expectations before taking limits
when dealing with asymptotics).
9 You might be surprised that we have not mentioned densities. For most of
us, our first exposure to probability on continuous spaces was by studying the
normal distribution and its density
1
p(x) = √ exp(−x2 /2) , (2.10)
2π
which can be integrated over intervals to obtain the probability that a Gaussian
random variable will take a value in that interval. The reader should notice
that p : R → R is Borel measurable and that the Gaussian measure associated
with this density is P on (R, B(R)) defined by
Z
P(A) = p dλ .
A

Here the integral is with respect to the Lebesgue measure λ on (R, B(R)). The
notion of a density can be generalised beyond this simple setup. Let P and Q
be measures (not necessarily probability measures) on arbitrary measurable
space (Ω, F). The Radon–Nikodym derivative of P with respect to Q is
an F-measurable random variable dQ dP
: Ω → [0, ∞) such that
Z
dP
P (A) = dQ for all A ∈ F . (2.11)
A dQ
R R
We can also write this in the form IA dP = IA dQ dP
dQ, A ∈ F, from which we
R R dP
may realise that for any X P -integrable random variable, XdP = X dQ dQ
must also hold. This is often called the change-of-measure formula. Another
word for the Radon–Nikodym derivative dQ dP
is the density of P with respect to
Q. It is not hard to find examples where the density does not exist. We say that
P is absolutely continuous with respect to Q if Q(A) = 0 =⇒ P (A) = 0
for all A ∈ F. When dQ dP
exists, it follows immediately that P is absolutely
continuous with respect to Q by Eq. (2.11). Except for some pathological cases,
it turns out that this is both necessary and sufficient for the existence of dP/dQ.
The measure Q is σ-finite if there exists a countable covering {Ai } of Ω with
F-measurable sets such that Q(Ai ) < ∞ for each i.
Theorem 2.13. Let P, Q be measures on a common measurable space (Ω, F)
and assume that Q is σ-finite. Then the density of P with respect to Q, dQ dP
,
exists if and only if P is absolutely continuous with respect to Q. Furthermore,
dQ is uniquely defined up to a Q-null set so that for any f1 , f2 satisfying (2.11),
dP

f1 = f2 holds Q-almost surely.

Densities work as expected. Suppose that Z is a standard Gaussian random
variable. We usually write its density as in Eq. (2.10), which we now know
is the Radon–Nikodym derivative of the Gaussian measure with respect to
the Lebesgue measure. The densities of ‘classical’ continuous distributions are
almost always defined with respect to the Lebesgue measure.
 2.7 Notes 40

10 In line with the literature, we will use P  Q to denote that P is absolutely

continuous with respect to Q. When P is absolutely continuous with respect
to Q, we also say that Q dominates P .
11 A useful result for Radon–Nikodym derivatives is the chain rule, which states
that if P  Q  S, then dQ dS = dS . The proof of this result follows from our
dP dQ dP
R R dQ
earlier observation that f dQ = f dS dS for any Q-integrable f . Indeed, the
chain rule is obtained from this by taking f = IA dQ dP
with A ∈ F and noting
R R
that this is indeed Q-integrable and IA dQ dQ = IA dQ. The chain rule is
dP

often used to reduce the calculation of densities to calculation with known

densities.
12 The Radon–Nikodym derivative unifies the notions of distribution (for discrete
spaces) and density (for continuous spaces). Let Ω be discrete (finite or
countable) and let ρ be the counting measure on (Ω, 2Ω ), which is defined
by ρ(A) = |A|. For any P on (Ω, F), it is easy to see that P  ρ and
dρ (i) = P ({i}), which is sometimes called the distribution function of P .
dP

13 The Radon–Nikodym derivative provides another way to define the conditional

expectation. Let X be an integrable random variable on (Ω, F, P) and H ⊂ F
be a sub-σ-algebra and P|RH be the restriction of P to (Ω, H). Define measure
µ on (Ω, H) by µ(A) = A XdP|H . It is easy to check that µ  P|H and
that E[X | H] = dP| dµ
H
satisfies Eq. (2.9). We note that the proof of the
Radon–Nikodym theorem is nontrivial and that the existence of conditional
expectations are more easily guaranteed via an ‘elementary’ but abstract
argument using functional analysis.
14 The Fubini–Tonelli theorem, which we will also refer to as Fubini’s theorem,
is a powerful result that allows one to exchange the order of integrations. This
result is needed for example for proving Proposition 2.8 (Exercise 2.19). To state
it, we need to introduce product measures. These work as expected: given two
probability spaces, (Ω1 , F1 , P1 ) and (Ω2 , F2 , P2 ), the product measure P of P1
and P2 is defined as any measure on (Ω1 ×Ω2 , F1 ⊗F2 ) that satisfies P(A1 , A2 ) =
P1 (A1 )P2 (A2 ) for all (A1 , A2 ) ∈ F1 × F2 (recall that F1 ⊗ F2 = σ(F1 × F2 ) is
the product σ-algebra of F1 and F2 ). Theorem 2.4 implies that this product
measure, which is often denoted by P1 × P2 (or P1 ⊗ P2 ) is uniquely defined.
(Think about what this product measure has to do with independence.) The
Fubini–Tonelli theorem (often just ‘Fubini’) states the following: let (Ω1 , F1 , P1 )
and (Ω2 , F2 , P2 ) be two probability spaces and consider a random variable
X on the product probability R space (Ω, F, P) R R= (Ω1 × Ω2 , F1 ⊗ F2 , P1 × P2 ).
If
R any
R of the three integrals |X(ω)| dP(ω), ( |X(ω1 , ω2 )| dP1 (ω1 )) dP2 (ω2 ),
( |X(ω1 , ω2 )| dP2 (ω2 )) dP1 (ω1 ) is finite, then
Z Z Z 
X(ω) dP(ω) = X(ω1 , ω2 ) dP1 (ω1 ) dP2 (ω2 )
Z Z 
= X(ω1 , ω2 ) dP2 (ω2 ) dP1 (ω1 ) .
 2.8 Bibliographic Remarks 41

15 For topological space X, the support of a measure µ on (X, B(X)) is

Supp(µ) = {x ∈ X : µ(U ) > 0 for all neighborhoods U of x} .
When X is discrete, this reduces to Supp(µ) = {x : µ({x}) > 0}.
16 Let X be a topological space. The weak* topology on the space of probability
R
measures P(X) on (X, B(X)) is the coarsest topology such that µ 7→ f dµ
is continuous for all bounded continuous functions f : X → R. In particular,
a sequence of probability
R measuresR (µn )n=1 converges to µ in this topology
∞

if and only if limn→∞ f dµn = f dµ for all bounded continuous functions

f : X → R.
Theorem 2.14. When X is compact and Hausdorff and P(X) is the space of
regular probability measures on (X, B(X)) with the weak* topology, then P(X)
is compact.
17 Mathematical terminology can be a bit confusing sometimes. Since E maps
(certain) functions to real values, it is also called the expectation operator.
‘Operator’ is just a fancy name for a function. In operator theory, the study
of operators, the focus is on operators whose domain is infinite dimensional,
hence the distinct name. However, most results of operator theory do not
hinge upon this property. If the image space is the set of reals, we talk about
functionals. The properties of functionals are studied in yet another subfield of
mathematics, functional analysis. The expectation operator is a functional
that maps the set of P-integrable functions (often denoted by L1 (Ω, P) or
L1 (P)) to reals. Its most important property is linearity, which was stated as
a requirement for integrals that define the expectation operator (Eq. (2.5)).
In line with the previous comment, when we use E, more often than not, the
probability space remains hidden. As such, the symbol E is further abused.

2.8 Bibliographic Remarks

Much of this chapter draws inspiration from David Pollard’s A user’s guide to
measure theoretic probability [Pollard, 2002]. We like this book because the author
takes a rigourous approach, but still explains the ‘why’ and ‘how’ with great
care. The book gets quite advanced quite fast, concentrating on the big picture
rather than getting lost in the details. Other useful references include the book by
Billingsley [2008], which has many good exercises and is quite comprehensive in
terms of its coverage of the ‘basics’. These books are both quite detailed. For an
outstanding shorter introduction to measure-theoretic probability, see the book
by Williams [1991], which has an enthusiastic style and a pleasant bias towards
martingales. We also like the book by Kallenberg [2002], which is recommended
for the mathematically inclined readers who already have a good understanding of
the basics. The author has put a major effort into organising the material so that
redundancy is minimised and generality is maximised. This reorganisation resulted
 2.9 Exercises 42

in quite a few original proofs, and the book is comprehensive. The factorisation
lemma (Lemma 2.5) is stated in the book by Kallenberg [2002] (Lemma 1.13
there). Kallenberg calls this lemma the ‘functional representation’ lemma and
attributes it to Joseph Doob. Theorem 2.4 is a corollary of Carathéodory’s
extension theorem, which says that probability measures defined on semi-rings of
sets have a unique extension to the generated σ-algebra. The remaining results can
be found in either of the three books mentioned above. Theorem 2.14 appears as
theorem 8.9.3 in the two-volume book by Bogachev [2007]. Finally, for something
older and less technical, we recommend the philosophical essays on probability
by Pierre Laplace, which was recently reprinted [Laplace, 2012].

2.9 Exercises

2.1 (Composing random elements) Show that if f is F/G-measurable and g

is G/H-measurable for sigma algebras F, G and H over appropriate spaces, then
their composition, g ◦ f (defined the usual way: (g ◦ f )(ω) = g(f (ω)), ω ∈ Ω), is
F/H-measurable.

2.2 Let X1 , . . . , Xn be random variables on (Ω, F). Prove that (X1 , . . . , Xn ) is

a random vector.

2.3 (Random variable induced σ-algebra) Let U be an arbitrary set and

(V, Σ) a measurable space and X : U → V an arbitrary function. Show that
ΣX = {X −1 (A) : A ∈ Σ} is a σ-algebra over U.

2.4 Let (Ω, F) be a measurable space and A ⊆ Ω and F|A = {A ∩ B : B ∈ F}.

(a) Show that (A, F|A ) is a measurable space.
(b) Show that if A ∈ F, then F|A = {B : B ∈ F, B ⊆ A}.

2.5 Let G ⊆ 2Ω be a non-empty collection of sets and define σ(G) as the smallest
σ-algebra that contains G. By ‘smallest’ we mean that F ∈ 2Ω is smaller than
F 0 ∈ 2Ω if F ⊂ F 0 .
(a) Show that σ(G) exists and contains exactly those sets A that are in every
σ-algebra that contains G.
(b) Suppose (Ω0 , F) is a measurable space and X : Ω0 → Ω be F/G-measurable.
Show that X is also F/σ(G)-measurable. (We often use this result to simplify
the job of checking whether a random variable satisfies some measurability
property).
(c) Prove that if A ∈ F where F is a σ-algebra, then I {A} is F-measurable.

2.6 (Knowledge and σ-algebras: a pathological example) In the context

of Lemma 2.5, show an example where Y = X and yet Y is not σ(X) measurable.

Hint As suggested after the lemma, this can be arranged by choosing

 2.9 Exercises 43

Ω = Y = X = R, X(ω) = Y (ω) = ω, F = H = B(R) and G = {∅, R} to

be the trivial σ-algebra.
2.7 Let (Ω, F, P) be a probability space, B ∈ F be such that P (B) > 0. Prove
that A 7→ P (A | B) is a probability measure over (Ω, F).

2.8 (Bayes law) Verify (2.2).

2.9 Consider the standard probability space (Ω, F, P) generated by two standard,
unbiased, six-sided dice that are thrown independently of each other. Thus,
Ω = {1, . . . , 6}2 , F = 2Ω and P(A) = |A|/62 for any A ∈ F so that Xi (ω) = ωi
represents the outcome of throwing dice i ∈ {1, 2}.

(a) Show that the events ‘X1 < 2’ and ‘X2 is even’ are independent of each
other.
(b) More generally, show that for any two events, A ∈ σ(X1 ) and B ∈ σ(X2 ),
are independent of each other.

2.10 (Serendipitous independence) The point of this exercise is to understand

independence more deeply. Solve the following problems:

(a) Let (Ω, F, P) be a probability space. Show that ∅ and Ω (which are events)
are independent of any other event. What is the intuitive meaning of this?
(b) Continuing the previous part, show that any event A ∈ F with P (A) ∈ {0, 1}
is independent of any other event.
(c) What can we conclude about an event A ∈ F that is independent of its
complement, Ac = Ω \ A? Does your conclusion make intuitive sense?
(d) What can we conclude about an event A ∈ F that is independent of itself?
Does your conclusion make intuitive sense?
(e) Consider the probability space generated by two independent flips of unbiased
coins with the smallest possible σ-algebra. Enumerate all pairs of events
A, B such that A and B are independent of each other.
(f) Consider the probability space generated by the independent rolls of two
unbiased three-sided dice. Call the possible outcomes of the individual dice
rolls 1, 2 and 3. Let Xi be the random variable that corresponds to the
outcome of the ith dice roll (i ∈ {1, 2}). Show that the events {X1 ≤ 2} and
{X1 = X2 } are independent of each other.
(g) The probability space of the previous example is an example when the
probability measure is uniform on a finite outcome space (which happens to
have a product structure). Now consider any n-element, finite outcome space
with the uniform measure. Show that A and B are independent of each other
if and only if the cardinalities |A|, |B|, |A ∩ B| satisfy n|A ∩ B| = |A| · |B|.
(h) Continuing with the previous problem, show that if n is prime, then no
non-trivial events are independent (an event A is trivial if P (A) ∈ {0, 1}).
(i) Construct an example showing that pairwise independence does not imply
mutual independence.
 2.9 Exercises 44

(j) Is it true or not that A, B, C are mutually independent if and only if

P (A ∩ B ∩ C) = P (A) P (B) P (C)? Prove your claim.

2.11 (Independence and random elements) Solve the following problems:

(a) Let X be a constant random element (that is, X(ω) = x for any ω ∈ Ω over
the outcome space over which X is defined). Show that X is independent of
any other random variable.
(b) Show that the above continues to hold if X is almost surely constant (that
is, P (X = x) = 1 for an appropriate value x).
(c) Show that two events are independent if and only if their indicator random
variables are independent (that is, A, B are independent if and only if
X(ω) = I {ω ∈ A} and Y (ω) = I {ω ∈ B} are independent of each other).
(d) Generalise the result of the previous item to pairwise and mutual
independence for collections of events and their indicator random variables.

2.12 Our goal in this exercise is to show that X is integrable if and only if |X|
is integrable. This is broken down into multiple steps. The first issue is to deal
with the measurability of |X|. While a direct calculation can also show this, it
may be worthwhile to follow a more general path:

(a) Any f : R → R continuous function is Borel measurable.

(b) Conclude that for any random variable X, |X| is also a random variable.
(c) Prove that for any random variable X, X is integrable if and only if |X|
is integrable. (The statement makes sense since |X| is a random variable
whenever X is).

Hint For (b) recall Exercise 2.1. For (c) examine the relationship between
|X| and (X)+ and (X)− .
2.13 (Infinite-valued integrals) Can we consistently extend the definition of
integrals so that for non-negative random variables, the integral is always defined
(it may be infinite)? Defend your view by either constructing an example (if you
are arguing against) or by proving that your definition is consistent with the
requirements we have for integrals.

2.14 Prove Proposition 2.6.

Hint You may find it useful to use Lebesgue’s dominated/monotone convergence

theorems.
2.15 Prove that if c ∈ R is a constant, then E [cX] = cE [X] (as long as X is
integrable).

2.16 Prove Proposition 2.7.

Hint Follow the ‘inductive’ definition of Lebesgue integrals, starting with

 2.9 Exercises 45

simple functions, then non-negative functions and finally arbitrary independent

random variables.
2.17 Suppose that G1 ⊂ G2 and prove that E[X | G1 ] = E[E[X | G1 ] | G2 ] almost
surely.

2.18 Demonstrate using an example that in general, for dependent random

variables, E [XY ] = E [X] E [Y ] does not hold.

2.19 Prove Proposition 2.8.

R
Hint Argue that X(ω) = [0,∞) I {[0, X(ω)]} (x) dx and exchange the integrals.
Use the Fubini–Tonelli theorem to justify the exchange of integrals.
2.20 Prove Theorem 2.12.
3 Stochastic Processes and Markov
Chains ( )

The measure-theoretic probability in the previous chapter covers almost all the
definitions required. Occasionally, however, infinite sequences of random variables
arise, and for these a little more machinery is needed. We expect most readers
will skip this chapter on the first reading, perhaps referring to it when necessary.
Before one can argue about the properties of infinite sequences of random
variables, it must be demonstrated that such sequences exist under certain
constraints on their joint distributions. For example, does there exist an infinite
sequence of random variables such that any finite subset of the random variables
are independent and distributed like a standard Gaussian? The first theorem
provides conditions under which questions like this can be answered positively.
This allows us to write, for example, ‘let (Xn )∞ n=1 be an infinite sequence of
independent standard Gaussian random variables’ and be comfortable knowing
there exists a probability space on which these random variables can be defined.
To state the theorem, we need the concept of Borel spaces.
Two measurable spaces (X , F) and (Y, G) are said to be isomorphic if there
exists a bijective function f : X → Y such that f is F/G-measurable and f −1 is
G/F-measurable. A Borel space is a measurable space (X , F) that is isomorphic
to (A, B(A)) with A ∈ B(R) a Borel measurable subset of the reals. This is not
a very strong assumption. For example, (Rn , B(Rn )) is a Borel space, along with
all of its measurable subsets.

Theorem 3.1. Let µ be a probability measure on a Borel space S and λ be the

Lebesgue measure on ([0, 1], B([0, 1]). Then there exists a sequence of independent
random elements X1 , X2 , . . . on ([0, 1], B([0, 1]), λ) such that the law λXt = µ for
all t.

We give a sketch of the proof because, although it is not really relevant for
the material in this book, it illustrates the general picture and dispels some of
the mystic about what is really going on. Exercise 3.1 asks you to provide the
missing steps from the proof.

Proof sketch of Theorem 3.1 For simplicity we consider only the case that
S = ([0, 1], B([0, 1])) and µ is the Lebesgue measure. For any x ∈ [0, 1], let
F1 (x), F2 (x), . . . be the binary expansion of x, which is the unique binary-valued
 3.1 Stochastic Processes 47

infinite sequence such that

∞
X
x= Ft (x)2−t .
t=1

We can view F1 , F2 , . . . as (binary-valued) random variables over the probability

space ([0, 1], B([0, 1]), λ). Viewed as such, a direct calculation shows that
F1 , F2 , . . . are independent. From this we can create an infinite sequence of
uniform random variables by reversing the process. To do this, we rearrange the
(Ft )∞
t=1 sequence into a grid. For example:

F1 , F2 , F4 , F7 , · · ·
F3 , F5 , F8 , · · ·
F6 , F9 , · · ·
F10 , · · ·
..
.
Letting Xm,t be the tth entry in the mth row of this grid, we define Xm =
P∞ −t
t=1 2 Xm,t , and again one can easily check that with this choice the sequence
X1 , X2 , . . . is independent and λXt = µ is uniform for each t.

3.1 Stochastic Processes

Let T be an arbitrary set. A stochastic process on probability space (Ω, F, P)

is a collection of random variables {Xt : t ∈ T }. In this book T will always
be countable, and so in the following we restrict ourselves to T = N. The first
theorem is not the most general, but suffices for our purposes and is more easily
stated than more generic alternatives.
Theorem 3.2. For each n ∈ N+ , let (Ωn , Fn ) be a Borel space and µn be a
measure on (Ω1 × · · · × Ωn , F1 ⊗ · · · ⊗ Fn ) and assume that µn and µn+1 are
related through
µn+1 (A × Ωn+1 ) = µn (A) for all A ∈ Ω1 ⊗ · · · ⊗ Ωn . (3.1)
Then there exists a probability space (Ω, F, P) and random elements X1 , X2 , . . .
with Xt : Ω → Ωt such that PX1 ,...,Xn = µn for all n.

Sequences of measures (µn )n satisfying Eq. (3.1) are called projective.

Theorem 3.1 follows immediately from Theorem 3.2. By assumption a random

variable takes values in (R, B(R)), which is Borel. Then let µn = ⊗nt=1 µ be
the n-fold product measure of µ with itself. That this sequence of measures is
projective is clear, and the theorem does the rest.
 3.2 Markov Chains 48

3.2 Markov Chains

A Markov chain is an infinite sequence of random elements (Xt )∞ t=1 where the
conditional distribution of Xt+1 given X1 , . . . , Xt is the same as the conditional
distribution of Xt+1 given Xt . The sequence has the property that given the last
element, the history is irrelevant to ‘predict’ the future. Such random sequences
appear throughout probability theory and have many applications besides. The
theory is too rich to explain in detail, so we give the basics and point towards the
literature for more details at the end. The focus here is mostly on the definition
and existence of Markov chains.
Let (X , F) and (Y, G) be measurable spaces. A probability kernel or Markov
kernel from (X , F) to (Y, G) is a function K : X × G → [0, 1] such that
(a) K(x, ·) is a measure for all x ∈ X ; and
(b) K(·, A) is F-measurable for all A ∈ G.
The idea here is that K describes a stochastic transition. Having arrived at x, a
process’s next state is sampled Y ∼ K(x, ·). Occasionally, we will use the notation
Kx (A) or K(A | x) rather than K(x, A).
If K1 is a (X , F) → (Y, G) probability kernel and K2 is a (Y, G) → (Z, H)
probability kernel, then the product kernel K1 ⊗ K2 is the probability kernel
from (X , F) → (Y × Z, G ⊗ H) defined by
Z Z
(K1 ⊗ K2 )(x, A) = IA ((y, z))K2 (y, dz)K1 (x, dy) .
Y Z

When P is a measure on (X , F) and K is a kernel from X to Y, then P ⊗ K is a

measure on (X × Y, F ⊗ G) defined by
Z Z
(P ⊗ K)(A) = IA ((x, y))K(x, dy)dP (x) .
X Y

These operations can be composed. When P is a probability measure on X and

K1 a kernel from X to Y and K2 a kernel from X × Y to Z, then P ⊗ K1 ⊗ K2
is a probability measure on X × Y × Z. The following provides a counterpart of
Theorem 3.2.
Theorem 3.3 (Ionescu–Tulcea). Let (Ωn , Fn )∞
n=1 be a sequence of measurable
spaces and K1 be a probability measure on (Ω1 , F1 ). For n ≥ 2, let Kn
Qn−1
be a probability kernel from t=1 Ωt to Ωn . Then there exists a probability
space (Ω, F, P) and random elements (Xt )∞
t=1 with Xt : Ω → Ωt such that
Nn
PX1 ,...,Xn = t=1 Kt for all n ∈ N+ .
A homogeneous Markov chain is a sequence of random elements (Xt )∞
t=1
taking values in state space S = (X , F) and with
P (Xt+1 ∈ · | X1 , . . . , Xt ) = P (Xt+1 ∈ · | Xt ) = µ(Xt , ·) almost surely ,
where µ is a probability kernel from (X , F) to (X , F) and we assume that
P (X1 ∈ ·) = µ0 (·) for some measure µ0 on (X , F).
 3.3 Martingales and Stopping Times 49

The word ‘homogeneous’ refers to the fact that the probability kernel does
not change with time. Accordingly, sometimes one writes ‘time homogeneous’
instead of homogeneous. The reader can no doubt see how to define a Markov
chain where µ depends on t, though doing so is purely cosmetic since the
state space can always be augmented to include a time component.

Note that if µ(x | ·) = µ0 (·) for all x ∈ X , then Theorem 3.3 is yet another
way to prove the existence of an infinite sequence of independent and identically
distributed random variables. The basic questions in Markov chains resolve around
understanding the evolution of Xt in terms of the probability kernel. For example,
assuming that Ωt = Ω1 for all t ∈ N+ , does the law of Xt converge to some fixed
distribution as t → ∞, and if so, how fast is this convergence? For now we make
do with the definitions, but in the special case that X is finite, we will discuss
some of these topics much later in Chapters 37 and 38.

3.3 Martingales and Stopping Times

Let X1 , X2 , . . . be a sequence of random variables on (Ω, F, P) and F = (Ft )nt=1

a filtration of F and where we allow n = ∞. Recall that the sequence (Xt )nt=1 is
F-adapted if Xt is Ft -measurable for all 1 ≤ t ≤ n.

Definition 3.4. A F-adapted sequence of random variables (Xt )t∈N+ is a F-

adapted martingale if

(a) E[Xt | Ft−1 ] = Xt−1 almost surely for all t ∈ {2, 3, . . .}; and
(b) Xt is integrable.

If the equality is replaced with a less-than (greater-than), then we call (Xt )t a

supermartingale (respectively, a submartingale).

The time index t need not run over N+ . Very often t starts at zero instead.

Example 3.5. A gambler repeatedly throws a coin, winning a dollar for each
heads and losing a dollar for each tails. Their total winnings over time is a
martingale. To model this situation, let Y1 , Y2 , . . . be a sequence of independent
Rademacher distributions, which means that P (Yt = 1) = P (Yt = −1) = 1/2.
Pt
The winnings after t rounds is St = s=1 Ys , which is a martingale adapted to
the filtration (Ft )∞
t=1 given by Ft = σ(Y1 , . . . , Yt ). The definition of super/sub-
martingales (the direction of inequality) can be remembered by remembering
that the definition favors the casino, not the gambler.

Can a gambler increase its expected winning by stopping cleverly? Precisely,

 3.3 Martingales and Stopping Times 50

the gambler at the end of round t can decide to stop (δt = 1) or continue (δt = 0)
based on the information available to them. Denoting by τ = min{t : δt = 1}
the time when the gambler stops, the question is whether by a clever choice of
(δt )t∈N , E [Sτ ] can be made positive. Here, (δt )t∈N , a sequence of binary, F-adapted
random variables, is called a stopping rule, while τ is a stopping time with
respect F.

Note that the stopping rule is not allowed to inject additional randomness
beyond what is already there in F.

Definition 3.6. Let F = (Ft )t∈N be a filtration. A random variable τ with values
in N ∪ {∞} is a stopping time with respect to F if I {τ ≤ t} is Ft -measurable
for all t ∈ N. The σ-algebra at stopping time τ is
Fτ = {A ∈ F∞ : A ∩ {τ ≤ t} ∈ Ft for all t} .

The filtration is usually indicated by writing ‘τ is a F-stopping time’. When

the underlying filtration is obvious from context, it may be omitted. This is
also true for martingales.

Using the interpretation of σ-algebras encoding information, if (Ft )t is thought

of as the knowledge available at time t, Fτ is the information available at the
random time τ . Exercise 3.7 asks you to explore properties of stopped σ-algebras;
amongst other things, it asks you to show that Fτ is in fact a σ-algebra.
Example 3.7. In the gambler example, the first time when the gambler’s
winnings hits 100 is a stopping time: τ = min{t : St = 100}. On the other
hand, τ = min{t : St+1 = −1} is not a stopping time because I {τ = t} is not
Ft -measurable.
Whether or not E [Sτ ] can be made positive by a clever choice of a stopping
time τ is answered in the negative by a fundamental theorem of Doob:
Theorem 3.8 (Doob’s optional stopping). Let F = (Ft )t∈N be a filtration and
(Xt )t∈N be an F-adapted martingale and τ an F-stopping time such that at least
one of the following holds:
(a) There exists an n ∈ N such that P (τ > n) = 0.
(b) E[τ ] < ∞, and there exists a constant c ∈ R such that for all t ∈ N,
E[|Xt+1 − Xt | | Ft ] ≤ c almost surely on the event that τ > t.
(c) There exists a constant c such that |Xt∧τ | ≤ c almost surely for all t ∈ N.
Then Xτ is almost surely well defined, and E[Xτ ] = E[X0 ]. Furthermore, when
(Xt ) is a super/sub-martingale rather than a martingale, then equality is replaced
with less/greater-than, respectively.
 3.3 Martingales and Stopping Times 51

The theorem implies that if Sτ is almost-surely well defined then either

E [τ ] = ∞ or E [Sτ ] = 0. Gamblers trying to outsmart the casino would need to
live a very long life! One application of Doob’s optional stopping theorem is a
useful and a priori surprising generalisation of Markov’s inequality to non-negative
supermartingales.

Theorem 3.9 (Maximal inequality). Let (Xt )∞ t=0 be a supermartingale with

Xt ≥ 0 almost surely for all t. Then for any ε > 0,
 
E[X0 ]
P sup Xt ≥ ε ≤ .
t∈N ε

Proof Let An be the event that supt≤n Xt ≥ ε and τ = (n + 1) ∧ min{t ≤

n : Xt ≥ ε}, where the minimum of an empty set is assumed to be infinite so
that τ = n + 1 if Xt < ε for all 0 ≤ t ≤ n. Clearly τ is a stopping time and
P (τ ≤ n + 1) = 1. Then by Theorem 3.8 and elementary calculation,

E[X0 ] ≥ E[Xτ ] ≥ E[Xτ I {τ ≤ n}] ≥ E[εI {τ ≤ n}] = εP (τ ≤ n) = εP (An ) ,

where the second inequality uses the definition of the stopping time and the non-
negativity of the supermartingale. Rearranging shows that P (An ) ≤ E[X0 ]/ε for
all n ∈ N. Since A1 ⊆ A2 ⊆ . . ., it follows that P (supt∈N Xt ≥ ε) = P (∪n∈N An ) ≤
E[X0 ]/ε.

Markov’s inequality (which we will cover in the next chapter) combined with
the definition of a supermartingale shows that
E[X0 ]
P (Xn ≥ ε) ≤ . (3.2)
ε
In fact, in the above we have effectively applied Markov’s inequality to the
random variable Xτ (the need for the proof arises when the conditions of
Doob’s optional sampling theorem are not met). The maximal inequality is
a strict improvement over Eq. (3.2) by replacing Xn with supt∈N Xt at no
cost whatsoever.

A similar theorem holds for submartingales. You will provide a proof in

Exercise 3.8.

Theorem 3.10. Let (Xt )nt=0 be a submartingale with Xt ≥ 0 almost surely for
all t. Then for any ε > 0,
 
E[Xn ]
P max Xt ≥ ε ≤ .
t∈{0,1,...,n} ε
 3.4 Notes 52

3.4 Notes

1 Some authors include in the definition of a stopping time τ that P (τ < ∞) = 1

and call random times without this property Markov times. We do not adopt
this convention and allow stopping times to be infinite with non-zero probability.
Stopping times are also called optional times.
2 There are several notations for probability kernels depending on the application.
The following are commonly seen and equivalent: K(x, A) = K(A | x) = Kx (A).
For example, in statistics a parametric family is often given by {Pθ : θ ∈ Θ},
where Θ is the parameter space and Pθ is a measure on some measurable space
(Ω, F). This notation is often more convenient than writing P(θ, ·). In Bayesian
statistics the posterior is a probability kernel from the observation space to
the parameter space, and this is often written as P(· | x).
3 There is some disagreement about whether or not a Markov chain on an
uncountable state space should instead be called a Markov process. In this
book we use Markov chain for arbitrary state spaces and discrete time. When
time is continuous (which it never is in this book), there is general agreement
that ‘process’ is more appropriate. For more history on this debate, see [Meyn
and Tweedie, 2012, preface].
4 A topological space X is Polish if it is separable and there exists a metric
d that is compatible with the topology that makes (X , d) a complete metric
space. All Polish spaces are Borel spaces. We follow Kallenberg [2002], but
many authors use standard Borel space rather than Borel space, and define
it as the σ-algebra generated by the open sets of a Polish space.
5 In Theorem 3.2 it was assumed that each µn was defined on a Borel space.
No such assumption was required for Theorem 3.3, however. One can derive
Theorem 3.2 from Theorem 3.3 by using the existence of regular conditional
probability measures when conditioning on random elements taking values
in a Borel space (see the next note). Topological assumptions often creep
into foundational questions relating to the existence of probability measures
satisfying certain conditions, and pathological examples show these assumptions
cannot be removed completely. Luckily, in this book we have no reason to
consider random elements that do not take values in a Borel space.
6 The fact that conditional expectation is only unique almost surely can be
problematic when you want a conditional distribution. Given random elements
X and Y on the same probability space, it seems reasonable to hope that
P (X ∈ · | Y ) is a probability kernel from the space of Y to that of X. A
version of the conditional distributions that satisfies this is called a regular
version. In general, there is no guarantee that such a regular version exist. The
basic properties of conditional expectation only guarantee that for any fixed
measurable A, P (X ∈ A | Y ) is unique up to a set of measure zero. The set of
measure zero can depend on A, which causes problems when there are ‘too
many’ measurable sets in the space of X. Assuming X lives in a Borel space,
the following theorem guarantees the existence of a conditional distribution.
 3.5 Bibliographic Remarks 53

Theorem 3.11 (Regular conditional distributions). Let X and Y be random

elements on the same probability space (Ω, F, P) taking values in measurable
spaces X and Y, and assume that X is Borel. Then there exists a probability
kernel K from Y to X such that K(· | Y ) = P (X ∈ · | Y ) P-almost surely.
Furthermore, K is unique in the sense that for any kernels K1 and K2 satisfying
this condition, it holds that K1 (· | y) = K2 (· | y) for all y in some set of PY -
measure one.

The theorem implies the useful relation that PX,Y = PY ⊗K (cf. Exercise 3.9)
where recall that for a random variable Z, PZ denotes its pushforward under
P. To make the origin K clear, we often write PX|Y instead of K. With this,
the above equality becomes PX,Y = PY ⊗ PX|Y , which can be viewed as the
converse of the Ionescu–Tulcea theorem (Theorem 3.3). Sometimes this is called
the chain rule of probabilities measures.
You can also condition on a σ-algebra G ⊂ F, in which case K is a probability
kernel from (Ω, G) to X . The condition that X be Borel is sufficient, but not
necessary. Some conditions are required, however. An example where no regular
version exists can be found in [Halmos, 1976, p210]. Regular versions play
a role in the following useful theorem for decomposing random variables on
product spaces.

Theorem 3.12 (Disintegration). Let X and Y be random elements on the

same probability space taking values in measurable spaces X and Y. Let f be
a random variable on X × Y so that E[|f (X, Y )|] < ∞. Suppose that K is a
regular version of P (X ∈ · | G) and Y is G-measurable. Then,
Z
E[f (X, Y ) | G] = f (x, Y )K(dx | ·) almost surely .
X

In many applications
R G = σ(Y ), in which case the theorem says that
E[f (X, Y ) | Y ] = X f (x, Y )K(dx | Y ) almost surely. Proofs of both theorems
appear in chapter 6 of Kallenberg [2002].

3.5 Bibliographic Remarks

There are many places to find the construction of a stochastic process. Like before,
we recommend Kallenberg [2002] for readers who want to refresh their memory
and Billingsley [2008] for a more detailed account. For Markov chains the recent
book by Levin and Peres [2017] provides a wonderful introduction. After reading
that, you might like the tome by Meyn and Tweedie [2012]. Theorem 3.1 can be
found as theorem 3.19 in the book by Kallenberg [2002], where the reader can also
find its proof. Theorem 3.2 is credited to Percy John Daniell by Kallenberg [2002]
(see Aldrich 2007). More general versions of this theorem exist. Readers looking
for these should look up Kolmogorov’s extension theorem [Kallenberg, 2002,
theorem 6.16]. The theorem of Ionescu–Tulcea (Theorem 3.3) is attributed to him
 3.6 Exercises 54

[Ionescu Tulcea, 1949–50] with a modern proof in the book by Kallenberg [2002,
theorem 6.17]. There are lots of minor variants of the optional stopping theorem,
most of which can be found in any probability book featuring martingales. The
most historically notable source is by the man himself [Doob, 1953]. A more
modern book that also gives the maximal inequalities is the book on optimal
stopping by Peskir and Shiryaev [2006].

3.6 Exercises

3.1 Fill in the details of Theorem 3.1:

(a) Prove that Ft ∈ {0, 1} is a Bernoulli random variable for all t ≥ 1.
(b) In what follows, equip S with P = λ, the uniform probability measure. Show
that for any t ≥ 1, Ft is uniformly distributed: P (Ft = 0) = P (Ft = 1) = 1/2.
(c) Show that (Ft )∞t=1 are independent.
(d) Show that (Xm,t )∞ t=1 is an independent sequence of Bernoulli random
variables that are uniformly distributed.
P∞
(e) Show that Xt = t=1 Xm,t 2−t is uniformly distributed on [0, 1].
(f) Show that (Xt )∞ t=1 are independent.

3.2 (Martingales and optional stopping) Let (Xt )∞ t=1 be an infinite

Pt
sequence of independent Rademacher random variables and St = s=1 Xs 2s−1 .
(a) Show that (St )∞
t=0 is a martingale.
(b) Let τ = min{t : St = 1} and show that P (τ < ∞) = 1.
(c) What is E[Sτ ]?
(d) Explain why this does not contradict Doob’s optional stopping theorem.

3.3 (Martingales and optional stopping (ii)) Give an example of a

martingale (Sn )∞
n=0 and stopping time τ such that

lim E[Sτ ∧n ] 6= E[Sτ ] .

n→∞

3.4 (Maximal inequality fails without non-negativity) Show that

Theorem 3.9 does not hold in general for supermartingales if the assumption that
it be non-negative is dropped.

3.5 Let (Ω, F) and (X , G) be measurable spaces, X : X → R be a random variable

and K : Ω × G → [0, 1] aR probability kernel from (Ω, F) to (X , G). Define function
U : Ω → R by U (ω) = X X(x)K(ω, dx) and assume that U (ω) exists for all ω.
Prove that U is F-measurable.

3.6 (Limits of increasing stopping times are stopping times) Let (τn )∞ n=1
be an almost surely increasing sequence of F-stopping times on probability space
(Ω, F, P) with filtration F = (Fn )∞
n=1 , which means that τn (ω) ≤ τn+1 (ω) for all
n ≥ 1 almost surely. Prove that τ (ω) = limn→∞ τn (ω) is a F-stopping time.
 3.6 Exercises 55

3.7 (Properties of stopping times) Let F = (Ft )t∈N be a filtration, and

τ, τ1 , τ2 be stopping times with respect to F. Show the following:
(a) Fτ is a σ-algebra.
(b) If τ = k for some k ≥ 1, then Fτ = Fk .
(c) If τ1 ≤ τ2 , then Fτ1 ⊂ Fτ2 .
(d) τ is Fτ -measurable.
(e) If (Xt ) is F-adapted, then Xτ is Fτ -measurable.
(f) Fτ is the smallest σ-algebra such that all F-adapted sequences (Xt ) satisfy
Xτ is Fτ -measurable.

3.8 Prove Theorem 3.10.

3.9 (Decomposing joint distributions) Let X and Y be random elements

on the same probability space (Ω, F, P) taking values in measurable spaces X
and Y respectively and assume that X is Borel. Show that P(X,Y ) = PY ⊗ PX|Y
where PX|Y denotes a regular conditional distribution of X and Y (the existence
of which is guaranteed by Theorem 3.11).
4 Stochastic Bandits

The goal of this chapter is to formally introduce stochastic bandits. The model
introduced here provides the foundation for the remaining chapters that treat
stochastic bandits. While the topic seems a bit mundane, it is important to be
clear about the assumptions and definitions. The chapter also introduces and
motivates the learning objectives, and especially the regret. Besides the definitions,
the main result in this chapter is the regret decomposition, which is presented in
Section 4.5.

4.1 Core Assumptions

A stochastic bandit is a collection of distributions ν = (Pa : a ∈ A), where A is

the set of available actions. The learner and the environment interact sequentially
over n rounds. In each round t ∈ {1, . . . , n}, the learner chooses an action At ∈ A,
which is fed to the environment. The environment then samples a reward Xt ∈ R
from distribution PAt and reveals Xt to the learner. The interaction between
the learner (or policy) and environment induces a probability measure on the
sequence of outcomes A1 , X1 , A2 , X2 , . . . , An , Xn . Usually the horizon n is finite,
but sometimes we allow the interaction to continue indefinitely (n = ∞). The
sequence of outcomes should satisfy the following assumptions:

(a) The conditional distribution of reward Xt given A1 , X1 , . . . , At−1 , Xt−1 , At

is PAt , which captures the intuition that the environment samples Xt from
PAt in round t.
(b) The conditional law of action At given A1 , X1 , . . . , At−1 , Xt−1 is
πt (· | A1 , X1 , . . . , At−1 , Xt−1 ), where π1 , π2 , . . . is a sequence of probability
kernels that characterise the learner. The most important element of this
assumption is the intuitive fact that the learner cannot use the future
observations in current decisions.

A mathematician might ask whether there even exists a probability space carrying
these random elements such that (a) and (b) hold. Specific constructions showing
this in the affirmative are given in Section 4.6. These constructions are also
valuable because they teach us important lessons about equivalent models. For
now, however, we move on.
 4.2 The Learning Objective 57

4.2 The Learning Objective

Pn
The learner’s goal is to maximise the total reward Sn = t=1 Xt , which is a
random quantity that depends on the actions of the learner and the rewards
sampled by the environment. This is not an optimisation problem for three
reasons:
1 What is the value of n for which we are maximising? Occasionally prior
knowledge of the horizon is reasonable, but very often the learner does not
know ahead of time how many rounds are to be played.
2 The cumulative reward is a random quantity. Even if the reward distributions
were known, then we require a measure of utility on distributions of Sn .
3 The learner does not know the distributions that govern the rewards for each
arm.
Of these points, the last is fundamental to the bandit problem and is discussed
in the next section. The lack of knowledge of the horizon is usually not a serious
issue. Generally speaking it is possible to first design a policy assuming the
horizon is known and then adapt it to account for the unknown horizon while
proving that the loss in performance is minimal. This is almost always quite easy,
and there exist generic approaches for making the conversion.
Assigning a utility to distributions of Sn is more challenging. Suppose
that Sn is the revenue of your company. Fig. 4.1 shows the distribution of
Sn for two different learners; call them A and B. Suppose you can choose
between learners A and B. Which one would you choose? One choice is to
go with the learner whose reward distribution has the larger expected value.
This will be our default choice for
stochastic bandits, but it bears remem-
bering that there are other consider- A B
ations, including the variance or tail
behaviour of the cumulative reward,
Density

which we will discuss occasionally. In

particular, in the situation shown on
in Fig. 4.1, learner B achieves a higher
expected reward than A. However B
has a reasonable probability of earning
Reward
less than the least amount that A can
earn, so a risk-sensitive user may prefer Figure 4.1 Alternative revenue distributions
learner A.

4.3 Knowledge and Environment Classes

Even if the horizon is known in advance and we commit to maximising the expected
value of Sn , there is still the problem that the bandit instance ν = (Pa : a ∈ A) is
 4.3 Knowledge and Environment Classes 58

unknown. A policy that maximises the expectation of Sn for one bandit instance
may behave quite badly on another. The learner usually has partial information
about ν, which we represent by defining a set of bandits E for which ν ∈ E is
guaranteed. The set E is called the environment class. We distinguish between
structured and unstructured bandits.

Unstructured Bandits
An environment class E is unstructured if A is finite and there exist sets of
distributions Ma for each a ∈ A such that

E = {ν = (Pa : a ∈ A) : Pa ∈ Ma for all a ∈ A} ,

or, in short, E = ×a∈A Ma . The product structure means that by playing action
a the learner cannot deduce anything about the distributions of actions b 6= a.
Some typical choices of unstructured bandits are listed in Table 4.1. Of course,
these are not the only choices, and the reader can no doubt find ways to construct
more, e.g. by allowing some arms to be Bernoulli and some Gaussian, or have
rewards being exponentially distributed, or Gumbel distributed, or belonging to
your favourite (non-)parametric family.
The Bernoulli, Gaussian and uniform distributions are often used as examples
for illustrating some specific property of learning in stochastic bandit problems.
The Bernoulli distribution is actually a natural choice. Think of applications like
maximising click-through rates in a web-based environment. A bandit problem
is often called a ‘distribution bandit’, where ‘distribution’ is replaced by the
underlying distribution from which the pay-offs are sampled. Some examples
are: Gaussian bandit, Bernoulli bandit or subgaussian bandit. Similarly we say
‘bandits with X’, where ‘X’ is a property of the underlying distribution from
which the pay-offs are sampled. For example, we can talk about bandits with
finite variance, meaning the bandit environment where the a priori knowledge of
the learner is that all pay-off distributions are such that their underlying variance
is finite.
Some environment classes, like Bernoulli bandits, are parametric, while others,
like subgaussian bandits, are non-parametric. The distinction is the number of
degrees of freedom needed to describe an element of the environment class. When
the number of degrees of freedom is finite, it is parametric, and otherwise it is
non-parametric. Of course, if a learner is designed for a specific environment class
E, then we might expect that it has good performance on all bandits ν ∈ E. Some
environment classes are subsets of other classes. For example, Bernoulli bandits
are a special case of bandits with a finite variance, or bandits with bounded
support. Something to keep in mind is that we expect that it will be harder to
achieve a good performance in a larger class. In a way, the theory of finite-armed
stochastic bandits tries to quantify this expectation in a rigourous fashion.
 4.3 Knowledge and Environment Classes 59

Name Symbol Definition

Bernoulli EBk {(B(µi ))i : µ ∈ [0, 1]k }

Uniform EUk {(U(ai , bi ))i : a, b ∈ Rk with ai ≤ bi for all i}

Gaussian (known var.) k

EN (σ 2 ) {(N (µi , σ 2 ))i : µ ∈ Rk }

Gaussian (unknown var.) k

EN {(N (µi , σi2 ))i : µ ∈ Rk and σ 2 ∈ [0, ∞)k }

Finite variance EVk (σ 2 ) {(Pi )i : VX∼Pi [X] ≤ σ 2 for all i}

Finite kurtosis k
EKurt (κ) {(Pi )i : KurtX∼Pi [X] ≤ κ for all i}

Bounded support k
E[a,b] {(Pi )i : Supp(Pi ) ⊆ [a, b]}

Subgaussian k
ESG (σ 2 ) {(Pi )i : Pi is σ-subgaussian for all i}

Table 4.1 Typical environment classes for stochastic bandits. Supp(P ) is the (topological)
support of distribution P . The kurtosis of a random variable X is a measure of its tail
behaviour and is defined by E[(X − E[X])4 ]/V[X]2 . Subgaussian distributions have similar
properties to the Gaussian and will be defined in Chapter 5.

Structured Bandits
Environment classes that are not unstructured are called structured. Relaxing the
requirement that the environment class is a product set makes structured bandit
problems much richer than the unstructured set-up. The following examples
illustrate the flexibility.

Example 4.1. Let A = {1, 2} and E = {(B(θ), B(1 − θ)) : θ ∈ [0, 1]}. In this
environment class, the learner does not know the mean of either arm, but can
learn the mean of both arms by playing just one. The knowledge of this structure
dramatically changes the difficulty of learning in this problem.

Example 4.2 (Stochastic linear bandit). Let A ⊂ Rd and θ ∈ Rd and

νθ = (N (ha, θi, 1) : a ∈ A) and E = {νθ : θ ∈ Rd } .

In this environment class, the reward of an action is Gaussian, and its mean is given
by the inner product between the action and some unknown parameter. Notice
that even if A is extremely large, the learner can deduce the true environment
by playing just d actions that span Rd .

Example 4.3. Consider an undirected graph G with vertices V = {1, . . . , |V |}

and edges E = {1, . . . , |E|}. In each round the learner chooses a path from
vertex 1 to vertex |V |. Then each edge e ∈ [E] is removed from the graph with
probability 1 − θe for unknown θ ∈ [0, 1]|E| . The learner succeeds in reaching
their destination if all the edges in their chosen path are present. This problem
 4.4 The Regret 60

can be formalised by letting A be the set of paths and

! !
Y
νθ = B θe : a ∈ A and E = {νθ : θ ∈ [0, 1]|E| } .
e∈a

An important feature of structured bandits is that the learner can often

obtain information about some actions while never playing them.

4.4 The Regret

In Chapter 1 we informally defined the regret as being the deficit suffered by the
learner relative to the optimal policy. Let ν = (Pa : a ∈ A) be a stochastic bandit
and define
Z ∞
µa (ν) = x dPa (x) .
−∞

Then let µ (ν) = maxa∈A µa (ν) be the largest mean of all the arms.
∗

We assume throughout that µa (ν) exists and is finite for all actions and
that argmaxa∈A µa (ν) is non-empty. The latter assumption could be relaxed
by carefully adapting all arguments using nearly optimal actions, but in
practice this is never required.

The regret of policy π on bandit instance ν is

" n #
X
Rn (π, ν) = nµ (ν) − E
∗
Xt , (4.1)
t=1

where the expectation is taken with respect to the probability measure on

outcomes induced by the interaction of π and ν. Minimising the regret is equivalent
to maximising the expectation of Sn , but the normalisation inherent in the
definition of the regret is useful when stating results, which would otherwise need
to be stated relative to the optimal action.

If the context is clear, we will often drop the dependence on ν and π in various
Pn
quantities. For example, by writing Rn = nµ∗ − E[ t=1 Xt ]. Similarly, the
limits in sums and maxima are abbreviated when we think you can work
out ranges of symbols in a unique way, e.g. µ∗ = maxi µi .

The regret is always non-negative, and for every bandit ν, there exists a policy
π for which the regret vanishes.
 4.4 The Regret 61

Lemma 4.4. Let ν be a stochastic bandit environment. Then,

(a) Rn (π, ν) ≥ 0 for all policies π;
(b) the policy π choosing At ∈ argmaxa µa for all t satisfies Rn (π, ν) = 0; and
(c) if Rn (π, ν) = 0 for some policy π, then P (µAt = µ∗ ) = 1 for all t ∈ [n].
We leave the proof for the reader (Exercise 4.1). Part (b) of Lemma 4.4 shows
that for every bandit ν, there exists a policy for which the regret is zero (the best
possible outcome). According to Part (c), achieving zero is possible if and only if
the learner knows which bandit it is facing (or at least, what is the optimal arm).
In general, however, the learner only knows that ν ∈ E for some environment
class E. So what can we hope for? A relatively weak objective is to find a policy
π with sublinear regret on all ν ∈ E. Formally, this objective is to find a policy π
such that
Rn (π, ν)
for all ν ∈ E , lim = 0.
n→∞ n
If the above holds, then at least the learner is choosing the optimal action almost
all of the time as the horizon tends to infinity. One might hope for much more,
however, for example, that for some specific choice of C > 0 and p < 1 that
for all ν ∈ E , Rn (π, ν) ≤ Cnp . (4.2)
Yet another alternative is to find a function C : E → [0, ∞) and f : N → [0, ∞)
such that
for all n ∈ N, ν ∈ E , Rn (π, ν) ≤ C(ν)f (n) . (4.3)
This factorisation of the regret into a function of the instance and a function
of the horizon is not uncommon in learning theory and appears in particular in
supervised learning.
We will spend a lot of time in the following chapters finding policies satisfying
Eq. (4.2) and Eq. (4.3) for different choices of E. The form of Eq. (4.3) is quite
general, so much time is also spent discovering what are the possibilities for f and
C, both of which should be ‘as small as possible’. All of the policies are inspired
by the simple observation that in order to make the regret small, the algorithm
must discover the action/arm with the largest mean. Usually this means the
algorithm should play each arm some number of times to form an estimate of
the mean of that arm, and subsequently play the arm with the largest estimated
mean. The question essentially boils down to discovering exactly how often the
learner must play each arm in order to have reasonable statistical certainty that
it has found the optimal arm.
There is another candidate objective called the Bayesian regret. If Q is a
prior probability measure on E (which must be equipped with a σ-algebra F),
then the Bayesian regret is the average of the regret with respect to the prior Q.
Z
BRn (π, Q) = Rn (π, ν) dQ(ν) , (4.4)
E
 4.5 Decomposing the Regret 62

which is only defined by assuming (or proving) that the regret is a measurable
function with respect to F. An advantage of the Bayesian approach is that
having settled on a prior and horizon, the problem of finding a policy that
minimises the Bayesian regret is just an optimisation problem. Most of this
book is devoted to analyzing the ‘frequentist’ regret in Eq. (4.1), which does not
integrate over all environments as Eq. (4.4) does. Bayesian methods are covered
in Chapters 34 to 36, where we also discuss the strengths and weaknesses of the
Bayesian approach.

4.5 Decomposing the Regret

We now present a lemma that forms the basis of almost every proof for
stochastic bandits. Let ν = (Pa : a ∈ A) be a stochastic bandit and define
∆a (ν) = µ∗ (ν) − µa (ν), which is called the suboptimality gap or action gap
or immediate regret of action a. Further, let
t
X
Ta (t) = I {As = a}
s=1

be the number of times action a was chosen by the learner after the end of round
t. In general, Ta (n) is random, which may seem surprising if we think about a
deterministic policy that chooses the same action for any fixed history. So why
is Ta (n) random in this case? The reason is because for all rounds t except for
the first, the action At depends on the rewards observed in rounds 1, 2, . . . , t − 1,
which are random, hence At will also inherit their randomness. We are now ready
to state the second and last lemmas of the chapter. In the statement of the lemma,
we use our convention that the dependence of the various quantities involved on
the policy π and the environment ν is suppressed.
Lemma 4.5 (Regret decomposition lemma). For any policy π and stochastic
bandit environment ν with A finite or countable and horizon n ∈ N, the regret
Rn of policy π in ν satisfies
X
Rn = ∆a E [Ta (n)] . (4.5)
a∈A

The lemma decomposes the regret in terms of the loss due to using each of the
arms. It is useful because it tells us that to keep the regret small, the learner
should try to minimise the weighted sum of expected action counts, where the
weights are the respective suboptimality gaps, (∆a )a∈A .

Lemma 4.5 tells us that a learner should aim to use an arm with a larger
suboptimality gap proportionally fewer times.

Note that the suboptimality gap for optimal arm(s) is zero.

 4.6 The Canonical Bandit Model ( ) 63

Proof of Lemma 4.5 Since Rn is based on summing over rounds, and the right-
hand side of the lemma statement is based on summing over actions, to convert one
sum into the other one, we introduce indicators. In particular, note that for any
P P P P
fixed t we have a∈A I {At = a} = 1. Hence Sn = t Xt = t a Xt I {At = a},
and thus
n
XX
Rn = nµ∗ − E [Sn ] = E [(µ∗ − Xt )I {At = a}] . (4.6)
a∈A t=1

The expected reward in round t conditioned on At is µAt , which means that

E [(µ∗ − Xt )I {At = a} | At ] = I {At = a} E [µ∗ − Xt | At ]

= I {At = a} (µ∗ − µAt )
= I {At = a} (µ∗ − µa )
= I {At = a} ∆a .

The result is completed by plugging this into Eq. (4.6) and using the definition
of Ta (n).

The argument fails when A is uncountable because you cannot introduce the
sum over actions. Of course the solution is to use an integral, but for this we need
to assume (A, G) is a measurable space. Given a bandit ν and policy π define
measure G on (A, G) by
" n #
X
G(U ) = E I {At ∈ U } ,
t=1

where the expectation is taken with respect to the measure on outcomes induced
by the interaction of π and ν.

Lemma 4.6. Provided that everything is well defined and appropriately measurable,
" n # Z
X
Rn = E ∆At = ∆a dG(a) .
t=1 A

For those worried about how to ensure everything is well defined, see Section 4.7.

4.6 The Canonical Bandit Model ( )

In most cases the underlying probability space that supports the random rewards
and actions is never mentioned. Occasionally, however, it becomes convenient to
choose a specific probability space, which we call the canonical bandit model.
 4.6 The Canonical Bandit Model ( ) 64

Finite Horizon
Let n ∈ N be the horizon. A policy and bandit interact to produce the outcome,
which is the tuple of random variables Hn = (A1 , X1 , . . . , An , Xn ). The first step
towards constructing a probability space that carries these random variables
is to choose the measurable space. For each t ∈ [n], let Ωt = ([k] × R)t ⊂ R2t
and Ft = B(Ωt ). The random variables A1 , X1 , . . . , An , Xn that make up the
outcome are defined by their coordinate projections:
At (a1 , x1 , . . . , an , xn ) = at and Xt (a1 , x1 , . . . , an , xn ) = xt .
The probability measure on (Ωn , Fn ) depends on both the environment and the
policy. Our informal definition of a policy is not quite sufficient now.
Definition 4.7. A policy π is a sequence (πt )nt=1 , where πt is a probability
kernel from (Ωt−1 , Ft−1 ) to ([k], 2[k] ). Since [k] is discrete, we adopt the notational
convention that for i ∈ [k],
πt (i | a1 , x1 , . . . , at−1 , xt−1 ) = πt ({i} | a1 , x1 , . . . , at−1 , xt−1 ) .
Let ν = (Pi )ki=1 be a stochastic bandit where each Pi is a probability measure
on (R, B(R)). We want to define a probability measure on (Ωn , Fn ) that respects
our understanding of the sequential nature of the interaction between the learner
and a stationary stochastic bandit. Since we only care about the law of the
random variables (Xt ) and (At ), the easiest way to enforce this is to directly list
our expectations, which are
(a) the conditional distribution of action At given A1 , X1 , . . . , At−1 , Xt−1 is
πt ( · | A1 , X1 , . . . , At−1 , Xt−1 ) almost surely.
(b) the conditional distribution of reward Xt given A1 , X1 , . . . , At is PAt almost
surely.
The sufficiency of these assumptions is asserted by the following proposition,
which we ask you to prove in Exercise 4.2.
Proposition 4.8. Suppose that P and Q are probability measures on an arbitrary
measurable space (Ω, F) and A1 , X1 , . . . , An , Xn are random variables on Ω, where
At ∈ [k] and Xt ∈ R. If both P and Q satisfy (a) and (b), then the law of the
outcome under P is the same as under Q:
PA1 ,X1 ,...,An ,Xn = QA1 ,X1 ,...,An ,Xn .
Next we construct a probability measure on (Ωn , Fn ) that satisfies (a) and
(b). To emphasise that what follows is intuitively not complicated, imagine that
Xt ∈ {0, 1} is Bernoulli, which means the set of possible outcomes is finite and
we can define the measure in terms of a distribution. Let pi (0) = Pi ({0}) and
pi (1) = 1 − pi (0) and define
n
Y
pνπ (a1 , x1 , . . . , an , xn ) = π(at | a1 , x1 , . . . , at−1 , xt−1 )pat (xt ) .
t=1
 4.6 The Canonical Bandit Model ( ) 65

The reader can check that pνπ is a distribution on ([k] × {0, 1})n and that the
associated measure satisfies (a) and (b) above. Making this argument rigourous
when (Pi ) are not discrete requires the use of Radon–Nikodym derivatives. Let λ
be a σ-finite measure on (R, B(R)) for which Pi is absolutely continuous with
respect to λ for all i. Next, let pi = dPi /dλ be the Radon–Nikodym
R derivative of
Pi with respect to λ, which is a function pi : R → R such that B pi dλ = Pi (B)
for all B ∈ B(R). Letting ρ be the counting measure with ρ(B) = |B|, the density
pνπ : Ω → R can now be defined with respect to the product measure (ρ × λ)n
by

n
Y
pνπ (a1 , x1 , . . . , an , xn ) = π(at | a1 , x1 , . . . , at−1 , xt−1 )pat (xt ) . (4.7)
t=1

The reader can again check (more abstractly) that (a) and (b) are satisfied by
the probability measure Pνπ defined by
Z
Pνπ (B) = pνπ (ω)(ρ × λ)n (dω) for all B ∈ Fn .
B

It is important to emphasise that this choice of (Ωn , Fn , Pνπ ) is not unique. Instead,
all that this shows is that a suitable probability space does exist. Furthermore, if
some quantity of interest depends on the law of Hn , by Proposition 4.8, there is
no loss in generality in choosing (Ωn , Fn , Pνπ ) as the probability space.

Pk
A choice of λ such that Pi  λ for all i always exists since λ = i=1 Pi
satisfies this condition. For direct calculations, another choice is usually
more convenient, e.g. the counting measure when (Pi ) are discrete and the
Lebesgue measure for continuous (Pi ).

There is another way to define the probability space, which can be useful.
Define a collection of independent random variables (Xsi )s∈[n],i∈[k] such that the
law of Xti is Pi . By Theorem 2.4 these random variables may be defined on
(Ω, F), where Ω = Rnk and F = B(Rnk ). Then let Xt = XtAt , where the actions
At are Ft−1 -measurable with Ft−1 = σ(A1 , X1 , . . . , At−1 , Xt−1 ). We call this the
random table model. Yet another way is to define (Xsi )s,i as above but let
Xt = XTAt (t),At . This corresponds to sampling a stack of rewards for each arm
at the beginning of the game, giving rise to the reward-stack model. Each time
the learner chooses an action, they receive the reward on top of the stack. All of
these models are convenient from time to time. The important thing is that it
does not matter which model we choose because the quantity of ultimate interest
(usually the regret) only depends on the law of A1 , X1 , . . . , An , Xn , and this is
the same for all choices.
 4.7 The Canonical Bandit Model for Uncountable Action Sets ( ) 66

Infinite Horizon
We never need the canonical bandit model for the case that n = ∞. It is comforting
to know, however, that there does exist a probability space (Ω, F, Pνπ ) and infinite
sequences of random variables X1 , X2 , . . . and A1 , A2 , . . . satisfying (a) and (b).
The result follows directly from the theorem of Ionescu–Tulcea (Theorem 3.3).

4.7 The Canonical Bandit Model for Uncountable Action Sets ( )

For uncountable action sets, a little more machinery is necessary to make things
rigourous. The first requirement is that the action set must be a measurable
space (A, G) and the collection of distribution ν = (Pa : a ∈ A) that defines a
bandit environment must be a probability kernel from (A, G) to (R, B(R)). A
policy is a sequence (πt )nt=1 , where πt is a probability kernel from (Ωt−1 , Ft−1 )
to (A, G) with
t
Y t
O
Ωt = (A × R) and Ft = (G ⊗ B(R)) .
s=1 s=1

The canonical bandit model is the probability measure Pνπ on (Ωn , Fn )

obtained by taking the product of the probability kernels π1 , P1 , . . . πn , Pn and
using Ionescu–Tulcea (Theorem 3.3), where Pt is the probability kernel from
(Ωt−1 × A, Ft ⊗ G) to (R, B(R)) given by Pt ( · | a1 , x1 , . . . , at−1 , xt−1 , at ) = Pat (·).

We did not define Pνπ in terms of a density because there may not exist a
common dominating measure for either (Pa : a ∈ A) or the policy. When
such measures exist, as they usually do, then Pνπ may be defined in terms
of a density in the same manner as the previous section.

You will check in Exercise 4.5 that the assumptions on ν and π in this section
are sufficient to ensure the quantities in Lemma 4.6 are well defined and that
Proposition 4.8 continues to hold in this setting without modification. Finally, in
none of the definitions above do we require that n be finite.

4.8 Notes

1 It is not obvious why the expected value is a good summary of the reward
distribution. Decision makers who base their decisions on expected values are
called risk-neutral. In the example shown on the figure above, a risk-averse
decision maker may actually prefer the distribution labelled as A because
occasionally distribution B may incur a very small (even negative) reward.
Risk-seeking decision makers, if they exist at all, would prefer distributions
with occasional large rewards to distributions that give mediocre rewards only.
 4.8 Notes 67

There is a formal theory of what makes a decision maker rational (a decision

maker in a nutshell is rational if they do not contradict themself). Rational
decision makers compare stochastic alternatives based on the alternatives’
expected utilities, according to the von-Neumann–Morgenstern utility theorem.
Humans are known not to do this. We are irrational. No surprise here.
2 The study of utility and risk has a long history, going right back to (at least)
the beginning of probability [Bernoulli, 1954, translated from the original
Latin, 1738]. The research can broadly be categorised into two branches. The
first deals with describing how people actually make choices (descriptive
theories), while the second is devoted to characterising how a rational decision
maker should make decisions (prescriptive theories). A notable example
of the former type is ‘prospect theory’ [Kahneman and Tversky, 1979], which
models how people handle probabilities (especially small ones) and earned
Daniel Kahneman a Nobel Prize (after the death of his long-time collaborator,
Amos Tversky). Further descriptive theories concerned with alternative aspects
of human decision-making include bounded rationality, choice strategies,
recognition-primed decision-making and image theory [Adelman, 2013].
3 The most famous example of a prescriptive theory is the von Neumann–
Morgenstern expected utility theorem, which states that under (reasonable)
axioms of rational behaviour under uncertainty, a rational decision maker
must choose amongst alternatives by computing the expected utility of the
outcomes [Neumann and Morgenstern, 1944]. Thus, rational decision makers,
under the chosen axioms, differ only in terms of how they assign utility to
outcomes (i.e. rewards). Finance is another field where attitudes towards
uncertainty and risk are important. Markowitz [1952] argues against expected
return as a reasonable metric that investors would use. His argument is based
on the (simple) observation that portfolios maximising expected returns will
tend to have a single stock only (unless there are multiple stocks with equal
expected returns, a rather unlikely outcome). He argues that such a complete
lack of diversification is unreasonable. He then proposes that investors should
minimise the variance of the portfolio’s return subject to a constraint on the
portfolio’s expected return, leading to the so-called mean-variance optimal
portfolio choice theory. Under this criteria, portfolios will indeed tend to
be diversified (and in a meaningful way: correlations between returns are taken
into account). This theory eventually won him a Nobel Prize in economics
(shared with two others). Closely related to the mean-variance criterion are the
‘value-at-risk’ (VaR) and the ‘conditional value-at-risk’, the latter of which has
been introduced and promoted by Rockafellar and Uryasev [2000] due to its
superior optimisation properties. The distinction between the prescriptive and
descriptive theories is important: human decision makers are in many ways
violating rules of rationality in their attitudes towards risk.
4 We defined the regret as an expectation, which makes it unusable in conjunction
with measures of risk because the randomness has been eliminated by the
expectation. When using a risk measure in a bandit setting, we can either base
 4.9 Bibliographical Remarks 68

this on the random regret or pseudo-regret defined by

n
X
R̂n = nµ∗ − Xt . (random regret)
t=1
Xn
R̄n = nµ∗ − µAt . (pseudo-regret)
t=1

While R̂n is influenced by the noise Xt − µAt in the rewards, the pseudo-regret
filters this out, which arguably makes it a better basis for measuring the ‘skill’
of a bandit policy. As these random regret measures tend to be highly skewed,
using variance to assess risk suffers not only from the problem of penalising
upside risk, but also from failing to capture the skew of the distribution.
5 What happens if the distributions of the arms are changing with time?
Such bandits are unimaginatively called non-stationary bandits. With no
assumptions, there is not much to be done. Because of this, it is usual to
assume the distributions change infrequently or drift slowly. We’ll eventually
see that techniques for stationary bandits can be adapted to this set-up (see
Chapter 31).
6 The rigourous models introduced in Sections 4.6 and 4.7 are easily extended to
more sophisticated settings. For example, the environment sometimes produces
side information as well as rewards or the set of available actions may change
with time. You are asked to formalise an example in Exercise 4.6.

4.9 Bibliographical Remarks

There is now a huge literature on stochastic bandits, much of which we will

discuss in detail in the chapters that follow. The earliest reference that we know
of is by Thompson [1933], who proposed an algorithm that forms the basis
of many of the currently practical approaches in use today. Thompson was a
pathologist who published broadly and apparently did not pursue bandits much
further. Sadly his approach was not widely circulated, and the algorithm (now
called Thompson sampling) did not become popular until very recently. Two
decades after Thompson, the bandit problem was formally restated in a short but
influential paper by Robbins [1952], an American statistician now most famous
for his work on empirical Bayes. Robbins introduced the notion of regret and
minimax regret in his 1952 paper. The regret decomposition (Lemma 4.5) has
been used in practically every work on stochastic bandits, and its origin is hard
to pinpoint. All we can say for sure is that it does not appear in the paper by
Robbins [1952], but does appear in the work of Lai and Robbins [1985]. Denardo
et al. [2007] considers risk in a (complicated) Bayesian setting. Sani et al. [2012]
consider a mean-variance approach to risk, while Maillard [2013] considers so-
called coherence risk measures (CVaR, is one example of such a risk measure),
 4.10 Exercises 69

and with an approach where the regret itself is redefined. VaR is considered in
the context of a specific bandit policy family by Audibert et al. [2007, 2009].

4.10 Exercises

4.1 (Positivity of the regret) Prove Lemma 4.4.

4.2 (Uniqueness of law) Prove Proposition 4.8.

4.3 (Definition of canonical probability measure) Prove that the measure

defined in terms of the density in Eq. (4.7) satisfies the conditions (a) and (b) in
Section 4.6.

Hint Use the properties of the Radon–Nikodym derivative in combination with

Fubini’s theorem.
4.4 (Mixing policies) Fix a horizon n and k. Let Π be a finite set of policies
for k-armed bandits on horizon n and p ∈ P(Π) be a distribution over Π. Show
there exists a policy π ◦ such that for any k-armed stochastic bandit ν,
X
Pνπ◦ = p(π)Pνπ .
π∈Π

Proof For action/reward sequence a1 , x1 , . . . , an , xn , syntactically abbreviate

ht = a1 , x1 , . . . , at , xt . Then define
P Qt
π∈Π p(π) s=1 πs (as | hs−1 )
πt (at | ht−1 ) = P
◦
Qt−1 .
π∈Π p(π) s=1 πs (as | hs−1 )

By the definition of the canonical probability space and the product of probability
kernels,
Xk Z Xk Z
Pνπ◦ (B) = ··· IB (hn )νan (dxn )πn◦ (an | hn−1 ) · · · νa1 (dx1 )π1◦ (a1 )
a1 =1 R an =1 R

X k Z
X k Z
X
= p(π) ··· IB (hn )νan (dxn )πn (an | hn−1 ) · · · νa1 (dx1 )π1 (a1 )
π∈Π a1 =1 R an =1 R
X
= p(π)Pνπ (B) ,
π∈Π

where the second equality follows by substituting the definition of π ◦ and

induction.
4.5 (Regret decomposition and canonical model for large action
spaces) Let ν be a bandit on measurable action space (A, G) and π1 , . . . , πn be
a policy satisfying the conditions in Section 4.7.
(a) Show that all quantities in Lemma 4.6 are appropriately defined and
measurable.
 4.10 Exercises 70

(b) Prove Lemma 4.6.

(c) Prove that Proposition 4.8 continues to hold.

4.6 (Canonical model for contextual bandit) Let A and C be finite sets.
A stochastic contextual bandit is like a normal stochastic bandit, but in each
round the learner first observes a context Ct ∈ C. They then choose an action
At ∈ A and receive a reward Xt ∼ PAt ,Ct .

(a) Suppose that C1 , . . . , Cn is sampled independently from distribution

ξ on C. Construct the canonical probability space that carries
C1 , A1 , X1 , . . . , Cn , An , Xn .
(b) What changes when Ct is allowed to depend on C1 , A1 , X1 , . . . , Ct−1 , At−1 , Xt−1 ?

4.7 (Bernoulli environment implementation) Implement a Bernoulli bandit

environment in Python using the code snippet below (or adapt to your favourite
language).
class B er no ul li B an di t :
# accepts a list of K >= 2 floats , each lying in [0 ,1]
def __init__ ( self , means ) :
pass

# F u n c t i o n should return the number of arms

def K ( self ) :
pass

# Accepts a p a r a m e t e r 0 <= a <= K -1 and returns the

# r e a l i s a t i o n of random v a r i a b l e X with P ( X = 1) being
# the mean of the ( a +1) th arm .
def pull ( self , a ) :
pass

# Returns the regret i n c u r r e d so far .

def regret ( self ) :
pass

4.8 (Follow-the-leader implementation) Implement the following simple

algorithm called ‘follow-the-leader’, which chooses each action once and
subsequently chooses the action with the largest average observed so far. Ties
should be broken randomly.
def F ol lo wT he L ea de r ( bandit , n ) :
# i m p l e m e n t the Follow - the - Leader a l g o r i t h m by r e p l a c i n g
# the code below that just plays the first arm in every round
for t in range ( n ) :
bandit . pull (0)

Depending on the literature you are reading, follow-the-leader may be called

‘stay with the winner’ or the ‘greedy algorithm’.
 4.10 Exercises 71

4.9 Suppose ν is a finite-armed stochastic bandit and π is a policy such that

Rn (π, ν)
lim = 0.
n→∞ n
Pn
Let T ∗ (n) = t=1 I {µAt = µ∗ } be the number of times an optimal arm is chosen.
Prove or disprove each of the following statements:
(a) limn→∞ E[T ∗ (n)]/n = 1.
(b) limn→∞ P (∆At > 0) = 0.

4.10 (one-armed bandits) Let M1 be a set of distributions on (R, B(R)) with

finite means and M2 = {δµ2 } be the singleton set with a Dirac at µ2 ∈ R. The
set of bandits E = M1 × M2 is called a one-armed bandit because, although
there are two arms, the second arm always yields a known reward of µ2 . A policy
π = (πt )t is called a retirement policy if once action 2 has been played once,
it is played until the end of the game. Precisely, if at = 2, then
πt+1 (2 | a1 , x1 , . . . , at , xt ) = 1 for all (as )t−1
s=1 and (xs )s=1 .
t

(a) Let n be fixed and π = (πt )nt=1 be any policy. Prove there exists a retirement
policy π 0 = (πt0 )nt=1 such that for all ν ∈ E.
Rn (π 0 , ν) ≤ Rn (π, ν) .
(b) Let M1 = {B(µ1 ) : µ1 ∈ [0, 1]} and suppose that π = (πt )∞
t=1 is a retirement
policy. Prove there exists a bandit ν ∈ E such that
Rn (π, ν)
lim sup > 0.
n→∞ n
4.11 (Failure of follow-the-leader (i)) Consider a Bernoulli bandit with
two arms and means µ1 = 0.5 and µ2 = 0.6.
(a) Using a horizon of n = 100, run 1000 simulations of your implementation
of follow-the-leader on the Bernoulli bandit above and record the (random)
Pn
pseudo regret, nµ∗ − t=1 µAt , in each simulation.
(b) Plot the results using a histogram. Your figure should resemble Fig. 4.2.
(c) Explain the results in the figure.

4.12 (Failure of follow-the-leader (ii)) Consider the same Bernoulli

bandit as used in the previous question.
(a) Run 1000 simulations of your implementation of follow-the-leader for each
horizon n ∈ {100, 200, 300, . . . , 1000}.
(b) Plot the average regret obtained as a function of n (see Fig. 4.3). Because the
average regret is an estimator of the expected regret, you should generally
include error bars to indicate the uncertainty in the estimation.
(c) Explain the plot. Do you think follow-the-leader is a good algorithm?
Why/why not?
 4.10 Exercises 72

Follow-the-leader

400
Frequency

200

0 2 4 6 8 10
Regret

Figure 4.2 Histogram of regret for follow-the-leader over 1000 trials on a Bernoulli bandit
with means µ1 = 0.5, µ2 = 0.6

Follow-the-leader
50

40
Expected Regret

30

20

10

200 400 600 800 1,000

n

Figure 4.3 The regret for Follow-the-leader over 1000 trials on Bernoulli bandit with
means µ1 = 0.5, µ2 = 0.6 and horizons ranging from n = 100 to n = 1000.
5 Concentration of Measure

Before we can start designing and analysing algorithms, we need one more tool
from probability theory, called concentration of measure. Recall that the
optimal action is the one with the largest mean. Since the mean pay-offs are
initially unknown, they must be learned from data. How long does it take to
learn about the mean reward of an action? In this section, after introducing
the notion of tail probabilities, we look at ways of obtaining upper bounds on
them. The main point is to introduce subgaussian random variables and the
Cramér–Chernoff exponential tail inequalities, which will play a central role in
the design and analysis of the various bandit algorithms.

5.1 Tail Probabilities

Suppose that X, X1 , X2 , . . . , Xn is a sequence of independent and identically

distributed random variables, and assume that the mean µ = E[X] and variance
σ 2 = V[X] exist. Having observed X1 , X2 , . . . , Xn , we would like to estimate the
common mean µ. The most natural estimator is

1X
n
µ̂ = Xi ,
n i=1

which is called the sample mean or empirical mean. Linearity of expectation

(Proposition 2.6) shows that E[µ̂] = µ, which means that µ̂ is an unbiased
estimator of µ. How far from µ do we expect µ̂ to be? A simple measure
of the spread
 of the distribution of a random variable Z is its variance,
V [Z] = E (Z − E [Z])2 . A quick calculation using independence shows that

  σ2
V [µ̂] = E (µ̂ − µ)2 = , (5.1)
n
which means that we expect the squared distance between µ and µ̂ to shrink as
n grows large at a rate of 1/n and scale linearly with the variance of X. While
the expected squared error is important, it does not tell us very much about the
distribution of the error. To do this we usually analyse the probability that µ̂
overestimates or underestimates µ by more than some value ε > 0. Precisely, how
 5.2 The Inequalities of Markov and Chebyshev 74

Figure 5.1 The figure shows a probability density, with the tails shaded indicating the
regions where X is at least ε away from the mean µ.

do the following quantities depend on ε?

P (µ̂ ≥ µ + ε) and P (µ̂ ≤ µ − ε) .
The expressions above (as a function of ε) are called the tail probabilities of
µ̂ − µ (Fig. 5.1). Specifically, the first is called the upper tail probability and the
second the lower tail probability. Analogously, P (|µ̂ − µ| ≥ ε) is called a two-sided
tail probability.

5.2 The Inequalities of Markov and Chebyshev

The most straightforward way to bound the tails is by using Chebyshev’s

inequality, which is itself a corollary of Markov’s inequality. The latter is
one of the golden hammers of probability theory, and so we include it for the
sake of completeness.
Lemma 5.1. For any random variable X and ε > 0, the following holds:
E [|X|]
(a) (Markov): P (|X| ≥ ε) ≤ .
ε
V [X]
(b) (Chebyshev): P (|X − E [X] | ≥ ε) ≤ .
ε2
We leave the proof of Lemma 5.1 as an exercise for the reader. By combining
(5.1) with Chebyshev’s inequality, we can bound the two-sided tail directly in
terms of the variance by
σ2
P (|µ̂ − µ| ≥ ε) ≤ . (5.2)
nε2
This result is nice because it was so easily bought and relied on no assumptions
other than the existence of the mean and variance. The downside is that when X is
well behaved, the inequality is rather loose. By assuming that higher moments of
X exist, Chebyshev’s inequality can be improved by applying Markov’s inequality
to |µ̂ − µ|k , with the positive integer k to be chosen so that the resulting bound is
 5.2 The Inequalities of Markov and Chebyshev 75

optimised. This is a bit cumbersome, and thus instead we present the continuous
analog of this, known as the Cramér-Chernoff method.
To calibrate our expectations on what improvement to expect relative to
Chebyshev’s inequality, let us start by recalling the central limit theorem
Pn
(CLT). Let Sn = t=1 (Xt − µ). The CLT says that under no additional
assumptions
√ than the existence of the variance, the limiting distribution of
Sn / nσ as n → ∞ is a Gaussian with mean zero and unit variance. If
2

Z ∼ N (0, 1), then

Z  2
∞
1 x
P (Z ≥ u) = √ exp − dx .
u 2π 2

The integral has no closed-form solution, but is easy to bound:

Z  2 Z ∞  2
∞
1 x 1 x
√ exp − dx ≤ √ x exp − dx
u 2π 2 u 2π u 2
r  2
1 u
= exp − , (5.3)
2πu2 2

which gives

 √ p   p 
P (µ̂ ≥ µ + ε) = P Sn / σ 2 n ≥ ε n/σ 2 ≈ P Z ≥ ε n/σ 2
r  
σ2 nε2
≤ exp − . (5.4)
2πnε2 2σ 2

This always improves on what we obtained with Chebyshev’s inequality, usually

by an enormous margin (Exercise 5.3). In particular, the bound on the right-hand
side of (5.4) decays slightly faster than the negative exponential of nε2 /(2σ 2 ),
which means that µ̂ rapidly concentrates around its mean.

An oft-taught rule of thumb is that the CLT provides a reasonable

approximation for n ≥ 30. We advise caution. Suppose that X1 , . . . , Xn
are independent Bernoulli with bias p = 1/n. As n tends to infinity the
Pn
distribution of t=1 Xt converges to a Poisson distribution with parameter
1, which does not look Gaussian at all.

The asymptotic nature of the CLT makes it unsuitable for designing bandit
algorithms. In the next section, we derive finite-time analogs, which are only
possible by making additional assumptions.
 5.3 The Cramér-Chernoff Method and Subgaussian Random Variables 76

5.3 The Cramér-Chernoff Method and Subgaussian Random

Variables

For the sake of moving rapidly towards bandits, we start with a straightforward
and relatively fundamental assumption on the distribution of X, known as the
subgaussian assumption.
Definition 5.2 (Subgaussianity). A random variable
X is σ-subgaussian if for
all λ ∈ R, it holds that E [exp(λX)] ≤ exp λ2 σ 2 /2 .
An alternative way to express the subgaussianity condition uses the moment-
generating function of X, which is a function MX : R → R defined by
MX (λ) = E [exp(λX)]. The condition in the definition can be written as
1 2 2
ψX (λ) = log MX (λ) ≤ λ σ for all λ ∈ R .
2
The function ψX is called the cumulant-generating function. It is not hard
to see that MX (or ψX ) need not exist for all random variables over the whole
range of real numbers. For example, if X is exponentially distributed and λ ≥ 1,
then
Z ∞
E [exp(λX)] = exp(−x) × exp(λx)dx = ∞ .
0 | {z }
density of exponential

The moment-generating function of X ∼ N (0, σ 2 ) satisfies MX (λ) = exp(λ2 σ 2 /2),

and so X is σ-subgaussian.

A random variable X is heavy tailed if MX (λ) = ∞ for all λ > 0. Otherwise

it is light tailed.

The following theorem explains the origin of the term ‘subgaussian’. The tails
of a σ-subgaussian random variable decay approximately as fast as that of a
Gaussian with zero mean and the same variance.
Theorem 5.3. If X is σ-subgaussian, then for any ε ≥ 0,
 
ε2
P (X ≥ ε) ≤ exp − 2 . (5.5)
2σ
Proof We take a generic approach called the Cramér–Chernoff method. Let
λ > 0 be some constant to be tuned later. Then
P (X ≥ ε) = P (exp (λX) ≥ exp (λε))
≤ E [exp (λX)] exp (−λε) (Markov’s inequality)
 2 2 
λ σ
≤ exp − λε . (Def. of subgaussianity)
2
Choosing λ = ε/σ 2 completes the proof.
5.3 The Cramér-Chernoff Method and Subgaussian Random Variables 77

A similar inequality holds for the left tail. By using the union bound
P (A ∪ B) ≤ P (A) + P (B), we also find that P (|X| ≥ ε) ≤ 2 exp(−ε2 /(2σ 2 )).
An equivalent form of these bounds is
 p   p 
P X ≥ 2σ 2 log(1/δ) ≤ δ P |X| ≥ 2σ 2 log(2/δ) ≤ δ .

This form is often more convenient and especially the latter, which for small δ
shows that with overwhelming probability X takes values in the interval
 p p 
− 2σ 2 log(2/δ), 2σ 2 log(2/δ) .

To study the tail behaviour of µ̂ − µ, we need one more lemma.

Lemma 5.4. Suppose that X is σ-subgaussian and X1 and X2 are independent

and σ1 and σ2 -subgaussian, respectively, then:

(a) E[X] = 0 and V [X] ≤ σ 2 .

(b) cX is |c|σ-subgaussian
p for all c ∈ R.
(c) X1 + X2 is σ12 + σ22 -subgaussian.

The proof of the lemma is left to the reader (Exercise 5.7). Combining
Lemma 5.4 and Theorem 5.3 leads to a straightforward bound on the tails
of µ̂ − µ.

Corollary 5.5. Assume that Xi − µ are independent, σ-subgaussian random

variables. Then for any ε ≥ 0,
   
nε2 nε2
P (µ̂ ≥ µ + ε) ≤ exp − 2 and P (µ̂ ≤ µ − ε) ≤ exp − 2 ,
2σ 2σ
1
Pn
where µ̂ = n t=1 Xt .
Pn √
Proof By Lemma 5.4, it holds that µ̂−µ = i=1 (Xi −µ)/n is σ/ n-subgaussian.
Then apply Theorem 5.3.

For x > 0, it holds that exp(−x) ≤ 1/(ex), which shows that the above
inequality is stronger than what we obtained via Chebyshev’s inequality except
when ε is very small. It is exponentially smaller if nε2 is large relative to σ 2 . The
deviation form of the above result says that under the conditions of the result,
for any δ ∈ [0, 1], with probability at least 1 − δ,
r
2σ 2 log(1/δ)
µ ≤ µ̂ + . (5.6)
n
Symmetrically, it also follows that with probability at least 1 − δ,
r
2σ 2 log(1/δ)
µ ≥ µ̂ − . (5.7)
n
Again, one can use a union bound to derive a two-sided inequality.

Example 5.6. The following random variables are subgaussian:

 5.4 Notes 78

(a) If X is Gaussian with mean zero and variance σ 2 , then X is σ-subgaussian.

(b) If X has mean zero and |X| ≤ B almost surely for B ≥ 0, then X is
B-subgaussian.

(c) If X has mean zero and X ∈ [a, b] almost surely, then X is (b − a)/2-
subgaussian.

If X is exponentially distributed with rate λ > 0, then X is not σ-subgaussian

for any σ ∈ R.

For random variables that are not centred (E [X] 6= 0), we abuse notation
by saying that X is σ-subgaussian if the noise X − E [X] is σ-subgaussian.
A distribution is called σ-subgaussian if a random variable drawn from that
distribution is σ-subgaussian. Subgaussianity is really a property of both a
random variable and the measure on the space on which it is defined, so the
nomenclature is doubly abused.

5.4 Notes

1 The Berry–Esseen theorem (independently discovered by Berry [1941] and

Esseen [1942]) quantifies the speed of convergence in the CLT. It essentially
says that the distance between the Gaussian and the actual distribution decays
√
at a rate of 1/ n under some mild assumptions (see Exercise 5.5). This is
known to be tight for the class of probability distributions that appear in the
Berry–Esseen result. However, this is a vacuous result when the tail probabilities
√
themselves are much smaller than 1/ n. Hence the need for concrete finite-time
results.

2 Theorem 5.3 shows that subgaussian random variables have tails that decay
almost as fast as a Gaussian. A version of the converse is also possible. That
is, if a centered random has tails that behave in a similar way to a Gaussian,
then it is subgaussian. In particular, the following holds: let X be a centered
random variable (E[X] = 0) with P (|X| ≥ ε) ≤ 2 exp(−ε2 /2). Then X is
 5.4 Notes 79

√
5-subgaussian:
" ∞
# ∞  i 
X λi X i X λ |X|i
E[exp(λX)] = E ≤1+ E
i=0
i! i=2
i!
Z  
i!1/i 1/i
∞
X ∞
≤1+ P |X| ≥ x dx (Exercise 2.19)
i=2 0
λ
∞ Z ∞  
X i!2/i x2/i
≤1+2 exp − dx (by assumption)
i=2 0
2λ2
    
√ λ
= 1 + 2πλ exp(λ2 /2) 1 + erf √ −1 (by Mathematica)
2
 2
5λ
≤ exp .
2
This bound is surely loose. At the same time, there is little room for
improvement: if X has density p(x) √ = |x| exp(−x2 /2)/2, then P (|X| ≥ ε) =
exp(−ε2 /2). And yet X is at best 2-subgaussian, so some degree of slack is
required (see Exercise 5.4).
3 We saw in (5.4) that if X1 , X2 , . . . , Xn are independent standard Gaussian
Pn
random variables and µ̂ = n1 t=1 , then
r  
σ2 nε2
P (µ̂ ≥ ε) ≤ exp − 2 .
2πnε2 2σ

If nε2 /σ 2 is relatively large, then this bound is marginally stronger than

exp(−nε2 /(2σ 2 )), which follows from the subgaussian analysis. One might ask
whether or not a similar improvement is possible more generally. And Talagrand
[1995] will tell you: yes! At least for bounded random variables (details in the
paper).
4 Hoeffding’s lemma states that for a zero-mean random variable X such that
X ∈ [a, b] almost surely for real values a < b, then MX (λ) ≤ exp(λ2 (b − a)2 /8).
Applying the Cramér–Chernoff method shows that if X1 , X2 , . . . , Xn are
independent and Xt ∈ [at , bt ] almost surely with at < bt for all t, then
!  
1X −2n2 ε2
n
P (Xt − E[Xt ]) ≥ ε ≤ exp Pn . (5.8)
t=1 (bt − at )
n t=1 2

The above is called Hoeffding’s inequality. For details see Exercise 5.11.
There are many variants of this result that provide tighter bounds when X
satisfies certain additional distributional properties like small variance (see
Exercise 5.14).
5 The Cramér–Chernoff method is applicable beyond the subgaussian case, even
when the moment-generating function is not defined globally. One example
where this occurs is when X1 , X2 , . . . , Xn are independent standard Gaussian
Pn
and Y = i=1 Xi2 . Then Y has a χ2 -distribution with n degrees of freedom.
 5.5 Bibliographical Remarks 80

An easy calculation shows that MY (λ) = (1 − 2λ)−n/2 for λ ∈ [0, 1/2) and
MY (λ) is undefined for λ ≥ 1/2. By the Cramér–Chernoff method, we have

P (Y ≥ n + ε) ≤ inf Mλ (Y ) exp(−λ(n + ε))

λ∈[0,1/2)
  n2
1
≤ inf exp(−λ(n + ε))
λ∈[0,1/2) 1 − 2λ

n

Choosing λ = 12 − 2(n+ε)
n
leads to P (Y ≥ n + ε) ≤ 1 + nε 2 exp − 2ε , which
turns out to be about the best you can do [Laurent and Massart, 2000].
6 The subgaussian concept provides a large class of distributions for which
concentration is easily analysed. As mentioned, however, many distributions
are not subgaussian, like the exponential and χ2 -distribution. There are other
general notions based on bounds on the moment generating function that
generalise these kinds of distributions. For more on these ideas, you should
look for keywords subexponential and subgamma.

5.5 Bibliographical Remarks

We return to concentration of measure many times, but note here that it is an

interesting (and still active) topic of research. What we have seen is only the tip
of the iceberg. Readers who want to learn more about this exciting field might
enjoy the book by Boucheron et al. [2013]. For matrix versions of many standard
results, there is a recent book by Tropp [2015]. The survey of McDiarmid [1998]
has many of the classic results. There is a useful type of concentration bound
that are ‘self-normalised’ by the variance. A nice book on this is by de la Peña
et al. [2008]. Another tool that is occasionally useful for deriving concentration
bounds in more unusual set-ups is called empirical process theory. There are
several references for this, including those by van de Geer [2000] or Dudley [2014].

5.6 Exercises

There are too many candidate exercises to list. We heartily recommend all the
exercises in chapter 2 of the book by Boucheron et al. [2013].
5.1 (Variance of average) Let X1 , X2 , . . . , Xn be a sequence of independent
and identically distributed random variables with mean µ and variance σ 2 < ∞.
Pn
Let µ̂ = n1 t=1 Xt and show that V[µ̂] = E[(µ̂ − µ)2 ] = σ 2 /n.

5.2 (Markov’s inequality) Prove Markov’s inequality (Lemma 5.1).

5.3 Prove that the Gaussian tail probability bound on the right-hand side
of Eq. (5.4) is smaller than the bound obtained with Chebyshev’s inequality
 5.6 Exercises 81

Eq. (5.2). In what regime is the improvement most dramatic and in what regime
are both bounds trivial?

5.4 Let X be a random variable on R with density with respect to the Lebesgue
measure of p(x) = |x| exp(−x2 /2)/2. Show the following:
2
(a) P (|X| ≥ ε)
p = exp(−ε /2).
(b) X is not (2 − ε)-subgaussian for any ε > 0.

5.5 (Berry–Esseen inequality) Let X1 , X2 , . . . , Xn be a sequence of

independent and identically distributed random variables with mean µ, variance
σ 2 and bounded third absolute moment:
ρ = E[|X1 − µ|3 ] < ∞ .
Pn
Let Sn = t=1 (Xt − µ)/σ. The Berry–Esseen theorem shows that
 
 
   Z x 
 Sn 1  Cρ
sup P √ ≤ x − √ exp(−y 2 /2)dy  ≤ √ ,
x  n 2π −∞ n
 | {z }
Φ(x)

where C < 1/2 is a universal constant.

Pn
(a) Let µ̂n = n1 t=1 Xt and derive a tail bound from the Berry–Esseen theorem.
That is, give a bound of the form P (µ̂n ≥ µ + ε) for positive values of ε.
(b) Compare your bound with the one that can be obtained from the Cramér–
Chernoff method. Argue pro- and contra- for the superiority of one over the
other.

5.6 (Central limit theorem) We mentioned that invoking the CLT to

approximate the distribution of sums of independent Bernoulli random variables
using a Gaussian can be a bad idea. Let X1 , . . . , Xn ∼ B(p) be independent
Bernoulli random variables with common mean p = pn = λ/n, where λ ∈ (0, 1).
For x ∈ N natural number, let Pn (x) = P (X1 + · · · + Xn = x).

(a) Show that limn→∞ Pn (x) = e−λ λx /(x!), which is a Poisson distribution with
parameter λ.
(b) Explain why this does not contradict the CLT, and discuss the implications
of the Berry–Esseen.
(c) In what way does this show that the CLT is indeed a poor approximation in
some cases?
(d) Based on Monte Carlo simulations, plot the distribution of X1 + · · · + Xn
for n = 30 and some well-chosen values of λ. Compare the distribution to
what you would get from the CLT. What can you conclude?

5.7 (Properties of subgaussian random variables (i)) Prove Lemma 5.4.

Hint Use Taylor series.

 5.6 Exercises 82

5.8 (Properties of subgaussian random variables (ii)) Let Xi be σi -

subgaussian for i ∈ {1, 2} with σi ≥ 0. Prove that X1 +X2 is (σ1 +σ2 )-subgaussian.
Do not assume independence of X1 and X2 .

5.9 (Properties of moment/cumulative-generating functions) Let X

be a real-valued random variable and let MX (λ) = E [exp(λX)] be its moment-
generating function defined over dom(MX ) ⊂ R, where the expectation takes on
finite values. Show that the following properties hold:

(a) MX is convex, and in particular dom(MX ) is an interval containing zero.

(b) MX (λ) ≥ eλE[X] for all λ ∈ dom(MX ).
(c) For any λ in the interior of dom(MX ), MX is infinitely many times
differentiable.
(k)
(d) Let MX (λ)  = dλ dk
k MX (λ). Then, for λ in the interior of dom(MX ),

M (k) (λ) = E X k exp(λX) .
(k)  
(e) Assuming 0 is in the interior of dom(MX ), MX (0) = E X k (hence the
name of MX ).
(f) ψX is convex (that is, MX is log-convex).

Hint For part (a), use the convexity of x 7→ ex .

5.10 (Large deviation theory) Let X, X1 , X2 , . . . , Xn be a sequence of
independent and identically distributed random variables with zero mean and
Pn
moment-generating function MX with dom(MX ) = R. Let µ̂n = n1 t=1 Xt .

(a) Show that for any ε > 0,

1
log P (µ̂n ≥ ε) ≤ −ψX
∗
(ε) = − sup (λε − log MX (λ)) . (5.9)
n λ

(b) Show that when X is a Rademacher variable (P (X = −1) = P (X = 1) =

1/2), ψX∗
(ε) = 1+ε 1−ε
2 log(1 + ε) + 2 log(1 − ε) when |ε| < 1 and ψX (ε) = +∞,
∗

otherwise.
(c) Show that when X is a centered Bernoulli random variable with parameter
p (that is, P (X = −p) = 1 − p and P (X = 1 − p) = p) then ψX ∗
(ε) = ∞
when ε is such that p + ε > 1 and ψX (ε) = d(p + ε, p) otherwise, where
∗

d(p, q) = p log(p/q) + (1 − p) log((1 − p)/(1 − q)) is the relative entropy

between the distributions B(p) and B(q).
(d) Show that when X ∼ N (0, σ 2 ) then ψX ∗
(ε) = ε2 /(2σ 2 ).
(e) Let σ 2 = V[X]. The (strong form of the) central limit theorem says that
  r  
 n 
lim sup P µ̂n
 ≥ x − (1 − Φ(x)) = 0 ,
n→∞ x∈R  σ2
Rx
where Φ(x) = √12π −∞ exp(−y 2 /2)dy is the cumulative distribution of the
 5.6 Exercises 83

standard Gaussian. Let Z be a random variable distributed like a standard

Gaussian. A careless application of this result might suggest that
 r 
1 ? 1 n
lim log P (µ̂n ≥ ε) = lim log P Z ≥ ε .
n→∞ n n→∞ n σ2
Evaluate the right-hand side. In light of the previous parts, what can you
conclude about the validity of the question-marked equality? What goes
wrong with the careless application of the central limit theorem? What do
you conclude about the accuracy of this theorem?

Hint For Part ((e)), consider using Eq. (13.4).

As it happens, the inequality in (5.9) may be replaced by an equality

as n → ∞. The assumption that the moment-generating function exists
everywhere may be relaxed significantly. We refer the interested reader to
the classic text by Dembo and Zeitouni [2009]. The function ψX ∗
is called the
Legendre transform, convex conjugate or Fenchel dual of the convex
function ψX . In probability theory, ψX
∗
is also called the Cramér transform
and is also known as a rate function. Convexity and the Fenchel dual will
play a role in some of the later chapters and will be discussed in more detail
in Chapter 26 and later.

The name “large deviation” originates from rewriting the tail probabilities in
terms of the partial sum Sn = X1 +· · ·+Xn , we see that the inequality in (5.9)
bounds the probability of the deviation of Sn from its mean (which is zero by
assumption) at a scale of Θ(n): P (µ̂n ≥ ε) = P (Sn ≥ nε). In contrast, the
central-limit theorem (CLT) gives the (limiting) probability of the deviation
√ √ √
of Sn from its mean at the scales of Θ( n): P (µ̂n n ≥ ε) = P (Sn ≥ nε).
√
Compared to nε, nε is thought of as a “large” deviation. The deviation
probabilities at this scale can decay to zero faster than what the CLT
predicts, as also showcased in the last part of the last exercise. But what
happens at intermediate scales? That is, when deviations are of size nα ε with
1/2 < α < n? This is studied on the formulaic name of moderate deviations.
As it turns out, in this case, the ruthless use of the large deviation formula
gives correct answers. The reader who wants to learn more about large
deviation theory can check out the lecture notes by Swart [2017].

5.11 (Hoeffding’s lemma) Suppose that X is zero mean and X ∈ [a, b] almost
surely for constants a < b.

(a) Show that X is (b − a)/2-subgaussian.

(b) Prove Hoeffding’s inequality (5.8).
 5.6 Exercises 84

Hint For part (a), it suffices to prove that ψX (λ) ≤ λ2 (b − a)2 /4. By Taylor’s
theorem, for some λ0 between 0 and λ, ψX (λ) = ψX (0) + ψX 0
(0)λ + ψX00
(λ0 )λ2 /2.
To bound the last term, introduce the distribution Pλ for λ ∈ R arbitrary:
Pλ (dz) = e−ψX (λ) eλz P (dz). Show that Ψ00X (λ) = V[Z], where Z ∼ Pλ . Now,
since Z ∈ [a, b] with probability one, argue (without relying on E [Z]) that
V[Z] ≤ (b − a)2 /4.

5.12 (Subgaussianity of Bernoulli distribution) Let X be a random

variable with Bernoulli distribution with mean p. That is X ∼ B(p): P (X = 1) = p
and P (X = 0) = 1 − p.

(a) Show that X is 1/2-subgaussian for all p.

q
1−2p
(b) Let Q : [0, 1] → [0, 1/2] be the function given by Q(p) = 2 ln((1−p)/p)
where undefined points are defined in terms of their limits. Show that X is
Q(p)-subgaussian.
(c) The subgaussianity constant of a random variable X is the smallest value of
σ such that X is σ-subgaussian. Show that the subgaussianity constant of
X ∼ B(p) is Q(p).
p p
(d) Plot Q(p) as a function of p. How does it compare to V[X] = p(1 − p)?
(e) Show that for λ ≥ 0 and p ≥ 1/2, Eexp(λX) ≤ exp(p(1 − p)λ2 /2). Think of
how these inequalities are used for bounding tails. What do you conclude?

Readers looking for a hint to parts (b), (c) and (e) in the previous exercise
might like to look at the papers by Berend and Kontorovich [2013] and
Ostrovsky and Sirota [2014]. The result that the subgaussianity constant of
X ∼ B(p) is upper bounded by Q(p) is known as the Kearn-Saul inequality
and is due to Kearns and Saul [1998].

5.13 (Central limit theorem for sums of Bernoulli random variables)

In this question we try to understand the concentration of the empirical mean
for Bernoulli random variables. Let X1 , X2 , . . . , Xn be independent Bernoulli
Pn
random variables with mean p ∈ [0, 1] and p̂n = t=1 Xt /n. Let Zn be normally
distributed random variable with mean p and variance p(1 − p)/n.

(a) Write down expressions for E[p̂n ] and V[p̂n ].

(b) What does the central limit theorem say about the relationship between p̂n
and Zn as n gets large?
(c) For each p ∈ {1/10, 1/2} and δ = 1/100 and ∆ = 1/10, find the minimum n
such that P (p̂n ≥ p + ∆) ≤ δ.
(d) Let p = 1/10 and ∆ = 1/10 and

nBer (δ, p, ∆) = min {n : P (p̂n ≥ p + ∆) ≤ δ} ,

nGauss (δ, p, ∆) = min {n : P (Zn ≥ p + ∆) ≤ δ} .
 5.6 Exercises 85

(i) Evaluate analytically the value of

nBer (δ, 1/10, 1/10)

lim .
δ→0 nGauss (δ, 1/10, 1/10)

(ii) In light of the central limit theorem, explain why the answer you got in (i)
was not 1.

Hint For Part (d.i) use large deviation theory (Exercise 5.10).
5.14 (Bernstein’s inequality) Let X1 , . . . , Xn be a sequence of independent
Pn
random variables with Xt − E[Xt ] ≤ b almost surely and S = t=1 (Xt − E[Xt ])
Pn
and v = t=1 V[Xt ].

1 x2
(a) Show that g(x) = 2 + x
3! + 4! + · · · = (exp(x) − 1 − x)/x2 is increasing.
(b) Let X be a random variable with E[X] = 0 and X ≤ b almost surely. Show
that E[exp(X)] ≤ 1 + g(b)V[X].
2
3α
(c) Prove that (1 + α) log(1 + α) − α ≥ 6+2α for all α ≥ 0. Prove that this is
the best possible approximation in the sense that the 2 in the denominator
cannot be increased.
(d) Let ε > 0 and α = bε/v and prove that
 v 
P (S ≥ ε) ≤ exp − 2 ((1 + α) log(1 + α) − α) (5.10)
b !
ε2
≤ exp −
. (5.11)
2v 1 + 3v
bε

(e) Use the previous result to show that

s    !
1 2b 1
P S≥ 2v log + log ≤ δ.
δ 3 δ

(f) Let X1 , X2 , . . . , Xn be a sequence of random variables adapted to filtration

F = (Ft )t . Abbreviate Et [·] = E[· | Ft ] and µt = Et−1 [Xt ]. Define S =
Pn Pn
X − µt and let V = t=1 Et−1 [(Xt − µt )2 ] be the predictable variation
Pp t
t=1
of ( t=1 Xt − µt )p . Show that if Xt − µt ≤ b holds almost surely for all
t ∈ [n] then with α = bε/v,
 v 
P (S ≥ ε, V ≤ v) ≤ exp − 2 ((1 + α) log(1 + α) − α) .
b

Note that the right-hand side of this inequality is the same as that shown in
Eq. (5.10).
 5.6 Exercises 86

The bound in Eq. (5.10) is called Bennett’s inequality and the one
in Eq. (5.11) is called Bernstein’s inequality. There are several
generalisations, the most notable of which is the martingale version that
slightly relaxes the independence assumption and which was presented in
Part (f). Martingale techniques appear in Chapter 20. Another useful variant
(under slightly different conditions) replaces the actual variance with the
empirical variance. This is useful when the variance is unknown. For more,
see the papers by Audibert et al. [2007], Mnih et al. [2008], Maurer and
Pontil [2009].

5.15 (Another Bernstein-type inequality) Let X1 , X2 , . . . , Xn be a

sequence of random variables adapted to the filtration F = (Ft )t . Abbreviate
Et [·] = E[· | Ft ] and µt = Et−1 [Xt ]. Prove the following

(a) If η > 0 and η(Xt − µt ) ≤ 1 almost surely, then

 !
1 1
Xn Xn
2
P (Xt − µt ) ≥ η Et−1 [(Xt − µt ) ] + log ≤ δ.
t=1 t=1
η δ

(b) If η > 0 and ηXt ≤ 1 almost surely, then

 !
1 1
Xn Xn
2
P (Xt − µt ) ≥ η Et−1 [Xt ] + log ≤ δ.
t=1 t=1
η δ

Hint Use the Cramér–Chernoff method and the fact that exp(x) ≤ 1 + x + x2
for all x ≤ 1 and exp(x) ≥ 1 + x for all x.

Pt
Let (Mt ) be the martingale defined by Mt = s=1 (Xs −µs ). The inequalities
in Exercise 5.15 can be viewed as a kind of Bernstein’s inequality because
they bound the tail of the martingale (Mt ) in terms of the predictable
Pn 2
variation of the martingale (Mt ), which is V = t=1 Et−1 [(Xt − µt ) ].
The main difference relative to well-known results is that the analysis has
stopped early. The next step is usually to choose η to minimise the bound
in some sense. Either by assuming bounds on the predictable variation,
union bounding or using the method of mixtures [de la Peña et al., 2008].
These techniques are covered in Chapter 20. Note, optimising η directly is
not possible because the bounds hold for any fixed η, but minimising the
right-hand side inside the probability with respect to η would lead to a
random η. For more martingale results with this flavour, see the notes by
McDiarmid [1998].

5.16 Let X1 , . . . , Xn be independent random variables with P (Xt ≤ x) ≤ x for

 5.6 Exercises 87

each x ∈ [0, 1] and t ∈ [n]. Prove for any ε > 0 that

!
Xn  ε n
P log(1/Xt ) ≥ ε ≤ exp(n − ε) .
t=1
n

5.17 (Concentration for categorical distributions) Let X1 , . . . , Xn be

an independent and identically distributed sequence taking values in [m]. For
Pn
i ∈ [m], let p(i) = P (X1 = i) and p̂(i) = n1 t=1 I {Xt = i}. Show that for any
δ ∈ (0, 1),
 s 

2 log 1δ + m log(2)
P kp − p̂k1 ≥  ≤ δ. (5.12)
n

Hint Use the fact that kp − p̂k1 = maxλ∈{−1,1}m hλ, p − p̂i.

5.18 (Expectation of maximum) Let X1 , . . . , Xn be a sequence of σ-
subgaussian random variables (possibly dependent) and Z = maxt∈[n] Xt . Prove
that
p
 ≤ p2σ log(n). 
(a) E[Z] 2

(b) P Z ≥ 2σ 2 log(n/δ) ≤ δ for any δ ∈ (0, 1).

Hint Use Jensen’s inequality to show that exp(λE[Z]) ≤ E[exp(λZ)], and then
provide a naive bound on the moment-generating function of Z.
5.19 (Almost surely bounded sums) Let X1 , X2 , . . . , Xn be a sequence of non-
Pn
negative random variables adapted to filtration (Ft )nt=0 such that t=1 Xt ≤ 1
almost surely. Prove that for all x > 1,
!  
X n  n−x n−1 , if x < n ;
.
P E[Xt | Ft−1 ] ≥ x ≤ fn (x) = n−1
0 , if x ≥ n ,
t=1

where the equality serves as the definition of fn (x).

Hint This problem does not use the techniques introduced in the chapter.
Prove that Bernoulli random variables are the worst case and use backwards
induction. Although this result is new to our knowledge, a weaker version was
derived by Kirschner and Krause [2018] for the analysis of information-directed
sampling. The bound is tight in the sense that there exists a sequence of random
variables and filtration for which equality holds.
Part II
Stochastic Bandits with
Finitely Many Arms
 89

Over the next few chapters, we introduce the fundamental algorithms and
tools of analysis for unstructured stochastic bandits with finitely many actions.
The keywords here are finite, unstructured and stochastic. The first of these
just means that the number of actions available is finite. The second is more
ambiguous, but roughly means that choosing one action yields no information
about the mean pay-off of the other arms. A bandit is stochastic if the sequence
of rewards associated with each action is independent and identically distributed
according to some distribution. This latter assumption will be relaxed in Part III.
There are several reasons to study this class of bandit problems. First, their
simplicity makes them relatively easy to analyse and permits a deep understanding
of the trade-off between exploration and exploitation. Second, many of the
algorithms designed for finite-armed bandits, and the principle underlying them,
can be generalised to other settings. Finally, finite-armed bandits already have
applications – notably as a replacement to A/B testing, as discussed in the
introduction.
6 The Explore-Then-Commit
Algorithm

The first bandit algorithm of the book is called explore-then-commit (ETC),

which explores by playing each arm a fixed number of times and then exploits by
committing to the arm that appeared best during exploration.

For this chapter, as well as Chapters 7 to 9, we assume that all bandit

instances are in ESG
k
(1), which means the reward distribution for all arms is
1-subgaussian.

The focus on subgaussian distributions is mainly for simplicity. Many of the

techniques in the chapters that follow can be applied to other stochastic bandits
such as those listed in Table 4.1. The key difference is that new concentration
analysis is required that exploits the different assumptions. The Bernoulli case is
covered in Chapter 10, where other situations are discussed along with references
to the literature. Notice that the subgaussian assumption restricts the subgaussian
constant to σ = 1, which saves us from endlessly writing σ. All results hold for
other subgaussian constants by scaling the rewards (see Lemma 5.4). Two points
are obscured by this simplification:

(a) All the algorithms that follow rely on the knowledge of σ.

(b) It may happen that Pi is subgaussian for all arms, but with a different
subgaussian constant for each arm. Algorithms are easily adapted to this
situation if the subgaussian constants are known, as you will investigate
in Exercise 7.2. The situation is more complicated when the subgaussian
constant is unknown (Exercise 7.7).

6.1 Algorithm and Regret Analysis

ETC is characterised by the number of times it explores each arm, denoted by a

natural number m. Because there are k actions, the algorithm will explore for mk
rounds before choosing a single action for the remaining rounds. Let µ̂i (t) be the
 6.1 Algorithm and Regret Analysis 91

average reward received from arm i after round t, which is written formally as

1 X
t
µ̂i (t) = I {As = i} Xs ,
Ti (t) s=1
Pt
where Ti (t) = s=1 I {As = i} is the number of times action i has been played
after round t. The ETC policy is given in Algorithm 1 below.

1: Input m.
2: In round t choose action
(
(t mod k) + 1 , if t ≤ mk ;
At =
argmaxi µ̂i (mk) , t > mk .
(ties in the argmax are broken arbitrarily)
Algorithm 1: Explore-then-commit.

Recall that µi is the mean reward when playing action i and ∆i = µ∗ − µi is

suboptimality gap between the mean of action i and the optimal action.

Theorem 6.1. When ETC is interacting with any 1-subgaussian bandit and
1 ≤ m ≤ n/k,
 
m∆2i
k
X k
X
Rn ≤ m ∆i + (n − mk) ∆i exp − .
i=1 i=1
4

Proof Assume without loss of generality that the first arm is optimal, which
means that µ1 = µ∗ = maxi µi . By the decomposition given in Lemma 4.5, the
regret can be written as
k
X
Rn = ∆i E [Ti (n)] . (6.1)
i=1

In the first mk rounds, the policy is deterministic, choosing each action exactly
m times. Subsequently it chooses a single action maximising the average reward
during exploration. Thus,

E [Ti (n)] = m + (n − mk)P (Amk+1 = i)

 
≤ m + (n − mk)P µ̂i (mk) ≥ max µ̂j (mk) . (6.2)
j6=i

The probability on the right-hand side is bounded by

 
P µ̂i (mk) ≥ max µ̂j (mk) ≤ P (µ̂i (mk) ≥ µ̂1 (mk))
j6=i

= P (µ̂i (mk) − µi − (µ̂1 (mk) − µ1 ) ≥ ∆i ) .

p
The next step is to check that µ̂i (mk) − µi − (µ̂1 (mk) − µ1 ) is 2/m-subgaussian,
 6.1 Algorithm and Regret Analysis 92

which by the properties of subgaussian random variables follows from the

definitions of (µ̂j )j and the algorithm. Hence by Corollary 5.5,
 
m∆2i
P (µ̂i (mk) − µi − µ̂1 (mk) + µ1 ≥ ∆i ) ≤ exp − . (6.3)
4
Substituting Eq. (6.3) into Eq. (6.2) and the regret decomposition (Eq. (6.1))
gives the result.
The bound in Theorem 6.1 illustrates the trade-off between exploration and
exploitation. If m is large, then the policy explores for too long, and the first
term will be large. On the other hand, if m is too small, then the probability
that the algorithm commits to the wrong arm will grow, and the second term
becomes large. The question is how to choose m. Assume that k = 2 and that
the first arm is optimal so that ∆1 = 0, and abbreviate ∆ = ∆2 . Then the bound
in Theorem 6.1 simplifies to
   
m∆2 m∆2
Rn ≤ m∆ + (n − 2m)∆ exp − ≤ m∆ + n∆ exp − . (6.4)
4 4
For large n the quantity on the right-hand side of Eq. (6.4) is minimised up to a
possible rounding error by
   
4 n∆2
m = max 1, log , (6.5)
∆2 4
and for this choice and any n, the regret is bounded by
    
4 n∆2
Rn ≤ min n∆, ∆ + 1 + max 0, log . (6.6)
∆ 4
In Exercise 6.2 you will show that Eq. (6.6) implies that
√
Rn ≤ ∆ + C n , (6.7)
where C > 0 is a universal constant. In particular, when ∆ ≤ 1 as is often
assumed, we get
√
Rn ≤ 1 + C n ,
Bounds of this type are called worst-case, problem free or problem
independent (see Eq. (4.2) or Eq. (4.3)). The reason is that the bound only
depends on the horizon and class of bandits for which the algorithm is designed,
and not the specific instance within that class. Because the suboptimality gap does
not appear, bounds like this are sometimes called gap-free. In contrast, bounds
like the one in Eq. (6.6) are called gap/problem/distribution/instance
dependent.
Note that without the condition ∆ ≤ 1, the worst-case bound for ETC is
infinite. In fact, without a bound on the reward range, the worst-case bound of
all reasonable algorithms (that try each action at least once) will also be infinite.
With the understanding that Eq. (6.7) gives rise to a meaningful worst-case
 6.1 Algorithm and Regret Analysis 93

bound for bandits with bounded reward range, we take the liberty and will also
call bounds like that in Eq. (6.7) a worst-case bound.
The bound in (6.6) is close to optimal (see Part IV), but there is a caveat. The
choice of m that defines the policy and leads to this bound depends on both the
suboptimality gap and the horizon. While the horizon is sometimes known in
advance, it is seldom reasonable to assume knowledge of the suboptimality gap.
You will show in Exercise 6.5 that there is a choice of m depending only on n, for
which Rn = O(n2/3 ) regardless of the value of ∆. Alternatively, the number of
plays before commitment can be made data dependent, which means the learner
plays arms alternately until it decides based on its observations to commit to
a single arm for the remainder (Exercise 6.5). ETC also has the property that
its immediate expected regret per time step is monotonically decreasing as time
goes by, though not in a nice smooth fashion. This monotone decreasing property
is a highly desirable property. In later chapters we will see policies where the
decrease is smoother.

Experiment 6.1 Fig. 6.1 shows the expected regret of ETC when playing a
Gaussian bandit with k = 2 and means µ1 = 0 and µ2 = −∆. The horizon is set
to n = 1000, and the suboptimality gap ∆ is varied between 0 and 1. Each data
point is the average of 105 simulations, which makes the error bars invisible. The
results show that the theoretical upper bound provided by Theorem 6.1 is quite
close to the actual performance.

Upper bound in (6.6)

80 ETC with m in (6.5)
Expected regret

60

40

20

0 0.2 0.4 0.6 0.8 1

∆

Figure 6.1 The expected regret of ETC and the upper bound in Eq. (6.6).
 6.2 Notes 94

6.2 Notes

1 An algorithm is called anytime if it does not require advance knowledge of

the horizon n. ETC is not anytime because the choice of commitment time
depends on the horizon. This limitation can be addressed by the doubling
trick, which is a simple way to convert a horizon-dependent algorithm into
an anytime algorithm (Exercise 6.6).
2 By allowing the exploration time m to be a data-dependent random variable,
it is possible to recover near-optimal regret without knowing the suboptimality
gap. For more details see Exercise 6.5. Another idea is to use an elimination
algorithm that acts in phases and eliminates arms using increasingly sensitive
hypothesis tests (Exercise 6.8). Elimination algorithms are often easy to analyse
and can work well in practice, but they also have inherent limitations, just like
ETC algorithms, as will be commented on later.
3 The ε-greedy algorithm is a randomised relative of ETC that in round t
plays the empirically best arm with probability 1 − εt and otherwise explores
uniformly at random. You will analyse this algorithm in Exercise 6.7.

6.3 Bibliographical Remarks

ETC has a long history. Robbins [1952] considered ‘certainty equivalence with
forcing’, which chooses the arm with the largest sample mean except at a fixed
set of times Ti ⊂ N when arm i is chosen for i ∈ [k]. By choosing the set
of times carefully, it is shown that this policy enjoys sublinear regret. While
ETC performs all the exploration at the beginning, Robbins’s policy spreads
the exploration over time. This is advantageous if the horizon is not known,
but disadvantageous otherwise. Anscombe [1963] considered exploration and
commitment in the context of medical trials or other experimental set-ups. He
already largely solves the problem in the Gaussian case and highlights many of
the important considerations. Besides this, the article is beautifully written and
well worth reading. Strategies based on exploration and commitment are simple
to implement and analyse. They can also generalise well to more complex settings.
For example, Langford and Zhang [2008] consider this style of policy under the
name ‘epoch-greedy’ for contextual bandits (the idea of exploring then exploiting
in epochs, or intervals, is essentially what Robbins [1952] suggested). We’ll return
to contextual bandits in Chapter 18. Abbasi-Yadkori et al. [2009], Abbasi-Yadkori
[2009b] and Rusmevichientong and Tsitsiklis [2010] consider ETC-style policies
under the respective names of ‘forced exploration’ and ‘phased exploration and
greedy exploitation’ (PEGE) in the context of linear bandits (which we shall meet
in Chapter 19). Other names include ‘forced sampling’, ‘explore-first’, ‘explore-
then-exploit’. Garivier et al. [2016b] have shown that ETC policies are necessarily
suboptimal in the limit of infinite data in a way that is made precise in Chapter 16.
This comment also applies to elimination-based strategies, which are described in
 6.4 Exercises 95

Exercise 6.8. The history of ε-greedy is unclear, but it is a popular and widely used
and known algorithm in reinforcement learning [Sutton and Barto, 1998]. Auer
et al. [2002a] analyse the regret of ε-greedy with slowly decreasing exploration
probabilities. There are other kinds of randomised exploration as well, including
Thompson sampling [1933] and Boltzmann exploration analysed recently by
Cesa-Bianchi et al. [2017].

6.4 Exercises

6.1 (Subgaussian empirical estimates) Let π be the policy of ETC and

P1 , . . . , Pk be the 1-subgaussian distributions associated with the k arms. Provide
a fully rigourous proof of the claim that

µ̂i (mk) − µi − µ̂1 (mk) + µ1

p
is 2/m-subgaussian. You should only use the definitions and the interaction
protocol, which states that

(a) P (At ∈ · | A1 , X1 , . . . , At−1 , Xt−1 ) = π(· | A1 , X1 , . . . , At−1 , Xt−1 ) a.s.

(b) P (Xt ∈ · | A1 , X1 , . . . , At−1 , Xt−1 , At ) = PAt (·) a.s.

6.2 (Minimax regret) Show that Eq. (6.6) implies the regret of an optimally
√
tuned ETC for subgaussian two-armed bandits satisfies Rn ≤ ∆ + C n where
C > 0 is a universal constant.

6.3 (High-probability bounds (i)) Assume that k = 2, and let δ ∈ (0, 1).
Modify the ETC algorithm to depend on δ and prove a bound on the pseudo-
Pn
regret R̄n = nµ∗ − t=1 µAt of ETC that holds with probability 1 − δ. The
algorithm is allowed to use the action suboptimality gaps.

6.4 (High-probability bounds (ii)) Repeat the previous exercise, but now
Pn
prove a high probability bound on the random regret: R̂n = nµ∗ − t=1 Xt .
Compare this to the bound derived for the pseudo-regret in the previous exercise.
What can you conclude?

6.5 (Adaptive commitment times) Suppose that ETC interacts with a two-
armed 1-subgaussian bandit ν ∈ E with means µ1 , µ2 ∈ R and ∆ν = |µ1 − µ2 |.

(a) Find a choice of m that only depends on the horizon n and not ∆ such that
there exists a constant C > 0 such that for any n and for any ν ∈ E, the
regret Rn (ν) of Algorithm 1 is bounded by

Rn (ν) ≤ (∆ν + C)n2/3 .

Furthermore, show that there is no C > 0 such that for any problem instance
ν and n ≥ 1, Rn (ν) ≤ ∆ν + Cn2/3 holds.
 6.4 Exercises 96

(b) Now suppose the commitment time is allowed to be data dependent, which
means the algorithm explores each arm alternately until some condition is
met and then commits to a single arm for the remainder. Design a condition
such that the regret of the resulting algorithm can be bounded by
C log n
Rn (ν) ≤ ∆ν + , (6.8)
∆ν
where C is a universal constant. Your condition should only depend on the
observed rewards and the time horizon. It should not depend on µ1 , µ2 or
∆ν .
(c) Show that
p any algorithm for which (6.8) holds also satisfies Rn (ν) ≤
∆ν + C n log(n) for any n ≥ 1 and ν ∈ E and a suitably chosen universal
constant C > 0.
(d) As for (b), but now the objective is to design a condition such that for any
n ≥ 1 and ν ∈ E, the regret of the resulting algorithm is bounded by

C log max e, n∆2ν
Rn (ν) ≤ ∆ν + . (6.9)
∆ν

(e) Show that any algorithm for which (6.9) holds also satisfies that for any
√
n ≥ 1 and ν ∈ E, Rn (ν) ≤ ∆ν + C n for suitably chosen universal constant
C > 0.

Hint For (a) start from Rn ≤ m∆ + n∆ exp(−m∆2 /2) and show an upper
bound on the second term which is independent of ∆. Then, choose m. For
(b) think about the simplest stopping policy and then make it robust by using
confidence intervals. Tune the failure probability. For (c) note that the regret
can never be larger than n∆.
6.6 (Doubling trick) The purpose of this exercise is to analyse a meta-
algorithm based on the so-called doubling trick that converts a policy depending
on the horizon to a policy with similar guarantees that does not. Let E be an
arbitrary set of bandits. Suppose you are given a policy π = π(n) designed for E
that accepts the horizon n as a parameter and has a regret guarantee of

max Rt (π(n), ν) ≤ fn (ν) , ∀ν ∈ E ,

1≤t≤n

where fn : E → [0, ∞) is a sequence of functions. Let n1 < n2 < n3 < · · · be a

fixed sequence of integers and consider the policy that runs π with horizon n1
until round t = min{n, n1 }, then runs π with horizon n2 until t = min{n, n1 +n2 },
and then restarts again with horizon n3 until t = min{n, n1 + n2 + n3 } and so-on.
Note that t is the real-time counter and is not reset on each restart. Let π ∗ be the
resulting policy. When n`+1 = 2n` , the length of periods when π is used double
with each phase, hence the name ‘doubling trick’.
P`
(a) Let n > 0 be arbitrary, `max = min{` : i=1 ni ≥ n}. Prove that for any
 6.4 Exercises 97

ν ∈ E, the n-horizon regret of π ∗ on ν is at most

`X
max

Rn (π ∗ , ν) ≤ fn` (ν) . (6.10)

`=1
√
(b) Suppose that fn (ν) ≤ n. Show that if n` = 2`−1 , then for any ν ∈ E and
horizon n the regret of π ∗ is at most
√
Rn (π ∗ , ν) ≤ C n ,
where C > 0 is a carefully chosen universal constant.
(c) Suppose that fn (ν) = g(ν) log(n) for some function g : E → [0, ∞). What is
the regret of π ∗ if n` = 2`−1 ? Can you find a better choice of (n` )` ?
(d) In light of this idea, should we bother trying to design algorithms that do not
depend on the horizon? Are there any disadvantages to using the doubling
trick? If so, what are they? Write a short summary of the pros and cons of
the doubling trick.

According to Besson and Kaufmann [2018], the doubling trick was first
applied to bandits by Auer et al. [1995]. Note, nowhere in this exercise did
we use that the bandit is stochastic. Nothing changes in the adversarial or
contextual settings studied later in the book.

6.7 (ε-greedy) For this exercise assume the rewards are 1-subgaussian and
there are k ≥ 2 arms. The ε-greedy algorithm depends on a sequence of
parameters ε1 , ε2 , . . .. First it chooses each arm once and subsequently chooses
At = argmaxi µ̂i (t − 1) with probability 1 − εt and otherwise chooses an arm
uniformly at random.
k
Rn εX
(a) Prove that if εt = ε > 0, then lim = ∆i .
n→∞ n k i=1
n o
(b) Let ∆min = min {∆i : ∆i > 0} and let εt = min 1, t∆Ck 2 , where C > 0 is
min
a sufficiently large universal constant. Prove that there exists a universal
C 0 > 0 such that
k   
X ∆i n∆2min
Rn ≤ C 0 ∆i + 2 log max e, .
i=1
∆min k

6.8 (Elimination algorithm) A simple way to generalise the ETC policy to

multiple arms and overcome the problem of tuning the commitment time is to
use an elimination algorithm. The algorithm operates in phases and maintains
an active set of arms that could be optimal. In the `th phase, the algorithm aims
to eliminate from the active set all arms i for which ∆i ≥ 2−` .
Without loss of generality, assume that arm 1 is an optimal arm. You may
assume that the horizon n is known.
 6.4 Exercises 98

1: Input: k and sequence (m` )`

2: A1 = {1, 2, . . . , k}
3: for ` = 1, 2, 3, . . . do
4: Choose each arm i ∈ A` exactly m` times
5: Let µ̂i,` be the average reward for arm i from this phase only
6: Update active set:
 
A`+1 = i : µ̂i,` + 2−` ≥ max µ̂j,`
j∈A`

7: end for
Algorithm 2: Phased elimination for finite-armed bandits

(a) Show that for any ` ≥ 1,

 
m` 2−2`
P (1 ∈
/ A`+1 , 1 ∈ A` ) ≤ k exp − .
4

(b) Show that if i ∈ [k] and ` ≥ 1 are such that ∆i ≥ 2−` , then
 
m` (∆i − 2−` )2
P (i ∈ A`+1 , 1 ∈ A` , i ∈ A` ) ≤ exp − .
4

(c) Let `i = min ` ≥ 1 : 2−` ≤ ∆i /2 . Choose m` in such a way that
P (exists ` : 1 ∈
/ A` ) ≤ 1/n and P (i ∈ A`i +1 ) ≤ 1/n.
(d) Show that your algorithm has regret at most
X  1

Rn ≤ C ∆i + log(n) ,
∆i
i:∆i >0

where C > 0 is a carefully chosen universal constant.

(e) Modify your choice of m` and show that the regret of the resulting algorithm
satisfies
X  1 

Rn ≤ C ∆i + log max e, n∆2i .
∆i
i:∆i >0

(f) Show that with an appropriate universal constant C 0 > 0, the regret satisfies
X p
Rn ≤ ∆i + C 0 nk log(k) .
i

Algorithm 2 is due to Auer and Ortner [2010]. The log(k) term in Part (f) can
be removed by modifying the algorithm to use the refined confidence intervals
in Chapter 9, but we would not recommend this for the reasons discussed
in Section 9.2 of that chapter. You could also use a more sophisticated
confidence level [Lattimore, 2018].
 6.4 Exercises 99

ETC

70

Expected regret 60

50

0 100 200 300 400

m

Figure 6.2 Expected regret for ETC over 105 trials on a Gaussian bandit with means
µ1 = 0, µ2 = −1/10

6.9 (Empirical study) In this exercise you will investigate the empirical
behaviour of ETC on a two-armed Gaussian bandit with means µ1 = 0 and
µ2 = −∆. Let
n
X
R̄n = ∆A t ,
t=1

which is chosen so that Rn = E[R̄n ]. Complete the following:

(a) Using programming language of your choice, write a function that accepts
an integer n and ∆ > 0 and returns the value of m that exactly minimises
the expected regret.
(b) Reproduce Fig. 6.1.
(c) Fix ∆ = 1/10 and plot the expected regret as a function of m with n = 2000.
Your plot should resemble Fig. 6.2.
(d) Plot the standard deviation V[R̄n ]1/2 as a function of m for the same bandit
as above. Your plot should resemble Fig. 6.3.
(e) Explain the shape of the curves you observed in Parts (b), (c) and (d) and
reconcile what you see with the theoretical results.
(f) Think, experiment and plot. Is it justified to plot V[R̄n ]1/2 as a summary of
how R̄n is distributed? Explain your thinking.
 6.4 Exercises 100

100
ETC
Standard deviation of the regret

80

60

40

0 100 200 300 400

m

Figure 6.3 Standard deviation of the regret for ETC over 105 trials on a Gaussian bandit
with means µ1 = 0, µ2 = −1/10
7 The Upper Confidence Bound
Algorithm

The upper confidence bound (UCB) algorithm offers several advantages over the
explore-then-commit (ETC) algorithm introduced in the last chapter.

(a) It does not depend on advance knowledge of the suboptimality gaps.

(b) It behaves well when there are more than two arms.
(c) The version introduced here depends on the horizon n, but in the next
chapter, we will see how to eliminate that as well.

The algorithm has many different forms, depending on the distributional

assumptions on the noise. Like in the previous chapter, we assume the noise is
1-subgaussian. A serious discussion of other options is delayed until Chapter 10.

7.1 The Optimism Principle

The UCB algorithm is based on the principle of optimism in the face of

uncertainty, which states that one should act as if the environment is as nice as
plausibly possible. As we shall see in later chapters, the principle is applicable
beyond the finite-armed stochastic bandit problem.
Imagine visiting a new country and making a choice between sampling the local
cuisine or visiting a well-known multinational chain. Taking an optimistic view of
the unknown local cuisine leads to exploration because without data, it could be
amazing. After trying the new option a few times, you can update your statistics
and make a more informed decision. On the other hand, taking a pessimistic
view of the new option discourages exploration, and you may suffer significant
regret if the local options are delicious. Just how optimistic you should be is a
difficult decision, which we explore for the rest of the chapter in the context of
finite-armed bandits.
For bandits, the optimism principle means using the data observed so far to
assign to each arm a value, called the upper confidence bound that with high
probability is an overestimate of the unknown mean. The intuitive reason why
this leads to sublinear regret is simple. Assuming the upper confidence bound
assigned to the optimal arm is indeed an overestimate, then another arm can only
be played if its upper confidence bound is larger than that of the optimal arm,
which in turn is larger than the mean of the optimal arm. And yet this cannot
 7.1 The Optimism Principle 102

happen too often because the additional data provided by playing a suboptimal
arm means that the upper confidence bound for this arm will eventually fall
below that of the optimal arm.
In order to make this argument more precise, we need to define the upper
confidence bound. Let (Xt )nt=1 be a sequence of independent 1-subgaussian random
Pn
variables with mean µ and µ̂ = n1 t=1 Xt . By Eq. (5.6),
r !
2 log(1/δ)
P µ ≥ µ̂ + ≤δ for all δ ∈ (0, 1) . (7.1)
n

When considering its options in round t, the learner has observed Ti (t − 1)

samples from arm i and received rewards from that arm with an empirical mean
of µ̂i (t − 1). Then a reasonable candidate for ‘as large as plausibly possible’ for
the unknown mean of the ith arm is

∞ if Ti (t − 1) = 0
UCBi (t − 1, δ) = q
2 log(1/δ)
(7.2)
µ̂i (t − 1) + otherwise .
Ti (t−1)

Great care is required when comparing (7.1) and (7.2) because in the former the
number of samples is the constant n, but in the latter it is a random variable
Ti (t − 1). By and large, however, this is merely an annoying technicality, and the
intuition remains that δ is approximately an upper bound on the probability of
the event that the above quantity is an underestimate of the true mean. More
details are given in Exercise 7.1.
At last we have everything we need to state a version of the UCB algorithm,
which takes as input the number of arms and the error probability δ.

1: Input k and δ
2: for t ∈ 1, . . . , n do
3: Choose action At = argmaxi UCBi (t − 1, δ)
4: Observe reward Xt and update upper confidence bounds
5: end for
Algorithm 3: UCB(δ).

Although there are many versions of the UCB algorithm, we often do not
distinguish them by name and hope the context is clear. For the rest of this
chapter, we’ll usually call UCB(δ) just UCB.

The value inside the argmax is called the index of arm i. Generally speaking,
an index algorithm chooses the arm in each round that maximises some value
(the index), which usually only depends on the current time step and the samples
from that arm. In the case of UCB, the index is the sum of the empirical mean
 7.1 The Optimism Principle 103

of rewards experienced so far and the exploration bonus, which is also known
as the confidence width.
Besides the slightly vague ‘optimism guarantees optimality or learning’ intuition
we gave before, it is worth exploring other intuitions for the choice of index. At
a very basic level, an algorithm should explore arms more often if they are (a)
promising because µ̂i (t − 1) is large or (b) not well explored because Ti (t − 1) is
small. As one can plainly see, the definition in Eq. (7.2) exhibits this behaviour.
This explanation is not completely satisfying, however, because it does not explain
why the form of the functions is just so.
A more refined explanation comes from thinking of what we expect of any
reasonable algorithm. Suppose at the start of round t the first arm has been
played much more frequently than the rest. If we did a good job designing our
algorithm, we would hope this is the optimal arm, and because it has been played
so often, we expect that µ̂1 (t − 1) ≈ µ1 . To confirm the hypothesis that arm 1 is
optimal, the algorithm had better be highly confident that other arms are indeed
worse. This leads quite naturally to the idea of using upper confidence bounds.
The learner can be reasonably certain that arm i is worse than arm 1 if
s s
2 log(1/δ) 2 log(1/δ)
µ̂i (t − 1) + ≤ µ1 ≈ µ̂1 (t − 1) + , (7.3)
Ti (t − 1) T1 (t − 1)

where δ is called the confidence level and quantifies the degree of certainty.
This means that choosing the arm with the largest upper confidence bound leads
to a situation where arms are only chosen if their true mean could reasonably be
larger than those of arms that have been played often. That this rule is indeed a
good one depends on two factors. The first is whether the width of the confidence
interval at a given confidence level can be significantly decreased, and the second
is whether the confidence level is chosen in a reasonable fashion. For now, we
will take a leap of faith and assume that the width of confidence intervals for
subgaussian bandits cannot be significantly improved from what we use here
(we shall see that this holds in later chapters), and concentrate on choosing the
confidence level now.

Choosing the confidence level is a delicate problem, and we will analyse a

number of choices in future chapters. The basic difficulty is that δ should
be small enough to ensure optimism with high probability, but not so large
that suboptimal arms are explored excessively.

Nevertheless, as a first cut, the choice of this parameter can be guided by

the following considerations. If the confidence interval fails and the index of an
optimal arm drops below its true mean, then it could happen that the algorithm
stops playing the optimal arm and suffers linear regret. This suggests we might
choose δ ≈ 1/n so that the contribution to the regret of this failure case is
relatively small. Unfortunately things are not quite this simple. As we have
 7.1 The Optimism Principle 104

already alluded to, one of the main difficulties is that the number of samples
Ti (t − 1) in the index (7.2) is a random variable, and so our concentration results
cannot be immediately applied. For this reason we will see that (at least naively)
δ should be chosen a bit smaller than 1/n.
Theorem 7.1. Consider UCB as shown in Algorithm 3 on a stochastic k-armed
1-subgaussian bandit problem. For any horizon n, if δ = 1/n2 , then
k
X X 16 log(n)
Rn ≤ 3 ∆i + .
i=1
∆i
i:∆i >0

Before the proof we need a little more notation. Let (Xti )t∈[n],i∈[k] be a collection
of independent random variables with the law of Xti equal to Pi . Then define
Ps
µ̂is = 1s u=1 Xui to be the empirical mean based on the first s samples. We
make use of the third model in Section 4.6 by assuming that the reward in round
t is
Xt = XTAt (t)At .
Then we define µ̂i (t) = µ̂iTi (t) to be the empirical mean of the ith arm after round
t. The proof of Theorem 7.1 relies on the basic regret decomposition identity,
k
X
Rn = ∆i E [Ti (n)] . (Lemma 4.5)
i=1

The theorem will follow by showing that E [Ti (n)] is not too large for suboptimal
arms i. The key observation is that after the initial period where the algorithm
chooses each action once, action i can only be chosen if its index is higher than
that of an optimal arm. This can only happen if at least one of the following is
true:
(a) The index of action i is larger than the true mean of a specific optimal arm.
(b) The index of a specific optimal arm is smaller than its true mean.
Since with reasonably high probability the index of any arm is an upper bound
on its mean, we don’t expect the index of the optimal arm to be below its
mean. Furthermore, if the suboptimal arm i is played sufficiently often, then its
exploration bonus becomes small and simultaneously the empirical estimate of
its mean converges to the true value, putting an upper bound on the expected
total number of times when its index stays above the mean of the optimal arm.
The proof that follows is typical for the analysis of algorithms like UCB, and
hence we provide quite a bit of detail so that readers can later construct their
own proofs.
Proof of Theorem 7.1 Without loss of generality, we assume the first arm is
optimal so that µ1 = µ∗ . As noted above,
k
X
Rn = ∆i E [Ti (n)] . (7.4)
i=1
 7.1 The Optimism Principle 105

The theorem will be proven by bounding E[Ti (n)] for each suboptimal arm i. We
make use of a relatively standard idea, which is to decouple the randomness from
the behaviour of the UCB algorithm. Let Gi be the ‘good’ event defined by
s  
n o n 2 1 o
Gi = µ1 < min UCB1 (t, δ) ∩ µ̂iui + log < µ1 ,
t∈[n] ui δ

where ui ∈ [n] is a constant to be chosen later. So Gi is the event when µ1 is

never underestimated by the upper confidence bound of the first arm, while at the
same time the upper confidence bound for the mean of arm i after ui observations
are taken from this arm is below the pay-off of the optimal arm. We will show
two things:

1 If Gi occurs, then arm i will be played at most ui times: Ti (n) ≤ ui .

2 The complement event Gci occurs with low probability (governed in some way
yet to be discovered by ui ).

Because Ti (n) ≤ n no matter what, this will mean that

E [Ti (n)] = E [I {Gi } Ti (n)] + E [I {Gci } Ti (n)] ≤ ui + P (Gci ) n . (7.5)

The next step is to complete our promise by showing that Ti (n) ≤ ui on Gi and
that P (Gci ) is small. Let us first assume that Gi holds and show that Ti (n) ≤ ui ,
which we do by contradiction. Suppose that Ti (n) > ui . Then arm i was played
more than ui times over the n rounds, and so there must exist a round t ∈ [n]
where Ti (t − 1) = ui and At = i. Using the definition of Gi ,
s
2 log(1/δ)
UCBi (t − 1, δ) = µ̂i (t − 1) + (definition of UCBi (t − 1, δ))
Ti (t − 1)
s
2 log(1/δ)
= µ̂iui + (since Ti (t − 1) = ui )
ui
< µ1 (definition of Gi )
< UCB1 (t − 1, δ) . (definition of Gi )

Hence At = argmaxj UCBj (t − 1, δ) 6= i, which is a contradiction. Therefore if

Gi occurs, then Ti (n) ≤ ui . Let us now turn to upper bounding P (Gci ). By its
definition,
 s 
   
2 log(1/δ)
Gci = µ1 ≥ min UCB1 (t, δ) ∪ µ̂iui + ≥ µ1 . (7.6)
t∈[n]  ui 
 7.1 The Optimism Principle 106

The first of these sets is decomposed using the definition of UCB1 (t, δ),
  ( r )
2 log(1/δ)
µ1 ≥ min UCB1 (t, δ) ⊂ µ1 ≥ min µ̂1s +
t∈[n] s∈[n] s
( r )
[ 2 log(1/δ)
= µ1 ≥ µ̂1s + .
s
s∈[n]

Then using a union bound and the concentration bound for sums of independent
subgaussian random variables in Corollary 5.5, we obtain:
 ( r )
  [ 2 log(1/δ)
P µ1 ≥ min UCB1 (t, δ) ≤ P  µ1 ≥ µ̂1s + 
t∈[n] s
s∈[n]
r !
2 log(1/δ)
Xn
≤ P µ1 ≥ µ̂1s + ≤ nδ . (7.7)
s=1
s

The next step is to bound the probability of the second set in (7.6). Assume that
ui is chosen large enough that
s
2 log(1/δ)
∆i − ≥ c∆i (7.8)
ui

for some c ∈ (0, 1) to be chosen later. Then, since µ1 = µi + ∆i , and using

Corollary 5.5,
 s   s 
2 log(1/δ) 2 log(1/δ)
P µ̂iui + ≥ µ1  = P µ̂iui − µi ≥ ∆i − 
ui ui
 
ui c2 ∆2i
≤ P (µ̂iui − µi ≥ c∆i ) ≤ exp − .
2
Taking this together with (7.7) and (7.6), we have
 
ui c2 ∆2i
P (Gi ) ≤ nδ + exp −
c
.
2
When substituted into Eq. (7.5), we obtain
  
ui c2 ∆2i
E [Ti (n)] ≤ ui + n nδ + exp − . (7.9)
2
It remains to choose ui ∈ [n] satisfying (7.8). A natural choice is the smallest
integer for which (7.8) holds, which is
 
2 log(1/δ)
ui = .
(1 − c)2 ∆2i
This choice of ui can be larger than n, but in this case Eq. (7.9) holds trivially
 7.1 The Optimism Principle 107

since Ti (n) ≤ n. Then, using the assumption that δ = 1/n2 and this choice of ui
leads via (7.9) to
 
2 2 2 log(n2 ) 2 2
E[Ti (n)] ≤ ui + 1 + n1−2c /(1−c) = + 1 + n1−2c /(1−c) . (7.10)
(1 − c)2 ∆2i
All that remains is to choose c ∈ (0, 1). The second term will contribute a
polynomial dependence on n unless 2c2 /(1 − c)2 ≥ 1. However, if c is chosen too
close to 1, then the first term blows up. Somewhat arbitrarily we choose c = 1/2,
which leads to
16 log(n)
E [Ti (n)] ≤ 3 + .
∆2i
The result follows by substituting the above display in Eq. (7.4).
As we saw for the ETC strategy, the regret bound in Theorem 7.1 depends
on the reciprocal of the gaps, which may be meaningless when even a single
suboptimal action has a very small suboptimality gap. As before, one can also
prove a sublinear regret bound that does not depend on the reciprocal of the
gaps.
Theorem 7.2. If δ = 1/n2 , then the regret of UCB, as defined in Algorithm 3,
on any ν ∈ ESG
k
(1) environment, is bounded by
p k
X
Rn ≤ 8 nk log(n) + 3 ∆i .
i=1

Proof Let ∆ > 0 be some value to be tuned subsequently, and recall from the
proof of Theorem 7.1 that for each suboptimal arm i, we can bound
16 log(n)
E[Ti (n)] ≤ 3 + .
∆2i
Therefore, using the basic regret decomposition again (Lemma 4.5), we have
k
X X X
Rn = ∆i E [Ti (n)] = ∆i E [Ti (n)] + ∆i E [Ti (n)]
i=1 i:∆i <∆ i:∆i ≥∆
X  16 log(n)

16k log(n) X
≤ n∆ + 3∆i + ≤ n∆ + +3 ∆i
∆i ∆ i
i:∆i ≥∆

p k
X
≤8 nk log(n) + 3 ∆i ,
i=1
P
where the first inequality follows because i:∆i <∆ Ti (n) ≤ n and the last line by
p
choosing ∆ = 16k log(n)/n.
P
The additive i ∆i term is unavoidable because no reasonable algorithm can
avoid playing each arm once (try to work out what would happen if it did not).
In any case, this term does not grow with the horizon n and is typically negligible.
 7.1 The Optimism Principle 108

100 ETC (m = 25)

ETC (m = 50)
Expected regret 80 ETC (m = 75)
ETC (m = 100)
60 ETC (optimal m)
UCB
40

20

0 0.2 0.4 0.6 0.8 1

∆

Figure 7.1 Experiment showing universality of UCB relative to fixed instances of ETC

As it happens, Theorem 7.2 is close to optimal.

√ We will see in Chapter 15k that
no algorithm can enjoy regret smaller than O( nk) over all problems in ESG (1).
In Chapter 9 we will also see a more complicated variant of Algorithm 3 that
shaves the logarithmic term from the upper bound given above.
Experiment 7.1 We promised that UCB would overcome the limitations
of ETC by achieving the same guarantees but without prior knowledge of
the suboptimality gaps. The theory supports this claim, but just because two
algorithms have similar theoretical guarantees does not mean they perform the
same empirically. The theoretical analysis might be loose for one algorithm and
maybe not the other, or by a different margin. For this reason it is always wise to
prove lower bounds (which we do later) and compare the empirical performance,
which we do (very briefly) now.
The set-up is the same as in Fig. 6.1, which has n = 1000 and k = 2 and
unit variance Gaussian rewards with means 0 and −∆ respectively. The plot in
Fig. 7.1 shows the expected regret of UCB relative to ETC for a variety of choices
of commitment time m. The expected regret of ETC with the optimal choice of
m (which depends on the knowledge of ∆ and that the pay-offs are Gaussian, cf.
Fig. 6.1) is also shown.

The results demonstrate a common phenomenon. If ETC is tuned with the

optimal choice of commitment time for each choice of ∆, then it outperforms
the parameter-free UCB, though only by a relatively small margin. If,
however, the commitment time must be chosen without the knowledge of
∆, then ETC will usually not outperform UCB. As it happens, a variant of
UCB introduced in the next chapter actually outperforms even the optimally
tuned ETC.
 7.2 Notes 109

7.2 Notes

1 The choice of δ = 1/n2 led to an easy analysis, but comes with two
disadvantages. First of all, it turns out that a slightly smaller value of δ
improves the regret (and empirical performance). Secondly, the dependence on
n means the horizon must be known in advance, which is often not reasonable.
Both of these issues are resolved in the next chapter, where δ is chosen to be
smaller and to depend on the current round t rather than n. Nonetheless – as
promised – Algorithm 3 with δ = 1/n2 does achieve a regret bound similar to
the ETC strategy, but without requiring knowledge of the gaps.
2 The assumption that the rewards generated by each arm are independent can
be relaxed significantly. All of the results would go through by assuming there
exists a mean reward vector µ ∈ Rk such that

E[Xt | X1 , A1 , . . . , At−1 , Xt−1 , At ] = µAt a.s. . (7.11)

2
E[exp(λ(Xt − µAt )) | X1 , A1 , . . . , At−1 , Xt−1 , At ] ≤ exp(λ /2) a.s. . (7.12)

Eq. (7.11) is just saying that the conditional mean of the reward in round t
only depends on the chosen action. Eq. (7.12) ensures that the tails of Xt are
conditionally subgaussian. That everything still goes through is proven using
martingale techniques, which we develop in detail in Chapter 20.
3 So is the optimism principle universal? Does it always lead to policies with
strong guarantees in more complicated settings? Unfortunately the answer turns
out to be no. The optimism principle usually leads to reasonable algorithms
when (i) any action gives feedback about the quality of that action and (ii) no
action gives feedback about the value of other actions. When (i) is violated, even
sublinear regret may not be guaranteed. When (ii) is violated, an optimistic
algorithm may avoid actions that lead to large information gain and low reward,
even when this trade-off is optimal. An example where this occurs is provided
in Chapter 25 on linear bandits. Optimism can work in more complex models as
well, but sometimes fails to appropriately balance exploration and exploitation.
4 When thinking about future outcomes, humans and some animals often have
higher expectations than are warranted by past experience or conditions of the
environment. This phenomenon, a form of cognitive bias, is known as the
optimism bias in the psychology and behavioural economics literature and is
in fact ‘one of the most consistent, prevalent, and robust biases documented in
psychology and behavioral economics’ [Sharot, 2011a]. While much has been
written about this bias in these fields, and one of the current explanations
of why the optimism bias is so prevalent is that it helps exploration, to our
best knowledge, the connection to the deeper mathematical justification of
optimism, pursued here and in other parts of this book, has so far escaped the
attention of researchers in all the relevant fields.
 7.3 Bibliographical Remarks 110

7.3 Bibliographical Remarks

The use of confidence bounds and the idea of optimism first appeared in the work
by Lai and Robbins [1985]. They analysed the asymptotics for various parametric
bandit problems (see the next chapter for more details on this). The first version
of UCB is by Lai [1987]. Other early work is by Katehakis and Robbins [1995],
who gave a very straightforward analysis for the Gaussian case, and Agrawal
[1995], who noticed that all that was needed is an appropriate sequence of
upper confidence bounds on the unknown means. In this way, their analysis is
significantly more general than what we have done here. These researchers also
focused on the asymptotics, which at the time was the standard approach in
the statistics literature. The UCB algorithm was independently discovered by
Kaelbling [1993], although with no regret analysis or clear advice on how to tune
the confidence parameter. The version of UCB discussed here is most similar to
that analysed by Auer et al. [2002a] under the name UCB1, but that algorithm
used t rather than n in the confidence level (see the next chapter). Like us, they
prove a finite-time regret bound. However, rather than considering 1-subgaussian
environments, Auer et al. [2002a] considers bandits where the pay-offs are confined
to the [0, 1] interval, which are ensured to be 1/2-subgaussian. See Exercise 7.2
for hints on what must change in this situation. The basic structure of the proof
of our Theorem 7.1 is essentially the same as that of theorem 1 of Auer et al.
[2002a]. The worst-case bound in Theorem 7.2 appeared in the book by Bubeck
and Cesa-Bianchi [2012], which also popularised the subgaussian set-up. We did
not have time to discuss the situation where the subgaussian constant is unknown.
There have been several works exploring this direction. If the variance is unknown,
but the noise is bounded, then one can replace the subgaussian concentration
bounds with an empirical Bernstein inequality [Audibert et al., 2007]. For details,
see Exercise 7.6. If the noise has heavy tails, then a more serious modification is
required, as discussed in Exercise 7.7 and the note that follows.
We found the article by Sharot [2011a] on optimism bias from the psychology
literature quite illuminating. Readers looking to dive deeper into this literature
may enjoy the book by the same author [Sharot, 2011b]. Optimism bias is also
known as ‘unrealistic optimism’, a term that is most puzzling to us – what bias
is ever realistic? The background of this is explained by Jefferson et al. [2017].

7.4 Exercises

7.1 (Concentration for sequences of random length) In this exercise,

we investigate one of the more annoying challenges when analyzing sequential
algorithms. Let X1 , X2 , . . . be a sequence of independent standard Gaussian
random variables defined on probability space (Ω, F, P). Suppose that T : Ω →
PT
{1, 2, 3, . . .} is another random variable, and let µ̂ = t=1 Xt /T be the empirical
mean based on T samples.
 7.4 Exercises 111

(a) Show that if T is independent from Xt for all t, then

r !
2 log(1/δ)
P µ̂ − µ ≥ ≤ δ.
T

(b) Now relax the assumption that T is independent from (Xt )t . Let Et =
I {T = t} be the event that T = t and Ft = σ(X1 , . . . , Xt ) be the σ-algebra
generated by the first t samples. Let δ ∈ (0, 1) and show there exists a T
such that for all t ∈ {1, 2, 3, . . .} it holds that Et is Ft -measurable and
r !
2 log(1/δ)
P µ̂ − µ ≥ = 1.
T

(c) Show that

r !
2 log(T (T + 1)/δ)
P µ̂ − µ ≥ ≤ δ. (7.13)
T

Hint For part (b) above, you may find it useful to apply the law of the iterated
logarithm, which says if X1 , X2 , . . . is a sequence of independent and identically
distributed random variables with zero mean and unit variance, then
Pn
t=1 Xt
lim sup √ =1 almost surely .
n→∞ 2n log log n
This result is especially remarkable because it relies on no assumptions other
than zero mean and unit variance. You might wonder if Eq. (7.13) might continue
to hold if log(T (T + 1)/δ) were replaced by log(log(T )/δ). It almost does, but
the proof of this fact is more sophisticated. For more details, see the paper by
Garivier [2013] or Exercise 20.9.
7.2 (Relaxing the subgaussian assumption) In this chapter, we assumed
the pay-off distributions were 1-subgaussian. The purpose of this exercise is to
relax this assumption.
(a) First suppose that σ 2 > 0 is a known constant and that ν ∈ ESG
k
(σ 2 ). Modify
the UCB algorithm and state and prove an analogue of Theorems 7.1 and 7.2
for this case.
(b) Now suppose that ν = (Pi )ki=1 is chosen so that Pi is σi -subgaussian where
(σi2 )ki=1 are known. Modify the UCB algorithm and state and prove an
analogue of Theorems 7.1 and 7.2 for this case.
(c) If you did things correctly, the regret bound in the previous part should not
depend on the values of {σi2 : ∆i = 0}. Explain why not.

7.3 (High-probability bounds) Recall from Chapter 4 that the pseudo-regret

is defined to be the random variable
n
X
R̄n = ∆A t .
t=1
 7.4 Exercises 112

The UCB policy in Algorithm 3 depends on confidence parameter δ ∈ (0, 1] that

determines the level of optimism. State and prove a bound on the pseudo-regret
of this algorithm that holds with probability 1 − f (n, k)δ, where f (n, k) is a
function, that depends on n and k only. More precisely show that for bandit
k
ν ∈ ESG (1) that


P R̄n ≥ g(n, ν, δ) ≤ f (n, k)δ ,

where g and f should be as small as possible (there are trade-offs – try and come
up with a natural choice).

7.4 (Phased UCB (i)) Fix a 1-subgaussian k-armed bandit environment and a
horizon n. Consider the version of UCB that works in phases of exponentially
increasing length of 1, 2, 4, . . .. In each phase, the algorithm uses the action that
would have been chosen by UCB at the beginning of the phase (see Algorithm 4
below).

(a) State and prove a bound on the regret for this version of UCB.
(b) Compare your result with Theorem 7.1.
 
(c) How would the result change if the `th phase had a length of α` with
α > 1?

1: Input k and δ
2: Choose each arm once
3: for ` = 1, 2, . . . do
4: Compute A` = argmaxi UCBi (t − 1, δ)
5: Choose arm A` exactly 2` times
6: end for
Algorithm 4: A phased version of UCB.

7.5 (Phased UCB (ii)) Let α > 1 and consider the version of UCB that first
plays each arm once. Thereafter it operates in the same way as UCB, but rather
than playing the chosen arm just once, it plays it until the number of plays of
that arm is a factor of α larger (see Algorithm 5 below).

(a) State and prove a bound on the regret for version of UCB with α = 2
(doubling counts).
(b) Compare with the result of the previous exercise and with Theorem 7.1.
What can you conclude?
(c) Repeat the analysis for α > 1. What is the role of α?
(d) Implement these algorithms and compare them empirically to UCB(δ).
 7.4 Exercises 113

1: Input k and δ
2: Choose each arm once
3: for ` = 1, 2, . . . do
4: Let t` = t
5: Compute A` = argmaxi UCBi (t` − 1, δ)
6: Choose arm A` until round t such that Ti (t) ≥ αTi (t` − 1)
7: end for
Algorithm 5: A phased version of UCB.

The algorithms of the last two exercises may seem ridiculous. Why would
you wait before updating empirical estimates and choosing a new action?
There are at least two reasons:
(a) It can happen that the algorithm does not observe its rewards
immediately, but rather they appear asynchronously after some delay.
Alternatively many bandits algorithms may be operating simultaneously
and the results must be communicated at some cost.
(b) If the feedback model has a more complicated structure than what we
examined so far, then even computing the upper confidence bound just
once can be quite expensive. In these circumstances, it’s comforting to
know that the loss of performance by updating the statistics only rarely
is not too severe.

7.6 (Adapting to reward variance in bandits with bounded rewards)

Let X1 , X2 , . . . , Xn be a sequence of independent and identically distributed
random variables with mean µ and variance σ 2 and bounded support so that
Pn Pn
Xt ∈ [0, b] almost surely. Let µ̂ = t=1 Xt /n and σ̂ 2 = t=1 (µ̂ − Xt )2 /n. The
empirical Bernstein inequality says that for any δ ∈ (0, 1),
s    !
2σ̂ 2 3 3b 3
P |µ̂ − µ| ≥ log + log ≤ δ.
n δ n δ

1
Pn
(a) Show that σ̂ 2 = n t=1 (Xt − µ)
2
− (µ̂ − µ)2 .
2 2 2
(b) Show that V[(Xt − µ) ] ≤ b σ .
(c) Use Bernstein’s inequality (Exercise 5.14) to show that
s    !
2 2 2b2 σ 2 1 2b2 1
P σ̂ ≥ σ + log + log ≤ δ.
n δ 3n δ

(d) Suppose that ν = (νi )ki=1 is a bandit where Supp(νi ) ⊂ [0, b] and the variance
of the ith arm is σi2 (with our earlier notation, ν ∈ E[0,b]
k
). Design a policy
 7.4 Exercises 114

that depends on b, but not σi2 such that

X  
σi2
 
Rn ≤ C ∆i + b + log(n) , (7.14)
∆i
i:∆i >0

where C > 0 is a universal constant.

If you did things correctly, then the policy you derived in Exercise 7.6
should resemble UCB-V by Audibert et al. [2007]. The proof of the empirical
Bernstein also appears there or (with slightly better constants) in the papers
by Mnih et al. [2008] and Maurer and Pontil [2009].

It is worth comparing (7.14) to the result of Theorem 7.1. In particular,

recall that if the rewards are bounded by b, the reward distributions are
b-subgaussian. The regret of UCB which adjusts the confidence intervals
P
accordingly can then be shown to be Rn = O( i:∆i >0 b log(n) ∆i ). Thus, the
main advantage of the policy of the previous exercise is the replacement of
σ2
b/∆i in this bound with b + ∆ii . In Exercise 16.7, you will show that this is
essentially unimprovable.

7.7 (Median of means and bandits with known finite variance)

Let n ∈ N+ and (Ai )m i=1 be a partition of [n] so that ∪i=1 Ai = [n] and
m

Ai ∩ Aj = ∅ for all i = 6 j. Suppose that δ ∈ (0, 1) and X1 , X2 , . . . , Xn is a

sequence of independent random variables with mean µ and variance σ 2 . The
median-of-means estimator µ̂M of µ is the median of µ̂1 , µ̂2 , . . . , µ̂m , where
P
µ̂i = t∈Ai Xt /|Ai | is the mean of the data in the ith block.

j n  1/8 ok
(a) Show that if m = min n2 , 8 log e δ and Ai are chosen as equally
sized as possible, then
s   !
192σ 2 e1/8
P µ̂M + log ≤µ ≤ δ.
n δ

(b) Use the median-of-means estimator to design an upper confidence bound

algorithm such that for all ν ∈ EVk (σ 2 ),

X  σ 2 log(n)

Rn ≤ C ∆i + ,
∆i
i:∆i >0

where C > 0 is a universal constant.

 7.4 Exercises 115

This exercise shows that the subgaussian assumption can be relaxed to

requiring only finite variance at the price of increased constant factors. The
result is only possible by replacing the standard empirical estimator with
something more robust. The median-of-means estimator is only one way to
do this. In fact, the empirical estimator can be made robust by truncating
the observed rewards and applying the empirical Bernstein concentration
inequality. The disadvantage of this approach is that choosing the location
of truncation requires prior knowledge about the approximate location of
the mean. Another approach is Catoni’s estimator, which also exhibits
excellent asymptotic properties [Catoni, 2012]. Yet another idea is to minimise
the Huber loss [Sun et al., 2017]. This latter paper is focussing on linear
models, but the results still apply in one dimension. The application of these
ideas to bandits was first made by Bubeck et al. [2013a], where the reader
will find more interesting results. Most notably, that things can still be made
to work even if the variance does not exist. In this case, however, there is a
price to be paid in terms of the regret. The median-of-means estimator is due
to Alon et al. [1996]. In case the variance is also unknown, then it may be
estimated by assuming a known bound on the kurtosis, which covers many
classes of bandits (Gaussian with arbitrary variance, exponential and many
more), but not some simple cases (Bernoulli). The policy that results from
this procedure has the benefit of being invariant under the transformations
of shifting or scaling the losses [Lattimore, 2017].

7.8 (Empirical comparison)

(a) Implement Algorithm 3.
(b) Reproduce Fig. 7.1.
(c) Explain the shape of the curves for ETC. In particular, when m = 50, we
see a bump, a dip and then a linear asymptote as ∆ grows. Why does the
curve look like this?
(d) Design an experiment to determine the practical effect of the choice of δ.
(e) Explain your results.
8 The Upper Confidence Bound
Algorithm: Asymptotic Optimality

The algorithm analysed in the previous chapter is not anytime. This shortcoming
is resolved via a slight modification and a refinement of the analysis. The improved
analysis leads to constant factors in the dominant logarithmic term that match a
lower bound provided later in Chapter 16.

8.1 Asymptotically Optimal UCB

The algorithm studied is shown in Algorithm 6. It differs from the one analysed
in the previous section (Algorithm 3) only by the choice of the confidence level,
the choice of which is dictated by the analysis of its regret.

1: Input k
2: Choose each arm once
3: Subsequently choose
s !
2 log f (t)
At = argmaxi µ̂i (t − 1) +
Ti (t − 1)

where f (t) = 1 + t log2 (t)

Algorithm 6: Asymptotically optimal UCB.

The regret bound for Algorithm 6 is more complicated than the bound for
Algorithm 3 (see Theorem 7.1). The dominant terms in the two results have the
same order, but the gain here is that in this result the leading constant, governing
the asymptotic rate of growth of regret, is smaller.

Theorem 8.1. For any 1-subgaussian bandit, the regret of Algorithm 6 satisfies
  p 
X 5 2 log f (n) + π log f (n) + 1
Rn ≤ inf ∆i 1 + 2 + . (8.1)
ε∈(0,∆i ) ε (∆i − ε)2
i:∆i >0
 8.1 Asymptotically Optimal UCB 117

Furthermore,
Rn X 2
lim sup ≤ . (8.2)
n→∞ log(n) ∆i
i:∆i >0

Choosing ε = ∆i /2 inside the sum shows that

X  1  p 
Rn ≤ ∆i + 8 log f (n) + 8 π log f (n) + 28 . (8.3)
∆i
i:∆i >0

Even more concretely, there exists some universal constant C > 0 such that
X  log(n)

Rn ≤ C ∆i + ,
∆i
i:∆i >0

which by the same argument as

pin the proof of Theorem 7.2 leads a worst-case
Pk
bound of Rn ≤ C i=1 ∆i + 2 Cnk log(n).

Taking the limit of the ratio of the bound in (8.3) and log(n) does not result
in the same constant as in the theorem, which is the main justification for
introducing the more complicated regret bound. You will see in Chapter 15
that the asymptotic bound on the regret given in (8.2) is unimprovable in a
strong sense.

We start with a useful lemma to bound the number of times the index of a
suboptimal arm will be larger than some threshold above its mean.
Lemma 8.2. Let X1 , . . . , Xn be a sequence of independent 1-subgaussian random
Pt
variables, µ̂t = 1t s=1 Xs , ε > 0, a > 0 and
( r ) ( r )
2a 2a
Xn X n
κ= I µ̂t + ≥ε , κ =u+
0
I µ̂t + ≥ε ,
t=1
t t
t=due

2 √
where u = 2aε−2 . Then it holds E[κ] ≤ E[κ0 ] ≤ 1 + 2
(a + πa + 1).
ε
The intuition for this result is as follows. Since the Xi are
p1-subgaussian and
independent we have E[µ̂t ] = 0, so we cannot expect µ̂t + 2a/t to be smaller
than ε until t is at least 2a/ε2 . The lemma confirms that this is the right order
as an estimate for E [κ].
Proof By Corollary 5.5 we have
  q 2 
r ! 2a
n
X 2a
n
X  t ε − t 
E[κ] ≤ E[κ0 ] = u + P µ̂t + ≥ε ≤u+ exp − 
t 2
t=due t=due
  q 2 
Z 2a
 t ε− 2
∞ √
t 
≤1+u+ exp −  dt = 1 + 2 (a + πa + 1) ,
u 2 ε
 8.1 Asymptotically Optimal UCB 118

√ √
where the final equality follows by making the substitution s = ε t − 2a and
substituting the value of u from the lemma statement.

Proof of Theorem 8.1 As usual, the starting point is the fundamental regret
decomposition (Lemma 4.5),

X
Rn = ∆i E[Ti (n)] .
i:∆i >0

The rest of the proof revolves around bounding E[Ti (n)]. Let i be a suboptimal
arm. The main idea is to decompose Ti (n) into two terms. The first measures the
number of times the index of the optimal arm is less than µ1 − ε. The second term
measures the number of times that At = i and its index is larger than µ1 − ε.

( s )
2 log f (t)
n
X n
X
Ti (n) = I {At = i} ≤ I µ̂1 (t − 1) + ≤ µ1 − ε
t=1 t=1
T1 (t − 1)
( s )
2 log f (t)
n
X
+ I µ̂i (t − 1) + ≥ µ1 − ε and At = i . (8.4)
t=1
Ti (t − 1)

The proof of the first part of the theorem is completed by bounding the expectation
of each of these two sums. Starting with the first, we again use Corollary 5.5:

" ( s )#
2 log f (t)
n
X
E I µ̂1 (t − 1) + ≤ µ1 − ε
t=1
T1 (t − 1)
r !
2 log f (t)
Xn X n
≤ P µ̂1s + ≤ µ1 − ε
t=1 s=1
s
 q 2 
2 log f (t)
Xn X n  s s +ε 
 
≤ exp − 
t=1 s=1
 2 

 
1 X sε2 5
Xn n
≤ exp − ≤ 2.
t=1
f (t) s=1 2 ε

The first inequality follows from the union bound over all possible values of
T1 (t − 1). The last inequality is an algebraic exercise (Exercise 8.1). The function
f (t) was chosen precisely so this bound would hold. For the second term in (8.4)
 8.2 Notes 119

we use Lemma 8.2 to get

" n ( s )#
X 2 log f (t)
E I µ̂i (t − 1) + ≥ µ1 − ε and At = i
t=1
Ti (t − 1)
" n ( s )#
X 2 log f (n)
≤E I µ̂i (t − 1) + ≥ µ1 − ε and At = i
t=1
Ti (t − 1)
" n ( r )#
X 2 log f (n)
≤E I µ̂is + ≥ µ1 − ε
s=1
s
" n ( r )#
X 2 log f (n)
=E I µ̂is − µi + ≥ ∆i − ε
s=1
s
2  p 
≤1+ log f (n) + π log f (n) + 1 .
(∆i − ε) 2

The first part of the theorem follows by substituting the results of the previous
two displays into (8.4). The second part follows by choosing ε = log−1/4 (n) and
taking the limit as n tends to infinity.

8.2 Notes

1 The improvement to the constants comes from making the confidence interval
slightly smaller, which is made possible by a more careful analysis. The main
trick is the observation that we do not need to show that µ̂1s ≥ µ1 for all s
with high probability, but instead that µ̂1s ≥ µ1 − ε for small ε.
2 The choice of f (t) = 1 + t log2 (t) looks quite odd. With a slightly messier
calculation we could have chosen f (t) = t logα (t) for any α > 0. If the rewards
are actually Gaussian, then a more careful concentration analysis allows one
to choose f (t) = t or even some slightly slower-growing function [Katehakis
and Robbins, 1995, Lattimore, 2016a, Garivier et al., 2016b].
3 The asymptotic regret is often indicative of finite-time performance. The reader
is advised to be cautious, however. The lower-order terms obscured by the
asymptotics can be dominant in all practical regimes.

8.3 Bibliographic Remarks

Lai and Robbins [1985] designed policies for which Eq. (8.2) holds. They also
proved a lower bound showing that no ‘reasonable’ policy can improve on this
bound for any problem, where ‘reasonable’ means that they suffer subpolynomial
regret on all problems (see Part IV). The policy proposed by Lai and Robbins
[1985] was based on upper confidence bounds, but was not a variant of UCB. The
asymptotics for variants of the policy presented here were given first by Lai [1987],
 8.4 Exercises 120

Katehakis and Robbins [1995] and Agrawal [1995]. None of these articles gave
finite-time bounds like what was presented here. When the reward distributions
lie in an exponential family, then asymptotic and finite-time bounds with the
same flavor to what is presented here are given by Cappé et al. [2013]. There are
now a huge variety of asymptotically optimal policies in a wide range of settings.
Burnetas and Katehakis [1996] study the general case and give conditions for
a version of UCB to be asymptotically optimal. Honda and Takemura [2010,
2011] analyse an algorithm called DMED, proving asymptotic optimality for noise
models where the support is bounded or semi-bounded. Kaufmann et al. [2012b]
prove asymptotic optimality for Thompson sampling (see Chapter 36) when
the rewards are Bernoulli, which is generalised to single-parameter exponential
families by Korda et al. [2013]. Kaufmann [2018] proves asymptotic optimality
for the BayesUCB class of algorithms for single-parameter exponential families.
Ménard and Garivier [2017] prove asymptotic optimality and minimax optimality
for exponential families (more discussion in Chapter 9).

8.4 Exercises

8.1 Do the algebra needed at the end of the proof of Theorem 8.1. Precisely,
show that
 
1 X sε2 5
X n n
exp − ≤ 2,
t=1
f (t) s=1 2 ε

where f (t) = 1 + t log2 (t).

Pn
Hint First bound F = s=1 exp(−sε2 /2) using a geometric series. Then show
that exp(−a)/(1 − exp(−a)) ≤ 1/a holds for any a > 0 and conclude that F ≤ ε22 .
Pn
Finish by bounding t=1 1/f (t) using the fact that 1/f (t) ≤ 1/(t log(t)2 ) and
bounding a sum by an integral.
8.2 (One-armed bandits) Consider the one-armed bandit problem: E =
{N (µ1 , 1) : µ1 ∈ R} × {N (0, 1)}. Suppose that ν = (P1 , P2 ) ∈ E and P1 has mean
µ1 = 1. Evaluate
Rn (π, ν)
lim sup ,
n→∞ log(n)
where π is the policy of Algorithm 6.

8.3 (One-armed bandits (ii)) Consider the setting of Exercise 8.2 and define
a policy by
 q
1 if µ̂ (t − 1) + 2 log f (t) ≥ 0
1 T1 (t−1)
At = (8.5)
2 otherwise .
 8.4 Exercises 121

Suppose that ν = (P1 , P2 ) where P1 = N (µ1 , 1) and P2 = N (0, 1). Prove that
for the modified policy,
(
Rn (ν) 0 if µ1 ≥ 0
lim sup ≤
n→∞ log(n)
2
−µ 1
if µ1 < 0 .

Hint Follow the analysis for UCB, but carefully adapt the proof by using the
fact that the index of the second arm is always zero.

The strategy proposed in the above exercise is based on the idea that
optimism is used to overcome uncertainty in the estimates of the quality of
an arm, but for one-armed bandits the mean of the second arm is known in
advance.

8.4 (One-armed bandits (iii)) The purpose of this question is to compare

UCB and the modified version in (8.5).
(a) Implement a simulator for the one-armed bandit problem and two algorithms:
UCB and the modified version analysed in Exercise 8.3.
(b) Use your simulator to estimate the expected regret of each algorithm for a
horizon of n = 1000 and µ1 ∈ [−1, 1].
(c) Plot your results with µ1 on the x-axis and the estimated expected regret on
the y-axis. Don’t forget to label the axis and include error bars and a legend.
(d) Explain the results. Why do the curves look the way they do?
(e) In your plot, for what values of µ1 does the worst-case expected regret
for each algorithm occur? What is the worst-case expected regret for each
algorithm?

8.5 (Different subgaussian constants) Let σ 2 ∈ [0, ∞)k be known and

2
suppose that the reward is Xt ∼ N (µAt , σA t
). Design an algorithm (that depends
2
on σ ) for which the asymptotic regret is
Rn X 2σ 2
lim sup = i
.
n→∞ log(n) ∆i
i:∆i >0
9 The Upper Confidence Bound
Algorithm: Minimax Optimality ( )

We proved that the variants p of UCB analysed in the last two chapters have a
worst-case regret of Rn = O( kn log(n)). Further, p in Exercise 6.8 you showed
that an elimination algorithm achieves Rn = O( kn log(k)). By modifying the
confidence levels of the algorithm it is possible to remove the log factor entirely.
Building on UCB, the directly named ‘minimax optimal strategy in the stochastic
case’ (MOSS) algorithm was the first to make this modification and is presented
below. MOSS again depends on prior knowledge of the horizon, a requirement
that may be relaxed, as we explain in the notes.

The term minimax is used because, except for constant factors, the worst-
case bound proven in this chapter cannot be improved on by any algorithm.
The lower bounds are deferred to Part IV.

9.1 The MOSS Algorithm

Algorithm 7 shows the pseudocode of MOSS, which is again an instance of the

UCB family. The main novelty is that the confidence level is chosen based on the
number of plays of the individual arms, as well as n and k.

1: Input n and k
2: Choose each arm once
3: Subsequently choose
s  
4 n
At = argmaxi µ̂i (t − 1) + log+ ,
Ti (t − 1) kTi (t − 1)
where log+ (x) = log max {1, x} .

Algorithm 7: MOSS.
 9.1 The MOSS Algorithm 123

Theorem 9.1. For any 1-subgaussian bandit, the regret of Algorithm 7 satisfies

√ k
X
Rn ≤ 39 kn + ∆i .
i=1

Before the proof we state and prove a strengthened version of Corollary 5.5.

Theorem 9.2. Let X1 , X2 , . . . , Xn be a sequence of independent σ-subgaussian

Pt
random variables and St = s=1 Xs . Then, for any ε > 0,
 
ε2
P (exists t ≤ n : St ≥ ε) ≤ exp − . (9.1)
2nσ 2

The bound in Eq. (9.1) is the same as the bound on P (Sn ≥ ε) that appears
in a simple reformulation of Corollary 5.5, so this new result is strictly stronger.

Proof From the definition of subgaussian random variables and Lemma 5.4,
 
nσ 2 λ2
E [exp (λSn )] ≤ exp .
2

Then, choosing λ = ε/(nσ 2 ) leads to

 
P (exists t ≤ n : St ≥ ε) = P max exp (λSt ) ≥ exp (λε)
t≤n
 2 2   
E [exp (λSn )] nσ λ ε2
≤ ≤ exp − λε = exp − .
exp (λε) 2 2nσ 2

The novel step is the first inequality, which follows from Doob’s submartingale
inequality (Theorem 3.10) and the fact that that exp(λSt ) is a submartingale
with respect to the filtration generated by X1 , X2 , . . . , Xn (Exercise 9.1).

Before the proof of Theorem 9.1, we need one more lemma to bound the
probability that the index of the optimal arm ever drops too far below the actual
mean of the optimal arm. The proof of this lemma relies on a tool called the
peeling device, which is an important technique in probability theory and has
many applications beyond bandits. For example, it can be used to prove the
celebrated law of the iterated logarithm.

Lemma 9.3. Let δ ∈ (0, 1) and X1 , X2 , . . . be independent and 1-subgaussian and

Pt
µ̂t = 1t s=1 Xs . Then, for any ∆ > 0,
s   !
4 + 1 15δ
P exists s ≥ 1 : µ̂s + log +∆≤0 ≤ 2 .
s sδ ∆
 9.1 The MOSS Algorithm 124

Proof Let St = tµ̂t . Then

s   !
4 + 1
P exists s ≥ 1 : µ̂s + log +∆≤0
s sδ
s   !
+ 1
= P exists s ≥ 1 : Ss + 4s log + s∆ ≤ 0
sδ
s   !
+ 1
X∞
≤ P exists s ∈ [2 , 2 ] : Ss + 4s log
j j+1
+ s∆ ≤ 0
j=0
sδ
s   !
1
X∞
+
≤ P exists s ≤ 2 j+1
: Ss + 4 · 2 log j +2 ∆≤0
j

j=0
2j+1 δ
 q
2 
∞ 2 j+2 log+ 1
+ 2 j
∆
X  2j+1 δ 
≤ exp − .
j=0
2j+2

The first inequality follows from a union bound over a geometric grid. The second
step is straightforward but important because it sets up to apply Theorem 9.2.
The rest is purely algebraic:
 q
2 
∞ 2 j+2 log+ 1
+ 2 j
∆ ∞
X  2 j+1 δ  X

exp −  ≤ δ 2j+1 exp −∆2 2j−2
j=0
2j+2
j=0
Z
8δ ∞
15δ
≤ +δ 2s+1 exp −∆2 2s−2 ds ≤ 2 .
e∆2 0 ∆
Above, the first inequality follows since (a + b)2 ≥ a2 + b2 for a, b ≥ 0, and
the second last step follows by noting that the integrand is unimodal and
2
has a maximum value of 8δ/(e∆ R b ). For such functions f , one has the bound
Pb
j=a f (j) ≤ maxs∈[a,b] f (s) + a f (s)ds.

Proof of Theorem 9.1 As usual, we assume without loss of generality that the
first arm is optimal, so µ1 = µ∗ . Arguing that the optimal arm is sufficiently
optimistic with high probability is no longer satisfactory because in this refined
analysis, the probability that an arm is played linearly often needs to depend
on its suboptimality gap. A way around this difficulty is to make an argument
in terms of the expected amount of optimism. Define a random variable ∆ that
measures how far below the index of the optimal arm drops below its true mean.
r !!+
4 n
∆ = µ1 − min µ̂1s + log+ .
s≤n s ks

Arms with suboptimality gaps much larger than ∆ will not be played too often,
while arms with suboptimality gaps smaller than ∆ may be played linearly often,
but ∆ is sufficiently small in expectation that this price is small. Using the basic
 9.1 The MOSS Algorithm 125

regret decomposition (Lemma 4.5) and splitting the actions based on whether or
not their suboptimality gap is smaller or larger than 2∆ leads to

X
Rn = ∆i E[Ti (n)]
i:∆i >0
 
X
≤ E 2n∆ + ∆i Ti (n)
i:∆i >2∆
 
 √ X 
≤ E 2n∆ + 8 kn + ∆i Ti (n) .
 √
i:∆i >max 2∆,8 k/n

The first term is easily bounded using Proposition 2.8 and Lemma 9.3:

Z Z  
∞ ∞
15k √
E[2n∆] = 2nE[∆] = 2n P (∆ ≥ x) dx ≤ 2n min 1, 2 dx ≤ 16 kn .
0 0 nx

For suboptimal arm i, define

( r )
n
X 4 n
κi = I µ̂is + log+ ≥ µi + ∆i /2 .
s=1
s ks

The reason for choosing κi in this way is that for arms i with ∆i > 2∆, it holds
that the index of the optimal arm is always larger than µi + ∆i /2, so κi is an
upper bound on the number of times arm i is played, Ti (n). If ∆i ≥ 8(k/n)1/2 ,
then the expectation of ∆i κi is bounded using Lemma 8.2 by

" n ( s   )#
1 X 4 2
+ n∆i
∆i E[κi ] ≤ + ∆i E I µ̂is + log ≥ µi + ∆i /2
∆i s=1
s k
 2
 s  2
 !
1 8 + n∆ + n∆i
≤ + ∆i + 2 log i
+ 2π log +1
∆i ∆i k k
r r   r
1 n n p n
≤ + ∆i + 4 log 8 + 2 π log 8 + 1 ≤ ∆i + 15 ,
8 k k k

where the first inequality follows by replacing the s in the logarithm with 1/∆2i
and adding the ∆i × 1/∆2i correction term to compensate for the first ∆−2 i
rounds where this fails to hold. Then we use Lemma 8.2 and the monotonicity of
√
x 7→ x−1−p log+ (ax2 ) for p ∈ [0, 1], positive a and x ≥ e/ a. The last inequality
 9.2 Two Problems 126

√
follows by naively bounding 1/8 + 4 log 8 + 2 π log 8 + 1 ≤ 15. Then
   
 X   X 
E ∆i Ti (n) ≤ E  ∆i κi 
 √ √
i:∆i >max 2∆,8 k/n i:∆i >8 k/n
 r 
X n
≤ ∆i + 15
√ k
i:∆i >8 k/n

√ k
X
≤ 15 nk + ∆i .
i=1
√ Pk
Combining all the results we have Rn ≤ 39 kn + i=1 ∆i .

9.2 Two Problems

MOSS is not the ultimate algorithm. Here we highlight two drawbacks.

Suboptimality Relative to UCB

Although MOSS is nearly asymptotically optimal (Note 1), all versions of MOSS
can be arbitrarily worse than UCB in some regimes. This unpleasantness is hidden
by both the minimax and asymptotic optimality criteria, which highlights the
importance of fully finite-time upper and lower bounds. The counter-example
witnessing the failure is quite simple. Let the rewards
p for all arms be Gaussian
with unit variance and n = k 3 , µ1 = 0, µ2 = − k/n and µi = −1 for all i > 2.
From Theorem 8.1, we have that

RnUCB = O(k log k) ,

while it turns out that MOSS has a regret of

√
RnMOSS = Ω( kn) = Ω(k 2 ) .

A rigourous proof of this claim is quite delicate, but we encourage readers to try
to understand why it holds intuitively.

Instability
There is a hidden cost of pushing too hard to reduce the expected regret, which
is that the distribution of the regret is less well-behaved. Consider a two-armed
Gaussian bandit with suboptimality gap ∆. The random (pseudo) regret is
Pn
R̂n = t=1 ∆At , which for a carefully tuned algorithm has a roughly bimodal
distribution:
(
n∆ with probability δ
R̂n ≈ 1 1


∆ log δ otherwise ,
 9.3 Notes 127

where δ is a parameter of the policy that determines the likelihood that the
optimal arm is misidentified. Integrating, one has
  
1 1
Rn = E[R̂n ] = O n∆δ + log ,
∆ δ
The choice of δ that minimises the expected regret depends on ∆ and is
approximately 1/(n∆2 ). With this choice, the regret is
 
1 2



Rn = O 1 + log n∆ .
∆
Of course ∆ is not known in advance, but it can be estimated online so that the
above bound is actually realisable by an adaptive policy that does not know ∆
in advance (Exercise 9.3). Let F be the (informal) event that R̂n = Ω(n∆). The
problem is that when δ = 1/(n∆2 ) is chosen to minimise the expected regret,
then the second moment due to failure is
E[IF R̂n2 ] = Ω(n) .
On the other hand, by choosing δ = (n∆)−2 , the regret increases only slightly to
  
1 1

Rn = O + log n2 ∆2 .
∆ n
The second moment of the regret due to failure, however, is E[IF R̂n2 ] = O(1).

9.3 Notes

1 MOSS is quite close to asymptotically optimal. You can prove that

Rn X 4
lim sup ≤ .
n→∞ log(n) ∆i
i:∆i >0

By modifying the algorithm slightly, it is even possible to replace the four

with a two and recover the optimal asymptotic regret. The trick is to increase
g slightly and replace the four in the exploration bonus by two. The major
task is then to re-prove Lemma 9.3, which is done by replacing the intervals
[2j , 2j+1 ] with smaller intervals [ξ j , ξ j+1 ], where ξ is tuned subsequently to be
fractionally larger than one. This procedure is explained in detail by Garivier
[2013]. When the reward distributions are actually Gaussian, there is a more
elegant technique that avoids peeling altogether (Exercise 9.4).
2 One way to mitigate the issues raised in Section 9.2 is to replace the index
used by MOSS with a less aggressive confidence level:
s  
4 + n
µ̂i (t − 1) + log . (9.2)
Ti (t − 1) Ti (t − 1)
The resulting algorithm is never worse than UCB, and you will show in
 9.4 Bibliographic Remarks 128

p
Exercise 9.3 that it has a distribution free regret of O( nk log(k)). An
algorithm that does almost the same thing in disguise is called ‘improved
UCB’, which operates in phases and eliminates arms for which the upper
confidence bound drops below a lower confidence bound for some arm [Auer
and Ortner, 2010]. This algorithm was the topic of Exercise 6.8.
3 Overcoming the failure of MOSS to be instance optimal without sacrificing
minimax optimality is possible by using an adaptive confidence level that tunes
the amount of optimism to match the instance. One of the authors has proposed
two ways to do this, using one of the following indices:
s
2(1 + ε) n
µ̂i (t − 1) + log , or (9.3)
Ti (t − 1) t
v !
u
u 2 n
µ̂i (t − 1) + t log Pk p .
Ti (t − 1) j=1 min{Ti (t − 1), Ti (t − 1)Tj (t − 1)}

The first of these algorithms is called the ‘optimally confident UCB’ [Lattimore,
2015b] while the second is AdaUCB [Lattimore, 2018]. Both algorithms are
minimax optimal up to constant factors and never worse than UCB. The
latter is also asymptotically optimal. If the horizon is unknown, then AdaUCB
can be modified by replacing n with t. It remains a challenge to provide a
straightforward analysis for these algorithms.

9.4 Bibliographic Remarks

MOSS is due to Audibert and Bubeck [2009], while an anytime modification

is by Degenne and Perchet [2016]. The proof that a modified version of MOSS
is asymptotically optimal may be found in the article by Ménard and Garivier
[2017]. There is also a variant of MOSS that adapts to the variance for rewards
bounded in [0, 1] [Mukherjee et al., 2018]. AdaUCB and its friends are by one of
the authors [Lattimore, 2015b, 2016b, 2018]. The idea to modify the confidence
level has been seen in several places, with the earliest by Lai [1987] and more
recently by Honda and Takemura [2010]. Kaufmann [2018] also used a confidence
level like in Eq. (9.2) to derive an algorithm based on Bayesian upper confidence
bounds.

9.5 Exercises

9.1 (Submartingale property) Let X1 , X2 , . . . , Xn be adapted to filtration

Pt
F = (Ft )t with E[Xt | Ft−1 ] = 0 almost surely. Prove that Mt = exp(λ s=1 Xs )
is a F-submartingale for any λ ∈ R.

9.2 (Problem-dependent bound) Let ∆min = mini:∆i >0 ∆i . Show there exists
 9.5 Exercises 129

a universal constant C > 0 such that the regret of MOSS is bounded by

  X
n∆2min
k
Ck
Rn ≤ log+ + ∆i .
∆min k i=1

9.3 (UCB*) Suppose we modify the index used by MOSS to be

s  
4 + n
µ̂i (t − 1) + log .
Ti (t − 1) Ti (t − 1)

(a) Show that for all 1-subgaussian bandits, this new policy suffers regret at
most
 
X 1
Rn ≤ C  ∆i + log+ (n∆2i ) ,
∆i
i:∆i >0

where C > 0 is a universal constant.

(b) Under the same conditions as the previous part, show there exists a universal
constant C > 0 such that
p k
X
Rn ≤ C kn log(k) + ∆i .
i=1

(c) Repeat parts (a) and (b) using the index

s  
4 t
µ̂i (t − 1) + log+ .
Ti (t − 1) Ti (t − 1)

9.4 (Gaussian noise and the tangent approximation) Let g(t) = at + b

with b > 0 and
 2  
1 x 1 (x − 2b)2
u(x, t) = √ exp − −√ exp −2ab − .
2πt 2t 2πt 2t
(a) Show that u(x, t) > 0 for x ∈ (−∞, g(t)) and u(x, t) = 0 for x = g(t).
(b) Show that u(x, t) satisfies the heat equation:
1 2
∂t u(x, t) = ∂ u(x, t) .
2 x
(c) Let Bt be a standard Brownian motion, which for any fixed t has density
with respect to the Lebesgue measure.
 2
1 x
p(x, t) = √ exp − .
2πt 2t
Define τ = min{t : Bt = g(t)} as the first time the Brownian motion hits the
boundary. Put on your physicists hat (or work hard) to argue that
Z g(t)
P (τ ≥ t) = u(x, t)dx .
−∞
 9.5 Exercises 130

(d) Let v(t) be the density

Rt of time τ with respect to the Lebesgue measure so
that P (τg ≤ t) = 0 v(t)dt. Show that
 
b g(t)2
v(t) = √ exp −
2πt3 2t
(e) In the last part, you established the exact density of the hitting time of a
Brownian motion approaching a linear boundary. We now generalise this
to nonlinear boundaries, but at the cost that now we only have a bound.
Suppose that f : [0, ∞) → [0, ∞) is concave and differentiable, and let
λ : R → R be the intersection of the tangent to f at t with the y-axis given
by λ(t) = f (t) − tf 0 (t). Let τ = min{t : Bt = f (t)} and v(t) be the density
of τ . Show that for t > 0,
 
λ(t) f (t)2
v(t) ≤ √ exp − .
2πt3 2t
(f) Suppose that X1 , X2 , . . . is a sequence of independent standard Gaussian
random variables. Show that
! Z  
f (t)2
X t n
λ(t)
P exists t ≤ n : Xs ≥ f (t) ≤ √ exp − dt .
s=1 0 2πt3 2t

(g) Let
p h : (0, ∞) → (1, ∞) be a concave increasing function p such that
log(h(a))/h(a) ≤ c/a for some constant c > 0 and f (t) = 2t log h(1/tδ)+
t∆. Show that
!
2cδ
Xt
P exists t : Xs ≥ f (t) ≤ √ 2 .
s=1
π∆
p
(h) Show that h(a) = 1 + (1 + a) log(1 + a) satisfies the requirements of the
previous part with c = 11/10.
(i) Use your results to modify MOSS for the case when the rewards are Gaussian.
Compare the algorithms empirically.
(j) Prove for your modified algorithm that
Rn X 2
lim sup ≤ .
n→∞ log(n) ∆i
i:∆i >0

Hint The above exercise has several challenging components and assumes
prior knowledge of Brownian motion and its interpretation in terms of the heat
equation. We recommend the book by Lerche [1986] as a nice reference on hitting
times for Brownian motion against concave barriers. The equation you derived in
Part (d) is called the Bachelier–Lévy formula , and the technique for doing
so is the method of images. The use of this theory in bandits was introduced
by one of the authors [Lattimore, 2018], which readers might find useful when
working through these questions.
9.5 (Asymptotic optimality and subgaussian noise) In the last exercise,
 9.5 Exercises 131

you modified MOSS to show asymptotic optimality when the noise is Gaussian.
This is also possible for subgaussian noise. Follow the advice in the notes of this
chapter to adapt MOSS so that for all 1-subgaussian bandits, it holds that
Rn X 2
lim sup ≤ ,
n→∞ log(n) ∆i
i:∆i >0
√
while maintaining the property that Rn ≤ C kn for universal constant C > 0.
10 The Upper Confidence Bound
Algorithm: Bernoulli Noise ( )

In previous chapters we assumed that the noise of the rewards was σ-subgaussian
for some known σ > 0. This has the advantage of simplicity and relative generality,
but stronger assumptions are sometimes justified and often lead to stronger results.
In this chapter the rewards are assumed to be Bernoulli, which just means that
Xt ∈ {0, 1}. This is a fundamental setting found in many applications. For
example, in click-through prediction, the user either clicks on the link or not. A
Bernoulli bandit is characterised by the mean pay-off vector µ ∈ [0, 1]k and the
reward observed in round t is Xt ∼ B(µAt ).
The Bernoulli distribution is 1/2-subgaussian regardless of its mean
(Exercise 5.12). Hence the results of the previous chapters are applicable, and an
appropriately tuned UCB enjoys logarithmic regret. The additional knowledge
that the rewards are Bernoulli is not being fully exploited by these algorithms,
however. The reason is essentially that the variance of a Bernoulli random
variable depends on its mean, and when the variance is small, the empirical mean
concentrates faster, a fact that should be used to make the confidence intervals
smaller.

10.1 Concentration for Sums of Bernoulli Random Variables

The first step when designing a new optimistic algorithm is to construct confidence
sets for the unknown parameters. For Bernoulli bandits, this corresponds to
analysing the concentration of the empirical mean for sums of Bernoulli random
variables. For this, the following definition will prove useful:

Definition 10.1 (Relative entropy between Bernoulli distributions). The

relative entropy between Bernoulli distributions with parameters p, q ∈ [0, 1] is

d(p, q) = p log(p/q) + (1 − p) log((1 − p)/(1 − q)) ,

where singularities are defined by taking limits: d(0, q) = log(1/(1 − q)) and
d(1, q) = log(1/q) for q ∈ [0, 1] and d(p, 0) = 0 if p = 0 and ∞ otherwise and
d(p, 1) = 0 if p = 1 and ∞ otherwise.
 10.1 Concentration for Sums of Bernoulli Random Variables 133

More generally, the relative entropy or Kullback–Leibler divergence

is a measure of similarity between distributions. See Chapter 14 for a generic
definition, interpretation and discussion.

Lemma 10.2. Let p, q, ε ∈ [0, 1]. The following hold:

(a) The functions d(·, q) and d(p, ·) are convex and have unique minimisers at q
and p, respectively.
(b) d(p, q) ≥ 2(p − q)2 (Pinsker’s inequality).
(c) If p ≤ q − ε ≤ q, then d(p, q − ε) ≤ d(p, q) − d(q − ε, q) ≤ d(p, q) − 2ε2 .

Proof We assume that p, q ∈ (0, 1). The corner cases are easily checked
separately. Part (a): d(·, q) is the sum of the negative binary entropy function
h(p) = p log p + (1 − p) log(1 − p) and a linear function. The second derivative
of h is h00 (p) = 1/p + 1/(1 − p), which is positive, and hence h is convex. For
fixed p the function d(p, ·) is the sum of h(p) and convex functions p log(1/q) and
(1 − p) log(1/(1 − q)). Hence d(p, ·) is convex. The minimiser property follows
because d(p, q) > 0 unless p = q in which case d(p, p) = d(q, q) = 0. A more
general version of (b) is given in Chapter 15. A proof of the simple version here
follows by considering the function g(x) = d(p, p + x) − 2x2 , which obviously
satisfies g(0) = 0. The proof is finished by showing that this is the unique
minimiser of g over the interval [−p, 1 − p]. The details are left to Exercise 10.1.
For (c), notice that
q 1−q
h(p) = d(p, q − ε) − d(p, q) = p log + (1 − p) log .
q−ε 1−q+ε
It is easy to see then that h is linear and increasing in its argument. Therefore,
since p ≤ q − ε,

h(p) ≤ h(q − ε) = −d(q − ε, q) ,

as required for the first inequality of (c). The second inequality follows by using
the result in (b).

The next lemma controls the concentration of the sample mean of a sequence
of independent and identically distributed Bernoulli random variables.

Lemma 10.3 (Chernoff’s bound). Let X1 , X2 , . . . , Xn be a sequence of independent

random variables that are Bernoulli distributed with mean µ, and let µ̂ =
1
Pn
n t=1 Xt be the sample mean. Then, for ε ∈ [0, 1 − µ], it holds that

P (µ̂ ≥ µ + ε) ≤ exp (−nd(µ + ε, µ)) (10.1)

and for ε ∈ [0, µ],

P (µ̂ ≤ µ − ε) ≤ exp (−nd(µ − ε, µ)) . (10.2)

 10.1 Concentration for Sums of Bernoulli Random Variables 134

Proof We will again use the Cramér–Chernoff method. Let λ > 0 be some
constant to be chosen later. Then,
n
! !
X
P (µ̂ ≥ µ + ε) = P exp λ (Xt − µ) ≥ exp (λnε)
t=1
Pn
E [exp (λ t=1 (Xt − µ))]
≤
exp (λnε)
= (µ exp(λ(1 − µ − ε)) + (1 − µ) exp(−λ(µ + ε))) .
n

This expression is minimised by λ = log (µ+ε)(1−µ)

µ(1−µ−ε) . Therefore,

P (µ̂ ≥ µ + ε)
 1−µ−ε  −µ−ε !n
(µ + ε)(1 − µ) (µ + ε)(1 − µ)
≤ µ + (1 − µ)
µ(1 − µ − ε) µ(1 − µ − ε)
 1−µ−ε !n
µ (µ + ε)(1 − µ)
=
µ+ε µ(1 − µ − ε)
= exp (−nd(µ + ε, µ)) .
The bound on the left tail is proven identically.
Using Pinsker’s inequality, it follows that P (µ̂ ≥ µ + ε) , P (µ̂ ≤ µ − ε) ≤
exp(−2nε2 ), which is the same as what can be obtained from Hoeffding’s lemma
(see (5.8)). Solving exp(−2nε2 ) = δ, we recover the usual 1 − δ confidence upper
bound. In fact, this cannot be improved when µ ≈ 1/2, but the Chernoff bound
is much stronger when µ is close to either zero or one. Can we invert the Chernoff
tail bound to get confidence intervals that get tighter automatically as µ (or µ̂)
approaches zero or one? The following corollary shows how to do this.
Corollary 10.4. Let µ, µ̂, n be as above. Then, for any a ≥ 0,
P (d(µ̂, µ) ≥ a, µ̂ ≤ µ) ≤ exp(−na) , (10.3)
and P (d(µ̂, µ) ≥ a, µ̂ ≥ µ) ≤ exp(−na) . (10.4)
Furthermore, defining
U (a) = max{u ∈ [0, 1] : d(µ̂, u) ≤ a} ,
and L(a) = min{u ∈ [0, 1] : d(µ̂, u) ≤ a} .
Then, P (µ ≥ U (a)) ≤ exp(−na) and P (µ ≤ L(a)) ≤ exp(−na).
Proof First, we prove (10.3). Note that d(·, µ) is decreasing on [0, µ], and thus,
for 0 ≤ a ≤ d(0, µ), {d(µ̂, µ) ≥ a, µ̂ ≤ µ} = {µ̂ ≤ µ − x, µ̂ ≤ µ} = {µ̂ ≤ µ − x},
where x is the unique solution to d(µ − x, µ) = a on [0, µ]. Hence, by Eq. (10.2)
of Lemma 10.3, P (d(µ̂, µ) ≥ a, µ̂ ≤ µ) ≤ exp(−na). When a ≥ d(0, µ), the
inequality trivially holds. The proof of (10.4) is entirely analogous and hence
is omitted. For the second part of the corollary, fix a and let U = U (a).
 10.1 Concentration for Sums of Bernoulli Random Variables 135

First, notice that U ≥ µ̂ and d(µ̂, ·) is strictly increasing on [µ̂, 1]. Hence,
{µ ≥ U } = {µ ≥ U, µ ≥ µ̂} = {d(µ̂, µ) ≥ d(µ̂, U ), µ ≥ µ̂} = {d(µ̂, µ) ≥ a, µ ≥ µ̂},
where the last equality follows by d(µ̂, U ) = a, which holds by the definition
of U . Taking probabilities and using the first part of the corollary shows that
P (µ ≥ U ) ≤ exp(−na). The statement concerning L = L(a) follows with a similar
reasoning.

Note that for δ ∈ (0, 1), U = U (log(1/δ)/n) and L = L(log(1/δ)/n) are upper
and lower confidence bounds for µ. Although the relative entropy has no closed-
form inverse, the optimisation problem that defines U and L can be solved to a
high degree of accuracy using Newton’s method (the relative entropy d is convex
in its second argument). The advantage of this confidence interval relative to
the one derived from Hoeffding’s bound is now clear. As µ̂ approaches one, the
width of the interval U (a) − µ̂ approaches zero,
p whereas the width of the interval
provided by Hoeffding’s bound stays at log(1/δ)/(2n). The same holds for
µ̂ − L(a) as µ̂ → 0.

Example 10.5. Fig. 10.1 shows a plot of d(3/4, x) and the lower bound given
by Pinsker’s inequality. The approximation degrades as |x − 3/4| grows large,
especially for x > 3/4. As explained in Corollary 10.4, the graph of d(µ̂, ·) can
be used to derive confidence bounds by solving for d(µ̂, x) = a = log(1/δ)/n.
Assuming µ̂ = 3/4 is observed, a confidence level of 90 per cent with n = 10,
a ≈ 0.23. The confidence interval can be read out from the figure by finding
those values where the horizontal dashed black line intersects the solid blue line.
The resulting confidence interval will be highly asymmetric. Note that in this
scenario, the lower confidence bounds produced by both Hoeffding’s inequality
and Chernoff’s bound are similar, while the upper bound provided by Hoeffding’s
bound is vacuous.

d(3/4, x)
0.6 2(x − 3/4)2
a = 0.23
0.4

0.2

0
0 0.25 0.5 0.75 1
x

Figure 10.1 Relative entropy and Pinsker’s inequality

 10.2 The KL-UCB Algorithm 136

10.2 The KL-UCB Algorithm

The difference between KL-UCB and UCB is that Chernoff’s bound is used to
define the upper confidence bound instead of Lemma 5.5.

1: Input k
2: Choose each arm once
3: Subsequently choose
 
log f (t)
At = argmaxi max µ̃ ∈ [0, 1] : d(µ̂i (t − 1), µ̃) ≤ ,
Ti (t − 1)
where f (t) = 1 + t log2 (t) .

Algorithm 8: KL-UCB.

Theorem 10.6. If the reward in round t is Xt ∼ B(µAt ), then the regret of

Algorithm 8 is bounded by
 
X log(f (n)) 1 2
Rn ≤ inf ∆i + + 2 .
ε1 ,ε2 >0 d(µi + ε1 , µ∗ − ε2 ) 2ε21 ε2
i:∆i >0 ε1 +ε2 ∈(0,∆i )

Rn X ∆i
Furthermore, lim sup ≤ .
n→∞ log(n) d(µi , µ∗ )
i:∆i >0

Comparing the regret in Theorem 10.6 to what would be obtained when using
UCB from Chapter 8, which for subgaussian constant σ = 1/2 satisfies
Rn X 1
lim sup ≤ .
n→∞ log(n) 2∆i
i:∆i >0

By Pinsker’s inequality (part (b) of Lemma 10.2) we see that d(µi , µ∗ ) ≥

2(µ∗ − µi )2 = 2∆2i , which means that the asymptotic regret of KL-UCB is
never worse than that of UCB. On the other hand, a Taylor’s expansion shows
that when µi and µ∗ are close (the hard case in the asymptotic regime),
∆2i
d(µi , µ∗ ) = + o(∆2i ) ,
2µi (1 − µi )
indicating that the regret of KL-UCB is approximately
Rn X 2µi (1 − µi )
lim sup ≈ . (10.5)
n→∞ log(n) ∆i
i:∆i >0

Notice that µi (1 − µi ) is the variance of a Bernoulli distribution with mean µi .

The approximation indicates that KL-UCB will improve on UCB in regimes
where µi is close to zero or one.
The proof of Theorem 10.6 relies on two lemmas. The first is used to show
that the index of the optimal arm is never too far below its true value, while the
 10.2 The KL-UCB Algorithm 137

second shows that the index of any other arm is not often much larger than the
same value. These results mirror those given for UCB, but things are complicated
by the non-symmetric and hard-to-invert divergence function.
For the next results, we define d(p, q) = d(p, q)I {p ≤ q}.

Lemma 10.7. Let X1 , X2 , . . . , Xn be independent Bernoulli random variables with

mean µ ∈ [0, 1], ε > 0 and
 
log f (t)
τ = min t : max d(µ̂s , µ − ε) − ≤0 .
1≤s≤n s
2
Then, E[τ ] ≤ .
ε2
Proof We start with a high-probability bound and then integrate to control the
expectation.
 
log f (t)
P (τ > t) ≤ P ∃1 ≤ s ≤ n : d(µ̂s , µ − ε) >
s
 
log f (t)
Xn
≤ P d(µ̂s , µ − ε) >
s=1
s
 
log f (t)
Xn
= P d(µ̂s , µ − ε) > , µ̂s < µ − ε
s=1
s
 
log f (t)
Xn
≤ P d(µ̂s , µ) > + 2ε2 , µ̂s < µ ((c) of Lemma 10.2)
s=1
s
  
log f (t)
Xn
2
≤ exp −s 2ε + (Eq. (10.3) of Corollary 10.4)
s=1
s
1 X
n


≤ exp −2sε2
f (t) s=1
1
≤ .
2f (t)ε2
To finish, we integrate the tail,
Z ∞ Z ∞
1 dt 2
E[τ ] = P (τ ≥ t) dt ≤ 2 ≤ 2.
0 2ε 0 f (t) ε
Lemma 10.8. Let X1 , X2 , . . . , Xn be independent Bernoulli random variables with
mean µ. Further, let ∆ > 0, a > 0 and define
Xn n ao
κ= I d(µ̂s , µ + ∆) ≤ .
s=1
s
 
a 1
Then, E[κ] ≤ inf + .
ε∈(0,∆) d(µ + ε, µ + ∆) 2ε2
 10.2 The KL-UCB Algorithm 138

Proof Let ε ∈ (0, ∆) and u = a/d(µ + ε, µ + ∆). Then,

Xn  a
E[κ] = P d(µ̂s , µ + ∆) ≤
s=1
s
Xn  a
≤ P µ̂s ≥ µ + ε or d(µ + ε, µ + ∆) ≤
s=1
s
(d(·, µ + ∆) is decreasing on [0, µ + ∆])
n
X
≤u+ P (µ̂s ≥ µ + ε)
s=due
∞
X
≤u+ exp (−sd(µ + ε, µ)) (Lemma 10.3)
s=1
a 1
≤ +
d(µ + ε, µ + ∆) d(µ + ε, µ)
a 1
≤ + 2 (Pinsker’s inequality/Lemma 10.2(b))
d(µ + ε, µ + ∆) 2ε
as required.

Proof of Theorem 10.6 As in other proofs, we assume without loss of generality

that µ1 = µ∗ and bound E[Ti (n)] for suboptimal arms i. To this end, fix a
suboptimal arm i and let ε1 + ε2 ∈ (0, ∆i ) with both ε1 and ε2 positive. Define
 
log f (t)
τ = min t : max d(µ̂1s , µ1 − ε2 ) − ≤ 0 , and
1≤s≤n s
 
log f (n)
Xn
κ= I d(µ̂is , µi + ∆i − ε2 ) ≤ .
s=1
s

Using a similar argument as in the proof of Theorem 8.1,

" n #
X
E[Ti (n)] = E I {At = i}
t=1
" n
#
X
≤ E[τ ] + E I {At = i}
t=τ +1
" n  #
X log f (t)
≤ E[τ ] + E I At = i and d(µ̂i,Ti (t−1) , µ1 − ε2 ) ≤
t=1
Ti (t − 1)
≤ E[τ ] + E[κ]
2 f (n) 1
≤ 2 + + 2,
ε2 d(µi + ε1 , µ − ε2 ) 2ε1
∗

where the second inequality follows, since by the definition of τ , if t > τ , then
the index of the optimal arm is at least as large as µ1 − ε2 . The third inequality
follows from the definition of κ as in the proof of Theorem 8.1. The final inequality
 10.3 Notes 139

follows from Lemmas 10.7 and 10.8. The first claim of the theorem is completed
by substituting the above into the standard regret decomposition
k
X
Rn = ∆i E[Ti (n)] .
i=1

The asymptotic claim for you in Exercise 10.2.

10.3 Notes

1 The new concentration inequality (Lemma 10.3) holds more generally for
any sequence of independent and identically distributed random variables
X1 , X2 , . . . , Xn for which Xt ∈ [0, 1] almost surely. Therefore all results in
this section also hold if the assumption that the noise is Bernoulli is relaxed
to the case where it is simply supported in [0, 1] (or other bounded sets by
shifting/scaling).
2 Expanding on the previous note, all that is required is a bound on the moment-
generating function for random variables X where, X ∈ [0, 1] almost surely.
Garivier and Cappé [2011, Lemma 9] noted that f (x) = exp(λx) − x(exp(λ) −
1) − 1 is negative on [0, 1], and so
E [exp(λX)] ≤ E [X(exp(λ) − 1) + 1] = µ exp(λ) + 1 − µ ,
which is precisely the moment-generating function of the Bernoulli distribution
with mean µ. Then the remainder of the proof of Lemma 10.3 goes through
unchanged. This shows that for any bandit ν = (Pi )i with Supp(Pi ) ∈ [0, 1] for
all i the regret of the policy in Algorithm 8 satisfies
Rn X ∆i
lim sup ≤ .
n→∞ log(n) d(µi , µ∗ )
i:∆i >0

3 The bounds obtained using the argument in the previous note are not quite
tight. Specifically one can show there exists an algorithm such that for all
bandits ν = (Pi )i with Pi , the reward distribution of the ith arm supported on
[0, 1], then
Rn X ∆i
lim sup = , where
n→∞ log(n) di
i:∆i >0

di = inf{D(Pi , P ) : µ(P ) > µ∗ and Supp(P ) ⊂ [0, 1]}

and D(P, Q) is the relative entropy between measures Pi and P , which we
define in Chapter 14. The quantity di is never smaller than d(µi , µ∗ ). For details
on this, see the paper by Honda and Takemura [2010].
4 The approximation in Eq. (10.5) was used to show that the regret for KL-UCB
is closely related to the variance of the Bernoulli distribution. It is natural to
ask whether or not this result could be derived, at least asymptotically, by
 10.4 Bibliographic Remarks 140

appealing to the central limit theorem. The answer is no. First, the quality of
the approximation in Eq. (10.5) does not depend on n, so asymptotically it is
not true that the Bernoulli bandit behaves like a Gaussian bandit with variances
tuned to match. The reason is that as n tends to infinity, the confidence level
should be chosen so that the risk of failure also tends to zero. But the central
limit theorem does not provide information about the tails with probability
mass less than O(n−1/2 ). See Note 1 in Chapter 5.
5 The analysis in this chapter is easily generalised to a wide range of alternative
noise models. You will do this for single-parameter exponential families in
Exercises 10.4, 10.5 and 34.5.
6 Chernoff credits Lemma 10.3 to his friend Herman Rubin [Chernoff, 2014], but
the name seems to have stuck.

10.4 Bibliographic Remarks

Several authors have worked on Bernoulli bandits, and the asymptotics have
been well understood since the article by Lai and Robbins [1985]. The earliest
version of the algorithm presented in this chapter is due to Lai [1987], who
provided asymptotic analysis. The finite-time analysis of KL-UCB was given by
two groups simultaneously (and published in the same conference) by Garivier
and Cappé [2011] and Maillard et al. [2011] (see also the combined journal article:
Cappé et al. 2013). Two alternatives are the DMED [Honda and Takemura, 2010]
and IMED [Honda and Takemura, 2015] algorithms. These works go after the
problem of understanding the asymptotic regret for the more general situation
where the rewards lie in a bounded interval (see Note 3). The latter work
covers even the semi-bounded case where the rewards are almost surely upper-
bounded. Both algorithms are asymptotically optimal. Ménard and Garivier
[2017] combined MOSS and KL-UCB to derive an algorithm that is minimax
optimal and asymptotically optimal for single-parameter exponential families.
While the subgaussian and Bernoulli examples are very fundamental, there has
also been work on more generic set-ups where the unknown reward distribution for
each arm is known to lie in some class F. The article by Burnetas and Katehakis
[1996] gives the most generic (albeit, asymptotic) results. These generic set-ups
remain wide open for further work.

10.5 Exercises

10.1 (Pinsker’s inequality) Prove Lemma 10.2(b).

Hint Consider the function g(x) = d(p, p + x) − 2x2 over the [−p, 1 − p] interval.
By taking derivatives, show that g ≥ 0.
10.2 (Asymptotic optimality) Prove the asymptotic claim in Theorem 10.6.
 10.5 Exercises 141

Hint Choose ε1 , ε2 to decrease slowly with n and use the first part of the
theorem.

10.3 (Concentration for bounded random variables) Let F = (Ft )t be a

filtration, (Xt )t be [0, 1]-valued, F-adapted sequence, such that E [Xt | Ft−1 ] = µt
Pn
for some µ1 , . . . , µn ∈ [0, 1] non-random numbers. Define µ = n1 t=1 µt ,
P
µ̂ = n1 t=1 Xt . Prove that the conclusion of Lemma 10.3 still holds.
n

Hint Read Note 2 at the end of this chapter. Let g(·, µ) be the cumulant-
generating function of the µ-parameter Bernoulli distribution. For X ∼ B(µ),
λ ∈ R, g(λ, µ) = log E [exp(λX)]. Show that g(λ, ·) is concave. Next, use this and
the tower rule to show that E [exp(λn(µ̂ − µ))] ≤ g(λ, µ)n .

The bound of the previous exercise is most useful when all µt are either all
close to zero or they are all close to one. When half of the {µt } are close to
zero and the other half close to one, then the bound degrades to Hoeffding’s
bound.

10.4 (KL-UCB for exponential families) Let M = {Pθ : θ ∈ Θ} be a

regular non-singular exponential family with sufficient statistic S(x) = x and
E = {(Pθi )ki=1 : θ ∈ Θk } be the set of bandits with reward distributions in M.
Design a policy π such that for all ν ∈ E, it holds that

Rn (π, ν) X ∆i
lim ≤ ,
n→∞ log(n) di,inf
i:∆i >0

R
where µ(θ) = R xdPθ (x) is the mean of Pθ and di,inf = inf{d(θ, φ) : µ(φ) >
µ∗ , φ ∈ Θ}, with d(θ, φ) the relative entropy between Pθ and Pφ .

Hint Readers not familiar with exponential families should skip ahead to
Section 34.3.1 and then do Exercise 34.5. For the exercise, repeat the proof of
Theorem 10.6, adapting as necessary. See also the paper by Cappé et al. [2013].

10.5 (KL-UCB for non-canonical exponential families) Repeat the

previous exercise, but relax the assumption that S(x) = x.

Hint This is a subtle problem. You should adapt the algorithm so that if there
are ties in the upper confidence bounds, then an arm with the largest number of
plays is chosen. A solution is available. Korda et al. [2013] analysed
R Thompson
sampling in this setting. Their result only holds when θ 7→ R xpθ (x)dh(x) is
invertible, which does not always hold.
 10.5 Exercises 142

In the analysis of KL-UCB for canonical exponential families, the asymptotic

rate is a good indicator of the finite-time regret in the sense that the o(log(n))
term hidden by the asymptotics has roughly the same leading constant as
the dominant term. By contrast, the analysis here indicates that
log(n) 1
E[Ti (n)] ≈ + ,
di,inf di,min
where di,min = di,min (0). Although the latter term is negligible asymptotically,
it may be the dominant term for all reasonable n.

10.6 (Comparison to UCB) In this exercise, you compare KL-UCB and UCB
empirically.
(a) Implement Algorithm 8 and Algorithm 6, where the latter algorithm should
be tuned for 1/2-subgaussian bandits so that
s
log(f (t))
At = argmaxi∈[k] µ̂i (t − 1) + .
2Ti (t − 1)
(b) Let n = 10000 and k = 2. Plot the expected regret of each algorithm as a
function of ∆ when µ1 = 1/2 and µ2 = 1/2 + ∆.
(c) Repeat the above experiment with µ1 = 1/10 and µ1 = 9/10.
(d) Discuss your results.
Part III
Adversarial Bandits with
Finitely Many Arms
 144

Statistician George E. P. Box is famous for writing that ‘all models are wrong,
but some are useful’. In the stochastic bandit model the reward is sampled from
a distribution that depends only on the chosen action. It does not take much
thought to realise this model is almost always wrong. At the macroscopic level
typically considered in bandit problems, there is not much that is stochastic
about the world. And even if there were, it is hard to rule out the existence of
other factors influencing the rewards.
The quotation suggests we should not care whether or not the stochastic bandit
model is right, only whether it is useful. In science, models are used for predicting
the outcomes of future experiments, and their usefulness is measured by the
quality of the predictions. But how can this be applied to bandit problems? What
predictions can be made based on bandit models? In this respect, we postulate
the following:

The point of bandit models is to facilitate predicting the performance of

bandit algorithms on future problem instances that one encounters in their
practice.

A model can fail in two fundamentally different ways. It can be too specific,
imposing assumptions so detached from reality that a catastrophic mismatch
between actual and predicted performance may arise. The second mode of failure
occurs when a model is too general, which makes the algorithms designed to do
well on the bandit model overly cautious, which can harm performance.
Not all assumptions are equally important. It is a critical assumption in
stochastic bandits that the mean reward of individual arms does not change
(significantly) over time. On the other hand, the assumption that a single, arm-
dependent distribution generates the rewards for a given arm plays a relatively
insignificant role. The reader is encouraged to think of cases when the constancy
of arm distributions plays no role, and also of cases when it does – furthermore, to
decide to what extent the algorithms can tolerate deviations from the assumption
that the means of arms stay the same. Stochastic bandits where the means of
the arms are changing over time are called non-stationary and are the topic of
Chapter 31.
If a highly specialised model is actually correct, then the resulting algorithms
usually dominate algorithms derived for a more general model. This is a general
manifestation of the bias-variance trade-off, well known in supervised learning
and statistics. The holy grail is to find algorithms that work ‘optimally’ across
a range of models. The reader should think about examples from the previous
chapters that illustrate these points.
The usefulness of the stochastic model depends on the setting. In particular,
the designer of the bandit algorithm must carefully evaluate whether stochasticity,
stability of the mean and independence are reasonable assumptions. For some
applications, the answer will probably be yes, while in others the practitioner
 145

may seek something more robust. This latter situation is the topic of the next
few chapters.

Adversarial Bandits

The adversarial bandit model abandons almost all the assumptions on how
the rewards are generated, so much so that the environment is often called the
adversary. The adversary has a great deal of power in this model, including the
ability to examine the code of the proposed algorithms and choose the rewards
accordingly. All that is kept from the previous chapters is that the objective will
be framed in terms of how well a policy is able to compete with the best action
in hindsight.
At first sight, it seems remarkable that one can say anything at all about such
a general model. And yet it turns out that this model is not much harder than
the stochastic bandit problem. Why this holds and how to design algorithms that
achieve these guarantees will be explained in the following chapters.
To give you a glimmer of hope, imagine playing the following simple bandit
game with a friend. The horizon is n = 1, and you have two actions. The game
proceeds as follows:
1 You tell your friend your strategy for choosing an action.
2 Your friend secretly chooses rewards x1 ∈ {0, 1} and x2 ∈ {0, 1}.
3 You implement your strategy to select A ∈ {1, 2} and receive reward xA .
4 The regret is R = max{x1 , x2 } − xA .
Clearly, if your friend chooses x1 = x2 , then your regret is zero no matter what.
Now let’s suppose you implement the deterministic strategy A = 1. Then your
friend can choose x1 = 0 and x2 = 1, and your regret is R = 1. The trick to
improve on this is to randomise. If you tell your friend, ‘I will choose A = 1 with
probability one half’, then the best she can do is choose x1 = 1 and x2 = 0 (or
reversed), and your expected regret is R = 1/2. You are forgiven if you did not
settle on this solution yourself because we did not tell you that a strategy may
be randomised. With such a short horizon, you cannot do better than this, but
for longer games the relative advantage of the adversary decreases, as we shall
see soon.
In the next two chapters, we investigate the k-armed adversarial model in detail,
providing both algorithms and regret analysis. Like the stochastic model, the
adversarial model has many generalisations, which we’ll visit in future chapters.

Bibliographic Remarks

The quote by George Box was used several times with different phrasings [Box,
1976, 1979]. The adversarial framework has its roots in game theory, with familiar
 146

names like Hannan [1957] and Blackwell [1954] producing some of the early
work. The non-statistical approach has enjoyed enormous popularity since the
1990’s and has been adopted wholeheartedly by the theoretical computer science
community [Vovk, 1990, Littlestone and Warmuth, 1994, and many, many others].
The earliest work on adversarial bandits is by Auer et al. [1995]. There is now a
big literature on adversarial bandits, which we will cover in more depth in the
chapters that follow. There has been a lot of effort to move away from stochastic
assumptions. An important aspect of this is to define a sense of regularity for
individual sequences. We refer the reader to some of the classic papers by Martin-
Löf [1966] and Levin [1973] and the more recent paper by Ivanenko and Labkovsky
[2013].
11 The Exp3 Algorithm

In this chapter we first introduce the formal model of adversarial bandit

environments and discuss the relationship to the stochastic bandit model. This is
followed by the discussion of importance-weighted estimation, the Exp3 algorithm
that uses this technique and the analysis of the regret of Exp3.

11.1 Adversarial Bandit Environments

Let k > 1 be the number of arms. A k-armed

adversarial bandit is an arbitrary sequence of
reward vectors (xt )nt=1 , where xt ∈ [0, 1]k . In each
round, the learner chooses a distribution over the
actions Pt ∈ Pk−1 . Then the action At ∈ [k] is
sampled from Pt , and the learner receives reward
xtAt . The interaction protocol is summarised in
Fig. 11.2.
A policy in this setting is a function π : ([k] ×
[0, 1])∗ → Pk−1 mapping history sequences to dis-
tributions over actions (regardless of measurability). Figure 11.1 Would you play
The performance of a policy π in environment x is with this multi-armed bandit?
measured by the expected regret, which is the expected loss in revenue of the
policy relative to the best fixed action in hindsight.
n
" n #
X X
Rn (π, x) = max xti − E xtAt , (11.1)
i∈[k]
t=1 t=1

Adversary secretly chooses rewards (xt )n

t=1 with xt ∈ [0, 1]
k

For rounds t = 1, 2, . . . , n:
Learner selects distribution Pt ∈ Pk−1 and samples At from Pt .
Learner observes reward Xt = xtAt .

Figure 11.2 Interaction protocol for k-armed adversarial bandits.

 11.1 Adversarial Bandit Environments 148

where the expectation is over the randomness of the learner’s actions. The
arguments π and x are omitted from the regret when they are clear from context.

The only source of randomness in the regret comes from the randomness in
the actions of the learner. Of course the interaction with the environment
means the action chosen in round t may depend on actions s < t as well as
the observed rewards until round t. As we noted, unlike the case of stochastic
bandits, here, there is no measurability restriction on the learner’s policy π.
This is actually by choice, see Note 12 for details.

The worst-case regret over all environments is

Rn∗ (π) = sup Rn (π, x) .
x∈[0,1]n×k

The main question is whether or not there exist policies π for which Rn∗ (π) is
sublinear in n. In Exercise 11.2 you will show that for deterministic policies
Rn∗ (π) ≥ n(1 − 1/k), which follows by constructing a bandit so that xtAt = 0 for
all t and xti = 1 for i 6= At . Because of this, sublinear worst-case regret is only
possible by using a randomised policy.

Readers familiar with game theory will not be surprised by the need for
randomisation. The interaction between learner and adversarial bandit can be
framed as a two-player zero-sum game between the learner and environment.
The moves for the environment are the possible reward sequences, and for
the player they are the policies. The pay-off for the environment/learner is
the regret and its negation respectively. Since the player goes first, the only
way to avoid being exploited is to choose a randomised policy.

While stochastic and adversarial bandits seem quite different, it turns out that the
optimal worst-case regret is the same up to constant factors and that lower bounds
for adversarial bandits are invariably derived in the same manner as for stochastic
bandits (see Part IV). In this chapter, we present a simple algorithm for which
the worst-case regret is suboptimal by just a logarithmic factor. First, however,
we explore the differences and similarities between stochastic and adversarial
environments.
We already noted that deterministic strategies will have linear regret for
some adversarial bandit. Since strategies in Part II like UCB and ‘Explore-then-
Commit’ were deterministic, they are not well suited for the adversarial setting.
This immediately implies that policies that are good for stochastic bandit can
be very suboptimal in the adversarial setting. What about the other direction?
Will an adversarial bandit strategy have small expected regret in the stochastic
setting? Let π be an adversarial bandit policy and ν = (ν1 , . . . , νk ) be a stochastic
bandit with Supp(νi ) ⊆ [0, 1] for all i. Next, let Xti be sampled from νi for each
 11.2 Importance-Weighted Estimators 149

i ∈ [k] and t ∈ [n], and assume these random variables are mutually independent.
By Jensen’s inequality and convexity of the maximum function, we have
" n #
X
Rn (π, ν) = max E (Xti − XtAt )
i∈[k]
t=1
" n
#
X
≤ E max (Xti − XtAt )
i∈[k]
t=1

= E [Rn (π, X)] ≤ Rn∗ (π) , (11.2)

where the regret in the first line is the stochastic regret (using the random table
model), and in the last it is the adversarial regret. Therefore the worst-case
stochastic regret is upper-bounded by the worst-case adversarial regret. Going
the other way, the above inequality also implies that the worst-case regret for
adversarial problems is lower-bounded by the worst-case regret on stochastic
problems with rewards bounded in [0, 1]. In Chapter 15, we √ prove that the worst-
case regret for stochastic Bernoulli bandits is at least c nk, where c > 0 is a
universal constant (Exercise 15.4). And so for the same universal constant, the
minimax regret for adversarial bandits satisfies
√
Rn∗ = inf sup Rn (π, x) ≥ c nk .
π x∈[0,1]n×k

There is a little subtlety here. In order to define the expectations in the stochastic
regret, the policy should be appropriately measurable. This can be resolved by
noting that lower bounds can be proven using Bernoulli bandits. For details, see
again Note 12.

11.2 Importance-Weighted Estimators

A key ingredient of all adversarial bandit algorithms is a mechanism for estimating

the reward of unplayed arms. Recall that Pt is the conditional distribution of the
action played in round t, and so for i ∈ [k], Pti is the conditional probability

Pti = P (At = i | A1 , X1 , . . . , At−1 , Xt−1 ) .

In what follows, we assume that for all t and i, Pti > 0 almost surely. As we
shall see later, this will be true for all policies considered in this chapter. The
importance-weighted estimator of xti is
I {At = i} Xt
X̂ti = . (11.3)
Pti
Let Et [·] = E[· | A1 , X1 , . . . , At , Xt ] denote the conditional expectation given the
history up to time t. The conditional mean of X̂ti satisfies

Et−1 [X̂ti ] = xti , (11.4)

 11.2 Importance-Weighted Estimators 150

which means that X̂ti is an unbiased estimate of xti conditioned on the history
observed after t − 1 rounds. To see why Eq. (11.4) holds, let Ati = I {At = i} so
that Xt Ati = xti Ati and
Ati
X̂ti = xti .
Pti
Now, Et−1 [Ati ] = Pti , and since Pti is σ(A1 , X1 , . . . , At−1 , Xt−1 )-measurable,
 
Ati xti xti
Et−1 [X̂ti ] = Et−1 xti = Et−1 [Ati ] = Pti = xti .
Pti Pti Pti
Being unbiased is a good start, but the variance of an estimator is also important.
For arbitrary random variable U , the conditional variance Vt−1 [U ] is the random
variable
 
Vt−1 [U ] = Et−1 (U − Et−1 [U ])2 .

So Vt−1 [X̂ti ] is a random variable that measures the variance of X̂ti conditioned
on the past. Calculating the conditional variance using the definition of X̂ti and
Eq. (11.4) shows that
 
2 Ati x2ti x2 (1 − Pti )
Vt−1 [X̂ti ] = Et−1 [X̂ti ] − x2ti = Et−1 2 − x2ti = ti . (11.5)
Pti Pti
This can be extremely large when Pti is small and xti is bounded away from zero.
In the notes and exercises, we shall see to what extent this can cause trouble.
The estimator in (11.3) is the first that comes to mind, but there are alternatives.
For example,
I {At = i}
X̂ti = 1 − (1 − Xt ) . (11.6)
Pti
This estimator is still unbiased. Rewriting the formula in terms of yti = 1 − xti
and Yt = 1 − Xt and Ŷti = 1 − X̂ti leads to
I {At = i}
Ŷti = Yt .
Pti
This is the same as (11.3) except that Yt has replaced Xt . The terms yti , Yt and
Ŷti should be interpreted as losses. Had we started with losses to begin with, then
this would have been the estimator that first came to mind. For obvious reasons,
the estimator in Eq. (11.6) is called the loss-based importance-weighted
estimator. The conditional variance of this estimator is essentially the same as
Eq. (11.5):
2 1 − Pti
Vt [X̂ti ] = Vt [Ŷti ] = yti .
Pti
2
The only difference is that the variance now depends on yti rather than x2ti . Which
is better depends on the rewards for arm i, with smaller rewards suggesting the
superiority of the first estimator and larger rewards (or small losses) suggesting
the superiority of the second estimator. Can we change the estimator (either one
 11.3 The Exp3 Algorithm 151

of them) so that it is more accurate for actions whose reward is close to some
specific value v? Of course! Just change the estimator so that v is subtracted
from the observed reward (or loss), then use the importance-sampling formula,
and subsequently add back v. The problem is that the optimal value of v depends
on the unknown quantity being estimated. Also note that the dependence of the
variance on Pti is the same for both estimators, and since the rewards are bounded,
it is this term that usually contributes most significantly. In Exercise 11.5, we ask
you to show that all unbiased estimators in this setting are importance-weighted
estimators.

Although the two estimators seem quite similar, it should be noted that the
first estimator takes values in [0, ∞) while the second takes values in (−∞, 1].
Soon we will see that this difference has a big impact on the usefulness of
these estimators when used in the Exp3 algorithm.

11.3 The Exp3 Algorithm

The simplest algorithm for adversarial bandits is called Exp3, which stands
for ‘exponential-weight algorithm for exploration and exploitation’. The reason
for this name will become clear after the explanation of the algorithm. Let
Pt
Ŝti = s=1 X̂si be the total estimated reward by the end of round t, where X̂si is
given in Eq. (11.6). It seems natural to play actions with larger estimated reward
with higher probability. While there are many ways to map Ŝti into probabilities,
a simple and popular choice is called exponential weighting, which for tuning
parameter η > 0 sets

exp(η Ŝt−1,i )
Pti = Pk . (11.7)
j=1 exp(η Ŝt−1,j )

The parameter η is called the learning rate. When the learning rate is large, Pt
concentrates about the arm with the largest estimated reward and the resulting
algorithm exploits aggressively. For small learning rates, Pt is more uniform,
and the algorithm explores more frequently. Note that as Pt concentrates, the
variance of the importance-weighted estimators for poorly performing arms
increases dramatically. There are many ways to tune the learning rate, including
allowing it to vary with time. In this chapter we restrict our attention to the
simplest case by choosing η to depend only on the number of actions k and the
horizon n. Since the algorithm depends on η, this means that the horizon must
be known in advance, a requirement that can be relaxed (see Note 10).
 11.4 Regret Analysis 152

1: Input: n, k, η
2: Set Ŝ0i = 0 for all i
3: for t = 1, . . . , n do
4: Calculate the sampling distribution Pt :
 
exp η Ŝt−1,i
Pti = P  
exp
k
j=1 η Ŝt−1,j

5: Sample At ∼ Pt and observe reward Xt

6: Calculate Ŝti :
I {At = i} (1 − Xt )
Ŝti = Ŝt−1,i + 1 −
Pti
7: end for
Algorithm 9: Exp3.

11.4 Regret Analysis

We are now ready to bound the expected regret of Exp3.

Theorem 11.1. Letp x ∈ [0, 1]
n×k
and π be the policy of Exp3 (Algorithm 9) with
learning rate η = log(k)/(nk). Then,
p
Rn (π, x) ≤ 2 nk log(k) .
As we will prove many variants of this result with various tools, here we give a
short algebraic proof, saving the development of intuition for later.
Proof For any arm i, define
n
" n #
X X
Rni = xti − E Xt ,
t=1 t=1

which is the expected regret relative to using action i in all the rounds. The
result will follow by bounding Rni for all i, including the optimal arm. For the
remainder of the proof, let i be some fixed arm. By the unbiasedness property of
the importance-weighted estimator X̂ti ,
n
X k
X k
X
E[Ŝni ] = xti and also Et−1 [Xt ] = Pti xti = Pti Et−1 [X̂ti ] .
t=1 i=1 i=1
(11.8)
The tower rule says that for any random variable X, E[Et−1 [X]] = E[X], which
together with the linearity of expectation and Eq. (11.8) means that
" n k #
h i XX h i
Rni = E Ŝni − E Pti X̂ti = E Ŝni − Ŝn , (11.9)
t=1 i=1
 11.4 Regret Analysis 153

P P
where the last equality serves as the definition of Ŝn = t i Pti X̂ti . To bound
the right-hand side of Eq. (11.9), let
k
X  
Wt = exp η Ŝtj .
j=1

By convention an empty sum is zero, which means that Ŝ0j = 0 and W0 = k.

Then,
k
X Yn
W1 Wn Wt
exp(η Ŝni ) ≤ exp(η Ŝnj ) = Wn = W0 ... =k . (11.10)
j=1
W0 Wn−1 t=1
Wt−1

The ratio in the product can be rewritten in terms of Pt by

Wt X exp(η Ŝt−1,j )
k X k
= exp(η X̂tj ) = Ptj exp(η X̂tj ) . (11.11)
Wt−1 j=1
Wt−1 j=1

We need the following facts:

exp(x) ≤ 1 + x + x2 for all x ≤ 1 and 1 + x ≤ exp(x) for all x ∈ R .

Using these two inequalities leads to

Xk Xk
Wt 2 2
≤1+η Ptj X̂tj + η Ptj X̂tj
Wt−1 j=1 j=1
 
Xk Xk
≤ exp η Ptj X̂tj + η 2 2
Ptj X̂tj . (11.12)
j=1 j=1

Notice that this was only possible because X̂tj is defined by Eq. (11.6), which
ensures that X̂tj ≤ 1 and would not have been true had we used Eq. (11.3).
Combining Eq. (11.12) and Eq. (11.10),
 
  n X
X k
exp η Ŝni ≤ k exp η Ŝn + η 2 2
Ptj X̂tj .
t=1 j=1

Taking the logarithm of both sides, dividing by η > 0 and reordering gives

log(k) Xn X k
2
Ŝni − Ŝn ≤ +η Ptj X̂tj . (11.13)
η t=1 j=1

As noted earlier, the expectation of the left-hand side is Rni . The first term on
the right-hand side is a constant, which leaves us to bound the expectation of the
second term. Letting ytj = 1 − xtj and Yt = 1 − Xt and expanding the definition
 11.4 Regret Analysis 154

2
of X̂tj leads to
   
 2
=
Xk X k
2 I {A j} y
= E Ptj 1 −
t tj
E Ptj X̂tj 
j=1 j=1
Ptj
 !
2
X k
I {A = j} y I {At = j} ytj
= E Ptj 1 − 2 +
t tj 
j=1
Ptj Ptj2
  
2
X k
I {At = j} ytj
= E 1 − 2Yt + Et−1  
j=1
Ptj
 
Xk
2 
= E 1 − 2Yt + ytj
j=1
 
X
= E (1 − Yt )2 + 2 
ytj
j6=At

≤ k.

Summing over t, and then substituting into Eq. (11.13), we get

log(k) p
Rni ≤ + ηnk = 2 nk log(k) ,
η
p
where the equality follows by substituting η = log(k)/(nk), which was chosen
to optimise this bound.

At the heart of the proof are the inequalities:

1 + x ≤ exp(x) for all x ∈ R and exp(x) ≤ 1 + x + x2 for x ≤ 1 .

The former of these inequalities is an ansatz derived from the first-order Taylor
expansion of exp(x) about x = 0. The latter, however, is not the second-order
Taylor expansion, which would be 1 + x + x2 /2. The problem is that the second-
order Taylor series is not an upper bound on exp(x) for x ≤ 1, but only for
x ≤ 0:
1
exp(x) ≤ 1 + x + x2 for all x ≤ 0 . (11.14)
2
But it is nearly an upper bound, and this can be exploited to improve the bound
in Theorem 11.1. The mentioned upper and lower bounds on exp(x) are shown
in Fig. 11.3, from which it is quite obvious that the bound in Eq. (11.14) is
significantly tighter when x ≤ 0.
Let us now put Eq. (11.14) to use in proving the following√improved version of
Theorem 11.1, for which the regret is smaller by a factor of 2. The algorithm is
unchanged except for a slightly increased learning rate.
 11.4 Regret Analysis 155

exp(x) − (1 + x)
exp(x) − (1 + x + x2 )
0.1
exp(x) − (1 + x + x2 /2)

−0.1

−0.5 0 0.5
x

Figure 11.3 Approximations for exp(x) on [−1/2, 1/2].

Theorem 11.2. Let x ∈ [0, 1]p

n×k
be an adversarial bandit and π be the policy of
Exp3 with learning rate η = 2 log(k)/(nk). Then,
p
Rn (π, x) ≤ 2nk log(k) .

Proof By construction, X̂tj ≤ 1. Therefore,

   
exp η X̂tj = exp(η) exp η(X̂tj − 1)
 
η2
≤ exp(η) 1 + η(X̂tj − 1) + (X̂tj − 1)2 .
2
P
Using the fact that j Ptj = 1 and the inequality 1 + x ≤ exp(x), we get
 
Xk Xk 2 Xk
Wt η
= Ptj exp(η X̂tj ) ≤ exp η Ptj X̂tj + Ptj (X̂tj − 1)2  ,
Wt−1 j=1 j=1
2 j=1

where the equality is from Eq. (11.11). We see that here we need to bound
P 2
j Ptj (X̂tj − 1) . Let Ŷtj = 1 − X̂tj . Then,

Ptj (X̂tj − 1)2 = Ptj Ŷtj Ŷtj = I {At = j} ytj Ŷtj ≤ Ŷtj ,

where the last inequality used Ŷtj ≥ 0 and ytj ≤ 1. Thus,

k
X k
X
Ptj (X̂tj − 1)2 ≤ Ŷtj .
j=1 j=1

With the same calculations as before, we get

log(k) η X X
n k
Ŝni − Ŝn ≤ + Ŷtj . (11.15)
η 2 t=1 j=1
 11.5 Notes 156

P
The result is completed by taking expectations of both sides, using E t,j Ŷtj =
P P
E t,j Et−1 Ŷtj = E t,j ytj ≤ nk and substituting the learning rate.

The reader may wonder about the somewhat ad hoc proof. The best we
can do for now is to point out a few things about the proof. It is natural to
replace the true rewards with the estimated ones. Then, to prove a regret
bound in terms of the estimated rewards, an alternative to the proof is
to start with the the trivial inequality that states that for any x = (xj )
P
vector and positive quantity η, the inequality xi ≤ η1 log j exp(ηxj ) holds.
Applying this with x = (Ŝni ) gives
1 X 1
Ŝni ≤ log( exp(η Ŝnj )) = log(Wn ) ,
η j
η

from where the proof can be continued by introducing the telescoping

argument.

11.5 Notes

1 Exp3 is nearly optimal in the sense that its expected regret cannot be improved
significantly in the worst case. The distribution of its regret, however, is very far
from optimal. Define the random regret to be the random variable measuring
the actual deficit of the learner relative to the best arm in hindsight:
n
X n
X n
X n
X
R̂n = max xti − Xt = Yt − min yti .
i∈[k] i∈[k]
t=1 t=1 t=1 t=1
| {z } | {z }
in terms of rewards in terms of losses

In Exercise 11.6 you will show that for all large enough n and reasonable
choices of η, there exists a bandit such that the random regret of Exp3 satisfies
P(R̂n ≥ n/4) > 1/131. In the same exercise, you should explain why this does
not contradict the upper bound. That Exp3 has such a high variance is a
serious limitation, which we address in the next chapter.
2 What happens when the range of the rewards is unbounded? This has been
studied by Allenberg et al. [2006], where some (necessarily much weaker)
positive results are presented.
3 In the full information setting, the learner observes the whole vector
xt ∈ [0, 1]k at the end of round t, but the reward is still xtAt . This setting is
also called prediction with expert advice. Exponential weighting is still
a good idea, but the estimated rewards can now be replaced by the actual
rewards. The resulting algorithm is sometimes called Hedge or the exponential
weights algorithm. The proof as written goes through in almost the same way,
 11.5 Notes 157

but one should replace the polynomial upper

p bound on exp(x) with Hoeffding’s
lemma. This analysis gives a regret of n log(k)/2, which is optimal in an
asymptotic sense [Cesa-Bianchi and Lugosi, 2006].
4 We assumed that the adversary chooses the rewards at the start of the game.
Such adversaries are called oblivious. An adversary is called reactive or
non-oblivious if xt is allowed to depend on the history x1 , A1 , . . . , xt−1 , At−1 .
Despite the fact that this is clearly a harder problem, the result we obtained
can be generalised to this setting without changing the analysis. It is another
question whether the definition of regret makes sense for reactive environments.
p
5 A more sophisticated algorithm and analysis shaves a factor of log(k) from
the regret upper bound in Theorem 11.2 [Audibert and Bubeck, 2009, 2010a,
Bubeck and Cesa-Bianchi, 2012]. It turns out that this algorithm, just like
Exp3, is an instantiation of mirror descent from convex optimisation, which
we present in Chapter 28. More details are in p Exercise 28.15. Interestingly,
this algorithm not only shaves off the extra log(k) factor from the regret,
but also achieves O(log(n))-regret
√ in the stochastic setting provided that one
uses a learning rate of 1/ t in round t [Zimmert and Seldin, 2019]. This
remarkable result improves in an elegant way on many previous attempts to
design algorithms for stochastic and adversarial bandits [Bubeck and Slivkins,
2012, Seldin and Slivkins, 2014, Auer and Chiang, 2016, Seldin and Lugosi,
2017]. There are some complications, however, depending on whether or not the
adversary is oblivious. The situation is best summarised by Auer and Chiang
[2016], where the authors present upper and lower bounds on what is possible
in various scenarios.
6 The initial distribution (the ‘prior’) P1 does not have to be uniform. By biasing
the prior towards a specific action, the regret can be reduced when the favoured
action turns out to be optimal. There is an unavoidable price for this, however,
if the optimal arm is not favoured [Lattimore, 2015a].
7 Building on the previous note, suppose the reward in round t is Xt =
ft (A1 , . . . , At ) and f1 , . . . , fn are a sequence of functions chosen in advance by
the adversary with ft : [k]t → [0, 1]. Let Π ⊂ [k]n be a set of action sequences.
Then the expected policy regret with respect to Π is
n
" n #
X X
max ft (a1 , . . . , at ) − E ft (A1 , . . . , At ) .
a1 ,...,an ∈Π
t=1 t=1

Even if Π only consists of constant sequences, there still does not exist a policy
guaranteeing sublinear regret. The reason is simple. Consider the two candidate
choices of f1 , . . . , fn . In the first choice, ft (a1 , . . . , at ) = I {a1 = 1}, and in
the second we have ft (a1 , . . . , at ) = I {a1 = 2}. Clearly the learner must suffer
linear regret in at least one of these two reactive bandit environments. The
problem is that the learner’s decision in the first round determines the rewards
available in all subsequent rounds, and there is no time for learning. By making
 11.5 Notes 158

additional assumptions, sublinear regret is possible, however – e.g. by assuming

the adversary has limited memory [Arora et al., 2012].
8 There is a common misconception that the adversarial framework is a good fit
for non-stationary environments. While the framework does not assume the
rewards are stationary, the regret concept used in this chapter has stationarity
built in. A policy designed for minimising the regret relative to the best action
in hindsight is seldom suitable for non-stationary bandits, where the whole point
is to adapt to changes in the optimal arm. In such cases a better benchmark is
to compete with a sequence of actions. For more on non-stationary bandits,
see Chapter 31.
9 The estimators in Eq. (11.3) and Eq. (11.6) both have conditional variance
Vt [X̂ti ] ≈ 1/Pti , which blows up for small Pti . It is instructive to think about
whether and how Pti can take on very small values. Consider the loss-based
estimator given by (11.6). For this estimator, when PtAt and Xt are both
small, X̂tAt can take on a large negative value. Through the update formula
(11.7), this then translates into Pt+1,At being squashed aggressively towards
zero. A similar issue arises with the reward-based estimator given by (11.3).
The difference is that now it will be a ‘positive surprise’ (PtAt small, Xt
large) that pushes the probabilities towards zero. But note that in this case,
Pt+1,i is pushed towards zero for all i 6= At . This means that dangerously
small probabilities are expected to be more frequent for the gains estimator
Eq. (11.3).
10 Exp3 requires advance knowledge of the horizon. The doubling trick can be
used to overcome this issue, but a more elegant solution is to use a decreasing
learning rate. The analysis in this chapter can be adapted to this case. More
discussion is provided in the notes and exercises of Chapter 28, where we give
a more generic solution to this problem (Exercise 28.13).
11 The calculation in Eq. (11.2) is a reduction, showing that algorithms with low
regret on finite-armed adversarial bandits also have low regret on stochastic
bandits where the reward distributions have appropriately bounded support.
Reductions play an important role throughout the bandit literature and we will
see many more examples. The reader should be careful not to generalise the idea
that adversarial algorithms work well on stochastic problems. The assumptions
must be checked (like boundedness of the support), and for different models
there can be subtleties. The whole of Chapter 29 is devoted to the linear case.
12 As we mentioned, a policy for k-armed adversarial bandits is defined by any
function π : ([k] × [0, 1])∗ → Pk−1 . There is no need to assume that π is
measurable because the actions are discrete and the rewards are deterministic.
The relations between the stochastic and adversarial regret are only well defined
for policies that are probability kernels as defined in Definition 4.7. You might
be worried that lower bounds for stochastic bandits only imply lower bounds for
measurable adversarial policies. Fortunately, the lower bounds are easily proven
for Bernoulli bandits, and in this case the space of reward sequences is finite
and measurability is no longer problematic. Later we study adversarial bandits
 11.6 Bibliographic Remarks 159

with an infinite action set A, which is equipped with a σ-algebra G. In this

case the reward vectors are replaced by functions (xt )nt=1 , where xt : A → [0, 1]
is G-measurable. Then, the measurability condition on the policy is that for all
choices of the adversary and all B ∈ B(A),
π(B | a1 , x1 (a1 ), . . . , at−1 , xt−1 (at−1 ))
must be measurable as a function of a1 , . . . , at−1 . In practice, of course, all the
policies you might ever propose would also be measurable as a function of the
rewards.

11.6 Bibliographic Remarks

Exponential weighting has been a standard tool in online learning since the
papers by Vovk [1990] and Littlestone and Warmuth [1994]. Exp3 and several
variations were introduced by Auer et al. [1995], which was also the first paper to
study bandits in the adversarial framework. The algorithm and analysis presented
here differs slightly because we do not add any additional exploration, while the
version of Exp3 in that paper explores uniformly with low probability. The fact
that additional exploration is not required was observed by Stoltz [2005].

11.7 Exercises

11.1 (Sampling from a multinomial) In order to implement Exp3, you need

a way to sample from the exponential weights distribution. Many programming
languages provide a standard way to do this. For example, in Python you can use
the Numpy library and numpy.random.multinomial. In more basic languages,
however, you only have access to a function rand() that returns a floating point
number ‘uniformly’ distributed in [0, 1]. Describe an algorithm that takes as input
a probability vector p ∈ Pk−1 and uses a single call to rand() to return X ∈ [k]
with P (X = i) = pi .

On most computers, rand() will return a pseudo-random number, and since

there are only finitely many floating point numbers, the resulting distribution
will not really be uniform on [0, 1]. Thinking about these issues is a worthy
endeavour, and sometimes it really matters. For this exercise you may ignore
these issues, however.

11.2 (Linear regret for deterministic policies) Show that for any
deterministic policy π there exists an environment x ∈ [0, 1]n×k such that
Rn (π, x) ≥ n(1 − 1/k). What does your result say about the policies designed in
Part II?
 11.7 Exercises 160

11.3 (Maximum and expectations) Show that the first inequality in (11.2)
holds: Moving the maximum inside the expectation increases the value of the
expectation.

11.4 (Alternative regret definition) Suppose we had defined the regret by

" n n
#
X X
Rntrack (π, x) = E max xti − xtAt .
i∈[k]
t=1 t=1

At first sight this definition seems like the right thing because it measures what
you actually care about. Unfortunately, however, it gives the adversary too much
power. Show that for any policy π (randomised or not), there exists a x ∈ [0, 1]n×k
such that
 
track 1
Rn (π, x) ≥ n 1 − .
k
11.5 (Unbiased estimators are importance weighted) Let P ∈ Pk−1
be a probability vector with nonzero components and let A ∼ P . Suppose
X̂ : [k] × R → R is a function such that for all x ∈ Rk ,
k
X
E[X̂(A, xA )] = Pi X̂(i, xi ) = x1 .
i=1

Show that there exists an a ∈ Rk such that ha, P i = 0 and for all i and z in their
I {i = 1} z
respective domains, X̂(i, z) = ai + .
P1
11.6 (Variance of Exp3) In this exercise, you will show that if η ∈ [n−p , 1]
for some p ∈ (0, 1), then for sufficiently large n, there exists a bandit on which
Exp3 has a constant probability of suffering linear regret. We work with losses so
that given a bandit y ∈ [0, 1]n×k , the learner samples At from Pt given by
 P 
exp −η s=1 Ŷsi
t−1

Pti = P  P ,
j=1 exp −η
k t−1
s=1 Ŷsj

where Ŷti = Ati yti /Pti . Let α ∈ [1/4, 1/2] be a constant to be tuned subsequently
and define a two-armed adversarial bandit in terms of its losses by
( (
0 if t ≤ n/2 α if t ≤ n/2
yt1 = and yt2 =
1 otherwise 0 otherwise .
For simplicity you may assume that n is even.

(a) Define the sequence of real-valued functions q1 , . . . , qn on domain [1/4, 1/2]

inductively by q0 (α) = 1/2 and
qs (α) exp(−ηα/qs (α))
qs+1 (α) = .
1 − qs (α) + qs (α) exp(−ηα/qs (α))
 11.7 Exercises 161

4,000

3,000

2,000
Regret

1,000

−1,000
8

85

0.3
0.2

0.2

9
0.2

0.2
α

Figure 11.4 Exp3 instability: Box and whisker plot of the distribution of the regret of
Exp3 for different values of α over a horizon of n = 104 with m = 500 repetitions for the
example of Exercise 11.6. The boxes represent the quartiles of the empirical distribution,
the diamond shows the average; the median is equal to the upper quartile (and thus
cannot be seen), while the dots show values outside of the “interquartile range”.

Pt
Show for t ≤ 1 + n/2 that Pt2 = qT2 (t−1) (α), where T2 (t) = s=1 As2 .
(b) Show that for sufficiently large n there exists an α ∈ [1/4, 1/2] and s ∈ N
such that

1 1
s−1
X n
qs (α) = and ≤ .
8n u=0
qu (α) 8

(c) Prove that P(T2 (n/2) ≥ s + 1) ≥ 1/65.

(d) Prove that P(R̂n ≥ n/4) ≥ (1 − n exp(−ηn)/2)/65.
(e) The previous part shows that the regret is linear with constant probability
for sufficiently large n. On the other hand, a dubious application of Markov’s
inequality and Theorem 11.1 shows that

4E[R̂n ]
P(R̂n ≥ n/4) ≤ = O(n−1/2 ) .
n
Explain the apparent contradiction.
(f) Validate the theoretical results of this exercise in an experimental fashion:
Implement Exp3 with the loss sequence suggested to reproduce Fig. 11.4.
The
p learning rate is set to the value computed in Theorem 11.2: η =
2 log(k)/(nk). Compare the figure with the theoretical results: Is there an
agreement between theory and the empirical results?
 11.7 Exercises 162

11.7 (Gumbel trick) Let a1 , . . . , ak be positive real values and U1 , . . . , Uk be

a sequence of independent and identically distributed uniform random variables
on [0, 1]. Then let Gi = − log(− log(Ui )), which follows a standard Gumbel
distribution. Prove that
 
ai
P log(ai ) + Gi = max(log(aj ) + Gj ) = Pk .
j=1 aj
j∈[k]

11.8 (Exp3 as follow-the-perturbed-leader) Let (Zti )ti be a collection

of independent and identically distributed random variables. The follow-the-
perturbed-leader algorithm chooses
t−1
!
X
At = argmaxi∈[k] Zti − η Ŷsi .
s=1

Show that if Zti is a standard Gumbel, then follow-the-perturbed-leader is the

same as Exp3.

11.9 (Exp3 on stochastic bandits) In this exercise we compare UCB and

Exp3 on stochastic data. Suppose we have a two-armed stochastic Bernoulli
bandit with µ1 = 0.5 and µ2 = µ1 + ∆ with ∆ = 0.05.
(a) Plot the regret of UCB and Exp3 on the same plot as a function of the
horizon n using the learning rate from Theorem 11.2.
(b) Now fix the horizon to n = 105 and plot the regret as a function of the
learning rate. Your plot should look like Fig. 11.5.
(c) Investigate how the shape of this graph changes as you change ∆.
(d) Find empirically the choice of η that minimises the worst-case regret over all
reasonable choices of ∆, and compare to the value proposed by the theory.
(e) What can you conclude from all this? Tell an interesting story.

Hint The performance of UCB depends greatly on which version you use. For
best results, remember that Bernoulli distributions are 1/2-subgaussian or use
the KL-UCB algorithm from Chapter 10.
 11.7 Exercises 163

Exp3
250

200
Expected regret

150

100

0 0.1
η

Figure 11.5 Expected regret for Exp3 for different learning rates over n = 105 rounds
on a Bernoulli bandit with means µ1 = 0.5 and µ2 = 0.55.
12 The Exp3-IX Algorithm

In the last chapter, we proved a sublinear bound on the expected regret of Exp3,
but with a dishearteningly large variance. The objective of this chapter is to
modify Exp3 so that the regret stays small in expectation and is simultaneously
well concentrated about its mean. Such results are called high-probability
bounds. By slightly modifying the algorithm, we show that for each δ ∈ (0, 1),
there exists an algorithm such that with probability at least 1 − δ,
s  !
Xn
k
R̂n = max (ytAt − yta ) = O nk log .
a∈A
t=1
δ
The poor behaviour of Exp3 occurs because the variance of the importance-
weighted estimators can become very large. In this chapter we modify the reward
estimates to control the variance at the price of introducing some bias.

12.1 The Exp3-IX Algorithm

We start by summarising what we know about the behaviour of the random regret
of Exp3. Because we want to use the loss-based estimator, it is more convenient
to switch to losses, which we do for the remainder of the chapter. Rewriting
Eq. (11.15) in terms of losses,

log(k) η X
k
L̂n − L̂ni ≤ + L̂nj , (12.1)
η 2 j=1

where L̂n and L̂ni are defined using the loss estimator Ŷtj by
n X
X k n
X
L̂n = Ptj Ŷtj and L̂ni = Ŷti .
t=1 j=1 t=1

Eq. (12.1) holds no matter how the loss estimators are chosen, provided
they satisfy 0 ≤ Ŷti ≤ 1/Pti for all t and i. Of course, the left-hand side of
Eq. (12.1) is not close to the regret unless Ŷti is a reasonable estimator of
the loss yti .
 12.1 The Exp3-IX Algorithm 165

We also need to define the sum of losses observed by the learner and for each
fixed action, which are
n
X n
X
L̃n = ytAt and Lni = yti
t=1 t=1

Like in the previous chapter, we need to define the (random) regret with respect
to a given arm i as follows:
n
X n
X
R̂ni = xti − Xt = L̃n − Lni . (12.2)
t=1 t=1

By substituting the above definitions into Eq. (12.1) and rearranging, the regret
with respect to arm i is bounded by

R̂ni = L̃n − Lni = (L̃n − L̂n ) + (L̂n − L̂ni ) + (L̂ni − Lni )

log(k)
k
η X
≤ + (L̃n − L̂n ) + (L̂ni − Lni ) + L̂nj . (12.3)
η 2 j=1

This means the random regret can be bounded by controlling L̃n − L̂n , L̂nj − Lnj
and L̂nj for each j. As promised we now modify the loss estimate. Let γ > 0 be
a small constant to be chosen later and define the biased estimator
I {At = i} Yt
Ŷti = . (12.4)
Pti + γ

First, note that Ŷti still satisfies 0 ≤ Ŷti ≤ 1/Pti , so (12.3) is still valid. As γ
increases, the predictable variance decreases, but the bias increases. The optimal
choice of γ depends on finding the sweet spot, which we will do once the dust
has settled in the analysis. When Eq. (12.4) is used in the exponential update in
Exp3, the resulting algorithm is called Exp3-IX (Algorithm 10). The suffix ‘IX’
stands for implicit exploration, a name justified by the following argument. A
simple calculation shows that
Pti yti γyti
Et [Ŷti ] = = yti − ≤ yti .
Pti + γ Pti + γ
Since small losses correspond to large rewards, the estimator is optimistically
biased. The effect is a smoothing of Pt so that actions with large losses for which
Exp3 would assign negligible probability are still chosen occasionally. In fact, the
smaller is Pti , the larger the bias is. As a result, Exp3-IX will explore more than
the standard Exp3 algorithm (see Exercise 12.5).

The reason for calling the exploration implicit is because the algorithm
explores more as a consequence of modifying the reward estimates, rather
than directly alternating Pt .
 12.2 Regret Analysis 166

1: Input: n, k, η, γ
2: Set L̂0i = 0 for all i
3: for t = 1, . . . , n do
4: Calculate the sampling distribution Pt :
 
exp −η L̂t−1,i
Pti = P  
exp
k
j=1 −η L̂ t−1,j

5: Sample At ∼ Pt and observe reward Xt

I {At = i} (1 − Xt )
6: Calculate L̂ti = L̂t−1,i +
Pti + γ
7: end for
Algorithm 10: Exp3-IX.

12.2 Regret Analysis

We now prove the following theorem bounding the random regret of Exp3-IX
with high probability.
Theorem 12.1. Let δ ∈ (0, 1) and define
r s
2 log(k + 1) δ )
log(k) + log( k+1
η1 = and η2 = .
nk nk
The following statements hold:
1 If Exp3-IX is run with parameters η = η1 and γ = η/2, then
s    !
p nk 1 k+1
P R̂n ≥ 8nk log(k + 1) + log + log ≤ δ.
2 log(k + 1) δ δ
(12.5)
2 If Exp3-IX is run with parameters η = η2 and γ = η/2, then
  
p k+1
P R̂n ≥ 2 (2 log(k + 1) + log(1/δ))nk + log ≤ δ. (12.6)
δ

The value of η1 is independent of δ, which means that using this choice of

learning rate leads to a single algorithm with a high-probability bound for
all δ. On the other hand, η2 does depend on δ, so the user must choose
a confidence level from the beginning. The advantage is that the bound
is improved, but only for the specified confidence level. We will show in
Chapter 17 that this trade-off is unavoidable.

The proof follows by bounding each of the terms in Eq. (12.3), which we do
via a series of lemmas. The first of these lemmas is a new concentration bound.
 12.2 Regret Analysis 167

To state the lemma, we recall two useful notions: Recall that given a filtration
F = (Ft )nt=0 , (Zt )nt=1 is F-adapted if for t ∈ [n], Zt is Ft -measurable and (Zt )nt=1
is F-predictable, if for t ∈ [n], Zt is Ft−1 -measurable.
Lemma 12.2. Let F = (Ft )nt=0 be a filtration and for i ∈ [k] let (Ỹti )t be F-adapted
such that:
Q  
 S ⊂[k] with |S| > 1, E i∈S Ỹti Ft−1 ≤ 0; and
1 for any 
2 E Ỹti  Ft−1 = yti for all t ∈ [n] and i ∈ [k].
Furthermore, let (αti )ti and (λti )ti be real-valued F-predictable random sequences
such that for all t, i it holds that 0 ≤ αti Ỹti ≤ 2λti . Then, for all δ ∈ (0, 1),
   !
1
Xn X k
Ỹti
P αti − yti ≥ log ≤ δ.
t=1 i=1
1 + λ ti δ

The proof relies on the Cramér–Chernoff method and is deferred until the
end of the chapter. Condition 1 states that the variables {Ỹti }i are negatively
correlated, and it helps us save a factor of k. Equipped with this result, we can
easily bound the terms L̂ni − Lni .
Lemma 12.3 (Concentration – variance). Let δ ∈ (0, 1). With probability at least
1 − δ, the following inequalities hold simultaneously:
  log( k+1 ) k 
X  log( k+1 )
max L̂ni − Lni ≤ δ
and L̂ni − Lni ≤ δ
. (12.7)
i∈[k] 2γ i=1
2γ

Proof Fix δ 0 ∈ (0, 1) to be chosen later and let Ati = I {At = i} as before. Then
X k Xn X k  
Ati yti
(L̂ni − Lni ) = − yti
i=1 t=1 i=1
Pti + γ
!
1 XX 1
n k
Ati yti
= 2γ − yti .
2γ t=1 i=1 1 + Pγti Pti

Introduce λti = Pγti , Ỹti = APtitiyti and αti = 2γ. Notice that the conditions of
Lemma 12.2 are now satisfied. In particular, for any S ⊂ [k] with |S| > 1, it holds
Q Q
that i∈S Ati = 0 and hence i∈S Ỹti = 0. Therefore,
!
log(1/δ 0 )
Xk
P (L̂ni − Lni ) ≥ ≤ δ0 . (12.8)
i=1
2γ

Similarly, for any fixed i,

 
log(1/δ 0 )
P L̂ni − Lni ≥ ≤ δ0 . (12.9)
2γ
To see this, use the previous argument with αtj = I {j = i} 2γ. The result follows
by choosing δ 0 = δ/(k + 1) and the union bound.
Pk
Lemma 12.4 (Bias). L̃n − L̂n = γ j=1 L̂nj .
 12.2 Regret Analysis 168

P
Proof Let Ati = I {At = i} as before. Writing Yt = j Atj ytj , we calculate
k
X k 
X  k
X Xk
Ptj Atj
Yt − Ptj Ŷtj = 1− Atj ytj = γ ytj = γ Ŷtj .
j=1 j=1
Ptj + γ j=1
Ptj + γ j=1
Pk
Therefore L̃n − L̂n = γ j=1 L̂nj as required.

Proof of Theorem 12.1 By Eq. (12.3) and Lemma 12.4, we have

log(k)
k
ηX
R̂n ≤ + (L̃n − L̂n ) + max(L̂ni − Lni ) + L̂nj
η i∈[k] 2 j=1

log(k) η X k
= + max(L̂ni − Lni ) + +γ L̂nj .
η i∈[k] 2 j=1

Therefore, by Lemma 12.3, with probability at least 1 − δ,

 

log(k) log k+1 η  X k
log k+1
R̂n ≤ + δ
+ γ+  Lnj + δ 
η 2γ 2 j=1
2γ

log(k)  η   η  log k+1

≤ + γ+ nk + γ + + 1 δ
,
η 2 2 2γ
where the second inequality follows since Lnj ≤ n for all j. The result follows by
substituting the definitions of η ∈ {η1 , η2 } and γ = η/2.

The attentive reader may be wondering whether proving the new

concentration inequality of Lemma 12.2, which looks a bit ad hoc, was
really necessary to get the bounds on L̂ni that were stated in the next
lemma. After all, we had a number of concentration inequalities available
to us that could be applied. As it turns out, one could also use Bernstein
inequality to get a result that only loses a factor of two compared to the
specialised lemma. The details are in Exercise 12.1. There are two important
lessons that are the basis of both proofs. The first is that since E[L̂ni ] < Lni
and the gap between these two quantities is large enough in a manner that
we make precise in Exercise 12.1, the deviation L̂ni − Lni can p be bounded
independently of Pti , Ati and yti . The price is that instead of log(1/δ), the
bound scales linearly with the generally larger quantity log(1/δ). The factor
1/γ here is the maximum scale of the individual summands in L̂ni . The
P
second lesson is specific to how in bounding i L̂ni − Lni a union bound over
i is avoided: this works because for a fixed time index t, (Ati )i are negatively
correlated. Negative dependence/association/correlation are known to be
good substitutes for independence, and by exploiting such properties one
can often demonstrate better concentration.
 12.2 Regret Analysis 169

12.2.1 Proof of Lemma 12.2

We start with a technical inequality:
 
x
Lemma 12.5. For any 0 ≤ x ≤ 2λ it holds that exp ≤ 1 + x.
1+λ
Note that 1+x ≤ exp(x). What the lemma shows is that by slightly discounting
the argument of the exponential function, in a bounded neighbourhood of zero,
1 + x can be an upper bound for the resulting function. Or, equivalently, slightly
inflating the linear term in 1 + x, the linear lower bound becomes an upper bound.
Proof of Lemma 12.5 We have
   
x x
exp ≤ exp ≤ 1 + x,
1+λ 1 + x/2
where the first inequality is because λ 7→ exp( 1+λx
) is decreasing in λ, and the
2u
second is because 1+u ≤ log(1 + 2u) holds for all u ≥ 0. This latter inequality
can be seen to hold by noting that for u = 0, the two sides are equal, while the
derivative of the left-hand side is smaller than that of the right-hand side at any
u ≥ 0.
Proof of Lemma 12.2 Fix t ∈ [n] and let Et [·] = E[· | Ft ] denote the conditional
expectation with respect to Ft . By Lemma 12.5 and the assumption that
0 ≤ αti Ỹti ≤ 2λti , we have
 
αti Ỹti
exp ≤ 1 + αti Ỹti .
1 + λti
Taking the product of these inequalities over i,
" k
!# " k # " k #
X αti Ỹti Y X
Et−1 exp ≤ Et−1 (1 + αti Ỹti ) ≤ 1 + Et−1 αti Ỹti
i=1
1 + λti i=1 i=1
k k
!
X X
=1+ αti yti ≤ exp αti yti , (12.10)
i=1 i=1
Qk P Qk
where the second inequality follows from i=1 (1 + ai ) = b∈{0,1}k i=1 abi i and
Q
the assumption that for S ⊂ [k] with |S| > 1, Et−1 [ i∈S Ỹti ] ≤ 0, the third one
follows from the assumption that Et−1 [Ỹti ] = yti , while the last one follows from
1 + x ≤ exp(x). Define
X k  !
Ỹti
Zt = exp αti − yti
i=1
1 + λti

and let Mt = Z1 . . . Zt , t ∈ [n] with M0 = 1. By (12.10), Et−1 [Zt ] ≤ 1. Therefore

E[Mt ] = E[Et−1 [Mt ]] = E[Mt−1 Et−1 [Zt ]] ≤ E[Mt−1 ] ≤ · · · ≤ E[M0 ] = 1 .
Setting t = n and combining the above display with Markov’s inequality leads to
P (log(Mn ) ≥ log(1/δ)) = P (Mn δ ≥ 1) ≤ E [Mn ] δ ≤ δ.
 12.3 Notes 170

12.3 Notes

1 An alternative to the somewhat custom-made Lemma 12.2 is to use a Bernstein-

type bound that simply bounds the deviation of a martingale from its mean
in terms of its quadratic variation. The slight disadvantage of this is that this
way we lose a factor of two. If this is not a concern, one may even prefer this
approach due to its greater transparency. For details, see Exercise 12.1.
2 An upper bound on the expected regret of Exp3-IX can be obtained by
integrating the tail:
Z ∞   Z ∞  
+ +
Rn ≤ E[(R̂n ) ] = P (R̂n ) ≥ x dx ≤ P R̂n ≥ x dx ,
0 0

where the first equality follows from Proposition 2.8. The result is completed
using either the high-probability bound in Theorem 12.1 and by straightforward
integration. We leave the details to the reader in Exercise 12.7.
3 The analysis presented here uses a fixed
p learning rate that depends on the
horizon. Replacing η and γ with ηt = log(k)/(kt) and γt = ηt /2 leads to an
anytime algorithm with about the same regret [Kocák et al., 2014, Neu, 2015a].
4 There is another advantage of the modified importance-weighted estimators
used by Exp3-IX, which leads to an improved regret in the special case that
one of the arms has small losses. Specifically, it is possible to show that
!
r
Rn = O k min Lni log(k) .
i∈[k]

In the worst case, Lni is linear in n and the usual bound is recovered. But
if the optimal arm enjoys low cumulative regret, then the above can be a
big improvement over the bounds given in Theorem 12.1. Bounds of this
kind are called first-order bounds. We refer the interested reader to the
papers by Allenberg et al. [2006], Abernethy et al. [2012] and Neu [2015b] and
Exercise 28.14.
5 Another situation where one might hope to have a smaller regret is when the
rewards/losses for each arm do not deviate too far from their averages. Define
the quadratic variation by
1X
n
X n
Qn = kxt − µk2 , where µ = xt .
t=1
n t=1
√
Hazan and Kale [2011] gave an algorithm for which Rn = O(k 2 Qn ), which can
be better than the worst-case bound of Exp3 or Exp3-IX when the quadratic
variation is very small. The factor of k 2 is suboptimal and can be removed
using a careful instantiation of the mirror descent algorithm [Bubeck et al.,
2018]. We do not cover this exact algorithm in this book, but the techniques
based on mirror descent are presented in Chapter 28.
6 An alternative to the algorithm presented here is to mix the probability
distribution computed using exponential weights with the uniform distribution,
 12.4 Bibliographic Remarks 171

while biasing the estimates. This leads to the Exp3.P algorithm due to Auer
et al. [2002b], who considered the case where δ is given and derived a bound that
is similar to Eq. (12.6) of Theorem 12.1. With an appropriate modification of
their proof, it is possible to derive a weaker bound similar to Eq. (12.5), where
the knowledge of δ is not needed by the algorithm. This has been explored by
Beygelzimer et al. [2010] in the context of a related algorithm, which will be
considered in Chapter 18. One advantage of this approach is that it generalises
to the case where the loss estimators are sometimes negative, a situation that
can arise in more complicated settings. For technical details, we advise the
reader to work through Exercise 12.3.

12.4 Bibliographic Remarks

The Exp3-IX algorithm is due to Kocák et al. [2014], who also introduced the
biased loss estimators. The focus of that paper was to improve algorithms for
more complex models with potentially large action sets and side information,
though their analysis can still be applied to the model studied in this chapter. The
observation that this algorithm also leads to high-probability bounds appeared in
a follow-up paper by Neu [2015a]. High-probability bounds for adversarial bandits
were first provided by Auer et al. [2002b] and explored in a more generic way by
Abernethy and Rakhlin [2009]. The idea to reduce the variance of importance-
weighted estimators is not new and seems to have been applied in various forms
[Uchibe and Doya, 2004, Wawrzynski and Pacut, 2007, Ionides, 2008, Bottou
et al., 2013]. All of these papers are based on truncating the estimators, which
makes the resulting estimator less smooth. Surprisingly, the variance-reduction
technique used in this chapter seems to be recent [Kocák et al., 2014].

12.5 Exercises

12.1 (Bernstein-type inequality and Lemma 12.3) Using the Berstein-type

inequality stated in Exercise 5.15, show the following:

(a) For any δ ∈ (0, 1), with probability at least 1 − δ, L̂ni − Lni < γ1 log(1/δ).
P P
(b) For any δ ∈ (0, 1), with probability at least 1 − δ, i L̂ni − i Lni <
1
γ log(1/δ).

12.2 Prove the claims made in Note 3.

Hint The source for this exercise is theorem 1 of the paper by Neu [2015a].
You can also read ahead and use the techniques from Exercise 28.13.
12.3 (Exp3.P) In this exercise we ask you to analyse the Exp3.P algorithm,
which as we mentioned in the notes is another way to obtain high probability
 12.5 Exercises 172

bounds. The idea is to modify Exp3 by biasing the estimators and introducing
some forced exploration. Let Ŷti = Ati yti /Pti − β/Pti be a biased version of the
loss-based importance-weighted estimator that was used in the previous chapter.
Pt
Define L̂ti = s=1 Ŷsi and consider the policy that samples At ∼ Pt , where
 
γ exp −η L̂t−1,i
Pti = (1 − γ)P̃ti + with P̃ti = P  .
k exp −η L̂
k
j=1 t−1,j

(a) Let δ ∈ (0, 1) and i ∈ [k]. Show that with probability 1 − δ, the random
regret R̂ni against i (cf. (12.2)) satisfies
r
n log(1/δ)
X n Xk Xn
β
R̂ni < nγ + (1 − γ) P̃tj (Ŷtj − yti ) + + .
t=1 j=1
P
t=1 tAt
2

(b) Show that

n X
X k n X
X k n
X
P̃tj (Ŷtj − yti ) = P̃tj (Ŷtj − Ŷti ) + (Ŷti − yti ) .
t=1 j=1 t=1 j=1 t=1

(c) Show that

log(k)
n X
X k Xn X k
P̃tj (Ŷtj − Ŷti ) ≤ +η P̃tj Ŷtj2 .
t=1 j=1
η t=1 j=1

(d) Show that

nk 2 β 2 X 1
n X
X k n
P̃tj Ŷtj2 ≤ + .
t=1 j=1
γ P
t=1 tAt

(e) Suppose that γ = kη and η = β. Apply the result of Exercise 5.15 to show
that for any δ ∈ (0, 1), the following hold:
 !
1 1
Xn
k
P ≥ 2nk + log ≤ δ.
P
t=1 tAt
γ δ
 !
1 1
Xn
P Ŷti − yti ≥ log ≤ δ.
t=1
β δ

(f) Combining the previous steps, show that there exists a universal constant
C > 0 such that for any δ ∈ (0, 1), for an appropriate choice of η, γ and β,
with probability at least 1 − δ it holds that the random regret R̂n of Exp3.P
satisfies
p
R̂n ≤ C nk log(k/δ) .
(g) In which step did you use the modified estimators?
(h) Show a bound where the algorithm parameters η, γ, β can only depend on
n, k, but not on δ.
(i) Compare the bounds with the analogous bounds for Exp3-IX in Theorem 12.1.
 12.5 Exercises 173

12.4 (Generic Exp3.P analysis) This exercise is concerned with a

generalisation of the core idea underlying Exp3.P of the previous exercise in that
rather than giving explicit expressions for the biased loss estimates, we focus
on the key properties required by the analysis of Exp3.P. To reduce clutter, we
assume for the remainder that t ranges in [n] and a ∈ [k]. Let (Ω, F, F, P) be a
filtered probability space with F = (Ft )nt=0 . Let (Zt ), (Ẑt ), (Z̃t ), (βt ) be sequences
of random elements in Rk , where Z̃t = Ẑt − βt and (Zt ), (βt ) are F-predictable,
whereas (Ẑt ) and therefore also (Z̃t ) are F-adapted. You should think of Ẑt as
the estimate of Zt that uses randomisation, and βt is the bias as in the previous
exercise. Given positive constant η, define the probability vector Pt ∈ Pk−1 by
 P 
exp −η s=1 Z̃sa
t−1

Pta = P  P .
b=1 exp −η
k t−1
s=1 Z̃sb

Let Et−1 [·] = E [· | Ft−1 ]. Assume the following hold for all a ∈ [k]:

(a) η|Ẑta | ≤ 1 , (b) ηβta ≤ 1 ,

2
(c) ηEt−1 [Ẑta ] ≤ βta almost surely , (d) Et−1 [Ẑta ] = Zta almost surely .

n X
X k
Pn
Let A∗ = argmina∈[k] t=1 Zta and Rn = Pta (Zta − ZtA∗ ).
t=1 a=1

(a) Show that

n X
X k
Pta (Zta − ZtA∗ )
t=1 a=1
n X
X k n X
X k n
X
= Pta (Z̃ta − Z̃tA∗ ) + Pta (Zta − Z̃ta ) + (Z̃tA∗ − ZtA∗ ) .
t=1 a=1 t=1 a=1 t=1
| {z } | {z } | {z }
(A) (B) (C)

(b) Show that

log(k) Xn X k Xn X k
2
(A) ≤ +η Pta Ẑta +3 Pta βta .
η t=1 a=1 t=1 a=1

(c) Show that with probability at least 1 − δ,

log(1/δ)
n X
X k
(B) ≤ 2 Pta βta + .
t=1 a=1
η

(d) Show that with probability at least 1 − kδ,

log(1/δ)
(C) ≤ .
η
 12.5 Exercises 174

(e) Conclude that for any δ ≤ 1/(k + 1), with probability at least 1 − (k + 1)δ,

3 log(1/δ) Xn X k Xn X k
2
Rn ≤ +η Pta Ẑta +5 Pta βta .
η t=1 a=1 t=1 a=1

Hint This is a long and challenging exercise. You may find it helpful to use
the result in Exercise 5.15. The solution is also available.
12.5 (Implementation) Consider the Bernoulli bandit with k = 5 arms and
n = 104 with means µ1 = 1/2 and µi = 1/2 − ∆ for i > 1. Plot the regret of Exp3
and Exp3-IX for ∆ ∈ [0, 1/2]. You should get a plot similar to that of Fig. 12.1.
Does the result surprise you?

300 Exp3
Exp3-IX

250
Regret

200

150

100
0 0.1 0.2 0.3 0.4 0.5
∆

Figure 12.1 Comparison between Exp3 and Exp3-IX on Bernoulli bandit

12.6 (Implementation: Variance of Exp3-IX) Repeat the experiment that

led to Fig. 11.4 but with Exp3 swapped to Exp3-IX. Use the confidence parameter
independent value of η and γ from Theorem 12.1. You should get a figure similar
to Fig. 12.2. Compare the new and the old figures and summarise your findings,
including the outcome of the results of Exercise 12.5.

12.7 (Expected regret of Exp3-IX) In this exercise, you will complete the
steps explained in Note 2 to prove a bound on the expected regret of Exp3-IX.
(a) Find a choice of η and universal constant C > 0 such that
p
Rn ≤ C kn log(k) .
(b) What happens as η grows? Write a bound on the expected regret of Exp3-IX
in terms of η and k and n.
 12.5 Exercises 175

−400

−500

−600
Regret

−700

−800
8

85

95

0.3
0.2

0.2
0.2

0.2

Figure 12.2 Box and whisker plot of the regret of Exp3-IX for the same setting as those
used to produce Fig. 11.4. For details of the experimental settings, see the text of
Exercise 11.6.
Part IV
Lower Bounds for Bandits
with Finitely Many Arms
 177

Until now, we have indulged ourselves by presenting algorithms and upper

bounds on their regret. As satisfying as this is, the real truth of a problem is
usually to be found in the lower bounds. There are several reasons for this:

1 An upper bound does not tell you much about what you could be missing
out on. The only way to demonstrate that your algorithm really is (close to)
optimal is to prove a lower bound showing that no algorithm can do better.
2 The second reason is that lower bounds are often more informative in the
sense that it usually turns out to be easier to get the lower bound right than
the upper bound. History shows a list of algorithms with steadily improving
guarantees until eventually someone hits upon the idea for which the upper
bound matches some known lower bound.
3 Finally, thinking about lower bounds forces you to understand what is hard
about the problem. This is so useful that the best place to start when attacking
a new problem is usually to try and prove lower bounds. Too often we have
not heeded our own advice and started trying to design an algorithm, only
to discover later that had we tackled the lower bound first, then the right
algorithm would have fallen in our laps with almost no effort at all.

So what is the form of a typical lower bound? In the chapters that follow, we
will see roughly two flavours. The first is the worst-case lower bound, which
corresponds to a claim of the form
‘For any policy you give me, I will give you an instance of a bandit problem ν on which
the regret is at least L’ .

Results of this kind have an adversarial flavour, which makes them suitable for
understanding the robustness of a policy. The second type is a lower bound on
the regret of an algorithm for specific instances. These bounds have a different
form that usually reads like the following:
‘If you give me a reasonable policy, then its regret on any instance ν is at least L(ν)’ .

The statement only holds for some policies – the ‘reasonable’ ones, whatever that
means. But the guarantee is also more refined because bound controls the regret
for these policies on every instance by a function that depends on this instance.
This kind of bound will allow us to show that the instance-dependent bounds
P
for stochastic bandits of O( i:∆i >0 ∆i + log(n)/∆i ) are not improvable. The
inclusion of the word ‘reasonable’ is unfortunately necessary. For every bandit
instance ν there is a policy that just chooses the optimal action in ν. Such policies
are not reasonable because they have linear regret for bandits with a different
optimal arm. There are a number of ways to define ‘reasonable’ in a way that is
simultaneously rigorous and, well, reasonable.
The contents of this part is roughly as follows. First we introduce the definition
of worst-case regret and discuss the line of attack for proving lower bounds
(Chapter 13). The next chapter takes us on a brief excursion into information
theory, where we explain the necessary mathematical tools (Chapter 14). Readers
 178

familiar with information theory could skim this chapter. The final three chapters
are devoted to applying information theory to prove lower bounds on the regret
for both stochastic and adversarial bandits.
13 Lower Bounds: Basic Ideas

The worst-case regret of a policy π on a set of stochastic bandit environments

E is

Rn (π, E) = sup Rn (π, ν) .

ν∈E

Let Π be the set of all policies. The minimax regret is

Rn∗ (E) = inf Rn (π, E) = inf sup Rn (π, ν) .

π∈Π π∈Π ν∈E

A policy is called minimax optimal for E if Rn (π, E) = Rn∗ (E). The value Rn∗ (E)
is of interest by itself. A small value of Rn∗ (E) indicates that the underlying bandit
problem is less challenging in the worst-case sense. A core activity in bandit
theory is to understand what makes Rn∗ (E) large or small, often focusing on its
behaviour as a function of the number of rounds n.

Minimax optimality is not a property of a policy alone. It is a property of a

policy together with a set of environments and a horizon.

Finding a minimax policy is generally too computationally expensive to be

practical. For this reason, we almost always settle for a policy that is nearly
minimax optimal.
One of the main results of this part is a proof of the following theorem, which
together with Theorem 9.1 shows that Algorithm 7 from Chapter 9 is minimax
optimal up to constant factors for 1-subgaussian bandits with suboptimality gaps
in [0, 1].

Theorem 13.1. Let E k be the set of k-armed Gaussian bandits with unit variance
and means µ ∈ [0, 1]k . Then there exists a universal
√ constant c > 0 such that for
all k > 1 and n ≥ k, it holds that Rn∗ (E k ) ≥ c kn.

We will prove this theorem in Chapter 15, but first we give an informal
justification.
 13.1 Main Ideas Underlying Minimax Lower Bounds 180

13.1 Main Ideas Underlying Minimax Lower Bounds

Let X1 , . . . , Xn be a sequence of independent Gaussian random variables with

unknown mean µ and known variance 1. Assume you are told that µ takes on
one of two values: µ = 0 or µ = ∆ for some known ∆ > 0. Your task is to guess
Pn
the value of µ based on your observation of X1 , . . . , Xn . Let µ̂ = n1 i=1 Xi be
the sample mean, which is Gaussian with mean µ and variance 1/n. While it is
not immediately obvious how easy this task is, intuitively we expect the optimal
decision is to predict that µ = 0 if µ̂ is closer to 0 than to ∆, and otherwise to
predict µ = ∆. For large n we expect our prediction will probably be correct.
Supposing that µ = 0 (the other case is symmetric), then the prediction will be
wrong only if µ̂ ≥ ∆/2. Using the fact that µ̂ is Gaussian with mean µ = 0 and
variance 1/n, combined with known bounds on the Gaussian tail probabilities
(see Eq. (13.4)), leads to
r    
1 8 n∆2 ∆
√ √ exp − ≤ P µ̂ ≥
n∆2 + n∆2 + 16 π 8 2
r  
1 8 n∆2
≤√ p exp − .
n∆2 + n∆2 + 32/π π 8
(13.1)
The upper and lower bounds only differ in the constant in the square root of the
denominator. One might believe that the decision procedure could be improved,
but the symmetry of the problem makes this seem improbable. The formula
exhibits the expected behaviour, which is that once n is large relative to 8/∆2 ,
then the probability that this procedure fails drops exponentially with further
increases in n. But the lower bound also shows that if n is small relative to 8/∆2 ,
then the procedure fails with constant probability.
The problem described is called hypothesis testing, and the ideas underlying
the argument above are core to many impossibility results in statistics. The next
task is to reduce our bandit problem to hypothesis testing. The high-level idea
is to select two bandit problem instances in such a way that the following two
conditions hold simultaenously:

1 Competition: An action, or, more generally, a sequence of actions that is good

for one bandit is not good for the other.
2 Similarity: The instances are ‘close’ enough that the policy interacting with
either of the two instances cannot statistically identify the true bandit with
reasonable statistical accuracy.

The two requirements are clearly conflicting. The first makes us want to choose
instances with means µ, µ0 ∈ [0, 1]k that are far from each other, while the second
requirement makes us want to choose them to be close to each other. The lower
bound will follow by optimising this trade-off.
Let us start to make things concrete by choosing bandits ν = (Pi )ki=1 and
 13.1 Main Ideas Underlying Minimax Lower Bounds 181

ν 0 = (Pi0 )ki=1 , where Pi = N (µi , 1) and Pi0 = N (µ0i , 1) are Gaussian and
µ, µ0 ∈ [0, 1]k . We will also assume that n is larger than k by some suitably
large constant factor. In order to prove a lower bound, it suffices to show that for
every strategy π, there exists a choice of µ and µ0 such that
√
max {Rn (π, ν), Rn (π, ν 0 )} ≥ c kn ,

where c > 0 is a universal constant. Let ∆ ∈ (0, 1/2] be a constant to be tuned

subsequently and choose µ = (∆, 0, 0, . . . , 0), which means that the first arm is
optimal in instance ν and

Rn (π, ν) = (n − E[T1 (n)])∆ , (13.2)

where the expectation is taken with respect to the induced measure on the
sequence of outcomes when π interacts with ν. Now we need to choose µ0 to
satisfy the two requirements above. Since we want ν and ν 0 to be hard to
distinguish and yet have different optimal actions, we should make µ0 as close to
µ except in a coordinate where π expects to explore the least. To this end, let

i = argminj>1 E[Tj (n)]

be the suboptimal arm in ν that π expects to play least often. From n =

P
E[T1 (n)] + j>1 E[Tj (n)] ≥ (k − 1)E[Ti (n)] we see that
n
E[Ti (n)] ≤
k−1
must hold. Then, define µ0 ∈ Rk by
(
µj , if j 6= i ;
µj =
0
2∆ , otherwise .

The regret in this bandit is

X
Rn (π, ν 0 ) = ∆E0 [T1 (n)] + 2∆E0 [Tj (n)] ≥ ∆E0 [T1 (n)] , (13.3)
j ∈1,i
/

where E0 [·] is the expectation operator on the sequence of outcomes when π

interacts with ν 0 . So now we have the following situation: the strategy π interacts
with either ν or ν 0 , and when interacting with ν, it expects to play arm i at
most n/(k − 1) times. But the two instances only differ when playing arm i. The
time has come to tune ∆. Because the strategy expects to play arm i only about
n/(k − 1) times, taking inspiration from the previous discussion on distinguishing
samples from Gaussian distributions with different means, we will choose
s r
1 k−1
∆= ≥ .
E[Ti (n)] n

If we are prepared to ignore the fact that Ti (n) is a random variable and take for
granted the claims in the first part of the chapter, then with this choice of ∆,
 13.2 Notes 182

the strategy cannot distinguish between instances ν and ν 0 , and in particular we

expect that E[T1 (n)] ≈ E0 [T1 (n)]. If E[T1 (n)] < n/2, then by Eq. (13.2) we have
r
n k−1 1p
Rn (π, ν) ≥ = n(k − 1) .
2 n 2
On the other hand, if E[T1 (n)] ≥ n/2, then
1p
n(k − 1) ,
Rn (π, ν 0 ) ≥ ∆E0 [T1 (n)] ≈ ∆E[T1 (n)] ≥
2
which completes our heuristic argument that there exists a universal constant
c > 0 such that
√
Rn∗ (E k ) ≥ c nk .
We have been sloppy in many places. The claims in the first part of the chapter
have not been proven yet, and Ti (n) is a random variable. Before we can present the
rigourous argument, we need a chapter to introduce some ideas from information
theory. Readers already familiar with these concepts can skip to Chapter 15 for
the proof of Theorem 13.1.

13.2 Notes

1 The worst-case regret has a game-theoretic interpretation. Imagine a game

between a protagonist and an antagonist that works as follows: for k > 1 and
n ≥ k the protagonist proposes a bandit policy π. The antagonist looks at the
policy and chooses a bandit ν from the class of environments considered. The
utility for the antagonist is the expected regret, and for the protagonist it is the
negation of the expected regret, which makes this a zero-sum game. Both players
aim to maximise their pay-offs. The game is completely described by n and E.
One characteristic value in a game is its minimax value. As described above,
this is a sequential game (the protagonist moves first, then the antagonist). The
minimax value of this game from the perspective of the antagonist is exactly
Rn∗ (E), while for the protagonist, it is supπ inf ν (−Rn (π, ν)) = −Rn∗ (E).
2 We mentioned that finding the minimax optimal policy is usually
computationally infeasible. In fact it is not clear we should even try. In classical
statistics, it often turns out that minimising the worst case leads to a flat
risk profile. In the language of bandits, this would mean that the regret is
the same for every bandit (where possible). What we usually want in practice
is to have low regret against ‘easy’ bandits and larger regret against ‘hard’
bandits. The analysis in Part II suggests that easy bandits are those where the
suboptimality gaps are large or very small. There is evidence to suggest that
the exact minimax optimal strategy may not exploit these easy instances, so
in practice one might prefer to find a policy that is nearly minimax optimal
and has much smaller regret on easy bandits. We will tackle questions of this
nature in Chapter 16.
 13.3 Bibliographic Remarks 183

3 The regret on a class of bandits E is a multi-objective criterion. Some policies

will be good for some instances and bad on others, and there are clear trade-
offs. One way to analyse the performance in a multi-objective setting is called
Pareto optimality. A policy is Pareto optimal if there does not exist another
policy that is a strict improvement – more precisely, if there does not exist a
π 0 such that Rn (π 0 , ν) ≤ Rn (π, ν) for all ν ∈ E and Rn (π 0 , ν) < Rn (π, ν) for
at least one instance ν ∈ E.
4 When we say a policy is minimax optimal up to constant factors for finite-armed
1-subgaussian bandits with suboptimality gaps in [0, 1], we mean there exists a
C > 0 such that
Rn (π, E k )
≤ C for all k and n ,
Rn∗ (E k )
where E k is the set of k-armed 1-subgaussian bandits with suboptimality gaps
in [0, 1]. We often say a policy is minimax optimal up to logarithmic factors,
by which we mean that
Rn (π, E k )
≤ C(n, k) for all k and n ,
Rn∗ (E k )
where C(n, k) is logarithmic in n and k. We hope the reader will forgive us
for not always specifying in the text exactly what is meant and promise that
statements of theorems will always be precise.

13.3 Bibliographic Remarks

The bound on Gaussian tails used in Eq. (13.1) is derived from §7.1.13 of the
reference book by Abramowitz and Stegun [1964], which bounds
Z ∞
exp(−x2 ) exp(−x2 )
√ ≤ exp(−t2 )dt ≤ p for all x ≥ 0 . (13.4)
x + x2 + 2 x x + x2 + 4/π

13.4 Exercises

13.1 (Minimax risk for hypothesis testing) Let Pµ = N (µ, 1) be the

Gaussian measure on (R, B(R)) with mean µ ∈ {0, ∆} and unit variance. Let
X : R → R be the identity random variable (X(ω) = ω). For decision rule
d : R → {0, ∆}, define the risk
R(d) = max Pµ (d(X) 6= µ) ,
µ∈{0,∆}

Prove that R(d) is minimised by d(x) = argminµ̃∈{0,∆} |x − µ̃| .

13.2 (Pareto optimal policies) Let k > 1 and E = EN k

(1) be the set of
Gaussian bandits with unit variance. Find a Pareto optimal policy for this class.
 13.4 Exercises 184

Hint Think about simple policies (not necessarily good ones) and use the
definition.
14 Foundations of Information Theory
( )

To make the arguments in the previous chapter rigourous and generalisable to

other settings, we need some tools from information theory and statistics. The
most important of these is the relative entropy, also known as the Kullback–
Leibler divergence named for Solomon Kullback and Richard Leibler (KL
divergence, for short).

14.1 Entropy and Optimal Coding

Alice wants to communicate with Bob. She wants to tell Bob the outcome of a
sequence n of independent random variables sampled from known distribution Q.
Alice and Bob agree to communicate using a binary code that is fixed in advance
in such a way that the expected message length is minimised. The entropy of Q
is the expected number of bits necessary per random variable using the optimal
code as n tends to infinity. The relative entropy between distributions P and Q
is the price in terms of expected message length that Alice and Bob have to pay
if they believe the random variables are sampled from Q when in fact they are
sampled from P .
Let P be a measure on [N ] with σ-algebra 2[N ] and X : [N ] → [N ] be the
identity random variable, X(ω) = ω. Alice observes a realisation of X and wants
to communicate the result to Bob using a binary code that they agree upon
in advance. For example, when N = 4, they might agree on the following code:
1 → 00, 2 → 01, 3 → 10, 4 → 11. Then if Alice observes a 3, she sends Bob a
message containing 10. For our purposes, a code is a function c : [N ] → {0, 1} ,
∗

where {0, 1} is the set of finite sequences of zeros and ones.

∗

Of course c must be injective so that no two numbers (or symbols) have the
same code. We also require that c be prefix free, which means that no code is a
prefix of any other. This is justified by supposing that Alice would like to tell
Bob about multiple samples. Then Bob needs to know where the message for one
symbol starts and ends.

Using a prefix code is not the only way to enforce unique decodability, but
all uniquely decodable codes have equivalent prefix codes (see Note 1).
 14.1 Entropy and Optimal Coding 186

The easiest choice is to use dlog2 (N )e bits

111 L 10101 B 011000
no matter the value of X. This simple code
E 010 D 01101 V 1100000
is sometimes effective, but is not entirely T 1101 C 00001 K 11000011
satisfactory if X is far from uniform. To A 1011 U 00000 X 110000100
understand why, suppose that N is extremely O 1001 F 110011 J 1100001011
large and P (X = 1) = 0.99, and the I 1000 M 110010 Q 11000010101
remaining probability mass is uniform over N 0111 W 110001 Z 11000010100
[N ] \ {1}. Then it seems preferable to have a S 0011 Y 101001
short code for one and slightly longer codes for H 0010 P 101000
the alternatives. With this in mind, a natural R 0001 G 011001

objective is to find a code that minimises the

expected code length. That is, Figure 14.1 A Huffman code for the
English alphabet, including space.
N
X
c∗ = argminc pi `(c(i)) , (14.1)
i=1

where the argmin is taken over valid codes and `(·) is a function that returns
the length of a code. The optimisation problem in (14.1) can be solved using
Huffman coding, and the optimal value satisfies
N
X
H2 (P ) ≤ pi `(c∗ (i)) ≤ H2 (P ) + 1 , (14.2)
i=1

where H2 (P ) is the entropy of P ,

 
X 1
H2 (P ) = pi log2 .
pi
i∈[N ]:pi >0

When pi = 1/N is uniform, the naive idea of using a code of uniform length is
recovered, but for non-uniform distributions, the code adapts to assign shorter
codes to symbols with larger probability. It is worth pointing out that the sum
is only over outcomes that occur with non-zero probability, which is motivated
by observing that limx→0+ x log(1/x) = 0 or by thinking of the entropy as an
expectation of the log probability with respect to P , and expectations should not
change when the value of the random variable is perturbed on a measure zero set.
It turns out that H2 (P ) is not just an approximation on the expected length of
the Huffman code, but is itself a fundamental quantity. Imagine that Alice wants
to transmit a long string of symbols sampled from P . She could use a Huffman
code to send Bob each symbol one at a time, but this introduces rounding errors
that accumulate as the message length grows. There is another scheme called
arithmetic coding for which the average number of bits per symbol approaches
H2 (P ) and the source coding theorem says that this is unimprovable.
The definition of entropy using base 2 makes sense from the perspective of
sending binary message. Mathematically, however, it is more convenient to define
 14.2 Relative Entropy 187

the entropy using the natural logarithm:

 
X 1
H(P ) = pi log . (14.3)
pi
i∈[N ]:pi >0

This is nothing more than a scaling of the H2 . Measuring information using base
2 logarithms has a unit of bits, and for the natural logarithm the unit is nats.
By slightly abusing terminology, we will also call H(P ) the entropy of P .

14.2 Relative Entropy

Suppose that Alice and Bob agree to use a code that is optimal when X is sampled
from distribution Q. Unbeknownst to them, however, X is actually sampled from
distribution P . The relative entropy between P and Q measures how much longer
the messages are expected to be using the optimal code for Q than what would be
obtained using the optimal code for P . Letting pi = P (X = i) and qi = Q(X = i),
assuming Shannon coding, working out the math while dropping d·e leads to the
definition of relative entropy as
     
X 1 X 1 X pi
D(P, Q) = pi log − pi log = pi log
qi pi qi
i∈[N ]:pi >0 i∈[N ]:pi >0 i∈[N ]:pi >0
(14.4)
From the coding interpretation, one conjectures that D(P, Q) ≥ 0. Indeed, this
is easy to verify using Jensen’s inequality. Still poking around the definition,
what happens when qi = 0 and pi = 0? This means that symbol i is superfluous
and the value of D(P, Q) should not be impacted by introducing superfluous
symbols. And again, it is not by the definition of the expectations. We also see
that the sufficient and necessary condition for D(P, Q) < ∞ is that for each i
with qi = 0, we also have pi = 0. The condition we discovered is equivalent to
saying that P is absolutely continuous with respect to Q. Note that absolute
continuity only implies a finite relative entropy when X takes on finitely many
values (Exercise 14.2).
This brings us back to defining relative entropy between probability measures
P and Q on arbitrary measurable spaces (Ω, F). When the support of P is
uncountable, defining the entropy via communication is hard because infinitely
many symbols are needed to describe some outcomes. This seems to be a
fundamental difficulty. Luckily, the impasse gets resolved automatically if we
only consider relative entropy. While we cannot communicate the outcome, for
any finite discretisation of the possible outcomes, the discretised values can be
communicated finitely, and all our definitions will work. Formally, a discretisation
to [N ] is specified by a F/2[N ] -measurable map X : Ω → [N ]. Then the entropy
of P relative Q can be defined as
D(P, Q) = sup sup D(PX , QX ) , (14.5)
N ∈N+ X
 14.2 Relative Entropy 188

where PX is the push-forward of P on [N ] defined by PX (A) = P (X ∈ A). The

inner supremum is over all F/2N -measurable maps. Informally we take all possible
discretisations X (with no limit on the ‘fineness’ of the discretisation) and define
D(P, Q) as the excess information when expecting to see X with X ∼ QX , while
in reality X ∼ PX . As we shall see soon, this is indeed a reasonable definition.

Theorem 14.1. Let (Ω, F) be a measurable space, and let P and Q be measures
on this space. Then,
R  
 log dP (ω) dP (ω) , if P  Q ;
D(P, Q) = dQ
∞ , otherwise .

Note that the relative entropy between P and Q can still be infinite even when
P  Q. Note also that in the case of discrete measures, the above expression
reduces to (14.4). For calculating relative entropies densities one often uses
densities: If λ is a common dominating σ-finite measure for P and Q (that is,
P  λ and Q  λ both hold), then letting p = dP dλ and q = dλ , if also P  Q,
dQ

the chain rule gives dQ dλ = dλ , which lets us write

dP dQ dP

Z  
p
D(P, Q) = p log dλ . (14.6)
q
This is probably the best-known expression for relative entropy and is often used
as a definition. Note that for probability measures, a common dominating σ-finite
measure can always be bound. For example, λ = P + Q always dominates both
P and Q.
Relative entropy is a kind of ‘distance’ measure between distributions P and
Q. In particular, D(P, Q) = 0 whenever P = Q, and otherwise D(P, Q) > 0.
However, strictly speaking, the relative entropy is not a distance because it
satisfies neither the triangle inequality nor is it symmetric. Nevertheless, it serves
the same purpose.
The relative entropy between many standard distributions is often quite easy
to compute. For example, the relative entropy between two Gaussians with means
µ1 , µ2 ∈ R and common variance σ 2 is
(µ1 − µ2 )2
D(N (µ1 , σ 2 ), N (µ2 , σ 2 )) = .
2σ 2
The dependence on the difference in means and the variance is consistent with
our intuition. If µ1 is close to µ2 , then the ‘difference’ between the distributions
should be small, but if the variance is very small, then there is little overlap, and
the difference is large. The relative entropy between two Bernoulli distributions
with means p, q ∈ [0, 1] is
   
p 1−p
D(B(p), B(q)) = p log + (1 − p) log ,
q 1−q
where 0 log(·) = 0. Due to its frequent appearance at various places, D(B(p), B(q))
 14.2 Relative Entropy 189

gets the honour of being abbreviated to d(p, q), which we have met before in
Definition 10.1.
We are nearing the end of our whirlwind tour of relative entropy. It remains
to state the key lemma that connects the relative entropy to the hardness of
hypothesis testing.

Theorem 14.2 (Bretagnolle–Huber inequality). Let P and Q be probability

measures on the same measurable space (Ω, F), and let A ∈ F be an arbitrary
event. Then,
1
P (A) + Q(Ac ) ≥ exp (− D(P, Q)) , (14.7)
2
where Ac = Ω \ A is the complement of A.

The proof may be found at the end of the chapter, but first some interpretation
and a simple application. Suppose that D(P, Q) is small; then P is close to Q
in some sense. Since P is a probability measure, we have P (A) + P (Ac ) = 1.
If Q is close to P , then we might expect that P (A) + Q(Ac ) should be large.
The purpose of the theorem is to quantify just how large. Note that if P is not
absolutely continuous with respect to Q, then D(P, Q) = ∞, and the result is
vacuous. Also note that the result is symmetric. We could replace D(P, Q) with
D(Q, P ), which sometimes leads to a stronger result because the relative entropy
is not symmetric.
Returning to the hypothesis-testing problem described in the previous chapter,
let X be normally distributed with unknown mean µ ∈ {0, ∆} and variance
σ 2 > 0. We want to bound the quality of a rule for deciding what is the real mean
from a single observation. The decision rule is characterised by a measurable
set A ⊆ R on which the predictor guesses µ = ∆ (it predicts µ = 0 on the
complement of A). Let P = N (0, σ 2 ) and Q = N (∆, σ 2 ). Then the probability
of an error under P is P (A), and the probability of error under Q is Q(Ac ). The
reader surely knows what to do next. By Theorem 14.2, we have
 
1 1 ∆2
P (A) + Q(A ) ≥ exp (− D(P, Q)) = exp − 2 .
c
2 2 2σ
If we assume that the signal-to-noise ratio is small, ∆2 /σ 2 ≤ 1, then
 
1 1 3
P (A) + Q(A ) ≥ exp −
c
≥ ,
2 2 10
which implies max {P (A), Q(Ac )} ≥ 3/20. This means that no matter how we
chose our decision rule, we simply do not have enough data to make a decision
for which the probability of error on either P or Q is smaller than 3/20.

Proof of Theorem 14.2 For reals a, b, we abbreviate max {a, b} = a ∨ b and

min {a, b} = a ∧ b. The result is trivial if D(P, Q) = ∞. On the other hand, by
Theorem 14.1, D(P, Q) < ∞ implies that P  Q. Let ν = P + Q. Then P, Q  ν,
which by Theorem 2.13 ensures the existence of the Radon–Nikodym derivatives
 14.3 Notes 190

R  
p = dP
dν and q = dν . By Eq. (14.6), D(P, Q) =
dQ
p log pq dν. For brevity, when
writing integrals with
R respect to ν, in this proof, we will drop dν. Thus, we will
write, for example, p log(p/q) for the above integral.
Instead of (14.7), we prove the stronger result that
Z
1
p ∧ q ≥ exp(− D(P, Q)) . (14.8)
2
R R R R R
This indeed is sufficient since p ∧ q = A p ∧ q + Ac p ∧ q ≤ A p + Ac q =
P (A) + Q(Ac ). We start with an inequality attributed to French mathematician
Lucien Le Cam, which lower-bounds the left-hand side of Eq. (14.8). The inequality
states that
Z Z 2
1 √
p∧q ≥ pq . (14.9)
2
How do we get this inequality? Starting from the right-hand side above, using
pq = (p ∧ q)(p ∨ q) and Cauchy–Schwarz we get
Z 2 Z 2 Z  Z 
√ p
pq = (p ∧ q)(p ∨ q) ≤ p∧q p∨q .

Now,
R using Rp ∧ q + p ∨ q = p + q, the proof is finished by substituting
p ∨ q = 2 − p ∧ q ≤ 2 and dividing both sides by two. It remains to lower-bound
the right-hand side of (14.9). For this, we use Jensen’s inequality. First, we write
(·)2 as exp(2 log(·)) and then move the log inside the integral:
Z 2  Z   Z r 
√ √ q
pq = exp 2 log pq = exp 2 log p
p>0 p
 Z    Z  
1 q p
≥ exp 2 p log = exp − p log
p>0 2 p pq>0 q
 Z  
p
= exp − p log = exp (− D(P, Q)) .
q
In the fourth and the last step, we used that since P  Q, q = 0 implies p = 0,
and so p > 0 implies q > 0, and eventually pq > 0. The result is completed by
chaining the inequalities.

14.3 Notes

1 A code c : N+ → {0, 1}∗ is uniquely decodable if i1 , . . . , in 7→ c(i1 ) · · · c(in )

is injective, where on the right-hand side the codes are simply concatenated.
Kraft’s inequality states that for any uniquely decodable code c,
∞
X
2−`(c(i)) ≤ 1 . (14.10)
i=1
 14.3 Notes 191

P∞ −`
Furthermore, for any (`n )∞ n=1 satisfying i=1 2 ≤ 1, there exists a prefix
code c : N+ → {0, 1}∗ such that `(c(i)) = `i . The second part justifies our
restriction to prefix codes rather than uniquely decodable codes in the definition
of the entropy.
2 The supremum in the definition given in Eq. (14.5) may often be taken over
a smaller set. Precisely, let (X , G) be a measurable space and suppose that
G = σ(F) where F is a field. Note that a field is defined by the same axioms
as a σ-algebra except that being closed under countable unions is replaced by
the condition that it be closed under finite unions. Then, for measures P and
Q on (X , G), it holds that

D(P, Q) = sup D(Pf , Qf ) ,

f

where the supremum is over F/2[n] -measurable functions. This result is known
as Dobrushin’s theorem.
R
3 In the proof of Theorem 14.2 we used the inequality P (A) + Q(Ac ) ≥ p ∧ q.
Looking at the proof, it is not hard to see that the inequality becomes an equality
when A = {p ≤ q} = {q/p ≥ 1}. Reader’s familiar with statistical decision
theory may recognize that this is a special case of the Neyman-Pearson
lemma which states that the most powerful test among all statistical tests in a
simple hypothesis testing problem at a given significance level is the likelihood
ratio test. Exercise 14.13 explores this connection.
4 How tight is Theorem 14.2? We remarked already that D(P, Q) = 0 if and only
if P = Q. But in this case, Theorem 14.2 only gives
1 1
1 = P (A) + Q(Ac ) ≥ exp (− D(P, Q)) = ,
2 2
which does not seem so strong. From where does the weakness arise? The
answer is in Le Cam’s inequality, Eq. (14.9), which can be refined by
Z 2 Z  Z  Z  Z 
√
pq ≤ p∧q p∨q = p∧q 2− p∧q .

By solving the quadratic inequality, we have

s Z 2
Z
√
P (A) + Q(A ) ≥
c
p∧q ≥1− 1− pq
p
≥ 1 − 1 − exp (− D(P, Q)) , (14.11)

which gives a modest improvement on Theorem 14.2 that becomes more

pronounced when D(P, Q) is close to zero, as demonstrated by Fig. 14.2. This
stronger bound might be useful for fractionally improving constant factors in
lower bounds, but we do not know of any application for which it is really
crucial, and the more complicated form makes it cumbersome to use. Part of
the reason for this is that the situation where D(P, Q) is small is better dealt
with using a different inequality, as explained in the next note.
 14.3 Notes 192

1 x/2
√
1− 1−x
p
1− log(1/x)/2

0.5

−0.5

0 1
x

Figure 14.2 Tightening the inequality of Le Cam. Here, x = exp(− D(P, Q)). Higher
values are better as the figure shows lower bounds on P (A) + Q(Ac ).

5 Another inequality from information theory is Pinsker’s inequality, which

states for measures P and Q on the same probability space (Ω, F) that
r
1
δ(P, Q) = sup P (A) − Q(A) ≤ D(P, Q) . (14.12)
A∈F 2
The quantity on the left-hand side is called the total variation distance
between P and Q, which is a distance on the space of probability measures on
a probability space. From this we can derive for any measurable A ∈ F that
r s  
1 1 1
P (A) + Q(A ) ≥ 1 −
c
D(P, Q) = 1 − log .
2 2 exp(− D(P, Q))
(14.13)
Examining Fig. 14.2 shows that this is an improvement on Eq. (14.11) when
D(P, Q) is small. However, we also see that in the opposite case, when D(P, Q)
is large, Eq. (14.13) is worse than Eq. (14.11), or the inequality in Theorem 14.2.
6 We saw the total variation distance in Eq. (14.12). There are two other
‘distances’ that are occasionally useful. These are the Hellinger distance
and the χ-squared distance, which, using the notation in the proof of
Theorem 14.2, are defined by
sZ s  Z 
√ √ 2 √
h(P, Q) = ( p − q) = 2 1 − pq (14.14)
Z 2 Z
(p − q) p2
χ2 (P, Q) = = − 1. (14.15)
q q
The Hellinger distance is bounded and exists for all probability measures P
and Q. A necessary condition for the χ2 -distance to exist is that P  Q. Like
 14.4 Bibliographic Remarks 193

the total variation distance, the Hellinger distance is actually a distance (it is
symmetric and satisfies triangle inequality), but the χ2 -‘distance’ is not. It is
possible to show (Tsybakov [2008], chapter 2) that
δ(P, Q)2 ≤ h(P, Q)2 ≤ D(P, Q) ≤ χ2 (P, Q) . (14.16)
All the inequalities are tight for some choices of P and Q, but the examples
do not chain together, as evidenced by Pinsker’s inequality, which shows that
δ(P, Q)2 ≤ D(P, Q)/2 (which is also tight for some P and Q).
7 The entropy for distribution P was defined as H(P ) in Eq. (14.3). If X is a
random variable, then H(X) is defined to be the entropy of the law of X. This
is a convenient notation because it allows one to write H(f (X)) and H(XY )
and similar expressions.

14.4 Bibliographic Remarks

There are many references for information theory. Most well known (and
comprehensive) is the book by Cover and Thomas [2012]. Another famous book
is the elementary and enjoyable introduction by MacKay [2003]. The approach
we have taken for defining and understanding the relative entropy is inspired by
an excellent shorter book by Gray [2011]. Theorem 14.1 connects our definition of
relative entropies to densities (the ‘classic definition’). It can be found in §5.2 of
the aforementioned book. Dobrushin’s theorem is due to him [Dobrushin, 1959].
An alternative source is lemma 5.2.2 in the book of Gray [2011]. Theorem 14.2 is
due to Bretagnolle and Huber [1979]. We also recommend the book by Tsybakov
[2008] as a good source for learning about information theoretic lower bounds in
statistical settings.

14.5 Exercises

14.1 Let P be a probability distribution on N+ and pi = P ({i}). Show that for

any prefix code c : N+ → {0, 1}∗ , it holds that
∞
X
pi `(c(i)) ≥ H2 (P ) .
i=1

Hint Use Kraft’s inequality from Note 1.

14.2 Find probability measures P and Q on N+ with P  Q and D(P, Q) = ∞.

14.3 Prove the inequality in Eq. (14.10) for prefix free codes c.

Hint Consider an infinite sequence of independent Bernoulli random variables

(Xn )∞
n=1 where Xn ∼ B(1/2). Viewing X as an infinite binary string, what is the
probability that X has a prefix that is a code for some symbol?
 14.5 Exercises 194

14.4 Let (Ω, F) be a measurable space, and let P, Q : F → [0, 1] be probability

measures. Let a < b and X : Ω → [a, b] be a F-measurable random variable.
Prove that
Z Z 
 
 X(ω)dP (ω) − X(ω)dQ(ω) ≤ (b − a)δ(P, Q) .

Ω Ω

14.5 (Entropy inequalities) Prove that each of the inequalities in Eq. (14.16)
is tight.

14.6 (Counting measure absolute continuity and derivatives) Let Ω be

P
a countable set and p : Ω → [0, 1] be a distribution on Ω so that ω∈Ω p(ω) = 1.
P
Let P be the measure associated with p, which means that P (A) = ω∈A p(ω).
Recall that the counting measure µ is the measure on (Ω, 2Ω ) given by µ(A) = |A|
if A is finite and µ(A) = ∞ otherwise.

(a) Show that P is absolutely continuous with respect to µ.

(b) Show that the Radon–Nykodim dP/dµ exists and that dP/dµ(ω) = p(ω).

14.7 (Relative entropy for Gaussian distributions) For each i ∈ {1, 2},
let µi ∈ R, σi2 > 0 and Pi = N (µi , σi2 ). Show that
   
1 σ22 σ12 (µ1 − µ2 )2
D(P1 , P2 ) = log + − 1 + .
2 σ12 σ22 2σ22

14.8 Let λ be the Lebesgue measure on (R, B(R)). Find

(a) a probability measure (R, B(R)) that is not absolutely continuous with
respect to λ; and
(b) a probability measure P on (R, B(R)) that is absolutely continuous to λ
with D(P, Q) = ∞ where Q = N (0, 1) is the standard Gaussian measure.

14.9 (Data processing inequality) Let P and Q be measures on (Ω, F), and
let G be a sub-σ-algebra of F and PG and QG be the restrictions of P and Q to
(Ω, G). Show that D(PG , QG ) ≤ D(P, Q).

14.10 Let (Ω, F) be a measurable space and P, Q : B(R) × Ω → [0, 1] be a pair

of probability kernels from (Ω, F) to (R, B(R)). Prove that

V = {ω ∈ Ω : D(P (· | ω), Q(· | ω)) = ∞} ∈ F .

Hint Apply Dobrushin’s theorem to the field of finite unions of rational-valued

intervals in R.
14.11 (Chain rule) Let P and Q be measures on (Rn , B(Rn )), and for t ∈ [n],
let Xt (x) = xt be the coordinate project from Rn → R. Then let Pt and Qt be
regular versions of Xt given X1 , . . . , Xt−1 under P and Q, respectively. Show
 14.5 Exercises 195

that

n
X
D(P, Q) = EP [D(Pt (· | X1 , . . . , Xt−1 ), Qt (· | X1 , . . . , Xt−1 ))] . (14.17)
t=1

Hint This is a rather technical exercise. You will likely need to apply a
monotone class argument [Kallenberg, 2002, theorem 1.1]. For the definition
of a regular version, see [Kallenberg, 2002, theorem 5.3] or Theorem 3.11.
Briefly, Pt is a probability kernel from (Rt−1 , B(Rt−1 )) to (R, B(R)) such that
Pt (A | x1 , . . . , xt−1 ) = P (Xt ∈ A | X1 , . . . , Xt−1 ) with P -probability one for all
A ∈ B(R).
14.12 (Chain rule (cont.)) Let P and Q be measures on (Rn , B(Rn )), and
for t ∈ [n], let Xt (x) = xt be the coordinate project from Rn → R. Then let Pt
and Qt be regular versions of Xt given X1 , . . . , Xt−1 under P and Q, respectively.
Let τ be a stopping time adapted to the filtration generated by X1 , . . . , Xn with
τ ∈ [n] almost surely. Show that

" τ #
X
D(P|Fτ , Q|Fτ ) = EP D(Pt (· | X1 , . . . , Xt−1 ), Qt (· | X1 , . . . , Xt−1 )) .
t=1

14.13 (Neyman-Pearson lemma) The simple hypothesis testing problem is

specified by a measurable space (X , F) and two distinct probability distributions
over this space, P and Q. The problem is to decide based on observing a random
element X taking values in X whether its distribution follows P or Q. The decision
rule, or test, is a measurable map f from X to (say) {P, Q}: when f (x) = P , the
rule decides in favor of P given the observation x, otherwise it decides in favor of
Q. There are two ways a test can be wrong and a test is characterized by the
probability of an incorrect decision in the two cases. In particular,
R the probability
of the test making an error when X follows P is e(f, P ) := R I {f (x) = Q} P (dx)
and the probability of error when X follows Q is e(f, Q) := I {f (x) = P } Q(dx).
Show that the following hold:

(a) If a test f is such that f (x) = P if p(x) > ηq(x) and f (x) = Q if p(x) < ηq(x)
and α = e(f, P ) then e(f, Q) = min{e(f 0 , Q) : e(f 0 , P ) ≤ α}. Here,
p = dP/dν and q = dQ/dν where ν is a common dominating measure
of P and Q.

(b) If f , as postulated in the previous part exist, and f 0 is a test such that
e(f 0 , Q) = e(f, Q) and e(f 0 , P ) ≤ α then e(f 0 , P ) = α. Furthermore, f 0 = f
holds except perhaps over the union of the set {x : p(x) = ηq(x)} and a set
that has zero measure under both P and Q.
 14.5 Exercises 196

In statistics, the upper bound α on e(f, P ) is called the significance level of

the test, while 1 − e(f, Q) is called its power, and breaking the symmetry,
e(f, P ) is called the type-I error, while e(f, Q) is called the type-II error.
Since the decision rule in the first part of the lemma can be expressed as a
function of the ratio p/q of densities (or likelihoods), the test in this part
is called a likelihood ratio test. The first part of the exercise says that the
most powerful test for a given significance level are the likelihood ratio
tests and the second part says that these are the only most powerful tests
(uniqueness).
15 Minimax Lower Bounds

After the short excursion into information theory, let us return to the world
of k-armed stochastic bandits. In what follows, we fix the horizon n > 0 and
the number of actions k > 1. This chapter has two components. The first is
an exact calculation of the relative entropy between measures in the canonical
bandit model for a fixed policy and different bandits. In the second component,
we prove a minimax lower bound that formalises the intuitive arguments given in
Chapter 13.

15.1 Relative Entropy Between Bandits

The following result will be used repeatedly. Some generalisations are provided
in the exercises.
Lemma 15.1 (Divergence decomposition). Let ν = (P1 , . . . , Pk ) be the reward
distributions associated with one k-armed bandit, and let ν 0 = (P10 , . . . , Pk0 ) be the
reward distributions associated with another k-armed bandit. Fix some policy π
and let Pν = Pνπ and Pν 0 = Pν 0 π be the probability measures on the canonical
bandit model (Section 4.6) induced by the n-round interconnection of π and ν
(respectively, π and ν 0 ). Then,
k
X
D(Pν , Pν 0 ) = Eν [Ti (n)] D(Pi , Pi0 ) . (15.1)
i=1

Proof Assume that D(Pi , Pi0 ) < ∞ for all i ∈ [k]. It follows that Pi  Pi0 . Define
Pk Pk
λ = i=1 Pi + Pi0 , which is the measure defined by λ(A) = i=1 (Pi (A) + Pi0 (A))
dPν
for any measurable set A. Theorem 14.1 shows that, as long as dP < +∞,
ν0
  
dPν
D(Pν , Pν 0 ) = Eν log .
dPν 0
Recalling that ρ is the counting measure over [k], we find that the Radon–Nikodym
derivative of Pν with respect to the product measure (ρ × λ)n is given in Eq. (4.7)
as
n
Y
pνπ (a1 , x1 , . . . , an , xn ) = πt (at | a1 , x1 , . . . , at−1 , xt−1 )pat (xt ) .
t=1
 15.2 Minimax Lower Bounds 198

The density of Pν 0 is identical except that pat is replaced by p0at . Then

pa (xt )
n X
dPν
log (a1 , x1 , . . . , an , xn ) = log 0 t ,
dPν 0 t=1
pat (xt )

where we used the chain rule for Radon–Nikodym derivatives and the fact that
the terms involving the policy cancel. Taking expectations of both sides,
  X  
pA (Xt )
n
dPν
Eν log (A1 , X1 , . . . , An , Xn ) = Eν log 0 t ,
dPν 0 t=1
pAt (Xt )

and
  " "  ##
pAt (Xt ) pAt (Xt )   
Eν log 0 = Eν Eν log 0  At = Eν D(PAt , PA0 t ) ,
pAt (Xt ) pAt (Xt ) 

where in the second equality we used that under Pν (·|At ), the distribution of Xt
is dPAt = pAt dλ. Plugging back into the previous display,
  X  
pAt (Xt )
n
dPν
Eν log (A1 , X1 , . . . , An , Xn ) = Eν log 0
dPν 0 t=1
pAt (Xt )
n k
" n
#
X   X X
= Eν D(PAt , PAt ) =0
Eν I {At = i} D(PAt , PAt )
0

t=1 i=1 t=1

k
X
= Eν [Ti (n)] D(Pi , Pi0 ) .
i=1

When the right-hand side of (15.1) is infinite, by our previous calculation, it is

not hard to see that the left-hand side will also be infinite.

We note in passing that the divergence decomposition holds regardless of

whether the action set is discrete or not. In its more general form, the sum
over the actions must be replaced by an integral with respect to an appropriate
non-negative measure, which generalises the expected number of pulls of arms.
For details, see Exercise 15.8.

15.2 Minimax Lower Bounds

Recall that EN
k
(1) is the class of Gaussian bandits with unit variance, which can
be parameterised by their mean vector µ ∈ Rk . Given µ ∈ Rk , let νµ be the
Gaussian bandit for which the ith arm has reward distribution N (µi , 1).

Theorem 15.2. Let k > 1 and n ≥ k − 1. Then, for any policy π, there exists a
mean vector µ ∈ [0, 1]k such that
1p
Rn (π, νµ ) ≥ (k − 1)n .
27
 15.2 Minimax Lower Bounds 199

Figure 15.1 The idea of the minimax lower bound. Given a policy and one environment,
the evil antagonist picks another environment so that the policy will suffer a large regret
in at least one environment.

Since νµ ∈ ENk
(1), it follows that the minimax regret for EN
k
(1) is lower-bounded
by the right-hand side of the above display as soon as n ≥ k − 1:
1p
Rn∗ (EN
k
(1)) ≥ (k − 1)n .
27
The idea of the proof is illustrated in Fig. 15.1.

Proof Fix a policy π. Let ∆ ∈ [0, 1/2] be some constant to be chosen later. As
suggested in Chapter 13, we start with a Gaussian bandit with unit variance
and mean vector µ = (∆, 0, 0, . . . , 0). This environment and π give rise to the
distribution Pνµ ,π on the canonical bandit model (Hn , Fn ). For brevity we will
use Pµ in place of Pνµ ,π , and expectations under Pµ will be denoted by Eµ . To
choose the second environment, let

i = argminj>1 Eµ [Tj (n)] .

Pk
Since j=1 Eµ [Tj (n)] = n, it holds that Eµ [Ti (n)] ≤ n/(k −1). The second bandit
is also Gaussian with unit variance and means

µ0 = (∆, 0, 0, . . . , 0, 2∆, 0, . . . , 0) ,

where specifically µ0i = 2∆. Therefore, µj = µ0j except at index i and the optimal
arm in νµ is the first arm, while in νµ0 arm i is optimal. We abbreviate Pµ0 = Pνµ0 ,π .
Lemma 4.5 and a simple calculation lead to
n∆ n∆
Rn (π, νµ ) ≥ Pµ (T1 (n) ≤ n/2) and Rn (π, νµ0 ) > Pµ0 (T1 (n) > n/2) .
2 2
Then, applying the Bretagnolle–Huber inequality from the previous chapter
 15.3 Notes 200

(Theorem 14.2),

n∆
Rn (π, νµ ) + Rn (π, νµ0 ) > (Pµ (T1 (n) ≤ n/2) + Pµ0 (T1 (n) > n/2))
2 (15.2)
n∆
≥ exp(− D(Pµ , Pµ0 )) .
4

It remains to upper-bound D(Pµ , Pµ0 ). For this, we use Lemma 15.1 and the
definitions of µ and µ0 to get

(2∆)2 2n∆2
D(Pµ , Pµ0 ) = Eµ [Ti (n)] D(N (0, 1), N (2∆, 1)) = Eµ [Ti (n)] ≤ .
2 k−1

Plugging this into the previous display, we find that

 
n∆ 2n∆2
Rn (π, νµ ) + Rn (π, νµ0 ) ≥ exp − .
4 k−1

p
The result is completed by choosing ∆ = (k − 1)/4n ≤ 1/2, where the inequality
follows from the assumptions in the theorem statement. The final steps are lower
bounding exp(−1/2) and using 2 max(a, b) ≥ a + b.

We encourage readers to go through the alternative proof outlined in

Exercise 15.2, which takes a slightly different path.

15.3 Notes

1 We used the Gaussian noise model because the KL divergences are so easily
calculated in this case, but all that we actually used was that D(Pi , Pi0 ) =
O((µi − µ0i )2 ) when the gap between the means ∆ = µi − µ0i is small. While
this is certainly not true for all distributions, it very often is. Why is that? Let
{Pµ : µ ∈ R} be some parametric family of distributions on Ω and assume that
distribution Pµ has mean µ. Assuming the densities are twice differentiable
and that everything is sufficiently nice that integrals and derivatives can be
exchanged (as is almost always the case), we can use a Taylor expansion about
 15.3 Notes 201

µ to show that
 
∂  1 ∂2 
D(Pµ , Pµ+∆ ) ≈ D(Pµ , Pµ+∆ ) ∆+ D(P , P )  ∆2
2 2 µ µ+∆ 
∂∆ ∆=0 ∂∆ ∆=0
Z   
∂ dPµ  1
= log dPµ  ∆ + I(µ)∆2
∂∆ Ω dPµ+∆ ∆=0 2
Z  
∂ dPµ+∆  1
=− log dPµ ∆ + I(µ)∆2
Ω ∂∆ dP µ

∆=0 2
Z 
∂ dPµ+∆  1
=− dPµ ∆ + I(µ)∆2
Ω ∂∆ dPµ

∆=0 2
Z 
∂ dPµ+∆  1
=− dPµ  ∆ + I(µ)∆2
∂∆ Ω dPµ ∆=0 2
Z 
∂  1
=− dPµ+∆  ∆ + I(µ)∆2
∂∆ Ω ∆=0 2
1
= I(µ)∆2 ,
2
where I(µ), introduced in the second line, is called the Fisher information
of the family (Pµ )µ at µ. Note that if λ is a common dominating measure for
(Pµ+∆ ) for ∆ small, dPµ+∆ = pµ+∆ dλ and we can write
Z 
∂2 
I(µ) = − 2
log pµ+∆  pµ dλ ,
∂∆ ∆=0

which is the form that is usually given in elementary texts. The upshot of all
this is that D(Pµ , Pµ+∆ ) for ∆ small is indeed quadratic in ∆, with the scaling
√
provided by I(µ), and as a result the worst-case regret is always O( nk),
provided the class of distributions considered is sufficiently rich and not too
bizarre.
√
2 We have now shown a lower bound that is Ω( nk), while many of the upper
bounds were O(log(n)). There is no contradiction because the logarithmic
bounds depended on the inverse suboptimality gaps, which may be very large.
3 Our lower bound was only proven for n ≥ k − 1. In Exercise 15.3, we ask you
to show that when n < k − 1, there exists a bandit such that
n(2k − n − 1) n
Rn ≥ > .
2k 2
4 The method used to prove Theorem 15.2 can be viewed as a generalisation
and strengthening of Le Cam’s method in statistics. Recall that Eq. (15.2)
establishes that for any µ and µ0 ,
n∆
inf sup Rn (π, ν) ≥ exp(− D(Pµ , Pµ0 )) .
π ν 8
To explain Le Cam’s method, we need a little notation. Let X be an outcome
space, P a set of measures on X and θ : P → Θ, where (Θ, d) is a metric space.
 15.4 Bibliographic Remarks 202

An estimator is a function θ̂ : X n → Θ. Le Cam’s method is used for proving

minimax lower bounds on the expected error of the estimator, which is
h i
inf sup EX1 ,...,Xn ∼P n d(θ̂(X1 , . . . , Xn ), θ(P )) . (15.3)
θ̂ P ∈P

The idea is to choose P0 , P1 ∈ P to maximise d(θ(P0 ), θ(P1 )) exp(−n D(P0 , P1 )),

on the basis that for any P0 , P1 ∈ P,
∆
Eq. (15.3) ≥ exp (−n D(P0 , P1 )) , (15.4)
8
where ∆ = d(θ(P0 ), θ(P1 )). There are two differences compared to the bandit
lower bound: (i) we deal with the sequential setting, and (ii) having chosen P0
we choose P1 in a way that depends on the algorithm. This provides a much
needed extra boost, without which the method would be unable to capture
how the characteristics of P are reflected in the minimax risk (or regret, in our
case).

15.4 Bibliographic Remarks

The first work on lower bounds that we know of was the remarkably precise
minimax analysis of two-armed Bernoulli bandits by Vogel [1960]. The Bretagnolle–
Huber inequality (Theorem 14.2) was first used for bandits by Bubeck et al.
[2013b]. As mentioned in the notes, the use of this inequality for proving lower
bounds is known as Le Cam’s method in statistics [Le Cam, 1973]. The proof
of Theorem 15.2 uses the same ideas as Gerchinovitz and Lattimore [2016],
while the alternative proof in Exercise 15.2 is essentially due to Auer et al.
[1995], who analysed the more difficult case where the rewards are Bernoulli (see
Exercise 15.4). Yu [1997] describes some alternatives to Le Cam’s method for the
passive, statistical setting. These alternatives can be (and often are) adapted to
the sequential setting.

15.5 Exercises

15.1 (Le Cam’s method) Establish the claim in Eq. (15.4).

15.2 (Alternative proof of Theorem 15.2) Here p you will prove

Theorem 15.2 with a different method. Let c > 0 and ∆ = 2c k/n, and for each
(i)
i ∈ {0, 1, . . . , k}, let µ(i) ∈ Rk satisfy µj = I {i = j} ∆. Further abbreviate the
notation in the proof of Theorem 15.2 by letting Ei [·] = Eµ(i) [·].
(a) Use Pinsker’s inequality (Eq. 14.12) and Lemma 15.1 and the result of
Exercise 14.4 to show
r
1 2 p
Ei [Ti (n)] ≤ E0 [Ti (n)] + n ∆ E0 [Ti (n)] = E0 [Ti (n)] + c nkE0 [Ti (n)] .
4
 15.5 Exercises 203

Pk
(b) Using the previous part, Jensen’s inequality and the identity i=1 E0 [Ti (n)] =
n, show that
k
X k p
X
Ei [Ti (n)] ≤ n + c nkE0 [Ti (n)] ≤ n + ckn .
i=1 i=1

(c) Let Ri = Rn (π, Gµ(i) ). Find a choice of c > 0 for which

k
X k
X
Ri = ∆ (n − Ei [Ti (n)]) ≥ ∆ (nk − n − ckn)
i=1 i=1
r r
k nk k
= 2c (nk − n − ckn) ≥
n 8 n
(d) Conclude that there exists an i ∈ [k] such that
1√
Ri ≥ kn .
8

The method used in this exercise is borrowed from Auer et al. [2002b] and
is closely related to the lower-bound technique known as Assouad’s method
in statistics [Yu, 1997].

15.3 (Lower bound for small horizons) Let k > 1 and n < k. Prove that
for any policy π there exists a Gaussian bandit with unit variance and means
µ ∈ [0, 1]k such that Rn (π, νµ ) ≥ n(2k − n − 1)/(2k) > n/2.

15.4 (Lower bounds for Bernoulli bandits) Recall from Table 4.1 that
EBk is the set of k-armed Bernoulli bandits. Show that there exists a universal
constant c > 0 such that for any 2 ≤ k ≤ n, it holds that:
√
Rn∗ (EBk ) = inf sup Rn (π, ν) ≥ c nk .
π ν∈E k
B

Hint Use the fact that KL divergence is upper bounded by the χ-squared
distance (Eq. (14.16)).
15.5 In Chapter 9 we proved that if π is the MOSS policy and ν ∈ ESG
k
(1), then
 
√ X
Rn (π, ν) ≤ C  kn + ∆i  ,
i:∆i >0

where C > 0 is a universal constant. Prove that the dependence on the sum
cannot be eliminated.

Hint You will have to use that Ti (t) is an integer for all t.
15.6 (Lower bound for explore-then-commit) Let ETCnm be the explore-
then-commit policy with inputs n and m respectively (Algorithm 1). Prove that
 15.5 Exercises 204

for all m, there exists a µ ∈ [0, 1]k such that

n o
Rn (ETCnm , νµ ) ≥ c min n, n2/3 k 1/3 ,

where c > 0 is a universal constant.

15.7 (Stopping-time version of divergence decomposition) Consider

the setting of Lemma 15.1, and let Ft = σ(A1 , X1 , . . . , At , Xt ) and τ be an
(Ft )-measurable stopping time. Then, for any random element X that is Fτ -
measurable,
k
X
D(PνX , Pν 0 X ) ≤ Eν [Ti (τ )] D(Pi , Pi0 ) ,
i=1

where PνX and Pν 0 X are the laws X under ν and ν 0 respectively.

15.8 (Divergence decomposition for more general action spaces) The

purpose of this exercise is to show that the divergence decomposition lemma
(Lemma 15.1) continues to hold for more general action spaces (A, G). Starting
from the set-up of Section 4.7, let Pν = Pνπ and Pν 0 = Pν 0 π be the measures on the
canonical bandit model induced by the interconnection of π and ν (respectively,
π and ν 0 ).
(a) Prove that
Z
D(Pν , Pν 0 ) = D(Pa , Pa0 ) dGν (a) , (15.5)
A
Pn
where Gν is a measure on (A, G) defined by Gν (B) = Eν [ t=1 I {At ∈ B}].
(b) Prove that
" n #
X
D(Pν , Pν 0 ) = E D(PAt , PAt ) .
0

t=1

Hint Use an appropriately adjusted form of the chain rule for relative entropy
from Exercise 14.11.
16 Instance-Dependent Lower Bounds

In the last chapter, we proved a lower bound on the minimax regret for subgaussian
bandits with suboptimality gaps in [0, 1]. Such bounds serve as a useful measure
of the robustness of a policy, but are often excessively conservative. This chapter
is devoted to understanding instance-dependent lower bounds, which try to
capture the optimal performance of a policy on a specific bandit instance.
Because the regret is a multi-objective criteria, an algorithm designer might
try and design algorithms that perform well on one kind of instance or another.
An extreme example is the policy that chooses At = 1 for all t, which suffers
zero regret when the first arm is optimal and linear regret otherwise. This is a
harsh trade-off, with the price for reducing the regret from logarithmic to zero
on just a few instances being linear regret on the remainder. Surprisingly, this
is the nature of the game in bandits. One can assign a measure of difficulty to
each instance such that policies performing overly well relative to this measure
on some instances pay a steep price on others. The situation is illustrated in
Fig. 16.1.

n over-specialised

reasonable, not instance optimal

instance optimal
Regret

√
n minimax optimal

Instances

Figure 16.1 On the x-axis, the instances are ordered according to the measure of difficulty,
and the y-axis shows the regret (on some scale). In the previous chapter, we proved
that no policy can be entirely below the horizontal ‘minimax optimal’ line. The results
in this chapter show that if the regret of a policy is below the ‘instance optimal’ line at
any point, then it must have regret above the shaded region for other instances. For
example, the ‘overly specified’ policy.
 16.1 Asymptotic Bounds 206

In finite time, the situation is a little messy, but if one pushes these ideas to
the limit, then for many classes of bandits one can define a precise notion of
instance-dependent optimality.

16.1 Asymptotic Bounds

We need to define exactly what is meant by a reasonable policy. If one is only

concerned with asymptotics, then a rather conservative definition suffices.

Definition 16.1. A policy π is called consistent over a class of bandits E if for

all ν ∈ E and p > 0, it holds that
Rn (π, ν)
lim = 0. (16.1)
n→∞ np
The class of consistent policies over E is denoted by Πcons (E).

Theorem 7.1 shows that UCB is consistent over ESG k

(1). The strategy that
always chooses the first action is not consistent on any class E unless the first
arm is optimal for every ν ∈ E.

Consistency is an asymptotic notion. A policy could be consistent and yet

play At = 1 for all t ≤ 10100 . For this reason, an assumption of consistency
is insufficient to derive non-asymptotic lower bounds. In Section 16.2, we
introduce a finite-time version of consistency that allows us to prove finite-
time instance-dependent lower bounds.

Recall that a class E of stochastic bandits is unstructured if E = M1 × · · · × Mk

with M1 , . . . , Mk sets of distributions. The main theorem of this chapter is a
generic lower bound that applies to any unstructured class of stochastic bandits.
After the proof, we will see some applications to specific classes. Let M be a set
of distributions with finite means, and let µ : M → R be the function that maps
P ∈ M to its mean. Let µ∗ ∈ R and P ∈ M have µ(P ) < µ∗ and define

dinf (P, µ∗ , M) = inf

0
{D(P, P 0 ) : µ(P 0 ) > µ∗ } .
P ∈M

Theorem 16.2. Let E = M1 × · · · × Mk and π ∈ Πcons (E) be a consistent policy

over E. Then, for all ν = (Pi )ki=1 ∈ E, it holds that
Rn X ∆i
lim inf ≥ c∗ (ν, E) = , (16.2)
n→∞ log(n) dinf (Pi , µ∗ , Mi )
i:∆i >0

where ∆i is the suboptimality gap of the ith arm in ν and µ∗ is the mean of the
optimal arm.

Proof Let µi be the mean of the ith arm in ν and di = dinf (Pi , µ∗ , Mi ). The
 16.1 Asymptotic Bounds 207

result will follow from Lemma 4.5, and by showing that for any suboptimal arm
i it holds that
Eνπ [Ti (n)] 1
lim inf ≥ .
n→∞ log(n) di
Fix a suboptimal arm i, and let ε > 0 be arbitrary and ν 0 = (Pj0 )kj=1 ∈ E be a
bandit with Pj0 = Pj for j 6= i and Pi0 ∈ Mi be such that D(Pi , Pi0 ) ≤ di + ε and
µ(Pi0 ) > µ∗ , which exists by the definition of di . Let µ0 ∈ Rk be the vector of means
of distributions of ν 0 . By Lemma 15.1, we have D(Pνπ , Pν 0 π ) ≤ Eνπ [Ti (n)](di + ε),
and by Theorem 14.2, for any event A,
1 1
Pνπ (A) + Pν 0 π (Ac ) ≥ exp (− D(Pνπ , Pν 0 π )) ≥ exp (−Eνπ [Ti (n)](di + ε)) .
2 2
Now choose A = {Ti (n) > n/2}, and let Rn = Rn (π, ν) and Rn0 = Rn (π, ν 0 ).
Then,
n
Rn + Rn0 ≥ (Pνπ (A)∆i + Pν 0 π (Ac )(µ0i − µ∗ ))
2
n
≥ min {∆i , µ0i − µ∗ } (Pνπ (A) + Pν 0 π (Ac ))
2
n
≥ min {∆i , µ0i − µ∗ } exp (−Eνπ [Ti (n)](di + ε)) .
4
Rearranging and taking the limit inferior leads to
 
n min{∆i ,µ0i −µ∗ }
log
Eνπ [Ti (n)] 1 4(Rn +Rn 0 )

lim inf ≥ lim inf

n→∞ log(n) di + ε n→∞ log(n)
 
1 log (Rn + Rn0 ) 1
= 1 − lim sup = ,
di + ε n→∞ log(n) di + ε
where the last equality follows from the definition of consistency, which says
that for any p > 0, there exists a constant Cp such that for sufficiently large n,
Rn + Rn0 ≤ Cp np , which implies that
log (Rn + Rn0 ) p log(n) + log(Cp )
lim sup ≤ lim sup = p,
n→∞ log(n) n→∞ log(n)
which gives the result since p > 0 was arbitrary and by taking the limit as ε
tends to zero.
Table 16.1 provides explicit formulas for dinf (P, µ∗ , M) for common choices of
M. The calculation of these quantities are all straightforward (Exercise 16.1).
The lower bound and definition of c∗ (ν, E) are quite fundamental quantities in
the sense that for most classes E, there exists a policy π for which
Rn (π, ν)
lim = c∗ (ν, E) for all ν ∈ E . (16.3)
n→∞ log(n)
This justifies calling a policy asymptotically optimal on class E if Eq. (16.3)
holds. For example, UCB from Chapter 8 and KL-UCB from Chapter 10 are
asymptotically optimal for EN k
(1) and EBk , respectively.
 16.2 Finite-Time Bounds 208

M P dinf (P, µ∗ , M)
(µ − µ∗ )2
{N (µ, σ 2 ) : µ ∈ R} N (µ, σ 2 )
2σ 2
 
1 (µ − µ∗ )2
{N (µ, σ 2 ) : µ ∈ R, σ 2 ∈ (0, ∞)} N (µ, σ 2 ) log 1+
2 σ2
   
µ 1−µ
{B(µ) : µ ∈ [0, 1]} B(µ) µ log + (1 − µ) log
µ∗ 1 − µ∗
 
2((a + b)/2 − µ∗ )2
{U(a, b) : a, b ∈ R} U(a, b) log 1+
b−a

Table 16.1 Expressions for dinf for different parametric families when the mean of P is less
than µ∗ .

16.2 Finite-Time Bounds

By making a finite-time analogue of consistency, it is possible to prove a finite-

time instance-dependent bound. First, a lemma that summarises what can be
obtained by chaining the Bretagnolle–Huber inequality (Theorem 14.2) with the
divergence decomposition lemma (Lemma 15.1).

Lemma 16.3. Let ν = (Pi ) and ν 0 = (Pi0 ) be k-armed stochastic bandits that
differ only in the distribution of the reward for action i ∈ [k]. Assume that i is
suboptimal in ν and uniquely optimal in ν 0 . Let λ = µi (ν 0 ) − µi (ν). Then, for
any policy π,
 
min{λ−∆i (ν),∆i (ν)}
log 4 + log(n) − log(Rn (ν) + Rn (ν 0 ))
Eνπ [Ti (n)] ≥ . (16.4)
D(Pi , Pi0 )

The lemma holds for finite n and any ν and can be used to derive finite-
time instance-dependent lower bounds for any environment class E that is rich
enough. The following result provides a finite-time instance-dependence bound
for Gaussian bandits where the asymptotic notion of consistency is replaced by
an assumption that the minimax regret is not too large. This assumption alone
is enough to show that no policy that is remotely close to minimax optimal can
be much better than UCB on any instance.

Theorem 16.4. Let ν ∈ EN k

be a k-armed Gaussian bandit with mean vector
µ ∈ Rk and suboptimality gaps ∆ ∈ [0, ∞)k . Let

E(ν) = {ν 0 ∈ EN
k
: µi (ν 0 ) ∈ [µi , µi + 2∆i ]} .

Suppose C > 0 and p ∈ (0, 1) are constants and π is a policy such that
 16.3 Notes 209

Rn (π, ν 0 ) ≤ Cnp for all n and ν 0 ∈ E(ν). Then, for any ε ∈ (0, 1],

!+
2 X (1 − p) log(n) + log ε∆i
8C
Rn (π, ν) ≥ . (16.5)
(1 + ε)2 ∆i
i:∆i >0

Proof Let i be suboptimal in ν, and choose ν 0 ∈ E(ν) such that µj (ν 0 ) = µj (ν)

for j 6= i and µj (ν 0 ) = µi + ∆i (1 + ε). Then, by Lemma 16.3 with λ = ∆i (1 + ε),
   
2 n  min {λ − ∆i , ∆i }
Eνπ [Ti (n)] ≥ log + log
∆2i (1 + ε)2 2Cnp 4
  
2 ε∆i
= 2 (1 − p) log (n) + log .
∆i (1 + ε)2 8C

Plugging this into the basic regret decomposition identity (Lemma 4.5) gives the
result.

When p = 1/2, the leading term in this lower bound is approximately half that
of the asymptotic bound. This effect may be real. The class of policies considered
is larger than in the asymptotic lower bound, and so there is the possibility that
the policy that is best tuned for a given environment achieves a smaller regret.

16.3 Notes

1 We mentioned that for most classes E there is a policy satisfying Eq. (16.3).
Its form is derived from the lower bound, and by making some additional
assumptions on the underlying distributions. For details, see the article
by Burnetas and Katehakis [1996], which is also the original source of
Theorem 16.2.
2 The analysis in this chapter only works for unstructured classes. Without this
assumption a policy can potentially learn about the reward from one arm
by playing other arms and this greatly reduces the regret. Lower bounds for
structured bandits are more delicate and will be covered on a case-by-case
basis in subsequent chapters.
3 The classes analysed in Table 16.1 are all parametric, which makes the
calculation possible analytically. There has been relatively little analysis
in the non-parametric case, but we know of three exceptions for which we
simply refer the reader to the appropriate source. The first is the class of
distributions with bounded support: M = {P : Supp(P ) ⊆ [0, 1]}, which has
been analysed exactly [Honda and Takemura, 2010]. The second is the class
of distributions with semi-bounded support, M = {P : Supp(P ) ⊆ (−∞, 1]}
[Honda and Takemura, 2015]. The third is the class of distributions with
bounded kurtosis, M = {P : KurtX∼P [X] ≤ κ} [Lattimore, 2017].
 16.4 Bibliographic Remarks 210

16.4 Bibliographic Remarks

Asymptotic optimality via a consistency assumption first appeared in the seminal

paper by Lai and Robbins [1985], which was later generalised by Burnetas and
Katehakis [1996]. In terms of upper bounds, there now exist policies that are
asymptotic optimal for single-parameter exponential families [Cappé et al., 2013].
Until recently, there were no results on asymptotic optimality for multi-parameter
classes of reward distributions. There has been some progress on this issue recently
for the Gaussian distribution with unknown mean and variance [Cowan et al.,
2018] and for the uniform distribution [Cowan and Katehakis, 2015]. There
are plenty of open questions related to asymptotically optimal strategies for
non-parametric classes of reward distributions. When the reward distributions
are discrete and finitely supported, an asymptotically optimal policy is given by
Burnetas and Katehakis [1996], though the precise constant is hard to interpret. A
relatively complete solution is available for classes with bounded support [Honda
and Takemura, 2010]. Already for the semi-bounded case, things are getting
murky [Honda and Takemura, 2015]. One of the authors thinks that classes with
bounded kurtosis are quite interesting, but here things are only understood up
to constant factors [Lattimore, 2017]. An asymptotic variant of Theorem 16.4 is
by Salomon et al. [2013]. Finite-time instance-dependent lower bounds have been
proposed by several authors, including Kulkarni and Lugosi [2000], for two arms,
and Garivier et al. [2019] and Lattimore [2018], for the general case. As noted
earlier, neither ETC policies, nor elimination-based algorithms are able to achieve
asymptotic optimality: as shown by Garivier et al. [2016b], these algorithms (no
matter how they are tuned) must incur an additional multiplicative penalty of
a factor of two on the standard Gaussian bandit problems as compared to the
optimal asymptotic regret.

16.5 Exercises

16.1 (Relative entropy calculations) Verify the calculations in Table 16.1.

16.2 (Rademacher noise) Let R(µ) be the shifted Rademacher distribution,

which for µ ∈ R and X ∼ R(µ) is characterised by P (X = µ + 1) =
P (X = µ − 1) = 1/2.
(a) Show that dinf (R(µ), µ∗ , M) = ∞ for any µ < µ∗ .
(b) Design a policy π for bandits with shifted Rademacher rewards such that
the regret is bounded by
k
X
Rn (π, ν) ≤ 3 ∆i for all n and ν ∈ Mk .
i=1

(c) The results from parts (a) and (b) seem to contradict the heuristic analysis
in Note 1 at the end of Chapter 15. Explain.
 16.5 Exercises 211

16.3 (Asymptotic lower bound for exponential families) Let M = {Pθ :

θ ∈ Θ} be an exponential family with sufficient statistic equal to the identity and
E = Mk and π be a consistent policy for E. Prove that the asymptotic upper
bound on the regret proven in Exercise 10.4 is tight.

16.4 (Unknown subgaussian constant) Let


M = P : there exists a σ 2 ≥ 0 such that P is σ 2 -subgaussian .
(a) Find a distribution P such that P ∈ / M.
(b) Suppose that P ∈ M has mean µ ∈ R. Prove that dinf (P, µ∗ , M) = 0 for all
µ∗ > µ.
(c) Let E = {(Pi ) : Pi ∈ M for all 1 ≤ i ≤ k}. Prove that if k > 1, then for all
consistent policies π,
Rn (π, ν)
lim inf =∞ for all ν ∈ E .
n→∞ log(n)
(d) Let f : N → [0, ∞) be any increasing function with limn→∞ f (n)/ log(n) =
∞. Prove there exists a policy π such that
Rn (π, ν)
lim sup =0 for all ν ∈ E ,
n→∞ f (n)
where E is as in the previous part.
(e) Conclude that there exists a consistent policy for E.

16.5 (Minimax lower bound) Use Lemma 16.3 to prove Theorem 15.2, possibly
with different constants.
2
16.6 (Refining the lower-order terms) Let k = 2, and for ν ∈ EN let
2
∆(ν) = max{∆1 (ν), ∆2 (ν)}. Suppose that π is a policy such that for all ν ∈ EN
with ∆(ν) ≤ 1, it holds that
C log(n)
Rn (π, ν) ≤ . (16.6)
∆(ν)
(a) Give an example of a policy satisfying Eq. (16.6).
(b) Assume that i = 2 is suboptimal for ν and that α ∈ (0, 1) be such that
1
Eνπ [T2 (n)] = 2∆(ν) 2 log(α). Let ν
0
be the alternative environment where
µ1 (ν 0 ) = µ1 (ν) and µ2 (ν 0 ) = µ1 (ν) + 2∆(ν). Show that
1
exp(− D(Pνπ , Pν 0 π )) =
.
α
(c) Let A be the event that T2 (n) ≥ n/2. Show that
2C log(n) 1 2C log(n)
Pνπ (A) ≤ and Pν 0 π (A) ≥ − .
n∆(ν)2 2α n∆(ν)2
(d) Show that
 
n∆(ν) 1 2C log(n)
Rn (π, ν ) ≥
0
− .
2 2α n∆(ν)2
 16.5 Exercises 212

n∆(ν)2
(e) Show that α ≥ 8C log(n) and conclude that
 
1 n∆(ν)2
Rn (π, ν) ≥ log .
2∆(ν) 8C log(n)

(f) Generalise the argument to an arbitrary number of arms.

In Exercise 7.6 you showed that there exists a bandit policy π such that for
some universal constant C > 0 and for any ν ∈ E[0,b] k
k-armed bandit with
rewards taking values in [0, b], the regret Rn (π, ν) of π on ν after n rounds
satisfies
X  
σi2
 
Rn (π, ν) ≤ C ∆i + b + log(n) ,
∆i
i:∆i >0

where ∆i = ∆i (ν) is the action gap of action i and σi2 = σi2 (ν) is the
variance of the reward of arm i. In particular, this is the inequality shown in
Eq. (7.14). The next exercise asks you to show that the appearance of both
σ2
b and ∆ii is necessary in this bound.

16.7 (Sharpness of Eq. (7.14)) Let k > 1, b > 0 and c > 0 be arbitrary. Show
that there is no policy π for which either
Rn (π, ν)
lim sup ≤ cb, k
∀ν ∈ E[0,b] (16.7)
n→∞ log(n)
or
Rn (π, ν) X σ 2 (ν)
lim sup ≤c i
, k
∀ν ∈ E[0,b] (16.8)
n→∞ log(n) ∆i (ν)
i:∆i >0

would hold true.

The intuition underlying this result is the following: Eq. (16.7) cannot hold
because this would mean that for some policy, the regret is logarithmic
with a constant independent of the gaps, while intuitively, if the variance
is constant, the coefficient of the logarithmic regret must increase as the
gaps get close. Similarly, Eq. (16.8) cannot hold either because we expect a
logarithmic regret with a coefficient proportional to the inverse gap even as
the variance gets zero, as the case of Bernoulli bandits shows. This exercise
is due to Audibert et al. [2007].

16.8 (Lower bound on regret variance) Let k > 1 and E ⊂ EN k

be the set
of k-armed Gaussian bandits with mean rewards in [0, 1] for all arms. Suppose
 16.5 Exercises 213

that π is a policy such that for all ν ∈ E,

Rn (π, ν) X 2(1 + p)
lim sup ≤ .
n→∞ log(n) ∆i
i:∆i >0

Prove that
log(V[R̂n (π, ν)])
lim sup sup ≥ 1,
n→∞ ν∈E (1 − p) log(n)
Pn
where R̂n (π, ν) = nµ∗ (ν) − t=1 µAt (ν).
17 High-Probability Lower Bounds

The lower bounds proven in the last two chapters were for stochastic bandits.
In this chapter, we prove high probability lower bounds for both stochastic and
adversarial bandits. Recall that for adversarial bandit x ∈ [0, 1]n×k , the random
regret is
n
X
R̂n = max xti − xtAt
i∈[k]
t=1

and the (expected) regret is Rn = E[R̂n ]. To set expectations, remember that in

Chapter 12 we proved two high-probability upper bounds on the regret of Exp3-
IX. In the first, we showed there exists a policy π such that for all adversarial
bandits x ∈ [0, 1]n×k and δ ∈ (0, 1), it holds with probability at least 1 − δ that
s  !
p kn 1
R̂n = O kn log(k) + log . (17.1)
log(k) δ

We also gave a version of the algorithm that depended on δ ∈ (0, 1) for which
with probability at least 1 − δ,
s  !
k
R̂n = O kn log . (17.2)
δ

The important difference is the order of quantifiers. In the first, we have a

single algorithm and a high-probability guarantee that holds simultaneously for
any confidence level. The second algorithm needs the confidence level to be
specified
p in advance. The price for using the generic algorithm appears to be
log(1/δ)/ log(k), which is usually quite small but not totally insignificant. We
will see that both bounds are tight up to constant factors, which implies that
knowing the desired confidence level in advance really does help. One reason
why choosing the confidence level in advance is not ideal is that the resulting
high-probability bound cannot be integrated to prove a bound in expectation.
For algorithms satisfying (17.1), the expected regret can be bounded by
Z ∞   p
Rn ≤ P R̂n ≥ u du = O( kn log(k)) . (17.3)
0
 17.1 Stochastic Bandits 215

On the other hand, if the high-probability bound only holds for a single δ, as in
(17.2), then it seems hard to do much better than
s  !
k
Rn ≤ nδ + O kn log ,
δ
p
which with the best choice of δ leads to a bound of O( kn log(n)).

17.1 Stochastic Bandits

For simplicity, we start with the stochastic setting before explaining how to
convert the arguments to the adversarial model. There is no randomness in the
expected regret, so in order to derive a high-probability bound, we define the
random pseudo-regret by
k
X
R̄n = Ti (n)∆i ,
i=1

which is a random variable through the pull counts Ti (n).

For all results in this section, we let E k ⊂ EN

k
denote the set of k-armed
Gaussian bandits with suboptimality gaps bounded by one. For µ ∈ [0, 1]d
we let νµ ∈ E k be the Gaussian bandit with means µ.

Theorem 17.1. Let n ≥ 1 and k ≥ 2 and B > 0 and π be a policy such that for
any ν ∈ E k ,
p
Rn (π, ν) ≤ B (k − 1)n . (17.4)

Let δ ∈ (0, 1). Then there exists a bandit ν in E k such that

   
1 1p 1
P R̄n (π, ν) ≥ min n, (k − 1)n log ≥ δ.
4 B 4δ
Proof Let ∆ ∈ (0, 1/2] be a constant to be tuned subsequently and ν = νµ where
the mean vector µ ∈ Rd is defined by µ1 = ∆ and µi = 0 for i > 1. Abbreviate
Rn = Rn (π, ν) and P = Pνπ and E = Eνπ . Let i = argmini>1 E[Ti (n)]. Then, by
Lemma 4.5 and the assumption in Eq. (17.4),
r
Rn B n
E[Ti (n)] ≤ ≤ . (17.5)
∆(k − 1) ∆ k−1

Define alternative bandit ν 0 = νµ0 where µ0 ∈ Rd is equal to µ except µ0i = µi +2∆.

Abbreviate P0 = Pν 0 π and R̄n = R̄n (π, ν) and R̄n0 = R̄n (π, ν 0 ). By Lemma 4.5, the
 17.1 Stochastic Bandits 216

Bretagnolle–Huber inequality (Theorem 14.2) and the divergence decomposition

(Lemma 15.1), we have
    
∆n ∆n n  n
P R̄n ≥ + P R̄n ≥
0 0
≥ P Ti (n) ≥ + P0 Ti (n) <
2 2 2 2
 r 
1 1 n
≥ exp (− D(P, P0 )) ≥ exp −2B∆ ≥ 2δ ,
2 2 k−1
where the last line follows by choosing
( r  )
1 1 k−1 1
∆ = min , log .
2 2B n 4δ

The result follows since max{a, b} ≥ (a + b)/2.

Corollary 17.2. Let n ≥ 1 and k ≥ 2. Then, for any policy π and δ ∈ (0, 1)
such that
s  
1
nδ ≤ n(k − 1) log , (17.6)
4δ

there exists a bandit problem ν ∈ E k such that

( s  )!
1 n(k − 1) 1
P R̄n (π, ν) ≥ min n, log ≥ δ. (17.7)
4 2 4δ

Proof We prove the result by contradiction. Assume that the conclusion does
not hold for π and let δ ∈ (0, 1) satisfy (17.6). Then, for any bandit problem
ν ∈ E k , the expected regret of π is bounded by
s   s  
n(k − 1) 1 1
Rn (π, ν) ≤ nδ + log ≤ 2n(k − 1) log .
2 4δ 4δ
p
Therefore, π satisfies the conditions of Theorem 17.1 with B = 2 log(1/(4δ)),
which implies that there exists some bandit problem ν ∈ E k such that (17.7)
holds, contradicting our assumption.

Corollary 17.3. Let k ≥ 2 and p ∈ (0, 1) and B > 0. Then, there does not
exist a policy π such that for all n ≥ 1, δ ∈ (0, 1) and ν ∈ E k ,
  
p 1
P R̄n (π, ν) ≥ B (k − 1)n logp < δ.
δ
Proof We proceed by contradiction. Suppose that such a policy exists. Choosing
δ sufficiently small and n sufficiently large ensures that
     
1 1 1 1p 1
log ≥ B log p
and n(k − 1) log ≤ n.
B 4δ δ B 4δ
 17.2 Adversarial Bandits 217

Now, by assumption, for any ν ∈ E k we have

Z ∞


Rn (π, ν) ≤ P R̄n (π, ν) ≥ x dx
0
p Z ∞   p
≤ B n(k − 1) exp −x1/p dx ≤ B n(k − 1) .
0

Therefore, by the Theorem 17.1, there exists a bandit ν ∈ E k such that

  
p 1
P R̄n (π, ν) ≥ B n(k − 1) log
δ
   
1 1 p 1
≥ P R̄n (π, ν) ≥ min n, n(k − 1) log ≥ δ,
4 B 4δ
which contradicts our assumption and completes the proof.

We suspect there exists a policy π and universal constant B > 0 such that
for all ν ∈ E k ,
  
√ 1
P R̄n (π, ν) ≥ B kn log ≤ δ.
δ

17.2 Adversarial Bandits

We now explain how to translate the ideas in the previous section to the adversarial
model. Let π = (πt )nt=1 be a fixed policy, and recall that for x ∈ [0, 1]n×k , the
random regret is
n
X
R̂n = max (xti − xtAt ) .
i∈[k]
t=1

Let Fx be the cumulative distribution function of the law of R̂n when policy π
interacts with the adversarial bandit x ∈ [0, 1]n×k .

Theorem 17.4. Let c, C > 0 be sufficiently small/large universal constants and

k ≥ 2, n ≥ 1 and δ ∈ (0, 1) be such that n ≥ Ck log(1/(2δ)). Then there exists a
reward sequence x ∈ [0, 1]n×k such that
s  !
1
1 − Fx c nk log ≥ δ.
2δ

The proof is a bit messy, but is not completely without interest. For the sake of
brevity, we explain only the high-level ideas and refer you elsewhere for the gory
details. There are two difficulties in translating the arguments in the previous
section to the adversarial model. First, in the adversarial model, we need the
 17.2 Adversarial Bandits 218

rewards to be bounded in [0, 1]. The second difficulty is we now analyse the
adversarial regret rather than the random pseudo-regret. Given a measure Q, let
X ∈ [0, 1]n×k and (At )nt=1 be a collection of random variables on a probability
space (Ω, F, PQ ) such that
(a) PQ (X ∈ B) = Q(B) for all B ∈ B([0, 1]n×k ); and
(b) PQ (At | A1 , X1 , . . . , At−1 , Xt−1 ) = πt (At | A1 , X1 , . . . , At−1 , Xt−1 ) almost
surely, where Xs = XtAs .
Then the regret is a random variable R̂n : Ω → R defined by
n
X
R̂n = max (Xti − XtAt ) .
i∈[k]
t=1

Suppose we sample X ∈ [0, 1]n×k from distribution Q on ([0, 1]n×k , B([0, 1]k )).
Claim 17.5. Suppose that X ∼ Q, where Q is a measure on [0, 1]n×k with the
Borel σ-algebra and that EQ [1 − FX (u)] ≥ δ. Then there exists an x ∈ [0, 1]n×k
such that 1 − Fx (u) ≥ δ.
The next step is to choose Q and argue that EQ [1 − FX (u)] ≥ δ for sufficiently
large u. To do this, we need a truncated normal distribution. Defining clipping
function

1 if x > 1


clip[0,1] (x) = 0 if x < 0


x otherwise .

Let σ and ∆ be positive constants to be chosen later and (ηt )nt=1 a sequence of
independent random variables with ηt ∼ N (1/2, σ 2 ). For each i ∈ [k], let Qi be
the distribution of X ∈ [0, 1]n×k , where

clip[0,1] (ηt + ∆) if j = 1


Xtj = clip[0,1] (ηt + 2∆) if j = i and i 6= 1


clip (η ) otherwise .
[0,1] t

Notice that under any Qi for fixed t, the random variables Xt1 , . . . , Xtk are not
independent, but for fixed j, the random variables X1j , . . . , Xnj are independent
and identically distributed. Let PQi be the law of X1 , A1 , . . . , An , Xn when policy
π interacts with adversarial bandit sampled from X ∼ Qi .
q

1
Claim 17.6. If σ > 0 and ∆ = σ k−1 2n log 8δ , then there exists an arm i such
that
PQi (Ti (n) < n/2) ≥ 2δ .
The proof of this claim follows along the same lines as the theorems in the
previous section. All that changes is the calculation of the relative entropy. The
last step is to relate Ti (n) to the random regret. In the stochastic model, this was
 17.3 Notes 219

straightforward, but for adversarial bandits there is an additional step. Notice

that under Qi , it holds that Xti − XtAt ≥ 0 and that if Xti , XtAt ∈ (0, 1) and
At 6= i, then Xti − XtAt ≥ ∆. From this we conclude that
n
!
X
R̂n ≥ ∆ n − Ti (n) − I {exists j ∈ [k] : Xtj ∈ {0, 1}} . (17.8)
t=1

The following claim upper-bounds the number of rounds in which clipping occurs
with high probability.

Claim 17.7. If σ = 1/10 and ∆ < 1/8 and n ≥ 32 log(1/δ), then

n
!
X n
PQi I {exists j ∈ [k] : Xtj ∈ {0, 1}} ≥ ≤ δ.
t=1
4

Combining Claim 17.6 and Claim 17.7 with Eq. (17.8) shows there exists an
arm i such that
 
n∆
PQi R̂n ≥ ≥ δ,
4
which by the definition of ∆ and Claim 17.5 implies Theorem 17.4.

17.3 Notes

1 The adversarial bandits used in Section 17.2 had the interesting property that
the same arm has the best reward in every round (not just the best mean).
This cannot be exploited by an algorithm, however, because it only gets a
single observation in each round.
2 In Theorem 17.4, we did not make any assumptions on the algorithm. If √ we
had assumed the algorithm enjoyed an expected regret bound of Rn ≤ B kn,
then we could conclude that for each sufficiently small δ ∈ (0, 1) there exists
an adversarial bandit such that
  
c√ 1
P R̂n ≥ kn log ≥ δ,
B 2δ
which shows that our high-probability upper bounds for Exp3-IX are nearly
tight.

17.4 Bibliographic Remarks

The results in this chapter are by Gerchinovitz and Lattimore [2016], who also
provide lower bounds on what is achievable when the loss matrix exhibits nice
structure such as low variance or similarity between losses of the arms.
 17.5 Exercises 220

17.5 Exercises

17.1 Prove each of the claims in Section 17.2.

Part V
Contextual and Linear
Bandits
 222

The algorithms introduced so far work well in stationary environments with

only a few actions. Real-world problems are seldom this simple. For example,
a bandit algorithm designed for targeted advertising may have thousands of
actions. Even more troubling, the algorithm has access to contextual information
about the user and the advertisement. Ignoring this information would make the
problem highly non-stationary, but algorithms introduced in the previous chapter
cannot make use of side information.
Large action sets are usually dealt with by introducing structure that allows the
algorithm to generalise from one action to another. For example, advertisements
can usually be associated with features describing their topic. Then the reward
can be described as a function of the features, usually assumed to be nice in some
way (linear or smooth, for example). Contextual information is dealt with in a
similar fashion by assuming the mean reward of an action is a function of the
context features and action features. As we explain, this leads to a model where
the action set is essentially changing in each round.
Of course the world is messy in other ways. Rewards are often delayed, and
may be unattributed, or the world may be non-stationary. The first of these
issues is discussed briefly in the introduction to Part VII while non-stationarity
is the subjection of Chapter 31.
Except for the first chapter, which is generic, the focus of this part will be on
the special case that the expected reward of each arm is a linear function of some
feature vector in a way that will be made precise in Chapter 19. Along the way,
we will discuss many generalisations and give references to the literature. One
aspect that will play a far larger role is computation. While finite-armed bandits
with few arms present no computation difficulties, when the number of actions
is very large or the information structure of the feedback model is not so easily
separable, then computation can be a serious challenge.
18 Contextual Bandits

In many bandit problems, the learner has access to additional information

that may help predict the quality of the actions. Imagine designing a movie
recommendation system where users sequentially request recommendations for
which movie to watch next. It would be inadvisable to ignore demographic
information about the user making the request, or other contextual history such
as previously watched movies or ratings. None of the algorithms presented so
far make use of this kind of additional information. Indeed, they optimise a
benchmark (the regret) that also disregards such contextual data. Essentially
they would try to identify the best single movie in hindsight. In this chapter,
we present an augmented framework and regret definition that better models
real-world problems where contextual information is available.

Whenever you design a new benchmark, there are several factors to consider.
Competing with a poor benchmark does not make sense, since even an
algorithm that perfectly matches the benchmark will perform poorly. At
the same time, competing with a better benchmark can be harder from a
learning perspective, and this penalty must be offset against the benefits.

The trade-off just described is fundamental to all machine learning problems.

In statistical estimation, the analoguous trade-off is known as the bias-variance
trade-off. We will not attempt to answer the question of how to resolve this trade-
off in this chapter because first we need to see how to effectively compete with
improved benchmarks. The good news is that many of the techniques developed
earlier are easily generalised.

18.1 Contextual Bandits: One Bandit per Context

While contextual bandits can be studied in both the adversarial and stochastic
frameworks, in this chapter we focus on the k-armed adversarial model. As usual,
the adversary secretly chooses (xt )nt=1 , where xt ∈ [0, 1]k with xti the reward
associated with arm i in round t. The adversary also secretly chooses a sequence
of contexts (ct )nt=1 , where ct ∈ C with C a set of possible contexts. In each round,
 18.1 Contextual Bandits: One Bandit per Context 224

Adversary secretly chooses rewards (xt )n

t=1 with xt ∈ [0, 1]
k

Adversary secretly chooses contexts (ct )n

t=1 with ct ∈ C

For rounds t = 1, 2, . . . , n:
Learner observes context ct ∈ C where C is an arbitrary fixed set of contexts.
Learner selects distribution Pt ∈ Pk−1 and samples At from Pt .
Learner observes reward Xt = xtAt .

Figure 18.1 Interaction protocol for k-armed contextual bandits.

the learner observes ct , chooses an action At and receives reward xtAt . The
interaction protocol is shown in Fig. 18.1.
A natural way to define the regret is to compare the rewards collected by
the learner with the rewards collected by the best context-dependent policy in
hindsight:
 
X X
Rn = E  max (xti − Xt ) . (18.1)
i∈[k]
c∈C t∈[n]:ct =c

If the set of possible contexts is finite, then a simple approach is to use a separate
instance of Exp3 for each context. Let
 
X
Rnc = E max (xti − Xt )
i∈[k]
t∈[n]:ct =c

be the regret due to context c ∈ C. When using a separate instance of Exp3 for
each context, we can use the results of Chapter 11 to bound
v
u n
u X
Rnc ≤ 2tk I {ct = c} log(k) , (18.2)
t=1

where the sum inside the square root counts the number of times context c ∈ C is
observed. Because this is not known in advance, it is important to use an anytime
version of Exp3 for which the above regret bound holds without needing to tune
a learning rate that depends on the number of times the context is observed (see
Exercise 28.13). Substituting (18.2) into the regret leads to
v
u n
X Xu X
Rn = Rnc ≤ 2 tk log(k) I {ct = c} . (18.3)
c∈C c∈C t=1

The magnitude of the right-hand side depends on the distribution of observed

contexts. On one extreme, there is only one observed context, and the bound is
the same as the standard finite-armed bandit problem. The other extreme occurs
 18.2 Bandits with Expert Advice 225

when all contexts are observed equally often, in which case we have
p
Rn ≤ 2 nk|C| log(k) . (18.4)
Jensen’s inequality applied to Eq. (18.3) shows that this really is the worst case
(Exercise 18.1).

The regret in Eq. (18.4) is different than the regret studied in Chapter 11. If
we ignore the context and run the standard Exp3 algorithm, then we would
have
" n # n
X X p
E Xt ≥ max xti − 2 kn log(k) .
i∈[k]
t=1 t=1

Using one version of Exp3 per context leads to

" n #
X X X p
E Xt ≥ max xti − 2 kn|C| log(k) .
i∈[k]
t=1 c∈C t∈[n]:ct =c

Which of these bounds is preferable depends on the magnitude of n and

how useful the context is. When n is very large, the second bound is more
likely to be preferable. On the other hand, the second bound is completely
vacuous when n ≤ 4k|C| log(k).

18.2 Bandits with Expert Advice

When the context set C is large, using one bandit algorithm per context will
almost always be a poor choice because the additional precision is wasted unless
the amount of data is enormous. Fortunately, however, it is seldom the case that
the context set is both large and unstructured. To illustrate a common situation,
we return to the movie recommendation theme, where the actions are movies
and the context contains user information such as age, gender and recent movie
preferences. In this case, the context space is combinatorially large, but there
is a lot of structure inherited from the fact that the space of movies is highly
structured and users with similar demographics are more likely to have similar
preferences.
We start by rewriting Eq. (18.1) in an equivalent form. Let Φ be the set of all
functions from C → [k]. Then,
" n
#
X
Rn = E max (xtφ(ct ) − Xt ) . (18.5)
φ∈Φ
t=1

The discussion above suggests that a slightly smaller set Φ may lead to more
reward. In what follows, we describe some of the most common ideas of how to
do this.
 18.2 Bandits with Expert Advice 226

Figure 18.2 Prediction with expert advice. The experts, upon seeing a foot give expert
advice on what socks should fit it best. If the owner of the foot is happy, the
recommendation system earns a cookie!

Partitions
Let P ⊂ 2C be a partition of C, which means that sets (or parts) in P are disjoint
and ∪P ∈P P = C. Then define Φ to be the set of functions from C to [k] that are
constant on each part in P. In this case, we can run a version of Exp3 for each
part, which means the regret depends on the number of parts |P| rather than on
the number of contexts.

Similarity Functions
Let s : C × C → [0, 1] be a function measuring the similarity between pairs of
contexts on the [0, 1]-scale. Then let Φ be the set of functions φ : C → [k] such
that the average dissimilarity
1 X
(1 − s(c, d))I {φ(c) 6= φ(d)}
|C|2
c,d∈C

is below a user-tuned threshold θ ∈ (0, 1). It is not clear anymore that we can
control the regret (18.5) using some simple meta-algorithm on Exp3, but keeping
the regret small is still a meaningful objective.

From Supervised Learning to Bandits with Expert Advice

Yet another option is to run your favorite supervised learning method, training
on batch data to find a collection of predictors φ1 , . . . , φM : C → [k]. Then we
could use a bandit algorithm to compete with the best of these in an online
fashion. This has the advantage that the offline training procedure can use the
power of batch data and the whole army of supervised learning, without relying
on potentially inaccurate evaluation methods that aim to pick the best of the
pack. And why pick if one does not need to?
The possibilities are endless, but ultimately we always end up with a set of
functions Φ with the goal of competing with the best of them. This suggests we
 18.2 Bandits with Expert Advice 227

Adversary secretly chooses rewards x ∈ [0, 1]n×k

Experts secretly choose predictions E (1) , . . . , E (n)
For rounds t = 1, 2, . . . , n:
Learner observes predictions of all experts, E (t) ∈ [0, 1]M ×k .
Learner selects a distribution Pt ∈ Pk−1 .
Action At is sampled from Pt and the reward is Xt = xtAt .

Figure 18.3 Interaction protocol for bandits with expert advice.

should think more generally about some subset Φ of functions without considering
the internal structure of Φ. In fact, once Φ has been chosen, the contexts play
very little role. All we need in each round is the output of each function.

18.2.1 Bandits with Expert Advice Framework

The bandits with expert advice setting is a k-armed adversarial bandit, but
with M experts making recommendations to the learner. At the beginning of each
round, the experts announce their predictions about which actions are the most
promising. For the sake of generality, the experts report a probability distribution
over the actions. The interpretation is that the expert, if the decision were left
to them, would choose the action for the round at random from the probability
distribution it reported. As discussed before, in an adversarial setting it is natural
to consider randomised algorithms, hence one should not be too surprised that the
experts are also allowed to randomise. An application to an important practical
problem is illustrated in Fig. 18.2.
The predictions of the M experts in round t are represented by a matrix
(t)
E (t) ∈ [0, 1]M ×k , where the mth row Em is a probability vector over [m]
(t)
representing the recommendations of expert m in round t. Since Em is a row
(t)
vector, for a k-dimensional vector x, the expression Em xt is well defined. The
learner and the environment interact according to the protocol in Fig. 18.3.
The regret measures the cumulative rewards collected by the learner relative
to the best expert in hindsight:
" n n
#
X X
(t)
Rn = E max Em xt − Xt . (18.6)
m∈[M ]
t=1 t=1

This framework assumes the experts are oblivious in the sense that their
predictions do not depend on the actions of the learner.
 18.3 Exp4 228

18.3 Exp4

The number 4 in Exp4 is not just an increased version number, but indicates
the four e’s in the long name of the algorithm, which is exponential weighting
for exploration and exploitation with experts. The idea of the algorithm is
very simple. Since exponential weighting worked so well in the standard bandit
problem, we aim to adopt it to the problem at hand. However, since the goal is to
compete with the best expert in hindsight, it is not the actions that we will score,
but the experts. Exp4 thus maintains a probability distribution Qt over experts
and uses this to come up with the next action in the obvious way, by first choosing
an expert Mt at random from Qt and then following the chosen expert’s advice
(t)
to choose At ∼ EMt . The reader is invited to check for themself that this is the
same as sampling At from Pt = Qt E (t) where Qt is treated as a row vector. Once
the action is chosen, one can use their favorite reward estimation procedure to
estimate the rewards for all the actions, which is then used to estimate how much
total reward the individual experts would have made so far. The reward estimates
are then used to update Qt using exponential weighting. The pseudocode of Exp4
is given in Algorithm 11.

1: Input: n, k, M , η, γ
2: Set Q1 = (1/M, . . . , 1/M ) ∈ [0, 1]1×M (a row vector)
3: for t = 1, . . . , n do
4: Receive advice E (t)
5: Choose the action At ∼ Pt , where Pt = Qt E (t)
6: Receive the reward Xt = xtAt
t =i}
7: Estimate the action rewards: X̂ti = 1 − I{A Pti +γ (1 − Xt )
8: Propagate the rewards to the experts: X̃t = E (t) X̂t
9: Update the distribution Qt using exponential weighting:
exp(η X̃ti )Qti
Qt+1,i = P for all i ∈ [M ]
j exp(η X̃tj )Qtj

10: end for

Algorithm 11: Exp4.

The algorithm uses O(M ) memory and O(M + k) computation per round
(when sampling in two steps). Hence it is only practical when both M and k are
reasonably small.

18.4 Regret Analysis

We restrict our attention to the case when γ = 0, which is the original algorithm.
The version where γ > 0 is called Exp4-IX and its analysis is left for Exercise 18.3.
 18.4 Regret Analysis 229

p
Theorem 18.1. Let γ = 0 and η = 2 log(M )/(nk), and denote by Rn the
expected regret of Exp4 defined in Algorithm 11 after n rounds. Then,
p
Rn ≤ 2nk log(M ) . (18.7)
After translating the notation, the proof of the following lemma can be extracted
from the analysis of Exp3 in the proof of Theorem 11.2 (Exercise 18.2).
Lemma 18.2. For any m∗ ∈ [M ], it holds that

log(M ) η X X
n
X n X
X M n M
X̃ − Qtm X̃tm ≤ + Qtm (1 − X̂tm )2 .
2 t=1 m=1
tm∗
t=1 t=1 m=1
η

Proof of Theorem 18.1 Let Ft = σ(E (1) , A1 , E (2) , A2 , . . . , At−1 , E (t) ) and
abbreviate Et [·] = E[ · | Ft ]. Let m∗ be the index of the best-performing expert in
hindsight:
n
X
(t)
m = argmaxm∈[M ]
∗
Em xt , (18.8)
t=1

which is not random by the assumption that the experts are oblivious. Applying
Lemma 18.2 shows that
log(M ) η X X
n
X n X
X M M M
X̃tm∗ − Qtm X̃tm ≤ + Qtm (1 − X̃tm )2 . (18.9)
t=1 t=1 m=1
η 2 t=1 m=1

When γ = 0 the estimator X̂ti is unbiased so that Et [X̂t ] = xt and

Et [X̃t ] = Et [E (t) X̂t ] = E (t) Et [X̂t ] = E (t) xt . (18.10)
Taking the expectation of both sides of Eq. (18.9) and using the tower rule for
conditional expectation and the fact that Qt is Ft -measurable leads to

log(M ) η X X 
n M

Rn ≤ + E Qtm (1 − X̃tm )2 . (18.11)
η 2 t=1 m=1

Like in Chapter 11, it is more convenient to work with losses. Let Ŷti = 1 − X̂ti ,
yti = 1 − xti and Ỹtm = 1 − X̃tm . Note that Ỹt = E (t) Ŷt and recall the notation
Ati = I {At = i}, which means that Ŷti = APtitiyti and
 !2   2
(t) (t)
Xk Emi yti Xk (t)
2
E ytA Emi
Et [Ỹtm ] = Et  mAt = (18.12)
t
≤ .
PtAt i=1
Pti i=1
Pti

Therefore, using the definition of Pti ,

" M # " M #
X X Xk (t)
2 Emi
E Qtm (1 − X̃tm ) ≤ E Qtm
m=1 m=1 i=1
Pti
" k P #
X M Qtm E (t)
=E m=1 mi
= k.
i=1
Pti
 18.5 Notes 230

Substituting into Eq. (18.11) leads to

log(M ) ηnk p
Rn ≤ + = 2nk log(M ) .
η 2
Let us see how this theorem can be applied to the contextual bandit where C is a
finite set and Φ is the set of all functions from C → [k]. To each of these functions
(t)
φ ∈ Φ, we associate an expert m with Emi = I {φ(ct ) = i}. Then M = k C , and
Theorem 18.1 says that
p
Rn ≤ 2nk|C| log(k) ,
which is the same bound we derived using an independent copy of Exp3 for each
context. More generally, if C is arbitrary (possibly infinite) and Φ is a finite set
of functions from C to [k], then the theorem ensures that
p
Rn ≤ 2nk log(|Φ|) .
These results seem quite promising already, but in fact there is another
improvement possible. The intuition is that learning should be easier if the
experts have a high degree of agreement. One way to measure this is by
t X
X k
(s)
Et∗ = max Emi .
m∈[M ]
s=1 i=1

In Exercise 18.7, you will show that if all experts make identical recommendations,
then Et∗ = t and that no matter how the experts behave,
En∗ ≤ n min(k, M ) . (18.13)
In this sense En∗ /n can be viewed as the effective number of experts, which
depends on the degree of disagreement in the expert’s recommendations. By
modifying the algorithm to use a time varying learning rate, one can prove the
following theorem.
Theorem
p 18.3. Assume the same conditions as in Theorem 18.1, except let
ηt = log(M )/Et∗ . Then there exists a universal constant C > 0 such that
p
Rn ≤ C En∗ log(M ) . (18.14)
The proof of Theorem 18.3 is not hard and is left to Exercise 18.4. The bound
tells us that Exp4 with the suggested learning rate is able to adapt to degree of
disagreement between the experts, which seems like quite an encouraging result.
As a further benefit, the learning rate does not depend on the horizon so the
algorithm is anytime.

18.5 Notes

1 The most important concept in this chapter is that there are trade-offs when
choosing the competitor class. A large class leads to a more meaningful definition
 18.5 Notes 231

of the regret, but also increases the regret. This is similar to what we have
observed in stochastic bandits. Tuning an algorithm for a restricted environment
class usually allows faster learning, but the resulting algorithms can fail when
interacting with an environment that does not belong to the restricted class.
2 The Exp4 algorithm serves as a tremendous building block for other bandit
problems by defining your own experts. An example is the application of Exp4
to non-stationary bandits that we explore in Chapter 31, which is one of the
rare cases where Exp4 can be computed efficiently with a combinatorially large
number of experts. When Exp4 does not have an efficient implementation, it
often provides a good starting place to derive regret bounds without worrying
about computation (for an example, see Exercise 18.5).
3 The bandits with expert advice framework is clearly more general than
contextual bandits. With the terminology of the bandits with expert advice
framework, the contextual bandit problem arises when the experts are given
by static C → [k] maps.
4 A significant challenge is that a naive implementation of Exp4 has running
time O(M + k) per round, which can be enormous if either M or k is large. In
general there is no solution to this problem, but in some cases the computation
can be reduced significantly. One situation where this is possible is when the
learner has access to an optimisation oracle that for any context/reward
sequence returns the expert that would collect the most reward in this sequence
(this is equivalent to solving the offline problem Eq. (18.8)). In Chapter 30
we show how to use an offline optimisation oracle to learn efficiently in
combinatorial bandit problems. The idea is to solve a randomly perturbed
optimisation problem (leading to the so-called follow-the-perturbed-leader class
of algorithms) and then show that the randomness in the outputs provides
sufficient exploration. However, as we shall see there, these algorithms will
have some extra information, which makes estimating the rewards possible.
5 In the stochastic contextual bandit problem, it is assumed that the
context/reward pairs form a sequence of independent and identically distributed
random variables. Let Φ be a set of functions from C to [k] and suppose the
learner has access to an optimisation oracle capable of finding
t
X
argmaxφ∈Φ xsφ(cs )
s=1

for any sequence of reward vectors x1 , . . . , xt and contexts c1 , . . . , ct . A simple

and efficient algorithm that exploits such an oracle is based on explore-then-
commit, which has O(n2/3 ) regret (Exercise 18.8). There is a more sophisticated
algorithm that is still polynomial-time and for which the regret is about the
same as the result in Theorem 18.1 [Agarwal et al., 2014]. The algorithm
computes importance-weighted estimates of the rewards in each round. These
are used to estimate the regret of all the experts. Based on this, a distribution
over the experts (with a small support) is computed by solving a feasibility
 18.6 Bibliographic Remarks 232

problem. The distribution is constrained so that the importance weights will not
be too large, while the regret estimates averaged over the chosen distribution
will stay small. To reduce the computation cost, this distribution is updated
periodically with the length of the interval between the updates exponentially
growing. The significance of this result is that it reduces contextual bandits
to (cost-sensitive) empirical risk minimisation (ERM), which means that any
advance in solving cost-sensitive ERM problems automatically translates to
bandits.
6 The development of efficient algorithms for ERM is a major topic in supervised
learning. Note that ERM can be NP-hard even in simple cases like linear
classification [Shalev-Shwartz and Ben-David, 2014, §8.7].
7 The bound on the regret stated in Theorem 18.3 is data dependent. Note
that in adversarial bandits the data and instance are the same thing, while
in stochastic bandits the instance determines the probability distributions
associated with each arm and the data corresponds to samples from those
distributions. In any case a data/instance-dependent bound should usually be
preferred if it is tight enough to imply the worst-case optimal bounds.
8 There are many points we have not developed in detail. One is high-probability
bounds, which we saw in Chapter 12 and can also be derived here. We also
have not mentioned lower bounds. The degree to which the bounds are tight
depends on whether or not there is additional structure in the experts. In later
chapters we will see examples when the results are essentially tight, but there
are also cases when they are not.
9 Theorem 18.3 is the first result where we used a time-varying learning rate.
As we shall see in later chapters, time-varying learning rates are a powerful
way to make online algorithms adapt to specific characteristics of the problem
instance.

18.6 Bibliographic Remarks

For a good account on the history of contextual bandits, see the article by Tewari
and Murphy [2017]. The Exp4 algorithm was introduced by Auer et al. [2002b],
and Theorem 18.1 essentially matches theorem 7.1 of their paper (the constant
in Theorem 18.1 is slightly smaller). McMahan and Streeter [2009] noticed that
neither the number of experts nor the size of the action set are what really matters
for the regret, but rather the extent to which the experts tend to agree. McMahan
and Streeter [2009] also introduced the idea of finding the distribution to be played
to be maximally ‘similar’ to Pt (i) while ensuring sufficient exploration of each of
the experts. The idea of explicitly optimising a probability distribution with these
objectives in mind is at the heart of several subsequent works [e.g. Agarwal et al.,
2014]. While Theorem 18.3 is inspired by this work, the result appears to be new
and goes beyond the work of McMahan and Streeter [2009] because it shows
that all one needs is to adapt the learning rate based on the degree of agreement
 18.7 Exercises 233

amongst the experts. Neu [2015a] proves high-probability bounds for Exp4-IX.
You can follow in his footsteps by solving Exercise 18.3. Another way to get
high-probability bounds is to generalise Exp3.P, which was done by Beygelzimer
et al. [2011]. As we mentioned in Note 5, there exist efficient algorithms for
stochastic contextual bandit problems when a suitable optimisation oracle is
available [Agarwal et al., 2014]. An earlier attempt to address the problem of
reducing contextual bandits to cost-sensitive ERM is by Dudı́k et al. [2011]. The
adversarial case of static experts is considered by Syrgkanis et al. [2016], who
√
prove suboptimal (worse than n) regret bounds under various conditions for
follow-the-perturbed-leader for the transductive setting when the contexts are
available at the start. The case when the contexts are independent and identically
distributed, but the reward is adversarial is studied by Lazaric and Munos [2009]
for the finite expert case, while Rakhlin and Sridharan [2016] considers the case
when an ERM oracle is available. The paper of Rakhlin and Sridharan [2016] also
considers the more realistic case when only an approximation oracle is available
for the ERM problem. What is notable about this work is that they demonstrate
regret bounds with a moderate blow-up, but without changing the definition
of the regret. Kakade et al. [2008] consider contextual bandit problems with
adversarial context-loss sequences, where all but one action suffers a loss of one in
every round. This can also be seen as an instance of multi-class classification
with bandit feedback where labels to be predicted are identified with actions
and the only feedback received is whether the label predicted was correct, with
the goal of making as few mistakes as possible. Since minimising the regret is in
general hard in this non-convex setting, just like most of the machine learning
literature on classification, Kakade et al. [2008] provide results in the form of
mistake bounds for linear classifiers where the baseline is not the number of
mistakes of the best linear classifier, but is a convex upper bound on it. The
recent book by Shalev-Shwartz and Ben-David [2014] lists some hardness results
for ERM. For a more comprehensive treatment of computation in learning theory,
the reader can consult the book by Kearns and Vazirani [1994].

18.7 Exercises

18.1 Let C be a finite context set, and let c1 , . . . , cn ∈ C be an arbitrary sequence

of contexts.
v
u n
X uX p
(a) Show that t I {ct = c} ≤ n|C|.
c∈C t=1
(b) Assume that n is an integer multiple of |C|. Show that the choice that
maximises the right-hand side of the previous inequality is the one when
each context occurs n/|C| times.

18.2 Prove Lemma 18.2.

 18.7 Exercises 234

18.3 In this exercise you will prove an analogue of Theorem 12.1 for Exp4-IX.
In the contextual setting, the random regret is
n 
X 
(t)
R̂n = max Em xt − Xt .
m∈[M ]
t=1

Design an algorithm that accepts a parameter δ ∈ (0, 1) such that

s  !!
p nk 1
P R̂n ≥ C nk log(m) + log ≤ δ.
log(m) δ

18.4 Prove Theorem 18.3.

Hint The key idea is to modify the analysis of Exp3 to handle decreasing
learning rates. Of course you can do this directly yourself, or you can peek ahead
to Chapter 28, and specifically Exercises 28.12 and 28.13.
18.5 Let x1 , . . . , xn be a sequence of reward vectors chosen in advance by
an adversary with xt ∈ [0, 1]k . Furthermore, let o1 , . . . , on be a sequence of
observations, also chosen in advance by an adversary with ot ∈ [O] for some
fixed O ∈ N+ . Then let H be the set of functions φ : [O]m → [k] where m ∈ N+ .
In each round the learner observes ot and should choose an action At based on
o1 , A1 , X1 , . . . , ot−1 , At−1 , Xt−1 , ot , and the regret is
n
X
Rn = max xtφ(ot ,ot−1 ,...,ot−m ) − xtAt ,
φ∈H
t=1

where ot = 1 for t ≤ 0. This means the learner is competing with the best
predictor in hindsight that uses only the last m observations. Prove there exists
an algorithm such that
p
E[Rn ] ≤ 2knOm log(k) .

18.6 In this problem we consider non-oblivious experts. Consider the following

modified regret definition:
" n n
#
X X
(t)
Rn0 = max E Em xt − Xt .
m∈[M ]
t=1 t=1

Show the following:

(a) Rn0 ≤ Rn regardless of whether the experts are oblivious or not.
(b) Theorem 18.1 remains valid for non-oblivious experts if in Eq. (18.7) we
replace Rn with Rn0 . In particular, explain how to modify the proof.
(c) Research question: give a non-trivial bound on Rn .

18.7 Prove Eq. (18.13).

18.8 (Explore-then-commit) Consider a stochastic contextual bandit

environment where (Ct )nt=1 is a sequence of contexts sampled from distribution
 18.7 Exercises 235

ξ on C and the rewards are (Xt )nt=1 , where the conditional law of Xt given Ct
and At is PCt At . The mean
R reward when choosing action i ∈ [k] having observed
context c ∈ C is µ(c, i) = x dPci (x). Let Φ be a subset of functions from C to
[k]. The regret is
" n #
X
Rn = n sup µ(φ) − E Xt ,
φ∈Φ t=1
R
where µ(φ) = µ(c, φ(c))dξ(c). Consider a variation of explore-then-commit,
which explores uniformly at random for the first m rounds. Then define
m
k X
µ̂(φ) = I {At = φ(Ct )} Xt .
m t=1

For rounds t > m, the algorithm chooses At = φ̂∗ (Ct ), where

n
X
φ̂ = argmaxφ∈Φ µ̂(φ) = argmaxφ∈Φ
∗
X̂tφ(Ct ) ,
t=1

where X̂ti = kI {At = φ(Ct )} Xt . When no maximiser exists you may assume
that µ̂(φ̂∗ ) ≥ supφ∈Φ µ̂(φ) − ε for any ε > 0 of your choice. Show that when Φ
is finite, then for appropriately tuned m the expected regret of this algorithms
satisfies
 
Rn = O n2/3 (k log(|Φ|))1/3 .

This algorithm is the explore-then-commit version of the epoch-greedy

algorithm by Langford and Zhang [2008]. You should not worry too much
about these details, but of course C should be associated with a σ-algebra
and the family of distributions (Pca : c ∈ C, a ∈ [k]) should be a probability
kernel from C × [k] to R.

18.9 Consider a stochastic contextual bandit problem with the same set-up as
the previous exercise and k = 2 arms. As before, let Φ be a set of functions from
C to [k]. Design a policy such that
" n # r
X n
Rn = n max µ(φ) − E Xt ≤ C ndk log ,
φ∈Φ
t=1
d

where C > 0 is a universal constant and d = VC(Φ) is the VC dimension of Φ.

Hint Use an initial period of exploration to choose a finite ‘representative’

subset of Φ, and then run Exp4 on this subset. The result that you need to know in
connection to the VC dimension is known as Sauer’s lemma, which states that if Φ
has VC dimension d, then for any sequence c = (ci )m i=1 ⊂ C, the cardinality of the
set Φc = {(φ(c1 ), . . . , φ(cm )) : φ ∈ Φ} is at most (em/d)d . You may also find it
 18.7 Exercises 236

useful that an i.i.d. sequence C1 , . . . , Cn is ‘exchangeable’: for any set A ⊂ C n and

any π : [n] → [n] bijection, P ((C1 , . . . , Cn ) ∈ A) = P (Cπ(1) , . . . , Cπ(n) ) ∈ A .
This property helps you to argue that the finite subset of Φ obtained by choosing
functions from Φ that all disagree on the first m elements on the contexts will be
representative of the behaviour of functions in Φ on the rest of the contexts.

We did not talk about VC dimension in this book. An introduction is given

by Shalev-Shwartz and Ben-David [2014], or there is the classic text by
Vapnik [1998]. The application to bandits is due to Beygelzimer et al. [2011].
19 Stochastic Linear Bandits

Contextual bandits generalise the finite-armed setting by allowing the learner

to make use of side information. This chapter focusses on a specific type of
contextual bandit problem in the stochastic set-up where the reward is assumed
to have a linear structure that allows for learning to transfer from one context to
another. This leads to a useful and rich model that will be the topic of the next
few chapters. To begin, we describe the stochastic linear bandit problem and
start the process of generalising the upper confidence bound algorithm.

19.1 Stochastic Contextual Bandits

The stochastic contextual bandit problem mirrors the adversarial contextual

bandit set-up discussed in Chapter 18. At the beginning of round t, the learner
observes a context Ct ∈ C, which may be random or not. Having observed the
context, the learner chooses their action At ∈ [k] based on the information
available. So far everything is the same as the adversarial setting. The difference
comes from the assumption that the reward Xt satisfies

Xt = r(Ct , At ) + ηt ,

where r : C × [k] → R is called the reward function and ηt is the noise, which
we will assume is conditionally 1-subgaussian. Precisely, let

Ft = σ(C1 , A1 , X1 , . . . , Ct−1 , At−1 , Xt−1 , Ct , At )

be the σ-field summarising the information available just before Xt is observed.

Then, we assume that
 2
λ
E [exp(ληt ) | Ft ] ≤ exp almost surely .
2

The noise could have been chosen to be σ-subgaussian for any known σ 2 , but
like in earlier chapters, we save ourselves some ink by fixing its value to σ 2 = 1.
Remember from Chapter 5 that subgaussian random variables have zero mean,
so the assumption also implies that E [ηt | Ft ] = 0 and E [Xt | Ft ] = r(Ct , At ).
 19.1 Stochastic Contextual Bandits 238

If F ⊂ G are σ-algebras, and E[exp(λη) | G] ≤ exp(λ2 /2) almost surely, then

by the tower rule, E[exp(λη) | F] ≤ exp(λ2 /2) almost surely. Hence, the
condition that ηt is subgaussian with respect to Ft can be ‘relaxed’ to the
condition that it be subgaussian with respect to any σ-algebra containing
Ft .

If r was given, then the action in round t with the largest expected return
is A∗t ∈ argmaxa∈[k] r(Ct , a). Notice that this action is now a random variable
because it depends on the context Ct . The loss due to the lack of knowledge of r
makes the learner incur the (expected) regret
" n n
#
X X
Rn = E max r(Ct , a) − Xt .
a∈[k]
t=1 t=1

Like in the adversarial setting, there is one big caveat in this definition of the
regret. Since we did not make any restrictions on how the contexts are chosen, it
could be that choosing a low-rewarding action in the first round might change
the contexts observed in subsequent rounds. Then the learner could potentially
achieve an even higher cumulative reward by choosing a ‘suboptimal’ arm initially.
As a consequence, this definition of the regret is most meaningful when the actions
of the learner do not (greatly) affect subsequent contexts.
One way to eventually learn an optimal policy is to estimate r(c, a) for each
(c, a) ∈ C × [k] pair. As in the adversarial setting, this is ineffective when the
number of context-action pairs is large. In particular, the worst-case regret over
all possible√contextual problems with M contexts and mean reward in [0, 1] is
at least Ω( nM k). While this may not look bad, M is often astronomical (for
example, 2100 ). The argument that gives rise to the mentioned lower bound
relies on designing a problem where knowledge of r(c, ·) for context c provides
no useful information about r(c0 , ·) for some different context c0 . Fortunately,
in most interesting applications, the set of contexts is highly structured, which
is often captured by the fact that r(·, ·) changes ‘smoothly’ as a function of its
arguments.
A simple, yet interesting assumption to capture further information about the
dependence of rewards on context is to assume that the learner has access to a
map ψ : C × [k] → Rd , and for an unknown parameter vector θ∗ ∈ Rd , it holds
that

r(c, a) = hθ∗ , ψ(c, a)i , for all (c, a) ∈ C × [k] . (19.1)

The map ψ is called a feature map, which is the standard nomenclature in

machine learning. The idea of feature maps is best illustrated with an example.
Suppose the context denotes the visitor of a website selling books, the actions are
books to recommend and the reward is the revenue on a book sold. The features
could indicate the interests of the visitors as well as the domain and topic of the
 19.2 Stochastic Linear Bandits 239

book. If the visitors and books are assigned to finitely many categories, indicator
variables of all possible combinations of these categories could be used to create
the feature map. Of course, many other possibilities exist. For example, you can
train a neural network (deep or not) on historical data to predict the revenue
and use the nonlinear map that we obtained by removing the last layer of the
neural network. The subspace Ψ spanned by the feature vectors {ψ(c, a)}c,a in
Rd is called the feature space.
If k · k is a norm on Rd then, an assumption on kθ∗ k implies smoothness of r.
In particular, from Hölder’s inequality,
|r(c, a) − r(c0 , a0 )| ≤ kθ∗ kkψ(c, a) − ψ(c0 , a0 )k∗ ,
where k · k∗ denotes the dual of k · k. Restrictions on kθ∗ k have a similar effect
to assuming that the dimensionality d is finite. In fact, one may push this to
the extreme and allow d to be infinite, an approach that can buy tremendous
flexibility and makes the linearity assumption less limiting.

19.2 Stochastic Linear Bandits

Stochastic linear bandits arise from realising that under Eq. (19.1), all that
matters is the feature vector that results from choosing a given action and not
the ‘identity’ of the action itself. This justifies studying the following simplified
model: in round t, the learner is given the decision set At ⊂ Rd , from which it
chooses an action At ∈ At and receives reward
Xt = hθ∗ , At i + ηt ,
where ηt is 1-subgaussian given A1 , A1 , X1 , . . . , At−1 , At−1 , Xt−1 , At and At . The
random (pseudo-)regret and regret are defined by
n
X
R̂n = max hθ∗ , a − At i ,
a∈At
t=1
" n #
h i X n
X
Rn = E R̂n = E max hθ∗ , ai − Xt ,
a∈At
t=1 t=1

respectively. Different choices of At lead to different settings, some of which we

have seen before. For example, if (ei )i are the unit vectors and At = {e1 , . . . , ed },
then the resulting stochastic linear bandit problem reduces to the finite-
armed setting. On the other hand, if At = {ψ(Ct , i) : i ∈ [k]}, then we have a
contextual linear bandit. Yet another possibility is a combinatorial action
set At ⊆ {0, 1} . Many combinatorial problems (such as matching, least-cost
d

problems in directed graphs and choosing spanning trees) can be written as linear
optimisation problems over some combinatorial set A obtained from considering
incidence vectors often associated with some graph. Some of these topics will be
covered later in Chapter 30.
 19.2 Stochastic Linear Bandits 240

As we have seen in earlier chapters, the UCB algorithm is an attractive approach

for finite-action stochastic bandits. Its best variants are nearly minimax optimal,
instance optimal and exactly optimal asymptotically. With these merits in mind,
it seems quite natural to try and generalise the idea to the linear setting.
The generalisation is based on the view that UCB implements the ‘optimism
in the face of uncertainty’ principle, which is to act in each round as if the
environment is as nice as plausibly possible. In finite-action stochastic bandits,
this means choosing the action with the largest upper confidence bound. In the
case of linear bandits, the idea remains the same, but the form of the confidence
bound is more complicated because rewards received yield information about
more than just the arm played.
The first step is to construct a confidence set Ct ⊂ Rd based on
(A1 , X1 , . . . , At−1 , Xt−1 ) that contains the unknown parameter vector θ∗ with
high probability. Leaving the details of how the confidence set is constructed
aside for a moment, and assuming that the confidence set indeed contains θ∗ , for
any given action a ∈ Rd , let

UCBt (a) = maxhθ, ai (19.2)

θ∈Ct

be an upper bound on the mean pay-off hθ∗ , ai of a. The UCB algorithm that
uses the confidence set Ct at time t then selects

At = argmaxa∈At UCBt (a) . (19.3)

UCB applied to linear bandits is known by various names, including LinRel

(linear reinforcement learning), LinUCB and OFUL (optimism in the f ace of
uncertainty for linear bandits). We will not be very dogmatic of this name and
call algorithms with the above construct instances of LinUCB.
The main question is how to choose the confidence set Ct ⊂ Rd . As usual, there
are conflicting desirable properties:

(a) Ct should contain θ∗ with high probability.

(b) Ct should be as small as possible.

At first sight it is not at all obvious what Ct should look like. After all, it is a
subset of Rd , not just an interval like the confidence intervals about the empirical
estimate of the mean reward for a single action that we saw in the previous
chapters. While we specify the analytic form of a possible construction for Ct here,
there are some details in choosing some of the parameters in this construction. As
they are both delicate and important, we dedicate the next chapter to discussing
them.
Following the idea for UCB, we need an analogue for the empirical estimate
of the unknown quantity, which in this case is θ∗ . There are several principles
one might use for deriving such an estimate. For now we use the regularised
 19.2 Stochastic Linear Bandits 241

least-squares estimator, which is

t
!
X
2
θ̂t = argminθ∈Rd (Xs − hθ, As i) + λkθk22 , (19.4)
s=1

where λ ≥ 0 is called the penalty factor. Choosing λ > 0 helps because it

ensures that the loss function has a unique minimiser even when A1 , . . . , At do
not span Rd , which simplifies the math. The solution to Eq. (19.4) is obtained
easily by differentiation and is

t
X
θ̂t = Vt−1 As Xs , (19.5)
s=1

where (Vt )t are d × d matrices given by

t
X
V0 = λI and Vt = V0 + As A>
s . (19.6)
s=1

So θ̂t is an estimate of θ∗ , which makes it natural to choose Ct to be centered at

θ̂t−1 . For what follows, we will simply assume that the confidence set Ct is closed
and satisfies
n o
Ct ⊆ Et = θ ∈ Rd : kθ − θ̂t−1 k2Vt−1 ≤ βt , (19.7)

where (βt )t is an increasing sequence of constants with β1 ≥ 1. The set Et is an

ellipsoid centred at θ̂t−1 and with principle axis being the eigenvectors of Vt with
corresponding lengths being the reciprocal of the eigenvalues. Notice that as t
grows, the matrix Vt has increasing eigenvalues, which means the volume of the
ellipse is also shrinking (at least, provided βt does not grow too fast). As noted
beforehand, the next chapter will be devoted to show that Ct = Et is a natural
choice for carefully chosen βt . In the rest of this chapter, we simply examine the
consequence of using a confidence set satisfying Eq. (19.7) and assume all the
desirable properties.

The impatient reader who is puzzled of the form Et may briefly think of
the case when ηs ∼ N (0, σ 2 ), A1 , . . . , At−1 are deterministic and span Rd so
that we can take λ = 0. In this case, one easily computes that with V = Vt−1 ,
Z = V 1/2 (θ̂t−1 − θ∗ ) ∼ N (0, I), or that kZk2 is the sum of d, independent
standard normal random variables, and thus it follows the χ2 -distribution
(with d degrees of freedom), from which one can find the appropriate value
of βt−1 . As we shall see, the expression one can get from this calculation,
will, more or less, be still correct in the general case.
 19.3 Regret Analysis 242

19.3 Regret Analysis

We prove a regret bound for LinUCB under the assumption that the confidence
intervals indeed contain the true parameter with high probability and boundedness
conditions on the action set and rewards.

Assumption 19.1. The following hold:

(a) 1 ≤ β1 ≤ β2 ≤ · · · ≤ βn .
(b) maxt∈[n] supa,b∈At hθ∗ , a − bi ≤ 1.
Sn
(c) kak2 ≤ L for all a ∈ t=1 At .
(d) There exists a δ ∈ (0, 1) such that with probability 1 − δ, for all t ∈ [n],
θ∗ ∈ Ct where Ct satisfies Eq. (19.7).

Theorem 19.2. Under the conditions of Assumption 19.1 with probability 1 − δ,

the regret of LinUCB satisfies
s   s  
det Vn dλ + nL2
R̂n ≤ 8nβn log ≤ 8dnβn log .
det V0 dλ

Theorem 20.5 in the next chapter shows that βn may be chosen to be

s    
p √ 1 dλ + nL2
βn = λm2 + 2 log + d log , (19.8)
δ dλ

where m2 is an upper bound on kθ∗ k2 . By choosing δ = 1/n, we obtain the

following corollary bounding the expected regret.

Corollary 19.3. Under the conditions of Assumption 19.1, the expected regret
of LinUCB with δ = 1/n is bounded by
√
Rn ≤ Cd n log(nL) ,

where C > 0 is a suitably large universal constant.

The proof of Theorem 19.2 depends on the following lemma, often called the
elliptical potential lemma.

Lemma 19.4. Let V0 ∈ Rd×d be positive definite and a1 , . . . , an ∈ Rd be a sequence

P
of vectors with kat k2 ≤ L < ∞ for all t ∈ [n], Vt = V0 + s≤t as a> s . Then,

n      
X det Vn trace V0 + nL2
1∧ kat k2V −1 ≤ 2 log ≤ 2d log .
t=1
t−1 det V0 d det(V0 )1/d

Proof Using that for any u ≥ 0, u ∧ 1 ≤ 2 ln(1 + u), we get

n 
X  X  
1 ∧ kat k2V −1 ≤2 log 1 + kat k2V −1 .
t−1 t−1
t=1 t
 19.3 Regret Analysis 243

 
det Vn
We now argue that this last expression is log det V0 . For t ≥ 1, we have
1/2 1/2
Vt = Vt−1 + at a>
t = Vt−1 (I + Vt−1 at at Vt−1 )Vt−1 .
> −1/2 −1/2

Now, since the determinant is a multiplicative map,

   
det(Vt ) = det(Vt−1 ) det I + Vt−1 at a> = det(Vt−1 ) 1 + kat k2V −1 ,
−1/2 −1/2
t Vt−1
t−1

where the second equality follows because the matrix I + yy has eigenvalues >

1 + kyk22 and 1 as well as the fact that the determinant of a matrix is the product
of its eigenvalues. Putting things together, we see that
n 
Y 
det(Vn ) = det(V0 ) 1 + kat k2V −1 , (19.9)
t−1
t=1

which is equivalent to the first inequality that we wanted to prove. To get the
second inequality, note that by the inequality of arithmetic and geometric means,
 d  d
1 trace V0 + nL2
Y d
det(Vn ) = λi ≤ trace Vn ≤ ,
i=1
d d

where λ1 , . . . , λd are the eigenvalues of Vn .

Proof of Theorem 19.2 By part (d) of Assumption 19.1, it suffices to prove the
bound on the event that θ∗ ∈ Ct for all rounds t ∈ [n]. Let A∗t = argmaxa∈At hθ∗ , ai
be an optimal action for round t and rt be the instantaneous regret in round t
defined by

rt = hθ∗ , A∗t − At i .

Let θ̃t ∈ Ct be the parameter in the confidence set for which hθ̃t , At i = UCBt (At ).
Then, using the fact that θ∗ ∈ Ct and the definition of the algorithm leads to

hθ∗ , A∗t i ≤ UCBt (A∗t ) ≤ UCBt (At ) = hθ̃t , At i .

Using Cauchy–Schwarz inequality and the assumption that θ∗ ∈ Ct and facts that
θ̃t ∈ Ct and Ct ⊆ Et leads to

rt = hθ∗ , A∗t − At i ≤ hθ̃t − θ∗ , At i ≤ kAt kV −1 kθ̃t − θ∗ kVt−1

t−1
p
≤ 2kAt kV −1 βt . (19.10)
t−1

By part (b) we also have rt ≤ 2, which, combined with βn ≥ max{1, βt }, yields

p p  
rt ≤ 2 ∧ 2 βt kAt kV −1 ≤ 2 βn 1 ∧ kAt kV −1 .
t−1 t−1

Then, by Cauchy–Schwarz inequality,

v v
n u n u n  
X u X u X
R̂n = rt ≤ tn rt2 ≤ 2tnβn 1 ∧ kAt k2V −1 . (19.11)
t−1
t=1 t=1 t=1
 19.4 Notes 244

The result is completed using Lemma 19.4, which depends on part (c) of
Assumption 19.1.

19.3.1 Computation
An obvious question is whether or not the optimisation problem in Eq. (19.3) can
be solved efficiently. First note that the computation of At can also be written as

(At , θ̃t ) = argmax(a,θ)∈At ×Ct hθ, ai . (19.12)

This is a bilinear optimisation problem over the set At × Ct . In general, not much
can be said about the computational efficiency of solving this problem. There are
two notable special cases, however.

(a) Suppose that a(θ) = argmaxa∈At hθ, ai can be computed efficiently for any θ
and that Ct = co(φ1 , . . . , φm ) is the convex hull of a finite set. Then At can
be computed by finding a(φ1 ), . . . , a(φm ) and choosing At = a(φi ), where i
maximises hφi , a(φi )i.
(b) Assume that Ct = Et is the ellipsoid given in Eq. (19.7) and At is a small
finite set. Then the action At from Eq. (19.12) can be found using
p
At = argmaxa∈At hθ̂t−1 , ai + βt kakV −1 , (19.13)
t−1

which may be solved by simply iterating over the arms and calculating
the term inside the argmax. Further implementation issues are explored in
Exercise 19.8.

19.4 Notes

1 It was mentioned that ψ may map its arguments to an infinite dimensional

space. There are several issues that arise in this setting. The first is whether or
not the algorithm can be computed efficiently. This is usually tackled via the
kernel trick, which assumes the existence of an efficiently computable kernel
function κ : (C × [k]) × (C × [k]) → R such that

hψ(c, a), ψ(c0 , a0 )i = κ((c, a), (c0 , a0 )) .

The trick is to rewrite all computations in terms of the kernel function so that
ψ(c, a) is neither computed, nor stored. The second issue is that the claim
made in Theorem 19.2 depends on the dimension d and becomes vacuous when
d is large or infinite. This dependence arises from Lemma 19.4. It is possible to
modify this result by replacing d with a data-dependent quantity that measures
the ‘effective dimension’ of the image of the data under φ. The final challenge
is to define an appropriate confidence set. See the bibliographic remarks for
further details and references.
 19.4 Notes 245

2 The bound given in Theorem 19.2 is essentially a worst-case style of bound,

with little dependence on the parameter θ∗ or the geometry of the action set.
Instance-dependent bounds for linear bandits are still an open topic of research,
and the asymptotics are only understood in the special case where the action
set is finite and unchanging (Chapter 25).
3 Theorem 20.5 in the next chapter shows that (βt )nt=1 as defined in Eq. (19.8)
can be replaced with a data-dependent quantity that is strictly smaller:
s    
1/2
√ 1 det(Vt−1 )
βt = m2 λ + 2 log + log . (19.14)
δ λd

Empirically this choice is never worse than the value suggested in Eq. (19.8)
and sometimes better, typically by a modest amount.
4 The application of Cauchy–Schwarzpin Eq. (19.11) often loses a logarithm,
as it does, for example, when rt = 1/t. Recently, however, a lower bound
for contextual linear bandits has been derived by constructing a sequence for
which this Cauchy–Schwarz is tight, as well as Lemma 19.4 [Li et al., 2019b].
5 In the worst case, the bound in Theorem 19.2 is tight up to logarithmic factors.
More details are in Chapter 24, which is devoted to lower bounds for stochastic
linear bandits. The environments for which the lower bound nearly matches the
upper bound have action sets that are either infinite or exponentially large in
the dimension. When |At | ≤ k for all rounds t, there are algorithms for which
the regret is
q 
3
Rn = O dn log (nk) .

The special case where the action set does not change with time is treated in
Chapter 22, where references to the literature are also provided.
6 The calculation in Eq. (19.13) shows that LinUCB has more than just a passing
resemblance to the UCB algorithm introduced in Chapter 7. The term hθ̂t−1 , ai
may be interpreted as an empirical estimate of the reward from choosing action
√
a, and βt kakV −1 is a bonus term that ensures sufficient exploration. If the
t−1

penalty term vanishes (λ = 0) and At = {e1 , . . . , ed } for all t ∈ [n], then θ̂i
becomes the empirical mean of action ei , and the matrix Vt is diagonal, with
its ith diagonal entry being the number of times action ei is used up to and
including round t. Then the bonus term has order
s
p βt
βt kei kV −1 = ,
t−1 Ti (t − 1)

where Ti (t − 1) is the number of times action ei has been chosen before the tth
round. So UCB for finite-armed bandits is recovered by choosing βt = 2 log(·),
where the term inside the logarithm can be chosen in a variety of ways as
discussed in earlier chapters. Notice now
p that the simple analysis given in this
chapter leads to a regret bound of O( dn log(·)), which is quite close to the
 19.5 Bibliographic Remarks 246

highly specialised analysis given in Chapters 7 to 9. Note however that the

dimension-free choice of βt does not satisfy Eq. (19.7), but this happens to be
unnecessary for the proof of Theorem 19.2 to go through.
7 An extension of considerably interest of the linear model is the generalised
linear model where the reward is
Xt = µ(hθ∗ , At i) + ηt , (19.15)
where µ−1 : R → R is called the link function. A common choice is the sigmoid
function: µ(x) = 1/(1 + exp(−x)) Bandits with rewards from a generalised
linear model have been studied by Filippi et al. [2010], who prove a bound with
a similar form as Theorem 19.2. Unfortunately, however, the bound depends in
a slightly unpleasant manner on the form of the link function, and it seems
there may be significant room for improvement. You will analyse and algorithm
for generalised linear bandits in Exercise 19.6.
8 Beyond optimism, there are at least three other principles for constructing
algorithms for stochastic linear bandits. The first is Thompson sampling, which
is a randomised Bayesian algorithm discussed at length in Chapter 36. The
second is a class of algorithms designed to achieve asymptotic optimality
in the special cases where the action set is fixed, or sampled i.i.d. from a
fixed distribution with finite support. These algorithms fall into the class of
‘optimisation-based’ algorithms that estimate the unknown parameter and
then solve an optimisation problem to determine an optimal allocation over
the actions [Lattimore and Szepesvári, 2017, Ok et al., 2018, Combes et al.,
2017, Hao et al., 2020]. A downside of optimisation-based approaches is that
so far the results have a very asymptotic nature and the algorithms are not
very practical. These ideas are discussed a little more in Chapter 25, where we
prove asymptotic lower bounds for linear bandits. The third design principle is
called information-directed sampling, which has a Bayesian version [Russo and
Van Roy, 2014a] and frequentist analogue [Kirschner and Krause, 2018]. In
rough terms, these algorithms choose a distribution over actions that minimises
the ratio of a squared expected instantaneous regret and the information gain
about the optimal action, which in the frequentist version is replaced by a
potential function that mimics the information gain.

19.5 Bibliographic Remarks

Stochastic linear bandits were introduced by Abe and Long [1999]. The first paper
to consider algorithms based on the optimism principle for linear bandits is by
Auer [2002], who considered the case when the number of actions is finite. The
core ideas of the analysis of optimistic algorithms (and more) is already present
in this paper. An algorithm based on confidence ellipsoids is described in the
papers by Dani et al. [2008], Rusmevichientong and Tsitsiklis [2010] and Abbasi-
Yadkori et al. [2011]. The regret analysis presented here, and the discussion of the
 19.6 Exercises 247

computational questions, is largely based on the former of these works, which also
√
stresses that an expected regret of Õ(d n) can be achieved regardless of the shape
of the decision sets At as long as the means are guaranteed to lie in a bounded
interval. Rusmevichientong and Tsitsiklis [2010] consider both optimistic and
explore-then-commit strategies, which they call ‘phased exploration and greedy
exploitation’ (PEGE). They focus on the case where At is the unit ball or some
other compact set with a smooth boundary and show that PEGE is optimal up to
logarithmic factors. The observation that explore-then-commit works for the unit
ball (and other action sets with a smooth boundary) was independently made
by Abbasi-Yadkori et al. [2009], further expanded in [Abbasi-Yadkori, 2009a].
Generalised linear models are credited to Nelder and Wedderburn [1972]. We
mentioned already that LinUCB was generalised to this model by Filippi et al.
[2010]. A more computationally efficient algorithm has recently been proposed by
Jun et al. [2017]. Nonlinear structured bandits where the pay-off function belongs
to a known set have also been studied [Anantharam et al., 1987, Russo and Van
Roy, 2013, Lattimore and Munos, 2014]. Kernelised versions of UCB have been
given by Srinivas et al. [2010], Abbasi-Yadkori [2012] and Valko et al. [2013b].
We mentioned early in the chapter that making assumptions on the norm θ∗ is
related to smoothness of the reward function with smoother functions leading
to stronger guarantees. For an example of where this is done, see the paper on
‘spectral bandits’ by Valko et al. [2014] and Exercise 19.7.

19.6 Exercises

19.1 (Least-squares solution) Prove that the solution given in Eq. (19.5) is
indeed the minimiser of Eq. (19.4).

19.2 (Action selection with ellipsoidal confidence sets) Show that the
action selection in LinUCB can indeed be done as shown in Eq. (19.13) when
Ct = Et is an ellipsoid given in Eq. (19.7).

19.3 (Elliptical potentials: You cannot have more than O(d) big
intervals) Let V0 = λI and a1 , . . . , an ∈ Rd be a sequence of vectors with
Pt
kat k2 ≤ L for all t ∈ [n]. Then let Vt = V0 + s=1 as a>s and show that the
number of times kat kV −1 ≥ 1 is at most
t−1

 
3d L2
log 1 + .
log(2) λ log(2)
 19.6 Exercises 248

The proof of Theorem 19.2 depended on part (b) of Assumption 19.1, which
asserts that the mean rewards are bounded by one. Suppose we replace this
assumption with the relaxation that there exists a B > 0 such that
max sup hθ∗ , a − bi ≤ B .
t∈[n] a,b∈At

Then, Exercise 19.3 allows you to bound the number of rounds when
kxt kV −1 ≥ 1, and in these rounds the naive bound of rt ≤ B is used. For
t−1
the remaining rounds, the analysis of Theorem 19.2 goes through unaltered.
As a consequence we see that the dependence on B is an additive constant
term that does not grow with the horizon.

19.4 (Computation cost savings with fixed large action set) When
the action set At = A is fixed and |A| = k, the total computation cost of
LinUCB after n rounds is O(kd2 n) in n rounds if the advice on implementation
of Exercise 19.8 is used. This can be reduced to O(k log(n) + d2 n) with almost
no increase of the regret, a significant reduction when k  d2 . For this, LinUCB
should be modified to work in phases, where in a given fixed it uses the same
action computed in the usual way at the beginning of the phase. A phase ends
when log det Vt (λ) increases by log(1 + ε).

kxk2A det A
(a) Prove that if 0 ≺ B  A then supx6=0 kxk2B
≤ det B .

(b) Let Assumption 19.1 hold. Let R̂n (βn ) be the regret bound of LinUCB
stated in Theorem 19.2. Show that with probability 1 − δ the random pseudo-
regret R̂n of the phased version of LinUCB, as described above, satisfies
R̂n ≤ R̂n ((1 + ε)βn ).

19.5 (Lipschitz reward functions) Consider the k-armed stochastic

contextual setting of Section 19.1, and assume that C = [0, 1] and that the
reward functions r(·, i) : C → [0, 1] are L-Lipschitz:

|r(x, i) − r(y, i)| ≤ L|x − y| for all x, y ∈ [0, 1] , i ∈ [k] .

(a) Construct an algorithm whose regret Rn after n rounds is O((Lk log k)1/3 n2/3 ).
(b) Show that the minimax optimal regret is of the order Ω((Lk)1/3 n2/3 ).
(c) Generalise the result to the case when C = [0, 1]d and in the definition
of Lipschitzness we use the Euclidean norm. Show the dependence on the
dimension in the lower and upper bounds. Discuss the influence of the choice
of the norm.

Hint Consider discretising C. Alternatively, use Exp4.

 19.6 Exercises 249

This exercise is inspired by the work of Perchet and Rigollet [2013], who focus
on improving the regret bound by adaptive discretisation when a certain
margin condition holds. There are many variations of the problem of the
previous exercise. For starters, the domain of contexts could be more general:
one may consider higher-order smoothness, continuous action and context
spaces. What is the role of the context distribution? In some applications,
the context distribution can be estimated for free, in which case you might
assume the context distribution is known. How to take a known context
distribution into account? To whet your appetite, if the context distribution
is concentrated on a handful of contexts, the discretisation should respect
which contexts the distribution is concentrated on. Instead of discretisation,
one may also consider function approximation. An interesting approach that
goes beyond discretisation is by Combes et al. [2017] (see also Magureanu
et al. 2014). The approach in these papers is to derive an asymptotic,
instance-dependent lower bound, which is then used to guide the algorithm
(much like in the track-and-stop algorithm in Section 33.2). An open problem
is to design algorithms that are simultaneously near minimax optimal and
asymptotically optimal. As described in Part II, this problem is now settled
for finite-armed stochastic bandits, the only case where we can say this in
the whole literature of bandits.

19.6 (Generalised linear bandits) In this exercise you will design and
analyse an algorithm for the generalised linear bandit problem mentioned in
Note 7. Let Θ be a convex compact subset of Rd and assume that θ∗ ∈ Θ. The
only difference relative to the standard model is that the reward is

Xt = µ(hθ∗ , At i) + ηt ,

where µ : R → R is a continuously differentiable function such that

 
c1 = min 1, min min µ (hθ, ai) > 0 and
0
a∈∪n
t=1 At θ∈Θ
 
c2 = max 1, max n
max µ (hθ, ai) < ∞ .
0
a∈∪t=1 At θ∈Θ

That c1 > 0 is assumed implies that µ is increasing on the relevant area of its
domain. Like in the standard model, for each t ≥ 1, ηt is 1-subgaussian given
A1 , X1 , . . . , At−1 , Xt−1 , At , and you may as well assume that rewards and feature
vectors are bounded:

max µ(hθ∗ , ai) − µ(hθ∗ , bi) ≤ 1 and max kak2 ≤ L and kθ∗ k2 ≤ m2 .
a,b∈∪n
t=1 At a∈∪n
t=1 At

Recall that λ is the regularisation parameter in the definition of Vt (see Eq. (19.6))
 19.6 Exercises 250

and let

t
X t
X

gt (θ) = λθ + µ(hθ, As i)As , Lt (θ) = gt (θ) − Xs As .

s=1 s=1 Vt−1

(a) Let βt be as in Eq. (19.8) and define a confidence set Ct by

n o
1/2
Ct = θ ∈ Θ : Lt−1 (θ) ≤ βt−1 .

Show that θ∗ ∈ Ct for all t with probability at least 1 − δ.

(b) Prove that for all θ, θ0 ∈ Θ, c1 kθ − θ0 kVt ≤ kgt (θ) − gt (θ0 )kV −1 .
t
(c) Consider the algorithm that chooses

At = argmaxa∈At max µ(hθ, ai) .

θ∈Ct

Prove that on the event that θ∗ ∈ Ct , for A∗t = argmaxa∈At µ(hθ∗ , ai),
1/2
2c2 βt−1
rt = µ(hθ∗ , A∗t i) − µ(hθ∗ , At i) ≤ kAt kV −1 .
c1 t−1

Pn
(d) Prove that with probability at least 1 − δ, the random regret R̂n = t=1 rt
is bounded by
s  
c2 nL2
R̂n ≤ 8ndβn log 1 + .
c1 d

Hint For (a), you should peek into the future and use Theorem 20.4. The
mean value theorem will help with Part (b).
19.7 (Spectral bandits) The regret of LinUCB can be improved considerably
if an appropriate norm of θ∗ is known to be small. In this exercise you will
investigate this phenomenon. Suppose that V0 is positive definite with eigenvalues
Pt
λ1 , . . . , λd , respective eigenvectors v1 , . . . , vd , and Vt = V0 + s=1 As A> s . All
other quantities are left unchanged, but the alternative value of V0 means that θ̂t
is heavily regularised in the direction of each vi for which λi is large. Without
loss of generality, assume that (λi )di=1 is increasing and let λ = λ1 be the smallest
eigenvalue. Define the ‘effective dimension’ by
 
n
deff = max i ∈ [d] : (i − 1)λi ≤ ∈ [d] .
log(1 + nL2 /λ)
   
det(Vt ) nL2
(a) Prove that log det(V 0)
≤ 2deff log 1 + λ .
(b) Let m > 0 be a user-defined constant and
s    
1/2 1 det(Vt )
βt = m + 2 log + log
δ det(V0 )

and let Ct = {θ : kθ − θ̂t−1 k2Vt−1 ≤ βt−1 }. Assume that kθ∗ kV0 ≤ m and
prove that θ∗ ∈ Ct for all t with probability at least 1 − δ.
 19.6 Exercises 251

(c) Prove that if kθ∗ kV0 ≤ m, then with probability at least 1 − δ, the random
regret of LinUCB in this setting is bounded by
s  
nL2
R̂n ≤ 8βn deff log 1 + .
λ
(d) Show that with an appropriate choice of δ,
√

E[R̂n ] = O deff n log(nL2 ) , (19.16)
where the last equality suppresses dependence on m and λ = λ1 .
(e) The result of the previous display explains the definition of the ‘effective
dimension’ deff . When V0 = I, which gives rise to uniform regularisation,
√
Corollary 19.3 states that for kθ∗ k ≤ m, E[R̂n ] = O(d n log(nL2 )). Given
Eq. (19.16), for a fixed n, deff can be thought of as replacing the dimension
d when V0 is chosen as any positive definite matrix with V0  λI. It follows
then that when deff ≤ d, the upper bound for non-uniform regularisation will
be smaller. Given this, explain the potential pros and cons of non-uniform
regularisation. What happens when kθ∗ kV0 ≤ m fails to hold? Show a bound
on the degradation of the expected regret as a function of max(0, kθ∗ kV0 −m).

Hint For Part (b), you should peek into the next chapter and modify
Theorem 20.5.

Valko et al. [2014] had a particular application in mind when designing

the algorithm in Exercise 19.7. Consider a large graph with k vertices
and similarity matrix between the vertices W ∈ [0, ∞)k×k . The graph
Laplacian is the matrix L = D − W , where D is diagonal with Dii =
Pk
j=1 Wij . Let µ ∈ [0, 1] be an unknown reward function and consider a
k

bandit algorithm with k actions corresponding to the vertices of the graph.

Without further assumptions, this is a finite-armed bandit, which for large
k is hopeless without further assumptions. Valko et al. [2014] assume the
rewards for well-connected vertices are similar – a kind of smoothness. Let
L = Q> ΛQ be the spectral decomposition of L and θ = Q> µ. Then,
1 X
Wij (µi − µj )2 = kθk2Λ .
2
i,j∈[k]

The left-hand side measures the variability of µ, weighted by connectivity of

the graph. Hence, assuming that kθkΛ is small corresponds to assuming that
µ changes only a little between well-connected vertices. Valko et al. [2014]
then let V0 = Λ + λI and analyse/implement the algorithm described in
Exercise 19.7. A more detailed exposition is by Valko [2016].

19.8 (Implementation) If the action set is the same in every round, then the
assumptions are satisfied for the various versions of UCB discussed in Chapters 7
 19.6 Exercises 252

LinUCB 4,000 LinUCB

40 UCB UCB
Expected regret

30 2,000

20
0
0 0.2 0.4 0.6 0.8 1 0 200 400 600 800 1,000
∆ k

Figure 19.1 The plot on the left compares the regret of UCB (Algorithm 6) and LinUCB
on a Gaussian bandit with k = 2, n = 1000 and varying suboptimality gaps ∆. The plot
on the right compares the same algorithms on a linear bandit with actions uniformly
distributed on the sphere and with d = 5 and n = 5000. The parameter θ is also
uniformly generated on the sphere.

to 9. How does LinUCB compare to UCB? Implement the version of LinUCB

using the value of βt given in Eq. (19.8) and/or Eq. (19.14) and compare it the
version of UCB given in Algorithm 6. In particular:
(a) Compare LinUCB with UCB on the 2-armed bandit with n = 1000 where the
reward distributions are Gaussian with unit variance and mean µ = (0, −∆)
where ∆ ∈ [0, 1], which for LinUCB corresponds to using At = {e1 , e2 } with
d = 2.
(b) Now compare LinUCB with UCB on k-armed stochastic linear bandits
where the d = 5 and At = A is composed of k unit vectors sampled from
the uniform distribution on the sphere (sample these vectors once). The
unknown parameter θ should also lie on the unit sphere and the noise should
be standard Gaussian. Plot the expected regret as a function of k ranging
from 2 to 1000 with a horizon of n = 5000.
(c) What conclusions can you draw from the experimental results you obtained?
(d) Show how one can use the Sherman-Morrison formula to implement
LinUCB using O(kd2 ) computation cost per round.
For parts (a) and (b) you should produce something comparable to Fig. 19.1.
20 Confidence Bounds for Least
Squares Estimators

In the last chapter, we derived a regret bound for a version of the upper confidence
bound algorithm that depended on a particular kind of confidence set. The purpose
of this chapter is to justify these choices.
Suppose a bandit algorithm has chosen actions A1 , . . . , At ∈ Rd and received
the rewards X1 , . . . , Xt with Xs = hθ∗ , As i + ηs where ηs is zero-mean noise.
Recall from the previous chapter that the penalised least-squares estimate of θ∗
is the minimiser of
t
X
Lt (θ) = (Xs − hθ, As i)2 + λkθk22 ,
s=1

where λ ≥ 0 is the penalty factor. This is minimised by

t
X t
X
θ̂t = Vt (λ)−1 Xs As with Vt (λ) = λI + As A>
s . (20.1)
s=1 s=1

It is convenient for the remainder to abbreviate Vt = Vt (0). Designing confidence

sets for θ∗ when A1 , . . . , At have been chosen by a bandit algorithm is a
surprisingly delicate matter. The difficulty stems from the fact that the actions
are neither fixed nor independent but are intricately correlated via the rewards.
We spend the first section of this chapter building intuition by making some
simplifying assumptions. Eager readers may skip directly to Section 20.1. For the
rest of this section, we assume the following:

1 No regularisation: λ = 0 and Vt is invertible.

2 Independent subgaussian noise: (ηs )s are independent and 1-subgaussian.
3 Fixed design: A1 , . . . , At are deterministically chosen without the knowledge of
X1 , . . . , Xt .

None of these assumptions is plausible in the bandit setting, but the simplification
eases the analysis and provides insight.

The assumption that λ = 0 means that in this section, θ̂t is just the ordinary
least squares estimator of θ. The requirement that Vt be non-singular means
that (As )ts=1 must span Rd , and so t must be at least d.
 Confidence Bounds for Least Squares Estimators 254

Comparing θ∗ and θ̂t in the direction x ∈ Rd , we have

* t
+ * t
+
X X

hθ̂t − θ∗ , xi = x, Vt−1
As Xs − θ∗ = x, Vt −1
As As θ ∗ + η s − θ ∗
>

s=1 s=1
* t
+ t
X X

= x, Vt−1 As η s = x, Vt−1 As ηs .
s=1 s=1

Since (ηs )s are independent and 1-subgaussian, by Lemma 5.4 and Theorem 5.3,
 v 
D E uu X t

2
 
1
P  x, θ̂t − θ∗ ≥ t2 x, Vt−1 As log  ≤ δ.
s=1
δ

Pt
2
A little linear algebra shows that s=1 x, Vt−1 As = kxk2V −1 and so
t

s  !
2 1
P hθ̂t − θ∗ , xi ≥ 2kxkV −1 log ≤ δ. (20.2)
t δ

If we only care about confidence bounds for one or a few vectors x, we could stop
here. For large action sets (with more than Ω(2d ) actions), one approach is to
convert this bound to a bound on kθ̂t − θ∗ kVt . To begin this process, notice that
1/2
1/2 Vt (θ̂t − θ∗ )
kθ̂t − θ∗ kVt = hθ̂t − θ∗ , Vt Xi , where X = .
kθ̂t − θ∗ kVt
The problem is that X is random, while we have only proven (20.2) for
deterministic x. The standard way of addressing problems like this is to use
a covering argument. First we identify a finite set Cε ⊂ Rd such that whatever
value X takes, there exists some x ∈ Cε that is ε-close to X. Then a union
bound and a triangle inequality allows one to finish. By its definition, we have
kXk22 = X > X = 1, which means that X ∈ S d−1 = {x ∈ Rd : kxk2 = 1}. Using
that X ∈ S d−1 , we see it suffices to cover S d−1 . The following lemma provides
the necessary guarantees on the size of the covering set.

Lemma 20.1. There exists a set Cε ⊂ Rd with |Cε | ≤ (3/ε)d such that for all
x ∈ S d−1 there exists a y ∈ Cε with kx − yk2 ≤ ε.

The proof of this lemma requires a bit work, but nothing really deep is needed.
This work is deferred to Exercises 20.3 and 20.4. Let Cε be the covering set given
by the lemma, and define event
( s  )
D E |Cε |
1/2
E = exists x ∈ Cε : Vt x, θ̂t − θ∗ ≥ 2 log .
δ

1/2
Using the fact that kVt xkV −1 = kxk2 = 1, and a union bound combined with
t
Eq. (20.2) shows that P (E) ≤ δ. When E does not occur, Cauchy–Schwarz shows
 20.1 Martingales and the Method of Mixtures 255

that
D E
1/2
kθ̂t − θ∗ kVt = max Vt x, θ̂t − θ∗
x∈S d−1
hD E D Ei
1/2 1/2
= max min Vt (x − y), θ̂t − θ∗ + Vt y, θ̂t − θ∗
x∈S d−1 y∈Cε
" s  #
|Cε |
< max min kθ̂t − θ∗ kVt kx − yk2 + 2 log
x∈S d−1 y∈Cε δ
s  
|Cε |
≤ εkθ̂t − θ∗ kVt + 2 log .
δ
Rearranging yields
s  
1 |Cε |
kθ̂t − θ∗ kVt < 2 log .
1−ε δ
Now there is a tension in the choice of ε > 0. The term in the denominator
suggests that ε should be small, but by Lemma 20.1 the cardinality of Cε grows
rapidly as ε tends to zero. By lazily choosing ε = 1/2,
s   !
1
P kθ̂t − θ∗ kVt ≥ 2 2 d log(6) + log ≤ δ. (20.3)
δ

Except for constants and other minor differences, this turns out to be about as
good as you can get. Unfortunately, however, this analysis only works because
Vt was assumed to be deterministic. When the actions are chosen by a bandit
algorithm, this assumption does not hold, and the ideas need to be modified.

20.1 Martingales and the Method of Mixtures

We now remove the limiting assumptions in the previous section. Of course some
conditions are still required. For the remainder of this section the following is
assumed:
1 There exists a θ∗ ∈ Rd such that Xt = hθ∗ , At i + ηt for all t ≥ 1.
2 The noise is conditionally 1-subgaussian:
 2
α
for all α ∈ R and t ≥ 1, E [exp(αηt ) | Ft−1 ] ≤ exp a.s. , (20.4)
2
where Ft−1 is such that A1 , X1 , . . . , At−1 , Xt−1 , At are Ft−1 -measurable.
3 In addition, we assume that λ > 0.
The inclusion of At in the definition of Ft−1 allows the noise to depend on past
choices, including the most recent action. This is often essential, as the case of
Bernoulli rewards shows. We have now dropped the assumption that (At )∞ t=1 are
fixed in advance.
 20.1 Martingales and the Method of Mixtures 256

The assumption that λ > 0 ensures that Vt (λ) is invertible and allows us
to relax the requirement that the actions span Rd . Notice also that in this
section, we allow the interaction sequence to be infinitely long.

Since we want exponentially decaying tail probabilities, one is tempted to try

the Cramér–Chernoff method:
  h  i
P kθ̂t − θ∗ k2Vt (λ) ≥ u2 ≤ inf E exp αkθ̂t − θ∗ k2Vt (λ) − αu2 .
α>0

Sadly, we do not know how to bound this expectation. Can we still somehow use
the Cramér–Chernoff method? We take inspiration from looking at the special
Pt
case of λ = 0 one last time, assuming that Vt = s=1 As A>
s is invertible. Let

t
X
St = η s As .
s=1

Pt
Recall that θ̂t = Vt−1 s=1 Xs As = θ∗ + Vt−1 St . Hence,
 
1 2 1 2 1 2
kθ̂t − θ∗ kVt = kSt kV −1 = max hx, St i − kxkVt .
2 2 t x∈Rd 2

The point of the second equality is to separate the martingale (St )t from Vt at
the price of the introduction of a maximum. This second equality is a special
case of (Fenchel) duality. As we shall see later in Chapter 26, for sufficiently
nice convex functions f one can show that with an appropriate function f ∗ ,
for any x ∈ Rd from the domain of f , f (x) = supu∈Rd hu, xi − f ∗ (u). The
advantage of this is that for any fixed u, x appears in a linear fashion.

The next lemma shows that the exponential of the term inside the maximum
is a supermartingale even when λ > 0.

Lemma 20.2. For all x ∈ Rd the process Mt (x) = exp(hx, St i − 12 kxk2Vt (λ) ) is an
F-adapted non-negative supermartingale with M0 (x) ≤ 1.

Proof of Lemma 20.2 That Mt (x) is Ft -measurable for all t and that it
is nonnegative are immediate from the definition. We need to show that
E[Mt (x) | Ft−1 ] ≤ Mt−1 (x) almost surely. The fact that (ηt ) is conditionally
1-subgaussian means that
! !
hx, At i
2 kxk2At A>
E [exp (ηt hx, At i) | Ft−1 ] ≤ exp = exp t
a.s.
2 2
 20.1 Martingales and the Method of Mixtures 257

Hence
   
1 2

E[Mt (x) | Ft−1 ] = E exp hx, St i − kxkVt  Ft−1

2
   
1 2

= Mt−1 (x)E exp ηt hx, At i − kxkAt A>  Ft−1

2 t

≤ Mt−1 (x) a.s.

Finally, note that M0 (x) ≤ 1 is immediate.

For simplicity, consider now again the case when λ = 0. Combining the lemma
and the linearisation idea almost works. The Cramér–Chernoff method leads to
      
1 1
P kθ̂t − θ∗ k2Vt ≥ log(1/δ) = P exp max hx, St i − kxk2Vt ≥ 1/δ
2 x∈Rd 2
   
1
≤ δE exp max hx, St i − kxk2Vt
x∈Rd 2
 
= δE max Mt (x) . (20.5)
x∈Rd

Lemma 20.2 shows that E[Mt (x)] ≤ 1. This seems quite promising, but
the presence of the maximum is a setback because E[maxx∈Rd Mt (x)] ≥
maxx∈Rd E[Mt (x)], which is the wrong direction to be used above. This means
we cannot directly use the lemma to bound Eq. (20.5). There are two ways to
proceed. The first is to use a covering argument over possible near-maximisers
of x, which eventually works. A more elegant way is to take inspiration from
Eq. (20.5) and use Laplace’s method for approximating integrals of well-behaved
exponentials, as we now explain.

20.1.1 Laplace’s Method ( )

We briefly review Laplace’s method for one-dimensional functions. Assume that
f : [a, b] → R is twice differentiable and has a unique maximum at x0 ∈ (a, b)
with −q = f 00 (x0 ) < 0. Laplace’s method for approximating f (x0 ) is to compute
the integral
Z b
Is = exp(sf (x))dx
a

for some large value of s > 0. From a Taylor expansion, we may write
q
f (x) = f (x0 ) − (x − x0 )2 + R(x) ,
2
where R(x) = o((x − x0 )2 ). Under appropriate technical assumptions,
Z b  
sq(x − x0 )2
Is ∼ exp(sf (x0 )) exp − dx as s → ∞ .
a 2
 20.1 Martingales and the Method of Mixtures 258

1 1

s=1 s=3

0 0
−5 0 5 −5 0 5

Figure 20.1 The plots depict Laplace’s approximation with f (x) = cos(x) exp(−x2 /20),
which is maximised at x0 = 0 and has q = −f 00 (x0 ) = 11/10. The solid line is a plot of
exp(sf (x))/ exp(sf (x0 )), and the dotted line is exp(−sq(x − x0 )2 ).

Furthermore, as s gets large,

Z b   Z ∞   r
sq(x − x0 )2 sq(x − x0 )2 2π
exp − dx ∼ exp − dx =
a 2 −∞ 2 sq
and hence
r
2π
Is ∼ exp(sf (x0 )) .
sq
It should also be clear that the fact that we integrate with respect to the Lebesgue
measure does not matter much. We could have integrated with respect to any
other measure as long as that measure puts a positive mass on the neighbourhood
of the maximiser. The method is illustrated in Fig. 20.1. The take-home message
is that if we integrate the exponential of a function that has a pronounced
maximum, then we can expect that the integral will be close to the exponential
function of the maximum.

20.1.2 Method of Mixtures

Laplace’s approximation suggests that
Z
max Mt (x) ≈ Mt (x)dh(x) , (20.6)
x Rd

where h is some measure on Rd chosen so that the integral can be calculated

in closed form. This is not a requirement of the method, but it does make the
argument shorter. The main benefit of replacing the maximum with an integral
is that we obtain the following lemma, which you will prove in Exercise 20.5.
R
Lemma 20.3. Let h be a probability measure on Rd ; then, M̄t = Rd Mt (x)dh(x)
is an F-adapted non-negative supermartingale with M̄0 = 1.
 20.1 Martingales and the Method of Mixtures 259

The following theorem is the key result from which the confidence set will be
derived.

Theorem 20.4. For all λ > 0 and δ ∈ (0, 1),

    
2 1 det(Vt (λ))
P exists t ∈ N : kSt kVt (λ)−1 ≥ 2 log + log ≤ δ.
δ λd
The proof will be given momentarily. First, though, the implications.

Theorem 20.5. Let δ ∈ (0, 1). Then, with probability at least 1 − δ, it holds that
for all t ∈ N,
s    
√ 1 det Vt (λ)
kθ̂t − θ∗ kVt (λ) < λkθ∗ k2 + 2 log + log .
δ λd

Furthermore, if kθ∗ k2 ≤ m2 , then P (exists t ∈ N+ : θ∗ ∈

/ Ct ) ≤ δ with
( s    )
√ 1 det Vt−1 (λ)
Ct = θ ∈ R : kθ̂t−1 − θkVt−1 (λ) < m2 λ + 2 log
d
+ log .
δ λd

Proof We only have to compare kSt kVt (λ)−1 and kθ̂t − θ∗ kVt (λ) :

kθ̂t − θ∗ kVt (λ) = kVt (λ)−1 St + (Vt (λ)−1 Vt − I)θ∗ kVt (λ)
≤ kSt kVt (λ)−1 + (θ∗> (Vt (λ)−1 Vt − I)Vt (λ)(Vt (λ)−1 Vt − I)θ∗ )1/2
= kSt kVt (λ)−1 + λ1/2 (θ∗> (I − Vt (λ)−1 Vt )θ∗ )1/2
≤ kSt kVt (λ)−1 + λ1/2 kθ∗ k ,

and the result follows from Theorem 20.4.

Proof of Theorem 20.4 Let H = λI ∈ Rd×d and h = N (0, H −1 ) and

Z
M̄t = Mt (x)dh(x)
Rd
Z  
1 1 1
=p exp hx, St i − kxk2Vt − kxk2H dx .
(2π)d det(H −1 ) Rd 2 2

By Lemma 20.3, M̄t is a non-negative supermartingale, and thus the maximal

inequality (Theorem 3.9) shows that
    
1 1
P sup log(M̄t ) ≥ log = P sup M̄t ≥ ≤ δ. (20.7)
t∈N δ t∈N δ

Now we turn to studying M̄t . Completing the square in the definition of M̄t
we get
1 1 1 1
hx, St i − kxk2Vt − kxk2H = kSt k2(H+Vt )−1 − kx − (H + Vt )−1 St k2H+Vt .
2 2 2 2
The first term kSt k2(H+Vt )−1 does not depend on x and can be moved outside the
 20.2 Notes 260

integral, which leaves a quadratic ‘Gaussian’ term that may be integrated exactly
and results in
 1/2  
det(H) 1 2
M̄t = exp kSt k(H+Vt )−1 . (20.8)
det(H + Vt ) 2
The result follows by substituting this expression into Eq. (20.7) and
rearranging.

20.2 Notes

1 Recall from the previous chapter that when kAt k2 ≤ L is assumed, then
  d  d
det Vt (λ) Vt (λ) nL2
≤ trace ≤ 1+ . (20.9)
λd λd λd
In general, the log determinant form should be preferred when confidence
intervals are used as part of an algorithm, but the right-hand side has a
concrete form that can be useful when stating regret bounds.
2 Plugging the bounds of the previous note into Theorem 20.5 and choosing
λ = 1 gives the confidence set
( s    )
1 nL 2
Ct = θ ∈ Rd : kθ̂t−1 − θkVt−1 (1) < m2 + 2 log + d log 1 + .
δ d
p
The dependence of the radius on n, d and δ, up to constants and a log(n)
factor, is the same as what we got in the fixed design case (cf. Eq. (20.3)),
which suggests that Theorem 20.5 can be quite tight. By considering the case
when each basis vector {e1 , . . . , ed } is played m times, then D = kθ̂t − θk2Vt
is distributed like a chi-squared distribution with d degrees of freedom. From
this, we see that the first term under the square root with the coefficient two
is stemming from variance of the noise, while the term that involved d log(n) √
is the bias (the expected value of D). In particular, this shows that the d
factor cannot be avoided.
3 If either of the above confidence sets is used (either the one from the theorem,
or that from Eq. (20.3)) to derive confidence bounds for the prediction error
h√θ̂t − θ, xi at some fixed x ∈ Rd , we get a confidence width that scales with
d (e.g., Eq. (19.13)), unlike the confidence width in Eq. (20.2), which is
independent of d. It follows that if one is interested in high-probability bounds
for the mean at a fixed input x, one should avoid going through a confidence
set for the whole parameter vector. What this leaves open is whether a bound
like in Eq. (20.2) is possible at a fixed input x, but with a sequential design.
In Exercise 20.2 you will answer this question in the negative. First note that
when the actions are chosen using a fixed design, integrating Eq. (20.2) shows
 20.3 Bibliographic Remarks 261

that E[hθ̂t − θ∗ , xi2 /kxk2V −1 ] = O(1). In the exercise, you will show that there
t
exists a sequential design such that
h i
E hθ̂t − θ∗ , xi2 /kxk2V −1 = Ω(d) ,
t
√
showing that for some sequential designs the factor d is necessary. It remains
an interesting open question to design confidence bounds for sequential design
for fixed x that adapts to the amount of dependence in the design.
4 Supermartingales arise naturally in proofs relying on the Cramér–Chernoff
method. Just one example is the proof of Lemma 12.2. One could rewrite
most of the proofs involving sums of random variables relying on the Cramér–
Chernoff method in a way that it would become clear that the proof hinges on
the supermartingale property of an appropriate sequence.

20.3 Bibliographic Remarks

Bounds like those given in Theorem 20.5 are called self-normalised bounds [de la
Peña et al., 2008]. The method of mixtures goes back to the work by Robbins
and Siegmund [1970]. In practice, the improvement provided by the method of
mixtures relative to the covering arguments is quite large. A historical account
of martingale methods in sequential analysis is by Lai [2009]. A simple proof of
Lemma 20.1 appears as lemma 2.5 in the book by van de Geer [2000]. Calculating
covering numbers (or related packing numbers) is a whole field by itself, with
open questions even in the most obvious examples. The main reference is by
Rogers [1964], which by now is a little old, but still interesting.

20.4 Exercises

20.1 (Lower bounds for fixed design) Let n = md for integer m and
A1 , . . . , An be a fixed design where each basis vector in {e1 , . . . , ed } is played
exactly m times. Then let (ηt )nt=1 be a sequence of independent standard Gaussian
random variables and Xt = hθ∗ , At i + ηt . Finally, let θ̂n be the ordinary least
squares estimator of θ∗ ∈ Rd . Show that
h i
E kθ̂n − θ∗ k2Vn = d .

This exercise shows that the d-dependence in Eq. (20.3) is unavoidable in

general for a self-normalised bound, even in the fixed design setting.

20.2 (Lower bounds for sequential design) Let n ≥ 2d and (ηt )nt=1 be a
sequence of independent standard Gaussian random variables. Find a sequence of
 20.4 Exercises 262

Pn
random vectors (At )nt=1 , with At ∈ Rd such that Vn = t=1 At A> t is invertible
almost surely and At is σ(A1 , η1 , . . . , At−1 , ηt−1 )-measurable for all t and
h i
E hθ̂n , 1i2 /k1k2V −1 ≥ cd ,
n

Pn
where c > 0 is a universal constant and Sn = t=1 ηt At and θ̂n = Vn−1 Sn .

Hint Choose A1 , . . . , Ad to be the standard basis vectors. Subsequently choose

selected basis vectors adaptively to push the estimate of hθ̂n , 1i away from zero.
For Exercise 20.4, where we ask you to prove Lemma 20.1, a few standard
definitions will be useful.

Definition 20.6 (Covering and packing). Let A ⊂ Rd . A subset C ⊂ A is said to

be an ε-cover of A if A ⊂ ∪x∈C B(x, ε), where B(x, ε) = {y ∈ Rd : kx − yk ≤ ε}
is the ε ball centered at x. An ε-packing of A is a subset P ⊂ A such that for any
x, y ∈ P, kx − yk > ε (note the strict inequality). The ε-covering number of A
is N (A, ε) = min{|C| : C is an ε-covering of A}, while the ε-packing number
of A is M (A, ε) = max{|P| : P is an ε-packing of A}, where we allow for both
the covering and packing numbers to take on the value of +∞.

The definitions can be repeated for pseudo-metric spaces. Let X be a set and
d : X × X → [0, ∞) be a function that is symmetric, satisfies the triangle
inequality and for which d(x, x) = 0 for all x ∈ X. Note that d(x, y) = 0 is
allowed for distinct x and y, so d need not be a metric. The basic results
concerning covering and packing stated in the next exercise remain valid
with this more general definition. In applications we often need the logarithm
of the covering and packing numbers, which are called the metric entropy
of X at scale ε. As we shall see, these are often close no matter whether we
consider packing or covering.

20.3 (Coverings and packings) Let A ⊂ Rd , B be the unit ball of Rd and

vol(·) the usual volume (measure under the Lebesgue measure). For brevity let
N (ε) = N (A, ε) and M (ε) = M (A, ε). Show that the following hold:

(a) ε → N (ε) is increasing as ε ≥ 0 is decreasing.

(b) M (2ε) ≤ N (ε) ≤ M (ε).
(c) We have
 d  d
1 vol(A) vol(A + 2ε B) (∗) vol( 32 A) 3 vol(A)
≤ N (ε) ≤ M (ε) ≤ ≤ ≤ ,
ε vol(B) vol( 2 B)
ε
vol( 2 B)
ε
ε vol(B)

where (∗) holds under the assumption that εB ⊂ A and that A is convex
and for U, V ⊂ Rd , c ∈ R, U + V = {u + v : u ∈ U , v ∈ V } and
cU = {cu : u ∈ U };
 20.4 Exercises 263

(d) Fix ε > 0. Then N (ε) < +∞ if and only if A is bounded. The same holds
for M (ε).

20.4 Use the results of the previous exercise to prove Lemma 20.1.

20.5 Prove Lemma 20.3.

Hint Use the ‘sections’ lemma [Kallenberg, 2002, Lemma 1.26] to established
that M̄t is Ft -measurable.
20.6 (Hoeffding–Azuma) Let X1 , . . . , Xn be a sequence of random variables
adapted to a filtration F = (Ft )t . Suppose that |Xt | ∈ [at , bt ] almost surely for
arbitrary fixed sequences (at ) and (bt ) with at ≤ bt for all t ∈ [n]. Show that for
any ε > 0,
!  
2n2 ε2
Xn
P (Xt − E[Xt | Ft−1 ]) ≥ ε ≤ exp − Pn .
t=1 (bt − at )
2
t=1

Hint It may help to recall Hoeffding’s lemma from Note 4 in Chapter 5, which
states that for a random variable X ∈ [a, b], the moment-generating function
satisfies

MX (λ) ≤ exp(λ2 (b − a)2 /8) .

20.7 (Extension of Hoeffding–Azuma) The following simple extension

of Hoeffding–Azuma is often useful. Let n ∈ N+ and (at ) and (bt ) be fixed
sequences with at ≤ bt for all t ∈ [n]. Let X1 , . . . , Xn be a sequence of random
variables adapted to a filtration F = (Ft )t and A be an event. Assume that
P (exists t ∈ [n] : A and Xt ∈
/ [at , bt ]) = 0 and ε > 0, and show that
!  
2n2 ε2
Xn
(a) P A ∩ (Xt − E[Xt | Ft−1 ]) ≥ ε ≤ exp − Pn .
t=1 (bt − at )
2
t=1
!  
2n2 ε2
Xn
(b) P (Xt − E[Xt | Ft−1 ]) ≥ ε ≤ P (A ) + exp − Pn
c
.
t=1 (bt − at )
2
t=1

The utility of this result comes from the fact that very often the range of
some adapted sequence is itself random and could be arbitrarily large with
low probability (when A does not hold). A reference for the above result is
the survey by McDiarmid [1998].

20.8 Let δ ∈ (0, 1) and F = (Ft )∞

t=1 be a filtration and (Xt )t=1 be F-adapted
∞

such that

for all λ ∈ R, E[exp(λXt ) | Ft−1 ] ≤ exp(λ2 σ 2 /2) a.s.

 20.4 Exercises 264

Pn
Let Sn = t=1 Xt . Show that
 v !
u √
u tσ 2 + 1 
P exists t : |St | ≥ t2σ 2 (t + 1) log
 ≤ δ.
δ

20.9 (Law of the iterated logarithm and method of mixtures) This

exercise uses the same notation as Exercise 20.8. Let f be a probability density
function supported on [0, ∞) and
Z ∞  
λ2 n
Mn = f (λ) exp λSn − dλ .
0 2

(a) Show that argmaxλ∈R λSn − λ2 n/2 = Sn /n.

(b) Suppose that f (λ) is decreasing for λ > 0. Show that for any ε > 0 and
Λn = Sn /n, that,
 
(1 − ε2 )S 2
Mn ≥ εΛn f (Λn (1 + ε)) exp
2n

(c) Use the previous result to show that for any δ ∈ (0, 1),
s     !
2n 1 1
P exists n : Sn ≥ inf log + log ≤ δ.
ε>0 (1 − ε2 ) δ εΛn f (Λn (1 + ε))
R∞
(d) Find an f such that 0
f (λ)dλ = 1 and f (λ) ≥ 0 for all λ ∈ R and
   
1 1
log = (1 + o(1)) log log
λf (λ) λ
as λ → 0.
(e) Use the previous results to show that
!
Sn
P lim sup p ≤1 = 1.
n→∞ 2n log log(n)

The last part of the previous exercises is one-half of the statement of the
law of iterated logarithm, which states that
Sn
lim sup p =1 almost surely .
n→∞ 2n log log(n)
In other words, the magnitude of the largest fluctuations of the partial sum
√
(Sn )n is almost surely of the order 2n log log n as n → ∞.

20.10 Let F = (Ft )nt=0 be a filtration and X1 , X2 , . . . , Xn be a sequence of

F-adapted random variables with Xt ∈ {−1, 0, 1} and µt = E[Xt | Ft−1 , Xt 6= 0],
 20.4 Exercises 265

which we define to be zero whenever P (Xt 6= 0 | Ft−1 ) = 0. Then, with St =

Pt Pt
s=1 (Xs − µs |Xs |) and Nt = s=1 |Xs |,
 s 
 √ 
c N
P exists t ≤ n : |St | ≥ 2Nt log and Nt > 0 ≤ δ ,
t
δ

where c > 0 is a universal constant.

This result
p appeared
√ in a paper by the authors and others with the constant
c = 4 2/π/ erf( 2) ≈ 3.43 [Lattimore et al., 2018].

20.11 (Sequential likelihood ratios and confidence sets) Let (Θ, G)

be a measurable space and (Pθ : θ ∈ Θ) be a probability kernel from (Θ, G)
to (R, B(R)). Assume there exists a common measure µ such that Pθ  µ for
all θ ∈ Θ, and let {pθ : θ ∈ Θ} be a family of densities with pθ = dPθ /dµ.
You may assume that pθ (x) is jointly measurable in θ and x. Such a choice
is guaranteed to exist, as we explain in Note 8 at the end of Chapter 34.
Fix θ∗ ∈ Θ and let (Xt )∞ t=1 be a sequence of independent random variables
with law Pθ∗ . Let θ̂t ∈ Θ be σ(X1 , . . . , Xt )-measurable. For θ ∈ Θ, define
Pt
Lt (θ) = s=1 log(pθ̂s−1 (Xs )/pθ (Xs )).


(a) Show that P supt≥1 Lt (θ∗ ) ≥ log(1/δ) ≤ δ for any δ ∈ (0, 1).
(b) Show that Θt = {θ : Lt (θ) < log(1/δ)} is a sequence of confidence sets such
that P (exists t ∈ N such that θ∗ 6∈ Θt ) ≤ δ.
(c) Let Θ = R and G = B(R) and Pθ = N (0, θ) and µ be the Lebesgue measure.
Pt
Then let θ̂t = 0 for t = 0 and 1t s=1 Xs otherwise. Write an expression
for Ct , and investigate how it compares to the usual confidence intervals for
Gaussian random variables.

Hint Use Cramér–Chernoff method and observe that (exp(Lt (θ∗ )))∞
t=1 is a
martingale.

The quantities pθ0 (Xs )/pθ (Xs ) are called likelihood ratios. That the
product of likelihood ratios forms a martingale is a cornerstone result of
classical parametric statistics. The sequential form that appears in the above
exercise is based on Lemma 2 of Lai and Robbins [1985], who cite Robbins
and Siegmund [1972] as the original source.
21 Optimal Design for Least Squares
Estimators

In the preceding chapter, we showed how to construct confidence intervals for

least squares estimators when the design is chosen sequentially. We now study
the problem of choosing actions for which the resulting confidence sets are small.
This plays an important role in the analysis of stochastic linear bandits with
finitely many arms (Chapter 22) and adversarial linear bandits (Part VI).

21.1 The Kiefer–Wolfowitz Theorem

Let η1 , . . . , ηn be a sequence of independent 1-subgaussian random variables and

a1 , . . . , an ∈ Rd be a fixed sequence with span(a1 , . . . , an ) = Rd and X1 , . . . , Xn
be given by Xt = hθ∗ , at i + ηt for some θ∗ ∈ Rd . The least squares estimator of
Pn Pn
θ∗ is θ̂ = V −1 t=1 at Xt with V = t=1 at a> t .

The least squares estimator used here is not regularised. This eases the
calculations, and the lack of regularisation will not harm us in future
applications.

Eq. (20.2) from Chapter 20 shows that for any a ∈ Rd and δ ∈ (0, 1),
s  !
2 1
P hθ̂ − θ∗ , ai ≥ 2kakV −1 log ≤ δ. (21.1)
δ

For our purposes, both a1 , . . . , an and a will be actions from some (possibly
infinite) set A ⊂ Rd and the question of interest is finding the shortest sequence
of exploratory actions a1 , . . . , an such that the confidence bound in the previous
display is smaller than some threshold for all a ∈ A. To solve this exactly
is likely an intractable exercise in integer programming. Finding an accurate
approximation turns out to be efficient for a broad class of action sets, however.
P
Let π : A → [0, 1] be a distribution on A so that a∈A π(a) = 1 and V (π) ∈ Rd×d
and g(π) ∈ R be given by
X
V (π) = π(a)aa> , g(π) = max kak2V (π)−1 . (21.2)
a∈A
a∈A
 21.1 The Kiefer–Wolfowitz Theorem 267

In the subfield of statistics called optimal experimental design, the distribution

π is called a design, and the problem of finding a design that minimises g is
called the G-optimal design problem. So how to use this? Suppose that π is
a design and a ∈ Supp(π) and
  
π(a)g(π) 1
na = 2
log . (21.3)
ε δ
Then, choosing each action a ∈ Supp(π) exactly na times ensures that
 
X g(π) 1
V = na aa ≥ 2 log
>
V (π) ,
ε δ
a∈Supp(π)

which by Eq. (21.1) means that for any a ∈ A, with probability 1 − δ,

s  
2 1
hθ̂ − θ∗ , ai ≤ 2kakV −1 log ≤ ε.
δ

By Eq. (21.3), the total number of actions required to ensure a confidence width
of no more than ε is bounded by
X X  π(a)g(π)  
1 g(π)
 
1
n= na = 2
log ≤ | Supp(π)| + 2 log .
ε δ ε δ
a∈Supp(π) a∈Supp(π)

The set Supp(π) is sometimes called the core set. The following theorem
characterises the size of the core set and the minimum of g.

Theorem 21.1 (Kiefer–Wolfowitz). Assume that A ⊂ Rd is compact and

span(A) = Rd . The following are equivalent:

(a) π ∗ is a minimiser of g.
(b) π ∗ is a maximiser of f (π) = log det V (π).
(c) g(π ∗ ) = d.

Furthermore, there exists a minimiser π ∗ of g such that | Supp(π ∗ )| ≤ d(d + 1)/2.

A design that maximises f is known as a D-optimal design, and thus the

theorem establishes the equivalence of G-optimal and D-optimal designs.

Proof We give the proof for finite A. The general case follows by passing to the
limit (Exercise 21.3). When it is convenient, distributions π on A are treated as
vectors in R|A| . You will show in Exercises 21.1 and 21.2 that f is concave and
that

(∇f (π))a = kak2V (π)−1 . (21.4)

Also notice that

!
X X
π(a)kak2V (π)−1 = trace π(a)aa V (π)
> −1
= trace(I) = d . (21.5)
a∈A a
 21.1 The Kiefer–Wolfowitz Theorem 268

(b)⇒(a): Suppose that π ∗ is a maximiser of f . By the first-order optimality

criterion (see Section 26.5), for any π distribution on A,
0 ≥ h∇f (π ∗ ), π − π ∗ i
X X
= π(a)kak2V (π∗ )−1 − π ∗ (a)kak2V (π∗ )−1
a∈A a∈A
X
= π(a)kak2V (π∗ )−1 − d.
a∈A

For an arbitrary a ∈ A, choosing π to be the Dirac at a ∈ A proves that

kak2V (π∗ )−1 ≤ d. Hence g(π ∗ ) ≤ d. Since g(π) ≥ d for all π by Eq. (21.5), it follows
that π ∗ is a minimiser of g and that minπ g(π) = d. (c) =⇒ (b): Suppose that
g(π ∗ ) = d. Then, for any π,
X
h∇f (π ∗ ), π − π ∗ i = π(a)kakV (π∗ )−1 − d ≤ 0 .
a∈A

And it follows that π is a maximiser of f by the first-order optimality conditions

∗

and the concavity of f . That (a) =⇒ (c) is now trivial. To prove the second
part of the theorem, let π ∗ be a minimiser of g, which by the previous part is a
maximiser of f . Let S = Supp(π ∗ ), and suppose that |S| > d(d + 1)/2. Since the
dimension of the subspace of d × d symmetric matrices is d(d + 1)/2, there must
be a non-zero function v : A → R with Supp(v) ⊆ S such that
X
v(a)aa> = 0 . (21.6)
a∈S

Notice that for any a ∈ S, the first-order optimality conditions ensure that
kak2V (π∗ )−1 = d (Exercise 21.5). Hence
X X
d v(a) = v(a)kak2V (π∗ )−1 = 0 ,
a∈S a∈S

where the last equality follows from Eq. (21.6). Let π(t) = π ∗ + tv and let
P
τ = max{t > 0 : π(t) ∈ PA }, which exists since v 6= 0 and a∈S v(a) = 0
and Supp(v) ⊆ S. By Eq. (21.6), V (π(t)) = V (π ∗ ), and hence f (π(τ )) = f (π ∗ ),
which means that π(τ ) also maximises f . The claim follows by checking that
| Supp(π(T ))| < | Supp(π ∗ )| and then using induction.

Geometric Interpretation
There is a geometric interpretation of the D-optimal design problem. Let π be a
P
D-optimal design for A and V = a∈A π(a)aa> and

E = x ∈ Rd : kxk2V −1 ≤ d ,
which is a centered ellipsoid. By Theorem 21.1, it holds that A ⊂ E with the core
set lying on the boundary (see Fig. 21.1). As you might guess from the figure,
the ellipsoid E is the minimum volume centered ellipsoid containing A. This is
known to be unique and the optimisation problem that characterises it is in fact
the dual of the log determinant problem that determines the D-optimal design.
 21.2 Notes 269

Figure 21.1 The minimum volume centered ellipsoid containing a point cloud. The points
on the boundary are the core set. The ellipse is E = {x : kxk2V (π)−1 = d}, where π is an
optimal design.

21.2 Notes

1 The letter ‘d’ in D-optimal design comes from the determinant in the objective.
The ‘g’ in G-optimal design stands for ‘globally optimal’. The names were coined
by Kiefer and Wolfowitz, though both problems appeared in the literature
before them.
2 In applications we seldom need an exact solution to the design problem. Finding
a distribution π such that g(π) ≤ (1 + ε)g(π ∗ ) will increase the regret of our
algorithms by a factor of just (1 + ε)1/2 .
3 The computation of an optimal design for finite action sets is a convex problem
for which there are numerous efficient approximation algorithms. The Frank–
Wolfe algorithm is one such algorithm, which can be used to find a near-optimal
solution for modestly sized problems. The algorithm starts with an initial π0
and updates according to

πk+1 (a) = (1 − γk )πk (a) + γk I {ak = a} , (21.7)

where ak = argmaxa∈A kak2V (πk )−1 and the step size is chosen to optimise f
along the line connecting πk and δak .
1 2
d kak kV (πk )−1 − 1
γk = argmaxγ∈[0,1] f ((1 − γ)πk + γδak ) = . (21.8)
kak k2V (πk )−1 − 1

If π0 is chosen to be the uniform distribution over A, then the number of

iterations before g(πk ) ≤ (1 + ε)g(π ∗ ) is at most O(d log log |A| + d/ε). For a
slightly more sophisticated choice of initialisation the dependence on |A| can
be eliminated entirely. More importantly, this other initialisation has a core
set of size O(d) and running the algorithm in Eq. (21.7) for just O(d log log d)
iterations is guaranteed to produce a design with g(π) ≤ 2g(π ∗ ) and a core set
of size O(d log log d).
 21.3 Bibliographic Remarks 270

4 If the action set is infinite, then approximately optimal designs can sometimes
still be found efficiently. Unfortunately the algorithms in the infinite case tend
to be much ‘heavier’ and less practical.
5 The smallest ellipsoid containing some set K ⊂ Rd is called the minimum
volume enclosing ellipsoid (MVEE) of K. As remarked, the D-optimal
design problem is equivalent to finding an MVEE of A with the added constraint
that the ellipsoid must be centred – or equivalently, finding the MVEE of the
symmetrised set A ∪ {−a : a ∈ A}. The MVEE of a convex set is also called
John’s ellipsoid, which has many applications in optimisation and beyond.
6 In Exercise 21.6, you will generalise Kiefer–Wolfowitz theorem to sets that do
not span Rd . When A is compact and dim(span(A)) = m ∈ [d], then there
exists a distribution π ∗ supported on at most m(m + 1)/2 points of A and for
which g(π ∗ ) = m = inf π g(π).

21.3 Bibliographic Remarks

The Kiefer–Wolfowitz theorem is due to Kiefer and Wolfowitz [1960]. The

algorithm in Note 3 is due to Fedorov [1972]. A similar variant was also proposed
by Wynn [1970], and the name Wynn’s method is sometimes used. The algorithm
is a specialisation of Frank–Wolfe’s algorithm, which was originally intended for
quadratic programming [Frank and Wolfe, 1956]. Todd [2016] wrote a nice book
about minimum volume ellipsoids and related algorithms, where you can also find
many more references and improvements to the basic algorithms. Chapter 3 of his
book also includes discussion of alternative initialisations and convergence rates
for various algorithms. The duality between D-optimal design and the MVEE
problem was shown by Silvey and Sibson [1972]. Although the connection between
minimum volume ellipsoids and experimental design is well known, previous
applications of these results to bandits used John’s theorem without appropriate
symmetrisation, which made the resulting arguments more cumbersome. For
more details on finding approximately optimal designs for infinite sets, see the
article by Hazan et al. [2016], references there-in and the book by Grötschel et al.
[2012].

21.4 Exercises

21.1 (Derivative of log determinant) Prove the correctness of the derivative

in Eq. (21.4).

Hint For square matrix A let adj(A) be the transpose of the cofactor matrix
of A. Use the facts that the inverse of a matrix A is A−1 = adj(A)> / det(A) and
 21.4 Exercises 271

that if A : R → Rd×d , then

 
d d
det(A(t)) = trace adj(A) A(t) .
dt dt
21.2 (Concavity of log determinant) Prove that H 7→ log det(H) is concave
where H is a symmetric, positive definite matrix.

Hint Consider t 7→ log det(H + tZ) for Z symmetric, and show that this is a
concave function.
21.3 (Kiefer–Wolfowitz for compact sets) Generalise the proof of
Theorem 21.1 to compact action sets.

21.4 Prove the second inequality in Eq. (21.8).

21.5 Let π ∗ be a G-optimal design and a ∈ Supp(π ∗ ). Prove that kak2V (π∗ )−1 = d.

21.6 Prove that if A is compact and dim(span(A)) = m ∈ [d], then there exists
a distribution π ∗ over A supported on at most m(m + 1)/2 points and for which
g(π ∗ ) = m.

21.7 (Implementation) Write a program that accepts as parameters a finite

set A ⊂ Rd and returns a design π : A → [0, 1] such that g(π) ≤ d + ε for some
given ε > 0. How robust is your algorithm? Experiment with different choices of
A and d, and report your results.

Hint The easiest pure way to do this is to implement the Frank–Wolfe algorithm
described in Note 3. All quantities can be updated incrementally using rank-one
update formulas, and this will lead to a significant speedup. You might like to
read the third chapter of the book by Todd [2016] and experiment with the
proposed variants.
22 Stochastic Linear Bandits with
Finitely Many Arms

The optimal design problem from the previous chapter has immediate applications
to stochastic linear bandits. In Chapter 19, we developed a linear version of the
√
upper confidence bound algorithm that achieves a regret of Rn = O(d n log(n)).
The only required assumptions were that the sequence of available action sets
were bounded. In this short chapter, we consider a more restricted setting where:

1 the set of actions available in round t is A ⊂ Rd and |A| = k for some natural
number k;
2 the reward is Xt = hθ∗ , At i + ηt where ηt is conditionally 1-subgaussian:

E[exp(ληt )|A1 , η1 , . . . , At−1 ] ≤ exp(λ2 /2) almost surely for all λ ∈ R; and

3 the suboptimality gaps satisfy ∆a = maxb∈A hθ∗ , b − ai ≤ 1 for all a ∈ A.

The key difference relative to Chapter 19 is that now the set of actions is finite
and does not change with time. Under these conditions, it becomes possible to
design a policy such that
p 
Rn = O dn log(nk) .

For moderately sized k, this bound improves the regret by a factor of d1/2 , which
in some regimes is large enough to be worth the effort. The policy is an instance
of phase-based elimination algorithms. As usual, at the end of a phase, arms that
are likely to be suboptimal with a gap exceeding the current target are eliminated.
In fact, this elimination is the only way the data collected in a phase is being
used. In particular, the actions to be played during a phase are chosen based
entirely on the data from previous phases: the data collected in the present phase
do not influence which actions are played. This decoupling allows us to make use
of the tighter confidence bounds available in the fixed design setting, as discussed
in the previous chapter. The choice of policy within each phase uses the solution
to an optimal design problem to minimise the number of required samples to
eliminate arms that are far from optimal.

Theorem 22.1. With probability at least 1 − δ, the regret of Algorithm 12 satisfies

s  
k log(n)
Rn ≤ C nd log ,
δ
 22.1 Notes 273

Input A ⊂ Rd and δ
Step 0 Set ` = 1 and let A1 = A
Step 1 Let t` = t be the current timestep and find G-optimal design π` ∈ P(A` )
with Supp(π` ) ≤ d(d + 1)/2 that maximises
X
log det V (π` ) subject to π` (a) = 1
a∈A`

Step 2 Let ε` = 2−` and

  
2dπ` (a) k`(` + 1) X
T` (a) = 2 log and T` = T` (a)
ε` δ
a∈A`

Step 3 Choose each action a ∈ A` exactly T` (a) times

Step 4 Calculate the empirical estimate:
+T`
t`X X
θ̂` = V`−1 At Xt with V` = T` (a)aa>
t=t` a∈A`

Step 5 Eliminate low rewarding arms:

 
A`+1 = a ∈ A` : maxhθ̂` , b − ai ≤ 2ε` .
b∈A`

Step 6 ` ← ` + 1 and Goto Step 1

Algorithm 12: Phased elimination with G-optimal exploration.

p
where C > 0 is a universal constant. If δ = O(1/n), then E[Rn ] ≤ C nd log(kn)
for an appropriately chosen universal constant C > 0.

The proof of this theorem follows relatively directly from the high-probability
correctness of the confidence intervals used to eliminate low-rewarding arms. We
leave the details to the reader in Exercise 22.1.

22.1 Notes

1 The assumption that the action set does not change is crucial for Algorithm 12.
Several complicated algorithms have been proposed and analysed for the case
where At is allowed to change from round to round under the assumption that
|At | ≤ k for all rounds. For these algorithms, it has been proven that
q 
Rn = O nd log3 (nk) . (22.1)

When k is small, these

√ results improve on the bound for LinUCB in Chapter 19
by a factor of up to d.
2 Algorithm 12 can be adapted to the case where k is infinite by using confidence
 22.2 Bibliographic Remarks 274

intervals derived in Chapter 20. Once the dust has settled, you should find the
regret is
 p 
Rn = O d n log(n) .

3 One advantage of Algorithm 12 is that it behaves well even when the linear
model is misspecified. Suppose the reward is Xt = hθ, At i + ηt + f (At ), where
ηt is noise as usual and f : A → R is some function with kf k∞ ≤ ε. Then the
regret of Algorithm 12 can be shown to be
p √ 
Rn = O dn log(nk) + nε d log(n) .

The linear dependence

√ on the horizon should be expected when k is large. The
presence of d in the second term is unfortunate, but unavoidable in many
regimes as discussed by Lattimore and Szepesvári [2019b].

22.2 Bibliographic Remarks

The algorithms achieving Eq. (22.1) for changing action sets are SupLinRel [Auer,
2002] and SupLinUCB [Chu et al., 2011]. Both introduce phases to decouple
the dependence of the design on the outcomes. Unfortunately the analysis of
these algorithms is long and technical, which prohibited us from presenting
the ideas here. These algorithms are also not the most practical relative to
LinUCB (Chapter 19) or Thompson sampling (Chapter 36). Of course this does
not diminish the theoretical breakthrough. Phased elimination algorithms have
appeared in many places, but the most similar to the algorithm presented here
is the work on spectral bandits by Valko et al. [2014] (and we have also met
them briefly in earlier chapters on finite-armed bandits). None of the works just
mentioned used the Kiefer–Wolfowitz theorem. This idea is apparently new, but
it is based on the literature on adversarial linear bandits where John’s ellipsoid
has been used to define exploration policies [Bubeck et al., 2012]. For more details
on adversarial linear bandits, read on to Part VI.
Ghosh et al. [2017] address misspecified (stochastic) linear bandits with a fixed
action set. In misspecified linear bandits, the reward is nearly a linear function of
the feature vectors associated with the actions. Ghosh et al. [2017] demonstrate
that in the favourable case when one can cheaply test linearity, an algorithm
that first runs a test and then switches to either
√ a linear bandit or a finite-armed
√
bandit based on the outcome will achieve ( k ∧ d) n regret up to log factors.
We will return to misspecified linear bandits a few more times in the book.

22.3 Exercises

22.1 In this exercise, you will prove Theorem 22.1.

 22.3 Exercises 275

(a) Use Theorem 21.1 to show that the length of the `th phase is bounded by
 
2d k`(` + 1) d(d + 1)
T` ≤ 2 log + .
ε` δ 2
(b) Let a∗ ∈ argmaxa∈A hθ∗ , ai be the optimal arm and use Eq. (21.1) to show
that
δ
P (exists phase ` such that a∗ ∈
/ A` ) ≤ .
k
(c) For action a define `a = min{` : 2ε` < ∆a } to be the first phase where the
suboptimality gap of arm a is smaller than 2ε` . Show that
δ
P (a ∈ A`a ) ≤ .
k
(d) Show that with probability at least 1 − δ the regret is bounded by
s  
k log(n)
Rn ≤ C dn log ,
δ
where C > 0 is a universal constant.
(e) Show that this implies Theorem 22.1 for the given choice of δ.

22.2 (Misspecified linear bandits) Assume the reward satisfies Xt =

hθ, At i + ηt + f (At ), where ηt is 1-subgaussian noise as usual and f : A → R is
some function with kf k∞ ≤ ε, show that the expected regret of Algorithm 12
with the choice δ = 1/n is
p √ 
Rn = O dn log(nk) + nε d log(n) .
23 Stochastic Linear Bandits with
Sparsity

In Chapter 19 we showed the linear variant of UCB has regret bounded by

√
Rn = O(d n log(n)) ,
which for fixed finite action sets can be improved to
p
Rn = O( dn log(nk)) .
For moderately sized action sets, these approaches lead to a big improvement
over what could be obtained by using the policies that do not make use of the
linear structure.
The situation is still not perfect, though. In typical applications, the features
are chosen by the user of the system, and one can easily imagine there are many
candidate features and limited information about which will be most useful. This
presents the user with a challenging trade-off. If they include many features, then
d will be large, and the algorithm may be slow to learn. But if a useful feature is
omitted, then the linear model will almost certainly be quite wrong. Ideally, one
should be able to add features without suffering much additional regret if the
added feature does not contribute in a significant way. This can be captured by
the notion of sparsity, which is the central theme of this chapter.

23.1 Sparse Linear Stochastic Bandits

Like in the standard stochastic linear bandit setting, at the beginning of round t,
the learner receives a decision set At ⊂ Rd . They then choose an action At ∈ At
and receive a reward
Xt = hθ∗ , At i + ηt , (23.1)
where (ηt )t is zero-mean noise and θ∗ ∈ Rd is an unknown vector. The only
difference in the sparse setting is that the parameter vector θ∗ is assumed to have
many zero entries. For θ ∈ Rd let
d
X
kθk0 = I {θi 6= 0} ,
i=1

which is sometimes called the zero-‘norm’ (quotations because it is not really a

norm; see Exercise 23.1).
 23.2 Elimination on the Hypercube 277

Assumption 23.1. The following hold:

(a) (Sparse parameter) There exist known constants m0 and m2 such that
kθ∗ k0 ≤ m0 and kθ∗ k2 ≤ m2 .
(b) (Bounded mean rewards) hθ∗ , ai ≤ 1 for all a ∈ At and all rounds t.
(c) (Subgaussian noise) The noise is conditionally 1-subgaussian:
for all λ ∈ R, E[exp(ληt ) | Ft−1 ] ≤ exp(λ2 /2) a.s. ,
where Ft = σ(A1 , X1 , . . . , At , Xt , At+1 ).
Much ink has been spilled on what can be said about the speed of learning in
linear models like (23.1) when (At )t are passively generated and the parameter
vector is known to be sparse. Most results are phrased about recovering θ∗ , but
there also exist a few results that quantify the error when predicting Xt . The
ideal outcome would be that the learning speed depends mostly on m0 , with only
a mild dependence on d. Almost all the results come under the assumption that
the covariance matrix of the actions (At )t is well conditioned.

The condition number of a positive definite matrix A is the ratio of its

largest and smallest eigenvalues. A matrix is well conditioned if it has a
small condition number.

The details are a bit more complicated than just the conditioning, but the
main point is that the usual assumptions imposed on the covariance matrix of
the actions for passive learning are never satisfied when the actions are chosen
by a good bandit policy. The reason is simple. Bandit algorithms want to choose
the optimal action as often as possible, which means the covariance matrix will
have an eigenvector that points (approximately) towards the optimal action with
a large corresponding eigenvalue. We need some approach that does not rely on
such strong assumptions.

23.2 Elimination on the Hypercube

As a warm-up, consider the case where the action set is the d-dimensional
hypercube: At = A = [−1, 1]d . To reduce clutter, we denote the true parameter
vector by θ = θ∗ . The hypercube is notable as an action set because it enjoys
perfect separability. For each dimension i ∈ [d], the value of Ati ∈ [−1, 1] can
be chosen independently of Atj for j 6= i. Because of this, the optimal action is
a∗ = sign(θ), where

1 , if θi > 0 ;


sign(θ)i = sign(θi ) = 0 , if θi = 0 ;


−1 , if θ < 0 .
i
 23.2 Elimination on the Hypercube 278

So learning the optimal action amounts to learning the sign of θi for each
dimension. A disadvantage of this structure is that in the worst case the sign
of each θi must be learned independently, which in Chapter 24 we show leads
√
to a worst-case regret of Rn = Ω(d n). On the positive side, the separability
means that θi can be estimated in each dimension independently while paying
absolutely no price for this experimentation when θi = 0. It turns out that this
√
allows us to design a policy for which Rn = O(kθk0 n), even without knowing
the value of kθk0 .
Let Gt = σ(A1 , X1 , . . . , At , Xt ) be the σ-algebra containing information up to
time t − 1 (this differs from Ft , which also includes information about the action
chosen). Now suppose that (Ati )di=1 are chosen to be conditionally independent
given Gt−1 , and further assume for some specific i ∈ [d] that Ati is sampled from
a Rademacher distribution so that P (Ati = 1 | Gt−1 ) = P (Ati = −1 | Gt−1 ) = 1/2.
Then
   
d 
X 
E[Ati Xt | Gt−1 ] = E Ati  Atj θj + ηt   Gt−1 
j=1 
X
= θi E[A2ti | Gt−1 ] + θj E[Atj Ati | Gt−1 ] + E[Ati ηt | Gt−1 ]
j6=i

= θi ,

where the first equality is the definition of Xt = hθ, At i+ηt , the second by linearity
of expectation and the third by the conditional independence of (Ati )i and the
fact that E[Ati | Gt−1 ] = 0 and E[A2ti | Gt−1 ] = 1. This looks quite promising, but
we should also check the variance. Using our assumptions that (ηt ) is conditionally
1-subgaussian and that hθ, ai ≤ 1 for all actions a, we have
2
V[Ati Xt | Gt−1 ] = E[A2ti Xt2 | Gt−1 ] − θi2 = E[(hθ, At i + ηt ) | Gt−1 ] − θi2 ≤ 2 .
(23.2)
And now we have cause for celebration. The value of θi can be estimated by
choosing Ati to be a Rademacher random variable independent of the choices in
other dimensions. All the policy does is treat all dimensions independently. For a
particular dimension (say i), it explores by choosing Ati ∈ {−1, 1} uniformly at
random until its estimate is sufficiently accurate to commit to either Ati = 1 or
Ati = −1 for all future rounds. How long this takes depends on |θi |, but note that
if |θi | is small, then the price of exploring is also limited. The policy that results
from this idea is called selective explore-then-commit (Algorithm 13, SETC).

Theorem 23.2. There exists a universal constants C, C 0 > 0 such that the regret
of SETC satisfies
X log(n) p
Rn ≤ 3kθk1 + C and Rn ≤ 3kθk1 + C 0 kθk0 n log(n) .
|θi |
i:θi 6=0

By appealing to the central limit theorem and the variance calculation in

 23.2 Elimination on the Hypercube 279

1: Input n and d
2: Set E1i = 1 and C1i = R for all i ∈ [d]
3: for t = 1, . . . , n do
4: For each i ∈ [d] sample Bti ∼ Rademacher
5: Choose action:

Bti if 0 ∈ Cti


(∀i) Ati = 1 if Cti ⊂ (0, ∞]


−1 if C ⊂ [−∞, 0) .
ti

6: Play At and observe Xt

7: Construct empirical estimators:
t
X Pt
Esi Asi Xs
(∀i) Ti (t) = Esi θ̂ti = s=1

s=1
Ti (t)

8: Construct confidence intervals:

s 
1 1  p 
(∀i) Wti = 2 + log n 2T (t) + 1
Ti (t) Ti (t)2
i

h i
(∀i) Ct+1,i = θ̂ti − Wti , θ̂ti + Wti

9: Update exploration parameters:

(
0 if 0 ∈
/ Ct+1,i or Eti = 0
(∀i) Et+1,i =
1 otherwise .
10: end for
Algorithm 13: Selective explore-then-commit.

Eq. (23.2), we should be hopeful that the confidence intervals used by the
algorithm are sufficiently large to contain the true θi with high probability, but
this still needs to be proven.

Lemma 23.3. Define τi = n ∧ max{t : Eti = 1}, and let Fi = I {θi ∈ / Cτi +1,i } be
the event that θi is not in the confidence interval constructed at time τi . Then
P (Fi ) ≤ 1/n.

The proof of Lemma 23.3 is left until after the proof of Theorem 23.2.

Proof of Theorem 23.2 Recalling the definition of the regret and using the
fact that the optimal action is a∗ = sign(θ), we have the following regret
 23.2 Elimination on the Hypercube 280

decomposition:

n
" # d
" n #!
X X X
Rn = maxhθ, ai − E hθ, At i = n|θi | − E Ati θi . (23.3)
a∈A
t=1 i=1 t=1
| {z }
Rni

Clearly, if θi = 0, then Rni = 0. And so it suffices to bound Rni for each i

with |θi | > 0. Suppose that |θi | > 0 for some i and the failure event Fi given
in Lemma 23.3 does not occur. Then θi ∈ Cτi +1,t , and by the definition of the
algorithm, Ati = sign(θi ) for all t ≥ τi . Therefore,
" n
# " n
#
X X
Rni = n|θi | − E Ati θi = E |θi | (1 − Ati sign(θi ))
t=1 t=1

≤ 2n|θi |P (Fi ) + |θi |E [I {Fic } τi ] . (23.4)

Since τi is the first round t when 0 ∈ / Ct+1,i it follows that if Fi does not occur,
then θi ∈ Cτi ,i and 0 ∈ Cτi ,i . Thus the width of the confidence interval Cτi ,i must
be at least |θi |, and so

s 
1 1 √

2Wτi −1,i = 4 + log n 2τi − 1 ≥ |θi | ,
τi − 1 (τi − 1)2

which after rearranging shows for some universal constant C > 0 that

C log(n)
I {Fic } (τi − 1) ≤ 1 + .
θi2

Combining this result with Eq. (23.4) leads to

C log(n)
Rni ≤ 2n|θi |P (Fi ) + |θi | + .
|θi |

Using Lemma 23.3 to bound P (Fi ) and substituting into the decomposition
Eq. (23.3) completes the proof of the first part. The second part is left as a treat
for you (Exercise 23.2).

P
Proof of Lemma 23.3 Let Sti = j6=i Atj θj and Zti = Ati ηt +Ati Sti . For t ≤ τi ,

1X
t
θ̂ti − θi = Zsi .
t s=1
 23.3 Online to Confidence Set Conversion 281

√
The next step is to show that Zti is conditionally 2-subgaussian for t ≤ τi :

E [exp(λZti ) | Gt−1 ] = E [E [exp(λZti ) | Ft−1 ] | Gt−1 ]

= E [exp(λAti Sti )E [exp(λAti ηt ) | Ft−1 ] | Gt−1 ]
  2 
λ 
≤ E exp(λAti Sti ) exp Gt−1
2 
 2
λ
= exp E [E [exp(λAti Sti ) | Gt−1 , Sti ] | Gt−1 ]
2
 2   2 2  
λ λ Sti 
≤ exp E exp  Gt−1
2 2
≤ exp(λ2 ) .

The first inequality used the fact that ηt is conditionally 1-subgaussian. The second-
to-last inequality follows because Ati is conditionally Rademacher for t ≤ τi ,
which is 1-subgaussian by Hoeffding’s lemma (5.11). The final inequality follows
because Sti ≤ kAt k∞ kθk1 ≤ 1. The result follows by applying the concentration
bound from Exercise 20.8.

23.3 Online to Confidence Set Conversion

A new plan is needed to relax the assumption that the action set is a hypercube.
The idea is to modify the ellipsoidal confidence set used in Chapter 19 to have a
smaller radius. We will see that modifying the algorithm in Chapter 19 to use
√
the smaller confidence intervals improves the regret to Rn = O( dpn log(n)).

Without assumptions
√ on the action set, one cannot hope to have a regret
smaller than O( dn). To see this, recall that d-armed bandits can be
represented as linear bandits with At = {e1 , . . . , ed }. For these problems,
Theorem 15.2 shows
√ that for any policy there exists a d-armed bandit for
which Rn = Ω( dn). Checking the proof reveals that when adapted to the
linear setting the parameter vector is 2-sparse.

The construction that follows makes use of a kind of duality between online
prediction and confidence sets. While we will only apply the idea to the sparse
linear case, the approach is generic.
The prediction problem considered is online linear prediction under the
squared loss. This is also known as online linear regression. The learner
interacts with an environment in a sequential manner where in each round
t ∈ N+ :

1 The environment chooses Xt ∈ R and At ∈ Rd in an arbitrary fashion.

2 The value of At is revealed to the learner (but not Xt ).
 23.3 Online to Confidence Set Conversion 282

3 The learner produces a real-valued prediction X̂t ∈ R in some way.

4 The environment reveals Xt to the learner and the loss is (Xt − X̂t )2 .
The regret of the learner relative to a linear predictor that uses the weights
θ ∈ Rd is
n
X n
X
ρn (θ) = (Xt − X̂t )2 − (Xt − hθ, At i)2 . (23.5)
t=1 t=1

We say that the learner enjoys a regret guarantee Bn relative to Θ ⊆ Rd if for

any strategy of the environment,
sup ρn (θ) ≤ Bn . (23.6)
θ∈Θ

The online learning literature has a number of powerful techniques for this
learning problem. Later we will give a specific result for the sparse case when
Θ = {x : kxk0 ≤ m0 }, but first we show how to use such a learning algorithm
to construct a confidence set. Take any learner for online linear regression, and
assume the environment generates Xt in a stochastic manner like in linear bandits:
Xt = hθ∗ , At i + ηt . (23.7)
Combining Eqs. (23.5) to (23.7) with elementary algebra,
n
X n
X
Qt = (X̂t − hθ∗ , At i)2 = ρn (θ∗ ) + 2 ηt (X̂t − hθ∗ , At i)
t=1 t=1
n
X
≤ Bn + 2 ηt (X̂t − hθ∗ , At i) , (23.8)
t=1

where the first equality serves as the definition of Qt . Let us now take stock for a
moment. If we could somehow remove the dependence on the noise ηt in the right-
hand side, then we could define a confidence set consisting of all θ that satisfy
the equation. Of course the noise has zero mean and is conditionally independent
of its multiplier, so the expectation of this term is zero. The fluctuations can be
controlled with high probability using a little concentration analysis. Let
t
X
Zt = ηs (X̂s − hθ∗ , As i) .
s=1

Since X̂t is chosen based on information available at the beginning of the round,
X̂t is Ft−1 -measurable, and so
for all λ ∈ R, E[exp(λ(Zt − Zt−1 )) | Ft−1 ] ≤ exp(λ2 σt2 /2) ,

where σt2 = (X̂t − hθ∗ , At i)2 . The uniform self-normalised tail bound
(Theorem 20.4) with λ = 1 implies that,
s  !
1 + Qt
P exists t ≥ 0 such that |Zt | ≥ (1 + Qt ) log ≤ δ.
δ2
 23.3 Online to Confidence Set Conversion 283

Provided this low-probability event does not occur, then from Eq. (23.8) we have
s  
1 + Qt
Qt ≤ Bt + 2 (1 + Qt ) log . (23.9)
δ2
While both sides depend on Qt , the left-hand side grows linearly, while the
right-hand side grows sublinearly in Qt . This means that the largest value of Qt
that satisfies the above inequality is finite. A tedious calculation then shows this
value must be less than
√ √ 
8 + 1 + Bt
βt (δ) = 1 + 2Bt + 32 log . (23.10)
δ
By piecing together the parts, we conclude that with probability at least 1 − δ
the following holds for all t:
t
X
Qt = (X̂s − hθ∗ , As i)2 ≤ βt (δ) .
s=1

We could define Ct+1 to be the set of all θ such that the above holds with θ∗
replaced by θ, but there is one additionally subtlety, which is that the resulting
Pt
confidence interval may be unbounded (think about the case that s=1 As A> s is
not invertible). In Chapter 19 we overcame this problem by regularising the least
squares estimator. Since we have assumed that kθ∗ k2 ≤ m2 , the previous display
implies that
t
X
kθ∗ k22 + (X̂s − hθ∗ , As i)2 ≤ m22 + βt (δ) .
s=1

All together, we have the following theorem:

Theorem 23.4. Let δ ∈ (0, 1) and assume that θ∗ ∈ Θ and supθ∈Θ ρt (θ) ≤ Bt . If
( t
)
X
2 2 2
Ct+1 = θ ∈ R : kθk2 +
d
(X̂s − hθ, As i) ≤ m2 + βt (δ) ,
s=1

then P (exists t ∈ N such that θ∗ 6∈ Ct+1 ) ≤ δ .

The confidence set in Theorem 23.4 is not in the most convenient form. By
Pt Pt
defining Vt = I + s=1 As A> s and St = s=1 As X̂s and θ̂t = Vt St and
−1

performing an algebraic calculation that we leave to the reader (see Exercise 23.5),
one can see that
t
X t
X
kθk22 + (X̂s − hθ, As i)2 = kθ − θ̂t k2Vt + (X̂s − hθ̂t , As i)2 + kθ̂t k22 . (23.11)
s=1 s=1

Using this, the confidence set can be rewritten in the familiar form of an ellipsoid:
( t
)
X
2 2 2 2 2
Ct+1 = θ ∈ R : kθ − θ̂t kVt ≤ m2 + βt (δ) − kθ̂t k2 −
d
(X̂s − hθ̂t , As i) .
s=1
 23.4 Sparse Online Linear Prediction 284

1: Input Online linear predictor and regret bound Bt , confidence parameter

δ ∈ (0, 1)
2: for t = 1, . . . , n do
3: Receive action set At
4: Computer confidence set:
( t−1
)
X
2 2 2
Ct = θ ∈ R : kθk2 +
d
(X̂s − hθ, As i) ≤ m2 + βt (δ)
s=1

5: Calculate optimistic action

At = argmaxa∈At maxhθ, ai
θ∈Ct

6: Feed At to the online linear predictor and obtain prediction X̂t

7: Play At and receive reward Xt
8: Feed Xt to online linear predictor as feedback
9: end for
Algorithm 14: Online linear predictor UCB (OLR-UCB).

It is not obvious that Ct+1 is not empty because the radius could be negative.
Theorem 23.4 shows, however, that with high probability θ∗ ∈ Ct+1 . At last we
have established all the conditions required for Theorem 19.2, which implies the
following theorem bounding the regret of Algorithm 14:
Theorem 23.5. With probability at least 1 − δ the pseudo-regret of OLR-UCB
satisfies
r  n
R̂n ≤ 8dn (m22 + βn−1 (δ)) log 1 + .
d

23.4 Sparse Online Linear Prediction

Theorem 23.6. There exists a strategy π for the learner such that for any
θ ∈ Rd , the regret ρn (θ) of π against any strategic environment such that
maxt∈[n] kAt k2 ≤ L and maxt∈[n] |Xt | ≤ X satisfies
n  o
ρn (θ) ≤ cX 2 kθk0 log(e + n1/2 L) + Cn log 1 + kθk 1
kθk0 + (1 + X 2 )Cn ,

where c > 0 is some universal constant and Cn = 2 + log2 log(e + n1/2 L).
Note that Cn = O(log log(n)), so by dropping the dependence on X and L, we
have
sup ρn (θ) = O(m0 log(n)) .
θ:kθk0 ≤m0 ,kθk2 ≤L

As a final catch, the rewards (Xt ) in sparse linear bandits with subgaussian noise
 23.5 Notes 285

are not necessarily bounded. However, the subgaussian property implies that with
probability 1 − δ, |ηt | ≤ log(2/δ). By choosing δ = 1/n2 and Assumption 23.1,
we have
 
2

1
P max |Xt | ≥ 1 + log 2n ≤ .
t∈[n] n

Putting all the pieces together shows that the expected regret of OLR-UCB when
using the predictor provided by Theorem 23.6 and when kθk0 ≤ m0 satisfies
p 
Rn = O dnm0 log(n)2 .

23.5 Notes

1 The strategy achieving the bound in Theorem 23.6 is not computationally

efficient. In fact we do not know of any polynomial time algorithm with
logarithmic regret for this problem. The consequence is that Algorithm 14 does
not yet have an efficient implementation.
2 While we focused on the sparse case, the results and techniques apply to other
settings. For example, we can also get alternative confidence sets from results
in online learning even for the standard non-sparse case. Or one may consider
additional or different structural assumptions on θ.
3 When the online linear regression results are applied, it is important to use the
tightest possible, data-dependent regret bounds Bn . In online learning most
regret bounds start as tight, data-dependent bounds, which are then loosened
to get further insight into the structure of problems. For our application,
naturally one should use the tightest available regret bounds (or modify the
existing proofs to get tighter data-dependent bounds). The gains from using
data-dependent bounds can be significant.
4 The confidence set used by Algorithm 14 depends on the sparsity parameter
, which must be known in advance. No algorithm can enjoy a regret of
m0p
O( kθ∗ k0 dn) for all kθ∗ k0 simultaneously (see Chapter 24).
5 The bound in Theorem 23.5 still depends on the ambient dimension. In general
this is unavoidable, as we show in Theorem 24.3. For this reason it recently
became popular to study the contextual setting with changing actions and
make assumptions on the distribution of the contexts so that techniques from
high-dimensional statistics can be brought to bear. These approaches are still
in their infancy and deciding on the right assumptions is a challenge. The
reader is referred to the recent papers by Kim and Paik [2019] and Bastani
and Bayati [2020].
 23.6 Bibliographical Remarks 286

23.6 Bibliographical Remarks

The selective explore-then-commit algorithm is due to the authors [Lattimore et al.,

2015]. The construction for the sparse case is from another paper co-authored
by one of the authors [Abbasi-Yadkori et al., 2012]. The online linear predictor
that competes with sparse parameter vectors and its analysis summarised in
Theorem 23.6 is due to Gerchinovitz [2013, theorem 10]. A recent paper by
Rakhlin and Sridharan [2017] also discusses the relationship between online
learning regret bounds and self-normalised tail bounds of the type given here.
Interestingly, what they show is that the relationship goes in both directions:
tail inequalities imply regret bounds, and regret bounds imply tail inequalities.
We are told by Francesco Orabona that confidence set constructions similar to
those in Section 23.3 have been used earlier in a series of papers by Claudio
Gentile and friends [Dekel et al., 2010, 2012, Crammer and Gentile, 2013, Gentile
and Orabona, 2012, 2014]. Carpentier and Munos [2012] consider a special case
where the action set is the unit sphere and the noise is vector valued so that the
reward is Xt = hAt , θ + ηt i. They prove bounds that essentially depend on the
sparsity of θ and E[kηt k22 ]. Our setting is recovered by choosing ηt to be a vector
of independent standard Gaussian random variables, but in this case the bounds
recovered by the proposed algorithm are suboptimal.

23.7 Exercises

23.1 (The zero-‘norm’) A norm on Rd is a function k · k : Rd → R such that

for all a ∈ R and x, y ∈ Rd , it holds that: (a) kxk = 0 if and only if x = 0 and
(b) kaxk = |a|kxk and (c) kx + yk ≤ kxk + kyk and (d) kxk ≥ 0. Show that k · k0
Pd
given by kxk0 = i=1 I {xi 6= 0} is not a norm.

23.2 (Minimax bound for SETC) Prove the second part of Theorem 23.2.

23.3 (Anytime algorithm) Algorithm 13 is not anytime (it requires advance

knowledge of the horizon). Design a modified version that does not require
this knowledge and prove a comparable regret bound to what was given in
Theorem 23.2.

Hint One way is to use the doubling trick, but a more careful approach will
lead to a more practical algorithm.
23.4 Complete the calculation to derive Eq. (23.10) from Eq. (23.9).

23.5 Prove the equality in Eq. (23.11).

24 Minimax Lower Bounds for
Stochastic Linear Bandits

Lower bounds for linear bandits turn out to be more nuanced than those for the
classical finite-armed bandit. The difference is that for linear bandits the shape
of the action set plays a role in the form of the regret, not just the distribution
of the noise. This should not come as a big surprise because the stochastic
finite-armed bandit problem can be modeled as a linear bandit with actions
being the standard basis vectors, A = {e1 , . . . , ek }. In this case the actions are
orthogonal, which means that samples from one action do not give information
about the rewards for other actions. Other action sets such as the unit ball
(A = B2d = {x ∈ Rd : kxk2 ≤ 1}) do not share this property. For example, if
d = 2 and A = B2d and an algorithm chooses actions e1 = (1, 0) and e2 = (0, 1)
many times, then it can deduce the reward it would obtain from choosing any
other action.
All results of this chapter have a worst-case flavour showing what is (not)
achievable in general, or under a sparsity constraint, or if the realisable assumption
is not satisfied. The analysis uses the information-theoretic tools introduced in
Part IV combined with careful choices of action sets. The hard part is guessing
what is the worst case, which is followed by simply turning the crank on the
usual machinery.
In all lower bounds, we use a simple model with Gaussian noise. For action
At ∈ A ⊆ Rd the reward is Xt = µ(At ) + ηt where ηt ∼ N (0, 1) is a sequence of
independent standard Gaussian noise and µ : A → R is the mean reward. We
will usually assume there exists a θ ∈ Rd such that µ(a) = ha, θi. We write Pµ to
indicate the measure on outcomes induced by the interaction of the fixed policy
and the Gaussian bandit paramterised by µ. Because we are now proving lower
bounds, it becomes necessary to be explicit about the dependence of the regret
on A and µ or θ. The regret of a policy is:
" n
#
X
Rn (A, µ) = n max µ(a) − Eµ Xt ,
a∈A
t=1

where the expectation is taken with respect to Pµ . Except in Section 24.4, we

assume the reward function is linear, which means there exists a θ ∈ Rd such
that µ(a) = ha, θi. In these cases, we write Rn (A, θ) and Eθ and Pθ . Recall the
Pt
notation used for finite-armed bandits by defining Tx (t) = s=1 I {As = x}.
 24.1 Hypercube 288

24.1 Hypercube

The first lower bound is for the hypercube action set and shows that the upper
bounds in Chapter 19 cannot be improved in general.
Theorem 24.1. Let A = [−1, 1]d and Θ = {−n−1/2 , n−1/2 }d . Then, for any
policy, there exists a vector θ ∈ Θ such that:
exp(−2) √
Rn (A, θ) ≥ d n.
8
Proof By the relative entropy identities in Exercise 15.8.(b) and Exercise 14.7,
we have for θ, θ0 ∈ Θ that
" n #
X
D(Pθ , Pθ0 ) = Eθ D(N (hAt , θi, 1), N (hAt , θ i, 1))
0

t=1

1X 
n

= Eθ hAt , θ − θ0 i2 . (24.1)
2 t=1

For i ∈ [d] and θ ∈ Θ, define

n
!
X
pθi = Pθ I {sign(Ati ) 6= sign(θi )} ≥ n/2 .
t=1

Now let i ∈ [d] and θ ∈ Θ be fixed, and let θj0 = θj for j 6= i and θi0 = −θi . Then,
by the Bretagnolle–Huber inequality (Theorem 14.2) and Eq. (24.1),
!
1 1X 1
n
pθi + pθ0 i ≥ exp − Eθ [hAt , θ − θ0 i2 ] ≥ exp (−2) . (24.2)
2 2 t=1 2

Applying an ‘averaging hammer’ over all θ ∈ Θ, which satisfies |Θ| = 2d , we get

X 1 X d
1 XX
d
d
pθi = pθi ≥ exp (−2) .
|Θ| i=1 |Θ| i=1 4
θ∈Θ θ∈Θ
Pd
This implies that there exists a θ ∈ Θ such that i=1 pθi ≥ d exp (−2) /4. By
the definition of pθi , the regret for this choice of θ is at least
" n d #
XX
Rn (A, θ) = Eθ (sign(θi ) − Ati )θi
t=1 i=1
r " n #
1X
d X
≥ Eθ I {sign(Ati ) 6= sign(θi )}
n i=1 t=1
√ X d n
!
n X
≥ Pθ I {sign(Ati ) 6= sign(θi )} ≥ n/2
2 i=1 t=1
√ X
exp(−2) √
d
n
= pθi ≥ d n,
2 i=1 8
 24.2 Unit Ball 289

where the first line follows since the optimal action satisfies a∗i = sign(θi ) for
i ∈ [d], the first inequality follows from a simple case-based analysis showing that
(sign(θi ) − Ati )θi ≥ |θi |I {sign(Ati ) 6= sign(θi )}, the second inequality is Markov’s
inequality (see Lemma 5.1), and the last inequality follows from the choice of
θ.

Except for logarithmic factors, this shows that the algorithm of Chapter 19
is near optimal for this action set. The same proof works when A = {−1, 1}d
is restricted to the corners of the hypercube, which is a finite-armed
p linear
bandit. In Chapter 22, we gave a policy with regret Rn = O( nd log(nk)),
where k = |A|. There is no contradiction because the action set in the above
proof has k = |A| = 2d elements.

24.2 Unit Ball

Lower-bounding the minimax regret when the action set is the unit ball presents
an additional challenge relative to the hypercube. The product structure of
the hypercube means that the actions of the learner in one dimension do not
constraint their choices in other dimensions. For the unit ball, this is not true, and
this complicates the analysis. Nevertheless, a small modification of the technique
allows us to prove a similar bound.

Theorem 24.2. Assume d ≤ 2n and let A = {x ∈ Rd : kxk2 ≤ 1}. Then

there exists a parameter
√ vector θ ∈ Rd with kθk22 = d2 /(48n) such that
√
Rn (A, θ) ≥ d n/(16 3).
1
p
Proof Let ∆ = 4√ 3
d/n and θ ∈ {±∆}d and for i ∈ [d], define τi = n ∧ min{t :
Pt 2
s=1 Asi ≥ n/d}. Then,
" n d  #
XX 1
Rn (A, θ) = ∆Eθ √ − Ati sign(θi )
t=1 i=1 d
√ " n d  2 #
∆ d XX 1
≥ Eθ √ − Ati sign(θi )
2 t=1 i=1 d
√ d "τ  2 #
∆ dX X i
1
≥ Eθ √ − Ati sign(θi ) ,
2 i=1 t=1 d

where the first inequality

√ uses that kAt k22 ≤ 1. Fix i ∈ [d]. For x ∈ {±1}, define
P τi
Ui (x) = t=1 (1/ d − Ati x)2 and let θ0 ∈ {±∆}d be another parameter vector
such that θj = θj0 for j 6= i and θi0 = −θi . Assume without loss of generality that
θi > 0. Let P and P0 be the laws of Ui (1) with respect to the bandit/learner
 24.2 Unit Ball 290

interaction measure induced by θ and θ0 , respectively. Then,

 r
4n 1
Eθ [Ui (1)] ≥ Eθ0 [Ui (1)] − +2 D(P, P0 )
d 2
v " #
 u
∆ 4n u X τi
≥ Eθ0 [Ui (1)] − +2 t E 2
Ati (24.3)
2 d t=1
 r
∆ 4n n
≥ Eθ [Ui (1)] − +2 +1 (24.4)
2
0
d d
√ r
4 3∆n n
≥ Eθ0 [Ui (1)] − , (24.5)
d d
where in the first inequality we used Pinsker’s inequality (Eq. (14.12)), the result
in Exercise 14.4, the bound
1 4n
Xτi Xτi Xτi
√ 2
Ui (1) = (1/ d − Ati ) ≤ 2 +2 A2ti ≤ + 2,
t=1 t=1
d t=1
d

and the assumption that d ≤ 2n. The inequality in Eq. (24.3) follows from the
chain rule for the relative entropy up to a stopping time (Exercise 15.7). Eq. (24.4)
is true by the definition of τi and Eq. (24.5) by the assumption that d ≤ 2n.
Then,
√ r
4 3n∆ n
Eθ [Ui (1)] + Eθ0 [Ui (−1)] ≥ Eθ0 [Ui (1) + Ui (−1)] −
d d
" # √ r √ r
4 3n∆ n 2n 4 3n∆ n
τ
τi X 2 i
n
= 2Eθ0 + Ati − ≥ − = .
d t=1
d d d d d d

The proof is completed using the randomisation hammer:

√ d
X ∆ dX X
Rn (A, θ) ≥ Eθ [Ui (sign(θi ))]
d
2 i=1 d
θ∈{±∆} θ∈{±∆}
√
∆ d
d
X X X
= Eθ [Ui (sign(θi ))]
2 i=1 θ−i ∈{±∆}d−1 θi ∈{±∆}
√ d
∆ dX X n √
≥ = 2d−2 n∆ d .
2 i=1 d
θ−i ∈{±∆}d−1
√ √
n∆ d d n
Hence there exists a θ ∈ {±∆} such that Rn (A, θ) ≥
d
= √ .
4 16 3

The same proof works when A = {x ∈ Rd : kxk2 = 1} is the unit sphere. In

fact, given a set X ⊂ Rd , a minimax lower bound that holds for A = co(X)
continues to hold for A = X.
 24.3 Sparse Parameter Vectors 291

24.3 Sparse Parameter Vectors

√
In Chapter 23 we gave an algorithm with Rn = Õ( dpn) where p ≥ kθk0 is a
known bound on the sparsity of the unknown parameter. Except for logarithmic
terms this bound cannot be improved. An extreme case is when p = 1, which
essentially√reduces to the finite-armed bandit problem where the minimax regret
has order dn (see Chapter 15). For this reason we cannot expect too much from
sparsity and in particular the worst-case bound will depend polynomially on the
ambient dimension d.
Constructing a lower bound for p > 1 is relatively straightforward. For simplicity
we assume that d = pk for some integer k > 1. A sparse linear bandit can mimic
the learner playing p finite-armed bandits simultaneously, each with k arms.
Rather than observing the reward for each bandit, however, the learner only
observes the sum of the rewards and the noise is added at the end. This is
sometimes called the multi-task bandit problem.

Theorem 24.3. Assume pd ≤ n and that d = pk for some integer k ≥ 2. Let

A = {ei ∈ Rk : i ∈ [k]}p ⊂ Rd . Then, for
p any policy there exists a parameter
√
vector θ ∈ Rd with kθk0 = p and kθk∞ ≤ d/(pn) such that Rn (A, θ) ≥ 81 pdn.

Proof Let ∆ > 0 and Θ = {∆ei : i ∈ [k]} ⊂ Rk . Given θ ∈ Θp ⊂ Rd and i ∈ [p],

(i)
let θ(i) ∈ Rk be defined by θk = θ(i−1)p+k , which means that

θ> = [θ(1)> , θ(2)> , . . . , θ(p)> ] .

Next define matrix V ∈ Rp×d to be a block-diagonal matrix with 1 × k blocks,

each containing the row vector (1, 2, . . . , k). For example, when p = 3, we have
 
1 ··· k 0 ··· 0 0 ··· 0
 
V =
0 ··· 0 1 ··· k 0 ··· 0.
 
0 ··· 0 0 ··· 0 1 ··· k

Let Bt = V At ∈ [k]p represent the vector of ‘base’ actions chosen by the learner
in each of the p bandits in round t. The optimal action in the ith bandit is
(i)
b∗i (θ) = argmaxb∈[k] θb .

The regret can be decomposed into the regrets in the p ‘base bandit’ problems (a
form of separability, again):
p
" n #
X X
Rn (θ) = ∆ Eθ I {Bti 6= bi } .
∗

i=1 t=1
| {z }
Rni (θ)
 24.4 Misspecified Models 292

For i ∈ [p], we abbreviate θ(−i) = (θ(1) , . . . , θ(i−1) , θ(i+1) , . . . , θ(p) ). Then,

p
1 X 1 X X
R n (θ) = Rni (θ)
|Θ|p p
|Θ|p i=1 p
θ∈Θ θ∈Θ
p
X 1 X 1 X
= Rni (θ)
|Θ|p−1 |Θ| (i)
i=1 θ (−i) ∈Θp−1 θ ∈Θ
p
1 X 1 X √
≥ kn (24.6)
8 |Θ|p−1
i=1 θ (−i) ∈Θp−1
1 √ 1p
= p kn = dpn .
8 8
Here, in the second equality, we use the convention that θ denotes the vector
obtained by ‘inserting’ θ(i) into θ(−i) at the ith ‘block’. Other
pthan this, the only
tricky step is the inequality, which follows by choosing ∆ ≈ k/n and repeating
the argument outlined in Exercise 15.2. We leave it to the reader to check the
details (Exercise 24.1).

24.4 Misspecified Models

An important generalisation of the linear model is the misspecified case, where

the mean rewards are not assumed to follow a linear model exactly. Suppose
that A ⊂ Rd is a finite set with |A| = k and that Xt = ηt + µ(At ), where
µ : A → R is an unknown function. Let θ ∈ Rd be the parameter vector for which
supa∈A |hθ, ai − µ(a)| is as small as possible:

θ = argminα∈Rd sup |hα, ai − µ(a)| .

a∈A

Then let ε = supa∈A |hθ, ai − µ(a)| be the maximum error. It would be very
pleasant to have an algorithm such that
" n #
X √ √
Rn (A, µ) = n max µ(a) − E µ(At ) = Õ(min{d n + εn, kn}) . (24.7)
a∈A
t=1

Unfortunately, it turns out that results of this kind are not achievable. To show
this, we will prove a generic bound for the classical finite-armed bandit problem
and afterwards show how this implies the impossibility of an adaptive bound like
the above.

Theorem 24.4. Let A = [k], and for µ ∈ [0, 1]k the reward is Xt = µAt + ηt and
the regret is
" n #
X
Rn (µ) = n max µi − Eµ µAt .
i∈A
t=1
 24.4 Misspecified Models 293

Define Θ, Θ0 ⊂ Rk by
 
Θ = µ ∈ [0, 1]k : µi = 0 for i > 1 Θ0 = µ ∈ [0, 1]k .
p
If V ∈ R is such that 2(k − 1) ≤ V ≤ n(k − 1) exp(−2)/8 and supµ∈Θ Rn (µ) ≤
V , then
n(k − 1)
sup Rn (µ0 ) ≥ exp(−2) .
µ0 ∈Θ0 8V
Pn
Proof Recall that Ti (n) = t=1 I {At = i} is the number of times arm i is
played after all n rounds. Let µ ∈ Θ be given by µ1 = ∆ = (k − 1)/V ≤ 1/2. The
regret is then decomposed as:
k
X
Rn (µ) = ∆ Eµ [Ti (n)] ≤ V .
i=2
Pk
Rearranging shows that i=2 Eµ [Ti (n)] ≤ ∆,
V
and so by the pigeonhole principle
there exists an i > 1 such that
V 1
Eµ [Ti (n)] ≤ = 2.
(k − 1)∆ ∆
Then, define µ0 ∈ Θ0 by

∆

 if j = 1
µ0j = 2∆ if j = i


0 otherwise .

Next, by Theorem 14.2 and Lemma 15.1, for any event A, we have
1 1
1
Pµ (A) + Pµ0 (Ac ) ≥ exp (D(Pµ , Pµ0 )) = exp −2∆2 E[Ti (n)] ≥ exp (−2) .
2 2 2
By choosing A = {T1 (n) ≤ n/2} we have
n∆ n(k − 1)
Rn (µ) + Rn (µ0 ) ≥ exp(−2) = exp(−2) .
4 4V
p
Therefore, by the assumption that Rn (µ) ≤ V ≤ n(k − 1) exp(−2)/8 we have
n(k − 1)
Rn (µ0 ) ≥ exp(−2) .
8V
As promised, we now relate this to the misspecified linear bandits. Suppose
that d = 1 (an absurd case) and that there are k arms A = {a1 , a2 , . . . , ak } ⊂ R1 ,
where a1 = (1) and ai = (0) for i > 1. Clearly, if θ > 0 and µ(ai ) = hai , θi, then
the problem can be modelled as a finite-armed bandit with means µ ∈ Θ ⊂ [0, 1]k .
In the general case, we just have a finite-armed bandit with µ ∈ Θ0 . If in the first
√
case we have Rn (A, µ) = O( n), then the theorem shows for large enough n that
√
sup Rn (A, µ) = Ω(k n) .
µ∈Θ0
 24.5 Notes 294

It follows that Eq. (24.7) is a pipe dream. To our knowledge, it is still an open
question of what is possible on this front. We speculate that for k ≥ d2 , there is
a policy for which
  
√ √ k√
Rn (A, θ) = Õ min d n + εn d, n .
d

24.5 Notes

1 The worst-case bound demonstrates the near optimality of the OFUL algorithm
for a specific action set. It is an open question to characterise the optimal
regret for a wide range of action sets. We will return to these issues in the next
part of the book, where we discuss adversarial linear bandits.
2 We return to misspecified bandits in the notes and exercises of Chapter 29,
where algorithms from the adversarial linear bandit framework are applied to
this problem in special cases. √
In many applications, the number of actions is so
√
large that Rn = Õ(d n + εn d) should be considered acceptable. There exist
algorithms achieving this bound, which for large k is essentially not improvable
in the worst case [Lattimore and Szepesvári, 2019b]. For small k,√recent work√by
Foster and Rakhlin [2020] shows that one can achieve Rn = Õ( dkn + εn k).

24.6 Bibliographic Remarks

Worst-case lower bounds for stochastic bandits have appeared in a variety of

places, all with roughly the same bound, but for different action sets. Our very
simple proof for the hypercube is new, but takes inspiration from the paper by
√
Shamir [2015]. Rusmevichientong and Tsitsiklis [2010] proved that Rn = Ω(d n)
when A is the unit sphere. Our proof for the unit ball strengthens their result
marginally and is much simpler. As far as we know, the first lower bound of
√
Ω(d n) was given by Dani et al. [2008] for an action set equal to the product
of two-dimensional disks. The results for the misspecified case are inspired by
the work of one of the authors on the Pareto-regret frontier for bandits, which
characterises what trade-offs are available when it is desirable to have a regret
that is unusually small relative to some specific arms [Lattimore, 2015a].

24.7 Exercises

24.1 Complete the missing steps to prove the inequality in Eq. (24.6).
25 Asymptotic Lower Bounds for
Stochastic Linear Bandits

The lower bounds in the previous chapter were derived by analysing the worst
case for specific action sets and/or constraints on the unknown parameter. In this
chapter, we focus on the asymptotics and aim to understand the influence of the
action set on the regret. We start with a lower bound, and argue that the lower
bound can be achieved. We finish by arguing that the optimistic algorithms (and
Thompson sampling) will perform arbitrarily worse than what can be achieved
by non-optimistic algorithms.

25.1 An Asymptotic Lower Bound for Fixed Action Sets

We assume that A ⊂ Rd is finite with |A| = k and that the reward is

Xt = hAt , θi + ηt , where θ ∈ Rd and (ηt )∞
t=1 is a sequence of independent
standard Gaussian random variables. Of course the regret of a policy in this
setting is
" n #
X
Rn (A, θ) = Eθ ∆At , ∆a = max
0
ha0 − a, θi ,
a ∈A
t=1

where the dependence on the policy is omitted for readability and Eθ [·] is the
expectation with respect to the measure on outcomes induced by the interaction
of the policy and the linear bandit determined by θ. Like the asymptotic lower
bounds in the classical finite-armed case (Chapter 16), the results of this chapter
are proven only for consistent policies. Recall that a policy is consistent in some
class of bandits E if the regret is sub-polynomial for any bandit in that class.
Here this means that

Rn (A, θ) = o(np ) for all p > 0 and θ ∈ Rd . (25.1)

The main objective of the chapter is to prove the following theorem on the
behaviour of any consistent policy and discuss the implications.

Theorem 25.1. Assume that A ⊂ Rd is finite and spans Rd , and suppose a

policy is consistent (satisfies Eq. 25.1). Let θ ∈ Rd be any parameter
 such
Pn
that there is a unique optimal action, and let Ḡn = Eθ t=1 t t . Then
A A >
 25.1 An Asymptotic Lower Bound for Fixed Action Sets 296

lim inf n→∞ λmin (Ḡn )/ log(n) > 0. Furthermore, for any a ∈ A, it holds that

∆2a
lim sup log(n)kak2Ḡ−1 ≤ .
n→∞ n 2

The reader should recognise kak2Ḡ−1 as the key term in the width of the
n
confidence interval for the least squares estimator (Chapter 20). This is quite
intuitive. The theorem is saying that any consistent algorithm must prove
statistically that all suboptimal arms are indeed suboptimal by making the
size of the confidence interval smaller than the suboptimality gap. Before the
proof of this result, we give a corollary that characterises the asymptotic regret
that must be endured by any consistent policy.

Corollary 25.2. Let A ⊂ Rd be a finite set that spans Rd and θ ∈ Rd be such

that there is a unique optimal action. Then, for any consistent policy,
Rn (A, θ)
lim inf ≥ c(A, θ) ,
n→∞ log(n)

where c(A, θ) is defined as

X
c(A, θ) = inf α(a)∆a
α∈[0,∞)A
a∈A
∆2a
subject to kak2H −1 ≤ for all a ∈ A with ∆a > 0 ,
α 2
P
with Hα = a∈A α(a)aa> .

The lower bound is complemented by a matching upper bound that we will

not prove.

Theorem 25.3. Let A ⊂ Rd be a finite set that spans Rd . Then there exists a
policy such that
Rn (A, θ)
lim sup ≤ c(A, θ) ,
n→∞ log(n)

where c(A, θ) is defined as in Corollary 25.2.

Proof of Theorem 25.1 The proof of the first part is simply omitted (see the
reference below for details). It follows along similar lines to what follows, essentially
that if Gn is not sufficiently large in every direction, then some alternative
parameter is not sufficiently identifiable. Let a∗ = argmaxa∈A ha, θi be the optimal
action, which we assumed to be unique. Let θ0 ∈ Rd be an alternative parameter
to be chosen subsequently, and let P and P0 be the measures on the sequence
of outcomes A1 , X1 , . . . , An , Xn induced by the interaction between the policy
and the bandit determined by θ and θ0 respectively. Let E[·] and E0 [·] be the
expectation operators of P and P0 , respectively. By Theorem 14.2 and Lemma 15.1,
 25.1 An Asymptotic Lower Bound for Fixed Action Sets 297

for any event E,

1
P (E) + P0 (E c ) ≥ exp (− D(P, P0 ))
2 " n #!  
1 1 X 2 1 1 0 2
= exp − E 0
hAt , θ − θ i = exp − kθ − θ kḠn .
2 2 t=1
2 2
(25.2)
A simple re-arrangement shows that
 
1 1
kθ − θ0 k2Ḡn ≥ log .
2 2P (E) + 2P0 (E c )
Now we follow the usual plan of choosing θ0 to be close to θ, but so that the
optimal action in the bandit determined by θ0 is not a∗ . Let ∆min = min{∆a :
a ∈ A, ∆a > 0} and ε ∈ (0, ∆min ) and H be a positive definite matrix to be
chosen later such that ka − a∗ k2H > 0. Then define
∆a + ε
θ0 = θ + H(a − a∗ ) ,
ka − a∗ k2H
which is chosen so that
ha − a∗ , θ0 i = ha − a∗ , θi + ∆a + ε = ε .
This means that a∗ is ε-suboptimal for bandit θ0 . We abbreviate Rn = Rn (A, θ)
and Rn0 = Rn (A, θ0 ). Then
" #
X n∆min nε
Rn = E Ta (n)∆a ≥ P (Ta∗ (n) < n/2) ≥ P (Ta∗ (n) < n/2) ,
2 2
a∈A
Pn
where Ta (n) = t=1 I {At = a}. Similarly, a∗ is ε-suboptimal in bandit θ0 so that
nε 0
Rn0 ≥ P (Ta∗ (n) ≥ n/2) .
2
Therefore,
2
P (Ta∗ (n) < n/2) + P0 (Ta∗ (n) ≥ n/2) ≤ (Rn + Rn0 ) . (25.3)
nε
Note that this holds for any choice of H with ka − a∗ kH > 0. The logical next
step is to select H (which determines θ0 ) to make (25.2) as large as possible. The
main difficulty is that this depends on n, so instead we aim to choose an H so
the quantity is large enough infinitely often. We start by just re-arranging things:
∗ 2
1 (∆a + ε)2 ka − a kH Ḡn H (∆a + ε)2
kθ − θ0 k2Ḡn = · 4 = ρn (H) ,
2 2 ∗
ka − a kH 2ka − a∗ k2Ḡ−1
n

where we introduced
ka − a∗ k2Ḡ−1 ka − a∗ k2H Ḡ
ρn (H) = n nH
.
ka − a∗ k4H
 25.1 An Asymptotic Lower Bound for Fixed Action Sets 298

Therefore, by choosing E to be the event that Ta∗ (n) < n/2 and using (25.3) and
(25.2), we have
 
(∆a + ε)2 nε
ρ (H) ≥ log ,
2ka − a∗ k2Ḡ−1 4Rn + 4Rn0
n
n

which after re-arrangement leads to

(∆a + ε)2 log((4Rn + 4Rn0 )/ε)
ρ (H) ≥ 1 − .
2 log(n)ka − a∗ k2Ḡ−1 log(n)
n
n

The definition of consistency means that Rn and Rn0 are both sub-polynomial,
which implies that the second term in the previous expression tends to zero for
large n and so by sending ε to zero,
ρn (H) 2
lim inf 2 ≥ 2. (25.4)
n→∞ log(n)ka − a kḠ−1
∗ ∆a
n

We complete the result using proof by contradiction. Suppose that

∆2a
lim sup log(n)ka − a∗ k2Ḡ−1 > . (25.5)
n→∞ n 2
Then there exists an ε > 0 and infinite set S ⊆ N such that
(∆a + ε)2
log(n)ka − a∗ k2Ḡ−1 ≥ for all n ∈ S .
n 2
Hence, by (25.4), lim inf n∈S ρn (H) > 1. We now choose H to be a cluster point of
the sequence (Ḡ−1
n /kḠn k)n∈S where kḠn k is the spectral norm of the matrix
−1 −1

Ḡn . Such a point must exist, since matrices in this sequence have unit spectral
−1

norm by definition and the set of such matrices is compact. We let S 0 ⊆ S be

a subset so that Ḡ−1n /kḠn k converges to H on n ∈ S . We now check that
−1 0

ka − a kH > 0:
∗

ka − a∗ k2Ḡ−1
ka − a∗ k2H = lim0 n
> 0,
n∈S kḠ−1
n k

where the last inequality follows from the assumption in (25.5) and the first part
of the theorem. Therefore,
ka − a∗ k2Ḡ−1 ka − a∗ k2H Ḡ
1 < lim inf ρn (H) ≤ lim inf n nH
= 1,
n∈S 0 n∈S ka − a∗ k4H
which is a contradiction, and hence (25.5) does not hold. Thus,
∆2a
lim sup log(n)ka − a∗ k2Ḡ−1 ≤ .
n→∞ n 2
We leave the proof of the corollary as an exercise for the reader. Essentially,
though, any consistent algorithm must choose its actions so that in expectation
∆2a
ka − a∗ k2Ḡ−1 ≤ (1 + o(1)) .
n 2 log(n)
 25.2 Clouds Looming for Optimism 299

Now, since a∗ will be chosen linearly, often it is easily shown for suboptimal a
that limn→∞ ka − a∗ kḠ−1
n
/kakḠ−1
n
→ 1. This leads to the required constraint on
the actions of the algorithm, and the optimisation problem in the corollary is
derived by minimising the regret subject to this constraint.

25.2 Clouds Looming for Optimism

The theorem and its corollary have disturbing

implications for policies based on the principle
of optimism in the face of uncertainty, which
is that they can never be asymptotically
optimal. The reason is that these policies
do not choose actions for which they have
collected enough statistics to prove they are
suboptimal, but in the linear setting it can
be worth playing these actions when they
are very informative about other actions for
which the statistics are not yet so clear. As
we shall see, a problematic example appears
in the simplest case where there is information sharing between the arms. Namely,
when the dimension is d = 2, and there are k = 3 arms.
Let A = {a1 , a2 , a3 }, where a1 = e1 and a2 = e2 and a3 = (1 − ε, γε) with
γ ≥ 1 and ε > 0 is small. Let θ = (1, 0) so that the optimal action is a∗ = a1
and ∆a2 = 1 and ∆a3 = ε. If ε is very small, then a1 and a3 point in nearly
the same direction, and so choosing only these arms does not provide sufficient
information to quickly learn which of a1 or a3 is optimal. On the other hand,
a2 and a1 − a3 point in very different directions, which means that choosing a2
allows a learning agent to quickly identify that a1 is in fact optimal. We now
show how the theorem and corollary demonstrate this. First we calculate the
optimal solution to the optimisation problem in Corollary 25.2. Recall we are
trying to minimise

X ∆2a
α(a)∆a subject to kak2H(α)−1 ≤ for all a ∈ A with ∆a > 0 ,
2
a∈A

P
where H(α) = a∈A α(a)aa> . Clearly we should choose α(a1 ) arbitrarily large,
then a computation shows that
 
 0 0 
lim H(α)−1 = .
α(a1 )→∞ 1
0 α(a3 )ε2 γ 2 +α(a2 )
 25.2 Clouds Looming for Optimism 300

The constraints mean that

1 1
= lim ka2 k2H(α)−1 ≤
α(a3 )ε γ + α(a2 ) α(a1 )→∞
2 2 2
2 2
γ ε ε2
= lim ka3 k2H(α)−1 ≤ .
α(a3 )ε γ + α(a2 ) α(a1 )→∞
2 2 2
Provided that γ ≥ 1, this reduces to the constraint that
α(a3 )ε2 + α(a2 ) ≥ 2γ 2 .
Since we are minimising α(a2 ) + εα(a3 ) we can easily see that α(a2 ) = 2γ 2 and
α(a3 ) = 0 provided that 2γ 2 ≤ 2/ε. Therefore, if ε is chosen sufficiently small
relative to γ, then the optimal rate of the regret is c(A, θ) = 2γ 2 , and so by
Theorem 25.3 there exists a policy such that
Rn (A, θ)
lim sup = 2γ 2 .
n→∞ log(n)
Now we argue that for γ sufficiently large and ε arbitrarily small that the regret
for any consistent optimistic algorithm is at least
Rn (A, θ)
lim sup = Ω(1/ε) ,
n→∞ log(n)
which can be arbitrarily worse than the optimal rate! So why is this so? Recall
that optimistic algorithms choose


At = argmaxa∈A max a, θ̃ ,
θ̃∈Ct

where Ct ⊂ Rd is a confidence set that we assume contains the true θ with high
probability. So far this does not greatly restrict the class of algorithms that we
might call optimistic. We now assume that there exists a constant c > 0 such
that
n p o
Ct ⊆ θ̃ : kθ̂t − θ̃kVt ≤ c log(n) ,
Pt
where Vt = s=1 As A> s . So now we ask how often we can expect the optimistic
algorithm to choose action a2 = e2 in the example described above. Since we
have assumed θ ∈ Ct with high probability, we have that
maxha1 , θ̃i ≥ 1 .
θ̃∈Ct

On the other hand, if Ta2 (t − 1) > 4c2 log(n), then

s
q log(n)
maxha2 , θ̃i = maxha2 , θ̃ − θi ≤ 2c ka2 kV −1 log(n) ≤ 2c < 1,
θ̃∈Ct θ̃∈Ct t Ta2 (t − 1)

which means that a2 will not be chosen more than 1 + 4c2 log(n) times. So if
γ = Ω(c2 ), then the optimistic algorithm will not choose a2 sufficiently often
and a simple computation shows it must choose a3 at least Ω(log(n)/ε2 ) times
 25.3 Notes 301

and suffers regret of Ω(log(n)/ε). The key take away from this is that optimistic
algorithms do not choose actions that are statistically suboptimal, but for linear
bandits it can be optimal to choose these actions more often to gain information
about other actions.

This conclusion generalises to structured bandit problems where choosing

one action allows you to gain information about the rewards of other actions.
In such models the optimism principle often provides basic guarantees, but
may fail to optimally exploit the structure of the problem.

25.3 Notes

1 All algorithms known to match the lower bound in Theorem 25.3 are based
on (or inspired by) solving the optimisation problem that defines c(A, θ) with
estimated value θ. Unfortunately, these algorithms are not especially practical
in finite time. As far as we know, none are simultaneously near-optimal in a
minimax sense. Constructing a practical asymptotically optimal algorithm for
linear bandits is a fascinating open problem.
2 In Chapter 36 we will introduce the randomised Bayesian algorithm called
Thompson sampling algorithm for finite-armed and linear bandits. While
Thompson sampling is often empirically superior to UCB, it does not overcome
the issues described here.

25.4 Bibliographic Remarks

The theorems of this chapter are by the authors: Lattimore and Szepesvári [2017].
The example in Section 25.2 first appeared in a paper by Soare et al. [2014],
which deals with the problem of best-arm identification for linear bandits (for an
introduction to best-arm identification, see Chapter 33). The optimisation-based
algorithms that match the lower bound are by Lattimore and Szepesvári [2017],
Ok et al. [2018], Combes et al. [2017] and Hao et al. [2020], with the latter
handling also the contextual case with finitely many contexts.

25.5 Exercises

25.1 Prove Corollary 25.2.

25.2 Prove the first part of Theorem 25.1.

 25.5 Exercises 302

25.3 Give examples of action sets A, parameter vectors θ ∈ Rd and vectors

a ∈ Rd such that:
(a) c(A ∪ {a}, θ) > c(A, θ); and
(b) c(A ∪ {a}, θ) < c(A, θ).
Part VI
Adversarial Linear Bandits
 304

The adversarial linear bandit is superficially a generalisation of the stochastic

linear bandit where the unknown parameter vector is chosen by an adversary.
There are many similarities between the two topics. Indeed, the techniques in
this part combine the ideas of optimal design presented in Chapter 22 with
the exponential weighting algorithm of Chapter 11. The intuitions gained by
studying stochastic bandits should not be taken too seriously, however. There are
subtle differences between the model of adversarial bandits introduced here and
the stochastic linear bandits examined in previous chapters. These differences
will be discussed at length in Chapter 29. The adversarial version of the linear
bandits turns out to be remarkably rich, both because of the complex information
structure and because of the challenging computational issues.
The part is split into four chapters, the first of which is an introduction to the
necessary tools from convex analysis and optimisation. In the first chapter on
bandits, we show how to combine the core ideas of the Exp3 policy of Chapter 11
with the optimal experimental design for least-squares estimators in Chapter 21.
When the number of actions is large (or infinite), the approach based on Exp3
is hard to make efficient. These shortcomings are addressed in the next chapter,
where we introduce the mirror descent and follow-the-regularised leader algorithms
for bandits and show how they can be used to design efficient algorithms. We
conclude the part with a discussion on the relationship between adversarial and
stochastic linear bandits, which is more subtle than the situation with finite-armed
bandits.
26 Foundations of Convex Analysis ( )

Our coverage of convex analysis is necessarily extremely brief. We introduce only

what is necessary and refer the reader to standard texts for the proofs.

26.1 Convex Sets and Functions

A set A ⊆ Rd is convex if for any x, y ∈ A it holds that αx + (1 − α)y ∈ A for all

α ∈ (0, 1). The convex hull of a collection of points x1 , x2 , . . . , xn ∈ Rd is the
smallest convex set containing the points, which also happens to satisfy
( n
)
X
co(x1 , x2 , . . . , xn ) = x∈R :x=
d
pi xi for some p ∈ Pn−1 .
i=1

The convex hull co(A) is also defined for an arbitrary set A ⊂ Rd and is still the
smallest convex set that contains A (see (c) in Figure 26.1). For the rest of the
section, we let A ⊆ Rd be convex. Let R̄ = R ∪ {−∞, ∞} be the extended real
number system and define operations involving infinities in the natural way (see
notes).

Definition 26.1. An extended real-valued function f : Rd → R̄ is convex if its

epigraph Ef = {(x, y) ∈ Rd × R : y ≥ f (x)} ⊂ Rd+1 is a convex set.

The term ‘epi’ originates in greek and it means upon or over: The epigraph of
a function is the set of points that sit on the top of the function’s graph.
The domain of an extended real-valued function on Rd is dom(f ) = {x ∈ Rd :
f (x) < ∞}. For S ⊂ Rd , a function f : S → R̄ is identified with the function
f¯ : Rd → R̄, which coincides with f on S and is defined to take the value ∞
outside of S. It follows that if f : S → R, then dom(f ) = S. A convex function is
proper if its range does not include −∞ and its domain is nonempty.

For the rest of the chapter, we will write ‘let f be a convex’ to mean that
f : Rd → R̄ is a proper convex function.
 26.1 Convex Sets and Functions 306

(a) (b) (c)

(d) (e) (f)

Figure 26.1 (a) is a convex set. (b) is a non-convex set. (c) is the convex hull of a
non-convex set. (d) is a convex function. (e) is non-convex, but all local minimums are
global. (f) is not convex.

Permitting convex functions to take values of −∞ is a convenient standard

because certain operations on proper convex functions result in improper
ones (infimal convolution, for example). These technicalities will never bother
us in this book, however.

A consequence of the definition is that for convex f , we have

f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)
for all α ∈ (0, 1) and x, y ∈ dom(f ) . (26.1)
In fact, the inequality holds for all x, y ∈ Rd .

Some authors use Eq. (26.1) as the definition of a convex function along
with a specification that the domain is convex: If A ⊆ Rd is convex, then
f : A → R is convex if it satisfies Eq. (26.1), with f (x) = ∞ assumed for
x∈/ A.

The reader is invited to prove that all convex functions are continuous on the
interior of their domain (Exercise 26.1).
A function is strictly convex if the inequality in Eq. (26.1) is always strict.
The Fenchel dual of a function f is f ∗ (u) = supx hx, ui − f (x), which is convex
because the maximum of convex functions is convex. The Fenchel dual has many
nice properties. Most important for us is that for sufficiently nice functions, ∇f ∗
is the inverse of ∇f (Theorem 26.6). Another useful property is that when f is
a proper convex function and its epigraph is closed, then f = f ∗∗ , where f ∗∗
denotes the bidual of f : f ∗∗ = (f ∗ )∗ . The Fenchel dual is also called the convex
 26.2 Jensen’s Inequality 307

conjugate. If f : Rd → R̄ is twice differentiable on the interior of its domain, then

convexity of f is equivalent to its Hessian having non-negative eigenvalues for
all x ∈ int(dom(f )). The field of optimisation is obsessed with convex functions
because all local minimums are global (see Fig. 26.1). This means that minimising
a convex function is usually possible (efficiently) using some variation of gradient
descent. A function f : Rd → R̄ is concave if −f is convex.

26.2 Jensen’s Inequality

One of the most important results for convex functions is Jensen’s inequality:
Theorem 26.2 (Jensen’s inequality). Let f : Rd → R̄ be a measurable convex
function and X be an Rd -valued random element on some probability space such
that E[X] exists and X ∈ dom(f ) holds almost surely. Then E[f (X)] ≥ f (E[X]).
If we allowed Lebesgue integrals to take on the value of ∞, the condition that
X is almost surely an element of the domain of f could be removed and the
result would still be true. Indeed, in this case we would immediately conclude
that E[f (X)] = ∞ and Jensen’s inequality would trivially hold.
The basic inequality of (26.1) is trivially
a special case of Jensen’s inequality. Jensen’s f (x)
inequality is so central to convexity that it
can actually be used as the definition (a Pn
(x̄, pk f (xk ))
function is convex if and only if it satisfies k=1

Jensen’s inequality). The proof of Jensen’s using

Definition 26.1 in full generality is left to the
reader (Exercise 26.2). However, we cannot resist (x̄, f (x̄))

to include here a simple ‘graphical proof’ that x1 x2 x3 x4 x5

works in the simple case when X is supported
Pn
on x1 , . . . , xn and P(X = xk ) = pk . Then, letting x̄ = k=1 pk xk , one can notice
P
that the point (x̄, k pk xk ) lies in the convex hull of {(xk , f (xk ))k }, which is a
convex subset of the epigraph Ef ⊂ Rd+1 . The result follows because (x̄, f (x̄)) is
on the boundary of Ef as shown in the figure. The direction of Jensen’s inequality
is reversed if ‘convex’ is replaced by ‘concave’.

26.3 Bregman Divergence

Let f : Rd → R be convex and x, y ∈ Rd with y ∈ dom(f ). The Bregman

divergence at y induced by f is defined by
Df (x, y) = f (x) − f (y) − ∇x−y f (y) ,
where ∇v f (y) = limh→0+ (f (y + hv) − f (y))/h ∈ R ∪ {−∞, ∞} is the directional
derivative of f at y in direction v. The directional derivative is always well defined
 26.3 Bregman Divergence 308

f (x)
f (y) + hx − y, ∇f (y)i

Df (x, y)

y x

Figure 26.2 The Bregman divergence Df (x, y) is the difference between f (x) and the
Taylor series approximation of f at y. When f is convex the, linear approximation is a
lower bound on the function and the Bregman divergence is positive.

for convex functions, but can be positive/negative infinity. When f is differentiable

at y, then ∇v f (y) = hv, ∇f (y)i and thus Df (x, y) = f (x) − f (y) − hx − y, ∇f (y)i,
which is the more usual definition. For the geometric intuition see Fig. 26.2. Let
dom(∇f ) denote the set of points in the domain of f where f is differentiable.
Theorem 26.3. The following hold:
(a) Df (x, y) ≥ 0 for all y ∈ dom(f ).
(b) Df (x, x) = 0 for all x ∈ dom(f ).
(c) Df (x, y) is convex as a function of x for any y ∈ dom(∇f ).
Part (c) does not hold in general when f is not differentiable at y, as you will
show in Exercise 26.14. The square root of the Bregman divergence shares many
properties with a metric, and for some choices of f , it actually is a metric. In
general, however, it is not symmetric and does not satisfy the triangle inequality.
Example 26.4. Let f (x) = 21 kxk22 . Then ∇f (x) = x and
1 1 1
Df (x, y) = kxk22 − kyk22 − hx − y, yi = kx − yk22 .
2 2 2
Example 26.5. Let A = [0, ∞)d , dom(f ) = A and for x ∈ A, f (x) =
Pd
i=1 (xi log(xi ) − xi ), where 0 log(0) = 0. Then, for y ∈ (0, ∞) , ∇f (y) = log(y)
d

and
d
X d
X d
X
Df (x, y) = (xi log(xi ) − xi ) − (yi log yi − yi ) − log(yi )(xi − yi )
i=1 i=1 i=1
d
X   d
X
xi
= xi log + (yi − xi ) .
i=1
yi i=1

Notice that if x, y ∈ Pd−1 are in the unit simplex, then Df (x, y) is the relative
entropy between probability vectors x and y. The function f is called the
unnormalised negentropy, which will feature heavily in many of the chapters
 26.4 Legendre Functions 309

that follow. When y 6> 0, the Bregman divergence is infinite if there exists an
P
i such that yi = 0 and xi > 0. Otherwise, Df (x, y) = i:xi >0 xi log(xi /yi ) +
Pd
i=1 (yi − xi ).

26.4 Legendre Functions

In this section we use various topological

notions such as the interior, closed set and 0 √
− x
boundary. The definitions of these terms are
given in the notes. Let f be a convex function
and A = dom(f ) and C = int(A). Then f is −1
Legendre if

(a) C is non-empty; −2
(b) f is differentiable and strictly convex on 0 1 2 3 4
C; and √
(c) limn→∞ k∇f (xn )k2 = ∞ for any Figure 26.3 f (x) = − x: the
archetypical Legendre function
sequence (xn )n with xn ∈ C for all n
and limn→∞ xn = x and some x ∈ ∂C.

The intuition is that the set {(x, f (x)) : x ∈ dom(A)} is a ‘dish’ with ever-
steepening edges towards the boundary of the domain. Legendre functions have
some very convenient properties:

Theorem 26.6. Let f : Rd → R̄ be a Legendre function. Then,

(a) ∇f is a bijection between int(dom(f )) and int(dom(f ∗ )) with the inverse

(∇f )−1 = ∇f ∗ ;
(b) Df (x, y) = Df ∗ (∇f (y), ∇f (x)) for all x, y ∈ int(dom(f )); and
(c) the Fenchel conjugate f ∗ is Legendre.

The next result formalises the ‘dish’ intuition by showing the directional
derivative along any straight path from a point in the interior to the boundary
blows up. You should supply the proof of the following results in Exercise 26.6.

Proposition 26.7. Let f be Legendre and x ∈ int(dom(f )) and y ∈

∂ int(dom(f )), then limα→1 hy − x, ∇f ((1 − α)x + αy)i = ∞.

Corollary 26.8. If f is Legendre and x∗ ∈ argminx∈dom(f ) f (x), then x∗ ∈

int(dom(f )).

Example 26.9. Let f be the Legendre function given by f (x) = 12 kxk22 , which
has domain dom(f ) = Rd . Then, f ∗ (x) = f (x) and ∇f and ∇f ∗ are the identity
functions.
 26.4 Legendre Functions 310

Pd √
Example 26.10. Let f (x) = −2 i=1 xi when xi ≥ 0 for all i and ∞
otherwise, which has dom(f ) = [0, ∞)d and int(dom(f )) = (0, ∞)d . The gradient
√
is ∇f (x) = −1/ x, which blows up (in norm) on any sequence (xn ) approaching
√
∂ int(dom(f )) = {x ∈ [0, ∞)d : xi = 0 for some i ∈ [d]}. Here, x stands for
√
the vector ( xi )i . In what follows we will often use the underlying convention
of extending univariate functions to vector by applying them componentwise.
Note that k∇f (x)k → 0 as kxk → ∞: ‘∞’ is not part of the boundary of dom(f ).
Strict convexity is also obvious so f is Legendre. In Exercise 26.8, we ask you
to calculate the Bregman divergences with respect to f and f ∗ and verify the
results of Theorem 26.6.
P
Example 26.11. Let f (x) = i xi log(xi ) − xi be the unnormalised negentropy,
which we met in Example 26.5. Similarly to the previous example, dom(f ) =
[0, ∞)d , int(dom(f )) = (0, ∞)d and ∂ int(dom(f )) = {x ∈ [0, ∞)d : xi =
0 for some i ∈ [d]}. The gradient is ∇f (x) = log(x), and thus k∇f (x)k → ∞ as
x → ∂ int(dom(f )). Strict convexity also holds, hence f is Legendre. You already
met the Bregman divergence Df (x, y), which turned out to be the relative entropy
when x, y belong to the simplex. Exercise 26.9 asks you to calculate the dual of f
(can you guess what this function will be?) and the Bregman divergence induced
by f ∗ and to verify Theorem 26.6.

The Taylor series of the Bregman divergence is often a useful approximation.

Let g(y) = Df (x, y), which for y = x has ∇g(y) = 0 and ∇2 g(y) = ∇2 f (x). A
second-order Taylor expansion suggests that
1 1
Df (x, y) = g(y) ≈ g(x) + hy − x, ∇g(x)i + ky − xk2∇2 f (x) = ky − xk2∇2 f (x) .
2 2
This approximation can be very poor if x and y are far apart. Even when x and y
are close, the lower-order terms are occasionally problematic, but nevertheless the
approximation can guide intuition. The next theorem, which is based on Taylor’s
theorem and measurable selections, gives an exact result (Exercise 26.15).

Theorem 26.12. If f is convex and twice differentiable in A = int(dom(f )) and

x, y ∈ A, then there exists an α ∈ [0, 1] and z = αx + (1 − α)y such that
1
Df (x, y) = kx − yk2∇2 f (z) .
2
Suppose furthermore that ∇2 f is continuous on int(dom(f )); then there exists a
measurable function g : int(dom(f )) × int(dom(f )) → int(dom(f )) such that for
all x, y ∈ int(dom(f )),
1
Df (x, y) = kx − yk2∇2 f (g(x,y)) .
2
The next result will be useful.

Theorem 26.13. Let η > 0 and f be Legendre and twice differentiable in

 26.5 Optimisation 311

A = int(dom(f )), x, y ∈ A, and let z ∈ [x, y] be the point such that Df (x, y) =
1 2
2 kx − yk∇2 f (z) . Then, for all u ∈ R ,
d

Df (x, y) η
hx − y, ui − ≤ kuk2(∇2 f (z))−1 .
η 2

Although the Bregman divergence is not symmetric, the right-hand side does
not depend on the order of x and y in the Bregman divergence except that
z ∈ [x, y] may be different.

Proof Strict convexity of f ensures that H = ∇2 f (z) is invertible. Applying

Cauchy–Schwarz,
q
hx − y, ui ≤ kx − ykH kukH −1 = kukH −1 2Df (x, y) .
Therefore,
q
Df (x, y) Df (x, y) η
hx − y, ui − ≤ kukH −1 2Df (x, y) − ≤ kuk2H −1 ,
η η 2
where the last step follows from the ever useful maxx∈R ax − bx2 = a2 /(4b) which
holds for any b > 0 and a ∈ R.

26.5 Optimisation

The first-order optimality condition states that if x ∈ Rd is the minimiser

of a differentiable function f : Rd → R, then ∇f (x) = 0. One of the things we
like about convex functions is that when f is convex, the first-order optimality
condition is both necessary and sufficient. In particular, if ∇f (x) = 0 for some
x ∈ Rd then x is a minimiser of f . The first-order optimality condition can also
be generalised to constrained minima: if f : Rd → R is convex and differentiable
and A ⊆ Rd is a non-empty convex set, then
x∗ ∈ argminx∈A f (x) ⇔ ∀x ∈ A : hx − x∗ , ∇f (x∗ )i ≥ 0 . (26.2)
The necessity of the condition on the right-hand side is easy to understand by
a geometric reasoning. If ∇f (x∗ ) = 0, then the said condition trivially holds. If
∇f (x∗ ) 6= 0, the hyperplane Hx∗ whose normal is ∇f (x∗ ) and goes through x∗
must be a supporting hyperplane of A at x∗ , with −∇f (x∗ ) being the outer
normal of A at x∗ otherwise x∗ could be moved by a small amount while staying
inside A and improving the value of f . Since A is convex, it thus lies entirely
on the side of Hx∗ that ∇f (x∗ ) points into. This is clearly equivalent to (26.2).
The sufficiency of the condition also follows from this geometric viewpoint as the
reader may verify from the figure.
The above statement continues to hold with a small modification even when f
 26.6 Projections 312

−∇f (x)
better point

−∇f (x∗ )

Figure 26.4 Illustration of first-order optimality conditions. The point at the top is not a
minimiser because the hyperplane with normal as gradient does not support the convex
set. The point at the right is a minimiser.

is not differentiable everywhere. In particular, in this case the equivalence (26.2)

holds for any x∗ ∈ dom(∇f ) with the modification that on the right side of the
equivalence, A should be replaced by A ∩ dom(f ):

Proposition 26.14. Let f : Rd → R̄ be a convex function and A ⊂ Rd a

non-empty convex set. Then, for any x∗ ∈ dom(∇f ), it holds that

x∗ ∈ argminx∈A f (x) ⇐⇒
∀x ∈ A ∩ dom(f ) : hx − x∗ , ∇f (x∗ )i ≥ 0 . (26.3)

Further, if f is Legendre, then x∗ ∈ argminx∈A f (x) implies x∗ ∈ dom(∇f ) and

hence also (26.3).

The part that concerns the Legendre objective f follows by noting that by
Corollary 26.8, x∗ ∈ int(dom(f )) combined with that by Theorem 26.6(a),
int(dom(f )) = dom(∇f ).

26.6 Projections

If A ⊂ Rd and x ∈ Rd , then the Euclidean projection of x on A is ΠA (x) =

argminy∈A kx − yk22 . One can also project with respect to a Bregman divergence
induced by convex function f . Let ΠA,f be defined by

ΠA,f (x) = argminy∈A Df (y, x) .

An important property of the projection is that minimising a Legendre function

f on a convex set A is (usually) equivalent to finding the unconstrained minimum
on the domain of f and then projecting that point on to A.

Theorem 26.15. Let f : Rd → R̄ be Legendre, A ⊂ Rd a non-empty, closed

convex set with A ∩ dom(f ) non-empty and assume that ỹ = argminz∈Rd f (z)
exists. Then the following hold:
 26.7 Notes 313

(a) y = argminz∈A f (z) exists and is unique;

(b) y = argminz∈A Df (z, ỹ).
√
The assumption that ỹ exists is necessary. For example f (x) = − x for x ≥ 0
and f (x) = ∞ for x < 0 is Legendre with domain dom(f ) = [0, ∞), but f does
not have a minimum on its domain.

26.7 Notes

1 The ‘infinity arithmetic’ on the extended real line is as follows:

α+∞=∞ for α ∈ (−∞, ∞]
α − ∞ = −∞ for α ∈ [−∞, ∞)
α · ∞ = ∞ and α · (−∞) = −∞ for α > 0
α · ∞ = −∞ and α · (−∞) = ∞ for α < 0
0 · ∞ = 0 · (−∞) = 0 .
Like α/0, the value of ∞ − ∞ is not defined. We also have α ≤ ∞ for all α
and α ≥ −∞ for all α.
2 There are many ways to define the topological notions used in this chapter.
The most elegant is also the most abstract, but there is no space for that here.
Instead we give the classical definitions that are specific to Rd and subsets.
Let A be a subset of Rd . A point x ∈ A is an interior point if there exists
an ε > 0 such that Bε (x) = {y : kx − yk2 ≤ ε} ⊂ A. The interior of A is
int(A) = {x ∈ A : x is an interior point}. The set A is open if int(A) = A
and closed if its complement Ac = Rd \ A is open. The boundary of A is
denoted by ∂A and is the set of points in x ∈ Rd such that for all ε > 0 the
set Bε (x) contains points from A and Ac . Note that points in the boundary
need not be in A. Some examples: ∂[0, ∞) = {0} = ∂(0, ∞) and ∂Rn = ∅.

26.8 Bibliographic Remarks

The main source for these notes is the excellent book by Rockafellar [2015]. The
basic definitions are in part I. The Fenchel dual is analysed in part III while
Legendre functions are found in part V. Convex optimisation is a huge topic. The
standard text is by Boyd and Vandenberghe [2004].

26.9 Exercises

26.1 Let f : Rd → R be convex. Prove that f is continuous on int(dom(f )).

26.2 Prove Jensen’s inequality (Theorem 26.2). Precisely, let X ∈ Rd be a

 26.9 Exercises 314

random variable for which E[X] exists and f : Rd → R̄ a measurable convex

function. Prove that E[f (x)] ≥ f (E[X]).

Hint Let x0 = E[X] ∈ Rd and define a linear function g : Rd → R such that

g(x0 ) = f (x0 ) and g(x) ≤ f (x) for all x ∈ Rd . To guarantee the existence of
g, you may use the supporting hyperplane theorem, which states that if
S ⊂ Rm is a convex set and s ∈ ∂S, then there exists a supporting hyperplane
containing s.
26.3 Let f : Rd → R ∪ {−∞, ∞}.

(a) Prove that f ∗∗ (x) ≤ f (x).

(b) Assume that f is convex and differentiable on int(dom(f )). Show that
f ∗∗ (x) = f (x) for x ∈ int(dom(f )).

As mentioned in the text, the assumption that f is differentiable can be

relaxed to an assumption that the epigraph of f is closed, in which case the
result holds over the whole domain. The proof is not hard, but you will need
to use the sub-differential rather than the gradient, and the boundary must
be treated with care.

26.4 For each of the real-valued functions below, decide whether or not it is
Legendre on the given domain:

(a) f (x) = x2 on [−1, 1].

√
(b) f (x) = − x on [0, ∞).
(c) f (x) = log(1/x) on [0, ∞) with f (0) = ∞.
(d) f (x) = x log(x) on [0, ∞) with f (0) = 0.
(e) f (x) = |x| on R.
(f) f (x) = max{|x|, x2 } on R.

26.5 Prove Theorem 26.3.

26.6 Prove Proposition 26.7 and Corollary 26.8.

26.7 Prove Proposition 26.14.

26.8 Let f be the convex function given in Example 26.10.

(a) For x, y ∈ dom(f ), find Df (x, y).

(b) Compute f ∗ (u) and ∇f ∗ (u).
(c) Find dom(∇f ∗ ).
(d) Show that for u, v ∈ (−∞, 0]d ,

(ui − vi )2
d
X
Df ∗ (u, v) = − .
i=1
ui vi2
 26.9 Exercises 315

(e) Verify the claims in Theorem 26.6.

26.9 Let f : Rd → R̄ be the unnormalised negentropy function from

P
Example 26.11. We have seen in Example 26.5 that Df (x, y) = i (xi log(xi /yi ) +
yi − xi ).
(a) Compute f ∗ (u) and ∇f ∗ (u).
(b) Find dom(∇f ∗ ).
(c) Show that for u, v ∈ Rd ,
d
X
Df ∗ (u, v) = exp(vi )(vi − ui ) + exp(ui ) − exp(vi ) .
i=1

(d) Verify the claims in Theorem 26.6.

26.10 Let f be Legendre. Show that f˜ given by f˜(x) = f (x) + hx, ui is also
Legendre for any u ∈ Rd .

26.11 Let f be the unnormalised negentropy function from Example 26.5.

(a) Prove that f is Legendre.
(b) Given y ∈ (0, ∞)d , prove that argminx∈Pd−1 Df (x, y) = y/kyk1 .

26.12 Let α ∈ [0, 1/d] and A = Pd−1 ∩ [α, 1]d and f be the unnormalised
negentropy function. Let y ∈ [0, ∞)d and x = argminx∈A Df (x, y) and assume
that y1 ≤ y2 ≤ · · · ≤ yd . Let m be the smallest value such that
d
X
ym (1 − (m − 1)α) ≥ α yj .
j=m

Show that
(
α if i < m
xi = Pd
(1 − (m − 1)α)yi / j=m yj otherwise .

26.13 (Generalised Pythagorean identity) Let A ⊂ Rd be convex and

closed and f : Rd → R̄ be a convex function with A ∩ dom(f ) non-empty.
(a) Suppose that x ∈ A and y ∈ Rd and z = ΠA,f (y) and f is differentiable at
y. Prove that
Df (x, y) ≥ Df (x, z) + Df (z, y) .
(b) Prove that the condition that f be differentiable at y cannot be relaxed.

26.14 Prove Theorem 26.3 and show that Part (c) does not hold in general
when f is not differentiable at y.

26.15 Prove Theorem 26.12

Hint For the first part, simply apply Taylor’s theorem. For the second part,
 26.9 Exercises 316

use a measurable selection theorem. For example, the theorem by Kuratowski

and Ryll-Nardzewski, which appears as Theorem 6.9.4 in the second volume of
the book by Bogachev [2007].
27 Exp3 for Adversarial Linear Bandits

The model for adversarial linear bandits is as follows. The learner is given an
action set A ⊂ Rd and the number of rounds n. As usual in the adversarial setting,
it is convenient to switch to losses. An instance of the adversarial problem is a
sequence of loss vectors y1 , . . . , yn taking values in Rd . In each round t ∈ [n], the
learner selects a possibly random action At ∈ A and observes a loss Yt = hAt , yt i.
The learner does not observe the loss vector yt . The regret of the learner after n
rounds is
" n # n
X X
Rn = E Yt − min ha, yt i .
a∈A
t=1 t=1

Clearly, the finite-armed adversarial bandits discussed in Chapter 11 is a special

case of adversarial linear bandits corresponding to the choice A = {e1 , . . . , ed },
where e1 , . . . , ed are the unit vectors of the d-dimensional standard Euclidean
basis.

For this chapter, we assume that


(a) for all t ∈ [n] the loss satisfies yt ∈ L = x ∈ Rd : supa∈A |ha, xi| ≤ 1 ;
and
(b) the action set A spans Rd .
The latter assumption is for convenience only and may be relaxed with a
little care (Exercise 27.7).

27.1 Exponential Weights for Linear Bandits

We adapt the exponential-weighting algorithm of Chapter 11. Like in that setting,

we need a way to estimate the individual losses for each action, but now we make
use of the linear structure to share information between the arms and decrease
the variance of our estimators. For now we assume that A is finite, which we
relax in Section 27.3. Let t ∈ [n] be the index of the current round. Assuming
the loss estimate for action a ∈ A in round s ∈ [n] is Ŷs (a), then the probability
distribution proposed by exponential weights is given by the probability mass
 27.1 Exponential Weights for Linear Bandits 318

function P̃t : A → [0, 1] given by

t−1
!
X
P̃t (a) ∝ exp −η Ŷs (a) ,
s=1

where η > 0 is the learning rate. To control the variance of the loss estimates,
it will be useful to mix this distribution with an exploration distribution π
P
(π : A → [0, 1] and a∈A π(a) = 1). The mixture distribution is
Pt (a) = (1 − γ)P̃t (a) + γπ(a) ,
where γ is a constant mixing factor to be chosen later. The algorithm then simply
samples its action At from Pt :
A t ∼ Pt .
Recall that Yt = hAt , yt i is the observed loss after taking action At . We need a
.
way to estimate yt (a) = ha, yt i. The idea is to use least squares to estimate yt with
Ŷt = Rt At Yt , where Rt ∈ Rd×d is selected so that Ŷt is an unbiased estimate of yt
given the history. Then the loss for a given action is estimated by Ŷt (a) = ha, Ŷt i.
To find the choice of Rt that makes Ŷt unbiased, let Et [·] = E [·|Pt ] and calculate
!
X
Et [Ŷt ] = Rt Et [At At ]yt = Rt
>
Pt (a)aa>
yt .
a∈A
| {z }
Qt

Using Rt = Q−1 t leads to Et [Ŷt ] = yt as desired. Of course Qt should be non-

singular, which will follow by choosing π so that
X
Q(π) = π(a)aa>
a∈A

is non-singular. The complete algorithm is summarised in Algorithm 15.

1: Input Finite action set A ⊂ Rd , learning rate η, exploration distribution π,

exploration parameter γ
2: for t = 1, 2, . . . , n do
3: Compute sampling distribution:
 P 
exp −η s=1 Ŷs (a)
t−1

Pt (a) = γπ(a) + (1 − γ) P  P .
a0 ∈A exp −η s=1 Ŷs (a )
t−1 0

4: Sample action At ∼ Pt
5: Observe loss Yt = hAt , yt i and compute loss estimates:
Ŷt = Q−1
t At Yt and Ŷt (a) = ha, Ŷt i .
6: end for
Algorithm 15: Exp3 for linear bandits.
 27.2 Regret Analysis 319

27.2 Regret Analysis

Theorem 27.1. Assume that A is non-empty and let k = |A|. For any exploration
distribution π, for some parameters η and γ, for all (yt )t with yt ∈ L, the regret
of Algorithm 15 satisfies
p
Rn ≤ 2 (2g(π) + d)n log(k) , (27.1)

where g(π) = maxa∈A kak2Q−1 (π) . Furthermore, there exists an exploration

distribution
p π and parameters η and γ such that g(π) ≤ d, and hence Rn ≤
2 3dn log(k).

The utility of (27.1) is that at times, calculating the distribution that minimises
g(π) or sampling from it may be difficult, in which case, one may employ a
distribution that trades off computation with the regret.

Proof Assume that the learning rate η is chosen so that for each round t the
loss estimates satisfy

η Ŷt (a) ≥ −1, ∀a ∈ A . (27.2)

Then, by adopting the proof of Theorem 11.1 (see Exercise 27.1), the regret is
bounded by
" #
log k Xn X
2
Rn ≤ + 2γn + η E Pt (a)Ŷt (a) . (27.3)
η t=1 a∈A

Note that we cannot use the proof that leads to the tighter constant (η getting
replaced by η/2 in the second term above) because we would loose too much in
other parts of the proof by guaranteeing that the loss estimates are bounded
by one (see below). To get a regrethP bound, it remains
i to set γ and η so that
2
(27.2) is satisfied and to bound E a Pt (a)Ŷt (a) . We start with the latter. Let
P 2
Mt = a Pt (a)Ŷt (a). By the definition of the loss estimate,

Ŷt2 (a) = (a> Q−1 2 2 > −1

t At Yt ) = Yt At Qt aa Qt At ,
> −1

P 2 2 > −1
which means that Mt = a Pt (a)Ŷt (a) = Yt At Qt At ≤ At Qt At =
> −1

trace(At At Qt ), and by the linearity of trace,

> −1

!
X
E[Mt | Pt ] ≤ trace Pt (a)aa Qt
> −1
= d.
a∈A

It remains to choose γ and η. Strengthen (27.2) to |η Ŷt (a)| ≤ 1 and note that
since |Yt | ≤ 1,

|η Ŷt (a)| = |ηa> Q−1 > −1

t At Yt | ≤ η|a Qt At | .
P
Recall that Q(π) = a∈A π(a)aa . Clearly Qt  γQ(π), and hence Qt
> −1

 27.3 Continuous Exponential Weights 320

Q(π)−1 /γ by Exercise 27.4. Using this and the Cauchy–Schwarz inequality shows
that
1 g(π)
t At | ≤ kakQ−1 kAt kQ−1 ≤ max ν Qt ν ≤
|a> Q−1 max ν > Q−1 (π)ν =
> −1
,
t t ν∈A γ ν∈A γ

which implies that

η
|η Ŷt (a)| ≤ g(π) . (27.4)
γ

Choosing γ = ηg(π) guarantees |η Ŷt (a)| ≤ 1. Plugging this choice into (27.3), we
get

log k p
Rn ≤ + ηn(2g(π) + d) = 2 (2g(π) + d)n log(k) ,
η
q
log(k)
where the last equality is derived by choosing η = (2g(π)+d)n finishing the proof
of (27.1).
For the second half, recall that by the Kiefer–Wolfowitz theorem (Theorem 21.1
and Exercise 21.6), there exists a sampling distribution π such that g(π) ≤ d.
Plugging this value into (27.1), finishes the proof.

27.3 Continuous Exponential Weights

The dependence on log(k) in the regret guarantee provided by Theorem 27.1

is objectionable when the number of arms is extremely large or infinite. One
approach is to find a finite subset C ⊆ A for which

sup min sup |ha − b, yi| ≤ 1/n .

a∈A b∈C y∈L

A standard calculation shows (Exercise 27.6) shows that C can always be chosen
so that log |C| ≤ d log(6dn). Then it is easy to check that
p Exp3 on C suffers regret
relative to the best action in A of at most Rn = O(d n log(nd)). The problem
with this approach is that C is exponentially large in d, which makes this algorithm
intractable in most situations. When A is convex, a more computationally
tractable approach is to use the continuous exponential weights algorithm.

For this section, we assume that A is convex and has positive Lebesgue
measure. The latter condition can be relaxed with some care (Exercise 27.10).

Let π be a probability measure supported on A. The continuous exponential

weights policy samples At from Pt = (1 − γ)P̃t + γπ, where P̃t is a measure
 27.3 Continuous Exponential Weights 321

supported on A defined by
R  P 
exp (a)
t−1
B
−η Ŷ
s=1 s da
P̃t (B) = R  P  . (27.5)
exp −η s=1 Ŷs (a) da
t−1
A

We will shortly see that the analysis in the previous section can be copied almost
verbatim to prove a regret bound for this strategy. But what has been bought here?
Rather than sampling from a discrete distribution on a large number of arms, we
now have to sample from a probability measure on a convex set. Sampling from
arbitrary probability measures is itself a challenging problem, but under certain
conditions there are polynomial time algorithms for this problem. The factors
that play the biggest role in the feasibility of sampling from a measure are (a)
the form of the measure or its density and (b) how the convex set is represented.
As it happens, the measure defined in the last display is log-concave, which
means that the logarithm of the density, with respect to the Lebesgue measure
on A, is a concave function.
Theorem 27.2. Let p(a) ∝ IA (a) exp(−f (a)) be a density with respect to the
Lebesgue measure on A such that f : A → R is a convex function. Then there
exists a polynomial-time algorithm for sampling from p, provided one can compute
the following efficiently:

1 (First-order information): ∇f (a) where a ∈ A.

2 (Euclidean projections): argminx∈A kx − yk2 where y ∈ Rd .
The probability distribution defined by Eq. (27.5) satisfies the first condition.
Efficiently computing a projection on to a convex set is a more delicate issue.
A general criterion that makes this efficient is access to a separation oracle,
which is a computational procedure φ that accepts a point x ∈ Rd as input and
responds φ(x) = true if x ∈ A and otherwise φ(x) = u, where hy, ui > hx, ui for
all y ∈ A (see Fig. 27.1).

x
A

Figure 27.1 Separation oracle returns the normal of a hyperplane that separates x from
A whenever x ∈
/ A. When x ∈ A, the separation oracle returns true.

Define log+ (x) = max(0, log(x)).

Theorem
R 27.3. Assume that A is compact, convex and has volume vol(A) =
A
da > 0. Then an appropriately tuned instantiation of the continuous
 27.4 Notes 322

exponential weights algorithm with Kiefer–Wolfowitz exploration has regret bounded

by
q
Rn ≤ 2d 3n(1 + log+ (2n/d)) .
The proof of Theorem 27.3 relies on the following proposition, which we leave
as an exercise (Exercise 27.11).
Proposition 27.4. Let K ⊂ Rd be a compact convex set with vol(K) > 0, u ∈ Rd
and let x∗ = argminx∈K hx, ui. Then,
    
vol(K) supx,y∈K hx − y, ui
log R ≤ d 1 + log + .
K
exp (−hx − x∗ , ui) dx d

The left-hand side in the above display is the logarithmic Laplace transform
of the uniform measure on K − {x∗ } evaluated at u.

Proof of Theorem 27.3 As before, choosing γ = dη ensures that |ηha, Ŷt i| ≤ 1 for
all a ∈ A (see the proof of Theorem 27.1). The standard argument (Exercise 27.9)
shows that
  
1   vol(A)
Rn ≤ E log R  P   + 3ηdn . (27.6)
η exp −η (Ŷ (a) − Ŷ (a∗ )) da
n
A t=1 t t

Pn
Using again that η|ha, Ŷt i| ≤ 1 and Proposition 27.4 with u = η t=1 Ŷt shows
that
d(1 + log+ (n/d)) q
Rn ≤ + 3ηdn ≤ 2d 3n(1 + log+ (2n/d)) .
η

27.4 Notes

1 A naive implementation of Algorithm 15 has computation complexity O(kd+d3 )

per round. There is also the one-off cost of computing the exploration
distribution, the complexity of which was discussed in Chapter 21. The real
problem is that k can be extremely large. This is especially true when the
action set is combinatorial. For example, when A = {a ∈ Rd : ai = ±1} is
the corners of the hypercube |A| = 2d , which is much too large unless the
dimension is small. Such problems call for a different approach that we present
in the next chapter and in Chapter 30.
2 It is not important to find exactly the optimal exploration distribution. All
that is needed is a bound on Eq. (27.4), which for the exploration distribution
based on the Kiefer–Wolfowitz theorem is just d. However, unlike in the finite
case, exploration is crucial and cannot be removed (Exercise 27.8).
 27.5 Bibliographic Remarks 323

√
3 The O( n) dependence of the regret on the horizon is not improvable, but the
linear dependence on the dimension is suboptimal for certain action sets and
optimal for others. An example where improvement is possible occurs when A
is the unit ball, which is analysed in the next chapter.
4 A slight modification of the set-up allows the action set to change in each
round, but where actions have identities. Suppose that k ∈ {1, 2, . . .} and At =
{a1 (t), . . . , ak (t)} and the adversary chooses losses so that maxa∈At |ha, yt i| ≤ 1
for all t. Then a straightforward adaptation of Algorithm 15 and Theorem 27.1
leads to an algorithm for which
" n #
X p
Rn = max E hAt − ai (t), yt i ≤ 2 3dn log(k) .
i∈[k]
t=1

The definition of the regret still compares the learner to the best single action
in hindsight, which makes it less meaningful than the definition of the regret
in Chapter 19 for stochastic linear bandits with changing action sets. These
differences are discussed in more detail in Chapter 29. See also Exercise 27.5.

27.5 Bibliographic Remarks

The results in Sections 27.1 and 27.2 follow the article by Bubeck et al. [2012],
with minor modifications to make the argument more pedagogical. The main
difference is that they used John’s ellipsoid over the action set for exploration,
which is only the right thing when John’s ellipsoid is also a central ellipsoid. Here
we use Kiefer–Wolfowitz, which is equivalent to finding the minimum volume
central ellipsoid containing the action set. Theorem 27.2, which guarantees
the existence of a polynomial time sampling algorithm for convex sets with
gradient information and projections is by Bubeck et al. [2015b]. We warn the
reader that these algorithms are not very practical, especially if theoretically
justified parameters are used. The study of sampling from convex bodies is quite
fascinating. There is an overview by Lovász and Vempala [2007], though it is a
little old. The continuous exponential weights algorithm is perhaps attributable
to Cover [1991] in the special setting of online learning called universal portfolio
optimisation. The first application to linear bandits is by Hazan et al. [2016].
Their algorithm and analysis are more complicated because they seek to improve
the computation properties by replacing the exploration distribution based on
Kiefer–Wolfowitz with an adaptive randomised exploration basis that can be
computed in polynomial time under weaker assumptions. Continuous exponential
weights for linear bandits using the core set of John’s ellipsoid for exploration
(rather than Kiefer–Wolfowitz) was recently p analysed by van der Hoeven et al.
[2018]. Another path towards an efficient O(d n log(·)) policy for convex action
sets is to use the tools from online optimisation. We explain some of these ideas
in more detail in the next chapter, but the reader is referred to the paper by
Bubeck and Eldan [2015].
 27.6 Exercises 324

27.6 Exercises

27.1 (‘Mixed’ Exp3 analysis) Prove Eq. (27.3).

27.2 (Dependence on the range of losses) Suppose that instead of assuming

yt ∈ L, we assume that yt ∈ {y ∈ Rd : supa∈A |ha, yi| ≤ b} for some known b > 0.
Modify the algorithm to accommodate this change, and explain how the regret
guarantee changes.

27.3 (Dependence on the range of losses (ii)) Now suppose that a < b
are known and yt ∈ {y ∈ Rd : ha, yi ∈ [a, b] for all a ∈ A}. How can you adapt
the algorithm now, and what is its regret?

27.4 (Inversion reverses Loewner orders) Let A, B ∈ Rd×d and suppose

that A  B and B is invertible. Show that A−1  B −1 .

27.5 (Changing action sets) Provide the necessary corrections to

Algorithm 15 and its analysis to prove the result claimed in Note 4.

Hint You will need to choose a new exploration distribution in every round.
Otherwise everything is more or less the same.
27.6 (Covering numbers for convex sets) For K ⊂ Rd let kxkK =
supy∈K |hx, yi|. Let A ⊂ Rd and L = {y : kykA ≤ 1}. Let N (A, ε) be the size of
the smallest subset C ⊆ A such that minx0 ∈C kx − x0 kL ≤ ε for all x ∈ A. Show
the following:

(a) When A = x ∈ Rd : kxkV −1 ≤ 1 , we have N (A, ε) ≤ (3/ε)d .
(b) When A is convex, bounded and span(A) = Rd we have N (A, ε) ≤ (3d/ε)d .
(c) For any bounded A ⊂ Rd we have N (A, ε) ≤ (6d/ε)d .

Hint For the first part, find a linear map from A to the Euclidean ball and
use the fact that the Euclidean ball can be covered with a set of size (3/ε)d . For
the second part use the fact that for any symmetric, convex and compact set K
there exists an ellipsoid E = {x : kxkV ≤ 1} such that E ⊆ K ⊆ dE.
27.7 (Low rank action sets (i)) In the definition of the algorithm and the
proof of Theorem 27.1, we assumed that A spans Rd and that it has positive
Lebesgue measure. Show that this assumption may be relaxed by carefully
adapting the algorithm and analysis.

27.8 (Necessity of exploration) We saw in Chapter 11 that the exponential

weights algorithm achieved near-optimal regret without mixing additional
exploration. Show that exploration is crucial here. More precisely, construct a
finite action set A and reward sequence yt ∈ L such that the regret of Algorithm 15
with γ = 0 becomes very poor (even with η optimally tuned) relative to the
optimal choice.
 27.6 Exercises 325

27.9 (Continuous exponential weights) Complete the missing steps in the

proof of Theorem 27.3.

27.10 (Low rank action sets (ii)) In the definition of the algorithm and
the proof of Theorem 27.3, we assumed that A spans Rd and that it has positive
Lebesgue measure. Show that this assumption may be relaxed by carefully
adapting the algorithm and analysis.

27.11 (Volume bounds) Prove Proposition 27.4.

28 Follow-the-Regularised-Leader and
Mirror Descent

In the last chapter, we showed that if A ⊂ Rd has k elements, then the regret of
Exp3 with a careful exploration distribution has regret
p
Rn = O( dn log(k)) .
We also showed the continuous version of this algorithm has regret at most
p
Rn = O(d n log(n)) .
Although this algorithm can often be made to run in polynomial time, the degree
tends to be high and the implementation complicated, making the algorithm
impractical. In many cases this can be improved, both in terms of the regret and
computation. In this chapter we demonstrate this in the case when A is the unit
ball by showing that for this case pthere is an efficient, low-complexity algorithm
for which the regret is Rn = O( dn log(n)). More importantly, however, we
introduce a pair of related algorithms called follow-the-regularised-leader and
mirror descent, which are powerful tools for the design and analysis of bandit
algorithms. In fact, the exponential weights algorithm turns out to be a special case.

28.1 Online Linear Optimisation

Mirror descent originated in the convex optimisa-

tion literature. The idea has since been adapted
to online learning and specifically to online lin-
ear optimisation. Online linear optimisation
is the full information version of the adversarial
linear bandit, where at the end of each round the
learner observes the full vector yt . Let A ⊂ Rd Figure 28.1 Mirror descent is a
be a convex set and L ⊂ Rd be an arbitrary set modern art, as well as science
called the loss space. Let y1 , . . . , yn be a sequence of loss vectors with yt ∈ L
for t ∈ [n]. In each round the learner chooses at ∈ A and subsequently observes
the vector yt . The regret relative to a fixed comparator a ∈ A is
n
X
Rn (a) = hat − a, yt i ,
t=1
 28.1 Online Linear Optimisation 327

and the regret is Rn = maxa∈A Rn (a). We emphasise that the only difference
relative to the adversarial linear bandit is that now yt is observed rather than
hat , yt i. Actions are not capitalised in this section because the algorithms presented
here do not randomise.

Mirror descent
The basic version of mirror descent has two extra parameters beyond n and A. A
learning rate η > 0 and a convex function F : Rd → R̄ with domain D = dom(F ).
Usually F will be Legendre. The function F is called a potential function or
regulariser. In the first round, mirror descent predicts

a1 = argmina∈A F (a) . (28.1)

Subsequently it predicts

at+1 = argmina∈A (ηha, yt i + DF (a, at )) , (28.2)

where DF (a, at ) is the F -induced Bregman divergence between a and at . Implicit

in the definition is that a1 , a2 , . . . are well-defined. The reader is invited to
construct examples when this is not the case (Exercise 28.2). A simple case when
(at )nt=1 are well-defined is when A is compact and F is Legendre.

Follow-the-Regularised-Leader
Like mirror descent, follow-the-regularised-leader depends on a convex potential
F with domain D = dom(F ) and predicts a1 = argmina∈A F (a). In subsequent
rounds t ∈ [n], the predictions are
t
!
X
at+1 = argmina∈A η ha, ys i + F (a) . (28.3)
s=1

The intuition is that the algorithm chooses at+1 to be the action that performed
best in hindsight with respect to the regularised loss. Again, the definition of
follow-the-regularised-leader implicitly assumes that (at )nt=1 are well-defined. As
for mirror descent, the regularisation serves to stabilise the algorithm, which
turns out to be a key property of good algorithms for online linear prediction.

Follow-the-leader chooses the action that appears best in hindsight,

Pt
at+1 = argmina∈A s=1 ha, ys i. In general this algorithm is not well suited
for online linear optimisation because the absence of regularisation makes it
unstable (Exercise 28.4).

Equivalence of Mirror Descent and Follow-the-Regularised-Leader

At first sight these algorithms do not look that similar. To clarify matters, let us
suppose that F is Legendre with domain D ⊆ A. In this setting, mirror descent
 28.1 Online Linear Optimisation 328

and follow-the-regularised-leader are identical. To see this, let

Φt (a) = ηha, yt i + DF (a, at ) = ηha, yt i + F (a) − F (at ) − ha − at , ∇F (at )i .

Now mirror descent chooses at+1 to minimise Φt . The reader should check that
the assumption that F is Legendre on domain D ⊆ A implies that the minimiser
occurs in the interior of D ⊆ A and that ∇Φt (at+1 ) = 0 (see Exercise 28.1). This
means that ηyt = ∇F (at ) − ∇F (at+1 ), and so
t
X t
X
∇F (at+1 ) = −ηyt + ∇F (at ) = ∇F (a1 ) − η ys = −η ys ,
s=1 s=1

where the last equality is true because a1 is chosen as the minimiser of F in

A ∩ D = D, and again the fact that F is Legendre ensures this minimum occurs
at an interior point where the gradient vanishes. Follow the regularised leader
Pt
chooses at+1 to minimise Φ0t (a) = η s=1 ha, ys i + F (a). The same argument
shows that ∇Φ0t (at+1 ) = 0, which means that
t
X
∇F (at+1 ) = −η ys .
s=1

The last two displays and the fact that the gradient for Legendre functions is
invertible shows that mirror descent and follow-the-regularised-leader are the
same in this setting.

The equivalence between these algorithms is far from universal. First of

all, it does not generally hold when F is not Legendre or its domain
is larger than A. Second, in many applications of these algorithms, the
learning rate or potential change with time, and in either case the
algorithms will typically produce different action sequences. For example, if
a learning rate ηt is used rather than η in the definition of Φt , then mirror
Pt
descent chooses ∇F (at+1 ) = − s=1 ηs ys , while follow-the-regularised-
Pt
leader chooses ∇F (at+1 ) = −ηt s=1 ys . We return to this issue in the
notes and exercises.

Example 28.1. Let A = Rd and F (a) = 12 kak22 . Then ∇F (a) = a and

D(a, at ) = 12 ka − at k22 . Clearly F is Legendre and D = A, so mirror descent and
follow-the-regularised-leader are the same. By simple calculus we see that
1
at+1 = argmina∈Rd ηha, yt i + ka − at k22 = at − ηyt ,
2
which may be familiar as online gradient descent with linear losses. For the
extension to nonlinear convex loss functions, see Note 12.
1 2
Example 28.2. Let A be a compact convex subset of Rd and F (a) = 2 kak2 .
 28.1 Online Linear Optimisation 329

Then mirror descent chooses

1
at+1 = argmina∈A ηha, yt i + ka − at k22 = Π(at − ηyt ) , (28.4)
2
where Π(a) is the Euclidean projection of a on to A. This algorithm is usually
called online projected gradient descent. On the other hand, for follow-the-
regularised-leader we have
!
1
Xt Xt
at+1 = argmina∈A η ha, ys i + ka − at k22 = Π −η ys ,
s=1
2 s=1

which may be a different choice than that of mirror descent.

Example 28.3. The exponential weights algorithm that appeared in various

forms on numerous occasions in earlier chapters is a special case of mirror descent
corresponding to choosing the constraint set A as the simplex in Rd and choosing
F to be the unnormalised negentropy function of Example 26.5. In this case
follow-the-regularised-leader chooses
t
X d
X
at+1 = argmina∈A η ha, ys i + ai log(ai ) − ai .
s=1 i=1

You will show in Exercise 28.8 that

 P 
exp −η s=1 ysi
t

at+1,i =P  P . (28.5)
exp
d t
j=1 −η s=1 ysj

28.1.1 A Two-Step Process for Implementation

Solving the optimisation problem in Eq. (28.2) is often made easier by using
Theorem 26.15 from Chapter 26. Assume F is Legendre and A is compact and
non-empty, and suppose that

∇F (a) − ηy ∈ int(dom(F ∗ )) for all a ∈ A ∩ D and y ∈ L . (28.6)

Then the solution to Eq. (28.2) can be found using the following two-step
procedure:

ãt+1 = argmina∈D ηha, yt i + DF (a, at ) and (28.7)

at+1 = argmina∈A DF (a, ãt+1 ) . (28.8)

Eq. (28.6) means the first optimisation problem can be evaluated explicitly as
the solution to

ηyt + ∇F (ãt+1 ) − ∇F (at ) = 0 . (28.9)

Since F is Legendre, Theorem 26.6 shows that ∇F is a bijection between

int(dom(F )) and int(dom(F ∗ )), which means that ãt+1 = (∇F )−1 (∇F (at ) − ηyt ).
 28.2 Regret Analysis 330

The optimisation problem in Eq. (28.8) is usually harder to calculate analytically,

but there are important exceptions, as we shall see.

All potentials and losses that appear in positive results in this book guarantee
that mirror descent (and also follow-the-regularised-leader) are well defined,
and that condition in Eq. (28.6) holds.

The two-step implementation of mirror descent also explains its name. The
update in round t can be seen as transforming the action at ∈ A into the ‘mirror’
(dual) space using ∇F , where it is combined with the most recent (scaled) loss
ηyt . Then ∇F −1 is used to transform the updated vector back to the original
(primal) space. The function ∇F is called the mirror map.

The same idea works for follow-the-regularised-leader. Assuming F is

Pt
Legendre, A is compact and nonempty and −η s=1 ys ∈ int(dom(F ∗ )),
then for follow-the-regularised-leader
t
!!
X
at+1 = ΠA,F ∇F −1
−η ys ,
s=1

where ΠA,F is the projection on to A with respect to DF as described in

Section 26.6.

Some of the differences between follow-the-regularised-leader and mirror descent

are illustrated in Fig. 28.2, which shows how the algorithms differ once projections
start to occur.

28.2 Regret Analysis

Although mirror descent and follow-the-regularised-leader are not the same, the
bounds presented here are identical. The theorem for mirror descent has two
parts, the first of which is a little stronger than the second. To minimise clutter,
we abbreviate DF by D.

Theorem 28.4 (Mirror descent regret bound). Let η > 0 and F be Legendre with
domain D and A ⊂ Rd be a non-empty convex set with int(dom(F )) ∩ A = 6 ∅. Let
a1 , . . . , an+1 be the actions chosen by mirror descent, which are assumed to be
well-defined. Then, for any a ∈ A, the regret of mirror descent is bounded by

F (a) − F (a1 ) X 1X
n n
Rn (a) ≤ + hat − at+1 , yt i − D(at+1 , at ) .
η t=1
η t=1

Furthermore, suppose that Eq. (28.6) holds and ã2 , ã3 , . . . , ãn+1 are given by
 28.2 Regret Analysis 331

Constraint set Primal Dual

a1 ∇F

−ηy1
∇F −1
a2
∇F
−ηy2
ΠA,F
a3 ∇F −1

∇F −ηy3
−ηy3
afollow-the-regularised-leader
4

∆F −1

∆F −1
aMD
4
∆F

Figure 28.2 Illustration of follow-the-regularised-leader and mirror descent. The

constraint set is A, and the function ΠA,F is the projection on to A with respect
to the Bregman divergence induced by Legendre function F . Follow-the-regularised-
leader accumulates the scaled losses in the dual space, mapping back to the primal
using the inverse map ∇F −1 . Mirror descent computes the next iterate by at+1 =
ΠA,F (∇F −1 (∇F (at ) − ηyt ). The algorithms generally behave differently in the presence
of projections. In the figure, the algorithms behave the same until the fourth iterate,
after which the projection that appeared in the computation of the third iterate breaks
the equivalence.

Eq. (28.7). Then,

!
1
n
X
Rn (a) ≤ F (a) − F (a1 ) + D(at , ãt+1 ) .
η t=1

Proof Fix a ∈ A. The result trivially holds when a 6∈ D. Hence, we assume that
a ∈ D. For the first part of the claim, we split the inner product:

hat − a, yt i = hat − at+1 , yt i + hat+1 − a, yt i .

In Exercise 28.1, you will show that at ∈ int(dom(F )), and hence the Bregman
divergence D(b, at ) = F (b) − F (at ) − hb − at , ∇F (at )i for any b ∈ dom(F ). By
definition, at+1 = argminb∈A ηhb, yt i + D(b, at ). Hence, the first-order optimality
conditions for at+1 (Proposition 26.14) show that

ha − at+1 , ηyt + ∇F (at+1 ) − ∇F (at )i ≥ 0 .

 28.2 Regret Analysis 332

Reordering and using the definition of the Bregman divergence,

1
hat+1 − a, yt i ≤ ha − at+1 , ∇F (at+1 ) − ∇F (at )i
η
1
= (D(a, at ) − D(a, at+1 ) − D(at+1 , at )) .
η
Using this, along with the definition of the regret,
n
X
Rn = hat − a, yt i
t=1

1X
Xn n
≤ hat − at+1 , yt i + (D(a, at ) − D(a, at+1 ) − D(at+1 , at ))
t=1
η t=1
!
1
Xn n
X
= hat − at+1 , yt i + D(a, a1 ) − D(a, an+1 ) − D(at+1 , at )
t=1
η t=1

F (a) − F (a1 ) 1 X
Xn n
≤ hat − at+1 , yt i + − D(at+1 , at ) , (28.10)
t=1
η η t=1

where the final inequality follows from the fact that D(a, an+1 ) ≥ 0 and
D(a, a1 ) ≤ F (a) − F (a1 ), the latter of which is true by the first-order optimality
conditions for a1 = argminb∈A F (b). To see the second part, note that
1
hat − at+1 , yt i = hat − at+1 , ∇F (at ) − ∇F (ãt+1 )i
η
1
= (D(at+1 , at ) + D(at , ãt+1 ) − D(at+1 , ãt+1 ))
η
1
≤ (D(at+1 , at ) + D(at , ãt+1 )) .
η
The result follows by substituting this into Eq. (28.10).

The assumption that a1 minimises the potential was only used to bound
D(a, a1 ) ≤ F (a) − F (a1 ). For a different initialisation, the following bound
still holds:
!
1 X n
Rn (a) ≤ D(a, a1 ) + D(at , ãt+1 ) . (28.11)
η t=1

As we shall see in Chapter 31, this is useful when using mirror descent to
analyse non-stationary bandits.

The first part of Theorem 28.4 also holds for follow-the-regularised-leader as

stated in the next result, the proof of which is left for Exercise 28.5.

Theorem 28.5 (Follow-the-regularised-leader regret bound). Let η > 0, F

be convex with domain D, A ⊆ Rd be a non-empty convex set. Assume that
 28.2 Regret Analysis 333

a1 , . . . , an+1 chosen by follow-the-regularised-leader are well defined. Then, for

any a ∈ A, the regret of follow-the-regularised-leader is bounded by
F (a) − F (a1 ) X 1X
n n
Rn (a) ≤ + hat − at+1 , yt i − D(at+1 , at ) .
η t=1
η t=1

We now give two applications of the regret bound of Theorem 28.4 for mirror
descent. The same results would hold for the same problems for FTRL just in this
case we would need to use Theorem 28.5. Let diamF (A) = maxa,b∈A F (a) − F (b)
be the diameter of A with respect to F .
Proposition 28.6 (Regret on the unit ball). Let A = B2d = {a ∈ Rd : kak2 ≤ 1}
be the standard unit ball and assume
pyt ∈ B2 for all t. Then mirror descent with
d
1 2
potential F (a) = 2 kak2 and η = 1/n is well defined and its regret satisfies
√
Rn ≤ n.
Proof That mirror descent is well defined follows by a direct calculation (cf.
Example 28.1). By Eq. (28.9), we have ãt+1 = at − ηyt so
1 η2
D(at , ãt+1 ) = kãt+1 − at k22 = kyt k22 .
2 2
Therefore, since diamF (A) = 1/2 and kyt k2 ≤ 1 for all t,

diamF (A) η X 1
n
ηn √
Rn ≤ + kyt k22 ≤ + = n.
η 2 t=1 2η 2

Proposition 28.7 (Regret on the simplex). Let A = Pd−1 be the probability

simplex and yt ∈ L = [0, 1]d forpall t. Then mirror descent with the unnormalised
p potential and η = 2 log(d)/n is well defined and its regret satisfies
negentropy
Rn ≤ 2n log(d).
Proof That mirror descent is well-defined follows because the simplex is compact.
The Bregman divergence with respect to the unnormalised negentropy potential
Pd
for a, b ∈ A is D(a, b) = i=1 ai log(ai /bi ). Therefore,

F (a) − F (a1 ) X 1X
n n
Rn (a) ≤ + hat − at+1 , yt i − D(at+1 , at )
η t=1
η t=1
log(d) X 1X1
n n
≤ + kat − at+1 k1 kyt k∞ − kat − at+1 k21
η t=1
η t=1
2
log(d) η X log(d) ηn p
n
≤ + kyt k2∞ ≤ + = 2n log(d) ,
η 2 t=1 η 2

where the first inequality follows from Theorem 28.4, the second from Pinsker’s
inequality and the facts that diamF (A) = log(d). In the third inequality, we
used ‘optimise to bound’. In particular, we used that for any a ∈ R and b > 0,
maxx∈R ax − bx2 /2 = a2 /(2b). The last inequality follows from the assumption
that kyt k∞ ≤ 1.
 28.3 Application to Linear Bandits 334

The last few steps in the above proof are so routine that we summarise their
use in a corollary, the proof of which we leave to the reader (Exercise 28.6).

Corollary 28.8. Let F be a Legendre potential and k · kt be a norm on Rd for

each t ∈ [n] such that DF (at+1 , at ) ≥ 12 kat+1 − at k2t . Then the regret of mirror
descent or follow-the-regularised-leader satisfies

diamF (A) η X
n
Rn ≤ + kyt k2t∗ ,
η 2 t=1

where kykt∗ = maxx:kxkt ≤1 hx, yi is the dual norm of k · kt .

It often happens that the easiest way to bound the regret of mirror descent is to
find a norm that satisfies the conditions of Corollary 28.8. Often, Theorem 26.13
provides a good approach.

Example 28.9. To illustrate a suboptimal application of mirror descent,

suppose we had chosen F (a) = 12 kak22 in the setting of Proposition 28.7. Then
DF (at+1 , at ) = 12 kat+1 −at k22 suggests choosing k·kt to be the standard Euclidean
norm. Since diamF (A) = 1/2 and k · k2∗ = k · k2 , applying Corollary 28.8 shows
that

1
n
ηX
Rn ≤ + kyt k22 .
2η 2 t=1

2
But now we√ see that kyt k2 canpbe as large as d, and tuning η would lead to a
rate of O( nd) rather than O( n log(d)).

Both Theorems 28.4 and 28.5 were presented for the oblivious case where
(yt )nt=1 are chosen in advance. This assumption was not used, however, and in
fact the bounds continue to hold when yt is chosen strategically as a function
of a1 , y1 , . . . , yt−1 , at . This is analogous to how the basic regret bound for
exponential weights continues to hold in the face of strategic losses. But
be cautioned, this result does not carry immediately to the application of
mirror descent to bandits, as discussed at the end in Note 9.

28.3 Application to Linear Bandits

We now show how mirror descent and follow-the-regularised-leader can be used to

construct algorithms for adversarial linear bandit problems. Like in the previous
chapter, the adversary chooses a sequence of vectors y1 , . . . , yn with yt ∈ L ⊂ Rd .
In each round the learner chooses At ∈ A ⊂ Rd and observes hAt , yt i. The regret
 28.3 Application to Linear Bandits 335

relative to action a ∈ A is
" n #
X
Rn (a) = E hAt − a, yt i .
t=1

The regret is Rn = maxa∈A Rn (a). The application of mirror descent and follow-
the-regularised-leader to linear bandits is straightforward. The only difficulty
is that the learner does not observe yt but instead hAt , yt i. The solution is to
replace yt with an estimator, which is typically some kind of importance-weighted
estimator as in the previous chapter. Because estimation of yt is only possible
using randomisation, the algorithm cannot play the suggested action of mirror
descent, but instead plays a distribution over actions with the same mean as the
proposed action. This is often necessary anyway, when A is not convex. Since
the losses are linear, the expected additional regret by playing according to the
distribution vanishes. The algorithm is summarised in Algorithm 16. We have
switched to capital letters because the actions are now randomised.

Theorem 28.10 (Regret of Mirror-Descent and FTRL with bandit feedback).

Suppose that Algorithm 16 is run with Legendre potential F , convex action set
A ⊂ Rd and learning rate η > 0 such that the loss estimators are unbiased:
E[Ŷt | Āt ] = yt for all t ∈ [n]. Then the regret for either variant of Algorithm 16,
provided that they are well defined, is bounded by
" #
F (a) − F (Ā1 ) X 1X
n n
Rn (a) ≤ E + hĀt − Āt+1 , Ŷt i − D(Āt+1 , Āt ) .
η t=1
η t=1

Furthermore, letting
Ãt+1 = argmina∈dom(F ) ηha, Ŷt i + DF (a, Āt )

and assuming that −η Ŷt + ∇F (a) ∈ ∇F (dom(F )) for all a ∈ A almost surely,
the regret of the mirror descent variation satisfies
diamF (A) 1 X 
n

Rn ≤ + E D(Āt , Ãt+1 ) .
η η t=1

Proof Using the definition of the algorithm and the assumption that Ŷt is
unbiased given Āt and that Pt has mean Āt leads to
     h h  ii

E [hAt , yt i] = E hĀt , yt i = E E hĀt , yt i | Āt = E E hĀt , Ŷt i  Āt ,

where the last equality used the linearity of expectations. Hence,

" n # " n #
X X
Rn (a) = E hAt , yt i − ha, yt i = E hĀt − a, Ŷt i ,
t=1 t=1

which is the expected random regret of mirror descent or follow-the-regularised-

leader on the recursively constructed sequence Ŷt . The result follows from
Theorem 28.4 or Theorem 28.5 and the note at the end of the last section
 28.4 Linear Bandits on the Unit Ball 336

that says these theorems continue to hold even for recursively constructed loss
sequences.

1: Input Legendre potential F , action set A and learning rate η > 0

2: Choose Ā1 = argmina∈A∩dom(F ) F (a)
3: for t = 1, . . . , n do
4: Choose measure Pt on A with mean Āt
5: Sample action At from Pt and observe hAt , yt i
6: Compute estimate Ŷt of the loss vector yt
7: Update:
Āt+1 = argmina∈A∩dom(F ) ηha, Ŷt i + DF (a, Āt ) (Mirror descent)
t
X
Āt+1 = argmina∈A∩dom(F ) η ha, Ŷs i + F (a)
s=1
(follow-the-regularised-leader)
8: end for
Algorithm 16: Online stochastic mirror descent/follow-the-regularised-leader.

28.4 Linear Bandits on the Unit Ball

To illustrate the power of these methods, we return to adversarial linear bandits

and the special case where the action set is the unit ball. In the previous chapter,
we showed that continuous exponential weights on the unit ball with Kiefer-
Wolfowitz exploration has a regret of
p
Rn = O(d n log(n)) .
Surprisingly, follow-the-regularised-leader
√ with a carefully chosen potential
improves on this bound by a factor of d.
For the remainder of this section, let k · k = k · k2 be the standard Euclidean
norm and A = B2d be the standard unit ball. In order to instantiate follow-the-
regularised-leader we need a potential, a sampling rule, an unbiased estimator and
a learning rate. Note that the only source of randomness is the randomisation in
the algorithm. Hence, let Et [·] = E[· | A1 , . . . , At−1 ]. We start with the sampling
rule and estimator. Recall that in round t we need to choose a distribution on
A with mean Āt and sufficient variability that the variance of the estimator is
not too large. Given the past, let Et and Ut be independent, where Et ∈ {0, 1} is
such that Et [Et ] = 1 − kĀt k and Ut is uniformly distributed on {±e1 , . . . , ±ed }.
The algorithm chooses
(1 − Et )Āt
At = Et Ut + .
kĀt k
 28.4 Linear Bandits on the Unit Ball 337

In other words, Et = 1 indicates that the algorithm explores, which happens with
probability 1 − kĀt k. Clearly, Et [At ] = Āt . (The sampling distribution Pt is just
the law of At given the past, which remains implicit.) For the estimator we use a
variant of the importance-weighted estimator from the last chapter:

d Et At hAt , yt i
Ŷt = . (28.12)
1 − kĀt k

The reader can check for themself that this estimator is unbiased. Next, we
inspect the contents of our magician’s hat and select the potential

F (a) = − log (1 − kak) − kak .

There is one more modification. Rather than instantiating follow-the-regularised-

leader with action set A, we use à = {x ∈ Rd : kxk2 ≤ r}, where r < 1 is a
radius to be tuned subsequently. The reason for this modification is to control
the variance of the estimator in Eq. (28.12), which blows up as Āt gets close to
the boundary. You will show in Exercise 28.7 that
! t−1
−η L̂t−1 X
Āt = Π with L̂t−1 = Ŷs , (28.13)
1 + ηkL̂t−1 k s=1

where Π(x) is the projection operator on to à with respect to k · k2 .

1: Input Learning rate η > 0

2: for t = 1, . . . , n do
3: Compute
! t−1
−η L̂t−1 X
Āt = Π with L̂t−1 = Ŷs
1 + ηkL̂t−1 k s=1

4: Sample Et ∈ {0, 1} from Binomial with bias 1 − kĀt k and Ut uniformly

on {e1 , . . . , ed }
5: Play action At = Et Ut + (1−E t )Āt
kĀt k
d Et At hAt , yt i
6: Observe hAt , yt i and estimate loss vector Ŷt =
1 − kĀt k
7: end for
Algorithm 17: Follow-the-regularised-leader for linear bandits on the unit ball

Theorem 28.11. Assume that (yt )nt=1 are a sequence of losses such that kyt k2 ≤ 1
for all t. Suppose that Algorithm 16 is run using the sampling rule, estimator
and potential as describedp above, shrunken action set à with r = 1 − 2ηd where
the learning rate is η = p log(n)/(3dn). Then, the algorithm is well defined and
its regret satisfies Rn ≤ 2 3nd log(n).
 28.4 Linear Bandits on the Unit Ball 338

You might notice that in some regimes this is smaller than the lower bound
for stochastic linear bandits (Theorem 24.2). There is no contradiction
because the adversarial and stochastic linear bandit models are actually
quite different. More details are in Chapter 29.

Proof That the algorithm is well defined follows because à is compact. Let
Pn
a∗ = argmina∈A t=1 ha, yt i be the optimal action. Then