Introduction Validity Precision Study Types In Summary

Chapter 2: Study design

MAST10010 Data Analysis

Department of Mathematics & Statistics

University of Melbourne

Slide 1/55
Introduction Validity Precision Study Types In Summary

Introduction
An example
Terminology
Validity
Comparison
Control
Randomisation & Confounding
Precision
Blocking
Replication
Balance
Study Types
Observational studies
Designed experiments
An Example
In Summary
Slide 2/55
Introduction Validity Precision Study Types In Summary

References

Pre-reading:
Utts and Heckard – Chapter 1: Case Studies

References:
Utts and Heckard – Chapter 6 (excellent coverage!)
DeVeaux, Velleman, Bock – Chapter 13

Slide 3/55
Introduction Validity Precision Study Types In Summary

Learning Outcomes:

At the end of this topic you should:

▶ Recognise observational studies and designed experiments,
and distinguish them
▶ Be able to explain and apply the principles of good study
design:
▶ Randomisation, comparison and control
▶ Replication, blocking/stratification and balance
▶ Explain confounding in the context of study design
▶ Understand when causal inferences may be made from a single
study

Slide 4/55
Introduction Validity Precision Study Types In Summary

How do we get good data?

Our purpose in this chapter is to understand how to design a study

so that we have meaningful data.

Meaningful data enables us to:

▶ draw conclusions that are valid (unbiased, address the
questions being asked)
▶ provide estimates (of effects) that have high
& measurable precision

Slide 4/55
Introduction Validity Precision Study Types In Summary

Bias & Precision

Slide 5/55
Introduction Validity Precision Study Types In Summary

An example

Bees & elephants

Are bees a natural deterrent for elephants?

▶ Lucy King, Iain Douglas-Hamilton and Fritz Vollrath are
zoologists who are concerned about the increasing conflict
between humans and elephants
▶ Elephants are herbivores and they have to eat a lot; they raid
crops
▶ Iain and Fritz had observed that there was less elephant
damage to acacia trees with beehives (empty or occupied)
than to trees without beehives.
▶ Other researchers have suggested elephants avoid beehives.

Slide 6/55
Introduction Validity Precision Study Types In Summary

An example

Questions

Question: Could bees be used to deter elephants, to stop them

from invading crops?

Study Design: How could we investigate this question?

Slide 7/55
Introduction Validity Precision Study Types In Summary

An example

Some Possibilities

Study Type 1

Play the sound of bees to the first 20 herds of elephants observed

at a particular location.

Slide 8/55
Introduction Validity Precision Study Types In Summary

An example

Some Possibilities

Study Type 1

Play the sound of bees to the first 20 herds of elephants observed

at a particular location.
▶ Are they responding to a sound . . . in general?
▶ Are they responding to the bee sound . . . in particular?
▶ Which herds are we observing? Might it be the same herds?

Slide 8/55
Introduction Validity Precision Study Types In Summary

An example

Possibilities

Study Type 2

Play the sound of bees to the first 10 herds of elephants observed

at a particular location, and another sound to the next 10 herds
found at the same location.

Slide 9/55
Introduction Validity Precision Study Types In Summary

An example

Possibilities

Study Type 2

Play the sound of bees to the first 10 herds of elephants observed

at a particular location, and another sound to the next 10 herds
found at the same location.
▶ If the two groups respond differently, can these differences be
attributed solely to the different sounds?
▶ Do the first 10 herds differ from the second 10 herds . . . in
some important way?
▶ Which herds are we observing? Might it be the same herds?

Slide 9/55
Introduction Validity Precision Study Types In Summary

An example

Possibilities

Study Type 3

Identify 20 different herds; choose 10 herds at random to hear the

sound of bees, and the remaining 10 to hear an alternative sound.
Play the sound at the place each herd is located.

Slide 10/55
Introduction Validity Precision Study Types In Summary

An example

Possibilities

Study Type 3

Identify 20 different herds; choose 10 herds at random to hear the

sound of bees, and the remaining 10 to hear an alternative sound.
Play the sound at the place each herd is located.
▶ Do there appear to be differences in responses to different
sounds?
▶ Can these differences be attributed to the different sounds?
▶ If the sounds are assigned at random, are the two sets of
herds likely to differ (systematically) in other ways?
▶ Are we observing independent responses of herds?

Slide 10/55
Introduction Validity Precision Study Types In Summary

Terminology

Some Terminology
1. Study units (experimental units, cases, subjects) — the units
or individuals on which data are obtained; one observation per
study unit.
2. Response variable — the variable of interest which we expect
may depend on other variables.
3. Explanatory variable (predictor, independent or treatment
variable) — a variable that is used to explain or predict (some
of) the variation in the response variable.
4. Other possible explanatory variables – explanatory variables
that may impact upon the response variable but are not of
primary interest to the study. We usually seek to either
control for these variables or measure them in some way so
that they don’t influence (bias) our findings.
Slide 11/55
Introduction Validity Precision Study Types In Summary

Terminology

For our example

▶ Study unit, (experimental unit):

▶ Response variable:
▶ Primary explanatory variable:
▶ Other possible explanatory variables:

Slide 12/55
Introduction Validity Precision Study Types In Summary

Terminology

For our example

▶ Study unit, (experimental unit): a herd of elephants

▶ Response variable: time for herd to respond to sound
▶ Primary explanatory variable: Sound stimulus (two types:
“bee sound” & “white noise”)
▶ Other possible explanatory variables: environmental factors
such as . . . .

Slide 13/55
Introduction Validity Precision Study Types In Summary

Terminology

Principles of Good Study Design

For validity (minimize possibility of bias):

▶ Randomization
▶ Comparison
▶ Control

For precision:
▶ Replication
▶ Blocking / stratification
▶ Balance

Slide 14/55
Introduction Validity Precision Study Types In Summary

Validity ≈ Unbiased

We say a study is biased if what we think we are measuring isn’t

what we are really measuring. To minimize the potential for bias
we use:
▶ comparison (placebo group, control group)
▶ control (possible confounding variables held constant)
▶ randomisation
▶ subjects representative of population under investigation
▶ groups to be compared are similar at baseline

Slide 15/55
Introduction Validity Precision Study Types In Summary

Comparison

Comparison

Some common comparisons:

Control group: is a set of experimental units which do not
receive the active treatment, but which are similar
in all other respects.
Placebo: a special type of control group, which is given a fake
treatment.
Current best: a group given the current standard treatment.
Natural groups: when there are groups in the data, such as
different ages, genders, family histories, etc.

Slide 16/55
Introduction Validity Precision Study Types In Summary

Comparison

Comparison

Placebo Effect
▶ A placebo is a treatment,
such as a sugar pill, that
(theoretically) has no active
therapeutic effect.
▶ A placebo effect is an effect
produced by a placebo
treatment, which cannot be
attributed to the properties
of the placebo itself, and
must therefore be due to the
patient’s belief.
Slide 17/55
Introduction Validity Precision Study Types In Summary

Comparison

Understanding Comparison

“Bees and Elephants” (Study Type 3)

– bee sounds played to 10 randomly selected herds and alternative
sound was played to another 10 randomly selected herds.
Comparison
▶ The use of two or more treatments so that a particular
treatment has something that it can be compared with.
▶ Comparison groups should be similar at baseline.

Slide 18/55
Introduction Validity Precision Study Types In Summary

Control

Control

Main ways to control a study are:

Restrictions: eg only considering people aged 20-25 will
control for age.
Protocols: being systematic, cleaning/calibrating equipment,
consistent dosage, same time of day, etc
Blinding: people are unaware of which treatment is given to
someone.

Slide 19/55
Introduction Validity Precision Study Types In Summary

Control

Control — using Blinding

Blinding in a study is used to protect against bias that arises from

experimenter bias (such as diagnosing bias) or the placebo effect.

Example: RCDB Salk Trial

Children, (of consenting parents) were randomly assigned to
receive either the vaccination or an injection of salt water.
▶ neither the child nor the diagnosing physicians were told
which treatment the child received.
The RCDB Salk trial study is an example of a double-blind study.
And the NFIP study?

Slide 20/55
Introduction Validity Precision Study Types In Summary

Control

Control — using Blinding

A study is said to be
▶ Blind if either the subject or the experimenter is unaware of
who gets which treatment
▶ Double-blind if both the subject and the experimenter are
unaware of who gets which treatment

Slide 21/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Confounding
Example: Subjects randomly assigned to either Treatment group
A or Treatment group B.

Treatment A Treatment B

Slide 22/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Confounding
Example: Subjects randomly assigned to either Treatment group
A or Treatment group B.

Treatment A Treatment B

Difference in mean response for two

groups can be attributed to ???

Slide 22/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

▶ We are actually estimating the difference between two

doctor/treatment combinations.
▶ Treatments are confounded with Doctors and Doctors
influence the outcome variable.
▶ So we can’t validly comment on the effectiveness of the
treatment.

Slide 23/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Visually. . .

Relationship ̸
=⇒ Causation
Relationship ⇐= Causation
Slide 24/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Confounding

▶ Occurs when the observed outcome can be due to more than

one explanatory variable and the effects of these explanatory
variables cannot be separated.
▶ A confounding variable is an explanatory variable that affects
the relation between the treatment/exposure and the
outcome.
▶ It is necessarily related to both the explanatory variable
(non-causal link) and the response variable (causal link).
A lurking variable is an unobserved (not measured) variable that
affects the relation between the exposure and the outcome.

Slide 25/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Dealing with possible confounding

Aim for similarity of the comparison groups at the start of the

study – similarity with respect to the (potential) confounding
variable. This can be achieved by:
▶ randomly assigning subjects to treatment groups
▶ randomising the order of treatments when one subject receives
both treatments (controls for practice effect)
▶ restriction of some form (eg. using just one doctor for both
treatments)
▶ blocking (eg. each doctor administers both treatments)

Slide 26/55
Introduction Validity Precision Study Types In Summary

Randomisation & Confounding

Randomisation and confounding

Randomisation
▶ Ensures that the groups are balanced, on average, on other
possible causes (known and unknown) of the response.
▶ Evens out the effect of uncontrolled potential confounding
variables
▶ We can check if the randomisation has been effective by
comparing measures (on known potential confounding
variables) for the different treatment groups prior to
introducing the intervention.

Slide 27/55
Introduction Validity Precision Study Types In Summary

Precision

We say a study is precise if we get close estimates. To maximize

the precision we use:
▶ blocking (mini-experiments)
▶ replication (multiple measurements for different conditions)
▶ balance (equal sized groups)

Slide 28/55
Introduction Validity Precision Study Types In Summary

Blocking

Blocking — dealing with confounding

Slide 29/55
Introduction Validity Precision Study Types In Summary

Blocking

Blocking

Blocking
▶ divide the study units into blocks - blocks similar within,
different between.
▶ randomly assign treatments to study units within a block.
▶ use a new randomisation for each block.

In observational studies we use stratification which is equivalent to

the blocking used in designed experiments.

Slide 30/55
Introduction Validity Precision Study Types In Summary

Blocking

Examples of blocks

A block is a collection of experimental units or subjects that before

the experiment are considered to similar in some way that is
expected to affect the response to treatment. Some examples:
▶ twins or litters of animals
▶ time of day
▶ batches of materials
▶ farm management practices – organic, non-organic
▶ gender, age group, socio-economic status, physical health,
cultural group etc.
▶ One subject receives both treatments (the order of the
treatments being randomised). Block = an individual.

Slide 31/55
Introduction Validity Precision Study Types In Summary

Blocking

Matched pairs Design

A special type of blocked design is a matched pairs study.

The most common example is twin studies where one of the twins
receives treatment A and the other receives treatment B. A block
consists of one pair of twins.
Another example is a Case-Control study (medical context).
In both instances, the pair are matched on all variables that could
affect the response.
Example: Upstream and Downstream readings (Week 1
Tute/lab)

Slide 32/55
Introduction Validity Precision Study Types In Summary

Replication

Replication
Precision: estimates are precise if the error is small

To increase precision:
▶ increase number of replicates in study. That is, we base
our estimates on more information. This usually amounts to
taking a larger sample size (more study units). We can then
estimate the variability between study units and hence
measure the precision in our estimates.
▶ replicate the measurements. That is, we take several
readings on one study unit and record the average for each
study unit. The analysis is then conducted on these averages.
This approach reduces ‘noise’ because we are average-ing out
variation within a study unit.
Slide 33/55
Introduction Validity Precision Study Types In Summary

Replication

Replication

NOTE:
▶ Replication in a study refers to the number of study units
. . . for example, number of replicates in each treatment group
▶ Repeatability/Reliability refers to being able to reproduce the
results in independent studies.

Slide 34/55
Introduction Validity Precision Study Types In Summary

Replication

Example: Fertilisers and Plant growth

Investigating the effect of 3 different fertilisers on seedling growth.

- 30 pots; each pot planted with 10 randomly selected seedlings.
- Fertilisers are then randomly assigned to the pots; 10 pots
receiving each fertiliser type.
▶ Study unit: a single pot
▶ Response variable: average seedling growth per pot.

Variation between seedlings within a pot is averaged out.

Greater precision in the measured response variable.

Slide 35/55
Introduction Validity Precision Study Types In Summary

Balance

Balance

Having equal numbers of replicates in each group.

▶ Is better (more precise) than unequal groups, if the total
sample size is the same.
▶ Usually the least important consideration.

Slide 36/55
Introduction Validity Precision Study Types In Summary

Study Types: example

An investigator wants to determine which feed type, A or B, is

best for increasing the weight gain of sheep on farms.

In an observational study, the data is collected by asking all the

farmers in a district what feed they used in recent times and what
was the average weight gain of their sheep over the last year.

In a designed experiment, the investigator randomly selects a

group of farms and then randomly assigns half of the farms to feed
type A and the other half to feed type B. After 12 months, the
average weight gain of the sheep is recorded for each farm and the
two feed types are compared.

Slide 37/55
Introduction Validity Precision Study Types In Summary

Observational or Experimental?

Observational studies
▶ the experimenter observes individuals and measures variables of
interest
▶ the experimenter does not determine which group a participant
(study unit) belongs to

Designed Experiment
▶ the experimenter deliberately imposes some treatment on the
study units in order to observe the response.
▶ the experimenter assigns the study units to the different treatment
groups.
▶ . . . “Evidence by design”

Slide 38/55
Introduction Validity Precision Study Types In Summary

Cause-and-effect relationships

Continuing with the previous example:

- The average weight gain for sheep on feed type A was found
to be 5kgm higher than the other feed type.
- Can we conclude that feed type A causes superior weight gain?

Slide 39/55
Introduction Validity Precision Study Types In Summary

Cause-and-effect relationships

Continuing with the previous example:

- The average weight gain for sheep on feed type A was found
to be 5kgm higher than the other feed type.
- Can we conclude that feed type A causes superior weight gain?
⃝
( ◦
◦ ) consider the design of the study!

Slide 39/55
Introduction Validity Precision Study Types In Summary

In our observational study:

▶ the researchers noticed (in hindsight) that farms using feed
type A have feeding pens that make it easier for animals to
access the food. The other farms had traditional feeding pens.

Is the increased weight gain due to (caused by) the feed type or
the type feeding pen?

Slide 40/55
Introduction Validity Precision Study Types In Summary

In our observational study:

▶ the researchers noticed (in hindsight) that farms using feed
type A have feeding pens that make it easier for animals to
access the food. The other farms had traditional feeding pens.

Is the increased weight gain due to (caused by) the feed type or
the type feeding pen?

There is no way to tell!

Slide 40/55
Introduction Validity Precision Study Types In Summary

In the designed experiment:

▶ farms were randomly allocated to a particular feed type
▶ for each type of feeding pen, approximately half of the farms
would receive feed Type A and the remainder would receive
Feed Type B.

Slide 41/55
Introduction Validity Precision Study Types In Summary

In the designed experiment:

▶ farms were randomly allocated to a particular feed type
▶ for each type of feeding pen, approximately half of the farms
would receive feed Type A and the remainder would receive
Feed Type B.

We can more confidently conclude that the feed type caused the
increased weight gain (the effect)

Slide 41/55
Introduction Validity Precision Study Types In Summary

Moral of the tale:

Observational studies can often show an association between two

variables, but never directly prove causation.

When our goal is to understand cause-and-effect relationships then

well-designed experiments are the optimal way of obtaining fully
convincing data.

Slide 42/55
Introduction Validity Precision Study Types In Summary

Image credit: [Link]

Slide 43/55
Introduction Validity Precision Study Types In Summary

Observational studies

Observational Studies . . . in practice

▶ Estimating frog population

▶ Estimating species diversity . . . in rock pools
▶ Evaluating the “form” of elite athletes
▶ Questionnaires / sample surveys
Evidence by “observation”

Slide 44/55
Introduction Validity Precision Study Types In Summary

Observational studies

Common problems

▶ Selection bias: sample selection process makes some

sub-groups of the population more likely to be selected (e.g.
treatment group in NFIP study was self-selecting)
▶ Response / reporting bias: different sub-groups being more or
less likely to respond (opinion polls); diagnosing bias (Salk
trial-NFIP)
▶ Question wording
▶ Presence of confounders
▶ conclusions about causation are more difficult to make
▶ the power of randomly assigning treatments is absent –
although we do often randomly select subjects.
▶ there may be other factors causing the observed association,
other than the exposure (treatment) being considered
Slide 45/55
Introduction Validity Precision Study Types In Summary

Designed experiments

Designed Experiments

Of course, the Study Design Principles apply equally to

observational studies and designed experiments (see slide 14).

However, with designed experiments we have a distinct advantage:

▶ The experimenter has control over the randomisation process
- randomly assign subjects to treatments (minimising possible
bias and confounding)
- randomise in such way that possible effects due to nuisance
variables are controlled for . . . and clearly measurable (blocked
designs)

Slide 46/55
Introduction Validity Precision Study Types In Summary

Designed experiments

Randomisation in designed experiments

In an experimental study subjects are randomly assigned to the

treatments.

A design that uses randomisation without any form of matching

(see next example) is referred to as a completely randomised
design.

Slide 47/55
Introduction Validity Precision Study Types In Summary

Designed experiments

. . . and how to achieve randomisation

▶ Use a mechanical device such as tossing a coin (won’t

necessarily lead to a balanced design)
▶ Use calculator/software (can be balanced/not depending on
how you do this)

Slide 48/55
Introduction Validity Precision Study Types In Summary

An Example

Example: Growth of native grasses

▶ Objective is to compare the growth of three different

Australian native grasses (Blue, Kangaroo and Weeping)
▶ Two fields are available for the study
▶ Each field consists of 9 plots (a total of 18 plots)

Slide 49/55
Introduction Validity Precision Study Types In Summary

An Example

▶ Research question

▶ Response variable

▶ Explanatory (treatment) Variable (of interest)

▶ Other possible explanatory variables

▶ Experimental unit

▶ Number of replicates of each treatment

Slide 50/55
Introduction Validity Precision Study Types In Summary

An Example

Completely Randomised Design: CRD

Comments?
What if Field 1 has better growing conditions than Field 2. . .

Slide 51/55
Introduction Validity Precision Study Types In Summary

An Example

( ⃝◦◦ )the CRD model

Data = signal+ noise (from Chapter1)

In this example:

Growth = Native plant type + noise

. . . and part of the noise is due to the difference between the

growing conditions in the two fields.

larger noise ⇒ less precision in estimates

To improve precision we explain part of the noise as due to

differences (in growing conditions) of the fields.

Slide 52/55
Introduction Validity Precision Study Types In Summary

An Example

( ⃝◦◦ )Randomised Block Design (RBD) model

Variation in growth due to the different fields is now removed from

the noise component of the model and becomes part of the signal.
So the model becomes;

Growth = Native plant type + Field type + noise

⇒ Improved precision . . . provided there really is a field effect!

We also have the added advantage that we can measure the
difference between the blocks (the Field effect)

Slide 53/55
Introduction Validity Precision Study Types In Summary

An Example

RBD–allocation

Slide 54/55
Introduction Validity Precision Study Types In Summary

Slide 55/55
Introduction Validity Precision Study Types In Summary

Learning Outcomes:

At the end of this topic you should:

▶ Recognise observational studies and designed experiments,
and distinguish them
▶ Be able to explain and apply the principles of good study
design:
▶ Randomisation, comparison and control
▶ Replication, blocking/stratification and balance
▶ Explain confounding in the context of study design
▶ Understand when causal inferences may be made from a single
study

Slide 4/55