Making effects manifest in randomized experiments

The Annals of Statistics, 2012

Randomized experiments are the "gold standard" for estimating causal effects, yet often in practice, chance imbalances exist in covariate distributions between treatment groups. If covariate data are available before units are exposed to treatments, these chance imbalances can be mitigated by first checking covariate balance before the physical experiment takes place. Provided a precise definition of imbalance has been specified in advance, unbalanced randomizations can be discarded, followed by a rerandomization, and this process can continue until a randomization yielding balance according to the definition is achieved. By improving covariate balance, rerandomization provides more precise and trustworthy estimates of treatment effects.

Pain, 1998

Variability in patients' response to interventions in pain and other clinical settings is large. Many explanations such as trial methods, environment or culture have been proposed, but this paper sets out to show that the main cause of the variability may be random chance, and that if trials are small their estimate of magnitude of effect may be incorrect, simply because of the random play of chance. This is highly relevant to the questions of 'How large do trials have to be for statistical accuracy?' and 'How large do trials have to be for their results to be clinically valid?' The true underlying control event rate (CER) and experimental event rate (EER) were determined from single-dose acute pain analgesic trials in over 5000 patients. Trial group size required to obtain statistically significant and clinically relevant (0.95 probability of number-needed-to-treat within ±0.5 of its true value) results were computed using these values. Ten thousand trials using these CER and EER values were simulated using varying group sizes to investigate the variation due to random chance alone. Most common analgesics have EERs in the range 0.4-0.6 and CER of about 0.19. With such efficacy, to have a 90% chance of obtaining a statistically significant result in the correct direction requires group sizes in the range 30-60. For clinical relevance nearly 500 patients are required in each group. Only with an extremely effective drug (EER Ͼ 0.8) will we be reasonably sure of obtaining a clinically relevant NNT with commonly used group sizes of around 40 patients per treatment arm. The simulated trials showed substantial variation in CER and EER, with the probability of obtaining the correct values improving as group size increased. We contend that much of the variability in control and experimental event rates is due to random chance alone. Single small trials are unlikely to be correct. If we want to be sure of getting correct (clinically relevant) results in clinical trials we must study more patients. Credible estimates of clinical efficacy are only likely to come from large trials or from pooling multiple trials of conventional (small) size.

Biometrics, 2014

We address estimation of intervention effects in experimental designs in which (a) interventions are assigned at the cluster level; (b) clusters are selected to form pairs, matched on observed characteristics; and (c) intervention is assigned to one cluster at random within each pair. One goal of policy interest is to estimate the average outcome if all clusters in all pairs are assigned control versus if all clusters in all pairs are assigned to intervention. In such designs, inference that ignores individual level covariates can be imprecise because cluster-level assignment can leave substantial imbalance in the covariate distribution between experimental arms within each pair. However, most existing methods that adjust for covariates have estimands that are not of policy interest. We propose a methodology that explicitly balances the observed covariates among clusters in a pair, and retains the original estimand of interest. We demonstrate our approach through the evaluation of t...

Journal of Causal Inference, 2016

We consider the conditional randomization test as a way to account for covariate imbalance in randomized experiments. The test accounts for covariate imbalance by comparing the observed test statistic to the null distribution of the test statistic conditional on the observed covariate imbalance. We prove that the conditional randomization test has the correct significance level and introduce original notation to describe covariate balance more formally. Through simulation, we verify that conditional randomization tests behave like more traditional forms of covariate adjustment but have the added benefit of having the correct conditional significance level. Finally, we apply the approach to a randomized product marketing experiment where covariate information was collected after randomization.

Acta Analytica, 2021

Despite the consensus promoted by the Evidence Based Medicine framework, many authors con-tinue to express doubts about the superiority of Randomized Controlled Trials. This paper evalu-ates four objections targeting the legitimacy, feasibility, and extrapolation problems linked to the experimental practice of random allocation. I argue that random allocation is a methodologically sound and feasible practice contributing to the internal validity of controlled experiments dealing with heterogeneous populations. I emphasize, however, that random allocation is solely designed to ensure the validity of causal inferences at the level of groups. By itself, random allocation can-not enhance test precision, doesn’t contribute to external validity, and limits the applicability of causal claims to individuals.

2008

This paper considers statistical tests that can be used to identify if a treatment is effective for a specific outcome variable over the entire distribution of a treated group when it is compared with a control group's distribution. Using only the average treatment effect to evaluate specific treatment programs ignores what happens in different regions of the distribution of interest. To address this problem, tests of equality of distributions, first and second order stochastic dominance are employed. To show how to implement the tests easily, an outline on how to estimate critical values using bootstrap method is presented. The tests are then applied to analyze the effectiveness of a treatment in a randomized experiment. I gratefully acknowledge comments and suggestions from Juan Carlos Escanciano, Kim Huynh and Ricardas Zitikis

The British Journal of Philosophy of Science, 2024

Experimental balance is usually understood as the control for the value of the conditions, other than the one under study, which are liable to affect the result of a test. We will discuss three different approaches to balance. 'Millean balance' requires to identify and equalize ex ante the value of these conditions in order to conduct solid causal inferences. 'Fisherian balance' measures ex post the influence of uncontrolled conditions through the analysis of variance. In 'efficiency balance' the value of the antecedent conditions is decided ex ante according to the efficiency they yield in the estimation of the treatment outcome. Against some old arguments by John Worrall, we will show that in both Fisherian and efficiency balance there are good reasons to randomize the allocation of treatments, in particular when there is no agreement among experimenters as to the antecedent conditions to be controlled for.

The Lancet, 2002

We cringe at the pervasive notion that a randomlsed trial needs to yield equal sample sizes In the comparison _ _ Unfortunately, that conceptual """understanding can lead to bias by investigators who force equality, especially if by n0nscientific means. In simple, unrestricted, randomlsed trials (analogous to repeated coif>.tosslng), the sizes of groups shou1d Indicate random variation. In other words, some discrepancy between the nurrtJers In the cort1)arison groups would be expected. The aPllOal of equal group sizes In a simple randorrised controlled trial Is cosmetic, not scientific. Moreover. other randomisation schemes, tenned restricted randomisation, force equality by departing from simple randomisation. Forcing equal group sizes, however, potentially harms the unpredictability of treatment assignments, especially when using pennuted-block randomisation In nol><louble-bllnded trials. Diminished unpredictability can allow bias to creep into a trial. Overall, Investigators underuse simple randomisation and overuse flxed-block randomisation. For non-double-bltnded trials larger than 200 participants, investigators should use simple randomisation more olten and accept modetate disparities in group sizes. SUCh unpredictability reflects the essence of randomness. We endorse the generation of mDdIy unequal group sizes and encourage an a"", .. letlon of such Inequalities. For nol><louble-bllnded randorrised controlled trials with a sample size of less than 200 overall or within any principal stratum or subgroup, urn randomisation enhances unpredictability compared with blocking. A simpler alternative, our mixed randomisation aJlP'08Ch, attaIns W1P'ed\Ctability within the context of the currently understood simple randomisation and pennuted-block methods. Simple randomisation contributes the unpredictability whereas pennuted-block rand_on contributes the balance, but avoids the peffect balance that can result in selection bias.

Psychological methods, 2017

Blinded randomized controlled trials (RCT) require participants to be uncertain if they are receiving a treatment or placebo. Although uncertainty is ideal for isolating the treatment effect from all other potential effects, it is poorly suited for estimating the treatment effect under actual conditions of intended use-when individuals are certain that they are receiving a treatment. We propose an experimental design, randomization to randomization probabilities (R2R), which significantly improves estimates of treatment effects under actual conditions of use by manipulating participant expectations about receiving treatment. In the R2R design, participants are first randomized to a value, π, denoting their probability of receiving treatment (vs. placebo). Subjects are then told their value of π and randomized to either treatment or placebo with probabilities π and 1-π, respectively. Analysis of the treatment effect includes statistical controls for π (necessary for causal inference)...

Evaluation Review, 2020

Background: Random design experiments are a powerful device for estimating average treatment effects, but evaluators sometimes seek to estimate the distribution of treatment effects. For example, an evaluator might seek to learn the proportion of treated units who benefit from treatment, the proportion who receive no benefit, and the proportion who are harmed by treatment. Method: Imbens and Rubin (I&R) recommend a Bayesian approach to drawing inferences about the distribution of treatment effects. Drawing on the I&R recommendations, this article explains the approach; provides computing algorithms for continuous, binary, ordered and countable outcomes; and offers simulated and real-world illustrations. Results: This article shows how the I&R approach leads to bounded uncertainty intervals for summary measures of the distribution of treatment effects. It clarifies the nature of those bounds and shows that they are typically informative. Conclusions: Despite identification issues,