Publications 2008
 

William R. Shadish, M.H. Clark, Peter M. Steiner

Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment

Articel in the magazine: Journal of the American Statistical Association

A key justification for the use of nonrandomized experiments is that, with proper adjustment, their results can well-approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies. (With comments by Little/Long/Lin, Hill, and Rubin, and a rejoinder.)

However, past methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized or a nonrandomized experiment. In the randomized experiment, participants were randomly assigned to mathematics or vocabulary training; in the nonrandomized experiment, they chose their training. The study held all other features of the experiment constant; it carefully measured pretest variables that might predict the condition that participants chose; and all participants were measured on vocabulary and mathematics outcome. Ordinary linear regression reduced bias in the nonrandomized experiment 84-94% using covariate-adjusted randomized results as the benchmark. Propensity score stratification, weighting and covariance adjustment reduced bias by about 58-96%, depending on the outcome measure and adjustment method. Propensity score adjustment performed poorly when the scores were constructed from predictors of convenience (sex, age, marital status and ethnicity) rather than from a broader set of predictors that might include these.

Suggested quote:
Shadish, William R.; Clark, M.H.; Steiner, Peter M. (2008), Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment, in: Journal of the American Statistical Association, 103, pp. 1334-1343.
top