Amos Tversky and Daniel Kahneman
Belief in the Law of Small Numbers
Amos Tversky and Daniel Kahneman. 1971. (View Paper → )
People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psychological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.
People often assume that small samples should reflect the population's characteristics as closely as large samples do, but this is not the case due to natural variability. This misconception is termed as the "law of small numbers."
Researchers often exhibit undue confidence in results derived from small samples, mistakenly believing that these results are highly replicable and reliable.
This bias affects the design of studies and the interpretation of statistical results. Researchers tend to overestimate the likelihood of replicating significant findings and underestimate the required sample size for reliable results.
The paper emphasises the importance of calculating statistical power and confidence intervals properly. It warns against the common research practice of relying on significance tests without considering the effect size or the likelihood of replicating results with smaller samples.