The problem of the type II statistical error.

Abstract

OBJECTIVE: To determine if type II statistical errors (also known as beta errors) are a common problem in published clinical research. METHODS: Type II statistical errors occur when sample sizes are too small to show an effect of treatment, even when an effect truly exists. Searching the Medline data base, we identified ten meta-analyses published during 1986-1994 in the American Journal of Obstetrics and Gynecology, Obstetrics and Gynecology, and The Journal of Reproductive Medicine. Meta-analyses were used as sources of component or individual studies for the following reason: When small component studies have negative findings that differ from the overall conclusions of the meta-analysis, the component studies may have type II statistical errors. RESULTS: We found that only 6.5% (15 of 231) of component studies provided any documentation that power calculations to determine sample sizes had been done a priori (before) the research began. Thus, many of these component studies with findings of no treatment effect may have had type II errors because of too-small sample sizes. When stratifying the component studies by year of publication, we found that 7.9% (14 of 178) of studies published in the 1980s and 1990s had any documented evidence of a priori power calculations. In the 1960s and 1970s, only one of 53 component studies had documented evidence of power calculations. CONCLUSION: To ensure that truly effective treatments are introduced into clinical practice as quickly as possible, we believe that a priori power calculations should always be done in quantitative clinical research.