Indexes and boundaries for “quantitative significance” in statistical decisions

Abstract

Boundaries for S, representing a “quantitatively significant” or “substantively impressive” distinction, have not been established, analogous to the boundary of α, usually set at 0.05, for the stochastic or probabilistic component of “statistical significance”. To determine what boundaries are being used for the “quantitative” decisions, we reviewed pertinent articles in three general medical journals. For each contrast of two means, contrast of two rates, or correlation coefficient, we noted the investigators' decisions about stochastic significance, stated in P values or confidence intervals, and about quantitative significance, indicated by interpretive comments.

The boundaries between impressive and unimpressive distinctions were best formed by a ratio of ⩾ 1.2 for the smaller to the larger mean in 546 comparisons, by a standardized increment of ⩾ 0.28 and odds ratio of ⩾ 2.2 in 392 comparisons of two rates; and by an r value of ⩾0.32 in 154 correlation coefficients. Additional boundaries were also identified for “substantially” and “highly” significant quantitative distinctions.

Although the proposed boundaries should be kept flexible, indexes and boundaries for decisions about “quantitative significance” are particularly useful when a value of δ must be chosen for calculating sample size before the research is done, and when the “statistical significance” of completed research is appraised for its quantitative as well as stochastic components.