The Power of a Sensitivity Analysis and Its Limit

Abstract

In an experiment, power and sample size calculations anticipate the outcome of a statistical test that will be performed when the experimental data are available for analysis. In parallel, in an observational study, the power of a sensitivity analysis anticipates the outcome of a sensitivity analysis that will be performed when the observational data are available for analysis. In both cases, it is imagined that the data will be generated by a particular model or distribution, and the outcome of the test or sensitivity analysis is anticipated for data from that model. Calculations of this sort guide many of the decisions made in designing a randomized clinical trial, and similar calculations may usefully guide the design of an observational study. In experiments, the power in large samples is used to judge the relative efficiency of competing statistical procedures. In parallel, the power in large samples of a sensitivity analysis is used to judge the ability of design features, such as those in Chap. 5, to distinguish treatment effects from bias due to unmeasured covariates. As the sample size increases, the limit of the power of a sensitivity analysis is a step function with a single step down from power 1 to power 0, where the step occurs at a value $\widetilde {\varGamma }$ of Γ called the design sensitivity. The design sensitivity is a basic tool for comparing alternative designs for an observational study.

Appendix: Technical Remarks and Proof of Proposition 15.1

This appendix contains a proof of Proposition 15.1 in Sect. 15.3. The formula and proof [3, 15] for Wilcoxon’s signed rank statistic are a special case of a general result [7, §3] with the attractive feature that the design sensitivity $\widetilde {\varGamma }$ has an explicit closed form (15.14).

The approximate power is , where μ _F is the expectation of T in the “favorable situation.” An intuitive, heuristic derivation [7, 15] uses the notion that if ζ _Γ,α > μ _F, then in large samples, as I →∞, the signed rank statistic T eventually will be less than ζ _Γ,α, whereas if ζ _Γ,α < μ _F, then T will eventually be greater than ζ _Γ,α. So the heuristic equates ζ _Γ,α in (15.10) and μ _F in (15.6) and solves for Γ. It is slightly better to equate ζ _Γ,α∕μ _F to 1, let I →∞, ignore terms that are small compared with I ², and solve for Γ,

(15.17)

(15.18)

where $1=\varGamma /\left \{ p_{1}^{{ }^{\prime }}\left ( 1+\varGamma \right ) \right \}$ has solution $\widetilde {\varGamma }=p_{1}^{{ }^{\prime }}/\left ( 1-p_{1}^{{ }^{\prime }}\right )$ as in (15.14). The heuristic is attractive because it uses only the expectation μ _F of T in the favorable situation, so the calculation is simple. Alas, the heuristic is “heuristic” precisely because the power depends on $\left ( \zeta _{\varGamma ,\alpha }-\mu _{F}\right ) /\sigma _{F}$, not on ζ _Γ,α − μ _F, and the heuristic ignores σ _F, hoping that things will work out. Indeed, things do work out, as the proof that follows demonstrates [3]. The general result in [7, §3] shows that the heuristic calculation works quite generally, but the proof that follows is self-contained and does not depend upon the general result.

Proof

For large I, the power, $\Pr \left ( \left . T\geq \zeta _{\varGamma ,\alpha }\right \vert \,\mathcal {Z}\right )$, is approximately equal to $1-\varPhi \left \{ \left ( \zeta _{\varGamma ,\alpha }-\mu _{F}\right ) /\sigma _{F}\right \}$, so the power tends to 1 as I →∞ if $\left ( \zeta _{\varGamma ,\alpha }-\mu _{F}\right ) /\sigma _{F}\rightarrow -\infty$ and the power tends to 0 as I →∞ if $\left ( \zeta _{\varGamma ,\alpha }-\mu _{F}\right ) /\sigma _{F}\rightarrow \infty$. Write $\kappa =\varGamma /\left ( 1+\varGamma \right )$. Using the expressions for μ _F and $\sigma _{F}^{2}$ from (15.6) and (15.7) in Sect. 15.1 and the expression for ζ _Γ,α from (15.10) in Sect. 15.2 yields

$$\displaystyle \begin{aligned} \frac{\zeta _{\varGamma ,\alpha }-\mu _{F}}{\sigma _{F}}=\end{aligned} $$

(15.19)

$$\displaystyle \begin{aligned} \frac{\kappa I\left( I+1\right) /2+\varPhi ^{-1}\left( 1-\alpha \right) \sqrt{ \kappa \left( 1-\kappa \right) I\left( I+1\right) \left( 2I+1\right) /6} -I\left( I-1\right) p_{1}^{{}^{\prime }}/2-Ip}{\sqrt{I\left( I-1\right) \left( I-2\right) \left( p_{2}^{{}^{\prime }}-\left. p_{1}^{{}^{\prime }}\right. ^{2}\right) +\frac{I\left( I-1\right) }{2}\left\{ 2\left( p-p_{1}^{{}^{\prime }}\right) ^{2}+3p_{1}^{{}^{\prime }}\left( 1-p_{1}^{{}^{\prime }}\right) \right\} +Ip\left( 1-p\right) }}. {} \end{aligned} $$

(15.20)

Expression (15.20) looks untidy at first. Notice, however, that every term in the numerator and denominator of (15.20) involves I, and I →∞. So divide the numerator and denominator of (15.20) by $I \sqrt {I}$, and let I →∞. The last term in the numerator, − Ip, becomes $-Ip/\left ( I\sqrt {I}\right ) \rightarrow 0$ as I →∞, whereas the first term in the numerator, $\kappa I\left ( I+1\right ) /2$, becomes $\kappa I\left ( I+1\right ) /\left ( 2I\sqrt {I}\right ) \approx \sqrt {I}\kappa /2$ in the sense that their ratio is tending to 1 as I →∞. Continuing in this way, we find that

$$\displaystyle \begin{aligned} \frac{\zeta _{\varGamma ,\alpha }-\mu _{F}}{\sigma _{F}}\approx \frac{\sqrt{I} \left( \kappa -p_{1}^{{}^{\prime }}\right) }{2\sqrt{p_{2}^{{}^{\prime }}-\left. p_{1}^{{}^{\prime }}\right. ^{2}}}\;\;\mathrm{as}\;\;I\rightarrow \infty \text{ ,} {} \end{aligned} $$

(15.21)

in the sense that the ratio of the left and right sides of (15.21) tends to 1 as I →∞ provided $\kappa \neq p_{1}^{{ }^{\prime }}$. If $\kappa >p_{1}^{{ }^{\prime }}$, then (15.21) tends to ∞ as I →∞, so the power tends to 0. If $\kappa <p_{1}^{{ }^{\prime }}$, then (15.21) tends to −∞ as I →∞, so the power tends to one. Moreover, $\kappa =\varGamma /\left ( 1+\varGamma \right ) >p_{1}^{{ }^{\prime }}$ if and only if $\varGamma >p_{1}^{{ }^{\prime }}/\left ( 1-p_{1}^{{ }^{\prime }}\right )$.