The probability of false positives in zero-dimensional analyses of one-dimensional kinematic, force and EMG trajectories

Abstract

A false positive is the mistake of inferring an effect when none exists, and although α controls the false positive (Type I error) rate in classical hypothesis testing, a given α value is accurate only if the underlying model of randomness appropriately reflects experimentally observed variance. Hypotheses pertaining to one-dimensional (1D) (e.g. time-varying) biomechanical trajectories are most often tested using a traditional zero-dimensional (0D) Gaussian model of randomness, but variance in these datasets is clearly 1D. The purpose of this study was to determine the likelihood that analyzing smooth 1D data with a 0D model of variance will produce false positives. We first used random field theory (RFT) to predict the probability of false positives in 0D analyses. We then validated RFT predictions via numerical simulations of smooth Gaussian 1D trajectories. Results showed that, across a range of public kinematic, force/moment and EMG datasets, the median false positive rate was 0.382 and not the assumed α=0.05, even for a simple two-sample t test involving N=10 trajectories per group. The median false positive rate for experiments involving three-component vector trajectories was p=0.764. This rate increased to p=0.945 for two three-component vector trajectories, and to p=0.999 for six three-component vectors. This implies that experiments involving vector trajectories have a high probability of yielding 0D statistical significance when there is, in fact, no 1D effect. Either (a) explicit a priori identification of 0D variables or (b) adoption of 1D methods can more tightly control α.

Introduction

In classical hypothesis testing p values represent the probability that a random process would produce an effect larger than the observed one. There are unfortunately many ways in which p value computations can go astray to yield “false positives” (Knudson, 2005, Kundson and Lindsey, 2014): the mistake of inferring an experimental effect when none exists in reality. This paper deals with one specific pitfall in p value computations which has not previously been quantified and is relevant to many branches of Biomechanics: zero-dimensional (0D) p values for one-dimensional (1D) (e.g. time varying) data.

Imagine a simple experiment which yields five scalar 1D force trajectories for each of two groups (Fig. 1b). Classical hypothesis testing can be conducted using either a “0D” or a “1D” approach (Pataky et al., 2015).

Zero-dimensional analysis: One could analyze the data using a 0D summary metric like the local maximum (Fig. 1a). In this case a one-tailed two-sample t test of the maxima yields t=2.357, p=0.023 and one would reject the null hypothesis at α=0.05. More completely, the 0D residuals (Fig. 1c) represent the variance about the group means, and one judges the effect size x (Fig. 1a) against this variance. If the null hypothesis (x=0) were true, random 0D data with the same variance would produce a distribution of t values over an infinite number of identical experiments (Fig. 1e) and only 2.3% of those values would be greater than the observed t=2.357. The null hypothesis is rejected because the observed t value exceeds the threshold $t_{0\mathrm{D}}^{⁎}$ corresponding to α (Fig. 1e and g). This $t_{0\mathrm{D}}^{⁎}$ value can be rapidly computed in statistical software packages, or it can be computed by iteratively simulating thousands of t tests on randomly generated samples of 0D Gaussian data.

One-dimensional analysis: One could alternatively analyze the data using 1D methods (Lenhoff et al., 1999, Pataky et al., 2015) (Fig. 1, right panels). Analogous to the 0D procedure, the 1D residuals (Fig. 1d) embody the variance about mean trajectories, and the null hypothesis is the null difference trajectory: $x\left(q\right)=0$, where q is time or the 1D measurement domain. If the null hypothesis were true, random 1D data with the same variance and same smoothness would produce a distribution of t trajectory maxima over an infinite number of identical experiments (Fig. 1f), and in this case random 1D data would produce the observed 0D effect of t=2.357 with a probability of approximately 55%, which is well above α. In other words, random 1D data produce particular t values with generally much greater probability than do random 0D data. The null hypothesis is not rejected because the observed maximum t value, across the whole trajectory, does not exceed the threshold $t_{1\mathrm{D}}^{⁎}$ corresponding to α (Fig. 1f and h). The $t_{1\mathrm{D}}^{⁎}$ value can be analytically calculated using random field theory (RFT) (Adler and Taylor, 2007, Friston et al., 2007), or it can be computed by iteratively simulating thousands of t tests on randomly generated samples of smooth 1D Gaussian data (Fig. 2) (Pataky, 2016).

The 0D and 1D procedures have yielded opposite hypothesis testing results, so which is correct? The answer depends on the a priori hypothesis. If, prior to conducting an experiment, one explicitly identified that particular 0D metric as the sole metric of empirical interest then the empirical question is inherently 0D and the 0D result is correct. If, however, one measured 1D data and did not specify that particular 0D metric prior to the experiment then the empirical question is inherently 1D and the 1D result is correct. Failing to specify a 0D metric prior to a 1D experiment and then adopting 0D methods has been termed “regional focus bias” (Pataky et al., 2013) and is a potential source of false positives. False positive prevalence for 1D biomechanics datasets has previously been estimated for 0D procedures (Knudson, 2005) but not, to our knowledge, in the context of 0D vs. 1D procedures.

The purpose of this study was to quantify the false positive rates that could be expected in real 1D biomechanical datasets when employing 0D statistical inference. To that end we analyzed nine public datasets (Table 2) representing a variety of experimental tasks (walking, running, cutting, and cycling) and data modalities (forces, kinematics, and EMG). Based on the data׳s temporal smoothness we estimated the likelihood of producing false positives using a simplified experimental design: a two-sample t test with N=10 for each group. Analyses for more complex designs like ANOVA are addressed in the Discussion (Section 4.3). The key theoretical concept we shall attempt to convey is that two parameters – the mean (μ) and standard deviation (σ) – describe 0D Gaussian behavior, and that only one additional parameter – 1D smoothness (FWHM) (Fig. 2) (Appendices A and B) – is needed to describe 1D Gaussian behavior.

To clarify our “0D” and “1D” terminology we shall also employ the term: “nDmD”, where n and m are the dimensionalities of the measurement domain and dependent variable, respectively (Table 1). In nDmD datasets the physical nature of the variables changes across the m components but not across the nD measurement domain. Biomechanics studies often measure 1DmD data but use 0D1D models of randomness to define critical statistical thresholds, and this paper quantifies false positive rates associated with that approach. Throughout this paper “0D” and “1D” represent to “0D1D” and “1DmD”, respectively.

We emphasize that this paper focusses on just a single statistical issue: the probability of false positives in a single 0D1D two-sample t test conducted on 1DmD data. We use only the two-sample t test because this simple test sufficiently demonstrates the magnitude of the false positives problem and because the problem is exacerbated in more complex designs like ANOVA. We acknowledge that many other issues must be considered when conducting statistical analyses including: small sample sizes, non-sphericity, normality, outliers, etc. Just as it is useful to consider each of these issues individually, we feel it is equally useful to consider 0D vs. 1D analysis individually because this issue is relevant to all 1D data analyses but has not been explicitly addressed in the literature.