Skip to main content
SpringerLink
Log in

Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

When the goal of prevention research is to capture in statistical models some measure of the dynamic complexity in structures and processes implicated in problem behavior and its prevention, approaches such as multilevel modeling (MLM) and structural equation modeling (SEM) are indicated. Yet the assumptions that must be satisfied if these approaches are to be used responsibly raise concerns regarding their use in prevention research involving smaller samples. In this article, we discuss in nontechnical terms the role of sample size in MLM and SEM and present findings from the latest simulation work on the performance of each approach at sample sizes typical of prevention research. For each statistical approach, we draw from extant simulation studies to establish lower bounds for sample size (e.g., MLM can be applied with as few as ten groups comprising ten members with normally distributed data, restricted maximum likelihood estimation, and a focus on fixed effects; sample sizes as small as N = 50 can produce reliable SEM results with normally distributed data and at least three reliable indicators per factor) and suggest strategies for making the best use of the modeling approach when N is near the lower bound.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Nested data may also emerge from analyses aimed at accounting for unobserved heterogeneity in outcomes such as in latent class analysis or growth mixture models with longitudinal data. Relatively little is known about sample size requirements for analyses of these types but they almost certainly require samples that are large. For that reason, those models fall outside the scope of this manuscript.

  2. A number of software packages can handle such analyses, including (but certainly not limited to): HLM, Mplus, R, SAS, and Stata.

  3. We omit a discussion of least squares approaches because these tend to be less efficient than maximum likelihood (Singer and Willett 2003).

  4. We recognized that 30 groups is unrealistic for many prevention studies. This simulation study did not consider fewer than 30 groups given that the results were already somewhat problematic for that number. Findings from work considering 10 level 2 units is presented below.

  5. However, progress is being made on this front. See Noh and Lee (2007) for an example.

  6. Bauer and Sterba (2011) also found that increasing the number of response categories resulted in less bias and greater efficiency. This result is not surprising given the well-known consequences of dichotomization (MacCallum et al. 2002). Whenever possible, researchers should maximize the number of response categories or use a continuous response scale to maximize power.

  7. Software packages that can estimate SEMs include, but are not limited to: EQS, LISREL, Mplus, R, and STATA.

References

  • Aitkin, M. A., & Longford, N. T. (1986). Statistical modeling issues in school effectiveness studies. Journal of the Royal Statistical Society, 149, 1–26. Retrieved from http://www.jstor.org/stable/2981882.

    Article  Google Scholar 

  • Anderson, T. W. (1957). Maximum likelihood estimates for a multivariate normal distribution when some observations are missing. Journal of the American Statistical Association, 52, 200–203. Retrieved from http://www.jstor.org/stable/2280845.

    Article  Google Scholar 

  • Austin, P. C. (2010). Estimating multilevel logistic regression models when the number of clusters is low: A comparison of different statistical software procedures. The International Journal of Biostatistics, 6, 1–18. doi:10.2202/1557-4679.1195.

    Google Scholar 

  • Bandalos, D. L., & Gagné, P. (2012). Simulation methods in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 92–108). New York: Guildford Press.

    Google Scholar 

  • Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal of Educational and Behavioral Statistics, 28, 135–167. doi:10.3102/10769986028002135.

    Article  Google Scholar 

  • Bauer, D. J., & Sterba, S. K. (2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16, 373–390. doi:10.1037/a0025813.

    Article  PubMed Central  PubMed  Google Scholar 

  • Bearden, W. O., Sharma, S., & Teel, J. E. (1982). Sample size effects on chi square and other statistics used in evaluating causal models. Journal of Marketing Research, 19, 425–430. doi:10.2307/3151716.

    Article  Google Scholar 

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. doi:10.1037/0033-2909.107.2.238.

    Article  CAS  PubMed  Google Scholar 

  • Bollen, K. A. (1990). Overall fit in covariance structure models: Two types of sample size effects. Psychological Bulletin, 107, 256–259. doi:10.1037/0033-2909.107.2.256.

    Article  Google Scholar 

  • Bollen, K. A., & Curran, P. J. (2006). Latent curve models: A structural equation approach. Hoboken: Wiley.

    Google Scholar 

  • Bollen, K. A., & Noble, M. D. (2011). Structural equation models and the quantification of behavior. Proceedings of the National Academy of Sciences of the United States of America, 108, 15639–15646. doi:10.1073/pnas.1010661108.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Breslow, N. E., & Clayton, D. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25. Retrieved from http://www.jstor.org/stable/2290687.

    Google Scholar 

  • Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38, 529–569. doi:10.1207/s15327906mbr3804_5.

    Article  Google Scholar 

  • Curran, P. J., Lee, T., Howard, A. L., Lane, S., & MacCallum, R. (2012). Disaggregating within-person and between-person effects in multilevel and structural equation growth models. In J. R. Harring & G. R. Hancock (Eds.), Advances in longitudinal models in the social and behavioral sciences (pp. 217–253). Charlotte: Information Age Publishing.

    Google Scholar 

  • Demidenko, E. (2004). Mixed models: Theory and applications. Hoboken: Wiley.

    Book  Google Scholar 

  • Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 44, 1–38. Retrieved from http://www.jstor.org/stable/2984875?origin=JSTOR-pdf.

    Google Scholar 

  • Duncan, S. C., Duncan, T. E., & Strycker, L. A. (2002). A multilevel analysis of neighborhood context and youth alcohol and drug problems. Prevention Science, 3, 125–133. doi:10.1023/A:1015483317310.

    Article  PubMed  Google Scholar 

  • Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling, 6, 56–83. doi:10.1080/10705519909540119.

    Article  Google Scholar 

  • Gagné, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41, 65–83. doi:10.1207/s15327906mbr4101_5.

    Article  Google Scholar 

  • Gillmore, M. R., Hawkins, J. D., Catalano, R. F., Jr., Day, L. E., Moore, M., & Abbott, R. (1991). Structure of problem behaviors in preadolescence. Journal of Consulting and Clinical Psychology, 59, 499–506. doi:10.1037/0022-006X.59.4.499.

    Article  CAS  PubMed  Google Scholar 

  • Goldstein, H. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73, 43–56. doi:10.1093/biomet/73.1.43.

    Article  Google Scholar 

  • Hopkin, C. R., Hoyle, R. H., & Gottfredson, N. C. (2013). Maximizing the yield of small samples in prevention research: A review of general strategies and best practices. Manuscript submitted for publication.

  • Hoyle, R. H. (2011). Structural equation modeling for social and personality psychology. London: Sage Publications.

    Google Scholar 

  • Hoyle, R. H., & Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 195–222). Thousand Oaks: Sage Publications.

    Google Scholar 

  • Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.

    Article  Google Scholar 

  • Jackson, D. L., Voth, J., & Frey, M. P. (2013). A note on sample size and solution propriety for confirmatory factor analytic models. Structural Equation Modeling, 20, 86–97. doi:10.1080/10705511.2013.742388.

    Article  Google Scholar 

  • Kaplan, D. (1995). Statistical power in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 100–117). Newbury Park: Sage Publications.

    Google Scholar 

  • Kim, K. H. (2005). The relation among fit indexes, power, and sample size in structural equation modeling. Structural Equation Modeling, 12, 368–390. doi:10.1207/s15328007sem1203_2.

    Article  Google Scholar 

  • Kim, S.-Y. (2012). Sample size requirements in single- and multiphase growth mixture models: A Monte Carlo simulation. Structural Equation Modeling, 19, 457–476. doi:10.1080/10705511.2012.687672.

    Article  Google Scholar 

  • Kreft, I. G. G., & de Leeuw, J. (1988). Introducing multilevel modeling. Thousand Oaks: Sage Publications.

    Google Scholar 

  • Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21. doi:10.1207/s15327906mbr3001_1.

    Article  Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229. doi:10.1037/a0012869.

    Article  PubMed  Google Scholar 

  • Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 85–91. doi:10.1027/1614-1881.1.3.86.

    Article  Google Scholar 

  • MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Retrieved from http://doi.apa.org/journals/met/1/2/130.pdf.

    Article  Google Scholar 

  • MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037/1082-989X.7.1.19.

    Article  PubMed  Google Scholar 

  • Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181–220. doi:10.1207/s15327906mbr3302_1.

    Article  Google Scholar 

  • McArdle, J. J., & Nesselroade, J. R. (2003). Growth curve analysis in contemporary psychological research. In J. Schinka & W. Velicer (Eds.), Comprehensive handbook of psychology: Research methods in psychology (pp. 447–480). New York: Wiley. doi:10.1002/0471264385.

    Google Scholar 

  • Meredith, W., & Tisak, J. (1990). Latent curve analysis. Psychometrika, 55, 107–122. doi:10.1007/BF02294746.

    Article  Google Scholar 

  • Noh, M., & Lee, Y. (2007). REML estimation for binary data in GLMMs. Journal of Multivariate Analysis, 98, 896–915. doi:10.1016/j.jmva.2006.11.009.

    Article  Google Scholar 

  • Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2002). Reliable estimation of generalized linear mixed models using adaptive quadrature. The Stata Journal, 2, 1–21. Retrieved from http://www.stata-journal.com/article.html?article=st0005.

    Google Scholar 

  • Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage Publications.

    Google Scholar 

  • Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213. doi:10.1037//1082-989X.5.2.199.

    Article  CAS  PubMed  Google Scholar 

  • Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2004). HLM 6 for Windows [Computer software]. Lincolnwood: Scientific Software.

    Google Scholar 

  • Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., & Martinez, A. (2011). Optimal Design Software for multi-level and longitudinal research (version 3.01) [Software]. Available from www.wtgrantfoundation.org.

  • Rodríguez, G., & Goldman, N. (2001). Improved estimation procedures for multilevel models with binary response: A case study. Journal of the Royal Statistical Society: Series A (Statistics in Society), 164, 339–355. doi:10.1111/1467-985X.00206.

    Article  Google Scholar 

  • Schabenberger, O. (2005). Introducing the GLIMMIX procedure for generalized linear mixed models. Cary: SAS Institute. Retrieved from http://www2.sas.com/proceedings/sugi30/196-30.pdf.

    Google Scholar 

  • Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. Retrieved from http://www.jstor.org/stable/2958889.

    Article  Google Scholar 

  • Shiyko, M. P., Lanza, S. T., Tan, X., Li, R., & Shiffman, S. (2012). Using the time-varying effect model (TVEM) to examine dynamic associations between negative affect and self-confidence on smoking urges: Differences between successful quitters and relapsers. Prevention Science, 13, 288–299. doi:10.1007/s11121-011-0264-z.

    Article  PubMed Central  PubMed  Google Scholar 

  • Simon, T. R., Ikeda, R. M., Smith, E. P., Reese, L. E., Rabiner, D. L., Miller-Johnson, S., et al. (2008). The multisite violence prevention project: Impact of a universal school-based violence prevention program on social-cognitive outcomes. Prevention Science, 9, 231–244. doi:10.1007/s11121-008-0101-1.

    Article  PubMed Central  Google Scholar 

  • Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press.

    Book  Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (2004). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London: Sage Publications.

    Google Scholar 

  • Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. Iowa City: Paper presented at the Meeting of the Psychometric Society.

    Google Scholar 

  • Tanaka, J. S. (1987). “How big is big enough?”: Sample size and goodness of fit in structural equation models with latent variables. Child Development, 58, 134–146. Retrieved from http://www.jstor.org/stable/1130296.

    Article  Google Scholar 

  • Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719–751. doi:10.1177/0011000006286345.

    Article  Google Scholar 

  • Whiteside, S. P., & Lynam, D. R. (2001). The five factor model and impulsivity: Using a structural model of personality to understand impulsivity. Personality and Individual Differences, 30, 669–689. doi:10.1016/S0191-8869(00)00064-7.

    Article  Google Scholar 

Download references

Acknowledgments

During the writing of this manuscript, the authors were supported by National Institute on Drug Abuse (NIDA) Grant P30 DA023026. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NIDA.

Author information

Authors and Affiliations

  1. Department of Psychology and Neuroscience, Duke University, Durham, NC, 27708, USA

    Rick H. Hoyle

  2. Center for Developmental Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, 27599, USA

    Nisha C. Gottfredson

Authors
  1. Rick H. Hoyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nisha C. Gottfredson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rick H. Hoyle.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hoyle, R.H., Gottfredson, N.C. Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling. Prev Sci 16, 987–996 (2015). https://doi.org/10.1007/s11121-014-0489-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-014-0489-8

Keywords

Search

Navigation