Skip to main content
Top
Gepubliceerd in: Quality of Life Research 7/2018

08-02-2018 | Special Section: Test Construction (by invitation only)

Scale development with small samples: a new application of longitudinal item response theory

Auteurs: Carrie R. Houts, Robert Morlock, Steven I. Blum, Michael C. Edwards, R. J. Wirth

Gepubliceerd in: Quality of Life Research | Uitgave 7/2018

Log in om toegang te krijgen
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail

Abstract

Purpose

Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible.

Methods

The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice.

Results

Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling.

Conclusion

Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.
Bijlagen
Alleen toegankelijk voor geautoriseerde gebruikers
Voetnoten
1
These models could be estimated in any program capable of fitting truly high-dimensional multidimensional IRT models (e.g., IRTPRO, the ‘mirt’ package in R, WINBUGS).
 
Literatuur
1.
go back to reference Walton, M. K., Powers, J. H., Hobart, J., Patrick, D., Marquis, P., Vamvakas, S., Isaac, M., Molsen, E., et al. (2015). Clinical outcome assessments: Conceptual foundation—Report of the ispor clinical outcomes assessment—Emerging good practices for outcomes research. Value in Health, 18, 741–752.CrossRefPubMedPubMedCentral Walton, M. K., Powers, J. H., Hobart, J., Patrick, D., Marquis, P., Vamvakas, S., Isaac, M., Molsen, E., et al. (2015). Clinical outcome assessments: Conceptual foundation—Report of the ispor clinical outcomes assessment—Emerging good practices for outcomes research. Value in Health, 18, 741–752.CrossRefPubMedPubMedCentral
3.
go back to reference Reeve, B. B., & Fayers, P. (2005). Applying item response theory modelling for evaluating questionnaire item and scale properties. In P. Fayers & R. Hay (Eds.), Assessing quality of life in clinical trials: Methods & practice (2nd ed.). Oxford: Oxford University Press. Reeve, B. B., & Fayers, P. (2005). Applying item response theory modelling for evaluating questionnaire item and scale properties. In P. Fayers & R. Hay (Eds.), Assessing quality of life in clinical trials: Methods & practice (2nd ed.). Oxford: Oxford University Press.
4.
go back to reference Houts, C. R., Edwards, M. C., Wirth, R. J., & Deal, L. (2016). A review of empirical research related to the use of small quantitative samples in clinical outcome scale development. Quality of Life Research, 25, 2685–2269.CrossRefPubMed Houts, C. R., Edwards, M. C., Wirth, R. J., & Deal, L. (2016). A review of empirical research related to the use of small quantitative samples in clinical outcome scale development. Quality of Life Research, 25, 2685–2269.CrossRefPubMed
5.
go back to reference Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133–144.CrossRef Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133–144.CrossRef
6.
go back to reference Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd edn.). New York: Marcel Decker, Inc.CrossRef Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd edn.). New York: Marcel Decker, Inc.CrossRef
7.
go back to reference Thissen, D., & Wainer, H. (Eds.). (2001). Test Scoring. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Thissen, D., & Wainer, H. (Eds.). (2001). Test Scoring. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
8.
go back to reference Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. New York: Psychology Press. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. New York: Psychology Press.
9.
go back to reference Linden, W. J. Van der, & Hambleton, R. K. (Eds.). Handbook of modern item response theory. New York: Springer. Linden, W. J. Van der, & Hambleton, R. K. (Eds.). Handbook of modern item response theory. New York: Springer.
10.
go back to reference Reckase, M. D. (2009). Multidimensional item response theory models. New York: Springer.CrossRef Reckase, M. D. (2009). Multidimensional item response theory models. New York: Springer.CrossRef
11.
go back to reference Cai, L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307–335.CrossRef Cai, L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307–335.CrossRef
12.
go back to reference Oort, F. (2005). Using structural equation modeling to detect response shifts and true change. Quality of Life Research, 14, 587–598.CrossRefPubMed Oort, F. (2005). Using structural equation modeling to detect response shifts and true change. Quality of Life Research, 14, 587–598.CrossRefPubMed
13.
go back to reference Millsap, R. E. (2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4, 5–9.CrossRef Millsap, R. E. (2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4, 5–9.CrossRef
14.
go back to reference Douglas, J. A. (1999). Item response models for longitudinal quality of life data in clinical trials. Statistics in Medicine, 18, 2917–2931.CrossRefPubMed Douglas, J. A. (1999). Item response models for longitudinal quality of life data in clinical trials. Statistics in Medicine, 18, 2917–2931.CrossRefPubMed
15.
go back to reference Cai, L. (2015). flexMIRT® version 3: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Chapel Hill, NC: Vector Psychometric Group. Cai, L. (2015). flexMIRT® version 3: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Chapel Hill, NC: Vector Psychometric Group.
16.
go back to reference Roberts, G. O., & Rosenthal, J. S. (2001). Optimal scaling for various Metropolis-Hastings algorithms. Statistical Science, 16, 351–367.CrossRef Roberts, G. O., & Rosenthal, J. S. (2001). Optimal scaling for various Metropolis-Hastings algorithms. Statistical Science, 16, 351–367.CrossRef
17.
go back to reference Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.CrossRef Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.CrossRef
18.
go back to reference Wirth, R. J., Edwards, M. C., Henderson, M., Henderson, T., Olivares, G., & Houts, C. R. (2016). Development of the contact lens user experience: CLUE Scales. Optometry and Vision Science, 93, 801–808.CrossRefPubMedPubMedCentral Wirth, R. J., Edwards, M. C., Henderson, M., Henderson, T., Olivares, G., & Houts, C. R. (2016). Development of the contact lens user experience: CLUE Scales. Optometry and Vision Science, 93, 801–808.CrossRefPubMedPubMedCentral
19.
go back to reference Edelen, M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16, 5–18.CrossRefPubMed Edelen, M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research, 16, 5–18.CrossRefPubMed
21.
go back to reference Brown, R. L. (1991). The effect of collapsing ordered polytomous scales on parameter estimates in structural equation measurement models. Educational and Psychological Measurement, 51(2), 317–328.CrossRef Brown, R. L. (1991). The effect of collapsing ordered polytomous scales on parameter estimates in structural equation measurement models. Educational and Psychological Measurement, 51(2), 317–328.CrossRef
22.
go back to reference Wollack, J. A., Bolt, D. M., Cohen, A. S., & Lee, Y.-S. (2002). Recovery of item parameters in the nominal response model: A comparison of marginal maximum likelihood estimation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 26, 339–352.CrossRef Wollack, J. A., Bolt, D. M., Cohen, A. S., & Lee, Y.-S. (2002). Recovery of item parameters in the nominal response model: A comparison of marginal maximum likelihood estimation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 26, 339–352.CrossRef
23.
go back to reference Hegade, V. S., Kendrick, S. F., Dobbins, R. L., Miller, S. R., Thompson, D., Richards, D., Storey, J., et al. (2017). Effect of ileal bile acid transporter inhibitor GSK2330672 on pruritus in primary biliary cholangitis: A double-blind, randomised, placebo-controlled, crossover, phase 2a study. The Lancet. https://doi.org/10.1016/S0140-6736(17)30319-7.CrossRef Hegade, V. S., Kendrick, S. F., Dobbins, R. L., Miller, S. R., Thompson, D., Richards, D., Storey, J., et al. (2017). Effect of ileal bile acid transporter inhibitor GSK2330672 on pruritus in primary biliary cholangitis: A double-blind, randomised, placebo-controlled, crossover, phase 2a study. The Lancet. https://​doi.​org/​10.​1016/​S0140-6736(17)30319-7.CrossRef
24.
go back to reference Talwalkar, J. A., & Lindor, K. D. (2003). Primary biliary cirrhosis. The Lancet, 362(9377), 53–61.CrossRef Talwalkar, J. A., & Lindor, K. D. (2003). Primary biliary cirrhosis. The Lancet, 362(9377), 53–61.CrossRef
26.
go back to reference Bergasa, N. V. (2014). Pruritus of cholestasis. In E. Carstens & T. Akiyama (Eds.), Itch: Mechanisms and treatment. Boca Raton: CRC Press/Taylor & Francis. Bergasa, N. V. (2014). Pruritus of cholestasis. In E. Carstens & T. Akiyama (Eds.), Itch: Mechanisms and treatment. Boca Raton: CRC Press/Taylor & Francis.
27.
go back to reference Beuers, U., Kremer, A. E., Bolier, R., & Elferink, R. P. (2014). Pruritus in cholestasis: Facts and fiction. Hepatology, 60(1), 399–407.CrossRefPubMed Beuers, U., Kremer, A. E., Bolier, R., & Elferink, R. P. (2014). Pruritus in cholestasis: Facts and fiction. Hepatology, 60(1), 399–407.CrossRefPubMed
28.
go back to reference Jones, E. A., & Bergasa, N. V. (1999). The pruritus of cholestasis. Hepatology, 29(4), 1003–1006.CrossRefPubMed Jones, E. A., & Bergasa, N. V. (1999). The pruritus of cholestasis. Hepatology, 29(4), 1003–1006.CrossRefPubMed
Metagegevens
Titel
Scale development with small samples: a new application of longitudinal item response theory
Auteurs
Carrie R. Houts
Robert Morlock
Steven I. Blum
Michael C. Edwards
R. J. Wirth
Publicatiedatum
08-02-2018
Uitgeverij
Springer International Publishing
Gepubliceerd in
Quality of Life Research / Uitgave 7/2018
Print ISSN: 0962-9343
Elektronisch ISSN: 1573-2649
DOI
https://doi.org/10.1007/s11136-018-1801-z

Andere artikelen Uitgave 7/2018

Quality of Life Research 7/2018 Naar de uitgave

Special Section: Test Construction (by invitation only)

Measurement invariance, the lack thereof, and modeling change