Skip to main content
SpringerLink
Log in

A Rationale for and Demonstration of the Use of DIF and Mixed Methods

  • Chapter
  • First Online:
Book cover Understanding and Investigating Response Processes in Validation Research

Part of the book series: Social Indicators Research Series ((SINS,volume 69))

Abstract

Item bias and Differential Item Functioning (DIF) research has been linked to the history of validity theory since it was first introduced. Forty years of research have provided a great arsenal of statistical methods, but few consolidated results on DIF causes. We argue for conducting DIF validation studies within a mixed methods research framework not just to identify DIF causes, but also to understand group differences in response processes and validation. In the chapter, we first introduce the main characteristics of mixed methods studies. Secondly, we present a general framework to conduct DIF mixed methods studies looking at response processes as source of validity evidence. Finally, the research framework is illustrated by a study case of mixed methods DIF study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Bazeley, P. (2012). Integrative analysis Strategies for mixed data sources. American Behavioral Scientist, 56(6), 814–828.

    Article  Google Scholar 

  • Benítez, I., & Padilla, J. L. (2014). Analysis of non-equivalent assessments across different linguistic groups using a mixed methods approach: Understanding the causes of differential item functioning by cognitive interviewing. Journal of Mixed Methods Research, 8(1), 52–68.

    Article  Google Scholar 

  • Benítez, I., Padilla, J. L., Hidalgo, M. D., & Sireci, S. (2015). Using mixed methods to interpret differential item functioning. Applied Measurement in Education, 29(1), 1–16.

    Article  Google Scholar 

  • Bryman, A. (1988). Quality and quantity in social research. London: Routledge.

    Book  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.

    Article  Google Scholar 

  • Creswell, J. W. (1995). Research design: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Creswell, J. W., & Plano Clark, W. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Creswell, J. W., & Plano Clark, W. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Creswell, J. W., & Tashakkori, A. (2007). Editorial: Developing publishable mixed methods manuscripts. Journal of Mixed Methods Research, 1(2), 107111.

    Article  Google Scholar 

  • Elosúa, P., & López-Jaúregui, A. (2007). Potential sources of differential item functioning in the adaptation of tests. International Journal of Testing, 7(1), 39–52.

    Article  Google Scholar 

  • Embretson, (Whitely) S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93(1), 179–197.

    Article  Google Scholar 

  • Ercikan, K., Arim, R., Law, D., Domene, J., Gagnon, F., & Lacroix, S. (2010). Application of think aloud protocols for examining and confirming sources of differential item functioning identified by expert reviews. Educational Measurement: Issues and Practice, 29(2), 24–35.

    Article  Google Scholar 

  • Ferne, T., & Rupp, A. A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4(2), 113–148.

    Article  Google Scholar 

  • Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs principles and practices. Health Services Research, 48(6), 2134–2156.

    Article  Google Scholar 

  • Fetters, M. D., & Freshwater, D. (2015). Publishing a methodological mixed methods research article. Journal of Mixed Methods Research, 9(3), 203–213.

    Article  Google Scholar 

  • Gorard, S., & Symonds, J. (2010). Death of mixed methods? Or the rebirth of research as a craft. Evaluation and Research in Education, 236(2), 121–136.

    Google Scholar 

  • Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274.

    Article  Google Scholar 

  • Hambleton, R. K. (2006). Good practice for identifying differential item functioning. Medical Care, 44(11), S182–S188.

    Article  Google Scholar 

  • Hidalgo, M. D., Benítez, I., Padilla, J. L., & Gómez-Benito, J. (2015). How much polytomous item bias make total-group survey score comparisons invalid? Sociological Methods and Research 1–19.

    Google Scholar 

  • Hidalgo, M. D., & Gómez-Benito, J. (2010). Education measurement: Differential item functioning. In P. Peterson, E. Baker, & B. McGaw (Eds.), International encyclopaedia of education (3rd ed., pp. 36–44). Oxford, UK: Academic Press.

    Chapter  Google Scholar 

  • Johnson, B., & Christensen, L. (2008). Educational research quantitative, qualitative, and mixed approaches. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.

    Article  Google Scholar 

  • Maddox, B., Zumbo, B. D., Tay-Lim, B., & Qu, D. (2015). An anthropologist among the psychometricians: Assessment events, ethnography, and differential item functioning in the Mongolian Gobi. International Journal of Testing, 15(4), 291–309.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: MacMillan.

    Google Scholar 

  • Millsap, R. E., & Everson, H. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17(4), 297–334.

    Article  Google Scholar 

  • OECD (2006). Database - PISA 2006. Retrieved from http://www.oecd.org/pisa/pisaproducts/databasepisa2006.htm

  • Padilla, J. L., & Benitez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136–144.

    Google Scholar 

  • Padilla, J. L., Benitez, I., & Castillo, M. (2013). Obtaining validity evidence by cognitive interviewing to interpret psychometric results. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 9(3), 113–122.

    Article  Google Scholar 

  • Penfield, R. D. (2010). Distinguishing between net and global DIF in polytomous items. Journal of Educational Measurement, 47(2), 129–149.

    Article  Google Scholar 

  • Penfield, R. D., Alvarez, K., & Lee, O. (2009). Using a taxonomy of differential step functioning to improve the interpretation of DIF in polytomous items: An illustration. Applied Measurement in Education, 22(1), 61–78.

    Article  Google Scholar 

  • Sireci, S. G. (2012, April). “De-constructing” test validation. Paper presented at the annual conference of the National Council on Measurement in Education, Vancouver, BC.

    Google Scholar 

  • Sireci, S. G., & Rios, J. A. (2013). Decisions that make a difference in detecting differential item functioning. Educational Research and Evaluation, 19(2–3), 170–187.

    Article  Google Scholar 

  • Tashakkori, A., & Teddlie, C. (2003). The past and future of mixed methods research: From data triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 671–701). Thousand Oaks, CA: Sage.

    Google Scholar 

  • van de Vijver, F. J. R., & Leung, K. (2011). Equivalence and bias: A review of concepts, models, and data analytic procedures. In D. R. Matsumoto & F. J. R. van de Vijver (Eds.), Cross-cultural research methods in psychology. New York: Cambridge University Press.

    Google Scholar 

  • Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233.

    Article  Google Scholar 

  • Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65–82). Charlotte, NC: IAP-Information Age Publishing, Inc..

    Google Scholar 

  • Zumbo, B. D. (2015, November). Consequences, side effects and the ecology of testing: keys to considering assessment ‘In Vivo’. Keynote address, the annual meeting of the Association for Educational Assessment – Europe (AEA-Europe), Glasgow, Scotland. [https://youtu.be/0L6Lr2BzuSQ]

  • Zumbo, B. D., & Gelin, M. N. (2005). A matter of test bias in educational policy research: Bringing the context into picture by investigating sociological/community moderated (or mediated) test and item bias. Journal of Educational Research and Policy Studies, 5(1), 1–23.

    Google Scholar 

  • Zumbo, B. D., Liu, Y., Wu, A. D., Shear, B. R., Olvera, O. L., & Tanvinder, K. A. (2015). A methodology for Zumbo’s third generation DIF analysis and the ecology of item responding. Language Assessment Quarterly, 12(1), 136–151.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Granada, 18071, Granada, Spain

    José-Luis Padilla

  2. Universidad Loyola Andalucía, 41014, Sevilla, Spain

    Isabel Benítez

Authors
  1. José-Luis Padilla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isabel Benítez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José-Luis Padilla .

Editor information

Editors and Affiliations

  1. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Bruno D. Zumbo

  2. Measurement, Evaluation, and Research Methodology (MERM) Program, Department of Educational and Counselling Psychology, and Special Education (ECPS), The University of British Columbia, Vancouver, British Columbia, Canada

    Anita M. Hubley

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Padilla, JL., Benítez, I. (2017). A Rationale for and Demonstration of the Use of DIF and Mixed Methods. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-56129-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-56128-8

  • Online ISBN: 978-3-319-56129-5

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation