Abstract
This study focused on how the design of a national student survey instrument was informed and improved through the combined use of student focus groups, cognitive interviews, and expert survey design advice. We were specifically interested in determining (a) how students interpret the items and response options, (b) the frequency of behaviors or activities associated with the response options, (c) if the items are clearly worded and specific enough to produce reliable and valid results, and (d) if the items and response categories accurately represent students' behaviors and perceptions. We collected focus group data from 8 colleges and universities as part of a nationally funded research project on student engagement. The findings provide additional insight into the importance of using focus groups and cognitive interviews to learn how students interpret various items and what different responses really mean.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
American Statistical Association (1997). How to Conduct Pretesting? American Statistical Association, Alexandria, VA.
Chickering, A. W., and Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin 39(7): 3-7.
Dillman, D. (2000). Mail and Internet Surveys: The Tailored Design Method, J. Wiley, New York.
Fetzer Institute and the National Institute on Aging (1999). Multidimensional Measurement of Religiousness/Spirituality for Use in Health Research, Fetzer Institute, Kalamzaoo, MI.
Krause, N. (2002). A comprehensive strategy for developing closed-ended survey items for use in studies of older adults. Journal of Gerontology 578(5): S263-S274.
Krueger, R. A. (1994). Focus Groups: A Practical Guide for Applied Research (2nd Ed.), Sage Publications, Thousand Oaks, CA.
Krueger, R. A. (1998). Moderating focus groups. In: Morgan, D. L., and Krueger, R. A. (eds.), The Focus Group Kit (Vol. 4), Sage Publications, Thousand Oaks, CA.
Kuh, G. D. (1999). The College Student Report, Indiana University Center for Postsecondary Research and Planning, Bloomington.
Kuh, G. D. (2001). Assessing what really matters to student learning: Inside the National Survey of Student Engagement. Change 33(3): 10-17, 66.
Kuh, G. D., Hayek, J. C., Carini, R. M., Ouimet, J. A., Gonyea, R. M., and Kennedy, J. (2001). NSSE Technical and Norms Report, Indiana University Center for Postsecondary Research and Planning, Bloomington.
Lazarsfeld, P. F. (1944). The controversy over detailed interviews: an offer for negotiation. Public Opinion Quarterly 8: 38-60.
Merriam, S. B. (1988). Qualitative Research and Case Study Application in Education, Jossey-Bass, San Francisco.
Morgan, D. L. (1997). Focus Groups as Qualitative Research, Sage Publications, Thousand Oaks, CA.
Morgan, D. L., and Krueger, R. A. (1998). The Focus Group Kit (Vols. 1–7), Sage Publications, Thousand Oaks, CA.
Pace, C. R. (1985). The Credibility of Student Self-Reports, University of California, The Center for the Study of Evaluation, Graduate School of Education, Los Angeles.
Pace, C. R., and Kuh, G. D. (1998). College Student Experience Questionnaire (4th Ed.), Center for Postsecondary Research and Planning, Indiana University, Bloomington.
Sax, L. J., Astin, A. W., Korn, W. S., and Mahoney, K. M. (1999). The American Freshman: National Norms for Fall 1999, University of California Los Angeles, Higher Education Research Institute, Los Angeles.
Tourangeau, R., Rips, L., and Rasinski, K. (2000). The Psychology of Survey Response, Cambridge University Press, New York.
University of North Carolina Program Assessment and Public Service Reports (n.d.). Retrieved September 27, 2002 from, http://www.northcarolina.edu/assessment/reports/reports.cfm
Van de Ven, A. H., and Delbecq, A. L. (1971). Nominal vs. interacting group processes for committee decision making effectiveness. Academy of Management Journal 14: 203-212.
Yin, R. K. (1984). Case Study Research, Sage Publications, Beverly Hills, CA.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ouimet, J.A., Bunnage, J.C., Carini, R.M. et al. Using Focus Groups, Expert Advice, and Cognitive Interviews to Establish the Validity of a College Student Survey. Research in Higher Education 45, 233–250 (2004). https://doi.org/10.1023/B:RIHE.0000019588.05470.78
Issue Date:
DOI: https://doi.org/10.1023/B:RIHE.0000019588.05470.78