Abstract
There are many excellent general textbooks available on doing research and you will find references at the end of this chapter. To find out more about qualitative and quantitative research design you are referred to them, and references are also given on specific methodologies such as questionnaire design, interview, and verbal protocols, statistical analysis procedures, and discourse analyses of testee performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Further reading
Brown, Anderson, Shillcock and Yule (1984)
Foster and Skehan (1996, 1999)
Mehnert (1998)
Norris et al. (1998)
Ortega (1999)
Skehan (1996)
Skehan and Foster (1997)
Wigglesworth (1997)
Lazaraton (2002), O’Loughlin, (2001), Pavlou (1997) and Shohamy (1994) analysed discourse features across different speaking tasks and detail useful methods for analysing such data.
Cumming (1997) provides a very useful summary of the literature that has looked at the characteristics of written texts in language tests.
Scott (2002) Wordsmith Tools website: http://www.lexically.net /wordsmith/version3/index.html
Urquhart and Weir (1998: 270–95) provide a full account of a range of procedures and methods of analysis, with examples which may be used in studies which attempt to unpack the reading process operationalized through a reading test.
On what happens in the test taking process or for further ideas on how to investigate this for yourself, it is worth looking at:
Buck (1991), Hale and Courtney (1994) and Wu Yian (1998) on listening.
Anderson et al (1991), Allan (1992), Crain-Thoreson et al (1997) Nevo (1989), Perkins (1992), Phakiti (2003), Storey (1995 and 1997) and Weir et al (2000) on reading
Ross (1992) on speaking
Smagorinsky (1994) on writing
On protocol analysis
Cohen (1984, 1988, 1994 and 1997) on verbal reports for investigating test-taking
Gass and Mackey (2000) for a useful theoretical and practical account of verbal protocol analysis
Green (1998) on verbal protocol analysis in language testing research.
Ericsson and Simon (1993) on protocol analysis: verbal report
Pressley and Afflerbach (1995) on verbal protocols for reading
Stratman and Hamp-Lyons (1994) on concurrent think-aloud protocols.
Baker (1997) is one of the few accessible accounts of using IRT analysis.
Bryman and Cramer (2001) is an accessible guide to using SPSS.
Crocker and Algina (1986) provide a solid and accessible account of the statistics you may need.
Fulcher (2003) Chapter 7 provides an accessible account of correlation, factor analysis, multi-trait, multi-method studies and generalizability studies.
Kim and Muller (1978a) There are no easy introductions to factor analysis but this is as close as it gets at the moment.
Kim and Muller (1978b) is the next step up on the previous reference.
McNamara (1996: Chapters 5–8 in particular) is one of the few readable and comprehensive accounts of using Rasch analysis, with examples of how to interpret the output.
Myford and Wolfe (2000, 2003, 2004) for accessible accounts of MFR.
Norris et al. (1998) for an extended discussion on how to estimate task difficulty along a number of parameters and a suggested procedure for operationalizing this SPSS Inc. (2002) The handbook that goes with the program.
O’Sullivan, Weir and Sa ville (2002). Describes the rationale and development of the observation checklists described above.
Weir and Milanovic (2003) provide a detailed coverage of what one Exam Board, Cambridge ESOL, does in this area. One of the few open and frank accounts of what goes on behind the scenes at one of the world’s leading test deliverers.
Yang and Weir (1998) for a discussion of how parallel forms are achieved for the College English Test (CET) in China taken by over 10 million candidates a year.
Washback studies
Green (2003) provides a very good survey of the literature in the field, some useful data collection instruments, and some innovative methods of analyses.
Cheng (2004) provides a recent comprehensive study of the effect of the introduction of a new examination with a useful methodology section.
Cheng and Watanabe (eds.) (2004) provide a useful and varied set of papers on recent developments in this area
Wall (2004) provides an interesting account of washback in the Sri Lankan context and sets it in the context of change theory
Recommended reading
Allan (1995) on questionnaires.
Allwright (1988) on observation.
Banerjee and Luoma (1997) on qualitative approaches to test validation.
Brown (1991) on research methods.
Cohen, Manion and Morrison (2000) on research methods in education.
Denicolo and Pope (eds.) (1997) on interviewing.
Foddy (1994) on constructing questions for interviews and questionnaires: language research.
Fulcher (2003: chapter 7) provides an accessible overview of correlation, factor analysis, multi-trait, multi-method studies, generalizability studies, multi-faceted Rasch analysis, expert judgement, questionnaires and interviews, discourse analysis and verbal protocol analysis
Hatch and Lazaraton (1997) on design and statistics for applied linguistics.
Miles and Huberman (1994) on qualitative data analysis.
Oppenheim (1992) on questionnaire design, interviewing and attitude measurement.
Patton (2002) on qualitative research methods.
Scwartz and Sudman (eds.) (1996) on the methodology for determining cognitive and communicative processes in survey research
Urquhart and Weir (1998: chapter 5) provide detailed accounts of how to do research in the area of reading and provide examples of instruments and analysis.
Weir and Roberts (1994) describe the advantages and disadvantages of a variety of survey instruments and how to construct and operationalize them
Weir, Yang and Jin (2000) investigate the construct of reading by a variety of methods.
Author information
Authors and Affiliations
Copyright information
© 2005 Cyril J. Weir
About this chapter
Cite this chapter
Weir, C.J. (2005). Research Methodologies for Exploring the Validity of a Test. In: Language Testing and Validation. Research and Practice in Applied Linguistics. Palgrave Macmillan, London. https://doi.org/10.1057/9780230514577_12
Download citation
DOI: https://doi.org/10.1057/9780230514577_12
Publisher Name: Palgrave Macmillan, London
Print ISBN: 978-1-4039-1189-6
Online ISBN: 978-0-230-51457-7
eBook Packages: Palgrave Language & Linguistics CollectionEducation (R0)