Abstract
The reactivity of a “think aloud” verbal protocol and the veridicality of different retrospective protocols were tested over four dissimilar tasks. Generating a concurrent protocol altered accuracy in two tasks, simple addition and a choice between two gambles, and generally prolonged response times. Such reactivity partially qualifies the dominant theory of protocol generation (Ericsson & Simon, 1984). Retrospective protocols yielded substantial forgetting or fabrication in all tasks, supporting the consensus on the nonveridicality of these methods. It is concluded that protocol validity should be based on an empirical check rather than on theory-based assurances.
Article PDF
Similar content being viewed by others
References
Anderson, J. R. (1979). Further arguments concern representations for mental imagery: A response to Hayes-Roth and Pylyshyn.Psychological Review,86, 395–406.
Arnolo, H., &Lee, B. (1978a).Jumble—That scrambled word game #12. New York: New American Library.
Arnold, H., &Lee, B. (1978b).Jumble—That scrambled word game #13. New York: New American Library.
Biehal, G., &Chakravarti, D. (1989). The effects of concurrent verbalization on choice processing.Journal of Marketing Research,26, 84–96.
Brooks, L. R. (1968). Spatial and verbal components m the act of recall.Canadian Journal of Psychology,22, 349–368.
Carroll, J. S., &Payne, J. W. (1977). Judgments about crime and the criminal: A model and a method for investigating parole decisions. In B. D. Sales (Ed.),Perspectives in law and psychology: Vol. 1. The criminal justice system (pp. 191–239). New York: Plenum.
Cochran, W. G., &Cox, G. M. (1957).Experimental designs (2nd ed.) New York: Wiley
Crowder, R. G. (1970). The role of one-s voice in immediate memoryCognitive Psychology,1, 157–178.
Dansereau, D. F. (1969). An information processing model of mental multiplication.Dissertation Abstracts International,30, 1916-B (University Microfilms No. 69-15746)
Dansereau, D. F., &Gregg, L. (1966). An information processing analysis of mental multiplication.Psychonomic Science,6, 71–72.
Ericsson, K. A., &Simon, H. A. (1980). Verbal reports as data.Psychological Review,87, 215–251.
Ericsson, K. A., &Simon, H. A. (1984).Protocol analysis: Verbal reports as data. Cambridge, MA: MIT. Press.
Feldman, J. (1959). An analysis of predictive behavior in a two-choice situation.Index to American Doctoral Dissertations (1958-59),19, 149
Filer, E. J. (1983). The reliability and validity of concurrent, retrospective, and interpretive verbal reports: An experimental study. In P. Humphreys, O. Svenson, & A. Vari (Eds.),Analyzing and aiding decision processes (pp. 429–440). Amsterdam: North-Holland.
Fryer, D. H. (1941). Articulation in automatic mental work.American Journal of Psychology,54, 504–517.
Hitch, G. J. (1978). The role of short-term working memory in mental arithmetic.Cognitive Psychology,10, 302–323.
Hunt, E. (1974). Quote the raven? Nevermore! In L. W. Gregg (Ed.),Knowledge and cognition (pp. 129–157). Potomac, MD: Erlbaum.
Johnson, E. J., &Payne, J. W. (1985). Effort and accuracy in choice.Management Science,31, 395–414.
Johnson, E. J., &Russo, J. E. (1978).What is remembered after a purchase decision? (C.I.P. Working Paper No. 379). Pittsburgh, PA: Carnegie-Mellon University, Graduate School of Industrial Administration.
Just, M. A., &Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension.Psychological Review,87, 329–354.
Kantowitz, B. H. (1987). Mental workload. In P. A. Hancock (Ed.),Human factors psychology. Amsterdam: Elsevier North-Holland.
Karpf, D. A. (1973). Thinking aloud in human description learning.Dissertation Abstracts International,33, 6111-B. (University Microfilms No. 73-13625.)
Kazdin, A. E. (1976). Assessment of imagery during covert modeling of assertive behavior.Journal of Behavior Therapy & Experimental Psychiatry,7, 213–219.
Merz, F. (1969). Der Einfluss des Verbalisiernens auf die Leistung bei Intelligenzaufgaben.Zeitschrift für Experimentelle und Angewandte Psychologie,16, 114–137.
Navos, D., &Gopher, D. (1979). On the economy of the human processing system.Psychological Review,86, 254–284.
Neter, J., &Wasserman, W. (1974).Applied linear statistical models. Homewood, IL: Irwin.
Newell, A., &Simon, H. A. (1972).Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
Nisbett, R. E., &Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes.Psychological Review,84, 231–259.
Payne, J. W., Braunstein, M. L., &Carroll, J. S. (1978). Exploring predecisional behavior: An alternative approach to decision research.Organizational Behavior & Human Performance,22, 17–44.
Pensey, C. G. (1975). Morality effects in short-term verbal memory.Psychological Bulletin,82, 68–84.
Quattrone, G. A. (1985). On the congruity between internal states and action.Psychological Bulletin,98, 3–40.
Raves, J. C. (1958).Standard progressive matrices. London: H. K. Lewis.
Pip, P. D. (1979). The basis and extent of self-awareness ~n decision making. A study in consumer behavior.Dissertation Abstracts International,40, 4206-A. (University Microfilms No. DDJ80-01997)
Roth, B. (1966). The effect of overt verbalization on problem solving.Dissertation Abstracts International,27, 957-B. (University Microfilms No. 65-9321)
Russo, J. E. (1979). A software system for the collection of retrospective protocols prompted by eye fixations.Behavior Research Methods & Instrumentation,11, 177–179.
Russo, J. E., &Dosher, B. A. (1983). Strategies for multiattribute binary choice.Journal of Experimental Psychology: Learning, Memory, & Cognition,9, 676–696.
Sabini J., &Silver, M. (1981). Introspection and causal accounts.Journal of Personality & Social Psychology,40, 171–179
Schweiger, D. M. (1983). Is the simultaneous verbal protocol a viable method for studying managerial problem solving and decision making?Academy of Management Journal,26, 185–192.
Simon, H. A. (1975). The functional equivalence of problem solving skills.Cognitive Psychology,7, 268–288.
Smead, R. J., Wilcox, J. B., &Wilkes, R. E. (1981). How valid are product descriptions and protocols in choice experiments?Journal of Consumer Research,8, 37–42.
Smith, E. R., &Miller, F. M. (1978). Limits on perception of cognitive processes: A reply to Nisbett and Wilson.Psychological Review,76, 211–224.
Tell, P. M. (1971). Influence of vocalization on short-term memory.Journal of Verbal Learning & Verbal Behavior,10, 149–156.
Tetlock, P. E., &Kim, J. I. (1987). Accountability and judgment processes in a personality prediction task.Journal of Personality & Social Psychology,52, 700–709.
Turner, C. K. (1986).The use of introspective reports for studying judgment processes (Working paper). Los Angeles: University of California, Department of Psychology.
Turer, C. K. (1988). Don-t blame memory for people-s faulty reports on what influences their judgments.Personality & Social Psychology Bulletin,14, 622–629.
Walker, W. H. (1982). Retrieval of knowledge from memory: A generalization of the Raaijmakers-Shiffrin retrieval model.Dissertation Abstracts International,43, 4183-B. (University Microfilms No. DEP83-09875.)
Weitz, B., &Wright, P. (1979). Retrospective self-insight on factors considered in product evaluation.Journal of Consumer Research,6, 280–294.
Weaner, H., &Kaplan, B. (1963).Symbol formation. New York: Wiley.
Wickens, C. D. (1980). The structure of attentional resources. In R. Nickerson (Ed.),Attention and performance VIII (pp. 239–257). Hillsdale, NJ: Erlbanm.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasuraman and D. R. Davis (Eds.),Varieties of attention (pp. 63–102). New York: Academic Press.
Wickess, C. D. (1987). Attention. In P. A. Hancock (Ed.),Human factors psychology (pp. 29–80). Amsterdam: Elsevier North-Holland.
Winikoef, A. (1967). Eye movements as an aid to protocol analysis of problem solving behavior. Index to American Doctoral Dissertations (1967–68), 129.
Wright, P., &Pip, R. D. (1981). Retrospective reports on the causes of decisions.Journal of Personality & Social Psychology,40, 601–614.
Zbrodoff, N. J., &Logan, G. D. (1986). On the autonomy of mental processes: A case study of arithmetic.Journal of Experimental Psychology: General,115, 118–130.
Author information
Authors and Affiliations
Additional information
This acknowledgement should not be taken to imply their agreement with our conclusions.
Rights and permissions
About this article
Cite this article
Russo, J.E., Johnson, E.J. & Stephens, D.L. The validity of verbal protocols. Mem Cogn 17, 759–769 (1989). https://doi.org/10.3758/BF03202637
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03202637