Marilyn Hughes Blackmon
Muneo Kitajima
ABSTRACT
The Cognitive Walkthrough for the Web (CWW) is a partially automated usability evaluation method for identifying and repairing website navigation problems. Building on five earlier experiments [3,4], we first conducted two new experiments to create a sufficiently large dataset for multiple regression analysis. Then we devised automatable problem-identification rules and used multiple regression analysis on that large dataset to develop a new CWW formula for accurately predicting problem severity. We then conducted a third experiment to test the prediction formula and refined CWW against an independent dataset, resulting in full cross-validation of the formula. We conclude that CWW has high psychological validity, because CWW gives us (a) accurate measures of problem severity, (b) high success rates for repairs of identified problems (c) high hit rates and low false alarms for identifying problems, and (d) high rates of correct rejections and low rates of misses for identifying non-problems.
- Blackmon, M. H. Cognitive Walkthrough. In W. S. Bainbridge (Ed.), Encyclopedia of Human-Computer Interaction, 2 volumes. Great Barrington, MA: Berkshire Publishing, 2004.Google Scholar
- Blackmon, M. H., Kitajima, M., Mandalia, D. R., & Polson, P. G. Automating Usability Evaluation: Cognitive Walkthrough for the Web Puts LSA to Work on Real-World HCI Design Problems. In T. K. Landauer, D. S. McNamara, S. J. Dennis, and W. Kintsch (Eds.), LSA: A Road to Meaning. Mahwah, NJ: Erlbaum (to appear, 2006).Google Scholar
- Blackmon, M. H., Kitajima, M., & Polson, P.G. (2003) Repairing usability problems identified by the Cognitive Walkthrough for the Web. Proc. of CHI 2003, ACM Press (2003), 497--504. Google ScholarDigital Library
- Blackmon, M. H., Polson, P. G., Kitajima, M., & Lewis, C. Cognitive Walkthrough for the Web. Proc. of CHI 2002, ACM Press (2002), 463--470. Google ScholarDigital Library
- Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. Using information scent to model user information needs and actions and the Web. Proc. of CHI 2001, ACM Press (2001), 490--497. Google ScholarDigital Library
- Chi, E., Pirolli, P., & Pitkow, J. The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a website. In Proceedings of CHI 2000, ACM Press (2000), 161--168. Google ScholarDigital Library
- Chi, E. H., Rosien, A., Supattanasiri, G., Williams, A., Royer, C., Chow, C., Robles, E., Dalal, B., Chen, J., & Cousins, S. (2003). The Bloodhound Project: Automating discovery of web usability issues using the InfoScent®® Simulator. Proc. of CHI 2003, ACM Press (2003), 505--512. Google ScholarDigital Library
- Furnas, G. W. Effective view navigation. Proc. of CHI'97, ACM Press (1997), 367--374. Google ScholarDigital Library
- Gray, W. D., & Salzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13 (1998), 203--261. Google ScholarDigital Library
- Gray, W. D., & Salzman, M. C. Repairing damaged merchandise: A rejoinder. Human-Computer Interaction, 13 (1998), 325--335. Google ScholarDigital Library
- Hertzum, M, & Jacobsen, N.E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15 (2003), 183--204.Google ScholarCross Ref
- Ivory-Ndiaye, M. Y. An empirical approach to automated website evaluation. Journal of Digital Information Management, 1 (2003), 75--102.Google Scholar
- Katz, M. A., & Byrne, M. D. Effects of scent and breadth on use of site-specific search on e-commerce websites. ACM Transactions on Computer-Human Interaction, 10 (2003), 198--220. Google ScholarDigital Library
- Kintsch, W. Comprehension: A paradigm for cognition. Cambridge, U.K. & New York: Cambridge University Press, 1998.Google Scholar
- Kitajima, M., Blackmon, M. H., & Polson, P. G. (2000). A Comprehension-based model of Web navigation and its application to Web usability analysis. Proc. of HCI 2000. Springer-Verlag (2000), 357--373.Google ScholarCross Ref
- Kitajima, M., & Polson, P. G. A comprehension-based model of exploration. Human-Computer Interaction, 12 (1997), 345--389. Google ScholarDigital Library
- Landauer, T. K. Learning and representing verbal meaning: Latent Semantic Analysis theory. Current Directions in Psychological Science, 7 (1998), 161--164.Google ScholarCross Ref
- Landauer, T. K. & Dumais, S. T. A solution to Plato's problem: The Latent Semantic Analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104 (1997), 211--240.Google ScholarCross Ref
- Landauer, T. K., Foltz, P., & Laham, D. An introduction to Latent Semantic Analysis. Discourse Processes, 25 (1998), 259--284.Google ScholarCross Ref
- Larson, K., & Czerwinski, M. Webpage design: Implications of memory, structure and scent for information retrieval. In Proceedings of CHI'98, ACM Press (1998), 25--32. Google ScholarDigital Library
- Miller, C. S., & Remington, R. W. Modeling Information Navigation: Implications for Information Architecture. Human-Computer Interaction, 19 (2004), 225--271. Google ScholarDigital Library
- Nielsen, J., & Mack, R. L. Usability Inspection Methods. New York: John Wiley & Sons, Inc., 1994. Google ScholarDigital Library
- Pirolli, P. The use of proximal information scent to forage for distal content on the World Wide Web. In Kirlik, A. (Ed.), Working with technology in mind: Brunswikian resources for cognitive science and engineering, in press.Google Scholar
- Pirolli, P., & Card, S. K. Information foraging. Psychological Review, 106 (1999), 643--675.Google ScholarCross Ref
- Pirolli, P. L., & Fu, W. SNIF-ACT: a model of information foraging on the World Wide Web (9th International Conference on User Modeling). Lecture Notes in Artificial Intelligence, 2702 (2003), 45--54. Google ScholarDigital Library
- Wharton, C., Rieman, J., Lewis, C., & Polson, P. The cognitive walkthrough method: A practitioner's guide. In J. Nielsen & R. L. Mack (Eds.), Usability Inspection Methods (pp. 105--140). New York: John Wiley, 1994. Google ScholarDigital Library
Index Terms
- Tool for accurately predicting website navigation problems, non-problems, problem severity, and effectiveness of repairs
Recommendations
Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsUsability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
Cognitive walkthrough for the web
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThis paper proposes a transformation of the Cognitive Walkthrough (CW), a theory-based usability inspection method that has proven useful in designing applications that support use by exploration. The new Cognitive Walkthrough for the Web (CWW) is ...
Repairing usability problems identified by the cognitive walkthrough for the web
CHI '03: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMethods for identifying usability problems in web page designs should ideally also provide practical methods for repairing the problems found. Blackmon et al. [2] proved the usefulness of the Cognitive Walkthrough for the Web (CWW) for identifying three ...
Comments