Combining principles of Cognitive Load Theory and diagnostic error analysis for designing job aids: Effects on motivation and diagnostic performance in a process control task
Introduction
In the management of industrial operations, job aids serve the objective of ensuring operator competence to perform tasks according to the standards of safety and productivity (Shepherd, 1997). Job aids are designed “to extend human capability to store and process information” (Swezey, 1987, p. 1040). A distinction is made between procedural and decision-making aids. Procedural aids provide step-by-step instructions for completing a task, and guide the user through sequences of actions (Salas et al., 2006; Swezey, 1987). Decision-making aids serve as a reference to help an operator to think along the right lines so as to determine the best decision or solution to a problem (Rossett and Gautier-Downes, 1991; Salas et al., 2006).
Empirical evidence for the usefulness of job aids dates back to the 1960s, when they proved to be especially useful for tasks that are a) infrequent and b) complex or have several steps, and when c) the costs of errors are high, and d) task performance depends on knowing a large amount of information (Rossett and Gautier-Downes, 1991). Job aids proved to be time-saving in terms of time spent on performing repair tasks (Elliot and Joyce, 1968; Kancler et al., 2007) and with regard to training time (Elliot and Joyce, 1971). Since the 1990s, the impact of usefulness and usability of different layout formats of job aids on performance has not been investigated either comprehensively or systematically, with the exception of a study by Spaulding and Dwyer (2001), who found that in comparison to several other job aids, a procedural aid was the most effective.
The present studies focus on procedural and decision-making aids, and address the question of how the design of a procedural job aid and the combination of procedural and decision-making aids affect performance.
As introduced above, job aids are especially useful when tasks are infrequent, complex, and costs of errors are high. These job characteristics apply prominently to the operator's task in process control (Ormerod et al., 1998; Ormerod and Shepherd, 2004; Shepherd, 1986; Shepherd and Marshall, 2005) with respect to diagnosing and repairing familiar and novel faults (Van Gog et al., 2005; Spath et al., 2006; Sauer et al., 2000a), as a high number of interactive information elements need to be processed simultaneously in working memory during learning (Chandler and Sweller, 1991; Van Gog et al., 2008).
Two perspectives on the design of a job aid for process control tasks can be taken.
- 1)
From a Human Factors/Ergonomics point of view, the deduction of job requirements is addressed (‘what’ to include in the job aid), including a task analysis, which results in a necessary set of knowledge, skills, and equipment and technical data, as well as the sequence of critical steps that need to be taken (Phipps et al., 2011; Swezey, 1987; Rossett and Gautier-Downes, 1991).
- 2)
From an instructional psychology point of view, the development of a job aid especially for complex tasks should address issues of cognitive load (“how” to present the content, Sweller et al., 1990). Cognitive Load Theory (CLT, Sweller, 1988, Sweller, 2006) raises the issue of element interactivity, which contributes to the complexity of learning materials. “An element is anything that needs to be understood and/or learned. If elements interact, they cannot be understood in isolation” (Sweller, 2006, p. 13).
Other than the intrinsic cognitive load imposed by the task itself, there is load imposed by cognitive processes induced by the instructional material or learning method (Van Gog et al., 2008). “This type of cognitive load is called extraneous when it is ineffective for learning and germane when it is effective for learning” (Van Gog et al., 2008, p. 212). Job aids should be designed in ways such that they reduce extraneous cognitive load and increase germane cognitive load. In this respect, representatives from CLT (Sweller, 1988), Cognitive Psychology (e.g. Hegarty and Just, 1993) or Human Factors (e.g. Dupont and Bestgen, 2006) recommend the integration of information sources (e.g. text and graphics) in order to avoid extraneous load. One source of avoidable extraneous load is “split attention”, which requires novice learners to attend to several sources of information simultaneously (for further details, see Cierniak et al., 2009; Purnell et al., 1991; Sweller, 2006; Sweller et al., 1990). Our hypothesis is that learners working with a procedural aid that takes into account the avoidance of the split-attention effect.
- -
Perceive lower extraneous load and higher germane load ratings compared to a control group. In contrast, there should be no differences in intrinsic load ratings as long as the task remains the same (H1).
- -
Outperform learners working with a control procedural aid regarding process control performance (i.e., system control and diagnosis) (H2).
However, the design of a procedural aid should address not only the cognitive aspects, but also the motivational and affective aspects as proposed by Swezey (1987), because germane load is assumed to depend on the learner's learning engagement (Sweller et al., 1998). Paas et al. (2005) argue that novices may not be sufficiently motivated to deal with complex learning tasks because learners perceive them as troublesome (Schnotz and Kürschner, 2007). Therefore, motivation “as a dimension that determines learning success” (Paas et al., 2005, p. 25) should also be considered as an important objective when designing procedural aids.
We presume that a procedural aid can also increase germane load by increasing the willingness to engage in exertive learning activities, therefore:
- -
Learners working with a well-designed procedural aid perceive the job aid more positively regarding directing attention, building confidence, and enhancing satisfaction (H3).
In order to investigate the hypotheses, two procedural aids are compared regarding the user opinion and performance criteria.
Section snippets
Participants and design
The experiment employed a between-subjects design with two conditions. Forty students (25 female), who were enrolled in bachelor and master programs in the engineering department at the University of Duisburg-Essen, participated in the experiment. The study was approved by the ethics committee. Subjects were informed about the purposes of the study and told that they could discontinue participation at any time (in terms of informed consent). Participants were randomly assigned to the
Study 2
The first study showed advantages of the newly designed procedural aid over the existing one. However, the post-hoc analysis also revealed that diagnosing novel faults is still more difficult and leads to more errors than diagnosing practised faults. Thus, we assumed that the procedural aid needs the support of a decision aid that addresses the issues of diagnosing novel faults.
Therefore, we reanalyzed the incorrectly diagnosed system faults of Study 1. In Table 4, the four system faults
General discussion
The aim of our research was to investigate how the design of a job aid affects performance in process control. The design of job aids has – to our knowledge – not been previously investigated from a comparative perspective, with the exception of one study conducted in the field of medicine (Spaulding and Dwyer, 2001). In the study by Spaulding and Dwyer (2001), five different job aids were comparatively investigated in terms of their effect on knowledge acquisition in a medical task (a
References (41)
- et al.
Comparative study of three training methods for enhancing process control performance: emphasis shift training, situation awareness training and drill and practice
Comput. Hum. Behav.
(2010) - et al.
Explaining the split-attention effect: is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load?
Comput. Hum. Behav.
(2009) - et al.
Spatiotemporal dynamics of component processes in human working memory
Electroencephalogr. Clin. Neurophysiol.
(1993) - et al.
Constructing mental models of machines from text and diagrams
J. Mem. Lang.
(1993) Towards a framework to select techniques for error prediction: supporting novice users in the health care sector
Appl. Ergon.
(2009)- et al.
Extending hierarchical task analysis to identify cognitive demands and information design requirements
Appl. Ergon.
(2011) - et al.
A conceptual framework for designing micro-worlds for complex work domains: a case study of the cabin air management system
Comput. Hum. Behav.
(2000) Issues in the training of process operators
Int. J. Ind. Ergonom.
(1986)- et al.
Timeliness and task specification in designing for human factors in railway operations
Appl. Ergon.
(2005) Cognitive load during problem solving: effects on learning
Cognitive Sci. Multidisciplinary J.
(1988)
Effects of studying sequences of process-oriented and product-oriented work examples on troubleshooting transfer efficiency
Learn. Instr.
Fundamentals of ergonomics in theory and practice
Appl. Ergon.
Task analysis and training design
J. Occup. Psychol.
Cognitive and personality variables of operators: individual characteristics and process control performance
Cognitive requirement analysis to derive training models for controlling complex systems
The predictive qualities of operator characteristics for process control performance: the influence of variables relating to cognition and personality
Ergonomics
Cognitive load theory and the format of instruction
Cognition Instruct.
Learning from technical documents: the role of intermodal referring expression
Hum. Factors
An Experimental Comparison of Procedural and Conventional Electronic Troubleshooting (AFHRL-TR-68–1)
An experimental evaluation of a method for simplifying electronic maintenance
Hum. Factors
Cited by (18)
A two-part evaluation approach for measuring the usability and user experience of an Augmented Reality-based assistance system to support the temporal coordination of spatially dispersed teams
2021, Cognitive Systems ResearchCitation Excerpt :168). Common examples for scripted coordination in the industrial sector are Standard Operating Procedures (SOPs), procedural job aids (Kluge, Grauel, & Burkolter, 2013), or schedules (Bardram, 2000). Basically, these artefacts represent action scripts that divide processes into subtasks, order them in the correct sequence, indicate start and end points, and allocate subtasks to the corresponding team member.
Can cued recall by means of gaze guiding replace refresher training? An experimental study addressing complex cognitive skill retrieval
2018, International Journal of Industrial ErgonomicsCitation Excerpt :Research on memory and retrieval shows that cued recall is less difficult than free recall (Tulving and Thompson, 1973; Watkins and Tulving, 1975). Job aids constitute a method for applying cued recall for complex cognitive skills, and are used to process, store and extend information (Kluge et al., 2013; Swezey, 1987). Job aids support skills that are used infrequently, consist of sequences, and/or contain a great amount of information, or skills that can cause serious consequences if performed incorrectly (Rossett and Gautier-Downes, 1991).
Human-centered design (HCD) of a fault-finding application for mobile devices and its impact on the reduction of time in fault diagnosis in the manufacturing industry
2017, Applied ErgonomicsCitation Excerpt :For the proceduralization of the testing sequence, training is important, as improvements in problem-solving accuracy may only be achieved through extended practice of sequences and in particular through the availability of information about the quality of moves during problem solving (Duncan, 1985). If training time is limited, the use and application of a fault-finding strategy can be enhanced by traditional and dynamic job aids (Kluge et al., 2013; Kluge, Greve, Borisov & Weyers, in press). However, the advantage of job aids depends on the quality of the guidance provided, which in turn depends on the identification of the relevant information required for successful fault finding.
Hierarchical Task Analysis Reimagined as a Planning Tool for Surgery During Exploration Space Flight
2022, Surgical InnovationEngineering psychology and human performance
2021, Engineering Psychology and Human PerformanceA systematized review of cognitive load theory in health sciences education and a perspective from cognitive neuroscience
2020, Journal of Education and Health Promotion