Elsevier

Applied Ergonomics

Volume 44, Issue 2, March 2013, Pages 285-296
Applied Ergonomics

Combining principles of Cognitive Load Theory and diagnostic error analysis for designing job aids: Effects on motivation and diagnostic performance in a process control task

https://doi.org/10.1016/j.apergo.2012.08.001Get rights and content

Abstract

Two studies are presented in which the design of a procedural aid and the impact of an additional decision aid for process control were assessed. In Study 1, a procedural aid was developed that avoids imposing unnecessary extraneous cognitive load on novices when controlling a complex technical system. This newly designed procedural aid positively affected germane load, attention, satisfaction, motivation, knowledge acquisition and diagnostic speed for novel faults. In Study 2, the effect of a decision aid for use before the procedural aid was investigated, which was developed based on an analysis of diagnostic errors committed in Study 1. Results showed that novices were able to diagnose both novel faults and practised faults, and were even faster at diagnosing novel faults. This research contributes to the question of how to optimally support novices in dealing with technical faults in process control.

Introduction

In the management of industrial operations, job aids serve the objective of ensuring operator competence to perform tasks according to the standards of safety and productivity (Shepherd, 1997). Job aids are designed “to extend human capability to store and process information” (Swezey, 1987, p. 1040). A distinction is made between procedural and decision-making aids. Procedural aids provide step-by-step instructions for completing a task, and guide the user through sequences of actions (Salas et al., 2006; Swezey, 1987). Decision-making aids serve as a reference to help an operator to think along the right lines so as to determine the best decision or solution to a problem (Rossett and Gautier-Downes, 1991; Salas et al., 2006).

Empirical evidence for the usefulness of job aids dates back to the 1960s, when they proved to be especially useful for tasks that are a) infrequent and b) complex or have several steps, and when c) the costs of errors are high, and d) task performance depends on knowing a large amount of information (Rossett and Gautier-Downes, 1991). Job aids proved to be time-saving in terms of time spent on performing repair tasks (Elliot and Joyce, 1968; Kancler et al., 2007) and with regard to training time (Elliot and Joyce, 1971). Since the 1990s, the impact of usefulness and usability of different layout formats of job aids on performance has not been investigated either comprehensively or systematically, with the exception of a study by Spaulding and Dwyer (2001), who found that in comparison to several other job aids, a procedural aid was the most effective.

The present studies focus on procedural and decision-making aids, and address the question of how the design of a procedural job aid and the combination of procedural and decision-making aids affect performance.

As introduced above, job aids are especially useful when tasks are infrequent, complex, and costs of errors are high. These job characteristics apply prominently to the operator's task in process control (Ormerod et al., 1998; Ormerod and Shepherd, 2004; Shepherd, 1986; Shepherd and Marshall, 2005) with respect to diagnosing and repairing familiar and novel faults (Van Gog et al., 2005; Spath et al., 2006; Sauer et al., 2000a), as a high number of interactive information elements need to be processed simultaneously in working memory during learning (Chandler and Sweller, 1991; Van Gog et al., 2008).

Two perspectives on the design of a job aid for process control tasks can be taken.

  • 1)

    From a Human Factors/Ergonomics point of view, the deduction of job requirements is addressed (‘what’ to include in the job aid), including a task analysis, which results in a necessary set of knowledge, skills, and equipment and technical data, as well as the sequence of critical steps that need to be taken (Phipps et al., 2011; Swezey, 1987; Rossett and Gautier-Downes, 1991).

  • 2)

    From an instructional psychology point of view, the development of a job aid especially for complex tasks should address issues of cognitive load (“how” to present the content, Sweller et al., 1990). Cognitive Load Theory (CLT, Sweller, 1988, Sweller, 2006) raises the issue of element interactivity, which contributes to the complexity of learning materials. “An element is anything that needs to be understood and/or learned. If elements interact, they cannot be understood in isolation” (Sweller, 2006, p. 13).

Other than the intrinsic cognitive load imposed by the task itself, there is load imposed by cognitive processes induced by the instructional material or learning method (Van Gog et al., 2008). “This type of cognitive load is called extraneous when it is ineffective for learning and germane when it is effective for learning” (Van Gog et al., 2008, p. 212). Job aids should be designed in ways such that they reduce extraneous cognitive load and increase germane cognitive load. In this respect, representatives from CLT (Sweller, 1988), Cognitive Psychology (e.g. Hegarty and Just, 1993) or Human Factors (e.g. Dupont and Bestgen, 2006) recommend the integration of information sources (e.g. text and graphics) in order to avoid extraneous load. One source of avoidable extraneous load is “split attention”, which requires novice learners to attend to several sources of information simultaneously (for further details, see Cierniak et al., 2009; Purnell et al., 1991; Sweller, 2006; Sweller et al., 1990). Our hypothesis is that learners working with a procedural aid that takes into account the avoidance of the split-attention effect.

  • -

    Perceive lower extraneous load and higher germane load ratings compared to a control group. In contrast, there should be no differences in intrinsic load ratings as long as the task remains the same (H1).

  • -

    Outperform learners working with a control procedural aid regarding process control performance (i.e., system control and diagnosis) (H2).

However, the design of a procedural aid should address not only the cognitive aspects, but also the motivational and affective aspects as proposed by Swezey (1987), because germane load is assumed to depend on the learner's learning engagement (Sweller et al., 1998). Paas et al. (2005) argue that novices may not be sufficiently motivated to deal with complex learning tasks because learners perceive them as troublesome (Schnotz and Kürschner, 2007). Therefore, motivation “as a dimension that determines learning success” (Paas et al., 2005, p. 25) should also be considered as an important objective when designing procedural aids.

We presume that a procedural aid can also increase germane load by increasing the willingness to engage in exertive learning activities, therefore:

  • -

    Learners working with a well-designed procedural aid perceive the job aid more positively regarding directing attention, building confidence, and enhancing satisfaction (H3).

In order to investigate the hypotheses, two procedural aids are compared regarding the user opinion and performance criteria.

Section snippets

Participants and design

The experiment employed a between-subjects design with two conditions. Forty students (25 female), who were enrolled in bachelor and master programs in the engineering department at the University of Duisburg-Essen, participated in the experiment. The study was approved by the ethics committee. Subjects were informed about the purposes of the study and told that they could discontinue participation at any time (in terms of informed consent). Participants were randomly assigned to the

Study 2

The first study showed advantages of the newly designed procedural aid over the existing one. However, the post-hoc analysis also revealed that diagnosing novel faults is still more difficult and leads to more errors than diagnosing practised faults. Thus, we assumed that the procedural aid needs the support of a decision aid that addresses the issues of diagnosing novel faults.

Therefore, we reanalyzed the incorrectly diagnosed system faults of Study 1. In Table 4, the four system faults

General discussion

The aim of our research was to investigate how the design of a job aid affects performance in process control. The design of job aids has – to our knowledge – not been previously investigated from a comparative perspective, with the exception of one study conducted in the field of medicine (Spaulding and Dwyer, 2001). In the study by Spaulding and Dwyer (2001), five different job aids were comparatively investigated in terms of their effect on knowledge acquisition in a medical task (a

References (41)

  • T. Van Gog et al.

    Effects of studying sequences of process-oriented and product-oriented work examples on troubleshooting transfer efficiency

    Learn. Instr.

    (2008)
  • J.R. Wilson

    Fundamentals of ergonomics in theory and practice

    Appl. Ergon.

    (2000)
  • J. Annett et al.

    Task analysis and training design

    J. Occup. Psychol.

    (1967)
  • D. Burkolter et al.

    Cognitive and personality variables of operators: individual characteristics and process control performance

  • D. Burkolter et al.

    Cognitive requirement analysis to derive training models for controlling complex systems

  • D. Burkolter et al.

    The predictive qualities of operator characteristics for process control performance: the influence of variables relating to cognition and personality

    Ergonomics

    (2009)
  • P. Chandler et al.

    Cognitive load theory and the format of instruction

    Cognition Instruct.

    (1991)
  • V. Dupont et al.

    Learning from technical documents: the role of intermodal referring expression

    Hum. Factors

    (2006)
  • T.K. Elliot et al.

    An Experimental Comparison of Procedural and Conventional Electronic Troubleshooting (AFHRL-TR-68–1)

    (1968)
  • T.K. Elliot et al.

    An experimental evaluation of a method for simplifying electronic maintenance

    Hum. Factors

    (1971)
  • Cited by (18)

    • A two-part evaluation approach for measuring the usability and user experience of an Augmented Reality-based assistance system to support the temporal coordination of spatially dispersed teams

      2021, Cognitive Systems Research
      Citation Excerpt :

      168). Common examples for scripted coordination in the industrial sector are Standard Operating Procedures (SOPs), procedural job aids (Kluge, Grauel, & Burkolter, 2013), or schedules (Bardram, 2000). Basically, these artefacts represent action scripts that divide processes into subtasks, order them in the correct sequence, indicate start and end points, and allocate subtasks to the corresponding team member.

    • Can cued recall by means of gaze guiding replace refresher training? An experimental study addressing complex cognitive skill retrieval

      2018, International Journal of Industrial Ergonomics
      Citation Excerpt :

      Research on memory and retrieval shows that cued recall is less difficult than free recall (Tulving and Thompson, 1973; Watkins and Tulving, 1975). Job aids constitute a method for applying cued recall for complex cognitive skills, and are used to process, store and extend information (Kluge et al., 2013; Swezey, 1987). Job aids support skills that are used infrequently, consist of sequences, and/or contain a great amount of information, or skills that can cause serious consequences if performed incorrectly (Rossett and Gautier-Downes, 1991).

    • Human-centered design (HCD) of a fault-finding application for mobile devices and its impact on the reduction of time in fault diagnosis in the manufacturing industry

      2017, Applied Ergonomics
      Citation Excerpt :

      For the proceduralization of the testing sequence, training is important, as improvements in problem-solving accuracy may only be achieved through extended practice of sequences and in particular through the availability of information about the quality of moves during problem solving (Duncan, 1985). If training time is limited, the use and application of a fault-finding strategy can be enhanced by traditional and dynamic job aids (Kluge et al., 2013; Kluge, Greve, Borisov & Weyers, in press). However, the advantage of job aids depends on the quality of the guidance provided, which in turn depends on the identification of the relevant information required for successful fault finding.

    • Engineering psychology and human performance

      2021, Engineering Psychology and Human Performance
    View all citing articles on Scopus
    View full text