The predictive validity of a computer-assisted career decision-making system: A six-year follow-up

Abstract

The present study tested the predictive validity of Making Better Career Decisions (MBCD), a computer-assisted career decision-making system. Seventy clients who had used MBCD six years earlier were interviewed by telephone about their present field of study or occupation and the degree of their satisfaction with their occupational choice. The participants were divided into two groups: one consisted of clients whose chosen occupation was on the list of occupations recommended by MBCD, whereas the other consisted of clients whose chosen occupation was not on the recommended list. In the first group, 84% were highly satisfied with their occupational choice and 16% were satisfied to a moderate extent. Among those whose chosen occupation was not on the recommended list, only 38% were highly satisfied with their chosen occupation, 44% were satisfied to a moderate extent, and 18% were dissatisfied with their choice. These findings support the predictive validity of MBCD. Implications for counseling and computer-assisted career guidance systems are discussed.

Introduction

Career decisions are among the most important decisions individuals make. Whereas some individuals who face difficulties in making career-decisions seek individual career-counseling, others use self-help tools such as the Self-Directed Search (Holland, Powell, & Fritzsche, 1994) or computer-based career information and guidance systems (e.g., CIS, SIGI Plus, DISCOVER, CHOICES Planner). In the present study, we focused on evaluating the predictive validity of the recommended list of occupations presented to clients by MBCD—a decision-theory-based computer-assisted career-guidance system.

Making Better Career Decisions. Making Better Career Decisions (MBCD) (www.kivunim.com/mbcd, retrieved July 22, 2005) is a computer-based self-help career decision-making system, based on the Prescreening, In-depth exploration, and Choice model (PIC; Gati & Asher, 2001a). MBCD is a unique combination of an expert system (Jenicke, 1988, Ozernoy, 1988) and a decision-support system (King, 1990), and represents the fifth generation of computer-assisted career guidance systems (CACGSs, Carson & Cartwright, 1997)—Internet-based Interactive Career Planning Systems (Gati, Kleiman, Saka, & Zakai, 2003). It facilitates the prescreening stage by helping clients locate promising alternatives that are worth in-depth exploration, based on their preferences in the aspects they regard as more important (Gati & Asher, 2001a). Then, the brief and extended occupational description makes it possible to begin the in-depth exploration stage. A schematic outline of the stages of the dialogue with MBCD and its components is presented in Fig. 1; more detailed information on MBCD is presented in Section 2.

One of the important outputs of the dialogue with MBCD is the list of occupations recommended for further exploration, based on the individual’s preferences. MBCD uses the sequential-elimination-based search (Gati, 1986, Gati and Asher, 2001a) as the default method for prescreening. During this search, users are first asked to prioritize the aspects according to their relative importance for them. Then users report their preferences in each aspect (in terms of optimal and acceptable levels), starting with the aspect ranked as most important. Occupations that do not suit the individuals’ preferences are eliminated from the list of promising alternatives. This process is repeated until the list of promising alternatives is reduced to about 7. The occupations on this list are those recommended for further exploration.

After receiving the list of promising alternatives, the clients can use also an optional compensatory-model based search to get an additional list of promising occupations. This model is based on the assumption that the advantages of an alternative can compensate for its disadvantages (Katz, 1966, Katz, 1993, Pitz and Harren, 1980). In the compensatory-model-based search, the client’s preferences in each aspect are compared to the respective characteristics of the occupations; the gaps, if any, are weighted according to the importance of the aspect to the individual. Thus, the disadvantages of an occupation (as reflected in the gaps between the individual’s preferences and the characteristics of the occupation in certain aspects) are compensated for by its advantages—the match between preferences and the given occupation’s characteristics in other aspects. Occupations that best suit the individual’s preferences (i.e., those with the minimal sum of gaps) are placed on the list of occupations recommended for further exploration in this model (Gati and Asher, 2001b, Katz, 1966). Typically, for those who use both search methods, some occupations are on the conjunction list, whereas other occupations appear on only one of the lists.

Testing the validity of MBCD. Previous studies focused on the concurrent validity and perceived effectiveness of MBCD. Gati, Saka, and Krausz (2001) found that most users reported progress in the career decision-making process, and that using MBCD contributed to a decrease in the career decision-making difficulties related to lack of information, as measured by the CDDQ (Gati, Krausz, & Osipow, 1996). Gati et al. (2003) focused on the perceived benefits of using the system (measured directly as the perceived benefit and indirectly as the willingness to recommend the use of MBCD to a friend). The present study was aimed at testing the predictive validity of the MBCD.

Testing the predictive validity of career-counseling instruments. Testing the predictive validity of career-counseling instruments, such as vocational interest inventories, is not only expensive but also challenging, mainly because of the difficulties involved in locating clients several years after they used the instrument. Consequently, follow-up studies of career-counseling outcomes are rare. There are two types of follow-up studies of the predictive validity of career-counseling instruments. The first group consists of studies in which the clients’ present occupations are compared to their vocational interests as reflected in their responses to the interest inventory in the past (e.g., Holland et al., 1994, Spokane, 1979, Zytowski, 1976, Zytowski and Laing, 1978). Studies of this type found that 40–60% of the participants were working in occupations compatible with their interest-inventory results (Fouad, 1999). The second group of studies are follow-up studies that show that clients whose occupational choice was compatible with the results of an interest inventory were more satisfied with their occupational choice than those whose choice was incompatible with their interest-inventory results (e.g., Barak and Meir, 1974, Bartling and Hood, 1981, Campbell, 1971, Dolliver et al., 1972, Schlezer, 1966). For example, Barak and Meir (1974) found moderate positive correlations (.29 and .40, for females and males, respectively) between the relevant RAMAK interest inventory field score (based on Roe’s occupational classification, Roe, 1956) and people’s satisfaction with choosing an occupation from that field after seven years. The advantage of occupational choice satisfaction as a criterion for predictive validity (used in the second group of studies) is that some individuals remain in their current job due to external constraints even if they are not satisfied with it; thus, simply working in the recommended field is not a sufficient criterion for the validity of the instrument.

So far, no studies have been carried out to test the predictive validity of the aspect approach for career decision-making and, in particular, the predictive validity of MBCD, which is based on this approach. Thus, the first goal of the present study was to test the predictive validity of the sequential elimination-based search, which is the core component of MBCD. As in previous research (e.g., Bartling and Hood, 1981, Kuder, 1970, Strong, 1943), the criterion used to evaluate the validity of the recommended list was the clients’ satisfaction with their occupational choice. To achieve this goal, we located clients who had used MBCD in the past, and asked them to report their chosen occupation and the degree of their satisfaction with it. We then divided these clients into two groups according to whether their occupational choice was or was not on the recommended list of occupations given them during their dialogue with MBCD six years ago. We hypothesized that clients’ degree of satisfaction with their occupational choice would be higher for those whose chosen occupation was on the list of occupations recommended by MBCD six years before than for those whose chosen occupation was not on the recommended list.

The second goal was to test the predictive validity of the recommended list of occupations based on the compensatory model. To achieve this goal, we compared the degree of occupational choice satisfaction of four groups of clients, divided into the groups according to the list of occupations their chosen occupation was on: (a) the sequential-elimination-based search list only, (b) the compensatory-model-based list only, (c) the conjunction list, which consists of those occupations on both previous lists, and (d) neither list. In addition, we hypothesized that clients whose chosen occupation was on the sequential-elimination search list would report greater occupational choice satisfaction than clients whose chosen occupation was on the compensatory search list. This hypothesis was based on the assumption that no advantage can fully compensate for a significant disadvantage (i.e., when the occupation has an unacceptable characteristic). Thus, an individual will probably be dissatisfied even with an occupation whose overall match is high if it has a significant disadvantage (e.g., working in shifts). Finally, we hypothesized that clients whose present occupation was on the lists generated by both search models would report the greatest satisfaction (i.e., greater than those whose chosen occupation was on the list generated by the sequential-elimination search but not on the list generated by the compensatory search). The rationale for this hypothesis is that the presence of an occupation on both lists reflects not only a fit in the most important aspects, but also in the less important ones.