The international cognitive ability resource: Development and initial validation of a public-domain measure
Introduction
The domain of cognitive ability assessment is now populated with dozens, possibly hundreds, of proprietary measures (Camara et al., 2000, Carroll, 1993, Cattell, 1943, Eliot and Smith, 1983, Goldstein and Beers, 2004, Murphy et al., 2011). While many of these are no longer maintained or administered, the variety of tests in active use remains quite broad, providing those who want to assess cognitive abilities with a large menu of options. In spite of this diversity, however, assessment challenges persist for researchers attempting to evaluate the structure and correlates of cognitive ability. We argue that it is possible to address these challenges through the use of well-established test development techniques and report on the development and validation of an item pool which demonstrates the utility of a public-domain measure of cognitive ability for basic intelligence research. We conclude by imploring other researchers to contribute to the on-going development, aggregation and maintenance of many more item types as part of a broader, public-domain tool — the International Cognitive Ability Resource (“ICAR”).
Section snippets
The case for a public domain measure
To be clear, the science of intelligence has historically been well-served by commercial measures. Royalty income streams (or their prospect) have encouraged the development of testing “products” and have funded their ongoing production, distribution and maintenance for decades. These assessments are broadly marketed for use in educational, counseling and industrial contexts and their administration and interpretation are a core service for many applied psychologists. Their proprietary nature
Study 1
We investigated the structural properties of the initial version of the International Cognitive Ability Resource based on internet administration to a large international sample. This investigation was based on 60 items representing four item types developed in various stages since 2006 (and does not include deprecated items or item types currently under development). We hypothesized that the factor structure would demonstrate four distinct but highly correlated factors, with each type of item
Study 2
Following the evidence for reliable variability in ICAR scores in Study 1, it was the goal of Study 2 to evaluate the validity of these scores when using the same administration procedures. While online administration protocols precluded validation against copyrighted commercial measures, it was possible to evaluate the extent to which ICAR scores correlated with (1) self-reported achievement test scores and (2) published rank orderings of mean scores by university major. In the latter case,
Study 3
The goal of the third study was to evaluate the construct validity of the ICAR items against a commercial measure of cognitive ability. Due to the copyrights associated with commercial measures, these analyses were based on administration to an offline sample of university students rather than an online administration.
General discussion
Reliability and validity data from these studies suggest that a public-domain measure of cognitive ability is a viable option. More specifically, they demonstrate that brief, un-proctored, and untimed administrations of items from the International Cognitive Ability Resource are moderately-to-strongly correlated with measures of cognitive ability and achievement. While this method of administration is inherently less precise and exhaustive than many traditional assessment methods, it offers
Conclusion
Public-domain measures of cognitive ability have considerable potential. We propose that the International Cognitive Ability Resource provides a viable foundation for collaborators who are interested in contributing extant or newly-developed public-domain tools. To the extent that these tools are well-suited for online administration, they will be particularly useful for large-scale cognitive ability assessment and/or use in research contexts beyond the confines of traditional testing
References (55)
- et al.
Proprietary versus public domain licensing of software and research products
Research Policy
(2006) - et al.
The International Personality Item Pool and the future of public-domain personality measures
Journal of Research in Personality
(2006) - et al.
ACT and general cognitive ability
Intelligence
(2008) - et al.
Development and initial validation of public domain basic interest markers
Journal of Vocational Behavior
(2008) Correction formulas for correlations restricted by selection on an unmeasured variable
Journal of Educational Measurement
(1990)- et al.
Automatic generation of quantitative reasoning items
Journal of Individual Differences
(2006) The basics of item response theory
(1985)- et al.
Correcting correlations for restrictions in range due to selection on an unmeasured variable
Educational and Psychological Measurement
(1972) - et al.
Psychological test usage: Implications in professional psychology
Professional Psychology: Research and Practice
(2000) Human cognitive abilities: A survey of factor-analytic studies
(1993)
Self-reported GPA and SAT: A methodological note
Practical Assessment, Research & Evaluation
The measurement of adult intelligence
Psychological Bulletin
Evaluation of Heckman selection model method for correcting estimates of HIV prevalence from sample surveys
Accuracy of self-reported SAT and ACT test scores: Implications for research
Research in Higher Education
2012 college-bound seniors total group profile report
Detecting and statistically correcting sample selection bias
Journal of Social Service Research
Approaches to modeling item-generative tests
Table of GRE scores by intended graduate major field
Manual for kit of factor-referenced cognitive tests
An international directory of spatial tests
The new rules of measurement
Psychological Assessment
Standardized tests: Recouping development costs and preserving integrity
Scholastic assessment or g? The relationship between the scholastic assessment test and general cognitive ability
Psychological Science
Further research on the accuracy of students' self-reported grade point averages, SAT scores, and course grades
Perceptual and Motor Skills
A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models
International Personality Item Pool: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences
Cited by (254)
Interactions between Machiavellianism and verbal reasoning in “bullshit” production
2024, Personality and Individual DifferencesTwo experiments on trading information goods in a network
2024, Games and Economic BehaviorYou better not tell me I'm not intelligent! Grandiose narcissism and reaction to negative intelligence feedback
2024, Personality and Individual DifferencesThe seductive beauty of latent variable models: Or why I don't believe in the Easter Bunny
2024, Personality and Individual DifferencesPsychological pillars of support for free speech: Tolerance for offensive, disagreeing, socially divisive, and heterodox speech
2024, Personality and Individual DifferencesAre we as intelligent as we think? Adaptation study of ICAR-16 into Turkish
2023, Personality and Individual Differences
- 1
With thanks to Melissa Mitchell.