General recognition theory with individual differences: a new method for examining perceptual and decisional interactions with an application to face perception

Abstract

A common question in perceptual science is to what extent different stimulus dimensions are processed independently. General recognition theory (GRT) offers a formal framework via which different notions of independence can be defined and tested rigorously, while also dissociating perceptual from decisional factors. This article presents a new GRT model that overcomes several shortcomings with previous approaches, including a clearer separation between perceptual and decisional processes and a more complete description of such processes. The model assumes that different individuals share similar perceptual representations, but vary in their attention to dimensions and in the decisional strategies they use. We apply the model to the analysis of interactions between identity and emotional expression during face recognition. The results of previous research aimed at this problem have been disparate. Participants identified four faces, which resulted from the combination of two identities and two expressions. An analysis using the new GRT model showed a complex pattern of dimensional interactions. The perception of emotional expression was not affected by changes in identity, but the perception of identity was affected by changes in emotional expression. There were violations of decisional separability of expression from identity and of identity from expression, with the former being more consistent across participants than the latter. One explanation for the disparate results in the literature is that decisional strategies may have varied across studies and influenced the results of tests of perceptual interactions, as previous studies lacked the ability to dissociate between perceptual and decisional interactions.

Appendix

Here we prove that the problem of non-identifiability of decisional separability described by Silbert and Thomas (2013) occurs in GRT-wIND only in the special case in which the decision bounds of all participants for each dimension are parallel to each other. We also describe procedures to: (1) estimate the parameters of a GRT-wIND model from identification data using maximum likelihood estimation, (2) run statistical tests for perceptual independence, perceptual separability and decisional separability, and (3) estimate parameters and test different types of independence as in previous applications of GRT.

Identifiability of decisional separability in the 2 × 2 GRT-wIND model

Silbert and Thomas (2013) showed analytically that a failure of decisional separability is non-identifiable in the Gaussian GRT model for a 2 × 2 identification experiment. That is, if the data from an experiment can be fit by a GRT model in which decisional separability fails, then it is always possible to find a different GRT model in which decisional separability holds and that predicts the exact same data pattern. We call this result the Silbert-Thomas non-identifiability, or STn for short.

We start by summarizing the proof offered by Silbert and Thomas (2013). Their theorem states that “Any perceptually separable but decisionally nonseparable configuration can be transformed to a configuration that is perceptually nonseparable, decisionally separable, and equivalent with respect to predicted response probabilities” (p. 17). Thus, they focus on the case in which the original configuration exhibits perceptual separability but violations of decisional separability. However, the more general result is that decisional separability is nonidentifiable in this model. As the authors indicate, “Any arbitrary (and, in general, not perceptually separable) linear bound model without decisional separability can be rotated and sheared to produce a model with decisional separability […], failure of decisional separability is never identifiable in this model” (pp. 4–5).

The proof for this theorem starts with a configuration without decisional separability and that has been translated so that the origin of the xy-plane coincides with the intersection of the two decision bounds h _A and h _B. The angle between h _B and the x-axis is represented by ϕ and the angle between the bounds h _A and h _B is represented by ω. Decisional separability holds when ϕ = 0 and ω = π/2. Rotation of the original configuration by ϕ degrees brings h _B to be parallel to the x-axis (and orthogonal to the y-axis), achieving decisional separability of component B from A. The horizontal shear transformation has the property of changing the angle between all lines in the plane except those parallel to the x-axis. Thus, for any value of ω, a horizontal shear transformation can be found that brings this angle to π/2 while keeping h _B parallel to the x-axis, thus achieving decisional separability of component A from B while also keeping decisional separability of component B from A.

The rotation and shear transformations can be represented by the transformation matrices L ₁ and L ₂, respectively, which combine to produce:

$$ \mathbf{\mathsf{L}}={\mathbf{\mathsf{L}}}_2{\mathbf{\mathsf{L}}}_1=\left[\begin{array}{cc}\hfill 1\hfill & \hfill -\frac{1}{\mathit{\mathsf{tan}}\omega}\hfill \\ {}\hfill 0\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{cc}\hfill \mathit{\mathsf{cos}}\varphi \hfill & \hfill -\mathit{\mathsf{sin}}\varphi \hfill \\ {}\hfill \mathit{\mathsf{sin}}\varphi \hfill & \hfill \mathit{\mathsf{cos}}\varphi \hfill \end{array}\right]=\left[\begin{array}{cc}\hfill \mathit{\mathsf{cos}}\varphi -\frac{\mathit{\mathsf{sin}}\varphi }{\mathit{\mathsf{tan}}\omega}\hfill & \hfill -\mathit{\mathsf{sin}}\varphi -\frac{\mathit{\mathsf{cos}}\varphi }{\mathit{\mathsf{tan}}\omega}\hfill \\ {}\hfill \mathit{\mathsf{sin}}\varphi \hfill & \hfill \mathit{\mathsf{cos}}\varphi \hfill \end{array}\right] $$

(A1)

This is an area-preserving affine transformation. The change-of-variables theorems for densities guarantees that probabilities will be preserved under such transformation (Billingsley 2012). This means that the predicted probabilities of correct responses in the original configuration and the decisionally-separable configuration are the same, as the values of the integrals involved do not change. The means and covariance matrices in the decisionally-separable configuration can be computed from the original means and covariance matrices by using the formulas:

$$ {\underset{\bar{\mkern6mu}}{\mu}}_{\mathit{\mathsf{T}}}=\mathbf{\mathsf{L}}\underset{\bar{\mkern6mu}}{\mu } $$

(A2)

$$ {\boldsymbol{\Sigma}}_{\mathit{\mathsf{T}}}=\mathbf{\mathsf{L}}\boldsymbol{\Sigma} {\mathbf{\mathsf{L}}}^{\mathsf{T}} $$

(A3)

In the remainder of this section, we show the conditions under which decisional separability is non-identifiable in GRT models with more than one bound per dimension. GRT-wIND and n x m GRT models with n > 2 and m > 2 are special cases of this general class. We start by identifying the conditions under which STn holds for a model with two bounds per dimension. It is then straightforward to see that the same conditions apply for any larger number of bounds per dimension.

Theorem

In a Gaussian GRT model with two dimensions and two linear bounds per dimension, where the ith bound for dimension A is represented as h _Ai and the jth bound for dimension B as h _Bj , the non-identifiability of decisional separability identified by Silbert and Thomas (2013) is true if and only if h _A1 ║ h _A2 and h _B1 ║ h _B2.

Proof

We first prove that if h _A1 ║ h _A2 and h _B1 ║ h _B2, then STn holds. As with the proof of STn, we start with a configuration without decisional separability that has been translated so that the origin of the xy-plane coincides with the intersection of h _A1 and h _B1. We represent the angle between h _Bj and the x-axis as ϕ _j and the angle between of h _Ai and h _Bj as ω _ij . Because h _B1 and h _B2 are parallel to each other, but not parallel to the x-axis, they intersect the latter at congruent angles; that is, ϕ ₁ = ϕ ₂ . Thus, rotation of the original configuration by ϕ ₁ degrees brings both h _B1 and h _B2 to be parallel to the x-axis and orthogonal to the y-axis, achieving decisional separability of component B from A. After rotation, it is still true that h _A1 ║ h _A2 and h _B1 ║ h _B2, because rotation preserves parallelism. This means that ω _ij = ω for all i and j. Thus, a single shear tranformation can bring this angle to π/2, achieving decisional separability of component A from B while also keeping decisional separability of component B from A.

To complete the proof, we must show that if STn holds, then h _A1 ║ h _A2 and h _B1 ║ h _B2. For STn to hold, a decisionally-separable configuration must exist that can be found by applying an affine transformation L to an original configuration without decisional separability. By definition, in this decisionally-separable configuration h _A1 ⊥ x, h _A2 ⊥ x, h _B1 ⊥ y and h _B2 ⊥ y. Because two lines that are both perpendicular to a third line are parallel to each other, with all lines in the same plane, h _A1 ║ h _A2 and h _B1 ║ h _B2 in the decisionally separable configuration. To go from the decisionally separable configuration to the original configuration, we must apply the transformation L ⁻¹. This inverse transformation exists because both shear and rotation are invertible transformations. The inverse of an affine transformation is itself an affine transformation that conserves parallelism, so application of L ⁻¹ to the decisionally-separable transformation conserves the property that h _A1 ║ h _A2 and h _B1 ║ h _B2. Thus, if STn holds, then bounds must be parallel in the decisionally separable configuration as well as in the original configuration.

This completes the proof for the case in which there are two linear bounds per dimension. A corollary is that for models with more than two bounds per dimension, STn holds if and only if each bound in one dimension is parallel to each of the other bounds in that specific dimension.

Here we have exclusively dealt with part (i) of the theorem proposed by Silbert and Thomas (2013). Part (ii) of this theorem proposes that a configuration with mean shift integrality and decisional separability is unidentifiable from a configuration with perceptual separability and without decisional separability. This theorem also deals with the non-identifiability of decisional separability, so as before it only holds for models with more than one bound per dimension if those bounds are parallel. Furthermore, an additional condition for this theorem to hold is that all covariance matrices in the model must be identical (Thomas and Silbert 2014). This in general is not the case in GRT-wIND or in traditional GRT models for designs larger than 2 × 2, which allow for estimation of different variances and covariances for each perceptual distribution.

In conclusion, STn is not generally true in GRT-wIND or any other model with more than one bound per dimension. The non-identifiability of decisional separability arises in such models only under very specific circumstances.

Maximum likelihood estimation for GRT-wIND

The data from each participant in an identification experiment are summarized in a confusion matrix, with rows corresponding to each stimulus in the experiment, columns corresponding to each response, and response frequencies reported in each cell of the matrix. Let S ₁ , S ₂ , …, S _n denote the n stimuli in an identification experiment and let R ₁ , R ₂ , …, R _n denote the n responses. Let r _ij denote the frequency with which the participant responded R _j on trials when stimulus S _i was presented. Finally, there are N participants in the experiment, indexed by k = 1, 2,…,N. Given a set of parameter values for the model, the likelihood of this confusion data is computed in two steps.

In the first step, the predicted confusion matrix of each participant is computed using standard methods. For example, the predicted probability that a participant responds R _j on trials when stimulus S _i was presented, denoted by P(R _j |S _i ), is computed by integrating the volume of the S _i perceptual distribution in response region R _j . A numerical approximation to this multiple integral can be computed efficiently using Cholesky factorization (Ennis and Ashby 2003; for a tutorial overview, see Ashby and Soto 2014).

The second step is to compute the log of the likelihood function for participant k:

$$ \log {L}_k={\displaystyle \sum_{i=1}^n{\displaystyle \sum_j^n{r}_{ij} \log P\left(\left.{R}_j\right|{S}_i\right)}} $$

(A4)

These log-likelihoods are then summed across all participants:

$$ \log L={\displaystyle \sum_{k=1}^N \log {L}_k} $$

(A5)

The maximum likelihood estimates of the parameters in a GRT-wIND model are those that maximize the expression in Equation A5.

Statistical tests of independence with GRT-wIND

The large number of parameters in a GRT-wIND model makes the computational cost of using likelihood ratio tests and model selection procedures prohibitive. Thus, we recommend a deviation from the custom of computing such tests in GRT analyses. The strategy used here consists of fitting the full GRT-wIND model and testing maximum-likelihood parameter estimates against expected values from null hypotheses using a Wald test (Wald 1943).

Let $\underset{\bar{\mkern6mu}}{\widehat{\theta}}$ be a column vector containing the maximum likelihood parameter estimates. The Wald test can be used to test any null hypothesis that can be expressed in the form of linear restrictions on $\underset{\bar{\mkern6mu}}{\widehat{\theta}}$:

$$ \begin{array}{c}\hfill {H}_0:\mathbf{R}\underset{\bar{\mkern6mu}}{\widehat{\theta}}-q=0\hfill \\ {}\hfill {H}_1:\mathbf{R}\underset{\bar{\mkern6mu}}{\widehat{\theta}}-q\ne 0\hfill \end{array} $$

where R is a matrix with number of columns equal to the number of parameters and number of rows equal to the number of restrictions being tested, and $\underset{\bar{\mkern6mu}}{q}$ is a column vector with number of rows equal to the number of restrictions being tested. For example, if we wanted to test the hypothesis that ${\widehat{\theta}}_1=0$, then R would have a single row (we are testing a single restriction) with a +1 in the first cell of that row and zeros in all other cells, while $\underset{\bar{\mkern6mu}}{q}$ would have a single cell with a zero in it. If we want to additionally test the hypothesis that ${\widehat{\theta}}_2-{\widehat{\theta}}_3=10$, then we would add a second row to R with a +1 in the second column (corresponding to $+{\widehat{\theta}}_2$) and –1 in the third column (corresponding to $-{\widehat{\theta}}_3$), while $\underset{\bar{\mkern6mu}}{q}$ would now have a second cell with the value 10 in it.

Null hypotheses are tested using the Wald statistic:

$$ W={\left[\mathbf{R}\underset{\bar{\mkern6mu}}{\widehat{\theta}}-\underset{\bar{\mkern6mu}}{q}\right]}^{\mathrm{T}}{\left[\mathbf{R}{\Sigma}_{\underset{\bar{\mkern6mu}}{\widehat{\theta}}}-{\mathbf{R}}^{\mathrm{T}}\right]}^{-1}\left[\mathbf{R}\underset{\bar{\mkern6mu}}{\widehat{\theta}}-\underset{\bar{\mkern6mu}}{q}\right] $$

(A6)

where []^T represents matrix transpose. The statistic W has a chi-squared distribution with degrees of freedom equal to the number of restrictions being tested (the length of $\underset{\bar{\mkern6mu}}{q}$). Computing W requires the covariance matrix of the maximum likelihood estimates, which can be estimated using the Hessian of the log-likelihood function at the solution:

$$ {\boldsymbol{\Sigma}}_{\underset{\bar{\mkern6mu}}{\widehat{\theta}}}=H{\left(\underset{\bar{\mkern6mu}}{\widehat{\theta}}\right)}^{-1} $$

(A7)

Usually the Hessian in Eq. A4 can be obtained from the same optimization software that is used to obtain the parameter estimates that maximize the log-likelihood, but better estimates are obtained from numerical differentiation software. In this study, we used the DERIVEST suite (D’Errico 2006) to obtain estimates of the Hessian.

For the 2 × 2 identification design used here, the restrictions imposed on the model by perceptual separability of dimension A from dimension B are the following:

$$ \begin{array}{ccc}\hfill {\mu}_{A_1{B}_21}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\sigma}_{A_1{B}_21}\hfill & \hfill =\hfill & \hfill 1\hfill \\ {}\hfill {\mu}_{A_2{B}_11}-{\mu}_{A_2{B}_21}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\sigma}_{A_2{B}_11}-{\sigma}_{A_2{B}_11}\hfill & \hfill =\hfill & \hfill 0\hfill \end{array} $$

The restrictions imposed in the model by perceptual separability of dimension B from dimension A are the following:

$$ \begin{array}{ccc}\hfill {\mu}_{A_2{B}_12}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\sigma}_{A_2{B}_12}\hfill & \hfill =\hfill & \hfill 1\hfill \\ {}\hfill {\mu}_{A_1{B}_22}-{\mu}_{A_2{B}_22}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\sigma}_{A_1{B}_22}-{\sigma}_{A_2{B}_22s}\hfill & \hfill =\hfill & \hfill 0\hfill \end{array} $$

The restrictions imposed in the model by perceptual independence in each of the perceptual distributions are the following:

$$ \begin{array}{ccc}\hfill {\rho}_{A_1{B}_1}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\rho}_{A_1{B}_2}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\rho}_{A_2{B}_1}\hfill & \hfill =\hfill & \hfill 0\hfill \\ {}\hfill {\rho}_{A_2{B}_2}\hfill & \hfill =\hfill & \hfill 0\hfill \end{array} $$

The Wald test allows tests of decisional separability for the whole group or for each participant individually. Here, we focus on the latter kind of test. Testing whether decisional separability of dimension A from dimension B holds in participant k involves a single restriction:

$$ {b}_{A_{k2}}=0 $$

Testing whether decisional separability of dimension B from dimension A holds in participant k involves the following restriction:

$$ {b}_{A_{k1}}=0 $$

Model fit and selection in the traditional GRT approach

To fit any GRT model to data (e.g., the models in the hierarchy shown in Fig. 5), the confusion matrix from a single participant is used to find the values of the free parameters that maximize Eq. A5.

A popular method to test assumptions about independence and separability is to fit a restricted and an unrestricted version of the model to data. The restricted model contains a number of parameters that are set to values reflecting the assumption under test. For example, testing perceptual independence would require setting all ρ parameters to zero. The same parameters would be free to vary in the unrestricted model. Once both models are fit to data, the likelihood of the data at the solutions (L _U and L _R for the unrestricted and unrestricted versions, respectively) can be used to run a likelihood ratio test, by computing the following statistic:

$$ \varDelta =-2\left( \log {L}_R- \log {L}_U\right), $$

(A8)

which follows a Chi-squared distribution with degrees of freedom equal to the difference in number of free parameters between the two models.

The likelihood ratio test can only be applied to select between two nested models. To select between two non-nested models, it is possible to use the Akaike information criterion (AIC, Akaike 1974) for model comparison. Here we use a version of AIC corrected for a bias problem present when the number of data points is small compared to the number of free parameters (see Burnham and Anderson 2004):

$$ AI{C}_C=\hbox{--} 2 \log L+2m+2m\left(m+1\right)/\left({n}^2\hbox{--} m\hbox{--} 1\right), $$

(A7)

where m is the number of free parameters in the model and n ² is the number of cells in the confusion matrix. The first two terms in Eq. A7 correspond to the traditional definition of AIC and the last term corresponds to the correction factor. A smaller value of AIC represents a better fit of the model to the data.

In the present study, as in previous model-based applications of GRT (e.g., Ashby and Lee 1991; Ashby et al. 2001; Fitousi and Wenger 2013 Thomas, 2001), a hierarchy of models was fit to the data from each participant (see Fig. 5). The procedure starts at the top of the hierarchy and compares nested models through likelihood ratio tests until the test results in a non-significant increase in fit. If more than one candidate model survives this process, the model with the smallest AIC_C is selected.