Abstracts

ORIGINAL ARTICLE

Reliability of the Social Phobia and Anxiety Inventory (SPAI) Portuguese version in a heterogeneous sample of Brazilian university students

Fidedignidade da versão em português do Inventário de Ansiedade e Fobia Social (SPAI) em amostra heterogênea de estudantes universitários brasileiros

Patrícia Picon^I,II; Gabriel J C Gauer^II; Vânia N Hirakata^III; Lourdes M Haggsträm^II; Deborah C Beidel^IV; Samuel M Turner^IV; Gisele G Manfro^I

^IPost-graduate Program in Medical Sciences: Psychiatry - Universidade Federal do Rio Grande do Sul (Federal University of Rio Grande do Sul, UFRGS), Porto Alegre, RS, Brazil

^IIDepartment of Psychiatry and Legal Medicine of the Pontifícia Universidade Católica do Rio Grande do Sul (Pontifical Catholic University of Rio Grande do Sul, PUCRS). Porto Alegre, RS, Brazil

^IIIResearch and Post-graduate Group of the Hospital de Clínicas de Porto Alegre, RS, Brazil

^IVDepartment of Psychology, University of Maryland. Baltimore, MD, USA.

^{Correspondence} Correspondence to Patrícia Picon Rua Padre Chagas 415/803 90570-080 Porto Alegre, RS, Brasil Phone: (55 51) 3346-9022 E-mail: ppicon@terra.com.br

ABSTRACT

OBJECTIVE: To evaluate internal consistency and temporal stability trough test re-test reliability of the Portuguese version of the Social Phobia and Anxiety Inventory (SPAI) in a heterogeneous Brazilian sample.

METHODS: After informed consent, the translated and adapted version of the SPAI Portuguese was tested and re-tested at a 14-day interval in a sample of 225 university students, both genders, from 4 different undergraduate courses. After double data entry, statistical analysis included Cronbach's alphas, Pearson and intra-class correlation coefficients.

RESULTS: The studied sample consisted of 213 students from 4 sub-samples: 95 law students, 31 social communication students, 54 civil engineering students and 33 dental medicine students. The mean age was 23 (±6) years and 110 (51.6%) were female. The Cronbach' alpha was 0.96, no difference for both genders. The differential (total) score of the SPAI Portuguese showed Pearson coefficient 0.83 (95% CI 0.78-0.87) and Intra-class coefficient 0.83 (95% CI 0.78-0.86). There was no statistical difference in Pearson coefficients between genders (p = 0.121) and the 4 sub-samples (p = 0.258).

CONCLUSION: The results do not ensure validity, however, the SPAI Portuguese version presented a good content homogeneity with satisfactory level of internal consistency. The temporal stability evaluated was substantial. The results showed that the Portuguese SPAI presents perfectly acceptable reliability for both genders and suggest that it can be used in Brazilian populations

Keywords: Phobic disorders/diagnosis; Reproducibility of results; Self assessment (Psychology); Anxiety disorders/diagnosis; Behavior therapy/methods

RESUMO

OBJETIVO: Avaliar consistência interna e estabilidade temporal, através de fidedignidade teste-reteste, da versão em português do Inventário de Ansiedade e Fobia Social (SPAI), em uma amostra brasileira heterogênea.

MÉTODOS: Após consentimento informado, a versão traduzida e adaptada para o português do SPAI foi testada e re-testada, com 14 dias de intervalo, em uma amostra de 225 estudantes universitários de ambos os gêneros, de quatro cursos de graduação. Após dupla entrada de dados, a análise estatística incluiu Alfa de Cronbach e os coeficientes de correlação de Pearson e Intra-classe.

RESULTADOS: A amostra estudada consistiu de 213 estudantes de quatro sub-amostras: 95 estudantes de Direito, 31 de Comunicação Social, 54 de Engenharia Civil e 33 de Odontologia. A média de idade foi de 23 anos (± 6) e 110 (51,6%) eram do gênero feminino. O Alpha de Cronbach foi de 0,96, sem diferenças entre os gêneros. O escore diferencial (total) do SPAI português apresentou coeficiente de Pearson de 0,83 (IC95% 0,78-0,87) e coeficiente Intra-classe de 0,83 (IC95% 0,78-0,86). Não houve diferença estatística nos coeficientes de Pearson entre os gêneros (p = 0,121) ou entre as quatro sub-amostras (p = 0,258).

CONCLUSÃO: Os resultados não asseguram validade; a versão do SPAI português apresentou boa homogeneidade de conteúdo com nível satisfatório consistência interna. A estabilidade temporal avaliada foi consistente. Os dados demonstram que o SPAI português apresenta fidedignidade perfeitamente aceitável para ambos os gêneros e sugerem sua utilização na população brasileira.

Descritores: Transtornos fóbicos/diagnóstico; Reprodutibilidade de resultados; Auto-avaliação (Psicologia); Transtornos da ansiedade/diagnóstico; Terapia comportamental/métodos

Introduction

Social anxiety disorder or social phobia is a relatively recent diagnostic category.¹ It presents lifetime prevalence rates ranging from 7 to 16% in epidemiological studies of the American and European population.² It affects very young individuals with a mean age of onset between 15 and 18 years, a chronic course and high comorbidity.^3-5 However, although this disorder responds to currently available pharmacological and psychotherapeutic treatments,^6-10 only 25% of social phobics receive appropriate treatment.¹¹ Early psychiatric diagnosis depends on a high degree of suspicion. Screening for social anxiety disorder should be part of the routine in any psychiatric evaluation.¹² The detection of probable cases of social anxiety disorder using screening instruments and the evaluation of the intensity of social anxiety symptoms in a clinical and research environment requires valid and reliable instruments.^13-15

Although in psychiatric clinical work it is traditional to emphasize measures or assessments performed by clinical evaluators, in social anxiety disorder the self-reported measures are more widely used and more extensively validated. ^7-9 Several of these instruments are currently available, being specifically developed to evaluate social phobia, with acceptable psychometric properties, and have been tested in clinical samples. They have been used as screening instruments to identify probable cases and evaluate the severity of symptoms in therapeutic effectiveness studies. Outstanding among these scales is the Social Phobia and Anxiety Inventory (SPAI), the most extensive self-assessment instrument that presents an excellent psychometric profile and has demonstrated its capacity to evaluate treatment outcome. It was validated both in population and clinical samples, in adults and adolescents aged 14 years or over. Its format allows its use for therapeutic planning.¹⁶ The SPAI has a subscale on agoraphobia that enables the exclusion of cases with panic disorder and agoraphobia as a differential diagnosis.^3,17

Social Phobia and Anxiety Inventory

The Social Phobia and Anxiety Inventory (SPAI) was developed by Turner and co-workers in 1989 to screen for probable cases of Social Phobia, evaluate the severity of symptoms of social anxiety and therapeutic results. The SPAI inventory looks at physiological, behavioral and cognitive aspects of the dimensional construct of anxiety and social phobia.¹⁸

The SPAI inventory consists of 45 items that quantitatively evaluate the severity of the symptoms of anxiety and social avoidance using a 7-point Likert scale. It has two subscales: social phobia (items 1 to 32) and agoraphobia (items 33 to 45). The differential or total score of the SPAI is calculated by subtracting the agoraphobia score from the social phobia score. This differential (total) score of the SPAI is indicated as the most accurate to discriminate between cases and non-cases of probable social phobia in clinical and non-clinical studies, minimizing the number of false positives or negatives.¹⁸

The SPAI Inventory may then be used as a screening instrument, with good discriminatory performance and a cutoff point equal to or greater than 80 to identify probable cases of social phobia.¹⁹

The original English version was tested in population samples of American university students, and adolescents, and in clinical samples. SPAI presented excellent internal consistency and test-retest reliability in these studies. The factorial analyses showed the presence of two factors: social phobia and agoraphobia.^18,20 The construct's concurrent, convergent, external and discriminant validities of the SPAI Inventory have been demonstrated in clinical and non-clinical samples.^21-23 The SPAI Inventory appropriately discriminates patients with social phobia from other anxiety disorders and from normal controls.² Its sensitivity to measure therapeutic results has been proved.^24-26

SPAI has translations into at least 10 languages.² There is already a study to validate the Spanish version in a non-clinical sample where the results of internal consistency, test- retest reliability and factorial analysis were similar to those in the original English version.^27-28

The lack of instruments to screen and evaluate the intensity of anxiety or social phobia symptoms adapted to the Portuguese language and Brazilian culture, with appropriate measurement of psychometric properties, has been felt as a serious gap. The validation of the SPAI Portuguese version would make it possible to design studies to identify probable cases, evaluate the severity of symptoms and follow up therapeutic results in social phobia in Brazilian samples of adults and adolescents aged 14 years or over. These studies would be comparable to those of other countries, populations and cultures, and thus of interest to the Brazilian clinical and scientific community. Furthermore, the need of further research to broaden the SPAI validation process in non-American samples, has been already indicated in the literature, as a form of confirming and expanding its generalization.²⁹

The aims of this study were to evaluate internal consistency and temporal stability, using the test-retest reliability measure of the Portuguese version of the Social Phobia and Anxiety Inventory (SPAI)^21,30 in a heterogeneous population sample of Brazilian university students.

Methods

1. Sample

Using a cross-sectional design, 213 university students of both genders were evaluated and re-evaluated to the study of internal consistency and test-retest reliability. The study included fresh and senior students of 9 classes allotted from four different undergraduate courses. Three hundred and twenty-six individuals were assessed in the first evaluation aiming to assure minimal sample size of 200 participants in the second evaluation,³¹ taking into account that students might be absent or refuse to participate. During the second evaluation 94 were absent and there were 7 refusals. Thus 225 students participated in the re-test. Among the 225 participants who accepted to participate, 12 (5.3%) had problems to complete the SPAI, either at the first or the second evaluation, which made it impossible to calculate the (total) differential score.¹⁸ The final sample studied then consisted of 213 students from 4 sub-samples: 95 law students, 31 social communication students, 54 civil engineering students and 33 dental medicine students.

2. Procedures

After the formal consent of the authors and the publisher, Multi-Health Systems Incorporation, which holds the copyrights, the original Social Phobia Anxiety Inventory (SPAI) English version¹⁸ was translated into Portuguese by Picon and Gauer in 1999.³⁰

The face and content validity was performed by a translation and back-translation process, evaluation study with bilingual volunteers and evaluation of the Portuguese version by a panel of experts.³²

The SPAI Portuguese version also consisted of two subscales: social phobia and agoraphobia, with a total of 45 items, scored using a 7-point Likert scale, similar to the English version.³⁰

After the approval by the São Lucas Hospital's Ethics in Research Committee at Pontifical Catholic University of Rio Grande do Sul, data collection was approved by the directors of the college courses allotted for the implementation of this study. After the end of routine academic activities, and with the verbal consent of the teachers responsible for the classes studied, the students in the classroom, who agreed to participate in the project, having signed the informed consent, completed the SPAI Portuguese.

The SPAI Portuguese was tested and re-tested at a 14-day interval,³¹ to all participants in the sample studied. The students who participated in the temporal stability study received a code number to perform the re-test.

3. Statistical analysis

The data base was performed by independent double entry and later checked by the Validate module of EPI INFO version 6.04d (World Health Organization, Geneva). The statistical analyses were performed using SPSS version 10.0 (SPSS Inc., Chicago), STATA version 7.0 (STATA Corp., College Station), and PEPI version 3.0 (PEPI, Station Building, UK) software. The scores of social phobia, agoraphobia subscales and of the SPAI Portuguese differential (total) score were calculated using SPSS version 10.0 and tested as to their normality by Shaphiro-Wilk test.³³

The comparisons of age means and the scores of social phobia, agoraphobia and differential (total) SPAI Portuguese between the four courses were performed using one-way analysis of variance (ANOVA).

Internal consistency for the items of the subscales of social phobia and agoraphobia and for all items of the SPAI Portuguese was evaluated using Cronbach's alpha coefficient.³⁴ The test-retest correlation and concordance estimates for the social phobia and agoraphobia scores and for the differential (total) score of the SPAI Portuguese were evaluated using Pearson correlation coefficient and intra-class correlation coefficient with their respective 95% confidence intervals.^31,35

The comparisons of Pearson correlation coefficient between genders, and between the four different sub-samples of undergraduate courses were performed using the Chi-square heterogeneity test.³⁶

Results

In the total sample of 213 participants, the mean age was 23 ± 6 years. The minimum age was 17, the maximum 53, with a median of 22 years. One hundred and three students were male (48.4%) and 110 female (51.6%). There was no statistical difference (p = 0.832) in the mean age between males (23 ± 5.4 y) and females (23 ± 6.5 y).

The Shapiro-Wilk normality test revealed asymmetry in the social phobia and agoraphobia scores, which underwent square root transformation. The results of analyses with and without transformation were similar and the later are presented.

The means of the social phobia, agoraphobia and differential (total) SPAI Portuguese scores obtained at the first evaluation were not different between genders (Table 1) or between the four different sub-samples of undergraduate courses studied (Table 2). The total sample (n = 213) was therefore homogeneous in relation to the means of the scores that are being studied.

The Internal Consistency estimated using Cronbach's alpha in the total sample and in both genders were identical for the social phobia subscale (items 1 to 32; alpha = 0.96) and for the total scale SPAI Portuguese (items 1 to 45; alpha = 0.96). The agoraphobia subscale (items 33 to 45) presented alpha values of 0.87 in the total sample, 0.88 in the female group and 0.85 in the male group. The Cronbach's alpha coefficients of the two subscales and of the SPAI Portuguese were similar in the four sub-samples, except for the agoraphobia subscale, in which it presented the greatest variation (Table 3).

The test-retest reliability of social phobia, agoraphobia and differential (total) SPAI Portuguese scores, estimated using Pearson correlation coefficients in the four sub-samples (Table 3) in the total sample and in both genders (Table 4) varied from: 0.82 to 0.93 for the social phobia score; 0.72 to 0.85 for the agoraphobia score; and 0.79 to 0.90 for the differential (total) SPAI Portuguese score. The Pearson correlation coefficients did not present a significant difference in the comparison between the four sub-samples (p = 0.833) or between genders (p = 0.740) for the social phobia score. The Pearson correlation coefficients also did not present a significant difference in the comparison between the four sub-samples (p = 0.258), or between genders (p = 0.121) for the differential (total) SPAI Portuguese score. However, there was a significant difference in the comparison of the Pearson correlation coefficients between genders for the agoraphobia score (p = 0.029) that did not repeat in the comparison between the sub-samples (p = 0.713).

The test-retest reliability of the three scores studied, estimated using the intra-class correlation coefficient in the four sub-samples (Table 3), in the total sample and in both genders (Table 4) varied from: 0.81 to 0.92 for the social phobia score; 0.68 to 0.85 for the agoraphobia score; and 0.79 to 0.90 for the (total) differential SPAI Portuguese score.

Discussion

The present study suggests that, in the sample studied, the Portuguese version of the Social Phobia and Anxiety Inventory (SPAI) presented results equivalent to the original English version as to its psychometric properties of internal consistency and test-retest reliability.

The analyses performed took into account the distribution of the variables studied, as previously described. Only after careful analysis of the data with and without normalization of the social phobia and agoraphobia scores a decision was made to use the analyses without transformation. The results are compared to those obtained in American and Spanish population samples of young adults and adolescents.

Means of the social phobia and agoraphobia scores and differential (total) SPAI Portuguese score

The current total sample (n = 213) consisted mostly of young adults. The means of the social phobia scores and the (total) differential SPAI Portuguese score, in the four sub-samples, in both genders and in the total sample studied were greater then those found in population studies of Spanish adolescents and American university students and adolescents without social anxiety.^18,20,27 The means of the agoraphobia scores found in the current study were very similar to those found in the American studies.

In a study with American adolescents, in the group without social anxiety, the mean score of social phobia and agoraphobia, and of the differential (total) SPAI score were 59.7 ± 31.7, 17.3 ± 11.9 and 43.0 ± 26.4, respectively. These means were statistically lower than the means of the social phobic group.²⁰

The mean of the differential (total) SPAI score in a sample of American psychology students with social phobia (n = 58) was 72.2 ± 20.4 and among students without social anxiety (n = 124) it was 32.7 ± 21.3.²¹ In the group with social anxiety the mean of the social phobia score and of the differential (total) SPAI score were higher among male individuals. The mean agoraphobia score was not different between genders.^18,22

In the present study, the means of the three scores did not present differences between genders, as opposed to what was observed in the American as well as in the Spanish studies. In the American study with a smaller sample size, in the group without social anxiety, only the agoraphobia score was the same between genders, and the others were higher among males.²² In the Spanish study, in the total sample of 3160 adolescents from 14 to 17 years, girls had higher means in all three scores.²⁷

The differences in the means of social phobia and differential (total) SPAI Portuguese scores, and the differences of means between genders as compared to the American and Spanish samples may be attributed to: the Brazilian sample does not discriminate students with and without social anxiety, different sample sizes, and cultural differences in the way of expressing social anxiety and agoraphobia symptoms among genders.

Of note, similarities of the three mean scores in the four sub-samples and between genders in the total sample (n = 213) give us indirect evidence of homogeneity of the final sample, as regards to a set of symptom expression of social anxiety and agoraphobia in our culture, even between groups with supposedly distinct profiles of professional choices.

Internal consistency of social phobia and agoraphobia subscales and of SPAI Portuguese

In the present study the results of Cronbach's alpha coefficients for the subscales of social phobia and agoraphobia and the total SPAI Portuguese version showed internal levels of consistency that are satisfactory to excellent and were similar in the four sub-samples.^15,31 However, for the agoraphobia subscale Cronbrach's alpha coefficient presented a small variation. In the four sub-samples the values obtained for the social phobia and agoraphobia subscales and the total SPAI Portuguese version were very similar or slightly higher then those obtained in the American and Spanish studies.

In the American sample of psychology students with a mean age of 19 years Cronbach's alpha coefficients were: 0.96 for the social phobia subscale and 0.85 for the agoraphobia subscale.^18,21-22 In the American sample of 223 adolescents from 12 to 18 years, the Cronbach's alpha coefficients were: 0.97 for the social phobia subscale and 0.91 for the agoraphobia subscale and total SPAI Inventory.²⁰ In the Spanish study, in a sample of 3160 adolescents, aged 14 to 17, Cronbach's alpha coefficients were: 0.95 for the social phobia subscale; 0.84 for the agoraphobia subscale and 0.95 for the total SPAI Inventory.²⁷

In the current total sample, the internal consistency results were the same between genders for the social phobia subscale (0.96), identical to the results in a sample of American university students. In the agoraphobia subscale, Cronbach's alpha coefficient was 0.88 among the women and 0.86 among the men, and in the American sample the alpha values were the same in both genders (0.85).^18,21-22 The Cronbach's alpha coefficient for the total SPAI Portuguese was also identical for both genders (0.96), but there are no comparable data in the American or Spanish samples according to gender.^18,28 The results showed that social phobia and agoraphobia subscales were equally reliable for men and women, as the result obtained in an American study. The total SPAI Portuguese Inventory also was equally reliable for men and women in this Brazilian student sample.

The findings reveal that the SPAI Portuguese version presents a good content homogeneity expressed by the correlation between its items, measured by the Cronbach's alpha coefficient. The values of Cronbach's alpha coefficient of the two subscales and the total SPAI Portuguese scale showed a high correlation between the items that evaluate the two constructs of the SPAI Inventory: social phobia and agoraphobia.^15,31,34 These results may be an indirect evidence of how well the translation and back-translation process were performed. The process appears to have preserved the dimensions contained in the original SPAI inventory, with a possible equivalent content, to be appropriately investigated in an ongoing content validation study using factorial analysis.²⁸

Test-retest reliability for social phobia and agoraphobia scores and the differential (total) score of the SPAI Portuguese

Pearson correlation coefficients for the social phobia and agoraphobia scores and differential (total) score of the SPAI Portuguese in the four sub-samples and in the total sample were similar to those of the American sample of 182 university students. In the American study the Pearson correlation coefficients were: 0.85 for the social phobia score; 0.74 for the agoraphobia score; and 0.86 for the differential (total) score of the SPAI, which presented no difference between genders, being 0.87 between males and 0.85 between females.^18,21

In the total sample Pearson correlation coefficients were: 0.88 for the social phobia score; 0.80 for the agoraphobia score and 0.83 for the differential (total) score of SPAI Portuguese. For the differential (total) score, the difference between genders in the Brazilian sample (see Table 4) did not present a significant difference (p = 0.163), as in the American study.^18-21

The data reveal identical temporal stability for the social phobia subscale and the SPAI Portuguese inventory for both genders and for the four groups studied.

The Pearson correlation coefficient in the total sample for agoraphobia score was 0.85 for females and 0.74 for males. Females show a higher temporal stability (p = 0.029), than males for agoraphobia items. It will be difficult to explain this difference as due to sample size, because no differences were detected between the four sub-samples that have a smaller number of individuals.

Concordance analyses through intra-class correlation coefficient were also performed, since this is the most appropriate form of evaluation of the temporal stability of an instrument whose score is treated as a continuous variable.^28,37 As to the test-retest reliability estimates, the intra-class correlation controls the effect of chance over variance between individuals and between test and retest more appropriately, when compared to the Pearson correlation coefficient method.^15,31

Intra-class correlation analyses showed results that were mostly identical or very similar to those of the analyses of Pearson correlation coefficients, in the four sub-samples, in both genders and in the total sample. These findings may be attributed to the adequate size of the sample and to the score variations between the test and retest being due mainly to biological variability and to the difference inherent to the passage of time between the two assessments, which may indicate the absence of measurement bias. The isolated use of the Pearson correlation coefficient may overestimate the results of reliability because it does not take into account a possible presence of systematic measurement errors.^28,35,38

The 95% confidence intervals of the Pearson correlation coefficients and the intra-class correlations for the three scores studied in the four sub-samples were rather narrow, once again indicating an appropriate sample size. The lower limits were mostly above 0.70 (Tables 3 and 4). In the total sample the 95% confidence intervals of results for r and ICC are always above 0.75 (Table 3).

The findings are compatible with test-retest correlation and concordance indexes from good to excellent and perfectly acceptable for the three scores: social phobia, agoraphobia and differential (total) score of the SPAI Portuguese.

Although good results for instrument reliability do not assure its validity, instrument validity is unfeasible when reliability results are poor.^28,38 The results presented here indicate appropriate reliability to use the SPAI Portuguese version in Brazilian population samples. However, further studies are still needed to show their construct validity in Brazilian population and clinical samples.

Thus the results do not assure the validity of the SPAI Portuguese version, currently under investigation in factorial validity studies in a population sample and as to its convergent and discriminant validities in a clinical sample. Therefore, this study is part of the broader validation process of the Portuguese language version of the Social Phobic and Anxiety Inventory in Brazilian samples.

Conclusions

The results in the four sub-samples and in the total sample studied show fully satisfactory levels of internal consistency of the social phobia and agoraphobia subscales and Portuguese language SPAI Inventory, with a substantial temporal stability of the social phobia, agoraphobia scores and differential (total) score of the SPAI Portuguese. The data show that the Portuguese-language version of SPAI shows perfectly acceptable reliability. These results are comparable to those found in population samples of American university students and adolescents where the English language original version was developed.

The heterogeneous sample studied allows us to suggest its applicability to non-clinical Brazilian populations of both genders, as well as to heterogeneous samples of over-14-year-old adolescents and young adults.

The advantages of the self-reported scales, both for clinical care and for research, are that they are easily applied and have a low cost. It is unnecessary to train raters, and the data collected are easy to compare over time and between different patients. Its use in different populations, different therapeutic settings and different cultures will depend on the capacity to generalize the research data about the psychometric properties of these instruments. Thus, validation studies in samples of different cultures increase the external validity of these instruments.

Due to the family, academic, occupational, social, affective and sexual environments limitations social phobia imposes on those who have it, the disorder has been recognized as a public health problem. The application of the SPAI Inventory by itself should not be used to confirm the definitive diagnosis of Social Phobia, and a psychiatric clinical evaluation is required.¹⁸ However, the early identification of probable cases of social phobia by means of duly validated screening instruments such as the SPAI Portuguese language version, reinforces the importance of the data presented here.

Acknowledgments

To Ana Carolina Seganfredo MD, Caroline Dei Ricardi MD, Ana Carolina Castro MD, Cassiane Bonato MD for their participation on field work. To Norberto Martins MD, for his cooperation on data bank work and computerized scores calculations.

Submitted: 21 September 2004

Accepted: 25 November 2004

Financing: This work was partly supported by The Research Foundation of the State of Rio Grande do Sul (FAPERGS; nº 97/50734.9).

Conflicts of interests: None

Original version accepted in English

This paper was elaborated from data set of the first author Ph. D. degree at Post-graduate Program in Medical Sciences: Psychiatry - Universidade Federal do Rio Grande do Sul (Federal University of Rio Grande do Sul, UFRGS).

Part of the results were presented as posters at: 28^th Annual Meeting of the Brazilian Psychiatric Association, October 25-28, 2000, Rio de Janeiro, RJ, Brazil and 157^th Annual Meeting of the American Psychiatric Association, May 1-6, 2004, New York, NY, USA.

Publication Dates

History