Skip to main content
SpringerLink
Log in

Good practice in testing for an association in contingency tables

  • Methods
  • Published:
Behavioral Ecology and Sociobiology Aims and scope Submit manuscript

Abstract

The testing for an association between two categorical variables using count data is commonplace in the behavioral sciences. Here, we present evidence that influential biostatistical textbooks give contradictory and incomplete advice on good practice in the analysis of such contingency table data. We survey the statistical literature and offer guidance on such analyses. Specifically, we call for greater use of exact testing rather than tests which use an asymptotic chi-squared distribution. That is, we suggest that researchers take a conservative approach and only perform asymptotic testing where there is little doubt that it is appropriate. We recommend a specific criterion for such decision-making. Where asymptotic testing is appropriate, we recommend chi-squared over the G-test and recommend against the implementation of Yates (or any other) correction. We also provide advice on the effective use of exact testing for associations in contingency tables. Lastly, we highlight issues that need to be considered when using the commonly recommended Fisher’s exact test.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Agresti A (1992) A survey of exact inference for contingency tables. Stat Sci 7:131–177

    Article  Google Scholar 

  • Agresti A (2001) Exact inference for categorical data: recent advances and continuing controversies. Stat Med 20:2709–2722

    Article  CAS  PubMed  Google Scholar 

  • Agresti A (2007) An introduction to categorical data analysis, 2nd edn. Wiley, New York

    Google Scholar 

  • Andres AM, Mato AS (1994) Choosing the optimal unconditioned test for comparing two independent proportions. Comput Stat Data Anal 17:555–574

    Article  Google Scholar 

  • Baglivo J, Oliver D, Pagano M (1988) Methods for analysis of contingency tables with large and small cell counts. J Am Stat Assoc 83:1006–1013

    Article  Google Scholar 

  • Bailey NJT (1995) Statistical methods in biology. Cambridge University Press, Cambridge

    Google Scholar 

  • Barnard GA (1945) A new test for 2 × 2 tables. Nature 156:783–784

    Article  Google Scholar 

  • Berger RL (1994) Mehta CR & Hilton JF (1993) comment. Am Stat 48:2

    Article  Google Scholar 

  • Berger VW (2004) On the generation and ownership of alpha in medical studies. Control Clin Trials 25:613–619

    Article  PubMed  Google Scholar 

  • Boschloo RB (1970) Raised conditional level of significance for the 2 × 2-table when testing the equality of two probabilities. Stat Neerl 24:1–35

    Article  Google Scholar 

  • Bradley DR, Bradley TD, McGrath SG, Cutcomb SD (1979) Type I error rate of the chi-square test of independence in R x C tables that have small expected frequencies. Psychol Bull 86:1290–1297

    Article  Google Scholar 

  • Camilli G, Hopkins KD (1978) Applicability of chi-square to 2 × 2 contingency tables with small expected cell frequencies. Psychol Bull 85:163–167

    Article  Google Scholar 

  • Camilli G, Hopkins KD (1979) Testing for association in 2 × 2 contingency tables with very small sample sizes. Psychol Bull 86:1011–1014

    Article  Google Scholar 

  • Clewer AG, Scarisbrik DH (2001) Practical statistics and experimental design for plant and crop sciences. Wiley, New York

    Google Scholar 

  • Cochran WG (1954) Some methods for strengthening the common chi-squared tests. Biometrics 10:417–451

    Article  Google Scholar 

  • Conahan MA (1970) The comparative accuracy of the likelihood ratio and chi-squared as approximations to the exact multinomial test. ED.D. diss. Lehigh Univ, p 64

  • Cressie N, Read TRC (1989) Pearson’s chi-squared and the log-likelihood ratio statistic G2: a comparative review. Int stat rev 57:19–43

    Article  Google Scholar 

  • Delucchi KL (1983) The use and misuse of chi-square: Lewis & Burke revisited. Psychol Bull 94:166–176

    Article  Google Scholar 

  • Fisher RA (1925) Statistical methods for research workers. Oliver & Boyd, Edinburgh

    Google Scholar 

  • Freeman GH, Halton JH (1951) Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika 38:141–149

    CAS  PubMed  Google Scholar 

  • Garamszegi LZ et al (2009) Changing philosophies and tools for statistical inferences in behavioural ecology. Behav Ecol 20:1363–1375

    Article  Google Scholar 

  • Haber M (1980) A comparison of some continuity corrections for the chi-squared test on 2 × 2 tables. J Am Stat Assoc 75:510–515

    Article  Google Scholar 

  • Haviland MG (1990) Yates’s correction for continuity and the analysis of 2 × 2 contingency-tables. Stat Med 9:363–367

    Article  CAS  PubMed  Google Scholar 

  • Hawkins D (2005) Biomeasurement. Oxford University Press, Oxford

    Google Scholar 

  • Hirji KF, Tan S-J, Elashoff RM (1991) A quasi-exact test for comparing 2 binomial proportions. Stat Med 10:1137–1153

    Article  CAS  PubMed  Google Scholar 

  • Larntz K (1978) Small-sample comparisons of exact levels for chi-squared goodness-of-fit statistics. J Am Stat Assoc 73:253–263

    Article  Google Scholar 

  • Lawal HB, Upton GJG (1984) On the use of chi-squared as a test of independence in contingency tables with small cell expectations. Aust J Stat 26:75–85

    Article  Google Scholar 

  • Lawson R (2004) Small sample confidence intervals for the odds ratio. Commun Stat simul comput 33:1095–1113

    Google Scholar 

  • Lloyd CJ (1988) Doubling the one-sided P-value in testing independence in 2 x 2 tables against a two-sided alternative. Stat Med 7:1297–1306

    Article  CAS  PubMed  Google Scholar 

  • Lloyd CJ, Moldovan MV (2007) Unconditional efficient one-sided confidence limits for the odds ratio based on conditional likelihood. Stat Med 26:5136–5146

    Article  PubMed  Google Scholar 

  • Lombardi CM, Hurlbert SH (2009) Misprescription and misuse of one-tailed tests. Aust Ecol 34:447–468

    Google Scholar 

  • Lydersen S, Pradhan V, Senchaudhuri P, Laake P (2005) Comparison of exact tests for association in unordered contingency tables using standard, mid-P, and randomized test versions. J Stat Comput Simul 75:447–458

    Article  Google Scholar 

  • Lydersen S, Pradhan V, Senchaudhuri P, Laake P (2007) Choice of test for association in small sample unordered r x c tables. Stat Med 26:4328–4343

    Article  CAS  PubMed  Google Scholar 

  • Lydersen S, Fagerland M, Laake P (2009) Tutorial in biostatistics: recommended tests for association in 2 × 2 tables. Stat Med 28:1159–1175

    Article  PubMed  Google Scholar 

  • Martin Andres A, Herranz Tejedor I (1995) Is Fisher’s exact test very conservative. Comput Stat Data Anal 19:579–591

    Article  Google Scholar 

  • Martin Andres A, Silva Mato A, Tapia Garcia JM, Sanchez Auevedo MJ (2004) Comparing the asymptotic power of exact tests in the 2 x 2 tables. Comput Stat Data Anal 47:745–756

    Article  Google Scholar 

  • Mehrotra DV, Chan ISF, Berger RL (2003) A cautionary note on exact unconditional inference for a difference between two independent binomial proportions. Biometrics 59:441–450

    Article  PubMed  Google Scholar 

  • Mehta CR, Patel NR (1983) A network algorithm for performing fisher’s exact test in r x c contingency tables. J Am Stat Assoc 78:427–434

    Article  Google Scholar 

  • Mehta CR, Patel NR (1986) A hybrid algorithm for Fisher’s exact test in unordered r×c contingency tables. Commun Stat Theory Methods 15:387–403

    Article  Google Scholar 

  • Mehta CR, Hilton JF (1993) Exact power of conditional and unconditional tests: doing beyond the 2 × 2 table. Am Stat 47:91–98

    Article  Google Scholar 

  • Meulepas E (1998) A two-tailed P-value for Fisher’s exact test. Biom J 40:3–10

    Article  Google Scholar 

  • Neuhäuser M (2004) The choice of α for one-sided tests. Drug Inf J 38:57–60

    Google Scholar 

  • Parshall CG, Kromrey JD (1996) Tests of independence in contingency tables with small samples: a comparison of statistical power. Educ Psychol Meas 56:26–44

    Article  Google Scholar 

  • Quinn GP, Keough MJ (2002) Experimental design and data analysis for biologists. Cambridge University Press, Cambridge

    Google Scholar 

  • Rice WR, Gaines SD (1994) “Heads I win, tails you lose”: testing directional alternative hypotheses in ecological and evolutionary research. Trends Ecol Evol 9:235–237

    Article  Google Scholar 

  • Roscoe JT, Byars JA (1971) Sample size restraints commonly imposed on the use of the chi-square statistic. J Am Stat Assoc 66:755–759

    Article  Google Scholar 

  • Ruxton GD, Neuhäuser M (2010) When should we use one-tailed hypothesis testing? Meth Ecol Evol 1:114–117

    Article  Google Scholar 

  • Senchaudhuri P, Mehta CR, Patel NR (1995) Estimating exact P-values by the method of control variates or Monte-Carlo rescue. J Am Stat Assoc 90:640–648

    Article  Google Scholar 

  • Sokal RR, Rohlf FJ (1995) Biometry, 3rd edn. Freeman, New York

    Google Scholar 

  • Stephens PA, Buskirk SW, Hayward GD, del Rio CM (2005) Information theory and hypothesis testing: a call for pluralism. J Appl Ecol 42:4–12

    Article  Google Scholar 

  • Stephens PA, Buskirk SW, Martínez del Rio C (2007) Inference in ecology and evolution. Trends Ecol Evol 22:192–197

    Article  PubMed  Google Scholar 

  • Suissa S, Shuster JJ (1985) Exact unconditional sample sizes for the 2 × 2 binomial trials. J R Stat Soc A 148:317–327

    Google Scholar 

  • Thompson B (1988) Misuse of chi-square contingency table test statistics. Educ Psychol Res 8:39–49

    Google Scholar 

  • Van der Meulen EA (2008) A nonrandomized, nonconservative version of the Fisher exact test. Commun Stat Theory Methods 37:699–708

    Article  Google Scholar 

  • Williams B (1993) Biostatistics. Chapman & Hall, New York

  • Yates F (1984) Tests of significance for 2 × 2 contingency tables. J R Stat Soc A 147:426–463

    Google Scholar 

  • Zar JH (1999) Biostatistical analysis, 4th edn. Prentice Hall, New York

    Google Scholar 

  • Zar JH (2008) Biostatistical analysis, 5th edn. Prentice Hall, New York

    Google Scholar 

Download references

Acknowledgement

We thank the referees for the very valuable suggestions on previous versions of this article.

Author information

Authors and Affiliations

  1. Division of Ecology & Evolutionary Biology, Faculty of Biomedical & Life Sciences, University of Glasgow, Graham Kerr Building, Glasgow, G12 8QQ, Scotland, UK

    Graeme D. Ruxton

  2. Department of Mathematics and Technique—RheinAhrCampus, Koblenz University of Applied Sciences, Südallee 2, 53424, Remagen, Germany

    Markus Neuhäuser

Authors
  1. Graeme D. Ruxton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markus Neuhäuser
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Graeme D. Ruxton.

Additional information

Communicated by L. Garamszegi

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ruxton, G.D., Neuhäuser, M. Good practice in testing for an association in contingency tables. Behav Ecol Sociobiol 64, 1505–1513 (2010). https://doi.org/10.1007/s00265-010-1014-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00265-010-1014-0

Keywords

Search

Navigation