Using MCMC chain outputs to efficiently estimate Bayes factors

doi:10.1016/j.jmp.2011.06.004

Journal of Mathematical Psychology

Volume 55, Issue 5, October 2011, Pages 368-378

https://doi.org/10.1016/j.jmp.2011.06.004 Get rights and content

Abstract

One of the most important methodological problems in psychological research is assessing the reasonableness of null models, which typically constrain a parameter to a specific value such as zero. Bayes factor has been recently advocated in the statistical and psychological literature as a principled means of measuring the evidence in data for various models, including those where parameters are set to specific values. Yet, it is rarely adopted in substantive research, perhaps because of the difficulties in computation. Fortunately, for this problem, the Savage–Dickey density ratio (Dickey & Lientz, 1970) provides a conceptually simple approach to computing Bayes factor. Here, we review methods for computing the Savage–Dickey density ratio, and highlight an improved method, originally suggested by Gelfand and Smith (1990) and advocated by Chib (1995), that outperforms those currently discussed in the psychological literature. The improved method is based on conditional quantities, which may be integrated by Markov chain Monte Carlo sampling to estimate Bayes factors. These conditional quantities efficiently utilize all the information in the MCMC chains, leading to accurate estimation of Bayes factors. We demonstrate the method by computing Bayes factors in one-sample and one-way designs, and show how it may be implemented in WinBUGS.

Highlights

► We demonstrate using conditional quantities in MCMC to estimate Bayes factors. ► We show how using conditional quantities substantially outperforms other methods. ► We apply the technique to point-null and area-null hypothesis tests. ► We provide WinBUGS code to implement the method in a simple t test case.

Section snippets

Bayes factor

In psychology, hypothesis testing is the most widely used method of making inferences from data. The goal of hypothesis testing is to assess the evidence provided by the data for or against a hypothesis. In frequentist null hypothesis significance testing, for instance, hypothesis tests assess the evidence against a null hypothesis. In this paper, we approach hypothesis testing from a model selection perspective, in which the null and alternative hypotheses are treated as separate models. The

The Savage–Dickey method

For the nested-model setup above, the Savage–Dickey method provides a convenient way to compute the Bayes factor, provided certain conditions are met. The marginal probability of the data under the null may be expressed as a restriction of the model $M_{1}$ : $p (y ∣ θ = θ_{0}, M_{1})$ . Consequently, the Bayes factor for the null model relative to the general one is: $B_{01} = \frac{p (y | M_{0})}{p (y | M_{1})} = \frac{p (y ∣ θ = θ_{0}, M_{1})}{p (y ∣ M_{1})} .$ Because all quantities are conditioned on $M_{1}$ , this dependence may be dropped from the notation without

Improved Savage–Dickey estimates

Logspline density estimates and normal approximations use marginal posterior samples of $θ$ as input but do not rely on samples of $ϕ$ , the parameters in common across the full and restricted models. At first glance, using samples from $θ$ may appear reasonable; after all, the marginal posterior density of $θ$ at $θ_{0}$ is exactly the quantity of interest. Yet, the sample of $ϕ$ , in conjunction with the data, provided all the information used to sample $θ$ in the MCMC chain. Gelfand and Smith (1990) noted that

Model and priors

We first outline the one-sample $t$ test model, and then describe how estimates of the Bayes factor may be obtained. As is conventional to assume, the likelihood of the data is normal. It is convenient to parameterize the model in terms of standardized effect size $δ = μ / σ$ : $y_{i} \overset{ind.}{\sim} Normal (σ δ, σ^{2})$ where $i = 1, \dots, N$ indexes participant. We place a conventional noninformative Jeffreys prior on $σ^{2}$ : $π (σ^{2}) \propto \frac{1}{σ^{2}} .$

We must also place a prior on $δ$ , the parameter of interest in the general model. In Bayesian parameter

One-way, between-subjects ANOVA

Our first example showed how the CMDE method can be applied to Rouder et al. ’s $t$ test Bayes factor. Although the $t$ test is one of the first statistical tests that students learn in introductory statistics classes, it is not as commonly used in practice as other statistical tests, such as ANOVA. The main feature of ANOVA is a multivariate null hypothesis in which all group effects are zero. We assess the performance of both normal approximation and CMDE by comparing each to the following

Discussion

In the preceding development, we have described the CMDE method for obtaining efficient estimates of posterior densities. The CMDE method is useful in computing the Bayes factor via the Savage–Dickey method in the case where the normalizing constant on the parameter of interest is known. In general, it will be expected to outperform other methods that do not make use of all the information in the MCMC chain.

We especially expect the CMDE to outperform the kernel density estimates, logsplines,

Conclusion

In foregoing examples, we have applied conditional marginal density estimation of Savage–Dickey ratios to compute Bayes factors. We show that this approach is tractable for a one-sample $t$ test and one-way, between-subjects ANOVA. Bayes factors obtained via CMDE are computationally convenient, and are far more accurate than previously recommended kernel density, logspline, or normal approximations methods. In cases where the necessary normalizing constant for computing the CMDE is unknown,

References (38)

M.D. Lee
How cognitive modeling can benefit from hierarchical Bayesian models
Journal of Mathematical Psychology
(2011)
E.-J. Wagenmakers et al.
Bayesian hypothesis testing for psychologists: a tutorial on the SavageDickey method
Cognitive Psychology
(2010)
Ruud Wetzels et al.
An encompassing prior generalization of the Savage–Dickey density ratio
Computational Statistics and Data Analysis
(2010)
J.O. Berger et al.
Testing a point null hypothesis: the irreconcilability of p values and evidence
Journal of the American Statistical Association
(1987)
J.M. Bernardo et al.
Bayesian theory
(2000)
D. Blackwell
Conditional expectation and unbiased sequential estimation
The Annals of Mathematical Statistics
(1947)
M.-H. Chen
Importance-weighted marginal Bayesian posterior density estimation
Journal of the American Statistical Association
(1994)
Chen, M.-H., & Shao, Q.-M. (1997). Performance study of marginal posterior density estimation via Kullback–Leibler...
S. Chib
Marginal likelihood from the Gibbs output
Journal of the American Statistical Association
(1995)
J.M. Dickey
The weighted likelihood ratio, linear hypotheses on normal location parameters
The Annals of Mathematical Statistics
(1971)

J.M. Dickey et al.

The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain

The Annals of Mathematical Statistics

(1970)

W. Edwards et al.

Bayesian statistical inference for psychological research

Psychological Review

(1963)

A. Gelfand et al.

Sampling based approaches to calculating marginal densities

Journal of the American Statistical Association

(1990)

A. Gelman et al.

Bayesian data analysis

(2004)

S. Geman et al.

Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images

IEEE Transactions on Pattern Analysis and Machine Intelligence

(1984)

P. Heidelberger et al.

Simulation run length control in the presence of an initial transient

Operations Research

(1983)

H. Jeffreys

Theory of probability

(1961)

R. Kass et al.

Bayes factors

Journal of the American Statistical Association

(1995)

I. Klugkist et al.

Bayesian model selection using encompassing priors

Statistica Neerlandica

(2005)

Cited by (64)

Imputation of data Missing Not at Random: Artificial generation and benchmark analysis
2024, Expert Systems with Applications
Experimental assessment of different missing data imputation methods often compute error rates between the original values and the estimated ones. This experimental setup relies on complete datasets that are injected with missing values. The injection process is straightforward for the Missing Completely At Random and Missing At Random mechanisms; however, the Missing Not At Random mechanism poses a major challenge, since the available artificial generation strategies are limited. Furthermore, the studies focused on this latter mechanism tend to disregard a comprehensive baseline of state-of-the-art imputation methods. In this work, both challenges are addressed: four new Missing Not At Random generation strategies are introduced and a benchmark study is conducted to compare six imputation methods in an experimental setup that covers 10 datasets and five missingness levels (10% to 80%). The overall findings are that, for most missing rates and datasets, the best imputation method to deal with Missing Not At Random values is the Multiple Imputation by Chained Equations, whereas for higher missingness rates autoencoders show promising results.
On the relationship of arousal and attentional distraction by emotional novel sounds
2023, Cognition
Unexpected and task-irrelevant sounds can impair performance in a task. It has been shown that highly arousing emotional distractor sounds impaired performance less compared to moderately arousing neutral distractor sounds. The present study tests whether these differential emotion-related distraction effects are directly related to an enhancement of arousal evoked by processing of emotional distractor sounds.
We disentangled costs of orienting of attention and benefits of increased arousal levels during the presentation of highly arousing emotional and moderately arousing neutral novel sounds that were embedded in a sequence of repeated standard sounds. We used sound-related pupil dilation responses as a marker of arousal and RTs as a marker of distraction in a visual categorization task in 57 healthy young adults. Multilevel analyses revealed increased RT and increased pupil dilation in response to novel vs. standard sounds. Emotional novel sounds reduced distraction effects on the behavioral level and increased pupil dilation responses compared to neutral novel sounds. Bayes Factors revealed strong evidence against an inverse proportional relationship between behavioral distraction effects and sound-related pupil dilation responses for emotional sounds. Given that the activity of the locus coeruleus has been linked to both changes in pupil diameter and arousal, it may embody an indirect relationship as a common antecedent by the release of norepinephrine into brain networks involved in attention control and control of the pupil. The present study provides new insights into the relation of changes in arousal and attentional distraction during the processing of emotional task-irrelevant novel sounds.
Seed shielding, an undescribed process that prevents seed from overheating (and dying) in extreme weather conditions
2023, Journal of Arid Environments
Sun exposure is a major threat to the survival of cacti seeds in desertic environments. During summer, fruits of the columnar cactus Pachycereus pringlei from San Esteban Island, northwestern Mexico, are eaten by herbivorous iguanas, and after digestion, seeds are deposited within their scats. We compared the germination percentage of wild seeds exposed to the sun while lying on the ground of the island, with that of seeds collected directly from the fruit, and seeds obtained from old and recently deposited iguana scats. Our results indicate that direct sun exposure causes drastic reduction in seed germination, and this negative effect is reduced in seeds shielded inside feces, although with a cost on germination. We discuss some of the possible implications of seed shielding in the population of P. pringlei.
From inference to design: A comprehensive framework for uncertainty quantification in engineering with limited information
2022, Mechanical Systems and Signal Processing
Citation Excerpt :
Further improvements have been identified with enhanced selection of task dependent distance metrics within the ABC regime [73,72]. Similar approaches include the Metropolis-adjusted Langevin algorithm [78], the Metropolis preconditioned Crank-Nicolson method [79], Hamiltonian MCMC [80], relativistic MCMC [81], and many others [82–86]. An appropriate likelihood metric is not readily available for the NASA challenge problem.
In this paper we present a framework for addressing a variety of engineering design challenges with limited empirical data and partial information. This framework includes guidance on the characterisation of a mixture of uncertainties, efficient methodologies to integrate data into design decisions, and to conduct reliability analysis, and risk/reliability based design optimisation. To demonstrate its efficacy, the framework has been applied to the NASA 2020 uncertainty quantification challenge. The results and discussion in the paper are with respect to this application.
Importance of elongation and organogenesis on the rhizome length of Zostera marina in an individual-based simulation model
2021, Estuarine, Coastal and Shelf Science
The aim of this study was to assess the relative importance of phytomer organogenesis, mortality, and elongation rates on the rhizome length of Eelgrass (Zostera marina). Rhizome length has both a discrete and a continuous component, which are the number of internodes and their lengths, respectively. We hypothesize that rhizome length would be more correlated with elongation rates than either phytomer organogenesis or mortality. An individual-based model was used to simulate the growth of Z. marina individuals in the Punta Banda estuary in Baja California, a low-latitude Eelgrass population. Individuals of the model were composed of a rhizome and lateral branches, both of which were composed of phytomers. The individuals changed by presenting horizontal growth, clonal reproduction, aging, and age-related mortality. Some growth rates were modeled with respect to environmental conditions. The model was validated by comparing the rhizome length of simulated and observed plants. A sensitivity analysis suggested that simulated rhizome length is most correlated to elongation rates, which are mainly affected by water temperature and irradiance.
Time matters: Feature-specific prioritization follows feature integration in visual object processing
2019, NeuroImage
Citation Excerpt :
Given that we tested for the null hypothesis, for this comparison, Bayes factors for dependent-group designs were computed (Rouder et al., 2009) via Gaussian quadrature, using the function ttestBF from the R package BayesFactor v0.9.12–2 (Morey et al., 2018). A non-informative Jeffreys prior was placed on the variance of the normal population (Jeffreys, 1961), while a Cauchy prior with a width of √{2}/2 was placed on the standardized effect size (Morey and Rouder, 2011; Morey et al., 2011; Rouder et al., 2012). Jeffreys (1961) categorization for the grade of evidence was used for the interpretation of Bayes factors.
Objects represent a fundamental selection unit of visual attention. However, at odds with the integrated competition account, our recent study demonstrated that attentional facilitation of constituent features does not spread automatically within an object, but instead depends on the specific task relevance of each feature. Here, we employed a novel experimental design, allowing simultaneous electrophysiological measurements of the allocation of attention to two distinct features (rotation and color) within one object (a square) during both trial-wise and block-wise cued shifts of attention. This was possible through the presentation of a square that evokes two distinct steady-state visual evoked potentials (SSVEPs) for its rotation and its color changes, respectively. Given the continuous oscillatory nature of SSVEPs, we were able to investigate the time course of neural activity in the early visual cortex of the human brain when subjects attended to one of the two features, compared to when the whole object was attended. This approach enabled us to uncover feature-based mechanisms of attention within one object, as well as their interaction with object-based mechanisms. Both behavioral and electrophysiological results indicate a biphasic process composed of an early transient integration of the constituent object features, followed by sustained mechanisms of feature selection with amplification of the to-be-attended feature, followed temporally by suppression of the to-be-ignored feature.

View all citing articles on Scopus

View full text

Using MCMC chain outputs to efficiently estimate Bayes factors

Abstract

Highlights

Section snippets

Bayes factor

The Savage–Dickey method

Improved Savage–Dickey estimates

Model and priors

One-way, between-subjects ANOVA

Discussion

Conclusion

Journal of Mathematical Psychology

Cognitive Psychology

Computational Statistics and Data Analysis

Testing a point null hypothesis: the irreconcilability of p values and evidence

Journal of the American Statistical Association

Bayesian theory

Conditional expectation and unbiased sequential estimation

The Annals of Mathematical Statistics

Importance-weighted marginal Bayesian posterior density estimation

Journal of the American Statistical Association

Marginal likelihood from the Gibbs output

Journal of the American Statistical Association

The weighted likelihood ratio, linear hypotheses on normal location parameters

The Annals of Mathematical Statistics

The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain

The Annals of Mathematical Statistics

Bayesian statistical inference for psychological research

Psychological Review

Sampling based approaches to calculating marginal densities

Journal of the American Statistical Association

Bayesian data analysis

Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images

IEEE Transactions on Pattern Analysis and Machine Intelligence

Simulation run length control in the presence of an initial transient

Operations Research

Theory of probability

Bayes factors

Journal of the American Statistical Association

Bayesian model selection using encompassing priors

Statistica Neerlandica