Elsevier

NeuroImage

Volume 54, Issue 3, 1 February 2011, Pages 2033-2044
NeuroImage

A reproducible evaluation of ANTs similarity metric performance in brain image registration

https://doi.org/10.1016/j.neuroimage.2010.09.025Get rights and content

Abstract

The United States National Institutes of Health (NIH) commit significant support to open-source data and software resources in order to foment reproducibility in the biomedical imaging sciences. Here, we report and evaluate a recent product of this commitment: Advanced Neuroimaging Tools (ANTs), which is approaching its 2.0 release. The ANTs open source software library consists of a suite of state-of-the-art image registration, segmentation and template building tools for quantitative morphometric analysis. In this work, we use ANTs to quantify, for the first time, the impact of similarity metrics on the affine and deformable components of a template-based normalization study. We detail the ANTs implementation of three similarity metrics: squared intensity difference, a new and faster cross-correlation, and voxel-wise mutual information. We then use two-fold cross-validation to compare their performance on openly available, manually labeled, T1-weighted MRI brain image data of 40 subjects (UCLA's LPBA40 dataset). We report evaluation results on cortical and whole brain labels for both the affine and deformable components of the registration. Results indicate that the best ANTs methods are competitive with existing brain extraction results (Jaccard = 0.958) and cortical labeling approaches. Mutual information affine mapping combined with cross-correlation diffeomorphic mapping gave the best cortical labeling results (Jaccard = 0.669 ± 0.022). Furthermore, our two-fold cross-validation allows us to quantify the similarity of templates derived from different subgroups. Our open code, data and evaluation scripts set performance benchmark parameters for this state-of-the-art toolkit. This is the first study to use a consistent transformation framework to provide a reproducible evaluation of the isolated effect of the similarity metric on optimal template construction and brain labeling.

Research Highlights

►A new, fast implementation of the cross-correlation that increases computational efficiency by a factor of 4 to 5 and allows larger correlation windows to be used for registration without excessive increase in computation time. ►Open-source implementation of the mutual information for symmetric diffeomorphic registration. ►A reproducible system for performance evaluation of the mean squares metric, cross-correlation metric and mutual information metric on optimal template-based brain extraction and regional brain labeling. The full evaluation system is documented in a bash script that is also released and available. The script is also being translated to python. ►Quantification of the similarity between optimal templates derived from different population subsets and with different similarity metrics.

Introduction

Rapid advancement in biological and medical imaging technologies increases demand for quantitative, computational anatomy tools. The principal tools of this emerging field are deformable mappings between images whether they be driven by similarity metrics which are intensity-based, point-set-based, or both. Several categories of mappings exist in the literature. Of particular recent interest are diffeomorphic transformations which, by definition, preserve topology. Topology preservation is fundamental to making comparisons between objects in the natural world that are thought to differ or change while preserving local neighborhood relations. Cytoarchitectonic brain mapping studies also suggest that the layout of cell types throughout the brain is generally preserved (Schleicher et al., 2009), further motivating diffeomorphic mapping in the context of the brain.

Our limited assessment of published research mirrors the experience of many others who prefer a working paradigm of reproducible research (Kovacevic, 2006). Dr. Kovacevic describes “[reproducible research as] the idea that, in ‘computational’ sciences, the ultimate product is not a published paper but, rather, the entire environment used to produce the results in the paper (data, software, etc.).” After an informal survey of 15 published papers, she finds “none had code available” and “in only about half the cases were the parameters [of the algorithm] specified.” The computational sciences research community also voices concerns about reproducibility (Yoo & Metaxas, 2005, Ibanez et al., 2006). In this paper, we discuss our contribution to the open source medical image analysis research community which we call ANTs (Advanced Neuroimaging Tools). Built on an Insight ToolKit (ITK) framework, this software package comprises a suite of tools for image registration, template building and segmentation based on previously published research. Here, we provide an overview of the package and detail recent technical advances, in the spirit of previous papers published in this journal (Neu et al., 2005, Zhang et al., 2008, Patel et al., 2010) and open source registration tools such as Elastix (Klein et al., 2010b).

The recent outcome from two large-scale comparative image registration algorithm assessments (Klein et al., 2009), http://empire10.isi.uu.nl is perhaps the most persuasive evidence motivating the contributions discussed in this paper. Our Symmetric Normalization (SyN) transformation model (Avants et al., 2008) performs consistently in the top rank across all tests in the Klein et al. (2009) study and finished first overall in the phase one Empire-10 evaluation study of intra-subject thoracic CT registration (http://empire10.isi.uu.nl). Unlike some of the other algorithms in these studies, all of our methods (not just SyN) are open source software.

One difficulty in interpreting the results of these evaluation studies is that each algorithm uses a different combination of transformation model (the geometric constraint on the mapping between brains), similarity metric (the measure that evaluates how similar two images appear), and multi-resolution, optimization, and resampling strategies and parameter settings. Thus, one cannot isolate the effect of transformation model from similarity metric or optimization strategy. Other aspects of implementation may also differ, including whether the authors recommend using whole head or whole brain data. For instance, the DARTEL algorithm (Ashburner, 2007) uses whole head data and segmentation to aid performance while the other methods did not incorporate segmentation. The follow-up evaluation study Klein et al. (2010a) evaluated ART2.0 (Ardekani et al., 2005), SyN, and Freesurfer (Fischl and Dale, 2000) on whole head data and found that both brain extraction and registration via an “optimal” (group-generated) template improve performance. However, Klein et al. (2010a) applied generic parameters for ANTs, including the similarity metric, which might have resulted in suboptimal performance for the whole head component of the study.

Consequently, here we study the effect of the similarity metric on whole head registration-based labeling via an optimal template. We evaluate ANTs affine as well as nonlinear registration performance because accuracy in both stages is critical for successful registration-based brain segmentation/labeling. Furthermore, this problem is faced routinely in brain image processing labs (Ségonne et al., 2004, Sadananthan et al., 2010, Park & Lee, 2009, Lim & Pfefferbaum, 1989, de Boer et al., 2010, Acosta-Cabronero et al., 2008). One advantage of a consistent and modular framework, such as constructed in ANTs, is that it is possible to evaluate a single component of the processing stream while holding all other aspects constant.

The paper organization: Theoretical overview of ANTs section gives an overview of the transformation models and similarity metrics in ANTs and their use with SyN in population mapping. Experimental evaluation section reports results on a series of large-scale experiments using the LPBA manually labeled dataset to evaluate ANTs registration applied to cortical and brain labeling. Finally, we close with a discussion of our findings.

Section snippets

Theoretical overview of ANTs

The following three components provide a common classification schema for registration methods (Brown, 1992, Ibanez et al., 2002):

  • the transformation model, which includes the regularization kernels,

  • the similarity (or correspondence) measures, and

  • the optimization strategy.

In general, image normalization computes the optimal transformation, ϕ, within a transformation space which maps each x of image I(x) to a location in image J(z) by minimizing a cost function, C, describing the similarity

Experimental evaluation

We now apply the above methods using Gaussian regularization of the velocity field, the SyN transformation model, the SyGN template building algorithm and the MSQ, CC and MI metrics to build templates via cross-validation, label the templates by majority voting and apply the templates to the LPBA40 validation dataset.

Results

We first establish template stability across population sub-divisions and metrics. We then detail performance differences by comparing evaluation results across metrics.

Summary

In this paper, we provide an overview of the ANTs toolkit and detail the ANTs implementation of MSQ, CC and MI deformable image registration metrics. We also contribute a new implementation of the CC metric that reduces computation time by a factor of 4–5 with default parameters in 1 mm3 3D brain image registration. We evaluate the impact of these metric choices—and their affine counterparts—on optimal template construction and template-based brain labeling. We use a conservative two-fold

Acknowledgments

ANTs is supported by Grant R01EB006266-01 from the National Institute of Biomedical Imaging and Bioengineering and administered through the UCLA Center for Computational Biology. We would also like to thank Ray Razlighi for checking the ANTs MI code and improving both its clarity and implementation.

References (83)

  • N. Fox et al.

    Imaging of onset and progression of Alzheimer's disease with voxel-compression mapping of serial magnetic resonance images

    Lancet

    (2001)
  • J.C. Gee

    On matching brain volumes

    Pattern Recognit.

    (1999)
  • R.A. Heckemann et al.

    Automatic anatomical brain MRI segmentation combining label propagation and decision fusion

    Neuroimage

    (2006)
  • R.A. Heckemann et al.

    Improving intersubject image registration using tissue-class information benefits robustness and accuracy of multi-atlas based anatomical segmentation

    Neuroimage

    (2010)
  • H. Jia et al.

    Absorb: atlas building by self-organized registration and bundling

    Neuroimage

    (2010)
  • S. Joshi et al.

    Unbiased diffeomorphic atlas construction for computational anatomy

    Neuroimage

    (2004)
  • A. Klein et al.

    Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration

    Neuroimage

    (2009)
  • A. Klein et al.

    Evaluation of volume-based and surface-based brain image registration methods

    Neuroimage

    (2010)
  • S.C. Neu et al.

    The loni debabeler: a mediator for neuroimaging software

    Neuroimage

    (2005)
  • V. Noblet et al.

    Retrospective evaluation of a topology preserving non-rigid registration method

    Med. Image Anal.

    (2006)
  • J.G. Park et al.

    Skull stripping based on region growing for magnetic resonance brain images

    Neuroimage

    (2009)
  • V. Patel et al.

    Loni mind: metadata in nifti for dwi

    Neuroimage

    (2010)
  • P. Rogelj et al.

    Symmetric image registration

    Med. Image Anal.

    (2006)
  • P. Rogelj et al.

    Point similarity measures for non-rigid registration of multi-modal data

    Comput. Vis. Image Underst.

    (2003)
  • T. Rohlfing et al.

    Evaluation of atlas selection strategies for atlas-based image segmentation with application to confocal microscopy images of bee brains

    Neuroimage

    (2004)
  • S.A. Sadananthan et al.

    Skull stripping using graph cuts

    Neuroimage

    (2010)
  • F. Ségonne et al.

    A hybrid approach to the skull stripping problem in MRI

    Neuroimage

    (2004)
  • D.W. Shattuck et al.

    Construction of a 3D probabilistic atlas of human cortical structures

    Neuroimage

    (2008)
  • G. Tao et al.

    Symmetric inverse consistent nonlinear registration driven by mutual information

    Comput. Methods Programs Biomed.

    (2009)
  • J.-P. Thirion

    Image matching as a diffusion process: an analogy with Maxwell's demons

    Med. Image Anal.

    (1998)
  • T. Vercauteren et al.

    Diffeomorphic demons: efficient non-parametric image registration

    Neuroimage

    (2009)
  • M.A. Yassa et al.

    High-resolution structural and functional MRI of hippocampal CA3 and dentate gyrus in patients with amnestic mild cognitive impairment

    Neuroimage

    (2010)
  • T.S. Yoo et al.

    Open science—combining open data and open source software: medical image analysis with the Insight Toolkit

    Med. Image Anal.

    (2005)
  • P.A. Yushkevich et al.

    A high-resolution computational atlas of the human hippocampus from postmortem magnetic resonance imaging at 9.4t

    Neuroimage

    (2009)
  • P.A. Yushkevich et al.

    Bias in estimation of hippocampal atrophy using deformation-based morphometry arises from asymmetric global normalization: an illustration in ADNI 3T MRI data

    Neuroimage

    (2010)
  • J. Zhang et al.

    Evaluation and comparison of glm- and cva-based fMRI processing pipelines with java-based fMRI processing pipeline evaluation system

    Neuroimage

    (2008)
  • V.I. Arnold

    Ordinary Differential Equations

    (1991)
  • B. Avants et al.

    Insight into images

  • B. Avants et al.

    Spatiotemporal normalization for longitudinal analysis of gray matter atrophy in frontotemporal dementia

    Med. Image Comput. Comput. Assist. Interv. Int. Conf. Med. Image Comput. Comput. Assist. Interv.

    (2007)
  • B.B. Avants et al.

    ANTS: Advanced Open-Source Normalization Tools for Neuroanatomy

    (2009)
  • M.F. Beg et al.

    Computing large deformation metric mappings via geodesic flows of diffeomorphisms

    Int. J. Comput. Vision

    (2005)
  • Cited by (3146)

    • Privacy preserving image registration

      2024, Medical Image Analysis
    View all citing articles on Scopus
    View full text