Abstract
This chapter uses the recently conducted evaluation of Building Strong Families as an example to explore two challenges for family research: (1) managing multiple comparisons when evaluating a program that aims to influence a wide range of outcomes and (2) dealing with potential bias introduced by examining truncated measures (i.e. measures that are not defined for all sample members). The evaluation’s strategy for dealing with multiple comparisons was to limit the number of outcomes that provide the key test of program effectiveness, and then use methods such as outcome domain composite indices and statistical adjustments for multiple comparisons to assess the strength of the pattern of impact findings. The evaluation’s strategy for dealing with truncated variables was to assess the risk of truncation bias using existing standards for sample attrition in experimental evaluations, and then use that assessment to determine which analysis may be included in evaluating program effectiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The members of the couple had to be able to speak one of the languages in which BSF was offered in their location. All locations offered BSF in English, and some also offered BSF in Spanish. At intake, couples were screened for intimate partner violence. If there was evidence of intimate partner violence that could be aggravated by participating in BSF, the couple was not considered eligible for BSF and was referred to other services.
References
Amato, P. (2001). Children of divorce in the 1990s: An update of the Amato and Keith meta-analysis. Journal of Family Psychology, 15, 355–370.
Amato, P. (2005). The impact of family formation change on the cognitive, social, and emotional well-being of the next generation. Future of Children, 15(2), 75–96.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological), 57, 289–300.
Brown, S. L. (2003). Relationship quality dynamics of cohabiting unions. Journal of Family Issues, 24(5), 583–601.
Brown, S. L. (2004). Family structure and child well-being: The significance of parental cohabitation. Journal of Marriage and Family, 66, 351–367.
Carlson, M., McLanahan, S., England, P., & Devaney, B. (2005). What we know about unmarried parents: Implications for building strong families programs. http://www.mathematica-mpr.com/publications/redirect_PubsDB.asp?strSite=PDFs/bsfisbr3.pdf. Accessed Aug 2012.
Center for Research on Child Well-being. (2007). Parents’ relationship status five years after a non-marital birth (Fragile Families Research Brief No. 39). Princeton: CRCW.
Graefe, D. R., & Lichter, D. T. (2002). Marriage among unwed mothers: Whites, blacks and Hispanics compared. Perspectives on Sexual and Reproductive Health, 34(6), 286–293.
McConnell, S., Stuart, E. A., & Devaney, B. (2008). The truncation by death problem. Evaluation Review, 32(2), 157–186.
McLanahan, S., & Sandefur, G. (1994). Growing up with a single parent: What hurts, what helps. Cambridge: Harvard University Press.
Schochet, P. Z. (2009). An approach for addressing the multiple testing problem in social policy impact evaluations. Evaluation Review, 33(6), 539–567.
Tucker, M. B., & Mitchell-Kernan, C. (1995). Trends in African American family formation: A theoretical and statistical overview. In M. B. Tucker, & C. Mitchell-Kernan (Eds.), The decline in marriage among African Americans (pp. 3–26). New York: Russell Sage.
U.S. Department of Education. (2008). WWC procedures and standards handbook. Washington, DC: Institute for Education Sciences. http://ies.ed.gov/ncee/wwc/references/idocviewer/Doc.aspx?docId=19 & tocId=7. Accessed Aug 2012.
Wood, R. G., McConnell, S., Moore, Q., Clarkwest, A., & Hsueh, J. (2010). Strengthening unmarried parent’s relationships: The early impacts of building strong families. Princeton: Mathematica Policy Research.
Wood, R. G., Moore, Q., & Clarkwest, A. (2011). BSF’s effects on couples who attended group relationship skills sessions: A special analysis of 15-month data. Princeton: Mathematica Policy Research.
Wood, R. G., Moore, Q., Clarkwest, A., Killewald, A., & Monahan, S. (2012). The long-lerm effects of Building Strong Families: A relationship skills education program for unmarried parents. Princeton: Mathematica Policy Research.
Zhang, J. L., & Rubin, D. B. (2003). Estimation of causal effects via principal stratification when some outcomes are truncated by “death”. Journal of Educational and Behavioral Statistics, 28(4), 353–368.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Moore, Q., Wood, R. (2014). Multiple Comparisons and Truncation Bias in Family Policy Research: Strategies from the Building Strong Families Evaluation. In: McHale, S., Amato, P., Booth, A. (eds) Emerging Methods in Family Research. National Symposium on Family Issues, vol 4. Springer, Cham. https://doi.org/10.1007/978-3-319-01562-0_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-01562-0_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-01561-3
Online ISBN: 978-3-319-01562-0
eBook Packages: Behavioral ScienceBehavioral Science and Psychology (R0)