Assessing performance of a randomized versus a non-randomized study design

https://doi.org/10.1016/j.cct.2007.07.006Get rights and content

Abstract

Introduction

Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.

Methods

The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality.

Results

In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.

Conclusion

Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.

Introduction

To evaluate the effectiveness of psychotherapeutic and pharmacotherapeutic treatment, randomized controlled trials have become a ‘golden standard’ [1], [2], [3]. Since participants are allocated to treatment group by chance, randomization minimizes the differences among groups and ensures approximate balance for both observable and unobservable covariates. As a consequence, there will be less confounding factors which might affect the intervention-effect and differences between groups can thus be attributed to the treatment received [4].

However, reality sometimes complicates the process of randomization, or even makes it impossible to use this strong experimental design [5], [2]. As a consequence, alternatives to randomization have been developed, for example quasi-experimental and case control designs. These kinds of designs can be used when political, practical or ethical barriers to a randomized experiment are present [4]. Besides, randomization does not always diminish the need for matching to reduce the influence of confounding variables [3], especially when the sample size is small [6]. Selection bias may still affect the results in properly randomized trials [7].

We investigated the preventive effect of a parent management training (PMT), the Incredible Years Parent Program (IY), BASIC [8] and ADVANCE [9], in preschool children who were at risk of the development of Disruptive Behavior Disorders (DBD). DBD is a term which covers both Oppositional Defiant Disorder (ODD) and Conduct Disorder (CD). These disorders are characterized by a persistent and pervasive pattern of antisocial behavior, including disobedience, tantrums, lying, destructiveness and stealing [10]. The IY parent training program aims to reduce the aggressive behavior of children by improving the parenting skills of their mothers and fathers. The therapeutic effect of this IY program in young children with ODD and CD has been shown in several studies [11], [12], [13]. Less evidence is provided for the preventive effect; this has been shown in a specific context only, i.e., Head Start [14], [15].

In the present study randomization of families was not feasible because of geographical and motivational reasons. The families lived scattered over 1449 km2 in the province of Utrecht, The Netherlands. This province consists of 29 clearly bounded cities, towns and intermediate agricultural areas and has 1.16 million inhabitants [16]. As motivation to participate is a recurrent problem in intervention studies, especially when families of children with conduct problems are involved [17], we wanted to make it as easy as possible for families to participate. It has been shown (e.g. [18]) that offering a preventive intervention for preschool children with disruptive behavior in a hospital results in a low attendance rate; less than half of the participants attended at least 50% of the sessions. To avoid this, we have chosen to deliver the IY program at four different sites which are within 15 km distance from the consenting families’ homes and which are also easy accessible, such as community centers. Moreover, the IY program requires at least 6 parents to participate in a parent group to optimize discussion and to foster a sense of support [8]. Consequently, the location of the IY program had to be close to the homes of the parents and sufficient parents had to live in the same area to form a group. In addition, parents in the control group had to be blind to their condition; they were not informed of the fact that the other group received parent management training. Therefore, to prevent the two groups from running into each other, control participants had to live at a considerable distance from the participants in the intervention group, preferably in another town or city.

Because randomization was not feasible, we have chosen to use a case control design. According to the Standards of Evidence given by the Society for Prevention Research [19], use of a case control design is permitted “as long as assignment was not by self-selection, but instead by some other factor (for instance geography)”. Also according to these standards, a case control design “is credible with demonstrated pretest equivalence using adequately powered tests on multiple baselines or pretests of multiple outcomes and important covariates”. This is necessary “to maximize confidence that the intervention, rather than some other alternative explanation causes reported outcomes”. Thus, variables which might not be equally distributed among the two conditions and which may have an effect on the outcome need to be controlled. A matching procedure can be used to remove the “overt bias” between treatment and control groups. Bias caused by “selection on unobservables” cannot be removed by matching, except to the extent that it is correlated with the observed variables, so in the remaining article we assume that “selection is on observables”. This assumption has been given different names, such as “unconfoundedness” and “ignorable treatment assignment” (for an exact mathematical description, see [20]). Procedures other than matching can also be used to ensure that effects found are due to the intervention, such as cluster-randomization or intention-to-treat-estimation.

In our study, the participating families were matched on six characteristics which have been proven to affect either the developmental course of the child's disruptive behavior or to be a moderator of treatment effect. These characteristics are the child's gender [21], [22], level of aggression [23], [24], IQ [25], [26], the parents' educational level [27], [28], stress level [29], [30], and address density of the place of residence of the family [31], [32]. Equally distributing these characteristics over the two groups will result in a minimization of the effect of these possible confounding variables. Results found will be mainly due to the intervention, with the exception of effects due to unobserved covariates.

In this study, we aimed to evaluate the pretest equivalence for randomization and matching by simulating the division into groups. However, in this study the required data were only available after the participants were divided into groups. We assessed the equivalence of the groups post hoc, i.e., we determined the difference in expected balance of the six characteristics, between two predefined groups, and the expected balance between two randomized groups. To calculate this expected balance between groups, we simulated a large number of predefined groups and randomized groups, using the data sample. The objective of this study was to assess the performance of pairwise matching on our data sample by simulating predefined and randomized groups and comparing the equivalence of predefined groups with the equivalence of randomized groups.

Section snippets

Participants

Out of a population-based sample of children from the province of Utrecht, the Netherlands (N = 8632), acquired by the office for screening and vaccination, 509 4-year-old children with a score at or above the 80th percentile on the Aggressive Behavior scale of the Child Behavior Check List 11/2–5 ([33]; Dutch version by Verhulst & Van der Ende) were considered to be at risk for DBD. Families were divided in a group in which the parents would participate in the IY program, delivered in 4

Results

We assessed the difference in balance for both the predefined and the randomized groups using the following settings: 20,000 simulations, with 75 best matches to store for every intervention family. In all simulations, all 74 intervention families were matched with exactly one control family. The distribution of the difference in matching distance (DiffV) over the 20000 simulations is shown in Fig. 1. Note that the differences in distance (depicted on the horizontal axis) are determined

Discussion

The objective of this study was to assess the performance of pairwise matching on our data sample by simulating predefined and randomized groups and comparing the equivalence of predefined groups with the equivalence of randomized groups. Findings revealed that matching using our predefined groups leads to a more equally balanced distribution of the six key characteristics than randomization in 34% of the simulated trials (with 50% of the trials as theoretical maximum). In the remaining 66% of

Acknowledgments

The original research project was funded by ZonMw Prevention (#2620.0001). The authors wish to express their thanks to Edwin Martens for his helpful comments on this manuscript.

References (42)

  • C. Webster-Stratton

    The incredible years: parents and children videotape series: a parenting course (BASIC)

    (2001)
  • C. Webster-Stratton

    The incredible years: parents and children videotape series: a parenting course (ADVANCE)

    (2002)
  • American Psychiatric Association

    Diagnostic and statistical manual of mental disorders

    (2002)
  • C. Webster-Stratton et al.

    Treating children with early-onset conduct problems: a comparison of child and parent training interventions

    J Consult Clin Psychol

    (1997)
  • S. Scott et al.

    Multicentre controlled trial of parenting groups for childhood antisocial behavior in clinical practice

    BMJ

    (2001)
  • M.J. Reid et al.

    Halting the development of conduct problems in Head Start children: the effects of parent training

    J Clin Child Adolesc Psychol

    (2004)
  • C. Webster-Stratton

    Preventing conduct problems in Head Start children: strengthening parenting competencies

    J Consult Clin Psychol

    (1998)
  • Centraal Bureau voor de Statistiek (CBS)

    Demografische Kerncijfers per Gemeente 2004 (Demographic figures by Municipality)

    (2004)
  • E.S.L. Luk et al.

    Children with persistent conduct problems who dropout of treatment

    Eur Child Adolesc Psychiatry

    (2001)
  • R.A. Barkley et al.

    Multi-method psycho-educational intervention for preschool children with disruptive behavior: preliminary results at post-treatment

    J Child Psychol Psychiatry

    (2000)
  • Society for Prevention Research

    Standards of evidence: criteria for efficacy, effectiveness and dissemination

    (2005)
  • Cited by (12)

    • Plant beneficial rhizospheric microorganism (PBRM) strategies to improve nutrients use efficiency: A review

      2017, Ecological Engineering
      Citation Excerpt :

      Since the elaboration of the rhizosphere concept by Lorenz Hiltner in 1904, many studies have reported that the soil environment attached to the root system is a hot spot of microbial abundance and activity due to the presence of root exudates and rhizodeposits (Kumar et al., 2014; Meena et al., 2016a). Some rhizosphere microorganisms may be neutral or deleterious in regard to plant growth, whereas other microbes support their hosts (Raaijmakers et al., 2008; Singh et al., 2010). Such PBRMs can stimulate plant growth, increase yield, reduce pathogen infection, as well as reduce biotic or abiotic plant stress, without conferring pathogenicity (Lugtenberg and Kamilova, 2009; Singh et al., 2010).

    • Impact of a medication management system on nursing home admission rate in a community-dwelling nursing home-eligible medicaid population

      2011, American Journal Geriatric Pharmacotherapy
      Citation Excerpt :

      Consequently, results may be attributable to factors other than the intervention. Research comparing randomized versus nonrandomized studies has shown that the use of matching in nonrandomized studies, as done in this study, can produce study groups with similar distributions of baseline covariates, a strength of traditional randomized studies.36,37 Clients were not randomly selected within pharmacies because of the clear danger of contamination between clients.

    View all citing articles on Scopus
    View full text