Assessing performance of a randomized versus a non-randomized study design
Introduction
To evaluate the effectiveness of psychotherapeutic and pharmacotherapeutic treatment, randomized controlled trials have become a ‘golden standard’ [1], [2], [3]. Since participants are allocated to treatment group by chance, randomization minimizes the differences among groups and ensures approximate balance for both observable and unobservable covariates. As a consequence, there will be less confounding factors which might affect the intervention-effect and differences between groups can thus be attributed to the treatment received [4].
However, reality sometimes complicates the process of randomization, or even makes it impossible to use this strong experimental design [5], [2]. As a consequence, alternatives to randomization have been developed, for example quasi-experimental and case control designs. These kinds of designs can be used when political, practical or ethical barriers to a randomized experiment are present [4]. Besides, randomization does not always diminish the need for matching to reduce the influence of confounding variables [3], especially when the sample size is small [6]. Selection bias may still affect the results in properly randomized trials [7].
We investigated the preventive effect of a parent management training (PMT), the Incredible Years Parent Program (IY), BASIC [8] and ADVANCE [9], in preschool children who were at risk of the development of Disruptive Behavior Disorders (DBD). DBD is a term which covers both Oppositional Defiant Disorder (ODD) and Conduct Disorder (CD). These disorders are characterized by a persistent and pervasive pattern of antisocial behavior, including disobedience, tantrums, lying, destructiveness and stealing [10]. The IY parent training program aims to reduce the aggressive behavior of children by improving the parenting skills of their mothers and fathers. The therapeutic effect of this IY program in young children with ODD and CD has been shown in several studies [11], [12], [13]. Less evidence is provided for the preventive effect; this has been shown in a specific context only, i.e., Head Start [14], [15].
In the present study randomization of families was not feasible because of geographical and motivational reasons. The families lived scattered over 1449 km2 in the province of Utrecht, The Netherlands. This province consists of 29 clearly bounded cities, towns and intermediate agricultural areas and has 1.16 million inhabitants [16]. As motivation to participate is a recurrent problem in intervention studies, especially when families of children with conduct problems are involved [17], we wanted to make it as easy as possible for families to participate. It has been shown (e.g. [18]) that offering a preventive intervention for preschool children with disruptive behavior in a hospital results in a low attendance rate; less than half of the participants attended at least 50% of the sessions. To avoid this, we have chosen to deliver the IY program at four different sites which are within 15 km distance from the consenting families’ homes and which are also easy accessible, such as community centers. Moreover, the IY program requires at least 6 parents to participate in a parent group to optimize discussion and to foster a sense of support [8]. Consequently, the location of the IY program had to be close to the homes of the parents and sufficient parents had to live in the same area to form a group. In addition, parents in the control group had to be blind to their condition; they were not informed of the fact that the other group received parent management training. Therefore, to prevent the two groups from running into each other, control participants had to live at a considerable distance from the participants in the intervention group, preferably in another town or city.
Because randomization was not feasible, we have chosen to use a case control design. According to the Standards of Evidence given by the Society for Prevention Research [19], use of a case control design is permitted “as long as assignment was not by self-selection, but instead by some other factor (for instance geography)”. Also according to these standards, a case control design “is credible with demonstrated pretest equivalence using adequately powered tests on multiple baselines or pretests of multiple outcomes and important covariates”. This is necessary “to maximize confidence that the intervention, rather than some other alternative explanation causes reported outcomes”. Thus, variables which might not be equally distributed among the two conditions and which may have an effect on the outcome need to be controlled. A matching procedure can be used to remove the “overt bias” between treatment and control groups. Bias caused by “selection on unobservables” cannot be removed by matching, except to the extent that it is correlated with the observed variables, so in the remaining article we assume that “selection is on observables”. This assumption has been given different names, such as “unconfoundedness” and “ignorable treatment assignment” (for an exact mathematical description, see [20]). Procedures other than matching can also be used to ensure that effects found are due to the intervention, such as cluster-randomization or intention-to-treat-estimation.
In our study, the participating families were matched on six characteristics which have been proven to affect either the developmental course of the child's disruptive behavior or to be a moderator of treatment effect. These characteristics are the child's gender [21], [22], level of aggression [23], [24], IQ [25], [26], the parents' educational level [27], [28], stress level [29], [30], and address density of the place of residence of the family [31], [32]. Equally distributing these characteristics over the two groups will result in a minimization of the effect of these possible confounding variables. Results found will be mainly due to the intervention, with the exception of effects due to unobserved covariates.
In this study, we aimed to evaluate the pretest equivalence for randomization and matching by simulating the division into groups. However, in this study the required data were only available after the participants were divided into groups. We assessed the equivalence of the groups post hoc, i.e., we determined the difference in expected balance of the six characteristics, between two predefined groups, and the expected balance between two randomized groups. To calculate this expected balance between groups, we simulated a large number of predefined groups and randomized groups, using the data sample. The objective of this study was to assess the performance of pairwise matching on our data sample by simulating predefined and randomized groups and comparing the equivalence of predefined groups with the equivalence of randomized groups.
Section snippets
Participants
Out of a population-based sample of children from the province of Utrecht, the Netherlands (N = 8632), acquired by the office for screening and vaccination, 509 4-year-old children with a score at or above the 80th percentile on the Aggressive Behavior scale of the Child Behavior Check List 11/2–5 ([33]; Dutch version by Verhulst & Van der Ende) were considered to be at risk for DBD. Families were divided in a group in which the parents would participate in the IY program, delivered in 4
Results
We assessed the difference in balance for both the predefined and the randomized groups using the following settings: 20,000 simulations, with 75 best matches to store for every intervention family. In all simulations, all 74 intervention families were matched with exactly one control family. The distribution of the difference in matching distance (DiffV) over the 20000 simulations is shown in Fig. 1. Note that the differences in distance (depicted on the horizontal axis) are determined
Discussion
The objective of this study was to assess the performance of pairwise matching on our data sample by simulating predefined and randomized groups and comparing the equivalence of predefined groups with the equivalence of randomized groups. Findings revealed that matching using our predefined groups leads to a more equally balanced distribution of the six key characteristics than randomization in 34% of the simulated trials (with 50% of the trials as theoretical maximum). In the remaining 66% of
Acknowledgments
The original research project was funded by ZonMw Prevention (#2620.0001). The authors wish to express their thanks to Edwin Martens for his helpful comments on this manuscript.
References (42)
- et al.
Psychotherapy in controlled psychopharmacology trials. Does it matter if we ignore it?
Contemp Clin Trials
(2005) - et al.
Ensuring the comparability of comparison groups: is randomization enough?
Control Clin Trials
(2004) - et al.
A comparison of eclectic treatment with Webster–Stratton's parents and children series in a children's mental health center: a randomized controlled trial
Behav Ther
(1998) - et al.
Group parent training: is it effective for children of all ages?
Behav Ther
(1996) - et al.
Increased rate of conduct problems in urban areas: what is the mechanism?
J Am Acad Child Adolesc Psych
(1996) Handbook of clinical trial and epidemiological research designs
(2000)- et al.
Annotation: randomised trials
J Child Psychol Psychiatry
(2002) - et al.
Research designs for studies evaluating the effectiveness of change and improvement strategies
Qual Saf Health Care
(2003) - et al.
Evidence based medicine and evaluation of mental health services: methodological issues and future directions
Arch Dis Child
(1999) Random sampling, randomization, and equivalence of contrasted groups in psychotherapy outcome research
J Consult Clin Psychol
(1989)
The incredible years: parents and children videotape series: a parenting course (BASIC)
The incredible years: parents and children videotape series: a parenting course (ADVANCE)
Diagnostic and statistical manual of mental disorders
Treating children with early-onset conduct problems: a comparison of child and parent training interventions
J Consult Clin Psychol
Multicentre controlled trial of parenting groups for childhood antisocial behavior in clinical practice
BMJ
Halting the development of conduct problems in Head Start children: the effects of parent training
J Clin Child Adolesc Psychol
Preventing conduct problems in Head Start children: strengthening parenting competencies
J Consult Clin Psychol
Demografische Kerncijfers per Gemeente 2004 (Demographic figures by Municipality)
Children with persistent conduct problems who dropout of treatment
Eur Child Adolesc Psychiatry
Multi-method psycho-educational intervention for preschool children with disruptive behavior: preliminary results at post-treatment
J Child Psychol Psychiatry
Standards of evidence: criteria for efficacy, effectiveness and dissemination
Cited by (12)
Plant beneficial rhizospheric microorganism (PBRM) strategies to improve nutrients use efficiency: A review
2017, Ecological EngineeringCitation Excerpt :Since the elaboration of the rhizosphere concept by Lorenz Hiltner in 1904, many studies have reported that the soil environment attached to the root system is a hot spot of microbial abundance and activity due to the presence of root exudates and rhizodeposits (Kumar et al., 2014; Meena et al., 2016a). Some rhizosphere microorganisms may be neutral or deleterious in regard to plant growth, whereas other microbes support their hosts (Raaijmakers et al., 2008; Singh et al., 2010). Such PBRMs can stimulate plant growth, increase yield, reduce pathogen infection, as well as reduce biotic or abiotic plant stress, without conferring pathogenicity (Lugtenberg and Kamilova, 2009; Singh et al., 2010).
Impact of a medication management system on nursing home admission rate in a community-dwelling nursing home-eligible medicaid population
2011, American Journal Geriatric PharmacotherapyCitation Excerpt :Consequently, results may be attributable to factors other than the intervention. Research comparing randomized versus nonrandomized studies has shown that the use of matching in nonrandomized studies, as done in this study, can produce study groups with similar distributions of baseline covariates, a strength of traditional randomized studies.36,37 Clients were not randomly selected within pharmacies because of the clear danger of contamination between clients.
Quality of Healthcare Services in Focus: The Role of Knowledge Transfer, Hierarchical Organizational Structure and Trust
2023, Knowledge Management Research and PracticeSources of anxiety among health care workers in Tehran during the COVID-19 pandemic
2022, Health Policy and PlanningRehabilitation of persons with dementia: using technology to improve participation
2021, Aging and Mental Health