The Effects of Lottery Incentive and Length of Questionnaire on Health Survey Response Rates: A Randomized Study

doi:10.1016/S0895-4356(99)00051-7

Journal of Clinical Epidemiology

Volume 52, Issue 11, November 1999, Pages 1117-1122

https://doi.org/10.1016/S0895-4356(99)00051-7 Get rights and content

Abstract

Maximizing the response rate of self-administered questionnaires is key in survey research. We aimed to evaluate the effects of lottery incentive and length of questionnaire on health survey response rates when used in isolation or combined. A random sample of 440 residents in Western Sydney, Australia was randomly allocated to four equal groups to receive or not receive an instant lottery ticket and a long (seven page) or short (one page) questionnaire. The overall response rate was 71.8%. The final response rates were higher among those receiving the short, rather than the long, questionnaire (75.6% versus 68.2%) (P = 0.08); and among those receiving the lottery incentive compared with those not receiving the incentive (75% versus 68.2%) (P = 0.09). By logistic regression analysis, the success of obtaining a completed questionnaire without any follow-up reminders was significantly associated with the lottery incentive but not the questionnaire length (P = 0.03 and P = 0.54, respectively). The difference between lottery and no lottery groups decreased gradually during the follow-up. A lottery incentive is associated with an increased response after the first mailing. A small up-front cost for a lottery ticket may be worthwhile, since it can save further costs by obviating the need for repeated follow-ups.

Introduction

Self-administered questionnaires are an important means of acquiring health information. They allow respondents to answer questions about sensitive subjects in their own environment with adequate time to reflect on their responses, and are economical compared with telephone surveys or personal interviews [1]. Traditionally, a major disadvantage of mail surveys has been a low response rate, which potentially impairs study validity [2]. High response rates are necessary to minimize non-responder bias [3], although even with response rates above 70% bias may still be present 4, 5.

A number of techniques have been applied in order to maximize the response rate. A meta-analysis indicated that repeated contacts, appeals, inclusion of a return envelope, paid return postage, shorter survey forms, and monetary incentives were effective in increasing survey response rates [6]. Survey topic may also be important, as sensitive topics can elicit poorer response rates [7].

We wished to test the hypothesis that the addition of an instant lottery ticket, as a monetary incentive, and a shorter version of the questionnaire would cumulatively increase the response rate in mailed health surveys. Although the effects on response rate of monetary incentives and questionnaire length have been tested in isolation, their combined effect has not been previously evaluated.

Section snippets

Methods

This study was approved by the Wentworth Area Health Services Ethics Committee. The sample was drawn randomly from the electoral rolls of Lindsay, a suburb in Western Sydney, Australia. Since voting is compulsory and universal for all citizens 18 years and older in Australia, the Electoral Commission in each state routinely registers and updates the lists of citizens in respective areas.

The survey consisted of a self-administered questionnaire sent out to 440 residents between April 1997 and

Results

Of the initial 440 questionnaires sent out, 17 questionnaires were returned as “not known at this address” at different stages of the mailing and were replaced. In total, seven surveys were not returned (address changed, language barrier discovered during the telephone follow-up), giving an effective sample of 433 residents. Overall, 311 completed questionnaires were returned for an eventual response rate of 71.8%.

At the telephone follow-up stage, 144 names were sought in the White pages on

Discussion

There have been many attempts to devise optimal strategies to increase response rates of mailed surveys. High response rates reduce non-response bias considerably but may not completely eliminate it [4]. Quantitative reviews of individual studies, using meta-analytic techniques suggested by Glass [9] have attempted to address the limitations of a diverse range of previous studies by examining the effects of factors hypothesized to affect response rates and by assessing these across a number of

Acknowledgements

We are indebted to Mr. Stuart Howell for his assistance with the statistical analyses. This work was supported by the National Health and Medical Research Council (NH&MRC), Australia.

References (24)

S Chapman et al.
Incentives for questionnaire respondents
Aust J Public Health
(1991)
C Moser et al.
Survey Methods in Social Investigation
(1972)
D.A Dillman
Mail and Telephone SurveysThe Total Design Method
(1975)
J Siemiatycki et al.
Non-response bias and early versus all responders in mail and telephone surveys
Am J Epidemiol
(1984)
E Guadagnoli et al.
The effects of non-response and late response on a survey of physician attitudes
Evaluation Health Professions
(1989)
S.H Berry et al.
Physician response to a mailed surveyAn experiment in timing of payment
Public Opinion Quarterly
(1987)
F.J Yammarino et al.
Understanding mail survey response behaviorA meta-analysis
Public Opinion Quarterly
(1991)
C Dommeyer
Does response to an offer of mail survey results with questionnaire interest?
Journal Market Research Society
(1985)
SPSS for Windows, SPSS Inc....
G Glass et al.
Meta-analysis in Social Research
(1981)

J.C Goyder

Further evidence on factors affecting response rates to mailed questionnaires

American Sociological Review

(1982)

B.R Worthen et al.

Relative effectiveness of personalized and form covering letters in initial and follow-up mail surveys

Psychological Reports

(1985)

Cited by (109)

The Single Assessment Numeric Evaluation (SANE) after anterior cervical discectomy and fusion: A pilot study
2021, Journal of Clinical Neuroscience
Retrospective cohort.
To evaluate the single assessment numerical evaluation (SANE) as a patient reported outcomes measure (PROM) after anterior cervical discectomy and fusion (ACDF), by comparing to legacy measures.
We included all patients undergoing ACDF with at least one year of follow up with complete PROM data. Patients completed the Neck Disability Index (NDI), the RAND-36 and the EuroQual Five Dimension (EQ-5D) scale, as well as the one-question SANE, pre- and post-operatively. Validity of SANE compared with other PROMs was determined utilizing Pearson’s correlation (ρ), proportional bias (B), responsiveness, minimal clinically important difference (MCID) and agreement.
Sixty-nine patients were included. There were moderate-to-strong correlations at a minimum of one-year follow-up between the SANE and NDI (ρ = −0.73, P < 0.0001), RAND (ρ = 0.80, P < 0.0001), and EQ-5D (ρ = −0.66, P < 0.0001). No significant proportional bias was found for the SANE when compared to the RAND (B = 0.03, p = 0.99), NDI (B = −0.003, p = 0.99), or EQ-5D (B = −0.0007, p = 0.99). Responsiveness for SANE was statistically similar to all other PROMs. The MCID for SANE was determined to be 10.5, with 42% of patients achieving the MCID. Bland-Altman plots demonstrated high agreement between all PROMs.
We found the SANE score provides clinically important patient outcomes data after ACDF, despite only requiring answering one question. The SANE performs comparably to more burdensome health questionnaires. The SANE score may offer spine surgeons the option to easily and quickly collect clinically relevant data on their surgical patients.
A SANE Approach to Outcome Collection? Comparing the Performance of Single- Versus Multiple-Question Patient-Reported Outcome Measures After Total Hip Arthroplasty
2020, Journal of Arthroplasty
Several patient-reported outcome measures (PROMs) exist to measure outcomes after total hip arthroplasty (THA) but can be limited by patient-perceived burden and completion rates. We analyzed whether the modified single assessment numerical evaluation (M-SANE), a one-question PROM, would perform similarly to multiple-question PROMs among patients undergoing primary THA.
Patients undergoing THA completed the Patient-Reported Outcomes Measurement Information System-10 (PROMIS-10), the Hip Disability and Osteoarthritis Outcomes Score Junior (HOOS-Jr), and M-SANE questionnaires both preoperatively and postoperatively. The M-SANE assessment asked patients to assess their hip on a scale from 0 to 10, with 10 being the best possible score. Validity of M-SANE compared with other PROMs was determined by Spearman’s correlation and floor and ceiling effects. Responsiveness was analyzed using standardized response mean (SRM).
One hundred and thirty six patients with at least 1-year follow-up were reviewed. The average M-SANE score improved from 3.3 preoperatively to 7.1 at one year postoperatively. There was moderate to strong correlation at one-year follow-up between the M-SANE and HOOS-Jr (ρ = 0.75, P < .001) and PROMIS-10 physical component summary (ρ = 0.63, P < .001). Floor and ceiling effects of the M-SANE (floor 2.0%, ceiling 21.3%) were comparable to the HOOS-Jr (floor 0.0%, ceiling 20.8%). The responsiveness of the M-SANE after THA (SRM = 1.06, 95% CI: 0.79-1.33) was comparable to HOOS-Jr (SRM = 1.33, 95% CI: 1.08-1.59) and superior to PROMIS-10 physical component summary (SRM = 0.65, 95% CI: 0.55-0.74).
The M-SANE has performed similarly across multiple psychometric properties compared with more burdensome PROMs in assessing longitudinal patient-reported outcomes after THA.
Behavioral economics implementation: Regret lottery improves mHealth patient study adherence
2019, Contemporary Clinical Trials Communications
Citation Excerpt :
Our findings add important new context to the literature on the use of lotteries to incentivize study participants. Previous studies on the effectiveness of lotteries to improve study adherence have largely focused on increasing response rates to one-time surveys [13–15,18,19]. Our use case differed in that the goal was to increase adherence to a long-term, intensive study protocol requiring daily participation over 52-weeks, and thus suggests an expanded role of lotteries in human subjects research.
Nonadherence to study protocols reduces the generalizability, validity, and statistical power of longitudinal studies.
To determine whether an automated electronically-delivered regret lottery would improve adherence to an intensive mHealth self-monitoring protocol as part of a longitudinal observational study.
We enrolled 77 adults into a 52-week study requiring five daily ecologic momentary assessments (EMA) of stress and daily accelerometer use. We performed a pre/post single-arm study to evaluate the efficacy of a lottery intervention in improving adherence to this protocol. Midway through the study, participants were invited to enter a weekly regret lottery ($50 prize, expected value <$1) in which prize collection was contingent upon meeting adherence thresholds for the prior week. Study protocol adherence before and after lottery initiation were compared using mixed models repeated measures analysis of variance.
62 participants consented to lottery participation. In the 12 weeks prior to lottery initiation, weekly adherence was declining (slope −1.4%/week). The weekly per-participant probability of adherence was higher after lottery initiation when comparing the 4-week (32% pre-lottery vs 50% post-lottery, p < 0.001), 8-week (37% vs 49%, p < 0.001), and 12-week periods (39% vs 45%, p = 0.001) before and after lottery initiation. However, the rate of decline in adherence over time was unchanged.
The implementation of an automated, electronically-delivered weekly regret lottery improved adherence with an intensive self-monitoring study protocol. Regret lotteries may represent a cost-effective tool to improve adherence and reduce bias caused by dropout or nonadherence.
Most response-inducing strategies do not increase participation in observational studies: a systematic review and meta-analysis
2018, Journal of Clinical Epidemiology
To evaluate response-inducing strategies for observational studies using health-related questionnaires or interviews.
We searched PubMed, EMBASE, CINAHL, PsycINFO, and Web of Science up to December 28, 2017. Studies evaluating the effect of a response-inducing strategy on participation rates of observational studies were included. For each strategy, we estimated pooled response ratios with 95% confidence intervals (CIs) in a Hartung-Knapp/Sidik-Jonkman random effects model with the final participation rate as outcome, stratified for type of participants and method of data collection.
The search yielded 168 eligible studies involving 367,616 potential participants and 33 strategies. Among patients, response-inducing strategies for paper-based questionnaires included unconditional monetary incentives (response ratio 1.15; 95% CI 1.09–1.21) and shorter questionnaires (1.04; 1.02–1.06). Among nonpatients, a personalized mode of delivery (1.47; 1.24–1.74), more expensive mailing type (1.25; 1.00–1.56), unconditional monetary incentives (1.24; 1.12–1.38), prenotification (1.12; 1.03–1.22), unconditional scratch lottery tickets (1.09; 1.01–1.18), and shorter questionnaires (1.06; 1.02–1.11) increased response rates to paper-based questionnaires. For Web-based questionnaires and interviews among nonpatients, response rates were increased by conditional lottery tickets (1.17; 1.02–1.34) and conditional monetary incentives (1.39; 1.01–1.91), respectively.
Although the majority of strategies evaluated were unsuccessful, some may increase response rates to observational studies, particularly among nonpatients.
Users evaluate a detailed familial risk questionnaire as valuable and no more time consuming than a simple enquiry in a web-based diabetes risk assessment tool
2016, Public Health
On-farm characteristics and biosecurity protocols for small-scale swine producers in eastern Australia
2015, Preventive Veterinary Medicine
Pigs are considered high risk for the introduction and spread of foot and mouth disease (FMD) in Australia. Facilities where animals from different origins are commingled, such as saleyards, pose a high risk for disease spread. Sound on-farm management practices and biosecurity protocols are the first line of defence against a potential on-farm disease outbreak. This study evaluated the practices of 104 producers (vendors who sold pigs and purchasers of live pigs for grow-out) who traded pigs at 6 peri-urban and rural saleyards in eastern Australia. Specifically, management and on-farm biosecurity practices were assessed using an in-depth questionnaire. Univariable and multivariable logistic regression analyses were used to investigate (1) producer associations: producer type, State, motivation to keep pigs, farm type, gender, years having owned pigs, and the acquisition of formal livestock qualifications; and (2) pig associations: herd size, housing, management (husbandry and feeding) practices and biosecurity (including pig movement) practices. Backyard operations (<20 sows) were undertaken by 60.6% of participants, followed by small-scale pig operations (28.8%; 21–100 sows). Few producers (16.3%) reported residing in close proximity (<5 km) to commercial operations; however, less rural producers had neighbouring hobby pig operations within 5 km of their property (P = 0.033). Motivation for keeping pigs was significantly associated with a number of biosecurity practices. Producers who kept pigs for primary income were more likely to provide footwear precautions (P = 0.007) and ask visitors about prior pig contacts (P = 0.004). Approximately 40% of backyard and small-scale producers reported not having any quarantine practices in place for incoming pigs, compared to only 9.1% among larger producers. The main reasons cited for not adopting on-farm biosecurity practices in this study included having no need on their property (43.1%) and a lack of information and support (by the industry and/or authorities; 18.5%). Up to three-quarters of all producers maintained an open breeding herd, regularly introducing new pigs to the main herd. Saleyards are an important source of income for backyard and small-scale producers as well as an important risk factor for the introduction and dissemination of endemic and emerging animal diseases. Differing management and biosecurity practices as well as the motivations of these producers keeping pigs in small numbers and trading pigs at saleyards need to be taken into account in the development of successful biosecurity extension programmes for this sector of the Australian pork industry.

View all citing articles on Scopus

View full text

Original articlesThe Effects of Lottery Incentive and Length of Questionnaire on Health Survey Response Rates: A Randomized Study

Abstract

Introduction

Section snippets

Methods

Results

Discussion

Acknowledgements

Aust J Public Health

Survey Methods in Social Investigation

Mail and Telephone SurveysThe Total Design Method

Non-response bias and early versus all responders in mail and telephone surveys

Am J Epidemiol

The effects of non-response and late response on a survey of physician attitudes

Evaluation Health Professions

Physician response to a mailed surveyAn experiment in timing of payment

Public Opinion Quarterly

Understanding mail survey response behaviorA meta-analysis

Public Opinion Quarterly

Does response to an offer of mail survey results with questionnaire interest?

Journal Market Research Society

Meta-analysis in Social Research

Further evidence on factors affecting response rates to mailed questionnaires

American Sociological Review

Relative effectiveness of personalized and form covering letters in initial and follow-up mail surveys

Psychological Reports

Original articles
The Effects of Lottery Incentive and Length of Questionnaire on Health Survey Response Rates: A Randomized Study