Validity of Self-reported Healthcare Utilization Data in the Community Health Survey in Korea

Identifying the current status of health indices sets the foundation for a series of processes in health programs, including selection of program priorities, the establishment of purposes and goals, and the development of strategy, program implementation, and evaluation. Efforts to identify the health status of the nation were initiated with the 'National Health Survey' in 1962 and the 'National Nutrition Survey' in 1969. Currently the 5th 'Korean National Health and Nutrition Examination Survey (KNHANES)' is in progress. As KNHANES considers only 16 wide-area self-governing districts, this nationwide sampling design has failed to cover the actual current status of 253 basic regional self-governing districts in which health programs are executed (1).

To overcome this limitation and to create health indices that are comparable among regions, the Community Health Survey (CHS) has been conducted annually since 2008 in individuals aged ≥ 19 yr in 253 regions around the country (2). The CHS has a two-stage sampling process. The first sampling stage is the process of selecting a sample area (tong/ban/ri) that is a primary sample unit, selected according to the number of households in dong/eup/myeon (the smallest administrative unit) using a probability proportional to the sampling method. In the second sampling stage, as a process of selecting sample households, the number of households in the selected sample tong/ban/ri is identified to create a household directory, and sample households are selected through systematic sampling methods. These methods are used to ensure the sample units are representative.

To evaluate reliability of CHS, a third organization conducts a telephone survey in 5% of the respondents 2 weeks after the interview. This identified how well the responses match. In fact, the kappa values were, on average, 0.68 (0.51 to 0.90) in 2008 and averaged 0.80 (0.70 to 0.90) in 2009 (3, 4).

However, the validity of the CHS has not yet been assessed. We evaluated the accuracy of self-reported CHS obtained by analysis of both sensitivity and specificity for some healthcare utilization.

The sensitivities for admission, outpatient visits, and dental visit were considerably lower than the specificities. Additionally, the sensitivity of admissions and outpatient visits were similar, despite the different utilization periods. Maybe, it seems likely that the respondents feel pressured to provide direct information regarding their medical care use, and not because they failed to remember. Although the sensitivities showed significant differences according to gender, age, marital status, and education in individual question, there were no variables with consistent pattern on sensitivity.

However, the sensitivity of dental visits was higher than those of admission and outpatient visits and the specificity was lower than those of admission and outpatient visits. The reason for the higher sensitivity appears that the respondents remembered the dental visits as a more specialized event than admission or outpatient visits. This may be also due to the methods of obtaining the answers, which differed, with the admission and outpatient answers being only yes/no, versus the dental visits, which had six options for the visit period. On the contrary, reasons for the lower specificity appear to be the dental clinic covers more non-benefit items than other general medical areas, meaning the dental clinic cannot claim insurance benefits for all their treatments. In addition, the respondents can use all healthcare institutions with their Industrial Accident Compensation Insurance, car-insurance, and non-insurance so this could lead to a higher false positive value for outpatient visits and dental visits.

A questionnaire survey is one of the data sources used to identify health behavior, health status, and healthcare utilization in a population, but its accuracy is known to be relatively low, when compared to either medical record review or insurance claim data (5-7). The accuracy of self-reported data in healthcare utilization depends, among other factors, on the subjects' knowledge and understanding of the relevant information, their ability to recall information, and their willingness to report it (8).

The range of sensitivity was 86-96% and the range of specificity was 77%-89% with a 6-48 month follow-up period in previous prospective studies that investigated the validity of data on self-reported dental visits in the Florida Dental Care Study (9). Among elderly aged ≥ 70 yr, the kappa value of hospital episodes and physician visits was 0.767 and 0.255, respectively, within the past year (10). Using the data from the 2001 Taiwan National Health Interview and the National Health Insurance, the crude kappa values for inpatients (within the past year), outpatients (within the past month), and dental visits (within the past month) were 0.75, 0.64, and 0.62, respectively (11). These values are higher than those reported in this study and may be explained by the discrepancies between the study design, questionnaire content, and the method of results presentation. In fact, the kappa values on admission, outpatients visits and dental visits in this study were 0.57, 0.40 and 0.44, respectively, although agreement rate of those were 90.9%, 73.4%, and 76.1% respectively.

Consequently, based on these study results, increasing the sensitivity of the respondents requires effort in the process of CHS. First, the recall time can be subdivided into periods of 1 month, 2-6 months, and 7 months to 1 yr for admissions, and 1 week, 2 weeks, and 3 4 weeks for outpatient visits, and reply methods can be changed from healthcare institute utilization (Yes/No) to the utilization period, like those used for the dental visits. Indeed, it has been reported that an optimal recall period for self-reported surveys occurs at 6 months or less (6). Second, when healthcare utilization questions are placed at the end of a long survey, the participant may want to finish the survey and not invest the time and effort needed for an accurate recall (12). Third, applying the correct probing methods for the respondents is required when obtaining information from the past to the present. For example, it has been reported that a 6-2 month approach, where the past 6-month utilization questions are asked initially, is more accurate than a 2-6 month approach (13).

The study has some limitations. First, is it right to use the insurance benefit claim data of the HIRA as the gold standard for validity test? Although NHI did not cover all healthcare utilization in Korea, NHI is a social insurance whereby all Korean take out a policy for medical security as a form of health insurance (96.7%) or medical aid (3.3%). In addition, 89.2% of an insured person visited the healthcare institution as admission or outpatient more than once in 2009 (14). That is, the gap which is about 10.8% included some people who did not use the healthcare institution, some people who used the healthcare institution by other insurance such as life-insurance, car-insurance, Industrial Accident Compensation Insurance, etc or some people who used the healthcare institution is not covered by any insurance. But we did not know exact amount of coverage of other insurance or non-insurance. Anyway, considering the amount of visit to the healthcare institution covered by NHI and access to data which we used as gold standard, we thought that using the insurance benefit claim data of the HIRA is the proper approach. Second, why did we use only 3 items, admission, outpatient visit, and dental visit for validity evaluation of CHS? Why did not we use more detailed healthcare utilization question such as outpatient visit by hypertension, admission by stroke, etc. In fact, we can evaluate the validity of specific utilization of CHS and compare CHS with HIRA about the accuracy of diagnosis. But they are the topics of the next study. In this study, we just want to evaluate the validity of CHS irrespective of specific disease, from a conservative point of view. Third, this study was a validity test conducted only for one broad area (a self-governing district); as a result, we did not analyze the regional difference of sensitivities and specificities and the study results should not be generalized regarding the validity of the entire CHS. This limitation could be overcome with validity testing for all respondents of the 2008 CHS (about 220,000 participants). In fact, the sensitivities and specificities of admission, outpatient visits and dental visits among 13 cities and counties of Chungbuk-do were very diverse. These were as follows: range of sensitivity and specificity of admission was 38.9%-67.7% and 94.1%-97.5%, range of sensitivity and specificity of outpatient visits was 31.2%-66.8% and 78.0%-92.0%, and range of sensitivity and specificity of admission was 39.2%-74.5% and 75.6%-88.0%. It needs the further task which contains a multilevel study to explain the regional variation of CHS validity.

In conclusion, despite some limitations, the present study is the first to evaluate the validity of nationwide health statistics resulting from questionnaire surveys and shows that CHS needs a lot of efforts to reflect the true health status, health behavior, and healthcare utilization of the population.