Preparation of name and address data for record linkage using hidden Markov models

BACKGROUND: Record linkage refers to the process of joining records that relate to the same entity or event in one or more data collections. In the absence of a shared, unique key, record linkage involves the comparison of ensembles of partially-identifying, non-unique data items between pairs of records. Data items with variable formats, such as names and addresses, need to be transformed and normalised in order to validly carry out these comparisons. Traditionally, deterministic rule-based data processing systems have been used to carry out this pre-processing, which is commonly referred to as "standardisation". This paper describes an alternative approach to standardisation, using a combination of lexicon-based tokenisation and probabilistic hidden Markov models (HMMs). METHODS: HMMs were trained to standardise typical Australian name and address data drawn from a range of health data collections. The accuracy of the results was compared to that produced by rule-based systems. RESULTS: Training of HMMs was found to be quick and did not require any specialised skills. For addresses, HMMs produced equal or better standardisation accuracy than a widely-used rule-based system. However, acccuracy was worse when used with simpler name data. Possible reasons for this poorer performance are discussed. CONCLUSION: Lexicon-based tokenisation and HMMs provide a viable and effort-effective alternative to rule-based systems for pre-processing more complex variably formatted data such as addresses. Further work is required to improve the performance of this approach with simpler data such as names. Software which implements the methods described in this paper is freely available under an open source license for other researchers to use and improve

Assessing the Validity of Insurance Coverage Data in Hospital Discharge Records: California OSHPD Data

OBJECTIVE: To assess the accuracy of data on “expected source of payment” in the patient discharge database compiled by the California Office of Statewide Health Planning and Development (OSHPD). DATA SOURCES: The OSHPD discharge data for the years 1993 to 1996 linked with administrative data from the University of California (UC) health benefits program for the same years. The linked dataset contains records for all stays in California hospitals by UC employees, retirees, and spouses. STUDY DESIGN: The accuracy of the OSHPD data is assessed using cross-tabulations of insurance type as coded in the two data sources. The UC administrative data is assumed to be accurate, implying that differences between the two sources represent measurement error in the OSHPD data. We cross-tabulate insurance categories and analyze the concordance of dichotomous measures of health maintenance organization (HMO) enrollment derived from the two sources. PRINCIPAL FINDINGS: There are significant coding errors in the OSHPD data on expected source of payment. A nontrivial percentage of patients with preferred provider organization (PPO) coverage are erroneously coded as being in HMOs, and vice versa. The prevalence of such errors increased after OSHPD introduced a new expected source of payment category for PPOs. Measurement problems are especially pronounced for older patients. Many patients over age 65 who are still covered by a commercial insurance plan are erroneously coded as having Medicare coverage. This, combined with the fact that during the period we analyzed, Medicare HMO enrollees and beneficiaries in the fee-for-service (FFS) program are combined in a single payment category, means that the OSHPD data provides essentially no information on insurance coverage for older patients. CONCLUSIONS: Researchers should exercise caution in using the expected source of payment in the OSHPD data. While measures of HMO coverage are reasonably accurate, it is not possible in these data to clearly identify PPOs as a distinct insurance category. For patients over age 65, it is not possible at all to distinguish among alternative insurance arrangements