5 research outputs found
Understanding the Evaluation Abilities of External Cluster Validity Indices to Internal Ones
Evaluating internal Cluster Validity Index (CVI) is a critical task in clustering research. Existing studies mainly employ the number of clusters (NC-based method) or external CVIs (external CVIs-based method) to evaluate internal CVIs, which are not always reasonable in all scenarios. Additionally, there is no guideline of choosing appropriate methods to evaluate internal CVIs in different cases. In this paper, we focus on the evaluation abilities of external CVIs to internal CVIs, and propose a novel approach, named external CVI\u27s evaluation Ability MEasurement approach through Ranking consistency (CAMER), to measure the evaluation abilities of external CVIs quantitatively, for assisting in selecting appropriate external CVIs to evaluate internal CVIs. Specifically, we formulate the evaluation ability measurement problem as a ranking consistency task, by measuring the consistency between the evaluation results of external CVIs to internal CVIs and the ground truth performance of internal CVIs. Then, the superiority of CAMER is validated through a real-world case. Moreover, the evaluation abilities of seven popular external CVIs to internal CVIs in six different scenarios are explored by CAMER. Finally, these explored evaluation abilities are validated on four real-world datasets, demonstrating the effectiveness of CAMER
CUBOS: An Internal Cluster Validity Index for Categorical Data
Internal cluster validity index is a powerful tool for evaluating clustering performance. The study on internal cluster validity indices for categorical data has been a challenging task due to the difficulty in measuring distance between categorical attribute values. While some efforts have been made, they ignore the relationship between different categorical attribute values and the detailed distribution information between data objects. To solve these problems, we propose a novel index called Categorical data cluster Utility Based On Silhouette (CUBOS). Specifically, we first make clear the superiority of the paradigm of Silhouette index in exploring the details of clustering results. Then, we raise the Improved Distance metric for Categorical data (IDC) inspired by Category Distance to measure distance between categorical data exactly. Finally, the paradigm of Silhouette index and IDC are combined to construct the CUBOS, which can overcome the aforementioned shortcomings and produce more accurate evaluation results than other baselines, as shown by the experimental results on several UCI datasets
Unsupervised learning methods for identifying and evaluating disease clusters in electronic health records
Introduction
Clustering algorithms are a class of algorithms that can discover groups of observations in
complex data and are often used to identify subtypes of heterogeneous diseases in electronic
health records (EHR). Evaluating clustering experiments for biological and clinical significance is
a vital but challenging task due to the lack of consensus on best practices. As a result, the
translation of findings from clustering experiments to clinical practice is limited.
Aim
The aim of this thesis was to investigate and evaluate approaches that enable the evaluation of
clustering experiments using EHR.
Methods
We conducted a scoping review of clustering studies in EHR to identify common evaluation
approaches. We systematically investigated the performance of the identified approaches using
a cohort of Alzheimer's Disease (AD) patients as an exemplar comparing four different
clustering methods (K-means, Kernel K-means, Affinity Propagation and Latent Class
Analysis.). Using the same population, we developed and evaluated a method (MCHAMMER)
that tested whether clusterable structures exist in EHR. To develop this method we tested
several cluster validation indexes and methods of generating null data to see which are the best
at discovering clusters. In order to enable the robust benchmarking of evaluation approaches,
we created a tool that generated synthetic EHR data that contain known cluster labels across a
range of clustering scenarios.
Results
Across 67 EHR clustering studies, the most popular internal evaluation metric was comparing
cluster results across multiple algorithms (30% of studies). We examined this approach
conducting a clustering experiment on AD patients using a population of 10,065 AD patients and
21 demographic, symptom and comorbidity features. K-means found 5 clusters, Kernel K means found 2 clusters, Affinity propagation found 5 and latent class analysis found 6. K-means
4
was found to have the best clustering solution with the highest silhouette score (0.19) and was
more predictive of outcomes. The five clusters found were: typical AD (n=2026), non-typical AD
(n=1640), cardiovascular disease cluster (n=686), a cancer cluster (n=1710) and a cluster of
mental health issues, smoking and early disease onset (n=1528), which has been found in
previous research as well as in the results of other clustering methods. We created a synthetic
data generation tool which allows for the generation of realistic EHR clusters that can vary in
separation and number of noise variables to alter the difficulty of the clustering problem. We
found that decreasing cluster separation did increase cluster difficulty significantly whereas
noise variables increased cluster difficulty but not significantly. To develop the tool to assess
clusters existence we tested different methods of null dataset generation and cluster validation
indices, the best performing null dataset method was the min max method and the best
performing indices we Calinksi Harabasz index which had an accuracy of 94%, Davies Bouldin
index (97%) silhouette score ( 93%) and BWC index (90%). We further found that when clusters
were identified using the Calinski Harabasz index they were more likely to have significantly
different outcomes between clusters. Lastly we repeated the initial clustering experiment,
comparing 10 different pre-processing methods. The three best performing methods were RBF
kernel (2 clusters), MCA (4 clusters) and MCA and PCA (6 clusters). The MCA approach gave
the best results highest silhouette score (0.23) and meaningful clusters, producing 4 clusters;
heart and circulatory( n=1379), early onset mental health (n=1761), male cluster with memory
loss (n = 1823), female with more problem (n=2244).
Conclusion
We have developed and tested a series of methods and tools to enable the evaluation of EHR
clustering experiments. We developed and proposed a novel cluster evaluation metric and
provided a tool for benchmarking evaluation approaches in synthetic but realistic EHR