2,740 research outputs found
Medoid Silhouette clustering with automatic cluster number selection
The evaluation of clustering results is difficult, highly dependent on the
evaluated data set and the perspective of the beholder. There are many
different clustering quality measures, which try to provide a general measure
to validate clustering results. A very popular measure is the Silhouette. We
discuss the efficient medoid-based variant of the Silhouette, perform a
theoretical analysis of its properties, provide two fast versions for the
direct optimization, and discuss the use to choose the optimal number of
clusters. We combine ideas from the original Silhouette with the well-known PAM
algorithm and its latest improvements FasterPAM. One of the versions guarantees
equal results to the original variant and provides a run speedup of .
In experiments on real data with 30000 samples and =100, we observed a
10464 speedup compared to the original PAMMEDSIL algorithm.
Additionally, we provide a variant to choose the optimal number of clusters
directly.Comment: arXiv admin note: substantial text overlap with arXiv:2209.1255
Graph-Based Automatic Feature Selection for Multi-Class Classification via Mean Simplified Silhouette
This paper introduces a novel graph-based filter method for automatic feature
selection (abbreviated as GB-AFS) for multi-class classification tasks. The
method determines the minimum combination of features required to sustain
prediction performance while maintaining complementary discriminating abilities
between different classes. It does not require any user-defined parameters such
as the number of features to select. The methodology employs the
Jeffries-Matusita (JM) distance in conjunction with t-distributed Stochastic
Neighbor Embedding (t-SNE) to generate a low-dimensional space reflecting how
effectively each feature can differentiate between each pair of classes. The
minimum number of features is selected using our newly developed Mean
Simplified Silhouette (abbreviated as MSS) index, designed to evaluate the
clustering results for the feature selection task. Experimental results on
public data sets demonstrate the superior performance of the proposed GB-AFS
over other filter-based techniques and automatic feature selection approaches.
Moreover, the proposed algorithm maintained the accuracy achieved when
utilizing all features, while using only to of the features.
Consequently, this resulted in a reduction of the time needed for
classifications, from to .Comment: 8 pages, 4 figure
On the use of Silhouette for cost based clustering
Clustering plays a fundamental role in Machine Learning. With clustering we refer to the problem of finding coherent groups in a dataset of elements. There are several algorithms to perform clustering that have been proposed in the literature, considering different costs for the optimization problems they consider. In this thesis we study the problem of clustering when the cost function is the silhouette coefficient, an index traditionally used for the internal validation of the results
Meta-optimizations for Cluster Analysis
This dissertation thesis deals with advances in the automation of cluster analysis.This dissertation thesis deals with advances in the automation of cluster analysis
Deep generative modeling for single-cell transcriptomics.
Single-cell transcriptome measurements can reveal unexplored biological diversity, but they suffer from technical noise and bias that must be modeled to account for the resulting uncertainty in downstream analyses. Here we introduce single-cell variational inference (scVI), a ready-to-use scalable framework for the probabilistic representation and analysis of gene expression in single cells ( https://github.com/YosefLab/scVI ). scVI uses stochastic optimization and deep neural networks to aggregate information across similar cells and genes and to approximate the distributions that underlie observed expression values, while accounting for batch effects and limited sensitivity. We used scVI for a range of fundamental analysis tasks including batch correction, visualization, clustering, and differential expression, and achieved high accuracy for each task
Incremental Cluster Validity Indices for Online Learning of Hard Partitions: Extensions and Comparative Study
Validation is one of the most important aspects of clustering, particularly when the user is designing a trustworthy or explainable system. However, most clustering validation approaches require batch calculation. This is an important gap because of the value of clustering in real-time data streaming and other online learning applications. Therefore, interest has grown in providing online alternatives for validation. This paper extends the incremental cluster validity index (iCVI) family by presenting incremental versions of Calinski-Harabasz (iCH), Pakhira-Bandyopadhyay-Maulik (iPBM), WB index (iWB), Silhouette (iSIL), Negentropy Increment (iNI), Representative Cross Information Potential (irCIP), Representative Cross Entropy (irH), and Conn_Index (iConn_Index). This paper also provides a thorough comparative study of correct, under- and over-partitioning on the behavior of these iCVIs, the Partition Separation (PS) index as well as four recently introduced iCVIs: incremental Xie-Beni (iXB), incremental Davies-Bouldin (iDB), and incremental generalized Dunn\u27s indices 43 and 53 (iGD43 and iGD53). Experiments were carried out using a framework that was designed to be as agnostic as possible to the clustering algorithms. The results on synthetic benchmark data sets showed that while evidence of most under-partitioning cases could be inferred from the behaviors of the majority of these iCVIs, over-partitioning was found to be a more challenging problem, detected by fewer of them. Interestingly, over-partitioning, rather then under-partitioning, was more prominently detected on the real-world data experiments within this study. The expansion of iCVIs provides significant novel opportunities for assessing and interpreting the results of unsupervised lifelong learning in real-time, wherein samples cannot be reprocessed due to memory and/or application constraints
EVALUATION OF THE CLUSTERING PERFORMANCE OF AFFINITY PROPAGATION ALGORITHM CONSIDERING THE INFLUENCE OF PREFERENCE PARAMETER AND DAMPING FACTOR
The identification of significant underlying data patterns such as image composition and spatial arrangements is fundamental in remote sensing tasks. Therefore, the development of an effective approach for information extraction is crucial to achieve this goal. Affinity propagation (AP) algorithm is a novel powerful technique with the ability of handling with unusual data, containing both categorical and numerical attributes. However, AP has some limitations related to the choice of initial preference parameter, occurrence of oscillations and processing of large data sets. This paper evaluates the clustering performance of AP algorithm taking into account the influence of preference parameter and damping factor. The study was conducted considering the AP algorithm, the adaptive AP and partition AP. According to the experiments, the choice of preference and damping greatly influences on the quality and the final number of clusters
An Improved Differential Evolution Algorithm for Data Stream Clustering
A Few algorithms were actualized by the analysts for performing clustering of data streams. Most of these algorithms require that the number of clusters (K) has to be fixed by the customer based on input data and it can be kept settled all through the clustering process. Stream clustering has faced few difficulties in picking up K. In this paper, we propose an efficient approach for data stream clustering by embracing an Improved Differential Evolution (IDE) algorithm. The IDE algorithm is one of the quick, powerful and productive global optimization approach for programmed clustering. In our proposed approach, we additionally apply an entropy based method for distinguishing the concept drift in the data stream and in this way updating the clustering procedure online. We demonstrated that our proposed method is contrasted with Genetic Algorithm and identified as proficient optimization algorithm. The performance of our proposed technique is assessed and cr eates the accuracy of 92.29%, the precision is 86.96%, recall is 90.30% and F-measure estimate is 88.60%
Comparison of Clustering Algorithms for Learning Analytics with Educational Datasets
Learning Analytics is becoming a key tool for the analysis and improvement of digital education processes, and its potential benefit grows with the size of the student cohorts generating data. In the context of Open Education, the potentially massive student cohorts and the global audience represent a great opportunity for significant analyses and breakthroughs in the field of learning analytics. However, these potentially huge datasets require proper analysis techniques, and different algorithms, tools and approaches may perform better in this specific context. In this work, we compare different clustering algorithms using an educational dataset. We start by identifying the most relevant algorithms in Learning Analytics and benchmark them to determine, according to internal validation and stability measurements, which algorithms perform better. We analyzed seven algorithms, and determined that K-means and PAM were the best performers among partition algorithms, and DIANA was the best performer among hierarchical algorithms
New internal and external validation indices for clustering in Big Data
Esta tesis, presentada como un compendio de artículos de investigación,
analiza el concepto de índices de validación de clustering y aporta nuevas
medidas de bondad para conjuntos de datos que podrían considerarse Big
Data debido a su volumen. Además, estas medidas han sido aplicadas en
proyectos reales y se propone su aplicación futura para mejorar algoritmos
de clustering.
El clustering es una de las técnicas de aprendizaje automático no supervisado
más usada. Esta técnica nos permite agrupar datos en clusters de
manera que, aquellos datos que pertenezcan al mismo cluster tienen características
o atributos con valores similares, y a su vez esos datos son disimilares
respecto a aquellos que pertenecen a los otros clusters. La similitud de los
datos viene dada normalmente por la cercanía en el espacio, teniendo en
cuenta una función de distancia. En la literatura existen los llamados índices
de validación de clustering, los cuales podríamos definir como medidas para
cuantificar la calidad de un resultado de clustering. Estos índices se dividen
en dos tipos: índices de validación internos, que miden la calidad del clustering
en base a los atributos con los que se han construido los clusters; e
índices de validación externos, que son aquellos que cuantifican la calidad del
clustering a partir de atributos que no han intervenido en la construcción de
los clusters, y que normalmente son de tipo nominal o etiquetas.
En esta memoria se proponen dos índices de validación internos para clustering
basados en otros índices existentes en la literatura, que nos permiten
trabajar con grandes cantidades de datos, ofreciéndonos los resultados en un
tiempo razonable. Los índices propuestos han sido testeados en datasets sintéticos
y comparados con otros índices de la literatura. Las conclusiones de
este trabajo indican que estos índices ofrecen resultados muy prometedores
frente a sus competidores.
Por otro lado, se ha diseñado un nuevo índice de validación externo de
clustering basado en el test estadístico chi cuadrado. Este índice permite
medir la calidad del clustering basando el resultado en cómo han quedado
distribuidos los clusters respecto a una etiqueta dada en la distribución. Los
resultados de este índice muestran una mejora significativa frente a otros
índices externos de la literatura y en datasets de diferentes dimensiones y características.
Además, estos índices propuestos han sido aplicados en tres proyectos
con datos reales cuyas publicaciones están incluidas en esta tesis doctoral.
Para el primer proyecto se ha desarrollado una metodología para analizar el
consumo eléctrico de los edificios de una smart city. Para ello, se ha realizado
un análisis de clustering óptimo aplicando los índices internos mencionados
anteriormente. En el segundo proyecto se ha trabajado tanto los índices internos
como con los externos para realizar un análisis comparativo del mercado
laboral español en dos periodos económicos distintos. Este análisis se realizó
usando datos del Ministerio de Trabajo, Migraciones y Seguridad Social, y
los resultados podrían tenerse en cuenta para ayudar a la toma de decisión
en mejoras de políticas de empleo. En el tercer proyecto se ha trabajado con
datos de los clientes de una compañía eléctrica para caracterizar los tipos
de consumidores que existen. En este estudio se han analizado los patrones
de consumo para que las compañías eléctricas puedan ofertar nuevas tarifas
a los consumidores, y éstos puedan adaptarse a estas tarifas con el objetivo
de optimizar la generación de energía eliminando los picos de consumo que
existen la actualidad.This thesis, presented as a compendium of research articles, analyses
the concept of clustering validation indices and provides new measures of
goodness for datasets that could be considered Big Data. In addition, these
measures have been applied in real projects and their future application is
proposed for the improvement of clustering algorithms.
Clustering is one of the most popular unsupervised machine learning
techniques. This technique allows us to group data into clusters so that the
instances that belong to the same cluster have characteristics or attributes
with similar values, and are dissimilar to those that belong to the other
clusters. The similarity of the data is normally given by the proximity in
space, which is measured using a distance function. In the literature, there
are so-called clustering validation indices, which can be defined as measures
for the quantification of the quality of a clustering result. These indices are
divided into two types: internal validation indices, which measure the quality
of clustering based on the attributes with which the clusters have been built;
and external validation indices, which are those that quantify the quality of
clustering from attributes that have not intervened in the construction of
the clusters, and that are normally of nominal type or labels.
In this doctoral thesis, two internal validation indices are proposed for
clustering based on other indices existing in the literature, which enable
large amounts of data to be handled, and provide the results in a reasonable
time. The proposed indices have been tested with synthetic datasets and
compared with other indices in the literature. The conclusions of this work
indicate that these indices offer very promising results in comparison with
their competitors.
On the other hand, a new external clustering validation index based on
the chi-squared statistical test has been designed. This index enables the
quality of the clustering to be measured by basing the result on how the
clusters have been distributed with respect to a given label in the distribution.
The results of this index show a significant improvement compared to
other external indices in the literature when used with datasets of different
dimensions and characteristics.
In addition, these proposed indices have been applied in three projects with real data whose corresponding publications are included in this doctoral
thesis. For the first project, a methodology has been developed to analyse
the electrical consumption of buildings in a smart city. For this study, an
optimal clustering analysis has been carried out by applying the aforementioned
internal indices. In the second project, both internal and external
indices have been applied in order to perform a comparative analysis of the
Spanish labour market in two different economic periods. This analysis was
carried out using data from the Ministry of Labour, Migration, and Social
Security, and the results could be taken into account to help decision-making
for the improvement of employment policies. In the third project, data from
the customers of an electric company has been employed to characterise the
different types of existing consumers. In this study, consumption patterns
have been analysed so that electricity companies can offer new rates to consumers.
Conclusions show that consumers could adapt their usage to these
rates and hence the generation of energy could be optimised by eliminating
the consumption peaks that currently exist
- …