Skip to main content
Article thumbnail
Location of Repository

Dynamic load balancing in parallel KD-tree k-means

By Giuseppe Di Fatta and David Pettinger

Abstract

One among the most influential and popular data mining methods is the k-Means algorithm for cluster analysis.\ud Techniques for improving the efficiency of k-Means have been\ud largely explored in two main directions. The amount of computation can be significantly reduced by adopting geometrical constraints and an efficient data structure, notably a multidimensional binary search tree (KD-Tree). These techniques allow to reduce the number of distance computations the algorithm performs at each iteration. A second direction is parallel processing, where data and computation loads are distributed over many processing nodes. However, little work has been done to provide a parallel formulation of the efficient sequential techniques based on KD-Trees. Such approaches are expected to have an irregular distribution of computation load and can suffer from load imbalance. This issue has so far limited the adoption of these efficient k-Means variants in parallel computing environments. In this work, we provide a parallel formulation of the KD-Tree based k-Means algorithm for distributed memory systems and address its load balancing\ud issue. Three solutions have been developed and tested. Two\ud approaches are based on a static partitioning of the data set and a third solution incorporates a dynamic load balancing policy

Publisher: IEEE
Year: 2010
OAI identifier: oai:centaur.reading.ac.uk:6127

Suggested articles


To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.