44,461 research outputs found
Modeling Scalability of Distributed Machine Learning
Present day machine learning is computationally intensive and processes large
amounts of data. It is implemented in a distributed fashion in order to address
these scalability issues. The work is parallelized across a number of computing
nodes. It is usually hard to estimate in advance how many nodes to use for a
particular workload. We propose a simple framework for estimating the
scalability of distributed machine learning algorithms. We measure the
scalability by means of the speedup an algorithm achieves with more nodes. We
propose time complexity models for gradient descent and graphical model
inference. We validate our models with experiments on deep learning training
and belief propagation. This framework was used to study the scalability of
machine learning algorithms in Apache Spark.Comment: 6 pages, 4 figures, appears at ICDE 201
Understanding and Comparing Scalable Gaussian Process Regression for Big Data
As a non-parametric Bayesian model which produces informative predictive
distribution, Gaussian process (GP) has been widely used in various fields,
like regression, classification and optimization. The cubic complexity of
standard GP however leads to poor scalability, which poses challenges in the
era of big data. Hence, various scalable GPs have been developed in the
literature in order to improve the scalability while retaining desirable
prediction accuracy. This paper devotes to investigating the methodological
characteristics and performance of representative global and local scalable GPs
including sparse approximations and local aggregations from four main
perspectives: scalability, capability, controllability and robustness. The
numerical experiments on two toy examples and five real-world datasets with up
to 250K points offer the following findings. In terms of scalability, most of
the scalable GPs own a time complexity that is linear to the training size. In
terms of capability, the sparse approximations capture the long-term spatial
correlations, the local aggregations capture the local patterns but suffer from
over-fitting in some scenarios. In terms of controllability, we could improve
the performance of sparse approximations by simply increasing the inducing
size. But this is not the case for local aggregations. In terms of robustness,
local aggregations are robust to various initializations of hyperparameters due
to the local attention mechanism. Finally, we highlight that the proper hybrid
of global and local scalable GPs may be a promising way to improve both the
model capability and scalability for big data.Comment: 25 pages, 15 figures, preprint submitted to KB
Optimistic Concurrency Control for Distributed Unsupervised Learning
Research on distributed machine learning algorithms has focused primarily on
one of two extremes - algorithms that obey strict concurrency constraints or
algorithms that obey few or no such constraints. We consider an intermediate
alternative in which algorithms optimistically assume that conflicts are
unlikely and if conflicts do arise a conflict-resolution protocol is invoked.
We view this "optimistic concurrency control" paradigm as particularly
appropriate for large-scale machine learning algorithms, particularly in the
unsupervised setting. We demonstrate our approach in three problem areas:
clustering, feature learning and online facility location. We evaluate our
methods via large-scale experiments in a cluster computing environment.Comment: 25 pages, 5 figure
Blockchain-enabled Clustered and Scalable Federated Learning (BCS-FL) Framework in UAV Networks
Privacy, scalability, and reliability are significant challenges in unmanned
aerial vehicle (UAV) networks as distributed systems, especially when employing
machine learning (ML) technologies with substantial data exchange. Recently,
the application of federated learning (FL) to UAV networks has improved
collaboration, privacy, resilience, and adaptability, making it a promising
framework for UAV applications. However, implementing FL for UAV networks
introduces drawbacks such as communication overhead, synchronization issues,
scalability limitations, and resource constraints. To address these challenges,
this paper presents the Blockchain-enabled Clustered and Scalable Federated
Learning (BCS-FL) framework for UAV networks. This improves the
decentralization, coordination, scalability, and efficiency of FL in
large-scale UAV networks. The framework partitions UAV networks into separate
clusters, coordinated by cluster head UAVs (CHs), to establish a connected
graph. Clustering enables efficient coordination of updates to the ML model.
Additionally, hybrid inter-cluster and intra-cluster model aggregation schemes
generate the global model after each training round, improving collaboration
and knowledge sharing among clusters. The numerical findings illustrate the
achievement of convergence while also emphasizing the trade-offs between the
effectiveness of training and communication efficiency.Comment: 6 pages, 7 figures, 2023 IEEE International Workshop on Computer
Aided Modeling and Design of Communication Links and Networks (IEEE CAMAD),
Edinburgh U
- …