2 research outputs found
Numerics of High Performance Computers and Benchmark Evaluation of Distributed Memory Computers
The internal representation of numerical data, their speed of manipulation to generate the desired result through efficient utilisation of central processing unit, memory, and communication links are essential steps of all high performance scientific computations. Machine parameters, in particular, reveal accuracy and error bounds of computation, required for performance tuning of codes. This paper reports diagnosis of machine parameters, measurement of computing power of several workstations, serial and parallel computers, and a component-wise test procedure for distributed memory computers. Hierarchical memory structure is illustrated by block copying and unrolling techniques. Locality of reference for cache reuse of data is amply demonstrated by fast Fourier transform codes. Cache and register-blocking technique results in their optimum utilisation with consequent gain in throughput during vector-matrix operations. Implementation of these memory management techniques reduces cache inefficiency loss, which is known to be proportional to the number of processors. Of the two Linux clusters-ANUP16, HPC22 and HPC64, it has been found from the measurement of intrinsic parameters and from application benchmark of multi-block Euler code test run that ANUP16 is suitable for problems that exhibit fine-grained parallelism. The delivered performance of ANUP16 is of immense utility for developing high-end PC clusters like HPC64 and customised parallel computers with added advantage of speed and high degree of parallelism
Sparse-SignSGD with Majority Vote for Communication-Efficient Distributed Learning
The training efficiency of complex deep learning models can be significantly
improved through the use of distributed optimization. However, this process is
often hindered by a large amount of communication cost between workers and a
parameter server during iterations. To address this bottleneck, in this paper,
we present a new communication-efficient algorithm that offers the synergistic
benefits of both sparsification and sign quantization, called GD-MV.
The workers in GD-MV select the top- magnitude components of
their local gradient vector and only send the signs of these components to the
server. The server then aggregates the signs and returns the results via a
majority vote rule. Our analysis shows that, under certain mild conditions,
GD-MV can converge at the same rate as signSGD while significantly
reducing communication costs, if the sparsification parameter is properly
chosen based on the number of workers and the size of the deep learning model.
Experimental results using both independent and identically distributed (IID)
and non-IID datasets demonstrate that the GD-MV attains higher
accuracy than signSGD, significantly reducing communication costs. These
findings highlight the potential of GD-MV as a promising solution
for communication-efficient distributed optimization in deep learning.Comment: 13 pages, 7 figure