2,342 research outputs found
Do optimization methods in deep learning applications matter?
With advances in deep learning, exponential data growth and increasing model
complexity, developing efficient optimization methods are attracting much
research attention. Several implementations favor the use of Conjugate Gradient
(CG) and Stochastic Gradient Descent (SGD) as being practical and elegant
solutions to achieve quick convergence, however, these optimization processes
also present many limitations in learning across deep learning applications.
Recent research is exploring higher-order optimization functions as better
approaches, but these present very complex computational challenges for
practical use. Comparing first and higher-order optimization functions, in this
paper, our experiments reveal that Levemberg-Marquardt (LM) significantly
supersedes optimal convergence but suffers from very large processing time
increasing the training complexity of both, classification and reinforcement
learning problems. Our experiments compare off-the-shelf optimization
functions(CG, SGD, LM and L-BFGS) in standard CIFAR, MNIST, CartPole and
FlappyBird experiments.The paper presents arguments on which optimization
functions to use and further, which functions would benefit from
parallelization efforts to improve pretraining time and learning rate
convergence
Parallel MATALAB Techniques
In this chapter, we show why parallel MATLAB is useful, provide a comparison
of the different parallel MATLAB choices, and describe a number of applications
in Signal and Image Processing: Audio Signal Processing, Synthetic Aperture
Radar (SAR) Processing and Superconducting Quantum Interference Filters
(SQIFs). Each of these applications have been parallelized using different
methods (Task parallel and Data parallel techniques). The applications
presented may be considered representative of type of problems faced by signal
and image processing researchers. This chapter will also strive to serve as a
guide to new signal and image processing parallel programmers, by suggesting a
parallelization strategy that can be employed when developing a general
parallel algorithm. The objective of this chapter is to help signal and image
processing algorithm developers understand the advantages of using parallel
MATLAB to tackle larger problems while staying within the powerful environment
of MATLAB
A system for routing arbitrary directed graphs on SIMD architectures
There are many problems which can be described in terms of directed graphs that contain a large number of vertices where simple computations occur using data from connecting vertices. A method is given for parallelizing such problems on an SIMD machine model that is bit-serial and uses only nearest neighbor connections for communication. Each vertex of the graph will be assigned to a processor in the machine. Algorithms are given that will be used to implement movement of data along the arcs of the graph. This architecture and algorithms define a system that is relatively simple to build and can do graph processing. All arcs can be transversed in parallel in time O(T), where T is empirically proportional to the diameter of the interconnection network times the average degree of the graph. Modifying or adding a new arc takes the same time as parallel traversal
- …