69,340 research outputs found
Recommended from our members
Final Report PetaScale Application Development Analysis Grant Number DE-FG02-04ER25629
The results obtained from this project will fundamentally change the way we look at computer performance analysis. These results are made possible by the precise definition of a consistent system of measurement with a set of primary units designed specifically for computer performance analysis. This system of units, along with their associated dimensions, allows us to apply the methods of dimensional analysis, based on the Pi Theorem, to define scaling and self-similarity relationships. These relationships reveal new insights into experimental results that otherwise seems only vaguely correlated. Applying the method to cache-miss data revealed scaling relationships that were not seen by those who originally collected the data. Applying dimensional analysis to the performance of parallel numerical algorithms revealed that computational force is a unifying concept for understanding the interaction between hardware and software. The efficiency of these algorithms depends, in a very intimate way, on the balance between hardware forces and software forces. Analysis of five different algorithms showed that performance analysis can be reduced to a study of the differential geometry of the efficiency surface. Each algorithm defines a set of curvilinear coordinates, specific to that algorithm, and different machines follow different paths along the surface depending on the difference in balance between hardware forces and software forces. Two machines with the same balance in forces follow the same path and are self-similar. The most important result from the project is the statement of the Principle of Computational Least Action. This principle follows from the identification of a dynamical system underlying computer performance analysis. Instructions in a computer are modeled as a classical system under the influence of computational forces. Each instruction generates kinetic energy during execution, and the sum of the kinetic energy for all instructions produces a kinetic energy spectrum as a function of time. These spectra look very much like the spectra used by chemists to analyze properties of molecules. Large spikes in the spectra reveal events during execution, like cache misses, that limit performance. The area under the kinetic energy spectrum is the computational action generated by the program. This computational action defines a normed metric space that measures the size of a program in terms of its action norm and the distance between programs in terms of the norm of the difference of their action. This same idea can be applied to a set of programmers writing code and leads to a computational action metric that measures programmer productivity. In both cases, experimental evidence suggests that highly efficient programs and highly productive programmers generate the least computational action
Do ideas have shape? Plato's theory of forms as the continuous limit of artificial neural networks
We show that ResNets converge, in the infinite depth limit, to a
generalization of image registration algorithms. In this generalization, images
are replaced by abstractions (ideas) living in high dimensional RKHS spaces,
and material points are replaced by data points. Whereas computational anatomy
aligns images via deformations of the material space, this generalization
aligns ideas by via transformations of their RKHS. This identification of
ResNets as idea registration algorithms has several remarkable consequences.
The search for good architectures can be reduced to that of good kernels, and
we show that the composition of idea registration blocks with reduced
equivariant multi-channel kernels (introduced here) recovers and generalizes
CNNs to arbitrary spaces and groups of transformations. Minimizers of
regularized ResNets satisfy a discrete least action principle implying the near
preservation of the norm of weights and biases across layers. The parameters of
trained ResNets can be identified as solutions of an autonomous Hamiltonian
system defined by the activation function and the architecture of the ANN.
Momenta variables provide a sparse representation of the parameters of a
ResNet. The registration regularization strategy provides a provably robust
alternative to Dropout for ANNs. Pointwise RKHS error estimates lead to
deterministic error estimates for ANNs.Comment: 56 page
Do ideas have shape? Plato's theory of forms as the continuous limit of artificial neural networks
We show that ResNets converge, in the infinite depth limit, to a generalization of image registration algorithms. In this generalization, images are replaced by abstractions (ideas) living in high dimensional RKHS spaces, and material points are replaced by data points. Whereas computational anatomy aligns images via deformations of the material space, this generalization aligns ideas by via transformations of their RKHS. This identification of ResNets as idea registration algorithms has several remarkable consequences. The search for good architectures can be reduced to that of good kernels, and we show that the composition of idea registration blocks with reduced equivariant multi-channel kernels (introduced here) recovers and generalizes CNNs to arbitrary spaces and groups of transformations. Minimizers of L2 regularized ResNets satisfy a discrete least action principle implying the near preservation of the norm of weights and biases across layers. The parameters of trained ResNets can be identified as solutions of an autonomous Hamiltonian system defined by the activation function and the architecture of the ANN. Momenta variables provide a sparse representation of the parameters of a ResNet. The registration regularization strategy provides a provably robust alternative to Dropout for ANNs. Pointwise RKHS error estimates lead to deterministic error estimates for ANNs
Singular solutions, momentum maps and computational anatomy
This paper describes the variational formulation of template matching
problems of computational anatomy (CA); introduces the EPDiff evolution
equation in the context of an analogy between CA and fluid dynamics; discusses
the singular solutions for the EPDiff equation and explains why these singular
solutions exist (singular momentum map). Then it draws the consequences of
EPDiff for outline matching problem in CA and gives numerical examples
Deriving Grover's lower bound from simple physical principles
Grover's algorithm constitutes the optimal quantum solution to the search
problem and provides a quadratic speed-up over all possible classical search
algorithms. Quantum interference between computational paths has been posited
as a key resource behind this computational speed-up. However there is a limit
to this interference, at most pairs of paths can ever interact in a fundamental
way. Could more interference imply more computational power? Sorkin has defined
a hierarchy of possible interference behaviours---currently under experimental
investigation---where classical theory is at the first level of the hierarchy
and quantum theory belongs to the second. Informally, the order in the
hierarchy corresponds to the number of paths that have an irreducible
interaction in a multi-slit experiment. In this work, we consider how Grover's
speed-up depends on the order of interference in a theory. Surprisingly, we
show that the quadratic lower bound holds regardless of the order of
interference. Thus, at least from the point of view of the search problem,
post-quantum interference does not imply a computational speed-up over quantum
theory.Comment: Updated title and exposition in response to referee comments. 6+2
pages, 5 figure
An Efficient Dual Approach to Distance Metric Learning
Distance metric learning is of fundamental interest in machine learning
because the distance metric employed can significantly affect the performance
of many learning methods. Quadratic Mahalanobis metric learning is a popular
approach to the problem, but typically requires solving a semidefinite
programming (SDP) problem, which is computationally expensive. Standard
interior-point SDP solvers typically have a complexity of (with
the dimension of input data), and can thus only practically solve problems
exhibiting less than a few thousand variables. Since the number of variables is
, this implies a limit upon the size of problem that can
practically be solved of around a few hundred dimensions. The complexity of the
popular quadratic Mahalanobis metric learning approach thus limits the size of
problem to which metric learning can be applied. Here we propose a
significantly more efficient approach to the metric learning problem based on
the Lagrange dual formulation of the problem. The proposed formulation is much
simpler to implement, and therefore allows much larger Mahalanobis metric
learning problems to be solved. The time complexity of the proposed method is
, which is significantly lower than that of the SDP approach.
Experiments on a variety of datasets demonstrate that the proposed method
achieves an accuracy comparable to the state-of-the-art, but is applicable to
significantly larger problems. We also show that the proposed method can be
applied to solve more general Frobenius-norm regularized SDP problems
approximately
- …