Adaptive Momentum for Neural Network Optimization

In this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore

Dynamics of a split torque helicopter transmission

A high reduction ratio split torque gear train has been proposed as an alternative to a planetary configuration for the final stage of a helicopter transmission. A split torque design allows a high ratio of power-to-weight for the transmission. The design studied in this work includes a pivoting beam that acts to balance thrust loads produced by the helical gear meshes in each of two parallel power paths. When the thrust loads are balanced, the torque is split evenly. A mathematical model was developed to study the dynamics of the system. The effects of time varying gear mesh stiffness, static transmission errors, and flexible bearing supports are included in the model. The model was demonstrated with a test case. Results show that although the gearbox has a symmetric configuration, the simulated dynamic behavior of the first and second compound gears are not the same. Also, results show that shaft location and mesh stiffness tuning are significant design parameters that influence the motions of the system