Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show $\sqrt{T}$-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.Comment: ICML 2017, 16 pages, 23 figure

Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show $\sqrt{T}$-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.Comment: ICML 2017, 16 pages, 23 figure

Bioengineered Scaffolds for Peripheral Nerve Regeneration

Nerve autografts are widely used clinically to repair nerve grafts. However, nerve grafts have many limitations, such as, availability of donor nerve grafts, and loss of function at donor site. To overcome these problems, we have used a tissue engineering approach to design three-dimensional (3D) agarose scaffolds containing gradients of laminin-1 (LN-1) and nerve growth factor (NGF) to mimic in vivo conditions to promote nerve regeneration in rats. To determine the effect of LN-1 gradients on neurite extension in vitro, dorsal root ganglia (DRG) from chick embryos were cultured in 3D hydrogels. A gradient of LN-1 molecules in agarose gels was made by diffusion technique. LN-1 was then immobilized to the agarose hydrogels using a photo-crosslinker, Sulfo-SANPAH (Sulfosuccinimidyl-6-[4-azido-2-nitrophenylamino] hexanoate). Anisotropic scaffolds with three different slopes of LN-1 gradients were used. Isotropic scaffolds with uniform concentrations of LN-1, at various levels, were used as a positive control. DRG cultured in anisotropic scaffolds with optimal slope of LN-1 gradient extended neurites twice as fast as DRG in optimal concentration in isotropic scaffolds. Also, in the anisotropic scaffolds the faster growing neurites were aligned along the direction of LN-1 gradient. To promote nerve regeneration in vivo, tubular polysulfone guidance channels containing agarose hydrogels with gradients of LN-1 and NGF (anisotropic scaffolds) were used to bridge 20-mm nerve gaps in rats. Nerve autografts were used as positive controls and isotropic scaffolds, with uniform concentration of LN-1 and NGF, were used as negative controls. After 4-months, the rats were sacrificed and nerve histology was done to test for nerve regeneration. Only anisotropic scaffolds and nerve autografts contained evidence of axonal regeneration. Both groups had similar numbers of myelinated axons and similar axonal-diameter distribution. However, nerve graft group performed better in functional outcome as measured by relative gastrocnemius muscle weight (RGMW) and electrophysiology. Optimization of performance of anisotropic scaffolds by varying the LN-1 and NGF concentration gradients might lead to development of scaffolds that can perform as well as nerve auotgrafts for nerve regeneration over long nerve gaps.Ph.D.Committee Chair: Bellamkonda, Ravi; Committee Member: English, Arthur; Committee Member: Garcia, Andres; Committee Member: LaPlaca, Michelle; Committee Member: McDevitt, Tod