Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with Applications to Neural Network Training

The Frobenius norm is a frequent choice of norm for matrices. In particular, the underlying Frobenius inner product is typically used to evaluate the gradient of an objective with respect to matrix variable, such as those occuring in the training of neural networks. We provide a broader view on the Frobenius norm and inner product for linear maps or matrices, and establish their dependence on inner products in the domain and co-domain spaces. This shows that the classical Frobenius norm is merely one special element of a family of more general Frobenius-type norms. The significant extra freedom furnished by this realization can be used, among other things, to precondition neural network training

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties when applied to quadratic problems and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework enables the potential inclusion of a preconditioner, thereby enabling the implementation of adaptive step sizes for stochastic second-order optimization methods

Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks

The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respect to the virtual parameters that additional layers, if inserted, would offer. We consider fully connected feedforward networks with selected activation functions as well as residual neural networks. In numerical experiments, the proposed sensitivity-based layer insertion technique exhibits improved training decay, compared to not inserting the layer. Furthermore, the computational effort is reduced in comparison to inserting the layer from the beginning. The code is available at \url{https://github.com/LeonieKreis/layer_insertion_sensitivity_based}

Cerebral blood flow and white matter alterations in adults with phenylketonuria

Background: Phenylketonuria (PKU) represents a congenital metabolic defect that disrupts the process of converting phenylalanine (Phe) into tyrosine. Earlier investigations have revealed diminished cognitive performance and changes in brain structure and function (including the presence of white matter lesions) among individuals affected by PKU. However, there exists limited understanding regarding cerebral blood flow (CBF) and its potential associations with cognition, white matter lesions, and metabolic parameters in patients with PKU, which we therefore aimed to investigate in this study. Method: Arterial spin labeling perfusion MRI was performed to measure CBF in 30 adults with early-treated classical PKU (median age 35.5 years) and 59 healthy controls (median age 30.0 years). For all participants, brain Phe levels were measured with 1H spectroscopy, and white matter lesions were rated by two neuroradiologists on T2 weighted images. White matter integrity was examined with diffusion tensor imaging (DTI). For patients only, concurrent plasma Phe levels were assessed after an overnight fasting period. Furthermore, past Phe levels were collected to estimate historical metabolic control. On the day of the MRI, each participant underwent a cognitive assessment measuring IQ and performance in executive functions, attention, and processing speed. Results: No significant group difference was observed in global CBF between patients and controls (F (1, 87) = 3.81, p = 0.054). Investigating CBF on the level of cerebral arterial territories, reduced CBF was observed in the left middle and posterior cerebral artery (MCA and PCA), with the most prominent reduction of CBF in the anterior subdivision of the MCA (F (1, 87) = 6.15, p = 0.015, surviving FDR correction). White matter lesions in patients were associated with cerebral blood flow reduction in the affected structure. Particularly, patients with lesions in the occipital lobe showed significant CBF reductions in the left PCA (U = 352, p = 0.013, surviving FDR correction). Additionally, axial diffusivity measured with DTI was positively associated with CBF in the ACA and PCA (surviving FDR correction). Cerebral blood flow did not correlate with cognitive performance or metabolic parameters. Conclusion: The relationship between cerebral blood flow and white matter indicates a complex interplay between vascular health and white matter alterations in patients with PKU. It highlights the importance of considering a multifactorial model when investigating the impact of PKU on the brain

The Witan, 1975-1976 Academic Year V. 3 No. 6, February 1976

Lawyer Advertising Controverted, Election Law Upheld, Why Pay Less?, Light News, Law Libraries Compared, Women's Work Working in America, Limited Legal Advertising, Court Appointments, Pep Talk, Rape and the Penal Code, Nicolini to Jr. State Bar, Hearsa

The Witan, 1975-1976 Academic Year V. 3 No. 6, February 1976

Lawyer Advertising Controverted, Election Law Upheld, Why Pay Less?, Light News, Law Libraries Compared, Women's Work Working in America, Limited Legal Advertising, Court Appointments, Pep Talk, Rape and the Penal Code, Nicolini to Jr. State Bar, Hearsa