Inductive Bias of Gradient Descent for Exponentially Weight Normalized Smooth Homogeneous Neural Nets

We analyze the inductive bias of gradient descent for weight normalized smooth homogeneous neural nets, when trained on exponential or cross-entropy loss. Our analysis focuses on exponential weight normalization (EWN), which encourages weight updates along the radial direction. This paper shows that the gradient flow path with EWN is equivalent to gradient flow on standard networks with an adaptive learning rate, and hence causes the weights to be updated in a way that prefers asymptotic relative sparsity. These results can be extended to hold for gradient descent via an appropriate adaptive learning rate. The asymptotic convergence rate of the loss in this setting is given by $\Theta(\frac{1}{t(\log t)^2})$, and is independent of the depth of the network. We contrast these results with the inductive bias of standard weight normalization (SWN) and unnormalized architectures, and demonstrate their implications on synthetic data sets.Experimental results on simple data sets and architectures support our claim on sparse EWN solutions, even with SGD. This demonstrates its potential applications in learning prunable neural networks.Comment: We have modified proposition 3, removing the extra assumptions, resulting in a slightly less sharp instability result. We have also added a figure showing the norm of the weights for SWN, EWN and NWN for the MNIST training procedure (Appendix N, Figure 11). A few more references that use SWN have been added to page 3. We have also fixed a few typos and grammatical error

Feature emergence via margin maximization: case studies in algebraic tasks

Understanding the internal representations learned by neural networks is a cornerstone challenge in the science of machine learning. While there have been significant recent strides in some cases towards understanding how neural networks implement specific target functions, this paper explores a complementary question -- why do networks arrive at particular computational strategies? Our inquiry focuses on the algebraic learning tasks of modular addition, sparse parities, and finite group operations. Our primary theoretical findings analytically characterize the features learned by stylized neural networks for these algebraic tasks. Notably, our main technique demonstrates how the principle of margin maximization alone can be used to fully specify the features learned by the network. Specifically, we prove that the trained networks utilize Fourier features to perform modular addition and employ features corresponding to irreducible group-theoretic representations to perform compositions in general groups, aligning closely with the empirical observations of Nanda et al. and Chughtai et al. More generally, we hope our techniques can help to foster a deeper understanding of why neural networks adopt specific computational strategies

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales

We study the effect of width on the dynamics of feature-learning neural networks across a variety of architectures and datasets. Early in training, wide neural networks trained on online data have not only identical loss curves but also agree in their point-wise test predictions throughout training. For simple tasks such as CIFAR-5m this holds throughout training for networks of realistic widths. We also show that structural properties of the models, including internal representations, preactivation distributions, edge of stability phenomena, and large learning rate effects are consistent across large widths. This motivates the hypothesis that phenomena seen in realistic models can be captured by infinite-width, feature-learning limits. For harder tasks (such as ImageNet and language modeling), and later training times, finite-width deviations grow systematically. Two distinct effects cause these deviations across widths. First, the network output has initialization-dependent variance scaling inversely with width, which can be removed by ensembling networks. We observe, however, that ensembles of narrower networks perform worse than a single wide network. We call this the bias of narrower width. We conclude with a spectral perspective on the origin of this finite-width bias

Correlation between Topographic Vessel Density and Retinal Thickness Changes in Patients with Diabetic Macular Edema Treated with Anti-VEGF Therapy: Is It a Suitable OCTA Biomarker?

The objective of this study was to determine the correlation between topographic vessel density (VD) and retinal thickness (RT) reductions induced by vascular endothelial growth factor inhibitors (anti-VEGF) in patients with diabetic macular edema (DME) using optical coherence tomography angiography (OCTA). This was a prospective, interventional case series. VD and RT measurements were separately taken in four parafoveal subfields at baseline and after six months of treatment. This correlation was statistically assessed using Spearman's rho correlation coefficient after adjustment for multiple comparisons. The study included a total of 48 eyes in the final analysis. Mean VD decreased from baseline to month 6 (from 45.2 (+/- 3.5) to 44.6% (+/- 3.2) in the superficial capillary plexus and from 50 (+/- 3.3) to 49% (+/- 3.9) in the deep capillary plexus). Statistically significant reductions in RT were observed in all ETDRS sectors (p < 0.0001). No significant association was found between RT and VD, even when analyzing responders and non-responders separately. After six months of anti-VEGF treatment, no significant correlation was observed between the topographic VD and RT values. These findings suggest that reductions in VD values may not solely result from a reduction in microaneurysms, also being affected by the repositioning of displaced vessels due to edema and a reduction in their caliber. Therefore, VD changes may not be a suitable indirect OCTA biomarker of microaneurysm turnover and treatment response

Gut microbiome changes due to sleep disruption in older and younger individuals: a case for sarcopenia?

Major hallmarks of functional loss, loss of metabolic and musculoskeletal health and (multi)morbidity with ageing are associated with sleep disturbances. With poor sleep shifts in gut microbial composition commonly manifest, which could mediate the proinflammatory state between sleep disturbances and sarcopenia. This systematic review presents the recent evidence on how sleep disturbances throughout the lifespan associate with and contribute to gut microbial composition changes, proposing a mechanism to understand the aetiology of sarcopenia through sleep disturbances. The relationship between disturbed sleep and clinically relevant gut microbiota composition on health aspects of ageing is discussed. A search was performed in PubMed, Cochrane Library, Scopus, Web of Science using keywords including (microbio* OR microflora) AND (sleep OR sleep disorder). Six cross-sectional population-based studies and five experimental clinical trials investigating healthy individuals with ages ranging from 4 to 71 were included. The cross-sectional studies reported similarities in associations with sleep disturbance and gut microbial diversity. In older adults, shorter sleep duration is associated with an increase in proinflammatory bacteria whereas increasing sleep quality is positively associated with an increase of beneficial Verrucomicrobia and Lentisphaerae phyla. In young adults, the effect of sleep disruption on gut microbiome composition, specifically the ratio of beneficial Firmicutes over Bacteroidetes phyla, remains contradictory and unclear. The findings of this review warrant further research in the modulation of the gut microbiome linking poor sleep with muscle-catabolic consequences throughout the lifespan

Sarcopenia is linked to higher levels of B-type natriuretic peptide and its N-terminal fragment in heart failure: a systematic review and meta-analysis

Aims: Sarcopenia is linked to impaired physical function and exercise tolerance. The aim of this systematic review and meta-analysis was to examine the association of sarcopenia and low appendicular skeletal muscle (ASM) with biomarkers of cardiac function, B-type natriuretic peptide (BNP) and its N-terminal fragment (NT-proBNP), in patients with heart failure (HF). Methods and results: From inception until May 2023, a systematic literature search of observational studies was undertaken utilizing the PubMed, Web of Science, Scopus, and Cochrane Library databases. A meta-analysis employing a random-effects model was used to compute the pooled effects (CRD42023418465). Overall, 16 studies were included in this systematic review and meta-analysis. Our main analysis showed that sarcopenia in HF was linked to significantly higher levels of BNP (MD: 87.76, 95% CI 20.74–154.78, I2 = 61%, P = 0.01) and NT-proBNP (MD: 947.45, 95% CI 98.97–1795.93, I2 = 35%, P = 0.03). Similarly, low ASM was associated with significantly higher levels of BNP (MD: 118.95, 95% CI 46.91–191.00, I2 = 93%, P < 0.01) and NT-proBNP (MD: 672.01, 95% CI 383.72–960.30, I2 = 2%, P < 0.01). The quality of the included cohort studies was considered moderate, using the binary AXIS checklist and the Cochrane Tool to Assess the Risk of Bias in Cohort Studies. Conclusions: In patients with HF, sarcopenia and reduced ASM are associated with considerably higher plasma levels of BNP and NT-proBNP. Future research is required to investigate whether sarcopenia may express dysregulated biomarkers of cardiac function

Impact of probiotics on muscle mass, muscle strength and lean mass: a systematic review and meta-analysis of randomized controlled trials

Probiotics have shown potential to counteract sarcopenia, although the extent to which they can influence domains of sarcopenia such as muscle mass and strength in humans is unclear. The aim of this systematic review and meta-analysis was to explore the impact of probiotic supplementation on muscle mass, total lean mass and muscle strength in human adults. A literature search of randomized controlled trials (RCTs) was conducted through PubMed, Scopus, Web of Science and Cochrane Library from inception until June 2022. Eligible RCTs compared the effect of probiotic supplementation versus placebo on muscle and total lean mass and global muscle strength (composite score of all muscle strength outcomes) in adults (>18 years). To evaluate the differences between groups, a meta-analysis was conducted using the random effects inverse-variance model by utilizing standardized mean differences. Twenty-four studies were included in the systematic review and meta-analysis exploring the effects of probiotics on muscle mass, total lean mass and global muscle strength. Our main analysis (k = 10) revealed that muscle mass was improved following probiotics compared with placebo (SMD: 0.42, 95% CI: 0.10–0.74, I2 = 57%, P = 0.009), although no changes were revealed in relation to total lean mass (k = 12; SMD: -0.03, 95% CI: −0.19 – 0.13, I2 = 0%, P = 0.69). Interestingly, a significant increase in global muscle strength was also observed among six RCTs (SMD: 0.69, 95% CI: 0.33–1.06, I2 = 64%, P = 0.0002). Probiotic supplementation enhances both muscle mass and global muscle strength; however, no beneficial effects were observed in total lean mass. Investigating the physiological mechanisms underpinning different ageing groups and elucidating appropriate probiotic strains for optimal gains in muscle mass and strength are warranted

Topical NSAIDs, intravitreal dexamethasone and peribulbar triamcinolone for pseudophakic macular edema

Background: the purpose of this study is to assess the effectiveness of topical nonsteroidal anti-inflammatory drugs (NSAIDs) and corticosteroids (intravitreal dexamethasone and peribulbar triamcinolone) in treating pseudophakic macular edema (PME). Methods: retrospective study of 33 eyes. Variables included best corrected visual acuity (BCVA; logMAR scale) and central retinal thickness (CRT) and central choroidal thickness (CCT) assessed with swept-source OCT. All patients were initially prescribed topical NSAIDs and reevaluated after 2 months. If improvement in BCVA or CRT was noted, topical NSAIDs were continued until resolution. If no improvement was observed at 2 months or subsequent visits, intravitreal dexamethasone implant was performed. Patients who refused intravitreal treatment were offered peribulbar triamcinolone. Results: after treatment with topical NSAIDs for a median of 2 months, BCVA increased significantly from 0.5 to 0.3 while CRT decreased significantly from 435 to 316 μm. PME resolved in 19 of the 33 eyes (57.6%). Of the 14 recalcitrant cases, 13 were treated with corticosteroids. Of these 13 cases, 9 (69.2%) resolved. BCVA increased non-significantly from 0.7 to 0.4. CRT and CCT decreased significantly from 492 to 317 μm and from 204 to 182 μm respectively. Conclusions: the overall success rate of the treatment algorithm was greater than 80%, a remarkable finding considering that no randomized study has yet been conducted to determine the optimal therapeutic protocol for PME. This is the first study to evaluate choroidal thickness in PME using SS-OCT, which could play a key role in its pathophysiology and provide useful information to improve the management of PME