Relative stability toward diffeomorphisms indicates performance in deep nets

Understanding why deep nets can classify data in large dimensions remains a challenge. It has been proposed that they do so by becoming stable to diffeomorphisms, yet existing empirical measurements support that it is often not the case. We revisit this question by defining a maximum-entropy distribution on diffeomorphisms, that allows to study typical diffeomorphisms of a given norm. We confirm that stability toward diffeomorphisms does not strongly correlate to performance on benchmark data sets of images. By contrast, we find that the stability toward diffeomorphisms relative to that of generic transformations $R_f$ correlates remarkably with the test error $\epsilon_t$. It is of order unity at initialization but decreases by several decades during training for state-of-the-art architectures. For CIFAR10 and 15 known architectures, we find $\epsilon_t\approx 0.2\sqrt{R_f}$, suggesting that obtaining a small $R_f$ is important to achieve good performance. We study how $R_f$ depends on the size of the training set and compare it to a simple model of invariant learning.Comment: NeurIPS 2021 Conferenc

Geometric compression of invariant manifolds in neural nets

We study how neural networks compress uninformative input space in models where data lie in $d$ dimensions, but whose label only vary within a linear manifold of dimension $d_\parallel < d$. We show that for a one-hidden layer network initialized with infinitesimal weights (i.e. in the \textit{feature learning} regime) trained with gradient descent, the uninformative $d_\perp=d-d_\parallel$ space is compressed by a factor $\lambda\sim \sqrt{p}$, where $p$ is the size of the training set. We quantify the benefit of such a compression on the test error $\epsilon$. For large initialization of the weights (the \textit{lazy training} regime), no compression occurs and for regular boundaries separating labels we find that $\epsilon \sim p^{-\beta}$, with $\beta_\text{Lazy} = d / (3d-2)$. Compression improves the learning curves so that $\beta_\text{Feature} = (2d-1)/(3d-2)$ if $d_\parallel = 1$ and $\beta_\text{Feature} = (d + d_\perp/2)/(3d-2)$ if $d_\parallel > 1$. We test these predictions for a stripe model where boundaries are parallel interfaces ($d_\parallel=1$) as well as for a cylindrical boundary ($d_\parallel=2$). Next we show that compression shapes the Neural Tangent Kernel (NTK) evolution in time, so that its top eigenvectors become more informative and display a larger projection on the labels. Consequently, kernel learning with the frozen NTK at the end of training outperforms the initial NTK. We confirm these predictions both for a one-hidden layer FC network trained on the stripe model and for a 16-layers CNN trained on MNIST, for which we also find $\beta_\text{Feature}>\beta_\text{Lazy}$. The great similarities found in these two cases support that compression is central to the training of MNIST, and puts forward kernel-PCA on the evolving NTK as a useful diagnostic of compression in deep nets

Discrepancy between FLC assays: Only a problem of quantification?

Immunoglobulin light chains not associated with heavy chains (free light chains, FLC) are found in serum. A growing clinical importance has been assigned to the quantification of the kappa and lambda FLC in serum in the management of plasma cell dyscrasias. At present, automated immunoassays are the only available techniques allowing quantitative determination of serum FLC. Unfortunately, the two reagents available for FLC assay, provide sometimes divergent results. It has been proposed that the different results, unpredictably affecting individual serum samples, are due the different reactivity of reagents against FLC oligomers that are known to be present to a variable extent in serum, especially when lambda FLC are involved. We report a case where we demonstrated that the two reagents recognized differently FLC monomer and dimers

PRDI-BF1 and PRDI-BF1P isoform expressions correlate with disease status in multiple myeloma patients

Human positive regulatory domain I binding factor 1 (PRDI-BF1 or BLIMP-1) is a transcription factor that acts as a master regulator and has crucial roles in the control of differentiation and in maintaining survival of plasma cells (PC). The PRDM1 gene, which codifies for PRDI-BF1, contains an alternative promoter capable of generating a PRDI-BF1 deleted protein (called PRDI-BF1β), which lacks 101 amino acids comprising most of the regulatory domain. PRDI-BF1β has been detected in relevant quantities especially in multiple myeloma cell lines (U266 and NCI- H929). The first aim of the study was to compare, using real time polymerase chain reaction (RT-PCR), the levels of PRDI-BF1 and PRDI-BF1β in myeloma patients and in normal human bone marrow. The second step was the examination of the expression of PRDI-BF1 and PRDI-BF1β isoform depending on disease status and treatment response. We demonstrate the correlation of PRDI-BF1 and the shorter PRDI-BF1β isoform protein levels with the clinical evolution and the management of myeloma patients

Mesangiogenic Progenitor Cells Derived from One Novel CD64brightCD31brightCD14neg Population in Human Adult Bone Marrow

Mesenchymal Stromal Cells (MSCs) have been the object of extensive research for decades, due to their intrinsic clinical value. Nonetheless, the unambiguous identification of a unique in vivo MSC progenitor is still lacking, and the hypothesis that these multipotent cells could possibly arise from different in vivo precursors has been gaining consensus in the last years. We identified a novel multipotent cell population in human adult bone marrow that we firstly named Mesodermal Progenitor Cells (MPCs) for the ability to differentiate toward the mesenchymal lineage while still retaining angiogenic potential. Despite extensive characterization, MPCs positioning within the differentiation pathway and whether they can be ascribed as possible distinctive progenitor of the MSC lineage is still unclear. Here we describe the ex vivo isolation of one novel bone marrow sub-population (Pop#8) with the ability to generate MPCs. Multicolor flow cytometry in combination with either FACS or MACS cell sorting were applied to characterize Pop#8 as CD64brightCD31brightCD14neg. We defined Pop#8 properties in culture, including the potential of Pop#8-derived MPCs to differentiate into MSCs. Gene expression data were suggestive of Pop#8 in vivo involvement in HSC niche constitution/maintenance. Pop#8 resulted over three logs more frequent than other putative MSC progenitors, corroborating the idea that most of the controversies regarding culture expanded MSCs could be the consequence of different culture conditions which select or promote particular sub-populations of precursors

Tympanic Membrane Collagen Expression by Dynamically Cultured Human Mesenchymal Stromal Cell/Star-Branched Poly(ε-Caprolactone) Nonwoven Constructs

The tympanic membrane (TM) primes the sound transmission mechanism due to special fibrous layers mainly of collagens II, III, and IV as a product of TM fibroblasts, while type I is less represented. In this study, human mesenchymal stromal cells (hMSCs) were cultured on star-branched poly("-caprolactone) (*PCL)-based nonwovens using a TM bioreactor and proper dierentiating factors to induce the expression of the TM collagen types. The cell cultures were carried out for one week under static and dynamic conditions. Reverse transcriptase-polymerase chain reaction (RT-PCR) and immunohistochemistry (IHC) were used to assess collagen expression. A Finite Element Model was applied to calculate the stress distribution on the scaolds under dynamic culture. Nanohydroxyapatite (HA) was used as a filler to change density and tensile strength of *PCL scaolds. In dynamically cultured *PCL constructs, fibroblast surface marker was overexpressed, and collagen type II was revealed via IHC. Collagen types I, III and IV were also detected. Von Mises stress maps showed that during the bioreactor motion, the maximum stress in *PCL was double that in HA/*PCL scaolds. By using a *PCL nonwoven scaold, with suitable physico-mechanical properties, an oscillatory culture, and proper dierentiative factors, hMSCs were committed into fibroblast lineage-producing TM-like collagens

Piezoelectric Signals in Vascularized Bone Regeneration

The demand for bone substitutes is increasing in Western countries. Bone graft substitutes aim to provide reconstructive surgeons with off-the-shelf alternatives to the natural bone taken from humans or animal species. Under the tissue engineering paradigm, biomaterial scaffolds can be designed by incorporating bone stem cells to decrease the disadvantages of traditional tissue grafts. However, the effective clinical application of tissue-engineered bone is limited by insufficient neovascularization. As bone is a highly vascularized tissue, new strategies to promote both osteogenesis and vasculogenesis within the scaffolds need to be considered for a successful regeneration. It has been demonstrated that bone and blood vases are piezoelectric, namely, electric signals are locally produced upon mechanical stimulation of these tissues. The specific effects of electric charge generation on different cells are not fully understood, but a substantial amount of evidence has suggested their functional and physiological roles. This review summarizes the special contribution of piezoelectricity as a stimulatory signal for bone and vascular tissue regeneration, including osteogenesis, angiogenesis, vascular repair, and tissue engineering, by considering different stem cell sources entailed with osteogenic and angiogenic potential, aimed at collecting the key findings that may enable the development of successful vascularized bone replacements useful in orthopedic and otologic surgery