59 research outputs found

    Foundation Model's Embedded Representations May Detect Distribution Shift

    Full text link
    Distribution shifts between train and test datasets obscure our ability to understand the generalization capacity of neural network models. This topic is especially relevant given the success of pre-trained foundation models as starting points for transfer learning (TL) models across tasks and contexts. We present a case study for TL on a pre-trained GPT-2 model onto the Sentiment140 dataset for sentiment classification. We show that Sentiment140's test dataset MM is not sampled from the same distribution as the training dataset PP, and hence training on PP and measuring performance on MM does not actually account for the model's generalization on sentiment classification.Comment: 14 pages, 8 figures, 5 table

    On CAT(Îș\kappa) surfaces

    Full text link
    We study the properties of CAT(Îș\kappa) surfaces: length metric spaces homeomorphic to a surface having curvature bounded above in the sense of satisfying the CAT(Îș\kappa) condition locally. The main facts about CAT(Îș\kappa) surfaces seem to be largely a part of mathematical folklore, and this paper is intended to rectify the situation. We provide three distinct proofs of the fact that CAT(Îș\kappa}) surfaces have bounded integral curvature. This fact allows us to use the established theory of surfaces of bounded curvature to derive further properties of CAT(Îș\kappa) surfaces. Among other results, we show that such surfaces can be approximated by smooth Riemannian surfaces of Gaussian curvature at most Îș\kappa. We do this by giving explicit formulas for smoothing the vertices of model polyhedral surfaces.Comment: 23 pages, 3 figure

    Efficient kernel surrogates for neural network-based regression

    Full text link
    Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the effectiveness and limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to assess their precise dependence on the training data and to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and classification using logistic regression, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework provides insights into understanding the robustness of the various approximants and suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription.Comment: 29 pages. software used to reach results available upon request, approved for release by Pacific Northwest National Laborator

    Lipoic acid plays a role in scleroderma: insights obtained from scleroderma dermal fibroblasts

    Get PDF
    Abstract Introduction Systemic sclerosis (SSc) is a connective tissue disease characterized by fibrosis of the skin and organs. Increase in oxidative stress and platelet-derived growth factor receptor (PDGFR) activation promote type I collagen (Col I) production, leading to fibrosis in SSc. Lipoic acid (LA) and its active metabolite dihydrolipoic acid (DHLA) are naturally occurring thiols that act as cofactors and antioxidants and are produced by lipoic acid synthetase (LIAS). Our goals in this study were to examine whether LA and LIAS were deficient in SSc patients and to determine the effect of DHLA on the phenotype of SSc dermal fibroblasts. N-acetylcysteine (NAC), a commonly used thiol antioxidant, was included as a comparison. Methods Dermal fibroblasts were isolated from healthy subjects and patients with diffuse cutaneous SSc. Matrix metalloproteinase (MMPs), tissue inhibitors of MMPs (TIMP), plasminogen activator inhibitor 1 (PAI-1) and LIAS were measured by enzyme-linked immunosorbent assay. The expression of Col I was measured by immunofluorescence, hydroxyproline assay and quantitative PCR. PDGFR phosphorylation and α-smooth muscle actin (αSMA) were measured by Western blotting. Student’s t-tests were performed for statistical analysis, and P-values less than 0.05 with two-tailed analysis were considered statistically significant. Results The expression of LA and LIAS in SSc dermal fibroblasts was lower than normal fibroblasts; however, LIAS was significantly higher in SSc plasma and appeared to be released from monocytes. DHLA lowered cellular oxidative stress and decreased PDGFR phosphorylation, Col I, PAI-1 and αSMA expression in SSc dermal fibroblasts. It also restored the activities of phosphatases that inactivated the PDGFR. SSc fibroblasts produced lower levels of MMP-1 and MMP-3, and DHLA increased them. In contrast, TIMP-1 levels were higher in SSc, but DHLA had a minimal effect. Both DHLA and NAC increased MMP-1 activity when SSc cells were stimulated with PDGF. In general, DHLA showed better efficacy than NAC in most cases. Conclusions DHLA acts not only as an antioxidant but also as an antifibrotic because it has the ability to reverse the profibrotic phenotype of SSc dermal fibroblasts. Our study suggests that thiol antioxidants, including NAC, LA, or DHLA, could be beneficial for patients with SSc.http://deepblue.lib.umich.edu/bitstream/2027.42/112060/1/13075_2014_Article_411.pd

    The significance of epigenetic alterations in lung carcinogenesis

    Full text link
    • 

    corecore