Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis

Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to characterize the clusters of the learned representations is still limited. In this paper, we aim to elucidate the characterization from theoretical perspectives. To this end, we consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL), where kernel functions play an important role when applying our theoretical results to other frameworks. We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We investigate the theoretical properties of the kernel-based contrastive loss via this formulation. We first prove that the formulation characterizes the structure of representations learned with the kernel-based contrastive learning framework. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning. We also establish a generalization error bound of KCL. Finally, we show a guarantee for the generalization ability of KCL to the downstream classification task via a surrogate bound

Denoising Cosine Similarity: A Theory-Driven Approach for Efficient Representation Learning

Representation learning has been increasing its impact on the research and practice of machine learning, since it enables to learn representations that can apply to various downstream tasks efficiently. However, recent works pay little attention to the fact that real-world datasets used during the stage of representation learning are commonly contaminated by noise, which can degrade the quality of learned representations. This paper tackles the problem to learn robust representations against noise in a raw dataset. To this end, inspired by recent works on denoising and the success of the cosine-similarity-based objective functions in representation learning, we propose the denoising Cosine-Similarity (dCS) loss. The dCS loss is a modified cosine-similarity loss and incorporates a denoising property, which is supported by both our theoretical and empirical findings. To make the dCS loss implementable, we also construct the estimators of the dCS loss with statistical guarantees. Finally, we empirically show the efficiency of the dCS loss over the baseline objective functions in vision and speech domains

Silica-associated systemic lupus erythematosus with lupus nephritis and lupus pneumonitis A case report and a systematic review of the literature

Introduction Several epidemiological studies have shown that silica exposure triggers the onset of systemic lupus erythematosus (SLE); however, the clinical characteristics of silica-associated SLE have not been well studied. Patient concerns A 67-year-old man with silicosis visited a primary hospital because of a fever and cough. His respiratory condition worsened, regardless of antibiotic medication, and he was referred to our hospital. Diagnosis The patient showed leukopenia, lymphopenia, serum creatinine elevation with proteinuria and hematuria, decreased serum C3 level, and was positive for anti-double stranded DNA antibody, anti-nuclear antibody, and direct Coombs test. He was diagnosed with SLE. Renal biopsy was performed, and the patient was diagnosed with lupus nephritis (class IV-G(A/C) + V defined by the International Society of Nephrology/Renal Pathology Society classification). Computed tomography revealed acute interstitial pneumonitis, bronchoalveolar lavage fluid showed elevation of the lymphocyte fraction, and he was diagnosed with lupus pneumonitis. Interventions Prednisolone (50 mg/day) with intravenous cyclophosphamide (500 mg/body) were initiated. Outcomes The patient showed a favorable response to these therapies. He was discharged from our hospital and received outpatient care with prednisolone slowly tapered off. He had cytomegalovirus and herpes zoster virus infections during treatment, which healed with antiviral therapy. Review: We searched for the literature on sSLE, and selected 11 case reports and 2 population-based studies. The prevalence of SLE manifestations in sSLE patients were comparative to that of general SLE, particularly that of elderly-onset SLE. Our renal biopsy report and previous reports indicate that lupus nephritis of sSLE patients show as various histological patterns as those of general SLE patients. Among the twenty sSLE patients reported in the case articles, three patients developed lupus pneumonitis and two of them died of it. Moreover, two patients died of bacterial pneumonia, one developed aspergillus abscesses, one got pulmonary tuberculosis, and one developed lung cancer. Conclusion Close attention is needed, particularly for respiratory system events and infectious diseases, when treating patients with silica-associated SLE using immunosuppressive therapies

Membrane microdomain switching: a regulatory mechanism of amyloid precursor protein processing

Neuronal activity has an impact on β cleavage of amyloid precursor protein (APP) by BACE1 to generate amyloid-β peptide (Aβ). However, the molecular mechanisms underlying this effect remain to be elucidated. Cholesterol dependency of β cleavage prompted us to analyze immunoisolated APP-containing detergent-resistant membranes from rodent brains. We found syntaxin 1 as a key molecule for activity-dependent regulation of APP processing in cholesterol-dependent microdomains. In living cells, APP associates with syntaxin 1–containing microdomains through X11–Munc18, which inhibits the APP–BACE1 interaction and β cleavage via microdomain segregation. Phosphorylation of Munc18 by cdk5 causes a shift of APP to BACE1-containing microdomains. Neuronal hyperactivity, implicated in Aβ overproduction, promotes the switching of APP microdomain association as well as β cleavage in a partially cdk5-dependent manner. We propose that microdomain switching is a mechanism of cholesterol- and activity-dependent regulation of APP processing in neurons

Studies of Enlargement of Farm Management in the Steep Slope Region of Mountainous Area : I. Research of the Actual Conditions of Farming with Special Reference of Dairying

わが国では酪農の規模は一般に小さく,特に急傾斜地体では粗飼料生産が酪農の規模拡大の制限因子となっており,酪農経営の規模は小さい. この研究調査は急傾斜地体の農業経営の規模拡大による所得増加の方法を研究するための基礎調査として徳島県脇町暮畑地区の8戸の酪農経営につき行なったものである

Studies of Enlargement of Farm Management in the Steep Slope Region of Mountainous Area : II. A Plan for Farm Management in the Steep Slope by Linner Programming Method

急傾斜地の農業経営の安定をはかり,農家の所得を増すために,徳島県美馬郡脇町暮畑部落について,2ヵ年間の農業経営の実態調査に基き,線型計画法によって将来の営農設計を行なった. この地区に適し,現在の基幹作目となっている稲作,煙草,乳牛について単体表を作成し,電子計算機を用いて計算した. この結果,他の作目は土地,労働力,その他の制約が大きいので利益を増すには,酪農の規模を拡大することがもっとも有利であることが示された. 高度の飼養,管理技術を前提とするならば,酪農の規模は粗飼料の給与率を低くする程,その拡大が可能であり,利益の増加をもたらすことが示された

Robustness of MEK-ERK Dynamics and Origins of Cell-to-Cell Variability in MAPK Signaling.

Cellular signaling processes can exhibit pronounced cell-to-cell variability in genetically identical cells. This affects how individual cells respond differentially to the same environmental stimulus. However, the origins of cell-to-cell variability in cellular signaling systems remain poorly understood. Here, we measure the dynamics of phosphorylated MEK and ERK across cell populations and quantify the levels of population heterogeneity over time using high-throughput image cytometry. We use a statistical modeling framework to show that extrinsic noise, particularly that from upstream MEK, is the dominant factor causing cell-to-cell variability in ERK phosphorylation, rather than stochasticity in the phosphorylation/dephosphorylation of ERK. We furthermore show that without extrinsic noise in the core module, variable (including noisy) signals would be faithfully reproduced downstream, but the within-module extrinsic variability distorts these signals and leads to a drastic reduction in the mutual information between incoming signal and ERK activity

Glycosylated porphyra-334 and palythine-threonine from the terrestrial cyanobacterium nostoc commune

Mycosporine-like amino acids (MAAs) are water-soluble UV-absorbing pigments, and structurally different MAAs have been identified in eukaryotic algae and cyanobacteria. In this study novel glycosylated MAAs were found in the terrestrial cyanobacterium Nostoc commune (N. commune). An MAA with an absorption maximum at 334 nm was identified as a hexose-bound porphyra-334 derivative with a molecular mass of 508 Da. Another MAA with an absorption maximum at 322 nm was identified as a two hexose-bound palythine-threonine derivative with a molecular mass of 612 Da. These purified MAAs have radical scavenging activities in vitro, which suggests multifunctional roles as sunscreens and antioxidants. The 612-Da MAA accounted for approximately 60% of the total MAAs and contributed approximately 20% of the total radical scavenging activities in a water extract, indicating that it is the major water-soluble UV-protectant and radical scavenger component. The hexose-bound porphyra-334 derivative and the glycosylated palythine-threonine derivatives were found in a specific genotype of N. commune, suggesting that glycosylated MAA patterns could be a chemotaxonomic marker for the characterization of the morphologically indistinguishable N. commune. The glycosylation of porphyra-334 and palythine-threonine in N. commune suggests a unique adaptation for terrestrial environments that are drastically fluctuating in comparison to stable aquatic environments. © 2013 by the authors