Theoretical Analysis of Primal-Dual Algorithm for Non-Convex Stochastic Decentralized Optimization

In recent years, decentralized learning has emerged as a powerful tool not only for large-scale machine learning, but also for preserving privacy. One of the key challenges in decentralized learning is that the data distribution held by each node is statistically heterogeneous. To address this challenge, the primal-dual algorithm called the Edge-Consensus Learning (ECL) was proposed and was experimentally shown to be robust to the heterogeneity of data distributions. However, the convergence rate of the ECL is provided only when the objective function is convex, and has not been shown in a standard machine learning setting where the objective function is non-convex. Furthermore, the intuitive reason why the ECL is robust to the heterogeneity of data distributions has not been investigated. In this work, we first investigate the relationship between the ECL and Gossip algorithm and show that the update formulas of the ECL can be regarded as correcting the local stochastic gradient in the Gossip algorithm. Then, we propose the Generalized ECL (G-ECL), which contains the ECL as a special case, and provide the convergence rates of the G-ECL in both (strongly) convex and non-convex settings, which do not depend on the heterogeneity of data distributions. Through synthetic experiments, we demonstrate that the numerical results of both the G-ECL and ECL coincide with the convergence rate of the G-ECL

Extra-embryonic endoderm cells derived from ES cells induced by GATA Factors acquire the character of XEN cells

<p>Abstract</p> <p>Background</p> <p>Three types of cell lines have been established from mouse blastocysts: embryonic stem (ES) cells, trophoblast stem (TS) cells, and extra-embryonic endoderm (XEN) cells, which have the potential to differentiate into their respective cognate lineages. ES cells can differentiate <it>in vitro </it>not only into somatic cell lineages but into extra-embryonic lineages, including trophectoderm and extra-embryonic endoderm (ExEn) as well. TS cells can be established from ES cells by the artificial repression of <it>Oct3/4 </it>or the upregulation of <it>Cdx2 </it>in the presence of FGF4 on feeder cells. The relationship between these embryo-derived XEN cells and ES cell-derived ExEn cell lines remains unclear, although we have previously reported that overexpression of <it>Gata4 </it>or <it>Gata6 </it>induces differentiation of mouse ES cells into extra-embryonic endoderm in vitro.</p> <p>Results</p> <p>A system in which GATA factors were conditionally activated revealed that the cells continue to proliferate while expressing a set of extra-embryonic endoderm markers, and, following injection into blastocysts, contribute only to the extra-embryonic endoderm lineage <it>in vivo</it>. Although the <it>in vivo </it>contribution is limited to cells of parietal endoderm lineage, Gata-induced extra-embryonic endoderm cells (gExEn) can be induced to differentiate into visceral endoderm-like cells <it>in vitro </it>by repression of <it>Gata6</it>. During early passage, the propagation of gExEn cells is dependent on the expression of the <it>Gata6 </it>transgene. These cells, however, lose this dependency following establishment of endogenous <it>Gata6 </it>expression.</p> <p>Conclusion</p> <p>We show here that Gata-induced extra-embryonic endoderm cells derived from ES cells mimic the character of XEN cells. These findings indicate that Gata transcription factors are sufficient for the derivation and propagation of XEN-like extra-embryonic endoderm cells from ES cells.</p

Clustering of Emission-line Stars in the W5E HII region

We have made a new survey of emission-line stars in the W5E HII region to investigate the population of PMS stars near the OB stars by using the Wide Field Grism Spectrograph 2 (WFGS2). A total of 139 H-alpha emission stars were detected and their g'i'-photometry was performed. The spatial distribution of them shows three aggregates, i.e., two aggregates near the bright-rimmed clouds at the edge of W5E HII region (BRC 13 and BRC 14) and one near the exciting O7V star. The age and mass of each H-alpha star were estimated from the extinction corrected color-magnitude diagram and theoretical evolutionary tracks. We found, for the first time in this region, that the young stars near the exciting star are systematically older (4 Myr) than those near the edge of the HII region (1 Myr). This result supports that the formation of stars proceed sequentially from the center of HII region to the eastern bright rim. We further suggest a possibility that the birth of low mass stars near the exciting star of HII region precede the production of massive OB stars in the pre-existing molecular cloud.Comment: 16 pages, 7 figures, 3 tables, accepted for publication in PAS

Embarrassingly Simple Text Watermarks

We propose Easymark, a family of embarrassingly simple yet effective watermarks. Text watermarking is becoming increasingly important with the advent of Large Language Models (LLM). LLMs can generate texts that cannot be distinguished from human-written texts. This is a serious problem for the credibility of the text. Easymark is a simple yet effective solution to this problem. Easymark can inject a watermark without changing the meaning of the text at all while a validator can detect if a text was generated from a system that adopted Easymark or not with high credibility. Easymark is extremely easy to implement so that it only requires a few lines of code. Easymark does not require access to LLMs, so it can be implemented on the user-side when the LLM providers do not offer watermarked LLMs. In spite of its simplicity, it achieves higher detection accuracy and BLEU scores than the state-of-the-art text watermarking methods. We also prove the impossibility theorem of perfect watermarking, which is valuable in its own right. This theorem shows that no matter how sophisticated a watermark is, a malicious user could remove it from the text, which motivate us to use a simple watermark such as Easymark. We carry out experiments with LLM-generated texts and confirm that Easymark can be detected reliably without any degradation of BLEU and perplexity, and outperform state-of-the-art watermarks in terms of both quality and reliability

Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

SGD with momentum acceleration is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum acceleration is Distributed SGD (DSGD) with momentum acceleration (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum acceleration that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and decrease when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the standard deep learning setting, where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta\in [0, 1)$. Through image classification tasks, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions are heterogeneous

The evolutionally-conserved function of group B1 Sox family members confers the unique role of Sox2 in mouse ES cells.

BACKGROUND: In mouse ES cells, the function of Sox2 is essential for the maintenance of pluripotency. Since the Sox-family of transcription factors are well conserved in the animal kingdom, addressing the evolutionary origin of Sox2 function in pluripotent stem cells is intriguing from the perspective of understanding the origin of pluripotency. RESULTS: Here we approach this question using a functional complementation assay in inducible Sox2-null ES cells. Assaying mouse Sox proteins from different Groups, we found that only Group B1 and Group G proteins were able to support pluripotency. Interestingly, invertebrate homologs of mammalian Group B1 Sox proteins were able to replace the pluripotency-associated function of mouse Sox2. Moreover, the mouse ES cells rescued by the Drosophila SoxNeuro protein are able to contribute to chimeric embryos. CONCLUSIONS: These data indicate that the function of mouse Sox2 supporting pluripotency is based on an evolutionally conserved activity of the Group B1 Sox family. Since pluripotent stem cell population in developmental process could be regarded as the evolutional novelty in vertebrates, it could be regarded as a co-optional use of their evolutionally conserved function

Pharmacokinetic study of pleural fluid penetration of carbapenem antibiotic agents in chemical pleurisy

SummaryStudy objectivesWe investigated pleural fluid penetration of carbapenem antibiotic agents [imipenem (IPM), panipenem (PAPM), meropenem (MEPM), and biapenem (BIPM)] using an experimental rabbit pleuritis model to clarify the usefulness of the carbapenem agents for the treatment of bacterial pleurisy or pyothorax.Measurements and resultsSerum and pleural fluid specimens were serially collected at 5, 10, 15, 30, 60, 90, 120, 180, 240, 300, and 360min after antibiotic administration for measurement of antibiotic levels. We investigated each agent alone as well as drug solutions containing each agent and a dehydropeptidase-I-specific inhibitor, cilastatin (CS), to remove the influence of dehydropeptidase-I-related hydrolysis. Groups of animals (n=3) received each carbapenem agent with or without CS. Serum and pleural fluid antibiotic levels were measured by high-performance liquid chromatography (HPLC). Because Cmax is not useful for evaluating the antimicrobial effects of carbapenem antibiotic agents due to their dose-dependent antimicrobial activity, we also investigated the AUC, which is correlated with the total drug levels in vivo.Among the drug solutions containing CS, MEPM/CS had the highest pleural fluid AUC0–360 (1594.8±510.3μgmin/ml), and the highest pleural fluid AUC0–360/plasma AUC0–360 ratio (0.79±0.04). BIPM/CS had the highest plasma AUC0–360 (3040.1±1525.9μgmin/ml). In pleural fluid AUC0–360/plasma AUC0–360 ratio MEPM/CS was significantly higher than those for the remaining agents. In pleural fluid AUC0–360 and plasma AUC0–360 there were no significant differences among these mixed solutions.ConclusionsMEPM had the most favorable pleural fluid penetration. Pleural fluid penetration should be examined in infection models and in clinical trials