Theoretical Analysis of Primal-Dual Algorithm for Non-Convex Stochastic Decentralized Optimization

In recent years, decentralized learning has emerged as a powerful tool not only for large-scale machine learning, but also for preserving privacy. One of the key challenges in decentralized learning is that the data distribution held by each node is statistically heterogeneous. To address this challenge, the primal-dual algorithm called the Edge-Consensus Learning (ECL) was proposed and was experimentally shown to be robust to the heterogeneity of data distributions. However, the convergence rate of the ECL is provided only when the objective function is convex, and has not been shown in a standard machine learning setting where the objective function is non-convex. Furthermore, the intuitive reason why the ECL is robust to the heterogeneity of data distributions has not been investigated. In this work, we first investigate the relationship between the ECL and Gossip algorithm and show that the update formulas of the ECL can be regarded as correcting the local stochastic gradient in the Gossip algorithm. Then, we propose the Generalized ECL (G-ECL), which contains the ECL as a special case, and provide the convergence rates of the G-ECL in both (strongly) convex and non-convex settings, which do not depend on the heterogeneity of data distributions. Through synthetic experiments, we demonstrate that the numerical results of both the G-ECL and ECL coincide with the convergence rate of the G-ECL

Tidal-flat sediment as an environment for benthic microalgae in southeast Ariake Bay, Kyushu, Japan

The physical and chemical properties of sediment in the tidal area of the Shirakawa River in Kyushu, Japan were studied to obtain knowledge of the habitat of benthic microalgae in Ariake Bay. Sediment samples from the foreshores of southeastern Ariake Bay and estuarine tidal flats in the Shirakawa River were collected during the normal stage of water during high tides in September and November 2016. The particle size distributions in sediment and the amounts of chlorophyll aand volcanic glass shards were examined in core samples. The elemental compositions of bulk sediment samples and agglomerated particles were measured by energy dispersive fluorescent Xray analysis (EDX) and scanning electron microscopy energy-dispersive X-ray spectrometry (SEMEDX), respectively. On the basis of the grain-size composition, the the estuary and the foreshoreright cores were identified as sandy tidal-flat sediment, and the foreshore-left as muddy tidal-flat sediment. Volcanic glass particles derived from Aso Caldera were more abundant in the estuarine cores and the foreshore-right cores than in the foreshore-left cores. The contents of iron and phosphorus were relatively high in the estuarine cores. The SEM-EDX mapping revealed that iron and phosphorus were correlated with each other on the aggregated particles, indicating that iron hydroxides derived from volcanic-ash soil and volcanic ash act as phosphorus carriers in the study area. It was suggested that dissolved iron from the upper Shirakawa River may rapidly aggregate and precipitate at the front edge of the estuary tidal flat and combine with phosphoric acid, providing a phosphorus source for benthic microalgae

Characterization of foreign exchange market using the threshold-dealer-model

We introduce a deterministic dealer model which implements most of the empirical laws, such as fat tails in the price change distributions, long term memory of volatility and non-Poissonian intervals. We also clarify the causality between microscopic dealers' dynamics and macroscopic market's empirical laws.Comment: 10pages, 5figures, 1table, Proceedings of APFA

Embarrassingly Simple Text Watermarks

We propose Easymark, a family of embarrassingly simple yet effective watermarks. Text watermarking is becoming increasingly important with the advent of Large Language Models (LLM). LLMs can generate texts that cannot be distinguished from human-written texts. This is a serious problem for the credibility of the text. Easymark is a simple yet effective solution to this problem. Easymark can inject a watermark without changing the meaning of the text at all while a validator can detect if a text was generated from a system that adopted Easymark or not with high credibility. Easymark is extremely easy to implement so that it only requires a few lines of code. Easymark does not require access to LLMs, so it can be implemented on the user-side when the LLM providers do not offer watermarked LLMs. In spite of its simplicity, it achieves higher detection accuracy and BLEU scores than the state-of-the-art text watermarking methods. We also prove the impossibility theorem of perfect watermarking, which is valuable in its own right. This theorem shows that no matter how sophisticated a watermark is, a malicious user could remove it from the text, which motivate us to use a simple watermark such as Easymark. We carry out experiments with LLM-generated texts and confirm that Easymark can be detected reliably without any degradation of BLEU and perplexity, and outperform state-of-the-art watermarks in terms of both quality and reliability

Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

SGD with momentum acceleration is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum acceleration is Distributed SGD (DSGD) with momentum acceleration (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum acceleration that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and decrease when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the standard deep learning setting, where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta\in [0, 1)$. Through image classification tasks, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions are heterogeneous