AdaLomo: Low-memory Optimization with Adaptive Learning Rate

Large language models have achieved remarkable success, but their extensive parameter size necessitates substantial memory for training, thereby setting a high threshold. While the recently proposed low-memory optimization (LOMO) reduces memory footprint, its optimization technique, akin to stochastic gradient descent, is sensitive to hyper-parameters and exhibits suboptimal convergence, failing to match the performance of the prevailing optimizer for large language models, AdamW. Through empirical analysis of the Adam optimizer, we found that, compared to momentum, the adaptive learning rate is more critical for bridging the gap. Building on this insight, we introduce the low-memory optimization with adaptive learning rate (AdaLomo), which offers an adaptive learning rate for each parameter. To maintain memory efficiency, we employ non-negative matrix factorization for the second-order moment estimation in the optimizer state. Additionally, we suggest the use of a grouped update normalization to stabilize convergence. Our experiments with instruction-tuning and further pre-training demonstrate that AdaLomo achieves results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.Comment: Fix some typ

Modeling Viscosity of High Titania Slag

TiO2-FeO-Ti2O3 slag system is the dominant system for industrial high-titania slag production. In the present work, viscosities of TiO2-FeO and TiO2-FeO-Ti2O3 systems were experimentally determined using the concentric rotating cylinder method under argon atmosphere. A viscosity model suitable for the TiO2-FeO-Ti2O3 slag system was then established based on the modification of the Vogel-Fulcher-Tammann (VFT) equation. The experimental results indicate that completely melted high-titania slags exhibit very low viscosity of around 0.8 dPa s with negligible dependence on temperature and compositions. However, it increases dramatically with decreasing temperature slightly below the critical temperature. Besides, the increase in FeO content was found to remarkably lower the critical temperature, while the addition of Ti2O3 increases it. The developed model can predict the viscosities of the TiO2-FeO-Ti2O3 and TiO2-FeO systems over wide ranges of compositions and temperatures within experimental uncertainties. The average relative error for the present model calculation is < 18.82 pct, which is better than the previously developed models for silicate slags reported in the literature. An iso-viscosity distribution diagram was made for the TiO2-FeO-Ti2O3 slag system, which can serve as a roadmap for the Ilmenite smelting reduction process as well as the high titania slags tapping process.publishedVersio

Link Prediction Based on Common-Neighbors for Dynamic Social Network

AbstractLink prediction is an important issue in social networks. Most of the existing methods aim to predict interactions between individuals for static networks, ignoring the dynamic feature of social networks. This paper proposes a link prediction method which considers the dynamic topology of social networks. Given a snapshot of a social network at time t (or network evolution between t1 and t2), we seek to accurately predict the edges that will be added during the interval from time t (or t2) to a given future time t′. Our approach utilizes three metrics, the time-varied weight, the change degree of common neighbor and the intimacy between common neighbors. Moreover, we redefine the common neighbors by finding them within two hops. Experiments on DBLP show that our method can reach better results

Research on the influence of virtual modeling and testing–based rubber track system on vibration performance of engineering vehicles

The rubbertrack system can be quickly swapped on the tyres, exerting a smaller ground pressure while generating a greater adhesion to solve the problem vehicles faced in traversing rough and difficult terrain. This paper will discuss the influence of rubber track system on the ride comfort of engineering vehicles with rigid suspension. First, a multi-body dynamic model of the rubber track system and a mathematical model of contact between the ground and the track are established, and then the macro commands are programmed to add many complex contact forces. Moreover, by using the method of physical prototype obstacle testing, the correctness of the simulation model is validated. The ride comfort of the engineering vehicle when equipped with rubber track system is explored by the method of the multi-body dynamics and real vehicle test. The research shows that a flexible roller wheel system can significantly improve the ride comfort of the engineering vehicle when compared to wheeled vehicles. When the vehicle speed is low, the weighted root-meansquare acceleration of the wheeled vehicle and tracked vehicle is almost the same. At the same time, it is verified that the ride comfort of the steelchain tracked vehicles is worse than that of rubber tracked vehicles, due to the polygon effect. Through the multi-body dynamics simulation of the virtual prototype, we can predict and evaluate the ride comfort of vehicles, saving the cost of testing and obtaining the actual experimental data, which has great significance for the research and development of vehicles

An Approach to Mismatched Disturbance Rejection Control for Continuous-Time Uncontrollable Systems

This paper focuses on optimal mismatched disturbance rejection control for linear continuoustime uncontrollable systems. Different from previous studies, by introducing a new quadratic performance index to transform the mismatched disturbance rejection control into a linear quadratic tracking problem, the regulated state can track a reference trajectory and minimize the influence of disturbance. The necessary and sufficient conditions for the solvability and the disturbance rejection controller are obtained by solving a forward-backward differential equation over a finite horizon. A sufficient condition for system stability is obtained over an infinite horizon under detectable condition. This paper details our novel approach for transforming disturbance rejection into a linear quadratic tracking problem. The effectiveness of the proposed method is provided with two examples to demonstrate.Comment: arXiv admin note: substantial text overlap with arXiv:2209.0701

Experiment and simulation research of the ground-borne vibration for a high-speed train

In order to study the effect of the operational loads on the ground-borne vibration of the high-speed train, a train-track coupling model with considering the vertical and horizontal effects is established and applied to calculate the impact of different operational speeds on vibration acceleration. As shown in the results, the vibration acceleration is largely affected by different frequencies generated from different train speeds. By means of an indoor dynamic triaxial test, the impact of different vibration frequencies of a train on soil body is simulated. And a large number of medium- and low-frequency vibration tests are conducted according to the settings of load form, drainage requirement and vibration number of train vibration loads. The experimental results are analyzed to study the effect of different frequencies on dynamic characteristics, and a dynamic strain-time calculation formula, that takes the frequency factor into consideration, is proposed. Meanwhile, the improved formula that considered frequency is substituted into the finite element model (FEM) of the train, so as to analyze the impact of different vibration frequencies on the settlement, is applied. As shown in the results, the proposed improved formula, that considered the frequency, is good at prediction. The effect of vibration efficiency on the engineering can be reflected by a simulated high-speed train model. Based on the simulation model, a reinforcement measure is conducted for the ground-borne, and it is calculated that the settlement is obviously reduced and the service time of the train ground-borne is increased. This paper can provide a reference for a theoretical research and engineering practice