AdaLomo: Low-memory Optimization with Adaptive Learning Rate

Large language models have achieved remarkable success, but their extensive parameter size necessitates substantial memory for training, thereby setting a high threshold. While the recently proposed low-memory optimization (LOMO) reduces memory footprint, its optimization technique, akin to stochastic gradient descent, is sensitive to hyper-parameters and exhibits suboptimal convergence, failing to match the performance of the prevailing optimizer for large language models, AdamW. Through empirical analysis of the Adam optimizer, we found that, compared to momentum, the adaptive learning rate is more critical for bridging the gap. Building on this insight, we introduce the low-memory optimization with adaptive learning rate (AdaLomo), which offers an adaptive learning rate for each parameter. To maintain memory efficiency, we employ non-negative matrix factorization for the second-order moment estimation in the optimizer state. Additionally, we suggest the use of a grouped update normalization to stabilize convergence. Our experiments with instruction-tuning and further pre-training demonstrate that AdaLomo achieves results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.Comment: Fix some typ

Effects of Annealing on Residual Stress in Ta2O5 Films Deposited by Dual Ion Beam Sputtering

Optical coatings deposited by the dual ion beam sputtering (DIBS) method usually show high compressive stress, which results in severe wavefront deformation of optical elements. Annealing post-treatment has been widely used to control the residual stress of optical coatings. However, the effect of annealing on the stress of Ta2O5 films deposited by the IBS method has not been reported in detail. In this study, different thicknesses of Ta2O5 films were deposited by IBS and annealed at different temperatures from 473 to 973 K in air, and the effect of annealing on the stress of Ta2O5 films was investigated. The as-deposited Ta2O5 films deposited by IBS show high compressive stress, which are about 160 MPa. The compressive stress decreases linearly with the increasing temperature, and the wavefront deformation of Ta2O5 films increases linearly with film thickness (within 20 μm) at the same annealing temperature. When the temperature rises to 591 K, Ta2O5 films with zero-stress can be obtained. Ta2O5 films show tensile stress instead of compressive stress with further increasing annealing temperature, and the tensile stress increases with increasing temperature. Meanwhile, with the increasing annealing temperature, the refractive index of Ta2O5 film decreases, indicating the decreasing packing density. The atomic force microscope (AFM) test results show that surface roughness of Ta2O5 films slowly increases with the increasing of annealing temperature. Moreover, X-ray photoelectron spectroscopy (XPS) analysis shows that the Ta in Ta2O5 films can be further oxidized with increasing annealing temperature, namely, the absorption of Ta2O5 film can be reduced. X-ray diffraction (XRD) analysis shows that the annealing temperature should be below 923 K to maintain the amorphous structure of the Ta2O5 film

Fabrication of broadband antireflection coatings using wavelength-indirect broadband optical monitoring

Multi-layer optical coatings with complex spectrum requirements, such as multi-band pass filters, notch filters, ultra-broadband antireflection coating and etc., whose working wavelength is out of monitoring wavelength range, are difficult to be fabricated using direct broadband optical monitoring (BBOM). In this paper, a broadband antireflection (AR) coating in the wavelength range from 1300 nm to 2000 nm at 45 incident was designed and deposited by dual ion beam sputtering (DIBS). Ta2O5 and SiO2 were chosen as high and low refractive index coating materials, respectively. The optimized coating structure contains 4 non-quarter-wave (QW) layers. In order to obtain high transmittance, the most important is to realize the thickness accurate control. Due to the limitation of the monitoring wavelength range, which is only from 450 nm to 1000 nm, a wavelength-indirect broadband optical monitoring strategy was successfully employed to control the layers thickness during the deposition process. At last, a good agreement between theoretical and measured transmittance is obtained. The maximum error (the first layer) is only about 5.3% and the minimum error (the third layer) is about -0.25% base on the results of reverse engineering analysis. (C) 2017 Elsevier GmbH. All rights reserved

Effects of cyclodextrins on the antimicrobial activity of plant-derived essential oil compounds

Essential oils (EOs) from plants are considered to be a safer alternative when compared to synthetic antimicrobial food additives. However, a major drawback of many EOs is their hydrophobic nature, which makes them insoluble in water based media and matrices. Although cyclodextrins (CDs) can increase the solubility of EO compounds, the effects of CDs on the antimicrobial activity of EOs have not been reported. In this paper, four different EO compounds (carvacrol, eugenol, linalool and 2-pentanoylfuran) were chosen to study the influence of CDs on the solubility and antimicrobial activity on bacteria and yeast. The greatest enhancement with regards to solubility of the four test compounds was achieved by hydroxypropyl-β-CD. In most instances, not only were the minimal antimicrobial concentrations of EO compounds decreased, but the interactivity of two combined EO compounds could be strengthened by the co-addition of CDs. Furthermore, the combination of carvacrol with hydroxypropyl-β-CD caused a marked change in the major membrane lipid composition of all microorganisms investigated; while scanning electron microscopy revealed that cellular integrity was significantly affected by 2× MIC, ultimately resulting in cell lysis. © 2012 Elsevier Ltd. All rights reserved

Construction of Virtual Interaction Location Prediction Model Based on Distance Cognition

Due to the difference in distance cognition between virtual and real symmetric space, it is difficult for users to accurately interact with the target in the Digital Twin system. In order to study the cross-effects of interaction task, target size and target location on the accuracy of egocentric peripersonal distance cognition, a 2 × 5 × 9 × 5 asymmetric experiment was designed and carried out. There were two kinds of interaction tasks, five kinds of interaction target widths and nine kinds of spatial locations set to estimate the five egocentric peripersonal distances. Based on the experimental data, with interaction task, target width and the actual spatial location as independent variables and virtual interaction location as a dependent variable, the mapping model between the actual physical location and virtual interaction location of different interaction targets was constructed and evaluated by multiple linear regression method. The results showed that the prediction model constructed by stepwise regression method was simple and less computationally intensive, but it had better stability and prediction ability. The correlation coefficients R2 on xp, yp and zp were 0.994, 0.999 and 0.998, RMSE values were 2.583 cm, 1.0774 cm and 1.3155 cm, rRMSE values were 26.57%, 12.60% and 1.15%, respectively. The research of relevant experiments and the construction of models are helpful to solve the layout optimization problem of virtual interactive space in the Digital Twin system

Study on the Properties of 1319 nm Ultra-High Reflector Deposited by Electron Beam Evaporation Assisted by an Energetic RF Ion Source

Ultra-high reflectors, working as a critical optical component, has been widely applied as a cavity mirror in fine optical systems such as laser gyro, F-P interferometer, etc. For decades, ion beam sputtering (IBS) technology, which can deposit ultra-low loss and dense layers, has been commonly believed to be the only and irreplaceable method to fabricate ultra-high reflectors. Thus, reports on other methods are rare and a reflectivity above 99.99% obtained by evaporation technology (including ion assisted evaporation) has not been seen yet. In the present study, an energetic radio frequency (RF) ion source was introduced during the electron beam evaporation process, which improved the layer quality dramatically. An ultra-high reflector at 1319 nm with reflectivity of 99.992% (measured by cavity-ring down method) was successfully deposited on a phi 100 mm x 25 mm single crystal silicon substrate whose surface roughness was approximately 0.420 nm. The surface figure of the reflector was accurately controlled superior to 1/6 lambda (lambda = 632.8 nm). The measured absorption was approximately 3-5 ppm and the calculated scatter based on surface roughness measurement was approximately 6.64 ppm. Total loss of the reflector was systematically discussed. This study showed that it is possible to apply electron beam evaporation in ultra-high reflector manufacture and the method is capable of depositing reflectors with an aperture larger than phi 600 mm which is the maximum capacity of current IBS technology

Is combined use of intravenous and intraarticular tranexamic acid superior to intravenous or intraarticular tranexamic acid alone in total knee arthroplasty? A meta-analysis of randomized controlled trials

Abstract Background Tranexamic acid (TXA) has been proven to be effective in reducing blood loss and transfusion rate after total knee arthroplasty (TKA) without increasing the risk of deep vein thrombosis (DVT) and pulmonary embolism (PE). Recently, an increasing number of studies have been interested in applying combined intravenous (IV) with intraarticular (IA) tranexamic acid in total knee arthroplasty. The purpose of this meta-analysis was to compare the blood loss and complications of combined TXA with IV TXA or IA TXA on TKA. Methods Systematic search of literatures were conducted to identify related articles that were published in PubMed, MEDLINE, Embase, the Cochrane Library, SpringerLink, ClinicalTrials.gov, and Ovid from their inception to September 2016. All studies that compare blood loss and complications of combined TXA and IV TXA or IA TXA on TKA were included. Main outcomes were collected and analyzed by the Review Manager 5.3. Results Five studies were included in the present meta-analysis. There was significant difference in total blood loss and blood volume of drainage when compared combined TXA group with IV TXA group or IA TXA group (P  0.05). Conclusions Compared with administration of IA TXA or IV TXA alone on TKA, combined use of TXA has advantages in reducing total blood loss and blood volume of drainage without increasing the incidence of thromboembolic complications. We recommend combined TXA as the preferred option for patients undergoing TKA

Transparent and water repellent ceria film grown by atomic layer deposition

Transparent and hydrophobic ceria film was fabricated by atomic layer deposition (ALD). The ceria coatings were characterized by goniometry, atomic force microscopy, X-ray photoelectron spectroscopy and variable angles spectroscopic ellipsometry. The hydrophobicity of the ceria coatings was investigated with water contact angles achieving as high as 105 degrees. The effect of annealing or surface relaxation on the hydrophobicity was studied. Surface chemistry analysis of the ceria surfaces was carried out to understand the surface treatment towards the wettability of the ALD coatings. The proposed ALD ceria film offers the advantages of hydrophobic coatings covering fine optical lens such as band-pass filter. (C) 2017 Published by Elsevier B.V

Elucidation of the Key Therapeutic Targets and Potential Mechanisms of Marmesine against Knee Osteoarthritis via Network Pharmacological Analysis and Molecular Docking

Background. Marmesine, a major active ingredient isolated from Radix Angelicae biseratae (Duhuo), has been reported to have multiple pharmacological activities. However, its therapeutic effects against knee osteoarthritis (OA) remain poorly investigated. The present study is aimed at uncovering the core targets and signaling pathways of marmesine against osteoarthritis using a combined method of bioinformatics and network pharmacology. Methods. We utilized SwissTargetPrediction and PharmMapper to collect the potential targets of marmesine. OA-related differentially expressed genes (DEGs) were identified from GSE98918 dataset. Then, the intersection genes between DEGs and candidate genes of marmesine were subjected to protein-protein interaction (PPI) network construction and functional enrichment analysis. The core targets were verified using the molecular docking technology. Results. A total of 320 marmesine-related genes and 5649 DEGs and 60 ingredient-disease targets between them were identified. The results of functional enrichment analyses revealed that response to oxygen levels, neuroinflammatory response, PI3K-Akt signaling pathway, MAPK signaling pathway, FoxO signaling pathway, and osteoclast differentiation was identified as the potential mechanisms of marmesine against OA. EGFR, CASP3, MMP9, PPARG, and MAPK1 served as hub genes regulated by marmesine in the treatment of OA, and the molecular docking further verified the results. Conclusion. Marmesine exerts the therapeutic effects against OA through multitarget and multipathways, in which EGFR, CASP3, MMP9, PPARG, and MAPK1 might be hub genes. Our research indicated that the combination of bioinformatics and network pharmacology could serve as an effective approach for investigating the potential mechanisms of natural product

Effects of Annealing on Residual Stress in Ta2O5 Films Deposited by Dual Ion Beam Sputtering

Optical coatings deposited by the dual ion beam sputtering (DIBS) method usually show high compressive stress, which results in severe wavefront deformation of optical elements. Annealing post-treatment has been widely used to control the residual stress of optical coatings. However, the effect of annealing on the stress of Ta2O5 films deposited by the IBS method has not been reported in detail. In this study, different thicknesses of Ta2O5 films were deposited by IBS and annealed at different temperatures from 473 to 973 K in air, and the effect of annealing on the stress of Ta2O5 films was investigated. The as-deposited Ta2O5 films deposited by IBS show high compressive stress, which are about 160 MPa. The compressive stress decreases linearly with the increasing temperature, and the wavefront deformation of Ta2O5 films increases linearly with film thickness (within 20 μm) at the same annealing temperature. When the temperature rises to 591 K, Ta2O5 films with zero-stress can be obtained. Ta2O5 films show tensile stress instead of compressive stress with further increasing annealing temperature, and the tensile stress increases with increasing temperature. Meanwhile, with the increasing annealing temperature, the refractive index of Ta2O5 film decreases, indicating the decreasing packing density. The atomic force microscope (AFM) test results show that surface roughness of Ta2O5 films slowly increases with the increasing of annealing temperature. Moreover, X-ray photoelectron spectroscopy (XPS) analysis shows that the Ta in Ta2O5 films can be further oxidized with increasing annealing temperature, namely, the absorption of Ta2O5 film can be reduced. X-ray diffraction (XRD) analysis shows that the annealing temperature should be below 923 K to maintain the amorphous structure of the Ta2O5 film