Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning

Offline-to-online reinforcement learning (RL) is a training paradigm that combines pre-training on a pre-collected dataset with fine-tuning in an online environment. However, the incorporation of online fine-tuning can intensify the well-known distributional shift problem. Existing solutions tackle this problem by imposing a policy constraint on the policy improvement objective in both offline and online learning. They typically advocate a single balance between policy improvement and constraints across diverse data collections. This one-size-fits-all manner may not optimally leverage each collected sample due to the significant variation in data quality across different states. To this end, we introduce Family Offline-to-Online RL (FamO2O), a simple yet effective framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O utilizes a universal model to train a family of policies with different improvement/constraint intensities, and a balance model to select a suitable policy for each state. Theoretically, we prove that state-adaptive balances are necessary for achieving a higher policy performance upper bound. Empirically, extensive experiments show that FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark. Codes are available at https://github.com/LeapLabTHU/FamO2O.Comment: NeurIPS 2023 spotlight. 24 pages, 13 figure

A Multilayer Perceptron-based Fast Sunlight Assessment for the Conceptual Design of Residential Neighborhoods under Chinese Policy

In Chinese building codes, it is required that residential buildings receive a minimum number of hours of natural, direct sunlight on a specified winter day, which represents the worst sunlight condition in a year. This requirement is a prerequisite for obtaining a building permit during the conceptual design of a residential project. Thus, officially sanctioned software is usually used to assess the sunlight performance of buildings. These software programs predict sunlight hours based on repeated shading calculations, which is time-consuming. This paper proposed a multilayer perceptron-based method, a one-stage prediction approach, which outputs a shading time interval caused by the inputted cuboid-form building. The sunlight hours of a site can be obtained by calculating the union of the sunlight time intervals (complement of shading time interval) of all the buildings. Three numerical experiments, i.e., horizontal level and slope analysis, and simulation-based optimization are carried out; the results show that the method reduces the computation time to 1/84~1/50 with 96.5%~98% accuracies. A residential neighborhood layout planning plug-in for Rhino 7/Grasshopper is also developed based on the proposed model. This paper indicates that deep learning techniques can be adopted to accelerate sunlight hour simulations at the conceptual design phase

Synthesis and electrochemical studies of iron oxide-based hybrids for lithium ion batteries

With increasing demand for enhanced lithium ion batteries applied in electronic vehicles, hybrid vehicles and mobile electronics, intensive research is being focused on exploring new generation anode materials with high storage capacity, low cost and great safety. Transition metal oxides, as a kind of promising candidates, have attracted much attention and shown much higher capacities, compared with carbon based anode materials (372 mAh g-1), through conversion or insertion/extraction reaction. Among them, iron oxide is of great interest due to its high corrosion resistance, abundance, nontoxicity and low cost. However, there still exist several issues (e.g. low electronic conductivity and large volume change) needed to be addressed before the large-scale commercialization of iron oxide based anodes. Therefore, it’s necessary to conduct scientific studies to further investigate and improve their electrochemical performance and behavior as anodes for lithium ion batteries. Design of novel hybrid nanostructure has been proved to be an effective approach to the technical bottleneck of electrode materials. The hybrid materials present desirable properties by integrating various functions into a single system. In view of the iron oxide/carbon hybrids, Fe3O4@porous carbon matrix and Fe3O4@porous carbon matrix/graphene, are synthesized and the nanostructure’s effect on their electrochemical performance is investigated in depth in this thesis. In the well-defined iron oxide/carbon hybrids, the conductivity and electrical contact of nanocomposites are improved with additive carbon materials. Moreover, hierarchical porous framework can offer large electrode/electrolyte contact area and shorten diffusion distance of lithium ions, which is crucial for the good rate capability. Meanwhile, the cushion to volume change can be provided from the reserved void and porosity during the charge/discharge process. However, the hybridization between metal oxide and carbon species still exhibits a limited improvement, due to the restricted theoretical capacity of carbon species. The mixture of metal oxides with well-fined micro/nanostructures can endow composites with much improved capacity and remarkable rate capability. Therefore, a series of Fe2O3/MnO2 nanocomposites are prepared via controllable annealing conditions. As demonstrated in characterization and performance evaluation, the hollow-structured oxygen-vacancy-rich Fe2O3/MnO2 provides superior electrochemical properites as anode materials for lithium ion batteries, ascribing to their increased contact area with electrolyte and enhanced electronic and ionic conductivity. To further reveal the structure and morphology evolution of iron oxide based anodes during long-term electrochemical cycling, the investigation of hollow-porous Fe2O3 microspheres is conducted via ex-situ X-ray diffraction (XRD) and ex-situ transmission electron microscopy (TEM) techniques. As revealed in the test results, a crystal size induced phase transition (α → γ → β) is found to be contributed to an abnormal performance fluctuation during cycling. These research results are expected to make insightful suggestions to other scientific studies.Doctor of Philosophy (IGS

Unusual one-step formation of Fe2O3/MnO2 core-shell hollow nanorods as a high performance anode material for lithium ion batteries

In the present paper, we propose a facile method toward tuning the interior structure of Fe2O3/MnO2 core-shell nanorods. Through thermal annealing, Fe2O3/MnO2 hollow and Fe2O3/(Fe0.42Mn0.58)2O3/MnO2 solid core-shell nanorods are obtained under the condition of 500 and 600 °C, respectively. We demonstrate that the unusual internal structure transform corresponds to the inward movement of Mn element from the nanorods surface. Owing to the unique core-shell hollow nanostructure, Fe2O3/MnO2 nanorods show excellent electrochemical performances as anodes for lithium ion batteries, whose reversible capacity maintains more than 700 mAhg-1 after 2000 cycles.MOE (Min. of Education, S’pore)Accepted versio

Feasibility of Stabilized Zn and Pb Contaminated Soils as Roadway Subgrade Materials

The authors have developed a new binder, KMP, which is made from oxalic acid-activated phosphate rock, monopotassium phosphate (KH2PO4), and reactive magnesia (MgO). This study explores the acid neutralization capacity, strength characteristics, water-soaking durability, resilient modulus, and pore size distribution of KMP stabilized soils with individual Zn, Pb, or coexisting Zn and Pb contaminants. For comparison purpose, Portland cement (PC) is also tested. The results show that KMP stabilized soils have a higher acid buffering capacity than PC stabilized soils, regardless of the soil contamination conditions. The water stability coefficient and resilient modulus of the KMP stabilized soils are found to be higher than PC stabilized soils. The reasons for the differences in these properties between KMP and PC stabilized soils are interpreted based on the stability and dissolubility of the main hydration products of the KMP and PC stabilized soils, the soil pore distribution, and concentration of Mg or Ca leached from the KMP and PC stabilized soils obtained from the acid neutralization capacity tests. Overall, this study demonstrates that the KMP is effective in stabilizing soils that are contaminated with Zn or Pb alone and mixed Zn and Pb contaminants, and the KMP stabilized soils are better suited as roadway subgrade material

Templated formation of porous Mn2O3 octahedra from Mn-MIL-100 for lithium-ion battery anode materials

Octahedral Mn-MIL-100 metal-organic frameworks (MOFs) are first synthesized, which are then used as templates to fabricate the porous Mn2O3 octahedra through a post-calcination strategy. The morphologies and crystalline structures of as-prepared Mn2O3 octahedra are performed by using field-emission scanning electron microscopy (FESEM), X-ray diffraction (XRD) and transmission electron microscopy (TEM). A reversible lithium storage capacity as high as 755 mA h/g at 0.2 C after 100 cycles is measured from Lithium-ion batteries (LIBs) where the porous Mn2O3 octahedra are acted as anode. Such a high performance indicates that the porous Mn2O3 structure is an excellent anode candidate of LIBs with high capacity and long-life cycling stability.MOE (Min. of Education, S’pore

Fabrication of GaN single crystals at 700°C using Na-Li-Ca mixed flux system

GaN single crystals were grown in Na-Li-Ca flux system of which Li-Ca gradually replaces Na. X-ray powder diffraction analysis confirmed that the structure of GaN samples was wurtzite. As concentration of Li-Ca in flux increased, the shape of GaN crystals was changed from pyramidal, prism to platelets in sequence, and the color of them became transparent gradually. Among of them, the transparent prism crystals were grown first at 700°C when the concentration of Li-Ca was 31.6 mol% and 48.9 mol% in flux solution, respectively. Raman spectra implied that these crystals were stress-free and the transparent crystal grown in the high concentration of Li-Ca solution had high structural quality or low impurity concentrations

Crystalline In–Sb–S framework for highly-performed lithium/sodium storage

info:eu-repo/semantics/publishe

Self-assembled Cu-Ni bimetal oxide 3D in-plane epitaxial structures for highly efficient oxygen evolution reaction

To improve the catalytic activity of copper based electrode for oxygen evolution reaction (OER), double cathodes liquid plasma discharge (DC-LPD) method was developed to fabricate three-dimensional (3D) nest-like structures electrode on the nickel foam (NF). The CuO-NiO/NF electrode obtained through annealing the in-plane epitaxial Cu(OH)2-Ni2O3H/NF in the oven, exhibits excellent electrocatalytic activities for the oxygen evolution reaction at a low overpotential of 319 mV to achieve a current density of 10 mA/cm2. The electrochemically active surface area (ECSA) of CuO-NiO/NF is over 2 times larger than that of Cu(OH)2-Ni2O3H/NF electrode. The synergistic effort between Cu and Ni makes the CuO-NiO/NF electrode a superior electrocatalyst with improved activity and stability as compared to the single NiO/NF electrode.MOE (Min. of Education, S’pore

Phase controllable fabrication of zinc cobalt sulfide hollow polyhedra as high-performance electrocatalysts for the hydrogen evolution reaction

Structure and phase modulations allow the development of highly active, cost-effective and stable electrocatalysts for the hydrogen evolution reaction (HER) but are rather challenging. In this paper, Zn–Co–S hollow/porous polyhedra with controllable phases were fabricated via solvent-based sulfidation at room temperature followed by thermal annealing. The obtained hollow structure Zn–Co–S-300 with an amorphous phase exhibits excellent electrocatalytic HER activity, which is higher than crystalline Zn–Co sulfides annealed at a higher temperature. Zn–Co–S-300 also shows a long-term working stability (91.7% current density retention over 10 hours) in alkaline media. This work provides a feasible approach for the fabrication of homogeneous ternary transition metal sulfide (TMS) electrocatalysts via the Kirkendall effect towards high-efficiency HER applications.MOE (Min. of Education, S’pore