Mixed-Integer Optimal Control via Reinforcement Learning: A Case Study on Hybrid Vehicle Energy Management

Many optimal control problems require the simultaneous output of continuous and discrete control variables. Such problems are usually formulated as mixed-integer optimal control (MIOC) problems, which are challenging to solve due to the complexity of the solution space. Numerical methods such as branch-and-bound are computationally expensive and unsuitable for real-time control. This paper proposes a novel continuous-discrete reinforcement learning (CDRL) algorithm, twin delayed deep deterministic actor-Q (TD3AQ), for MIOC problems. TD3AQ combines the advantages of both actor-critic and Q-learning methods, and can handle the continuous and discrete action spaces simultaneously. The proposed algorithm is evaluated on a hybrid electric vehicle (HEV) energy management problem, where real-time control of the continuous variable engine torque and discrete variable gear ratio is essential to maximize fuel economy while satisfying driving constraints. Simulation results on different drive cycles show that TD3AQ can achieve near-optimal solutions compared to dynamic programming (DP) and outperforms the state-of-the-art discrete RL algorithm Rainbow, which is adopted for MIOC by discretizing continuous actions into a finite set of discrete values.Comment: 12 pages, 12 figure

Short-Wave Near-Infrared Spectrometer for Alcohol Determination and Temperature Correction

A multichannel short-wave near-infrared (SW-NIR) spectrometer module based on charge-coupled device (CCD) detection was designed. The design relied on a tungsten lamp enhanced by light emitting diodes, a fixed grating monochromator and a linear CCD array. The main advantages were high optical resolution and an optimized signal-to-noise ratio (0.24 nm and 500, resp.) in the whole wavelength range of 650 to 1100 nm. An application to alcohol determination using partial least squares calibration and the temperature correction was presented. It was found that the direct transfer method had significant systematic prediction errors due to temperature effect. Generalized least squares weighting (GLSW) method was utilized for temperature correction. After recalibration, the RMSEP found for the 25°C model was 0.53% v/v and errors of the same order of magnitude were obtained at other temperatures (15, 35 and 40°C). And an r2 better than 0.99 was achieved for each validation set. The possibility and accuracy of using the miniature SW-NIR spectrometer and GLSW transfer calibration method for alcohol determination at different temperatures were proven. And the analysis procedure was simple and fast, allowing a strict control of alcohol content in the wine industry

Privacy-Preserving Joint Edge Association and Power Optimization for the Internet of Vehicles via Federated Multi-Agent Reinforcement Learning

Proactive edge association is capable of improving wireless connectivity at the cost of increased handover (HO) frequency and energy consumption, while relying on a large amount of private information sharing required for decision making. In order to improve the connectivity-cost trade-off without privacy leakage, we investigate the privacy-preserving joint edge association and power allocation (JEAPA) problem in the face of the environmental uncertainty and the infeasibility of individual learning. Upon modelling the problem by a decentralized partially observable Markov Decision Process (Dec-POMDP), it is solved by federated multi-agent reinforcement learning (FMARL) through only sharing encrypted training data for federatively learning the policy sought. Our simulation results show that the proposed solution strikes a compelling trade-off, while preserving a higher privacy level than the state-of-the-art solutions.Comment: 6 pages, 4 figures, IEEE Trans. on Veh. Techno

Mapping Soil Alkalinity and Salinity in Northern Songnen Plain, China with the HJ-1 Hyperspectral Imager Data and Partial Least Squares Regression

In arid and semi-arid regions, identifying and monitoring of soil alkalinity and salinity are in urgently need for preventing land degradation and maintaining ecological balances. In this study, physicochemical, statistical, and spectral analysis revealed that potential of hydrogen (pH) and electrical conductivity (EC) characterized the saline-alkali soils and were sensitive to the visible and near infrared (VIS-NIR) wavelengths. On the basis of soil pH, EC, and spectral data, the partial least squares regression (PLSR) models for estimating soil alkalinity and salinity were constructed. The R2 values for soil pH and EC models were 0.77 and 0.48, and the root mean square errors (RMSEs) were 0.95 and 17.92 dS/m, respectively. The ratios of performance to inter-quartile distance (RPIQ) for the soil pH and EC models were 3.84 and 0.14, respectively, indicating that the soil pH model performed well but the soil EC model was not considerably reliable. With the validation dataset, the RMSEs of the two models were 1.06 and 18.92 dS/m. With the PLSR models applied to hyperspectral data acquired from the hyperspectral imager (HSI) onboard the HJ-1A satellite (launched in 2008 by China), the soil alkalinity and salinity distributions were mapped in the study area, and were validated with RMSEs of 1.09 and 17.30 dS/m, respectively. These findings revealed that the hyperspectral images in the VIS-NIR wavelengths had the potential to map soil alkalinity and salinity in the Songnen Plain, China

Impact of HBeAg on the maturation and function of dendritic cells

AbstractObjectivesHBV infection typically leads to chronic hepatitis in newborns and some adults with weakened immune systems. The mechanisms by which virus escapes immunity remain undefined. Regulatory dendritic cells (DCregs) contributing to immune escape have been described. We wondered whether or not HBeAg as an immunomodulatory protein could induce DCreg which might subsequently result into HBV persistence.MethodsThe immunophenotyping, T-cell activation and cytokine production were analyzed in HBeAg-treated DCs from normal or cyclophosphamide (Cy)-induced immunocompromised mice.ResultsHBeAg tended to promote bone marrow derived DCs (BMDCs) from Cy-treated mice into CD11bhighPIR-B+ regulatory DCs exhibiting the lowest T-cell stimulatory capacity and interleukin (IL)-12p70 production compared with controls. Neutralization of IL-10 significantly inhibited the regulatory effect of these DCs on T-cell stimulation of mature DCs. After lipopolysaccharides (LPS) stimulation, marked phosphorylation of Akt was detected in HBeAg treated DCs from immunocompromised mice. Blocking the PI3K-Akt pathway by LY294002 led to an enhancement of IL-12 production. PI3K signalling pathway appears to be involved in the decreased IL-12 secretion by HBeAg treated DCs.ConclusionsThese findings suggest that HBeAg may program the generation of a new DC subset with regulatory capacity under the condition of immunosuppression, which may presumably contribute to the persistent HBV infection

Nano-scale characterization of white layer in broached Inconel 718

The formation mechanism of white layers during broaching and their mechanical properties are not well investigated and understood to date. In the present study, multiple advanced characterization techniques with nano-scale resolution, including transmission electron microscopy (TEM), transmission Kikuchi diffraction (TKD), atom probe tomography (APT) as well as nano-indentation, have been used to systematically examine the microstructural evolution and corresponding mechanical properties of a surface white layer formed when broaching the nickel-based superalloy Inconel 718. TEM observations showed that the broached white layer consists of nano-sized grains, mostly in the range of 20–50 nm. The crystallographic texture detected by TKD further revealed that the refined microstructure is primarily caused by strong shear deformation. Co-located Al-rich and Nb-rich fine clusters have been identified by APT, which are most likely to be γ′ and γ′′ clusters in a form of co-precipitates, where the clusters showed elongated and aligned appearance associated with the severe shearing history. The microstructural characteristics and crystallography of the broached white layer suggest that it was essentially formed by adiabatic shear localization in which the dominant metallurgical process is rotational dynamic recrystallization based on mechanically-driven subgrain rotations. The grain refinement within the white layer led to an increase of the surface nano-hardness by 14% and a reduction in elastic modulus by nearly 10% compared to that of the bulk material. This is primarily due to the greatly increased volume fraction of grain boundaries, when the grain size was reduced down to the nanoscale

Study on Differences in the Pathology, T Cell Subsets and Gene Expression in Susceptible and Non-Susceptible Hosts Infected with Schistosoma japonicum

More than 40 kinds of mammals in China are known to be naturally infected with Schistosoma japonicum (S. japonicum); Microtus fortis (M. fortis), a species of vole, is the only mammal in which the schistosomes cannot mature or cause significant pathogenic changes. In the current study, we compared the differences in pathology by Hematoxylin-eosin staining and in changes in the T cell subsets with flow cytometry as well as gene expression using genome oligonucleotide microarrays in the lung and liver, before challenge and 10 days post-infection with schistosomes in a S. japonicum-susceptible mouse model of infection, a non-susceptible rat model and the non-permissive host, M. fortis. The results demonstrated that S. japonicum promoted a more intensive immune response and more pathological lesions in M. fortis and rats than in mice. Hematoxylin-eosin staining revealed that the immune effector cells involved were mainly eosinophilic granulocytes supplemented with heterophilic granulocytes and macrophages. The analysis of splenic T cell subsets showed that CD4+ T cell subsets and the CD4+/CD8+ ratio were increased, while the CD8+ T cell subsets decreased remarkably in rats; whereas the CD8+ T cell subsets were increased, but the CD4+/CD8+ ratio was decreased significantly in mice. The analysis of the pattern of gene expression suggested that some immune-associated genes and apoptosis-inducing genes up-regulated, while some development-associated genes were down-regulated in the infected M. fortis compared to the uninfected controls; the three different hosts have different response mechanisms to schistosome infection. The results of this study will be helpful for identifying the key molecules in the immune response to S. japonicum in M. fortis and for understanding more about the underlying mechanism of the response, as well as for elucidating the interaction between S. japonicum and its hosts

Method for Extracting the Glueball Wave Function

We describe a nonperturbative method for calculating the QCD vacuum and glueball wave functions, based on an eigenvalue equation approach to Hamiltonian lattice gauge theory. Therefore, one can obtain more physical information than the conventional simulation methods. For simplicity, we take the 2+1 dimensional U(1) model as an example. The generalization of this method to 3+1 dimensional QCD is straightforward.Comment: 3 pages, Latex. Presented at Lattice 97: 15th International Symposium on Lattice Field Theory, Edinburgh, Scotland, 22-26 Jul 1997, to appear in Nucl. Phys. B(Proc. Suppl.