Admissible Policy Teaching through Reward Design

We study reward design strategies for incentivizing a reinforcement learning agent to adopt a policy from a set of admissible policies. The goal of the reward designer is to modify the underlying reward function cost-efficiently while ensuring that any approximately optimal deterministic policy under the new reward function is admissible and performs well under the original reward function. This problem can be viewed as a dual to the problem of optimal reward poisoning attacks: instead of forcing an agent to adopt a specific policy, the reward designer incentivizes an agent to avoid taking actions that are inadmissible in certain states. Perhaps surprisingly, and in contrast to the problem of optimal reward poisoning attacks, we first show that the reward design problem for admissible policy teaching is computationally challenging, and it is NP-hard to find an approximately optimal reward modification. We then proceed by formulating a surrogate problem whose optimal solution approximates the optimal solution to the reward design problem in our setting, but is more amenable to optimization techniques and analysis. For this surrogate problem, we present characterization results that provide bounds on the value of the optimal solution. Finally, we design a local search algorithm to solve the surrogate problem and showcase its utility using simulation-based experiments

Effects of image charges, interfacial charge discreteness, and surface roughness on the zeta potential of spherical electric double layers

We investigate the effects of image charges, interfacial charge discreteness, and surface roughness on spherical electric double layers in electrolyte solutions with divalent counter-ions in the setting of the primitive model. By using Monte Carlo simulations and the image charge method, the zeta potential profile and the integrated charge distribution function are computed for varying surface charge strengths and salt concentrations. Systematic comparisons were carried out between three distinct models for interfacial charges: 1) SURF1 with uniform surface charges, 2) SURF2 with discrete point charges on the interface, and 3) SURF3 with discrete interfacial charges and finite excluded volume. By comparing the integrated charge distribution function (ICDF) and potential profile, we argue that the potential at the distance of one ion diameter from the macroion surface is a suitable location to define the zeta potential. In SURF2 model, we find that image charge effects strongly enhance charge inversion for monovalent interfacial charges, and strongly suppress charge inversion for multivalent interfacial charges. For SURF3, the image charge effect becomes much smaller. Finally, with image charges in action, we find that excluded volumes (in SURF3) suppress charge inversion for monovalent interfacial charges and enhance charge inversion for multivalent interfacial charges. Overall, our results demonstrate that all these aspects, i.e., image charges, interfacial charge discreteness, their excluding volumes have significant impacts on the zeta potential, and thus the structure of electric double layers.Comment: 11 pages, 10 figures, some errors are change

Practice Makes Perfect: an iterative approach to achieve precise tracking for legged robots

Precise trajectory tracking for legged robots can be challenging due to their high degrees of freedom, unmodeled nonlinear dynamics, or random disturbances from the environment. A commonly adopted solution to overcome these challenges is to use optimization-based algorithms and approximate the system with a simplified, reduced-order model. Additionally, deep neural networks are becoming a more promising option for achieving agile and robust legged locomotion. These approaches, however, either require large amounts of onboard calculations or the collection of millions of data points from a single robot. To address these problems and improve tracking performance, this paper proposes a method based on iterative learning control. This method lets a robot learn from its own mistakes by exploiting the repetitive nature of legged locomotion within only a few trials. Then, a torque library is created as a lookup table so that the robot does not need to repeat calculations or learn the same skill over and over again. This process resembles how animals learn their muscle memories in nature. The proposed method is tested on the A1 robot in a simulated environment, and it allows the robot to pronk at different speeds while precisely following the reference trajectories without heavy calculations.Comment: 6 pages, 4 figure

Microbial communities in the rhizosphere of pulse crops under semi-arid growing conditions

Non-Peer Reviewe

FFT-LB modeling of thermal liquid-vapor systems

We further develop a thermal LB model for multiphase flows. In the improved model, we propose to use the FFT scheme to calculate both the convection term and external force term. The usage of FFT scheme is detailed and analyzed. By using the FFT algorithm spatiotemporal discretization errors are decreased dramatically and the conservation of total energy is much better preserved. A direct consequence of the improvement is that the unphysical spurious velocities at the interfacial regions can be damped to neglectable scale. Together with the better conservation of total energy, the more accurate flow velocities lead to the more accurate temperature field which determines the dynamical and final states of the system. With the new model, the phase diagram of the liquid-vapor system obtained from simulation is more consistent with that from theoretical calculation. Very sharp interfaces can be achieved. The accuracy of simulation results are also verified by the Laplace law. The FFT scheme can be easily applied to other models for multiphase flows.Comment: 34 pages, 21 figure

An Intelligent Advisor for City Traffic Policies

Nowadays, city streets are populated not only by private vehicles but also by public transport, fleets of workers, and deliveries. Since each vehicle class has a maximum cargo capacity, we study in this article how authorities could improve the road traffic by endorsing long term policies to change the different vehicle proportions: sedans, minivans, full size vans, trucks, and motorbikes, without losing the ability of moving cargo throughout the city. We have performed our study in a realistic scenario (map, road traffic characteristics, and number of vehicles) of the city of Malaga and captured the many details into the SUMO microsimulator. After analyzing the relationship between travel times, emissions, and fuel consumption, we have defined a multiobjective optimization problem to be solved, so as to minimize these city metrics. Our results provide a scientific evidence that we can improve the delivery of goods in the city by reducing the number of heavy duty vehicles and fostering the use of vans instead.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech. This research has been partially funded by the Spanish MINECO and FEDER projects TIN2014-57341-R, TIN2016-81766-REDT, and TIN2017-88213-R. University of Malaga, Andalucia TECH. Daniel H. Stolfi is supported by a FPU grant (FPU13/00954) from the Spanish MECD. Christian Cintrano is supported by a FPI grant (BES-2015-074805) from Spanish MINECO

Evolution of turbulence and in-plane vortices in the near field flow behind multi-scale planar grids

In this experimental work, we carry out detailed two-dimensional particle image velocimetry investigations for the near field wakes behind a conventional and two multi-scale planar grids, using stitched camera fields of view. Statistical independent measurements are conducted focusing on the first few mesh distances downstream of the grid. It is found that the multiple integral length scales originated from the grids loose their importance on the turbulence development after about three mesh distances downstream, much earlier than the distance where the turbulence becomes homogeneous. The largest eddy size, represented by the integral length scales, does not show clear differences in its growth rate among the three grids after an initial development of three times the largest grid size downstream. Nevertheless, when examining individual vortex behaviours using conditional averaging and filtering processes, clear differences are found. The grids are found to have different decay rates of peak vorticity and projected vortex strengths. Despite these differences, the in-plane vorticity correlation function reveals that the mean vortex shape of all the grids shows a universal near-Gaussian pattern which does not change much as the turbulence decays

Pair Production of Charged Higgs Bosons from Bottom-Quark Fusion

For very large values of $\tan\beta$, charged Higgs boson pair production at the Large Hadron Collider (LHC) from the scattering of two bottom quarks can proceed dominantly. We investigated the cross sections of charged Higgs boson pair production via the subprocess $b\bar{b} \to H^+H^-$ at the LHC including the next-to-leading order (NLO) QCD corrections in the minimal supersymmetric standard model (MSSM). We find that the NLO QCD corrections can significantly reduce the dependence of the cross sections on the renormalization and factorization scales.Comment: small changes are mad

Transcriptional and Post-Transcriptional Regulation of Autophagy

Autophagy is a widely conserved process in eukaryotes that is involved in a series of physiological and pathological events, including development, immunity, neurodegenerative disease, and tumorigenesis. It is regulated by nutrient deprivation, energy stress, and other unfavorable conditions through multiple pathways. In general, autophagy is synergistically governed at the RNA and protein levels. The upstream transcription factors trigger or inhibit the expression of autophagyor lysosome-related genes to facilitate or reduce autophagy. Moreover, a significant number of noncoding RNAs (microRNA, circRNA, and lncRNA) are reported to participate in autophagy regulation. Finally, post-transcriptional modifications, such as RNA methylation, play a key role in controlling autophagy occurrence. In this review, we summarize the progress on autophagy research regarding transcriptional regulation, which will provide the foundations and directions for future studies on this self-eating process