Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Evolutionary testing supported by slicing and transformation

Evolutionary testing is a search based approach to the automated generation of systematic test data, in which the search is guided by the test data adequacy criterion. Two problems for evolutionary testing are the large size of the search space and structural impediments in the implementation of the program which inhibit the formulation of a suitable fitness function to guide the search. In this paper we claim that slicing can be used to narrow the search space and transformation can be applied to the problem of structural impediments. The paper presents examples of how these two techniques have been successfully employed to make evolutionary testing both more efficient and more effective

Exploring semantic inter-class relationships (SIR) for zero-shot action recognition

© Copyright 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Automatically recognizing a large number of action categories from videos is of significant importance for video understanding. Most existing works focused on the design of more discriminative feature representation, and have achieved promising results when the positive samples are enough. However, very limited efforts were spent on recognizing a novel action without any positive exemplars, which is often the case in the real settings due to the large amount of action classes and the users' queries dramatic variations. To address this issue, we propose to perform action recognition when no positive exemplars of that class are provided, which is often known as the zero-shot learning. Different from other zero-shot learning approaches, which exploit attributes as the intermediate layer for the knowledge transfer, our main contribution is SIR, which directly leverages the semantic inter-class relationships between the known and unknown actions followed by label transfer learning. The inter-class semantic relationships are automatically measured by continuous word vectors, which learned by the skip-gram model using the large-scale text corpus. Extensive experiments on the UCF101 dataset validate the superiority of our method over fully-supervised approaches using few positive exemplars

Coded Merkle Tree: Solving Data Availability Attacks in Blockchains

In this paper, we propose coded Merkle tree (CMT), a novel hash accumulator that offers a constant-cost protection against data availability attacks in blockchains, even if the majority of the network nodes are malicious. A CMT is constructed using a family of sparse erasure codes on each layer, and is recovered by iteratively applying a peeling-decoding technique that enables a compact proof for data availability attack on any layer. Our algorithm enables any node to verify the full availability of any data block generated by the system by just downloading a $\Theta(1)$ byte block hash commitment and randomly sampling $\Theta(\log b)$ bytes, where $b$ is the size of the data block. With the help of only one connected honest node in the system, our method also allows any node to verify any tampering of the coded Merkle tree by just downloading $\Theta(\log b)$ bytes. We provide a modular library for CMT in Rust and Python and demonstrate its efficacy inside the Parity Bitcoin client.Comment: To appear in Financial Cryptography and Data Security (FC) 202

A post-placement side-effect removal algorithm

Side-effects are widely believed to impede program comprehension and have a detrimental effect upon software maintenance. This paper introduces an algorithm for side-effect removal which splits the side-effects into their pure expression meaning and their state-changing meaning. Symbolic execution is used to determine the expression meaning, while transformation is used to place the state-changing part in a suitable location in a transformed version of the program. This creates a program which is semantically equivalent to the original but guaranteed to be free from side-effects. The paper also reports the results of an empirical study which demonstrates that the application of the algorithm causes a significant improvement in program comprehension

A saposin-lipoprotein nanoparticle system for membrane proteins.

A limiting factor in membrane protein research is the ability to solubilize and stabilize such proteins. Detergents are used most often for solubilizing membrane proteins, but they are associated with protein instability and poor compatibility with structural and biophysical studies. Here we present a saposin-lipoprotein nanoparticle system, Salipro, which allows for the reconstitution of membrane proteins in a lipid environment that is stabilized by a scaffold of saposin proteins. We demonstrate the applicability of the method on two purified membrane protein complexes as well as by the direct solubilization and nanoparticle incorporation of a viral membrane protein complex from the virus membrane. Our approach facilitated high-resolution structural studies of the bacterial peptide transporter PeptTSo2 by single-particle cryo-electron microscopy (cryo-EM) and allowed us to stabilize the HIV envelope glycoprotein in a functional state

Sale-based estimation of pharmaceutical concentrations and associated environmental risk in the Japanese wastewater system

Information on sales and emission of selected pharmaceuticals were used to predict their concentrations in Japanese wastewater influent through a >300 of pharmaceuticals data sink. A combined wastewater-based epidemiology and environmental risk analysis follow was established. By comparing predicted environmental concentrations (PECs) of pharmaceuticals in wastewater influent against measured environmental concentrations (MECs) reported in previous studies, it was found that the model gave accurate results for 17 pharmaceuticals (0.5 1 μg/L), and the PECs of 6 pharmaceuticals were extremely high (>10 μg/L) in wastewater effluent, which could be attributed to their high usage rates by consumers and poor removal rates in wastewater treatment plants (WWTPs). Furthermore, environmental risk assessment (ERA) was carried out by calculating the ratio of predicted no effect concentration (PNEC) to PEC of different pharmaceuticals, and it was found that 9 pharmaceuticals were likely to have high toxicity, and 54 pharmaceuticals were likely to have potential toxicity. It is recommended that this is further investigated in detail. The priority screening and environmental risk assessment results on pharmaceuticals can provide reliable basis for policy-making and environmental management

Generalized Mittag-Leffler Distributions and Processes for Applications in Astrophysics and Time Series Modeling

Geometric generalized Mittag-Leffler distributions having the Laplace transform $\frac{1}{1+\beta\log(1+t^\alpha)},00$ is introduced and its properties are discussed. Autoregressive processes with Mittag-Leffler and geometric generalized Mittag-Leffler marginal distributions are developed. Haubold and Mathai (2000) derived a closed form representation of the fractional kinetic equation and thermonuclear function in terms of Mittag-Leffler function. Saxena et al (2002, 2004a,b) extended the result and derived the solutions of a number of fractional kinetic equations in terms of generalized Mittag-Leffler functions. These results are useful in explaining various fundamental laws of physics. Here we develop first-order autoregressive time series models and the properties are explored. The results have applications in various areas like astrophysics, space sciences, meteorology, financial modeling and reliability modeling.Comment: 12 pages, LaTe

From Rotating Atomic Rings to Quantum Hall States

Considerable efforts are currently devoted to the preparation of ultracold neutral atoms in the emblematic strongly correlated quantum Hall regime. The routes followed so far essentially rely on thermodynamics, i.e. imposing the proper Hamiltonian and cooling the system towards its ground state. In rapidly rotating 2D harmonic traps the role of the transverse magnetic field is played by the angular velocity. For particle numbers significantly larger than unity, the required angular momentum is very large and it can be obtained only for spinning frequencies extremely near to the deconfinement limit; consequently, the required control on experimental parameters turns out to be far too stringent. Here we propose to follow instead a dynamic path starting from the gas confined in a rotating ring. The large moment of inertia of the fluid facilitates the access to states with a large angular momentum, corresponding to a giant vortex. The initial ring-shaped trapping potential is then adiabatically transformed into a harmonic confinement, which brings the interacting atomic gas in the desired quantum Hall regime. We provide clear numerical evidence that for a relatively broad range of initial angular frequencies, the giant vortex state is adiabatically connected to the bosonic $\nu=1/2$ Laughlin state, and we discuss the scaling to many particles.Comment: 9 pages, 5 figure

3D mapping from partial observations: An application to utility mapping

Precise mapping of buried utilities is critical to managing massive urban underground infrastructure and preventing utility incidents. Most current research only focuses on generating such maps based on complete information of underground utilities. However, in real-world practice, it is rare that a full picture of buried utilities can be obtained for such mapping. Therefore, this paper explores the problem of generating maps from partial observations of a scene where the actual world is not fully observed. In particular, we focus on the problem of generating 2D/3D maps of buried utilities using a probabilistic based approach. This has the advantage that the method is generic and can be applied to various sources of utility detections, e.g. manhole observations, sensors, and existing records. In this paper, we illustrate our novel methods based on detections from manhole observations and sensor measurements. This paper makes the following new contributions. It is the first time that partial observations have been used to generate utility maps using optimization based approaches. It is the first time that such a large variety of utilities' properties have been considered, such as location, directions, type and size. Another novel contribution is that different kinds of connections are included to reflect the complex layout and structure of buried utilities. Finally, for the first time to the best of our knowledge, we have integrated utility detection, probability calculation, model formulation and map generation into a single framework. The proposed framework represents all detections using a common language of probability distributions and then formulates the mapping problem as an Integer Linear Programming (ILP) problem and the final map is generated based on the solution with the highest probability sum. The effectiveness of this system is evaluated on synthetic and real data using appropriate evaluation metrics