Defense Against Reward Poisoning Attacks in Reinforcement Learning

We study defense strategies against reward poisoning attacks in reinforcement learning. As a threat model, we consider attacks that minimally alter rewards to make the attacker's target policy uniquely optimal under the poisoned rewards, with the optimality gap specified by an attack parameter. Our goal is to design agents that are robust against such attacks in terms of the worst-case utility w.r.t. the true, unpoisoned, rewards while computing their policies under the poisoned rewards. We propose an optimization framework for deriving optimal defense policies, both when the attack parameter is known and unknown. Moreover, we show that defense policies that are solutions to the proposed optimization problems have provable performance guarantees. In particular, we provide the following bounds with respect to the true, unpoisoned, rewards: a) lower bounds on the expected return of the defense policies, and b) upper bounds on how suboptimal these defense policies are compared to the attacker's target policy. We conclude the paper by illustrating the intuitions behind our formal results, and showing that the derived bounds are non-trivial

Dimension Reduction via Colour Refinement

Colour refinement is a basic algorithmic routine for graph isomorphism testing, appearing as a subroutine in almost all practical isomorphism solvers. It partitions the vertices of a graph into "colour classes" in such a way that all vertices in the same colour class have the same number of neighbours in every colour class. Tinhofer (Disc. App. Math., 1991), Ramana, Scheinerman, and Ullman (Disc. Math., 1994) and Godsil (Lin. Alg. and its App., 1997) established a tight correspondence between colour refinement and fractional isomorphisms of graphs, which are solutions to the LP relaxation of a natural ILP formulation of graph isomorphism. We introduce a version of colour refinement for matrices and extend existing quasilinear algorithms for computing the colour classes. Then we generalise the correspondence between colour refinement and fractional automorphisms and develop a theory of fractional automorphisms and isomorphisms of matrices. We apply our results to reduce the dimensions of systems of linear equations and linear programs. Specifically, we show that any given LP L can efficiently be transformed into a (potentially) smaller LP L' whose number of variables and constraints is the number of colour classes of the colour refinement algorithm, applied to a matrix associated with the LP. The transformation is such that we can easily (by a linear mapping) map both feasible and optimal solutions back and forth between the two LPs. We demonstrate empirically that colour refinement can indeed greatly reduce the cost of solving linear programs

An Introduction to Patchouli (Pogostemon cablin Benth.) – A Medicinal and Aromatic Plant: It’s Importance to Mankind

Patchouli (Pogostemon cablin Benth.) is a plant from Lamiaceae family, well known for its medicinal and aromatic properties.  Patchouli is grown for its essential oil.  Patchouli essential oil is mainly obtained by steam distillation of the shade dried leaves.  It is widely appreciated for its characteristic pleasant and long lasting woody, earthy, camphoraceous odour.  It is especially notable as the essential oil extracted is internationally important and valuable, principally for the aromatherapy, perfumery, cosmetics, incense stick production and food flavouring industries.  This review attempted to give an overview of the relationship between aromatherapy and essential oils, importance of patchouli, harvesting pattern of patchouli, basics behind drying and steam distillation of patchouli crop, as well as trends existing in the various markets for essential oil application and its importance to mankind.   Keywords: patchouli, aromatherapy, drying, steam distillation, essential oil, application

Admissible Policy Teaching through Reward Design

We study reward design strategies for incentivizing a reinforcement learning agent to adopt a policy from a set of admissible policies. The goal of the reward designer is to modify the underlying reward function cost-efficiently while ensuring that any approximately optimal deterministic policy under the new reward function is admissible and performs well under the original reward function. This problem can be viewed as a dual to the problem of optimal reward poisoning attacks: instead of forcing an agent to adopt a specific policy, the reward designer incentivizes an agent to avoid taking actions that are inadmissible in certain states. Perhaps surprisingly, and in contrast to the problem of optimal reward poisoning attacks, we first show that the reward design problem for admissible policy teaching is computationally challenging, and it is NP-hard to find an approximately optimal reward modification. We then proceed by formulating a surrogate problem whose optimal solution approximates the optimal solution to the reward design problem in our setting, but is more amenable to optimization techniques and analysis. For this surrogate problem, we present characterization results that provide bounds on the value of the optimal solution. Finally, we design a local search algorithm to solve the surrogate problem and showcase its utility using simulation-based experiments

Comparison of the Effects of Zonisamide, Ethosuximide and Pregabalin in the Chronic Constriction Injury Induced Neuropathic Pain in Rats

Background: Evidence has been generated that various anticonvulsant agents provide relief of several chronic pain syndromes and therefore as an alternative to opioids, nonsteroidal anti.inflammatory, and tricyclic antidepressant drugs in the treatment of neuropathic pain. The results of these studies thus raise the question of whether all anticonvulsant drugs or particular mechanistic classes may be efficacious in the treatment of neuropathic pain syndromes.Aim: The aim was to compare the clinically used anticonvulsant drugs which are differ in their mechanism of action in a chronic pain model, the chronic constriction injury, in order to determine if all anticonvulsants or only particular mechanistic classes of anticonvulsants are analgesic.Materials and Methods: The study included zonisamide, ethosuximide and pregabalin. All compounds were anticonvulsant with diverse mechanism of actions. The peripheral neuropathic pain was induced by chronic constriction injury of the sciatic nerve in male Sprague.Dawley rats. Zonisamide (80 and 40 mg/kg), ethosuximide (300 and 100 mg/kg), pregabalin (50 and 20 mg/kg), and saline was administered intraperitoneally in respective groups in a blinded, randomized manner from postoperative day (POD) 7.13. Paw withdrawal duration to spontaneous pain, chemical allodynia and  mechanical hyperalgesia and paw withdrawal latency to mechanical  allodynia and thermal hyperalgesia were tested before drug administration on POD7 and after administration on POD 7, 9, 11 and 13.Results: The present study suggests that these drugs could provide an effective alternative in the treatment of neuropathic pain. However, zonisamide and pregabalin appears to have suitable efficacy to treat a wide spectrum of neuropathic pain condition. Conclusion: The present findings suggest that the inhibition of N.type calcium channels or voltage.gated sodium and T.type calcium channels provides better analgesic potential instead of inhibition of T.type calcium channels alone.Keywords: Chronic constriction injury model, Ethosuximide, Neuropathic pain, Pregabalin, Zonisamid