Denumerable-Armed Bandits

This paper studies the class of denumerable-armed (i.e. finite- or countably infinitearmed) bandit problems with independent arms and geometric discounting over an infinite horizon, in which each arm generates rewards according to one of a finite number of distributions, or "types." The number of types in the support of an arm, as also the types themselves, are allowed to vary across the arms. We derive certain continuity and curvature properties of the dynamic allocation (or Gittins) index of Gittins and Jones (1974), and provide necessary and sufficient conditions under which the Gittins-Jones result identifying all optimal strategies for finite-armed bandits may be extended to infinite-armed bandits. We then establish our central result: at each point in time, the arm selected by an optimal strategy will, with strictly positive probability, remain an optimal selection forever. More specifically, for every such arm, there exists (at least) one type of that arm such that, when conditioned on that type being the arm's "true" type, the arm will survive forever and continuously with nonzero probability. When the reward distributions of an arm satisfy the monotone likelihood ratio property (MLRP), the survival prospects of an arm improve when conditioned on types generating higher expected rewards; however, we show how this need not be the case in the absence of MLRP. Implications of these results are derived for the theories of job search and matching, as well as other applications of the bandit paradigm

Switching Costs and the Gittins Index

The Theorem of Gittins and Jones (1974) is, perhaps, the single most powerful result in the literature on Bandit problems. This result establishes that in independent-armed Bandit problems with geometric discounting over an infinite horizon, all optimal strategies may be obtained by solving a family of simple optimal stopping problems that associate with each arm an index known as the dynamic allocation index or, more popularly, as the Gittins index. Importantly, the Gittins index of an arm depends solely on the characteristics of that arm and the rate of discounting, and is otherwise completely independent of the problem under consideration. These features simplify significantly the task of characterizing optimal strategies in this class of problems

A Class of Bandit Problems Yielding Myopic Optimal Strategies

We consider the class of bandit problems in which each of the n ≧ 2 independent arms generates rewards according to one of the same two reward distributions, and discounting is geometric over an infinite horizon. We show that the dynamic allocation index of Gittins and Jones (1974) in this context is strictly increasing in the probability that an arm is the better of the two distributions. It follows as an immediate consequence that myopic strategies are the uniquely optimal strategies in this class of bandit problems, regardless of the value of the discount parameter or the shape of the reward distributions. Some implications of this result for bandits with Bernoulli reward distributions are given

Generalization of Linearized Gouy-Chapman-Stern Model of Electric Double Layer for Nanostructured and Porous Electrodes: Deterministic and Stochastic Morphology

We generalize linearized Gouy-Chapman-Stern theory of electric double layer for nanostructured and morphologically disordered electrodes. Equation for capacitance is obtained using linear Gouy-Chapman (GC) or Debye-$\rm{\ddot{u}}$ckel equation for potential near complex electrode/electrolyte interface. The effect of surface morphology of an electrode on electric double layer (EDL) is obtained using "multiple scattering formalism" in surface curvature. The result for capacitance is expressed in terms of the ratio of Gouy screening length and the local principal radii of curvature of surface. We also include a contribution of compact layer, which is significant in overall prediction of capacitance. Our general results are analyzed in details for two special morphologies of electrodes, i.e. "nanoporous membrane" and "forest of nanopillars". Variations of local shapes and global size variations due to residual randomness in morphology are accounted as curvature fluctuations over a reference shape element. Particularly, the theory shows that the presence of geometrical fluctuations in porous systems causes enhanced dependence of capacitance on mean pore sizes and suppresses the magnitude of capacitance. Theory emphasizes a strong influence of overall morphology and its disorder on capacitance. Finally, our predictions are in reasonable agreement with recent experimental measurements on supercapacitive mesoporous systems

Perturbation expansions and series acceleration procedures: Part-II. Extrapolation techniques

Three new procedures for the extrapolation of series coefficients from a given power series expansion are proposed. They are based on (i) a novel resummation identity, (ii) parametrised Euler transformation (pet) and (iii) a modifiedpet. Several examples taken from the Ising model series expansions, ferrimagnetic systems, etc., are illustrated. Apart from these applications, the higher order virial coefficients for hard spheres and hard discs have also been evaluated using the new techniques and these are compared with the estimates obtained by other methods. A satisfactory agreement is revealed between the two

Switching Costs and the Gittins Index

The Theorem of Gittins and Jones (1974) is, perhaps, the single most powerful result in the literature on Bandit problems. This result establishes that in independent-armed Bandit problems with geometric discounting over an infinite horizon, all optimal strategies may be obtained by solving a family of simple optimal stopping problems that associate with each arm an index known as the dynamic allocation index or, more popularly, as the Gittins index. Importantly, the Gittins index of an arm depends solely on the characteristics of that arm and the rate of discounting, and is otherwise completely independent of the problem under consideration. These features simplify significantly the task of characterizing optimal strategies in this class of problems

Optimal Retention in Principal/Agent Models

This paper studies the interaction between a single long-lived principal and a series of short-lived agents in the presence of both moral hazard and adverse selection. We assume that the principal can influence the agents' behavior only through her choice of a retention rule; this rule is further required to be sequentially rational (i.e., no precommitment is allowed). We provide general conditions under which equilibria exist in which (a) the principal adopts a 'cut-off' rule under which agents are retained only when the reward they generate exceeds a critical bound; and (b) agent separate according to type, with better agents taking superior actions. We show that in equilibrium, a retained agent's productivity is necessarily declining over time, but that retained agents are also more productive on average than untried agents due to selection effects. Finally, we show that for each given type, agents of that type are more productive in the presence of adverse selection than when there is pure moral hazard (i.e., when that type is the sole type of agent in the model); nonetheless, adding uncertainty about agent-types cannot benefit the principal except in uninteresting cases

Denumerable-Armed Bandits

This paper studies the class of denumerable-armed (i.e. finite- or countably infinitearmed) bandit problems with independent arms and geometric discounting over an infinite horizon, in which each arm generates rewards according to one of a finite number of distributions, or "types." The number of types in the support of an arm, as also the types themselves, are allowed to vary across the arms. We derive certain continuity and curvature properties of the dynamic allocation (or Gittins) index of Gittins and Jones (1974), and provide necessary and sufficient conditions under which the Gittins-Jones result identifying all optimal strategies for finite-armed bandits may be extended to infinite-armed bandits. We then establish our central result: at each point in time, the arm selected by an optimal strategy will, with strictly positive probability, remain an optimal selection forever. More specifically, for every such arm, there exists (at least) one type of that arm such that, when conditioned on that type being the arm's "true" type, the arm will survive forever and continuously with nonzero probability. When the reward distributions of an arm satisfy the monotone likelihood ratio property (MLRP), the survival prospects of an arm improve when conditioned on types generating higher expected rewards; however, we show how this need not be the case in the absence of MLRP. Implications of these results are derived for the theories of job search and matching, as well as other applications of the bandit paradigm

A Class of Bandit Problems Yielding Myopic Optimal Strategies

We consider the class of bandit problems in which each of the n ≧ 2 independent arms generates rewards according to one of the same two reward distributions, and discounting is geometric over an infinite horizon. We show that the dynamic allocation index of Gittins and Jones (1974) in this context is strictly increasing in the probability that an arm is the better of the two distributions. It follows as an immediate consequence that myopic strategies are the uniquely optimal strategies in this class of bandit problems, regardless of the value of the discount parameter or the shape of the reward distributions. Some implications of this result for bandits with Bernoulli reward distributions are given