Extreme State Aggregation Beyond MDPs

We consider a Reinforcement Learning setup where an agent interacts with an environment in observation-reward-action cycles without any (esp.\ MDP) assumptions on the environment. State aggregation and more generally feature reinforcement learning is concerned with mapping histories/raw-states to reduced/aggregated states. The idea behind both is that the resulting reduced process (approximately) forms a small stationary finite-state MDP, which can then be efficiently solved or learnt. We considerably generalize existing aggregation results by showing that even if the reduced process is not an MDP, the (q-)value functions and (optimal) policies of an associated MDP with same state-space size solve the original problem, as long as the solution can approximately be represented as a function of the reduced states. This implies an upper bound on the required state space size that holds uniformly for all RL problems. It may also explain why RL algorithms designed for MDPs sometimes perform well beyond MDPs.Comment: 28 LaTeX pages. 8 Theorem

Probabilities on Sentences in an Expressive Logic

Automated reasoning about uncertain knowledge has many applications. One difficulty when developing such systems is the lack of a completely satisfactory integration of logic and probability. We address this problem directly. Expressive languages like higher-order logic are ideally suited for representing and reasoning about structured knowledge. Uncertain knowledge can be modeled by using graded probabilities rather than binary truth-values. The main technical problem studied in this paper is the following: Given a set of sentences, each having some probability of being true, what probability should be ascribed to other (query) sentences? A natural wish-list, among others, is that the probability distribution (i) is consistent with the knowledge base, (ii) allows for a consistent inference procedure and in particular (iii) reduces to deductive logic in the limit of probabilities being 0 and 1, (iv) allows (Bayesian) inductive reasoning and (v) learning in the limit and in particular (vi) allows confirmation of universally quantified hypotheses/sentences. We translate this wish-list into technical requirements for a prior probability and show that probabilities satisfying all our criteria exist. We also give explicit constructions and several general characterizations of probabilities that satisfy some or all of the criteria and various (counter) examples. We also derive necessary and sufficient conditions for extending beliefs about finitely many sentences to suitable probabilities over all sentences, and in particular least dogmatic or least biased ones. We conclude with a brief outlook on how the developed theory might be used and approximated in autonomous reasoning agents. Our theory is a step towards a globally consistent and empirically satisfactory unification of probability and logic.Comment: 52 LaTeX pages, 64 definiton/theorems/etc, presented at conference Progic 2011 in New Yor

Optimistic Agents are Asymptotically Optimal

We use optimism to introduce generic asymptotically optimal reinforcement learning agents. They achieve, with an arbitrary finite or compact class of environments, asymptotically optimal behavior. Furthermore, in the finite deterministic case we provide finite error bounds.Comment: 13 LaTeX page

Visualization of leukocyte transendothelial and interstitial migration using reflected light oblique transillumination in intravital video microscopy

Dynamic visualization of the intravascular events leading to the extravasation of leukocytes into tissues by intravital microscopy has significantly expanded our understanding of the underlying molecular processes. In contrast, the detailed observation of leukocyte transendothelial and interstitial migration in vivo has been hampered by the poor image contrast of cells within turbid media that is obtainable by conventional brightfield microscopy. Here we present a microscopic method, termed reflected light oblique transillumination microscopy, that makes use of the optical interference phenomena generated by oblique transillumination to visualize subtle gradients of refractive indices within tissues for enhanced image contrast. Using the mouse cremaster muscle, we demonstrate that this technique makes possible the reliable quantification of extravasated leukocytes as well as the characterization of morphological phenomena of leukocyte transendothelial and interstitial migration

Optimising poly(lactic-co-glycolic acid) microparticle fabrication using a Taguchi orthogonal array design-of-experiment approach

© 2019 Mensah et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.The objective of this study was to identify, understand and generate a Taguchi orthogonal array model for the formation of 10–50 μm microparticles with applications in topical/ocular controlled drug delivery. Poly(lactic-co-glycolic acid) (PLGA) microparticles were fabricated by the single emulsion oil-in-water method and the particle size was characterized using laser diffraction and scanning electronic microscopy (SEM). Sequential Taguchi L 12 and L 18 orthogonal array (OA) designs were employed to study the influence of ten and eight parameters, respectively, on microparticle size (response). The first optimization step using the L 12 design showed that all parameters significantly influenced the particle size of the prepared PLGA microparticles with exception of the concentration of poly(vinyl alcohol) (PVA) in the hardening bath. The smallest mean particle size obtained from the L 12 design was 54.39 μm. A subsequent L 18 design showed that the molecular weight of PLGA does not significantly affect the particle size. An experimental run comprising of defined parameters including molecular weight of PLGA (89 kDa), concentration of PLGA (20% w/v), concentration of PVA in the emulsion (0.8% w/v), solvent type (ethyl acetate), organic/aqeuous phase ratio (1:1 v/v), vortexing speed (9), vortexing duration (60 seconds), concentration of PVA in hardening bath (0.8% w/v), stirring speed of hardening bath (1200 rpm) and solvent evaporation duration (24 hours) resulted in the lowest mean particle size of 23.51 μm which was predicted and confirmed by the L 18 array. A comparable size was demonstrated during the fabrication of BSA-incorporated microparticles. Taguchi OA design proved to be a valuable tool in determining the combination of process parameters that can provide the optimal condition for microparticle formulation. Taguchi OA design can be used to correctly predict the size of microparticles fabricated by the single emulsion process and can therefore, ultimately, save time and costs during the manufacturing process of drug delivery formulations by minimising experimental runs.Peer reviewedFinal Published versio