Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?

Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during training if most of the training data contain such spurious correlations. This phenomenon is particularly pronounced in domains such as robotics, with potentially large gaps between the open- and closed-loop performance of an agent. In such settings, causally confused models may appear to perform well according to open-loop metrics during training but fail catastrophically when deployed in the real world. In this paper, we study causal confusion in offline reinforcement learning. We investigate whether selectively sampling appropriate points from a dataset of demonstrations may enable offline reinforcement learning agents to disambiguate the underlying causal mechanisms of the environment, alleviate causal confusion in offline reinforcement learning, and produce a safer model for deployment. To answer this question, we consider a set of tailored offline reinforcement learning datasets that exhibit causal ambiguity and assess the ability of active sampling techniques to reduce causal confusion at evaluation. We provide empirical evidence that uniform and active sampling techniques are able to consistently reduce causal confusion as training progresses and that active sampling is able to do so significantly more efficiently than uniform sampling.Comment: Published in Proceedings of the 2nd Conference on Causal Learning and Reasoning (CLeaR 2021

Outcome-Driven Reinforcement Learning via Variational Inference

While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.Comment: Published in Advances in Neural Information Processing Systems 34 (NeurIPS 2021

Amorphous calcium carbonate as a novel potential treatment for osteoarthritis in dogs: a pilot clinical study

BackgroundAmorphous calcium carbonate (ACC) is a potential new treatment for canine osteoarthritis (OA) with novel mechanisms based on local pH modulation and targeting bone remodeling, inflammation, and pain. The aim of this pilot exploratory clinical study was to obtain initial data on the potential efficacy and safety of ACC in OA dogs and to determine if further investigation was appropriate using similar assessment methods.Materials and methodsIn this prospective, randomized, double-blind, controlled pilot study, 41 client-owned dogs were allocated in a 2:1 ratio to ACC: placebo given orally for 56 days. Efficacy assessments included improvements in pain and mobility using owner questionnaires [Canine Brief Pain Inventory (CBPI), Client Specific Outcome Measure (CSOM), and Veterinary Orthopedic Scores (VOS)]. Safety in the study population was monitored by veterinary examinations, clinical pathology, and adverse events.ResultsFifty-three dogs were screened, of which 41 enrolled and served for the safety assessment. Thirty-six dogs were found evaluable for initial efficacy assessment. Three dogs given placebo (21.4%) and one given ACC (4.5%) were removed before day 56 due to owner-perceived pain and were considered treatment failures. There were no serious adverse events or clinically significant treatment-related effects in the study. Overall, ACC was found safe in the small study population. On day 56, proportionally more ACC than placebo dogs were treatment successes based on CBPI (45.5% vs. 21.4%) and CSOM (63.6% vs. 30.8%, respectively); however, these differences were not statistically significant (p = 0.15 and 0.06, respectively). On day 56, within the ACC group but not the placebo group, the CBPI, CSOM, and VOS assessments were lower compared to day 0 and day 14 (p < 0.05).LimitationsThe relatively small number of dogs limited the statistical power of the pilot study in evaluating the efficacy and safety of ACC.ConclusionStudy results support the conduct of larger, appropriately powered studies using similar assessments to confirm whether ACC may be a safe and effective treatment for OA in dogs

Unveiling Dust-enshrouded Star Formation in the Early Universe: a Sub-mm Survey of the Hubble Deep Field

The advent of sensitive sub-mm array cameras now allows a proper census of dust-enshrouded massive star-formation in very distant galaxies, previously hidden activity to which even the faintest optical images are insensitive. We present the deepest sub-mm survey of the sky to date, taken with the SCUBA camera on the James Clerk Maxwell Telescope and centred on the Hubble Deep Field. The high source density found in this image implies that the survey is confusion-limited below a flux density of 2 mJy. However, within the central 80 arcsec radius independent analyses yield 5 reproducible sources with S(850um) > 2 mJy which simulations indicate can be ascribed to individual galaxies. We give positions and flux densities for these, and furthermore show using multi-frequency photometric data that the brightest sources in our map lie at redshifts z~3. These results lead to integral source counts which are completely inconsistent with a no-evolution model, and imply that massive star-formation activity continues at redshifts > 2. The combined brightness of the 5 most secure sources in our map is sufficient to account for 30 - 50% of the previously unresolved sub-mm background, and we estimate statistically that the entire background is resolved at about the 0.3 mJy level. Finally we discuss possible optical identifications and redshift estimates for the brightest sources. One source appears to be associated with an extreme starburst galaxy at z~1, whilst the remaining four appear to lie in the redshift range 2 < z < 4. This implies a star-formation density over this redshift range that is at least five times higher than that inferred from the ultraviolet output of HDF galaxies.Comment: 19 pages, 6 figures (to appear as a Nature Article

Increased upconversion performance for thin film solar cells: A trimolecular composition

Photochemical upconversion based on triplet-triplet annihilation (TTA-UC) is employed to enhance the short-circuit currents generated by two varieties of thin-film solar cells, a hydrogenated amorphous silicon (a-Si:H) solar cell and a dye-sensitized solar cell (DSC). TTA-UC is exploited to harvest transmitted sub-bandgap photons, combine their energies and re-radiate upconverted photons back towards the solar cells. In the present study we employ a dual-emitter TTA-UC system which allows for significantly improved UC quantum yields as compared to the previously used single-emitter TTA systems. In doing so we achieve record photo-current enhancement values for both the a-Si:H device and the DSC, surpassing 10-3 mA cm-2 sun-2 for the first time for a TTA-UC system and marking a record for upconversion-enhanced solar cells in general. We discuss pertinent challenges of the TTA-UC technology which need to be addressed in order to achieve its viable device application

a trimolecular composition

Photochemical upconversion based on triplet–triplet annihilation (TTA-UC) is employed to enhance the short-circuit currents generated by two varieties of thin-film solar cells, a hydrogenated amorphous silicon (a-Si:H) solar cell and a dye-sensitized solar cell (DSC). TTA-UC is exploited to harvest transmitted sub-bandgap photons, combine their energies and re-radiate upconverted photons back towards the solar cells. In the present study we employ a dual-emitter TTA-UC system which allows for significantly improved UC quantum yields as compared to the previously used single-emitter TTA systems. In doing so we achieve record photo-current enhancement values for both the a-Si:H device and the DSC, surpassing 10−3 mA cm−2 sun−2 for the first time for a TTA-UC system and marking a record for upconversion-enhanced solar cells in general. We discuss pertinent challenges of the TTA-UC technology which need to be addressed in order to achieve its viable device application

Improving the light-harvesting of amorphous silicon solar cells with photochemical upconversion

Single-threshold solar cells are fundamentally limited by their ability to harvest only those photons above a certain energy. Harvesting below-threshold photons and re-radiating this energy at a shorter wavelength would thus boost the efficiency of such devices. We report an increase in light harvesting efficiency of a hydrogenated amorphous silicon (a-Si:H) thin-film solar cell due to a rear upconvertor based on sensitized triplet–triplet-annihilation in organic molecules. Low energy light in the range 600–750 nm is converted to 550–600 nm light due to the incoherent photochemical process. A peak efficiency enhancement of (1.0 ± 0.2)% at 720 nm is measured under irradiation equivalent to (48 ± 3) suns (AM1.5). We discuss the pathways to be explored in adapting photochemical UC for application in various single threshold devices

IR and UV Galaxies at z=0.6 -- Evolution of Dust Attenuation and Stellar Mass as Revealed by SWIRE and GALEX

We study dust attenuation and stellar mass of $\rm z\sim 0.6$ star-forming galaxies using new SWIRE observations in IR and GALEX observations in UV. Two samples are selected from the SWIRE and GALEX source catalogs in the SWIRE/GALEX field ELAIS-N1-00 ($\Omega = 0.8$ deg$^2$). The UV selected sample has 600 galaxies with photometric redshift (hereafter photo-z) $0.5 \leq z \leq 0.7$ and NUV$\leq 23.5$ (corresponding to \rm L_{FUV} \geq 10^{9.6} L_\sun). The IR selected sample contains 430 galaxies with $f_{24\mu m} \geq 0.2$ mJy (\rm L_{dust} \geq 10^{10.8} L_\sun) in the same photo-z range. It is found that the mean $\rm L_{dust}/L_{FUV}$ ratios of the z=0.6 UV galaxies are consistent with that of their z=0 counterparts of the same $\rm L_{FUV}$. For IR galaxies, the mean $\rm L_{dust}/L_{FUV}$ ratios of the z=0.6 LIRGs (\rm L_{dust} \sim 10^{11} L_\sun) are about a factor of 2 lower than local LIRGs, whereas z=0.6 ULIRGs (\rm L_{dust} \sim 10^{12} L_\sun) have the same mean $\rm L_{dust}/L_{FUV}$ ratios as their local counterparts. This is consistent with the hypothesis that the dominant component of LIRG population has changed from large, gas rich spirals at z$>0.5$ to major-mergers at z=0. The stellar mass of z=0.6 UV galaxies of \rm L_{FUV} \leq 10^{10.2} L_\sun is about a factor 2 less than their local counterparts of the same luminosity, indicating growth of these galaxies. The mass of z=0.6 UV lunmous galaxies (UVLGs: \rm L_{FUV} > 10^{10.2} L_\sun) and IR selected galaxies, which are nearly exclusively LIRGs and ULIRGs, is the same as their local counterparts.Comment: 27 pages, 8 figures, to be published in the Astrophysical Journal Supplement series dedicated to GALEX result

Improving the light-harvesting of second generation solar cells with photochemical upconversion

Photovoltaics (PV) offer a solution for the development of sustainable energy sources, relying on the sheer abundance of sunlight: More sunlight falls on the Earth’s surface in one hour than is required by its inhabitants in a year. However, it is imperative to manage the wide distribution of photon energies available in order to generate more cost efficient PV devices because single threshold PV devices are fundamentally limited to a maximum conversion efficiency, the Shockley-Queisser (SQ) limit. Recent progress has enabled the production of c-Si cells with efficiencies as high as 25%,1 close to the limiting efficiency of ∼30%. But these cells are rather expensive, and ultimately the cost of energy is determined by the ratio of system cost and efficiency of the PV device. A strategy to radically decrease this ratio is to circumvent the SQ limit in cheaper, second generation PV devices. One promising approach is the use of hydrogenated amorphous silicon (a-Si:H), where film thicknesses on the order of several 100nm are sufficient. Unfortunately, the optical threshold of a-Si:H is rather high (1.7-1.8 eV) and the material suffers from light-induced degradation. Thinner absorber layers in a-Si:H devices are generally more stable than thicker films due to the better charge carrier extraction, but at the expense of reduced conversion efficiencies, especially in the red part of the solar spectrum (absorption losses). Hence for higher bandgap materials, which includes a-Si as well as organic and dye-sensitized cells, the major loss mechanism is the inability to harvest low energy photons

Diminazene resistance in Trypanosoma congolense is not caused by reduced transport capacity but associated with reduced mitochondrial membrane potential

Trypanosoma congolense is a principal agent causing livestock trypanosomiasis in Africa, costing developing economies billions of dollars and undermining food security. Only the diamidine diminazene and the phenanthridine isometamidium are regularly used, and resistance is widespread but poorly understood. We induced stable diminazene resistance in T. congolense strain IL3000 in vitro. There was no cross‐resistance with the phenanthridine drugs, melaminophenyl arsenicals, oxaborole trypanocides, or with diamidine trypanocides, except the close analogues DB829 and DB75. Fluorescence microscopy showed that accumulation of DB75 was inhibited by folate. Uptake of [3H]‐diminazene was slow, low affinity and partly but reciprocally inhibited by folate and by competing diamidines. Expression of T. congolense folate transporters in diminazene‐resistant T. b. brucei significantly sensitized the cells to diminazene and DB829, but not to oxaborole AN7973. However, [3H]‐diminazene transport studies, whole genome sequencing and RNA‐seq found no major changes in diminazene uptake, folate transporter sequence or expression. Instead, all resistant clones displayed a moderate reduction in the mitochondrial membrane potential. We conclude that diminazene uptake in T. congolense proceed via multiple low affinity mechanisms including folate transporters; while resistance is associated with a reduction in Ψm it is unclear whether this is the primary cause of the resistance