Avoiding Wireheading with Value Reinforcement Learning

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.Comment: Artificial General Intelligence (AGI) 201

The Neutral Hydrogen Distribution in Merging Galaxies: Differences between Stellar and Gaseous Tidal Morphologies

We have mapped the neutral atomic gas (HI) in the three disk-disk merger systems NGC 520, Arp 220, and Arp 299. These systems differ from the majority of the mergers mapped in HI, in that their stellar and gaseous tidal features do not coincide. In particular, they exhibit large stellar tidal features with little if any accompanying neutral gas and large gas-rich tidal features with little if any accompanying starlight. On smaller scales, there are striking anti-correlations where the gaseous and stellar tidal features appear to cross. We explore several possible causes for these differences, including dust obscuration, ram pressure stripping, and ionization effects. No single explanation can account for all of the observed differences. The fact that each of these systems shows evidence for a starburst driven superwind expanding in the direction of the most striking anti-correlations leads us to suggest that the superwind is primarily responsible for the observed differences, either by sweeping the features clear of gas via ram pressure, or by excavating a clear sightline towards the starburst and allowing UV photons to ionize regions of the tails.Comment: 16 pages, 5 figures, uses emulateapj.sty. To appear in the March 2000 issue of AJ. Version with full resolution figures is available via http://www.cv.nrao.edu/~jhibbard/HIdisp/HIdisp.htm

Computer mediated communication: Elements impacting sustained usage by Iowa elementary and middle school teachers

The purpose of the study was to examine, using a case study methodology, interactions among a group of Iowa teachers involved in a computer mediated communication project, to better understand those elements which may impact sustained usage of computer mediated communication. The study involved examination of one year\u27s worth of on-line journal postings and interactive conferences, and interviews with the 22 teachers involved in the project. The multiple sources of data collected were analyzed using a methodology known as constant comparative, and the analysis presented according to the following themes: What topics did the participants discuss while using computer mediated communication? Were there differences in language used by the teachers in the weekly journals versus the on-line conferences? What were the reflective responses of these teachers when questioned about their involvement with computer mediated communication? The study found that topics of discussion evolving from the teacher\u27s use of computer mediated communication fell into four areas: social, school related, class related, and personal. The study also found a significant difference in the language used by the teachers in their weekly journals versus the on-line conferences. Analysis of the reflective responses to the interview questions revealed behavioral or attitudinal patterns in the areas of home use, frequency of use, sharing of experiences, relevance to teaching, viability for inservice activities, time commitment, and support. Examination of the above findings revealed differences impacting sustained usage in the areas of gender, the teachers\u27 perception of the innovative aspects of the new technology, and their attitude toward time commitment. Recommendations were made in the areas of (a) time for integration of new technologies, (b) working collaboratively, (c) technical support, (d) school district commitment, (e) funding, and (f) additional research

Wiretapping the Internet: The Expansion of the Communications Assistance to Law Enforcement Act to Extend Government Surveillance

Criminal use of the Internet to circumvent traditional government phone wiretaps has inspired the Obama Administration to create a proposal to expand the Communications Assistance to Law Enforcement Act ( CALEA ). CALEA was passed in 1994 to regulate telephone and broadband companies to ensure compliance with standards to enable government wiretapping. The proposed amendment of CALEA would allow the government to require all communications service providers to meet technical standards necessary to comply with a wiretap order. The expansion of CALEA would likely widen its scope to social networking sites, instant messaging, gaming consoles that allow conversation among multiple players, and to word processing software that allows communication through Internet access. The unique architecture of the Internet lends it to particular vulnerabilities with the consequence that an expansion of CALEA to all Internet communications could create problems regarding the innovative nature of the Internet, national security, free speech, and privacy. This Note will examine the competing interests related to expanding CALEA and will weigh the potential benefits and consequences of CALEA. The Note concludes that substantially more information is needed to justify a change of the law

Wiretapping the Internet: The Expansion of the Communications Assistance to Law Enforcement Act to Extend Government Surveillance

Criminal use of the Internet to circumvent traditional government phone wiretaps has inspired the Obama Administration to create a proposal to expand the Communications Assistance to Law Enforcement Act ( CALEA ). CALEA was passed in 1994 to regulate telephone and broadband companies to ensure compliance with standards to enable government wiretapping. The proposed amendment of CALEA would allow the government to require all communications service providers to meet technical standards necessary to comply with a wiretap order. The expansion of CALEA would likely widen its scope to social networking sites, instant messaging, gaming consoles that allow conversation among multiple players, and to word processing software that allows communication through Internet access. The unique architecture of the Internet lends it to particular vulnerabilities with the consequence that an expansion of CALEA to all Internet communications could create problems regarding the innovative nature of the Internet, national security, free speech, and privacy. This Note will examine the competing interests related to expanding CALEA and will weigh the potential benefits and consequences of CALEA. The Note concludes that substantially more information is needed to justify a change of the law

The Economic Impacts of the Regional Greenhouse Gas Initiative on Ten Northeast and Mid-Atlantic States

Assesses outcomes of the first U.S. market-based program to reduce emissions of carbon dioxide from power plants, including impact on electricity markets, power companies' costs, and consumer prices; use of auction proceeds; and states' economic benefits

NGC 3310, a galaxy merger?

The HI structure and kinematics of the peculiar starburst galaxy NGC 3310 (Arp 217, UGC 5786) are discussed. New evidence bearing on the origin of the starburst is presented. The bulk of HI coincides with the bright optical disk and shows differential rotation. Its velocity dispersion is, however, unusually large for a spiral galaxy (up to 40 km/s), suggesting that the disk is highly perturbed as already indicated by optical emission line spectroscopy. There are, in addition, two prominent HI tails, one extending to the north-west and the other, somewhat patchy, to the south. These HI tails, the perturbed kinematics and the peculiar optical morphology strongly suggest a recent merger between two gas-rich galaxies. This seems to have been a major merger in which most of the gas in the inner parts has been preserved in neutral atomic form and either one of the progenitor disks has survived or a new disk has formed.Comment: 10 pages, 11 figures, accepted for publication in A&

The HI content of extremely metal-deficient blue compact dwarf galaxies

We have obtained new HI observations with the 100m Green Bank Telescope (GBT) for a sample of 29 extremely metal-deficient star-forming Blue Compact Dwarf (BCD) galaxies, selected from the Sloan Digital Sky Survey spectral data base to be extremely metal-deficient (12+logO/H<7.6). Neutral hydrogen was detected in 28 galaxies, a 97% detection rate. Combining the HI data with SDSS optical spectra for the BCD sample and adding complementary galaxy samples from the literature to extend the metallicity and mass ranges, we have studied how the HI content of a galaxy varies with various global galaxian properties. There is a clear trend of increasing gas mass fraction with decreasing metallicity, mass and luminosity. We obtain the relation M(HI)/L(g)~L(g)^{-0.3}, in agreement with previous studies based on samples with a smaller luminosity range. The median gas mass fraction f(gas) for the GBT sample is equal to 0.94 while the mean gas mass fraction is 0.90+/-0.15, with a lower limit of ~0.65. The HI depletion time is independent of metallicity, with a large scatter around the median value of 3.4 Gyr. The ratio of the baryonic mass to the dynamical mass of the metal-deficient BCDs varies from 0.05 to 0.80, with a median value of ~0.2. About 65% of the BCDs in our sample have an effective yield larger than the true yield, implying that the neutral gas envelope in BCDs is more metal-deficient by a factor of 1.5-20, as compared to the ionized gas.Comment: 21 pages, 13 figures, accepted for publication in MNRA

Bipartite entangled stabilizer mutually unbiased bases as maximum cliques of Cayley graphs

We examine the existence and structure of particular sets of mutually unbiased bases (MUBs) in bipartite qudit systems. In contrast to well-known power-of-prime MUB constructions, we restrict ourselves to using maximally entangled stabilizer states as MUB vectors. Consequently, these bipartite entangled stabilizer MUBs (BES MUBs) provide no local information, but are sufficient and minimal for decomposing a wide variety of interesting operators including (mixtures of) Jamiolkowski states, entanglement witnesses and more. The problem of finding such BES MUBs can be mapped, in a natural way, to that of finding maximum cliques in a family of Cayley graphs. Some relationships with known power-of-prime MUB constructions are discussed, and observables for BES MUBs are given explicitly in terms of Pauli operators.Comment: 8 pages, 1 figur