Avoiding Wireheading with Value Reinforcement Learning

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.Comment: Artificial General Intelligence (AGI) 201

Depth perception not found in human observers for static or dynamic anti-correlated random dot stereograms

One of the greatest challenges in visual neuroscience is that of linking neural activity with perceptual experience. In the case of binocular depth perception, important insights have been achieved through comparing neural responses and the perception of depth, for carefully selected stimuli. One of the most important types of stimulus that has been used here is the anti-correlated random dot stereogram (ACRDS). In these stimuli, the contrast polarity of one half of a stereoscopic image is reversed. While neurons in cortical area V1 respond reliably to the binocular disparities in ACRDS, they do not create a sensation of depth. This discrepancy has been used to argue that depth perception must rely on neural activity elsewhere in the brain. Currently, the psychophysical results on which this argument rests are not clear-cut. While it is generally assumed that ACRDS do not support the perception of depth, some studies have reported that some people, some of the time, perceive depth in some types of these stimuli. Given the importance of these results for understanding the neural correlates of stereopsis, we studied depth perception in ACRDS using a large number of observers, in order to provide an unambiguous conclusion about the extent to which these stimuli support the perception of depth. We presented observers with random dot stereograms in which correlated dots were presented in a surrounding annulus and correlated or anti-correlated dots were presented in a central circular region. While observers could reliably report the depth of the central region for correlated stimuli, we found no evidence for depth perception in static or dynamic anti-correlated stimuli. Confidence ratings for stereoscopic perception were uniformly low for anti-correlated stimuli, but showed normal variation with disparity for correlated stimuli. These results establish that the inability of observers to perceive depth in ACRDS is a robust phenomenon

Mechanisms for similarity matching in disparity measurement

Early neural mechanisms for the measurement of binocular disparity appear to operate in a manner consistent with cross-correlation-like processes. Consequently, cross-correlation, or cross-correlation-like procedures have been used in a range of models of disparity measurement. Using such procedures as the basis for disparity measurement creates a preference for correspondence solutions that maximize the similarity between local left and right eye image regions. Here, we examine how observers' perception of depth in an ambiguous stereogram is affected by manipulations of luminance and orientation-based image similarity. Results show a strong effect of coarse-scale luminance similarity manipulations, but a relatively weak effect of finer-scale manipulations of orientation similarity. This is in contrast to the measurements of depth obtained from a standard cross-correlation model. This model shows strong effects of orientation similarity manipulations and weaker effects of luminance similarity. In order to account for these discrepancies, the standard cross-correlation approach may be modified to include an initial spatial frequency filtering stage. The performance of this adjusted model most closely matches human psychophysical data when spatial frequency filtering favors coarser scales. This is consistent with the operation of disparity measurement processes where spatial frequency and disparity tuning are correlated, or where disparity measurement operates in a coarse-to-fine manner. © 2014 Goutcher and Hibbard

Impairment of cyclopean surface processing by disparity-defined masking stimuli.

Binocular disparity signals allow for the estimation of three-dimensional shape, even in the absence of monocular depth cues. The perception of such disparity-defined form depends, however, on the linkage of multiple disparity measurements over space. Performance limitations in cyclopean tasks thus inform us about errors arising in disparity measurement and difficulties in the linkage of such measurements. We used a cyclopean orientation discrimination task to examine the perception of disparity-defined form. Participants were presented with random-dot sinusoidal modulations in depth and asked to report whether they were clockwise or counter-clockwise rotated. To assess the effect of different noise structures on measurement and linkage processes, task performance was measured in the presence of binocular, random-dot masks, structured as either antiphase depth sinusoids, or as random distributions of dots in depth. For a fixed number of surface dots, the ratio of mask-to-surface dots was varied to obtain thresholds for orientation discrimination. Antiphase masks were found to be more effective than random depth masks, requiring a lower mask-to-surface dot ratio to inhibit performance. For antiphase masks, performance improved with decreased cyclopean frequency, increased disparity amplitude, and/or an increase in the total number of stimulus dots. Although a cross-correlation model of disparity measurement could account for antiphase mask performance, random depth masking effects were consistent with limitations in relative disparity processing. This suggests that performance is noise-limited for antiphase masks and complexity-limited for random masks. We propose that use of differing mask types may prove effective in understanding these distinct forms of impairment

Evaluation of the accuracy of the Leap Motion controller for measurements of grip aperture

The Leap Motion controller allows for a mouse-free alternative to general computing. With 200 frames/second infrared cameras, a 150 field of view and an 8 ft2 umbrella of interactive space, the Leap Motion has many potential practical applications. The device is advertised as aiming to be placed in new cars, laptops and hospitals, for example, to provide contact-free device control, while reducing the need for attentive button pressing and averting eye focus

The effect of image position on the Independent Components of natural binocular images

Human visual performance degrades substantially as the angular distance from the fovea increases. This decrease in performance is found for both binocular and monocular vision. Although analysis of the statistics of natural images has provided significant insights into human visual processing, little research has focused on the statistical content of binocular images at eccentric angles. We applied Independent Component Analysis to rectangular image patches cut from locations within binocular images corresponding to different degrees of eccentricity. The distribution of components learned from the varying locations was examined to determine how these distributions varied across eccentricity. We found a general trend towards a broader spread of horizontal and vertical position disparity tunings in eccentric regions compared to the fovea, with the horizontal spread more pronounced than the vertical spread. Eccentric locations above the centroid show a strong bias towards far-tuned components, eccentric locations below the centroid show a strong bias towards near-tuned components. These distributions exhibit substantial similarities with physiological measurements in V1, however in common with previous research we also observe important differences, in particular distributions of binocular phase disparity which do not match physiologypublishersversionPeer reviewe

Ideal binocular disparity detectors learned using independent subspace analysis on binocular natural image pairs

This work was funded by the Biotechnology and Biological Sciences Research Council (BBSRC) grant [BB/K018973/1].An influential theory of mammalian vision, known as the efficient coding hypothesis, holds that early stages in the visual cortex attempts to form an efficient coding of ecologically valid stimuli. Although numerous authors have successfully modelled some aspects of early vision mathematically, closer inspection has found substantial discrepancies between the predictions of some of these models and observations of neurons in the visual cortex. In particular analysis of linear-non-linear models of simple-cells using Independent Component Analysis has found a strong bias towards features on the horoptor. In order to investigate the link between the information content of binocular images, mathematical models of complex cells and physiological recordings, we applied Independent Subspace Analysis to binocular image patches in order to learn a set of complex-cell-like models. We found that these complex-cell-like models exhibited a wide range of binocular disparity-discriminability, although only a minority exhibited high binocular discrimination scores. However, in common with the linear-non-linear model case we found that feature detection was limited to the horoptor suggesting that current mathematical models are limited in their ability to explain the functionality of the visual cortex.Publisher PDFPeer reviewe

AGI and the Knight-Darwin Law: why idealized AGI reproduction requires collaboration

Can an AGI create a more intelligent AGI? Under idealized assumptions, for a certain theoretical type of intelligence, our answer is: “Not without outside help”. This is a paper on the mathematical structure of AGI populations when parent AGIs create child AGIs. We argue that such populations satisfy a certain biological law. Motivated by observations of sexual reproduction in seemingly-asexual species, the Knight-Darwin Law states that it is impossible for one organism to asexually produce another, which asexually produces another, and so on forever: that any sequence of organisms (each one a child of the previous) must contain occasional multi-parent organisms, or must terminate. By proving that a certain measure (arguably an intelligence measure) decreases when an idealized parent AGI single-handedly creates a child AGI, we argue that a similar Law holds for AGIs

Repellent Effects of Methyl Anthranilate on Western Corn Rootworm Larvae (Coleoptera: Chrysomelidae) in Soil Bioassays

Methyl anthranilate (MA), a compound in maize roots that is repellent to western corn rootworm larvae (Diabrotica virgifera virgifera LeConte) was tested in behavioral bioassays in a soil environment. MA prevented larvae from locating roots of a maize seedling, and the repellency strengthened with increasing rates of MA. In a simple push– pull strategy between an MA-treated seedling and an untreated seedling, granules containing 0.1 mg/g MA pushed larvae to the untreated seedling. This push effect increased with dose, with 90% repellency observed for the highest dose tested (100 mg/g). Chemical analysis showed that MA concentrations remained high for 4 wk in dry, sterilized or unsterilized soil, but declined rapidly in moist soil. After 7 d, 50% less MA was recovered in moist, sterilized soil than in dry soil, and only a trace of MA remained in unsterilized moist soil, suggesting that both moisture and microbial activity contributed to the loss of MA. Various (MA) carrier granules were tested in bioassays after aging in moist soil. After 1 d, all of the MA granules were repellent at the 10 mg/g rate and clay granules were also effective at 1 mg/g. After 1 wk, only molecular sieve granules elicited repellency, but that activity disappeared after 2 wk. These results demonstrate that MA is repellent to western corn rootworm larvae in the soil environment and may have potential as a rootworm treatment if formulations can be developed that protect the material from decomposition in the soil