Hülse, Martin

Lee, Mark

McBride, Sebastian

English

Based on the biological phenomenon of inhibition of return, we introduce an architecture developed for an active robotic vision system where continually updated global information is used to modulate the action selection process for saccadic camera movements. This facilitates, in an extremely efficient way, the fundamental process of avoiding re-saccading to objects previously visited and, thus, is considered to have a wide-ranging application within active vision systems

Aberystwyth Research Portal

Aberystwyth UniversityImplementing inhibition of return: embodied visual memory for robotic systemsHülse, Martin; McBride, Sebastian; Lee, MarkPublished in:Proc. 9th Int. Conf. on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund Uni.Cognitive StudiesPublication date:2009Citation for published version (APA):Hülse, M., McBride, S., & Lee, M. (2009). Implementing inhibition of return: embodied visual memory for roboticsystems. In L. Canamero, P. -Y. Oudeyer, & C. Balkenius (Eds.), Proc. 9th Int. Conf. on Epigenetic Robotics:Modeling Cognitive Development in Robotic Systems. Lund Uni. Cognitive Studies (pp. 213-214)General rightsCopyright and moral rights for the publications made accessible in the Aberystwyth Research Portal (the Institutional Repository) areretained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights.            • Users may download and print one copy of any publication from the Aberystwyth Research Portal for the purpose of private study orresearch.            • You may not further distribute the material or use it for any profit-making activity or commercial gain            • You may freely distribute the URL identifying the publication in the Aberystwyth Research PortalTake down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.tel: +44 1970 62 2400email: is@aber.ac.ukDownload date: 03. Oct. 2019Implementing inhibition of return; embodiedvisual memory for robotic systemsMartin Hu¨lse Sebastian McBride Mark LeeDept. of Computer Science, Aberystwyth University, SY23 3DB, Wales, UKAbstractBased on the biological phenomenon of inhibi-tion of return, we introduce an architecture de-veloped for an active robotic vision system wherecontinually updated global information is used tomodulate the action selection process for saccadiccamera movements. This facilitates, in an ex-tremely efcient way, the fundamental process ofavoiding re-saccading to objects previously vis-ited and, thus, is considered to have a wide-ranging application within active vision systems.Inhibition of return (IOR) refers to the suppressionof stimuli (objects and events) processing where thosestimuli have previously (and recently) been the focus ofspatial attention (Lupianez et al., 2006). In this sense, itforms the basis of attentional (and thus visual) bias to-wards novel objects. Although the neural mechanismunderpinning IOR is not completely understood, it iswell established that the dorsal frontoparietal network,including frontal eye elds (FEF) and superior parietalcortex are the primary structures mediating its control(Mayer et al., 2004). These are some of the many mod-ulatory and affecting structures of the deep superior col-liculus (optic tectum in non-mammals), the primary mo-tor structure controlling saccade. Although visual in-formation from the retina starts at the supercial su-perior colliculus, and there are direct connections be-tween the supercial and deep layers, the former can-not elicit saccade directly (Stein and Meredith, 1991).This information has to be subsequently processed by anumber of cortical and sub-cortical structures that placeit: 1) in context of attentional bias within egocentricsaliency maps (posterior parietal cortex) (Gottlieb, 2007),2) the aforementioned IOR inputs from other modali-ties (Stein et al., 2002), 3) overriding voluntary saccades(frontal eye elds) (Stein et al., 2002) and 4) basal gan-glia action selection (McHafe et al., 2005). Thus, bio-logically there exists a highly developed, context specicmethod for facilitating the most appropriate saccade asa form of attention selection. All of the above saccade-affecting attributes have valuable robotic application butinhibition of return is potentially the most useful in theearlier stages of constructing a saccade system that isattention rather than visual-input driven. For example,within the most basic of active vision system tasks wherestatic objects of the same shape and color are systemat-ically saccaded to (i.e. brought to the centre of image),there is a consistent need for a mechanism whereby ob-jects already scanned are ignored (i.e. inhibition of re-turn). The primary issue here is that similar image datacan emerge in very different image locations, thus theonly way of knowing whether an image feature has pre-viously been saccaded to or not, is to store that informa-tion at the global level. In the following we introducean architecture developed for a robotic active vision sys-tem where that architecture enables the system to inte-grate and update global information which can in turnmodulate the action selection process for saccadic cameramovements.The active vision system consists of two cameras (bothprovide RGB 1032x778 image data) mounted on a mo-torised pan-tilt-verge unit. Three degrees of freedom(DOF) are used: one verge movement for each cameraand one tilt which moves both cameras. Each motor iscontrolled by determining its position in radians (rad)where the state of the active vision system is fully de-termined by the motor positions of the tilt, left and rightverge axis, (ptilt ; pvL; pvR).The overall computational architecture is illustrated inFigure 1. It consists of three main parts implementing:1) ltering image data, 2) action selection and executionand 3) the operation of the visual memory. The latter isthe central feature of this architecture and main objectiveof this paper. Without the visual memory, action selectionand the resulting saccadic eye movements are determinedsolely by the current retina image data. Hence, similarvisual inputs (RGB image) lead to the same saccade, nomatter how often this specic saccade has been executedbefore. With a visual memory in place, however, specicmotor positions (ptilt ; pvL; pvR) resulting from a success-ful saccadic camera movement can be stored. This infor-mation can then be used to merge the camera image datawith the data representing the items present in the visualmemory (i.e. those previously saccaded to). The inhibi-tion of return process can then be simply carried out bysubtracting the latter from the former, essentially trans-forming the original camera input into a `retina-basedsaliency map' where, objects in the visual memory havebeen inhibited leaving unsaccaded objects to compete forwX∆ ∆ ∆       tilt         vL        vR(   p   ,    p   ,    p   )saliency mapoverlaid withvisual memory             tilt     vL     vR(X,Y, p   , p   , p    )*        tilt         vL        vR(X,Y,    p   ,    p   ,    p   )*∆ ∆ ∆list of non−zero pixels(X,Y)*   tilt     vL     vR(p   , p   , p    )   tilt     vL     vR(p   , p   , p    )RGB imagewRet x hRetfilter Cfilter Bfilter Aw wA CwBBG (WTA)wRet x hRet({0/1},X,Y)*visual memoryin gaze space(X,Y)filter XwRet x hRetmappingfor eye−saccade−wRet x hRetactivevisionsystemmappingsaccadefor eye−wRet x hRet++saliency mapretina basedFilterandAction−Integrationand updatememoryof the visualselectionexecutionRBSMLVMMOSMFigure 1: Architecture for embodied visual memory.the next saccade. In the following the core function ofthis architecture shall be described in more detail.A visual buffer (local visual memory map or LVMM).and the mapping for the saccadic eye movement (retinabased saliency map or RBSM) are the essential elementsnecessary to create the so called overlaid saliency map(OSM), see Figure 1. The OSM then feeds into an actionselection process: Basal Ganglia, (BG). The LVMM rep-resents stimuli which have corresponding entries in thevisual memory. The creation of the LVMM is, thus, acrucial part of the architecture. This process starts withRBSMwhere, for each no-zero pixel in RBSM, the corre-sponding D values (Dptilt ; DpvL;DpvR) are derived. TheseD values are learnt beforehand through a mapping pro-cess previously described (Lee et al., 2007). Hence, foreach non-zero pixel in RBSM we get the relative mo-tor positions (Dptilt ;DpvL; DpvR) which drives the par-ticular pixel into the image center. The result of thisstep is stored as a list where each entry is written as:(X ;Y;Dptilt ;DpvL;DpvR). Notice, in Figure 1 an aster-isk signies a list. Adding these D-values to the cur-rent absolute motor positions (ptilt ; pvL; pvR) provided bythe active vision system delivers the nal absolute motorpositions of the active vision system if a saccade move-ment was executed. This is again represented as a list:(X ;Y; ptilt ; pvL; pvR). Thus, the D-values are replaced bythe nal absolute motor positions. With this global infor-mation the system can now easily ask if a specic pixel(X ;Y ) in the current RBSM has a corresponding item inthe visual memory. If the derived absolute motor posi-tions of pixel (X ;Y ) can be found in the visual memorythen this pixel is labelled with value of 1 otherwise it is la-beled as 0. Thus, all list entries appear as: (X ;Y;f0;1g):From this list we can then create the LVMM which hasthe same dimensions as RBSM. Since LVMM containsall previously saccaded to pixels (value 1:0), substractionfrom RBSM results in the aforementioned `retina-basedsaliency map' and an accurate mapping of objects thathave not yet been saccaded to.Although several computational models of inhibitionof return of have been put forward e.g. (Sun et al., 2008),the robotic implementation of such a process has, untilnow, not been fully described. It is noted, however, thatthe implementation of the architecture presented here, byusing lists to store and process data, is not biologicallyplausible. However, this provides a good real-time per-formance, and thus is considered as a pragmatic balancebetween biological constraint and robotic efciency.AcknowledgmentThanks for support from EC-FP7 projects IM-CLeVeRand ROSSI, and EPSRC, grant EP/C516303/1.ReferencesGottlieb, J. (2007). From thought to action: The parietalcortex as a bridge between perception, action, andcognition. Neuron, 53, 1, 9-16.Lee, M., Meng, Q., and Chao, F. (2007). Developmen-tal learning for autonomous robots. Robotics andAutonomous Systems, 55 (9), 750-759.Lupianez, J., Klein, R., and Bartolomeo, P. (2006). Inhi-bition of return: Twenty years after. Cognitive Neu-ropsychology, 23, 7, 1003-1014.Mayer, A., Seidenberg, M., Doringer, J., and Rao, S.(2004). An event-related fmri study of exogenousorienting: Supporting evidence for the cortical basisof inhibition of return? Journal of Cognitive Neuro-science, 16, 7, 1262-1271.McHafe, J., Stanford, T., Stein, B., Coizet, W., andRedgrave, P. (2005). Subcortical loops through thebasal ganglia. Trends in Neurosciences, 28, 8, 401-407.Stein, B. and Meredith, M. (1991). Functional organi-zation of the superior colliculus. In A.G., L., (Ed.),The neural bases if visual function, pages 85100.Macmillan,, Hampshire.Stein, B., Wallace, M., Stanford, T., and Jiang, W.(2002). Cortex governs multisensory integration inthe midbrain. Neuroscientist, 8, 4, 306-314.Sun, Y., Fisher, R., Wang, F., and Gomes, H. (2008).A computer vision model for visual-object-based at-tention and eye movements. Computer Vision andImage Understanding, 112, 2, 126-142.

Implementing inhibition of return: embodied visual memory for robotic systems

Aberystwyth UniversityImplementing inhibition of return: embodied visual memory for robotic systemsHülse, Martin; McBride, Sebastian; Lee, MarkPublished in:Proc. 9th Int. Conf. on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund Uni.Cognitive StudiesPublication date:2009Citation for published version (APA):Hülse, M., McBride, S., & Lee, M. (2009). Implementing inhibition of return: embodied visual memory for roboticsystems. In L. Canamero, P. -Y. Oudeyer, & C. Balkenius (Eds.), Proc. 9th Int. Conf. on Epigenetic Robotics:Modeling Cognitive Development in Robotic Systems. Lund Uni. Cognitive Studies (pp. 213-214)http://hdl.handle.net/2160/3979General rightsCopyright and moral rights for the publications made accessible in the Aberystwyth Research Portal (the Institutional Repository) areretained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights.            • Users may download and print one copy of any publication from the Aberystwyth Research Portal for the purpose of private study orresearch.            • You may not further distribute the material or use it for any profit-making activity or commercial gain            • You may freely distribute the URL identifying the publication in the Aberystwyth Research PortalTake down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.tel: +44 1970 62 2400email: is@aber.ac.ukDownload date: 09. Jul. 2020Implementing inhibition of return; embodiedvisual memory for robotic systemsMartin Hu¨lse Sebastian McBride Mark LeeDept. of Computer Science, Aberystwyth University, SY23 3DB, Wales, UKAbstractBased on the biological phenomenon of inhibi-tion of return, we introduce an architecture de-veloped for an active robotic vision system wherecontinually updated global information is used tomodulate the action selection process for saccadiccamera movements. This facilitates, in an ex-tremely efcient way, the fundamental process ofavoiding re-saccading to objects previously vis-ited and, thus, is considered to have a wide-ranging application within active vision systems.Inhibition of return (IOR) refers to the suppressionof stimuli (objects and events) processing where thosestimuli have previously (and recently) been the focus ofspatial attention (Lupianez et al., 2006). In this sense, itforms the basis of attentional (and thus visual) bias to-wards novel objects. Although the neural mechanismunderpinning IOR is not completely understood, it iswell established that the dorsal frontoparietal network,including frontal eye elds (FEF) and superior parietalcortex are the primary structures mediating its control(Mayer et al., 2004). These are some of the many mod-ulatory and affecting structures of the deep superior col-liculus (optic tectum in non-mammals), the primary mo-tor structure controlling saccade. Although visual in-formation from the retina starts at the supercial su-perior colliculus, and there are direct connections be-tween the supercial and deep layers, the former can-not elicit saccade directly (Stein and Meredith, 1991).This information has to be subsequently processed by anumber of cortical and sub-cortical structures that placeit: 1) in context of attentional bias within egocentricsaliency maps (posterior parietal cortex) (Gottlieb, 2007),2) the aforementioned IOR inputs from other modali-ties (Stein et al., 2002), 3) overriding voluntary saccades(frontal eye elds) (Stein et al., 2002) and 4) basal gan-glia action selection (McHafe et al., 2005). Thus, bio-logically there exists a highly developed, context specicmethod for facilitating the most appropriate saccade asa form of attention selection. All of the above saccade-affecting attributes have valuable robotic application butinhibition of return is potentially the most useful in theearlier stages of constructing a saccade system that isattention rather than visual-input driven. For example,within the most basic of active vision system tasks wherestatic objects of the same shape and color are systemat-ically saccaded to (i.e. brought to the centre of image),there is a consistent need for a mechanism whereby ob-jects already scanned are ignored (i.e. inhibition of re-turn). The primary issue here is that similar image datacan emerge in very different image locations, thus theonly way of knowing whether an image feature has pre-viously been saccaded to or not, is to store that informa-tion at the global level. In the following we introducean architecture developed for a robotic active vision sys-tem where that architecture enables the system to inte-grate and update global information which can in turnmodulate the action selection process for saccadic cameramovements.The active vision system consists of two cameras (bothprovide RGB 1032x778 image data) mounted on a mo-torised pan-tilt-verge unit. Three degrees of freedom(DOF) are used: one verge movement for each cameraand one tilt which moves both cameras. Each motor iscontrolled by determining its position in radians (rad)where the state of the active vision system is fully de-termined by the motor positions of the tilt, left and rightverge axis, (ptilt ; pvL; pvR).The overall computational architecture is illustrated inFigure 1. It consists of three main parts implementing:1) ltering image data, 2) action selection and executionand 3) the operation of the visual memory. The latter isthe central feature of this architecture and main objectiveof this paper. Without the visual memory, action selectionand the resulting saccadic eye movements are determinedsolely by the current retina image data. Hence, similarvisual inputs (RGB image) lead to the same saccade, nomatter how often this specic saccade has been executedbefore. With a visual memory in place, however, specicmotor positions (ptilt ; pvL; pvR) resulting from a success-ful saccadic camera movement can be stored. This infor-mation can then be used to merge the camera image datawith the data representing the items present in the visualmemory (i.e. those previously saccaded to). The inhibi-tion of return process can then be simply carried out bysubtracting the latter from the former, essentially trans-forming the original camera input into a `retina-basedsaliency map' where, objects in the visual memory havebeen inhibited leaving unsaccaded objects to compete forwX∆ ∆ ∆       tilt         vL        vR(   p   ,    p   ,    p   )saliency mapoverlaid withvisual memory             tilt     vL     vR(X,Y, p   , p   , p    )*        tilt         vL        vR(X,Y,    p   ,    p   ,    p   )*∆ ∆ ∆list of non−zero pixels(X,Y)*   tilt     vL     vR(p   , p   , p    )   tilt     vL     vR(p   , p   , p    )RGB imagewRet x hRetfilter Cfilter Bfilter Aw wA CwBBG (WTA)wRet x hRet({0/1},X,Y)*visual memoryin gaze space(X,Y)filter XwRet x hRetmappingfor eye−saccade−wRet x hRetactivevisionsystemmappingsaccadefor eye−wRet x hRet++saliency mapretina basedFilterandAction−Integrationand updatememoryof the visualselectionexecutionRBSMLVMMOSMFigure 1: Architecture for embodied visual memory.the next saccade. In the following the core function ofthis architecture shall be described in more detail.A visual buffer (local visual memory map or LVMM).and the mapping for the saccadic eye movement (retinabased saliency map or RBSM) are the essential elementsnecessary to create the so called overlaid saliency map(OSM), see Figure 1. The OSM then feeds into an actionselection process: Basal Ganglia, (BG). The LVMM rep-resents stimuli which have corresponding entries in thevisual memory. The creation of the LVMM is, thus, acrucial part of the architecture. This process starts withRBSMwhere, for each no-zero pixel in RBSM, the corre-sponding D values (Dptilt ; DpvL;DpvR) are derived. TheseD values are learnt beforehand through a mapping pro-cess previously described (Lee et al., 2007). Hence, foreach non-zero pixel in RBSM we get the relative mo-tor positions (Dptilt ;DpvL; DpvR) which drives the par-ticular pixel into the image center. The result of thisstep is stored as a list where each entry is written as:(X ;Y;Dptilt ;DpvL;DpvR). Notice, in Figure 1 an aster-isk signies a list. Adding these D-values to the cur-rent absolute motor positions (ptilt ; pvL; pvR) provided bythe active vision system delivers the nal absolute motorpositions of the active vision system if a saccade move-ment was executed. This is again represented as a list:(X ;Y; ptilt ; pvL; pvR). Thus, the D-values are replaced bythe nal absolute motor positions. With this global infor-mation the system can now easily ask if a specic pixel(X ;Y ) in the current RBSM has a corresponding item inthe visual memory. If the derived absolute motor posi-tions of pixel (X ;Y ) can be found in the visual memorythen this pixel is labelled with value of 1 otherwise it is la-beled as 0. Thus, all list entries appear as: (X ;Y;f0;1g):From this list we can then create the LVMM which hasthe same dimensions as RBSM. Since LVMM containsall previously saccaded to pixels (value 1:0), substractionfrom RBSM results in the aforementioned `retina-basedsaliency map' and an accurate mapping of objects thathave not yet been saccaded to.Although several computational models of inhibitionof return of have been put forward e.g. (Sun et al., 2008),the robotic implementation of such a process has, untilnow, not been fully described. It is noted, however, thatthe implementation of the architecture presented here, byusing lists to store and process data, is not biologicallyplausible. However, this provides a good real-time per-formance, and thus is considered as a pragmatic balancebetween biological constraint and robotic efciency.AcknowledgmentThanks for support from EC-FP7 projects IM-CLeVeRand ROSSI, and EPSRC, grant EP/C516303/1.ReferencesGottlieb, J. (2007). From thought to action: The parietalcortex as a bridge between perception, action, andcognition. Neuron, 53, 1, 9-16.Lee, M., Meng, Q., and Chao, F. (2007). Developmen-tal learning for autonomous robots. Robotics andAutonomous Systems, 55 (9), 750-759.Lupianez, J., Klein, R., and Bartolomeo, P. (2006). Inhi-bition of return: Twenty years after. Cognitive Neu-ropsychology, 23, 7, 1003-1014.Mayer, A., Seidenberg, M., Doringer, J., and Rao, S.(2004). An event-related fmri study of exogenousorienting: Supporting evidence for the cortical basisof inhibition of return? Journal of Cognitive Neuro-science, 16, 7, 1262-1271.McHafe, J., Stanford, T., Stein, B., Coizet, W., andRedgrave, P. (2005). Subcortical loops through thebasal ganglia. Trends in Neurosciences, 28, 8, 401-407.Stein, B. and Meredith, M. (1991). Functional organi-zation of the superior colliculus. In A.G., L., (Ed.),The neural bases if visual function, pages 85100.Macmillan,, Hampshire.Stein, B., Wallace, M., Stanford, T., and Jiang, W.(2002). Cortex governs multisensory integration inthe midbrain. Neuroscientist, 8, 4, 306-314.Sun, Y., Fisher, R., Wang, F., and Gomes, H. (2008).A computer vision model for visual-object-based at-tention and eye movements. Computer Vision andImage Understanding, 112, 2, 126-142.

http://pure.aber.ac.uk/ws/files/2309619/epiRob09Sub.pdf

Implementing inhibition of return: embodied visual memory for robotic systems

Abstract

Similar works

Full text

Available Versions

Aberystwyth Research Portal

Aberystwyth Research Portal