The neuroscience of vision-based grasping: a functional review for computational modeling and bio-inspired robotics

The topic of vision-based grasping is being widely studied using various techniques and with different goals in humans and in other primates. The fundamental related findings are reviewed in this paper, with the aim of providing researchers from different fields, including intelligent robotics and neural computation, a comprehensive but accessible view on the subject. A detailed description of the principal sensorimotor processes and the brain areas involved in them is provided following a functional perspective, in order to make this survey especially useful for computational modeling and bio-inspired robotic application

Structural connectivity and functional properties of the macaque superior parietal lobule

Despite the consolidated belief that the macaque superior parietal lobule (SPL) is entirely occupied by Brodmann’s area 5, recent data show that macaque SPL also hosts a large cortical region with structural and functional features similar to that of Brodmann’s area 7. According to these data, the anterior part of SPL is occupied by a somatosensory-dominated cortical region that hosts three architectural and functional distinct regions (PE, PEci, PEip) and the caudal half of SPL by a bimodal somato-visual region that hosts four areas: PEc, MIP, PGm, V6A. To date, the most studied areas of SPL are PE, PEc, and V6A. PE is essentially a high-order somatomotor area, while PEc and V6A are bimodal somatomotor–visuomotor areas, the former with predominant somatosensory input and the latter with predominant visual input. The functional properties of these areas and their anatomical connectivity strongly suggest their involvement in the control of limb movements. PE is suggested to be involved in the preparation/execution of limb movements, in particular, the movements of the upper limb; PEc in the control of movements of both upper and lower limbs, as well as in their interaction with the visual environment; V6A in the control of reach-to-grasp movements performed with the upper limb. In humans, SPL is traditionally considered to have a different organization with respect to macaques. Here, we review several lines of evidence suggesting that this is not the case, showing a similar structure for human and non-human primate SPLs

Optic Ataxia: From Balint’s Syndrome to the Parietal Reach Region

Optic ataxia is a high-order deficit in reaching to visual goals that occurs with posterior parietal cortex (PPC) lesions. It is a component of Balint’s syndrome that also includes attentional and gaze disorders. Aspects of optic ataxia are misreaching in the contralesional visual field, difficulty preshaping the hand for grasping, and an inability to correct reaches online. Recent research in nonhuman primates (NHPs) suggests that many aspects of Balint’s syndrome and optic ataxia are a result of damage to specific functional modules for reaching, saccades, grasp, attention, and state estimation. The deficits from large lesions in humans are probably composite effects from damage to combinations of these functional modules. Interactions between these modules, either within posterior parietal cortex or downstream within frontal cortex, may account for more complex behaviors such as hand-eye coordination and reach-to-grasp

Remembering Forward: Neural Correlates of Memory and Prediction in Human Motor Adaptation

We used functional MR imaging (FMRI), a robotic manipulandum and systems identification techniques to examine neural correlates of predictive compensation for spring-like loads during goal-directed wrist movements in neurologically-intact humans. Although load changed unpredictably from one trial to the next, subjects nevertheless used sensorimotor memories from recent movements to predict and compensate upcoming loads. Prediction enabled subjects to adapt performance so that the task was accomplished with minimum effort. Population analyses of functional images revealed a distributed, bilateral network of cortical and subcortical activity supporting predictive load compensation during visual target capture. Cortical regions – including prefrontal, parietal and hippocampal cortices – exhibited trial-by-trial fluctuations in BOLD signal consistent with the storage and recall of sensorimotor memories or “states” important for spatial working memory. Bilateral activations in associative regions of the striatum demonstrated temporal correlation with the magnitude of kinematic performance error (a signal that could drive reward-optimizing reinforcement learning and the prospective scaling of previously learned motor programs). BOLD signal correlations with load prediction were observed in the cerebellar cortex and red nuclei (consistent with the idea that these structures generate adaptive fusimotor signals facilitating cancelation of expected proprioceptive feedback, as required for conditional feedback adjustments to ongoing motor commands and feedback error learning). Analysis of single subject images revealed that predictive activity was at least as likely to be observed in more than one of these neural systems as in just one. We conclude therefore that motor adaptation is mediated by predictive compensations supported by multiple, distributed, cortical and subcortical structures

Vision for action: thalamic and cortical inputs to the macaque superior parietal lobule

open5noOpen access funding provided by Alma Mater Studiorum - Università di Bologna within the CRUI-CARE Agreement. This work was supported by grants from Ministero dell’Università e della Ricerca (2017KZNZLN), Fondazione Cassa di Risparmio di Bologna (Bando Internazionalizzazione), Italy, and by the European Commission funded project H2020-EIC-FETPROACT-2019-951910-MAIA. The support is gratefully acknowledged.The dorsal visual stream, the cortical circuit that in the primate brain is mainly dedicated to the visual control of actions, is split into two routes, a lateral and a medial one, both involved in coding different aspects of sensorimotor control of actions. The lateral route, named “lateral grasping network”, is mainly involved in the control of the distal part of prehension, namely grasping and manipulation. The medial route, named “reach-to-grasp network”, is involved in the control of the full deployment of prehension act, from the direction of arm movement to the shaping of the hand according to the object to be grasped. In macaque monkeys, the reach-to-grasp network (the target of this review) includes areas of the superior parietal lobule (SPL) that hosts visual and somatosensory neurons well suited to control goal-directed limb movements toward stationary as well as moving objects. After a brief summary of the neuronal functional properties of these areas, we will analyze their cortical and thalamic inputs thanks to retrograde neuronal tracers separately injected into the SPL areas V6, V6A, PEc, and PE. These areas receive visual and somatosensory information distributed in a caudorostral, visuosomatic trend, and some of them are directly connected with the dorsal premotor cortex. This review is particularly focused on the origin and type of visual information reaching the SPL, and on the functional role this information can play in guiding limb interaction with objects in structured and dynamic environments.openGamberini M.; Passarelli L.; Filippini M.; Fattori P.; Galletti C.Gamberini M.; Passarelli L.; Filippini M.; Fattori P.; Galletti C

Cortico-spinal modularity in the parieto-frontal system: a new perspective on action control

: Classical neurophysiology suggests that the motor cortex (MI) has a unique role in action control. In contrast, this review presents evidence for multiple parieto-frontal spinal command modules that can bypass MI. Five observations support this modular perspective: (i) the statistics of cortical connectivity demonstrate functionally-related clusters of cortical areas, defining functional modules in the premotor, cingulate, and parietal cortices; (ii) different corticospinal pathways originate from the above areas, each with a distinct range of conduction velocities; (iii) the activation time of each module varies depending on task, and different modules can be activated simultaneously; (iv) a modular architecture with direct motor output is faster and less metabolically expensive than an architecture that relies on MI, given the slow connections between MI and other cortical areas; (v) lesions of the areas composing parieto-frontal modules have different effects from lesions of MI. Here we provide examples of six cortico-spinal modules and functions they subserve: module 1) arm reaching, tool use and object construction; module 2) spatial navigation and locomotion; module 3) grasping and observation of hand and mouth actions; module 4) action initiation, motor sequences, time encoding; module 5) conditional motor association and learning, action plan switching and action inhibition; module 6) planning defensive actions. These modules can serve as a library of tools to be recombined when faced with novel tasks, and MI might serve as a recombinatory hub. In conclusion, the availability of locally-stored information and multiple outflow paths supports the physiological plausibility of the proposed modular perspective

Specific Roles Of Macaque Parietal Regions In Making Saccades And Reaches

A principle task of our brain is to guide movements, includng saccade: fast eye movements) and reaches towards things that we see. Regions in the parietal cortex such as LIP and PRR are active during visually-guided movements. Neurons in these areas respond differentially for saccades versus reaches, but in most parietal areas there is some response: in single unit recording as well as in fMRI imaging) with either type of movement. This raises an important question. What is the functional significance of the neuronal activity in parietal areas? Recording and imaging studies can only show correlations; causal roles must be inferred. The activity in any particular area could reflect where the subject\u27s spatial attention is directed, without regard for what behavior the subject will perform. Stronger activity in one task compared to another could reflect differential allocation of attention. For example, we might attend more strongly to a target for an eye movement than to a target for an arm movement, or vice versa. Alternatively, might play a causal role in driving only one type of movement. In this case, the weaker activity evoked during a different type of movement might serve no purpose at all; it might represent a contingency plan to perform the non-selected movement; or it might be serve some other function unrelated to the specific movement - for example, weak saccade-related activity in an area with strong arm movement related signals might support play no role in driving eye movements, but instead provide timing information to the reaching system to support eye-hand coordination. To help resolve this mystery, we used an interventional approach. We asked what happens to reaches and saccades when we reversibly lesioned specific areas in the monkey parietal cortex. In order to establish what brain regions were affected in each inactivation experiment, we developed a novel technique to image the location of the lesions in vivo. The results of this causal manipulation were clear: LIP lesions delay the initiation of saccades and have no effect on reaches, while PRR lesions delay the initiation of reaches and have no effect on saccades. We obtained further evidence for a more motoric role for parietal areas than previously suspected. PRR was active for reaches of only the contralateral arm, aimed at targets in either hemisphere - similar to the typical profiles of motor but not visual sensory areas. Interestingly, LIP lesions did influence reaches, but only when the animals were allowed to first look at the target before reaching for it. We believe that in this case, the reaching movement waits for the saccade system, and so the direct effect of the lesion on the saccades has an indirect effect on the reaches. These results are important for several reasons. First, they resolve a long-standing debate regarding the functional specificity of parietal areas with regard to particular movements and attention. They provide new information on the circuits guiding eye movements, arm movements and eye-hand coordination. Finally, our results underscore the fact that measurements of neuronal activity can be misleading, and are only one of several tools that must be used in order to understand brain function

The Role of the Dorsal Premotor and Superior Parietal Cortices in Decoupled Visuomotor Transformations

In order to successfully interact with objects located within our environment, the brain must be capable of combining visual information with the appropriate felt limb position (i.e. proprioception) in order compute an appropriate coordinated muscle plan for accurate motor control. Eye-hand coordination is essential to our independence as a species and relies heavily on the reciprocally-connected regions of the parieto-frontal reach network. The dorsal premotor cortex (PMd) and the superior parietal lobule (SPL) remain prime candidates within this network for controlling the transformations required during visually-guided reaching movements. Our brains are primed to reach directly towards a viewed object, a situation that has been termed a “standard” or coupled reach. Such direct eye-hand coordination is common across species and is crucial for basic survival. Humans, however, have developed the capacity for tool-use and thus have learned to interact indirectly with an object. In such “non-standard” or decoupled situations, the directions of gaze and arm movement have been spatially decoupled and rely on both the implementation of a cognitive rule and on online feedback of the decoupled limb. The studies included within this dissertation were designed to further characterize the role of PMd and SPL during situations in which when a reach requires a spatial transformation between the actions of the eyes and the hand. More specifically, we were interested in examining whether regions within PMd (PMdr, PMdc) and SPL (PEc, MIP) responded differently during coupled versus decoupled visuomotor transformations. To address the relative contribution of these various cortical regions during decoupled reaching movements, we trained two female rhesus macaques on both coupled and decoupled visually-guided reaching tasks. We recorded the neural activity (single units and local field potentials) within each region while the animals performed each condition. We found that two separate networks emerged each contributing in a distinct ways to the performance of coupled versus decoupled eye-hand reaches. While PMdr and PEc showed enhanced activity during decoupled reach conditions, PMdc and MIP were more enhanced during coupled reaches. Taken together, these data presented here provide further evidence for the existence of alternate task-dependent neural pathways for visuomotor integration