Connections between articulations and grasping

The idea that hand gestures and speech are connected is quite old. Some of these theories even suggest that language is primarily based on a manual communication system. In this thesis, I present four studies in which we studied the connections between articulatory gestures and manual grasps. The work is based on an earlier finding showing systematic connections between specific articulatory gestures and grasp types. For example, uttering a syllable such as [kɑ] can facilitate power grip responses, whereas uttering a syllable such as [ti] can facilitate precision grip responses. I will refer to this phenomenon as the articulation-grip congruency effect. Similarly, to the original work, we used special power and precision grip devices that the participants held in their hand to perform responses. In Study I, we measured response times and accuracy of grip responses and vocalisations to investigate whether the effect can be also observed in vocal responses, and to which extent the effect operates in the action selection processes. In Study II, grip response times were measured to investigate whether the effect persists when the syllables are only heard or read silently. Study III investigated the influence of grasp planning and/or execution on categorizing perceived syllables. In Study IV, we measured electrical activity in the brain during listening of syllables that were either congruent or incongruent with the precision or power grip, and we investigated how performing different grips affected the auditory processing of the heard syllables. The results of Study I showed that besides manual facilitation, the effect is observed also in vocal responses, both when a simultaneous grip is executed and when it is only prepared, meaning that overt execution is not needed for the effect. This suggests that the effect operates in action planning. In addition, the effect was also observed when the participants knew beforehand which response they should execute, suggesting that the effect is not based on the action selection processes. Study II showed that the effect was also observed when the syllables were heard or read silently, supporting the view that articulatory simulation of a perceived syllable can activate the motor program of the grasp which is congruent with the syllable. Study III revealed that grip preparation can influence categorization of perceived syllables. The participants were biased to categorize noise-masked syllables as being [ke] rather than [te] when they were prepared to execute the power grip, and vice versa when they were prepared to execute the precision grip. Finally, Study IV showed that grip performance also modulates early auditory processing of heard syllables. These results support the view that articulatory and hand motor representations form a partly shared network, where activity from one domain can induce activity in the other. This is in line with earlier studies that have shown more general linkage between mouth and manual processes and expands this notion of hand-mouth interaction by showing that these connections can also operate between very specific hand and articulatory gestures.Ajatus käden eleiden ja puheen välisistä yhteyksistä on melko vanha. Jotkut teoriat jopa ehdottavat, että kieli pohjautuu pääosin käsillä tapahtuvaan kommunikointijärjestelmään. Tässä väitöskirjassa esittelen neljä osatyötä, joissa tutkimme artikulatoristen eleiden ja tarttumisotteiden välisiä yhteyksiä. Työ perustuu aiempaan löydökseen, joka paljasti systemaattisia yhteyksiä tiettyjen artikulatoristen eleiden ja tarttumisotteiden välillä. Esimerkiksi [kɑ] tavun lausuminen nopeuttaa voimaotteen tekemistä, kun taas esimerkiksi [ti] tavun lausuminen nopeuttaa pinsettiotteen tekemistä. Väitöskirjan osatyöt hyödynsivät tätä perusefektiä muokkaamalla koeasetelmaa kuhunkin tutkimuskysymykseen sopivaksi. Osatyön I tulokset osoittivat, että yhteensopivuusefekti on havaittavissa myös lausutuissa vastauksissa. Efekti havaittiin myös, kun otteen suorittamiseen oli vain valmistauduttu. Tämä viittaa siihen, että efekti toimii toimintojen suunnittelun tasolla. Lisäksi efekti havaittiin silloinkin, kun osallistujat tiesivät etukäteen, mikä vastaus heidän tulisi suorittaa, mikä viittaa siihen, ettei efekti perustu toimintojen valintaan liittyviin prosesseihin. Osatyössä II efekti havaittiin, vaikka tavut vain kuultiin tai luettiin äänettömästä. Tämä tukee näkemystä, että havaittujen tavujen artikulatorinen simulointi voi aktivoida tavun kanssa yhteensopivan otteen motorista ohjelmaa. Osatyö III osoitti, että käden otteet voivat vaikuttaa havaittujen tavujen luokitteluun. Osallistujat olivat biasoituneet luokittelemaan esitettyjen kohinaisten tavujen olevan ennemmin [ke] kuin [te], kun he olivat valmistautuneet suorittamaan voimaotteen ja päinvastoin, kun he olivat valmistautuneet pinsettiotteen suorittamiseen. Viimeisimpänä osatyö IV osoitti, että otteiden suorittaminen vaikuttaa myös havaittujen tavujen varhaiseen auditoriseen prosessointiin. Nämä tulokset tukevat näkemystä, että artikulatoriset ja käden motoriset edustukset muodostavat osittain jaetun verkoston, jossa aktiivisuus yhdellä osa-alueella voi aiheuttaa aktiivisuutta myös toisella. Tämä on linjassa aiheen aiempien tutkimusten kanssa, jotka ovat osoittaneet yleisempiä yhteyksiä käden ja suun toimintojen välillä. Nämä tulokset laajentavat käden ja suun välisen yhteyden ajatusta osoittamalla, että yhteydet voivat toimia myös hyvin tarkasti rajattujen artikulatoristen ja käden eleiden välillä

Inter-limb coupling of proximal and distal hand actions

Past studies have revealed connections in directional programming between hands. The present study investigated whether there could also be interaction in programming proximal and distal components of a prehensile hand action. In Experiment 1, the participants performed simultaneously either a push or pull response with the left hand and the grip closure or opening with the right hand. In Experiment 2, the push and pull responses of the left hand were performed together with the precision or power grip responses of the right hand. The participants showed preference, measured in reaction times, to couple the push response with the grip opening and the precision grip, whereas the pull response was associated with the grip closure and the power grip. The study shows for the first time a systematic interaction in proximal and distal prehensile components between two hands. We propose that these effects reflect inter- and intra-limb connections between the representations that prepare the arm extension for the outward reaching, the finger extension for the grip opening, and the motor processes that prepare the precision grip. Conversely, there appear to be connections between the representations that prepare the arm flexion for the inward directed hand movements, the flexion of the thumb and the fingers for the grip closure, and flexion of four fingers for the power grip.Peer reviewe

Connecting directional limb movements to vowel fronting and backing

It has been shown recently that when participants are required to pronounce a vowel at the same time with the hand movement, the vocal and manual responses are facilitated when a front vowel is produced with forward-directed hand movements and a back vowel is produced with backward-directed hand movements. This finding suggests a coupling between spatial programing of articulatory tongue movements and hand movements. The present study revealed that the same effect can be also observed in relation to directional leg movements. The study suggests that the effect operates within the common directional processes of movement planning including at least tongue, hand and leg movements, and these processes might contribute sound-to-meaning mappings to the semantic concepts of 'forward' and 'backward'.Peer reviewe

Kansallisen Pitkäaikaissäilytyksen hiilijalanjälki

Pistäytymisklinikan esittely Digime2022-seminaarissa ”Tarjolla tänään ja huomenna: Digitaalinen kulttuuriperintö” 14.10.2022

Sharp and round shapes of seen objects have distinct influences on vowel and consonant articulation

The shape and size-related sound symbolism phenomena assume that, for example, the vowel [i] and the consonant [t] are associated with sharp-shaped and small-sized objects, whereas [E] and [m] are associated with round and large objects. It has been proposed that these phenomena are mostly based on the involvement of articulatory processes in representing shape and size properties of objects. For example, [i] might be associated with sharp and small objects, because it is produced by a specific front-close shape of articulators. Nevertheless, very little work has examined whether these object properties indeed have impact on speech sound vocalization. In the present study, the participants were presented with a sharp- or round-shaped object in a small or large size. They were required to pronounce one out of two meaningless speech units (e.g., [i] or [E]) according to the size or shape of the object. We investigated how a task-irrelevant object property (e.g., the shape when responses are made according to size) influences reaction times, accuracy, intensity, fundamental frequency, and formant 1 and formant 2 of vocalizations. The size did not influence vocal responses but shape did. Specifically, the vowel [i] and consonant [t] were vocalized relatively rapidly when the object was sharp-shaped, whereas [u] and [m] were vocalized relatively rapidly when the object was round-shaped. The study supports the view that the shape-related sound symbolism phenomena might reflect mapping of the perceived shape with the corresponding articulatory gestures.Peer reviewe

Mismatch negativity (MMN) to speech sounds is modulated systematically by manual grip execution

Manual actions and speech are connected: for example, grip execution can influence simultaneous vocalizations and vice versa. Our previous studies show that the consonant [k] is associated with the power grip and the consonant [t] with the precision grip. Here we studied whether the interaction between speech sounds and grips could operate already at a pre-attentive stage of auditory processing, reflected by the mismatch-negativity (MMN) component of the event-related potential (ERP). Participants executed power and precision grips according to visual cues while listening to syllable sequences consisting of [ke] and [te] utterances. The grips modulated the MMN amplitudes to these syllables in a systematic manner so that when the deviant was [ke], the MMN response was larger with a precision grip than with a power grip. There was a converse trend when the deviant was [te]. These results suggest that manual gestures and speech can interact already at a pre-attentive processing level of auditory perception, and show, for the first time that manual actions can systematically modulate the MMN. (C) 2017 Elsevier B.V. All rights reserved.Peer reviewe

Interaction in planning vocalizations and grasping

Peer reviewe

Reaching and grasping with the tongue : Shared motor planning between hand actions and articulatory gestures

Research has shown connections between articulatory mouth actions and manual actions. This study investigates whether forward-backward hand movements could be associated with vowel production processes that programme tongue fronting/backing, lip rounding/spreading (Experiment 1), and/or consonant production processes that programme tongue tip and tongue dorsum actions (Experiment 2). The participants had to perform either forward or backward hand movement and simultaneously pronounce different vowels or consonants. The results revealed a response benefit, measured in vocal and manual reaction times, when the responses consisted of front vowels and forward hand movements. Conversely, back vowels were associated with backward hand movements. Articulation of rounded versus unrounded vowels or coronal versus dorsal consonants did not produce the effect. In contrast, when the manual responses of forward-backward hand movements were replaced by precision and power grip responses, the coronal consonants [t] and [r] were associated with the precision grip, whereas the dorsal consonant [k] was associated with the power grip. We propose that the movements of the tongue body, operating mainly for vowel production, share the directional action planning processes with the hand movements. Conversely, the tongue articulators related to tongue tip and dorsum movements, operating mainly for consonant production, share the action planning processes with the precision and power grip, respectively.Peer reviewe

Systematic Influence of Perceived Grasp Shape on Speech Production

Previous research has shown that precision and power grip performance is consistently influenced by simultaneous articulation. For example, power grip responses are performed relatively fast with the open-back vowel [a], whereas precision grip responses are performed relatively fast with the close-front vowel [i]. In the present study, the participants were presented with a picture of a hand shaped to the precision or power grip. They were required to pronounce speech sounds according to the front/above perspective of the hand. The results showed that not only the grip performance is affected by simultaneously pronouncing the speech sound but also the production of speech sound can be affected by viewing an image of a grip. The precision grip stimulus triggered relatively rapid production of the front-close vowel [i]. In contrast, the effect related to the power grip stimulus was mostly linked to the vertical dimension of the pronounced vowel since this stimulus triggered relatively rapid production of the back-open vowel [a] and back-mid-open vowel [o] while production of the back-close vowel [u] was not influenced by it. The fact that production of the dorsal consonant [k] or coronal consonant [t] were not influenced by these stimuli suggests that the effect was not associated with a relative front-back tongue shape of the articulation in the absence of changes in any vertical articulatory components. These findings provide evidence for an intimate interaction between certain articulatory gestures and grip types, suggesting that an overlapping visuomotor network operates for planning articulatory gestures and grasp actions.Peer reviewe

Preparation and execution of teeth clenching and foot muscle contraction influence on corticospinal hand-muscle excitability

Contraction of a muscle modulates not only the corticospinal excitability (CSE) of the contracting muscle but also that of different muscles. We investigated to what extent the CSE of a hand muscle is modulated during preparation and execution of teeth clenching and ipsilateral foot dorsiflexion either separately or in combination. Hand-muscle CSE was estimated based on motor evoked potentials (MEPs) elicited by transcranial magnetic stimulation (TMS) and recorded from the first dorsal interosseous (FDI) muscle. We found higher excitability during both preparation and execution of all the motor tasks than during mere observation of a fixation cross. As expected, the excitability was greater during the execution phase than the preparation one. Furthermore, both execution and preparation of combined motor tasks led to higher excitability than individual tasks. These results extend our current understanding of the neural interactions underlying simultaneous contraction of muscles in different body parts.Peer reviewe