Multimodal language processing in human communication

Multiple layers of visual (and vocal) signals, plus their different onsets and offsets, represent a significant semantic and temporal binding problem during face-to-face conversation. Despite this complex unification process, multimodal messages appear to be processed faster than unimodal messages. Multimodal gestalt recognition and multilevel prediction are proposed to play a crucial role in facilitating multimodal language processing. The basis of the processing mechanisms involved in multimodal language comprehension is hypothesized to be domain general, coopted for communication, and refined with domain-specific characteristics. A new, situated framework for understanding human language processing is called for that takes into consideration the multilayered, multimodal nature of language and its production and comprehension in conversational interaction requiring fast processing

Análise de dados altimétricos obtidos por carta topográfica SRTM, Aster GDEM e Topodata para a região do município de Canela, RS.

Modelos digitais de superfície terrestre são utilizados para realizar diversas análises relacionadas à modelagem ambiental. Analisar diferentes recursos disponíveis, como SRTM, ASTER GDEM e TOPODATA, faz-se necessário, uma vez que esses dados são obtidos e disponibilizados gratuitamente. Como forma de identificar os melhores recursos, este estudo obteve os resultados e os erros gerados pela manipulação dos modelos digitais de elevação (MDE) do Município de Canela, no Rio Grande do Sul, obtidos por comparação de carta topográfica 1:50.000 com dados do SRTM, ASTER GDEM e TOPODATA, a fim de avaliar a confiabilidade dos dados e determinar sua discrepância. Para a análise da região do Município de Canela, concluímos que os dados SRTM apresentaram menor variação do erro em comparação com ASTER GDEM e TOPODATA, considerando como referência dados altimétricos na escala 1:50.000

Eyebrow movements as signals of communicative problems in human face-to-face interaction

Repair is a core building block of human communication, allowing us to address problems of understanding in conversation. Past research has uncovered the basic mechanisms by which interactants signal and solve such problems. However, the focus has been on verbal interaction, neglecting the fact that human communication is inherently multimodal. Here, we focus on a visual signal particularly prevalent in signaling problems of understanding: eyebrow frowns and raises. We present a corpus study showing that verbal repair initiations with eyebrow furrows are more likely to be responded to with clarifications as repair solutions, repair initiations that were preceded by eyebrow actions as preliminaries get repaired faster (around 230 ms), and eyebrow furrows alone can be sufficient to occasion clarification. We also present an experiment based on virtual reality technology, revealing that addressees’ eyebrow frowns have a striking effect on speakers’ speech, leading them to produce answers to questions several seconds longer than when not perceiving addressee eyebrow furrows. Together, the findings demonstrate that eyebrow movements play a communicative role in initiating repair in spoken language rather than being merely epiphenomenal. Thus, they should be considered as core coordination devices in human conversational interaction

The Inverse Shapley Value Problem

For $f$ a weighted voting scheme used by $n$ voters to choose between two candidates, the $n$ \emph{Shapley-Shubik Indices} (or {\em Shapley values}) of $f$ provide a measure of how much control each voter can exert over the overall outcome of the vote. Shapley-Shubik indices were introduced by Lloyd Shapley and Martin Shubik in 1954 \cite{SS54} and are widely studied in social choice theory as a measure of the "influence" of voters. The \emph{Inverse Shapley Value Problem} is the problem of designing a weighted voting scheme which (approximately) achieves a desired input vector of values for the Shapley-Shubik indices. Despite much interest in this problem no provably correct and efficient algorithm was known prior to our work. We give the first efficient algorithm with provable performance guarantees for the Inverse Shapley Value Problem. For any constant \eps > 0 our algorithm runs in fixed poly$(n)$ time (the degree of the polynomial is independent of \eps) and has the following performance guarantee: given as input a vector of desired Shapley values, if any "reasonable" weighted voting scheme (roughly, one in which the threshold is not too skewed) approximately matches the desired vector of values to within some small error, then our algorithm explicitly outputs a weighted voting scheme that achieves this vector of Shapley values to within error \eps. If there is a "reasonable" voting scheme in which all voting weights are integers at most \poly(n) that approximately achieves the desired Shapley values, then our algorithm runs in time \poly(n) and outputs a weighted voting scheme that achieves the target vector of Shapley values to within error $\eps=n^{-1/8}.

Visual information in computer-mediated interaction matters: Investigating the association between the availability of gesture and turn transition timing in conversation

Natural human interaction involves the fast-paced exchange of speaker turns. Crucially, if a next speaker waited with planning their turn until the current speaker was finished, language production models would predict much longer turn transition times than what we observe. Next speakers must therefore prepare their turn in parallel to listening. Visual signals likely play a role in this process, for example by helping the next speaker to process the ongoing utterance and thus prepare an appropriately-timed response. To understand how visual signals contribute to the timing of turn-taking, and to move beyond the mostly qualitative studies of gesture in conversation, we examined unconstrained, computer-mediated conversations between 20 pairs of participants while systematically manipulating speaker visibility. Using motion tracking and manual gesture annotation, we assessed 1) how visibility affected the timing of turn transitions, and 2) whether use of co-speech gestures and 3) the communicative kinematic features of these gestures were associated with changes in turn transition timing. We found that 1) decreased visibility was associated with less tightly timed turn transitions, and 2) the presence of gestures was associated with more tightly timed turn transitions across visibility conditions. Finally, 3) structural and salient kinematics contributed to gesture’s facilitatory effect on turn transition times. Our findings suggest that speaker visibility--and especially the presence and kinematic form of gestures--during conversation contributes to the temporal coordination of conversational turns in computer-mediated settings. Furthermore, our study demonstrates that it is possible to use naturalistic conversation and still obtain controlled results

Ptychographic X-ray computed tomography of extended colloidal networks in food emulsions

As a main structural level in colloidal food materials, extended colloidal networks are important for texture and rheology. By obtaining the 3D microstructure of the network, macroscopic mechanical properties of the material can be inferred. However, this approach is hampered by the lack of suitable non-destructive 3D imaging techniques with submicron resolution. We present results of quantitative ptychographic X-ray computed tomography applied to a palm kernel oil based oil-in-water emulsion. The measurements were carried out at ambient pressure and temperature. The 3D structure of the extended colloidal network of fat globules was obtained with a resolution of around 300 nm. Through image analysis of the network structure, the fat globule size distribution was computed and compared to previous findings. In further support, the reconstructed electron density values were within 4% of reference values.Comment: 19 pages, 4 figures, to be published in Food Structur

Forced Fox-P3 expression can improve the safety and antigen-specific function of engineered regulatory T cells

Regulatory T cells (Treg) are potent inhibitors of autoreactive T cells. The intracellular transcription factor FoxP3 controls the expression levels of a diverse set of genes and plays a critical role in programming functional Tregs. Although, antigen-specific Tregs are more potent than polyclonal Tregs in treating ongoing autoimmunity, phenotype plasticity associated with loss of FoxP3 expression in Tregs can lead to the conversion into antigen-specific effector T cells which might exacerbate autoimmune pathology. In this study, we designed a retroviral vector driving the expression of FoxP3 and a human HLA-DR-restricted TCR from the same promoter. Transduction of purified human Tregs revealed that all TCR-positive cells had elevated levels of FoxP3 expression, increased CD25 and CTLA4 expression and potent suppressive function. Elevated FoxP3 expression did not impair the in vitro expansion of engineered Tregs. Adoptive transfer into HLA-DR transgenic mice revealed that FoxP3+TCR engineered Tregs showed long-term persistence with stable FoxP3 and TCR expression. In contrast, adoptive transfer of Tregs engineered with TCR only resulted in the accumulation of TCR-positive, FoxP3-negative T cells which displayed antigen-specific effector function when stimulated with the TCR-recognised peptides. Our data indicate that forced expression of FoxP3 can prevent accumulation of antigen-specific effector T cells without impairing the engraftment and persistence of engineered Tregs

Life history strategy and intelligence:Commonality and personality profile differences

Previous work on individual and group differences in life history (LH) strategy posited a central role for intelligence. Yet, empirical results failed to support the hypothesized positive association between a slow LH strategy and intelligence. The current investigation (N = 102) represents an attempt to not only re-examine the LH/intelligence hypothesis, but also to conduct an in-depth examination on how LH strategy and intelligence are expressed in personality profiles. The California Adult Q-sort measure of slow LH strategy exhibited a significant positive correlation with performance (r = 0.32), verbal (r = 0.34), and full (r = 0.38) IQ test scores. Additional findings suggest that a slow LH strategy and intelligence both include personality characteristics reflecting ambition and, possibly, social perceptiveness. Alternatively, intelligence is more closely aligned with a personality profile including intellectual ability, independence, and creativity while LH strategy was uniquely associated with interpersonal warmth, conformity, and reticence.</p

The influence of gaze direction on the comprehension of speech and gesture in triadic communication

Human face-to-face communication is a multi-modal activity. Recent research has shown that, during comprehension, recipients integrate information from speech with that contained in co-speech gestures (e.g., Kelly et al., 2010). The current studies take this research one step further by investigating the influence of another modality, namely eye gaze, on speech and gesture comprehension, to advance our understanding of language processing in more situated contexts. In spite of the large body of literature on processing of eye gaze, very few studies have investigated its processing in the context of communication (but see, e.g., Staudte & Crocker, 2011 for an exception). In two studies we simulated a triadic communication context in which a speaker alternated their gaze between our participant and another (alleged) participant. Participants thus viewed speech-only or speech + gesture utterances either in the role of addressee (direct gaze) or in the role of unaddressed recipient (averted gaze). In Study 1, participants (N = 32) viewed video-clips of a speaker producing speech-only (e.g. “she trained the horse”) or speech+gesture utterances conveying complementary information (e.g. “she trained the horse”+WHIPPING gesture). Participants were asked to judge whether a word displayed on screen after each video-clip matched what the speaker said or not. In half of the cases, the word matched a previously uttered word, requiring a “yes” answer. In all other cases, the word matched the meaning of the gesture the actor had performed, thus requiring a ‘no’ answer

Proposta de organização e visualização de grande quantidade de imagens orbitais através de web services.

The organization of satellite imagery seeks to facilitate the presentation and manipulation of spatial information in projects of Land Management's Strategic area from Embrapa Satellite Monitoring, enabling integration between mapped data and interoperability of image data and mapped regions. As a way to standardize this organization, we followed the recommendations of national and international bodies that deal with the standardization of geographic metadata, specifically satellite images. To view this integration, a webmapping interface was created. A webmapping must be prepared to meet the needs of relationship between user and product. In this paper we propose a standardized structure for storage and maintenance of images obtained by satellite sensors aiming its availability on the web through web map services, examples of interoperability. The non-interoperability prevents sharing of information and computing resources. The visualization of web map services takes place through a webmapping that benefits from the organization structure of satellite images proposed in this paper. The hierarchical organization proposed research assistance of satellite images. As they are entered into the database, now new images will be organized in a logical structure, without affecting the search for listings older images. The webmapping allows a direct link between the metadata and geospatial data visualization. Combining web services based on Geographic Information Systems (GIS) in conjunction with the Flex technology allows the creation of Rich Internet Applications (RIA) giving the user more possibilities of interaction via the Web