Multimodal language processing in human communication

Multiple layers of visual (and vocal) signals, plus their different onsets and offsets, represent a significant semantic and temporal binding problem during face-to-face conversation. Despite this complex unification process, multimodal messages appear to be processed faster than unimodal messages. Multimodal gestalt recognition and multilevel prediction are proposed to play a crucial role in facilitating multimodal language processing. The basis of the processing mechanisms involved in multimodal language comprehension is hypothesized to be domain general, coopted for communication, and refined with domain-specific characteristics. A new, situated framework for understanding human language processing is called for that takes into consideration the multilayered, multimodal nature of language and its production and comprehension in conversational interaction requiring fast processing

Visual information in computer-mediated interaction matters: Investigating the association between the availability of gesture and turn transition timing in conversation

Natural human interaction involves the fast-paced exchange of speaker turns. Crucially, if a next speaker waited with planning their turn until the current speaker was finished, language production models would predict much longer turn transition times than what we observe. Next speakers must therefore prepare their turn in parallel to listening. Visual signals likely play a role in this process, for example by helping the next speaker to process the ongoing utterance and thus prepare an appropriately-timed response. To understand how visual signals contribute to the timing of turn-taking, and to move beyond the mostly qualitative studies of gesture in conversation, we examined unconstrained, computer-mediated conversations between 20 pairs of participants while systematically manipulating speaker visibility. Using motion tracking and manual gesture annotation, we assessed 1) how visibility affected the timing of turn transitions, and 2) whether use of co-speech gestures and 3) the communicative kinematic features of these gestures were associated with changes in turn transition timing. We found that 1) decreased visibility was associated with less tightly timed turn transitions, and 2) the presence of gestures was associated with more tightly timed turn transitions across visibility conditions. Finally, 3) structural and salient kinematics contributed to gesture’s facilitatory effect on turn transition times. Our findings suggest that speaker visibility--and especially the presence and kinematic form of gestures--during conversation contributes to the temporal coordination of conversational turns in computer-mediated settings. Furthermore, our study demonstrates that it is possible to use naturalistic conversation and still obtain controlled results

Eyebrow movements as signals of communicative problems in human face-to-face interaction

Repair is a core building block of human communication, allowing us to address problems of understanding in conversation. Past research has uncovered the basic mechanisms by which interactants signal and solve such problems. However, the focus has been on verbal interaction, neglecting the fact that human communication is inherently multimodal. Here, we focus on a visual signal particularly prevalent in signaling problems of understanding: eyebrow frowns and raises. We present a corpus study showing that verbal repair initiations with eyebrow furrows are more likely to be responded to with clarifications as repair solutions, repair initiations that were preceded by eyebrow actions as preliminaries get repaired faster (around 230 ms), and eyebrow furrows alone can be sufficient to occasion clarification. We also present an experiment based on virtual reality technology, revealing that addressees’ eyebrow frowns have a striking effect on speakers’ speech, leading them to produce answers to questions several seconds longer than when not perceiving addressee eyebrow furrows. Together, the findings demonstrate that eyebrow movements play a communicative role in initiating repair in spoken language rather than being merely epiphenomenal. Thus, they should be considered as core coordination devices in human conversational interaction

The Inverse Shapley Value Problem

For $f$ a weighted voting scheme used by $n$ voters to choose between two candidates, the $n$ \emph{Shapley-Shubik Indices} (or {\em Shapley values}) of $f$ provide a measure of how much control each voter can exert over the overall outcome of the vote. Shapley-Shubik indices were introduced by Lloyd Shapley and Martin Shubik in 1954 \cite{SS54} and are widely studied in social choice theory as a measure of the "influence" of voters. The \emph{Inverse Shapley Value Problem} is the problem of designing a weighted voting scheme which (approximately) achieves a desired input vector of values for the Shapley-Shubik indices. Despite much interest in this problem no provably correct and efficient algorithm was known prior to our work. We give the first efficient algorithm with provable performance guarantees for the Inverse Shapley Value Problem. For any constant \eps > 0 our algorithm runs in fixed poly$(n)$ time (the degree of the polynomial is independent of \eps) and has the following performance guarantee: given as input a vector of desired Shapley values, if any "reasonable" weighted voting scheme (roughly, one in which the threshold is not too skewed) approximately matches the desired vector of values to within some small error, then our algorithm explicitly outputs a weighted voting scheme that achieves this vector of Shapley values to within error \eps. If there is a "reasonable" voting scheme in which all voting weights are integers at most \poly(n) that approximately achieves the desired Shapley values, then our algorithm runs in time \poly(n) and outputs a weighted voting scheme that achieves the target vector of Shapley values to within error $\eps=n^{-1/8}.

Forced Fox-P3 expression can improve the safety and antigen-specific function of engineered regulatory T cells

Regulatory T cells (Treg) are potent inhibitors of autoreactive T cells. The intracellular transcription factor FoxP3 controls the expression levels of a diverse set of genes and plays a critical role in programming functional Tregs. Although, antigen-specific Tregs are more potent than polyclonal Tregs in treating ongoing autoimmunity, phenotype plasticity associated with loss of FoxP3 expression in Tregs can lead to the conversion into antigen-specific effector T cells which might exacerbate autoimmune pathology. In this study, we designed a retroviral vector driving the expression of FoxP3 and a human HLA-DR-restricted TCR from the same promoter. Transduction of purified human Tregs revealed that all TCR-positive cells had elevated levels of FoxP3 expression, increased CD25 and CTLA4 expression and potent suppressive function. Elevated FoxP3 expression did not impair the in vitro expansion of engineered Tregs. Adoptive transfer into HLA-DR transgenic mice revealed that FoxP3+TCR engineered Tregs showed long-term persistence with stable FoxP3 and TCR expression. In contrast, adoptive transfer of Tregs engineered with TCR only resulted in the accumulation of TCR-positive, FoxP3-negative T cells which displayed antigen-specific effector function when stimulated with the TCR-recognised peptides. Our data indicate that forced expression of FoxP3 can prevent accumulation of antigen-specific effector T cells without impairing the engraftment and persistence of engineered Tregs

Ptychographic X-ray computed tomography of extended colloidal networks in food emulsions

As a main structural level in colloidal food materials, extended colloidal networks are important for texture and rheology. By obtaining the 3D microstructure of the network, macroscopic mechanical properties of the material can be inferred. However, this approach is hampered by the lack of suitable non-destructive 3D imaging techniques with submicron resolution. We present results of quantitative ptychographic X-ray computed tomography applied to a palm kernel oil based oil-in-water emulsion. The measurements were carried out at ambient pressure and temperature. The 3D structure of the extended colloidal network of fat globules was obtained with a resolution of around 300 nm. Through image analysis of the network structure, the fat globule size distribution was computed and compared to previous findings. In further support, the reconstructed electron density values were within 4% of reference values.Comment: 19 pages, 4 figures, to be published in Food Structur

The influence of gaze direction on the comprehension of speech and gesture in triadic communication

Human face-to-face communication is a multi-modal activity. Recent research has shown that, during comprehension, recipients integrate information from speech with that contained in co-speech gestures (e.g., Kelly et al., 2010). The current studies take this research one step further by investigating the influence of another modality, namely eye gaze, on speech and gesture comprehension, to advance our understanding of language processing in more situated contexts. In spite of the large body of literature on processing of eye gaze, very few studies have investigated its processing in the context of communication (but see, e.g., Staudte & Crocker, 2011 for an exception). In two studies we simulated a triadic communication context in which a speaker alternated their gaze between our participant and another (alleged) participant. Participants thus viewed speech-only or speech + gesture utterances either in the role of addressee (direct gaze) or in the role of unaddressed recipient (averted gaze). In Study 1, participants (N = 32) viewed video-clips of a speaker producing speech-only (e.g. “she trained the horse”) or speech+gesture utterances conveying complementary information (e.g. “she trained the horse”+WHIPPING gesture). Participants were asked to judge whether a word displayed on screen after each video-clip matched what the speaker said or not. In half of the cases, the word matched a previously uttered word, requiring a “yes” answer. In all other cases, the word matched the meaning of the gesture the actor had performed, thus requiring a ‘no’ answer

Protective effect of ultrathin alumina film against diffusion of iron into carbon fiber during growth of carbon nanotubes for hierarchical composites investigated by ptychographic X-ray computed tomography

Composite materials based on carbon fiber (CF) are prone to failure at the fiber-matrix interface upon compression or stress transverse to the fiber axis. The direct growth of carbon nanotubes on CF constitutes a novel approach to enhance the mechanical properties of the interface. However, the challenge is that, during the growth, tensile properties of the fiber are altered due to the diffusion effect of iron nanoparticles used in the process, leading to CF surface defect formation. In this work, we deliver and discuss an analysis methodology on ptychographic X-ray computed tomography (PXCT) images in order to assess the iron nanoparticle abundance within CFs. PXCT provides 50 nm - resolved 3D electron density maps of the CFs. We evidence the protective effect of an ultrathin alumina film against iron infiltration into CF during the CNT growth. This method potentially allows to evaluate the efficiency of other diffusion-minimizing approaches. The conclusions of the PXCT examination are validated by energy-dispersive X-ray spectroscopy and scanning transmission electron microscopy carried out on thin sample slices cut with a focused ion beam. The results provide a new insight into the mechanical performance of CFs and therefore constitute valuable knowledge for the development of hierarchical composites

Splenic pooling and loss of VCAM-1 causes an engraftment defect in patients with myelofibrosis after allogeneic hematopoietic stem cell transplantation

Myelofibrosis is a myeloproliferative neoplasm that results in cytopenia, bone marrow fibrosis and extramedullary hematopoiesis. Allogeneic hematopoietic stem cell transplantation is the only curative treatment but is associated with a risk of delayed engraftment and graft failure. In this study, patients with myelofibrosis (n=31) and acute myeloid leukemia (n=31) were analyzed for time to engraftment, graft failure and engraftment-related factors. Early and late neutrophil engraftment and late thrombocyte engraftment were significantly delayed in patients with myelofibrosis as compared to acute myeloid leukemia, and graft failure only occurred in myelofibrosis (6%). Only spleen size had a significant influence on engraftment efficiency in myelofibrosis patients. To analyze the cause for the engraftment defect, clearance of hematopoietic stem cells from peripheral blood was measured and immunohistological staining of bone marrow sections was performed. Numbers of circulating CD34+ were significantly reduced at early time points in myelofibrosis patients, whereas CD34+CD38- and colony-forming cells showed no significant difference in clearance. Staining of bone marrow sections for homing proteins revealed a loss of VCAM-1 in myelofibrosis with a corresponding significant increase in the level of soluble VCAM-1 within the peripheral blood. In conclusion, our data suggest that reduced engraftment and graft failure in myelofibrosis patients is caused by an early pooling of CD34+ hematopoietic stem cells in the spleen and a bone marrow homing defect caused by the loss of VCAM-1. Improved engraftment in myelofibrosis might be achieved by approaches that reduce spleen size and cleavage of VCAM-1 in these patients prior to hematopoietic stem cell transplantation