ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition

Situation Recognition is the task of generating a structured summary of what is happening in an image using an activity verb and the semantic roles played by actors and objects. In this task, the same activity verb can describe a diverse set of situations as well as the same actor or object category can play a diverse set of semantic roles depending on the situation depicted in the image. Hence a situation recognition model needs to understand the context of the image and the visual-linguistic meaning of semantic roles. Therefore, we leverage the CLIP foundational model that has learned the context of images via language descriptions. We show that deeper-and-wider multi-layer perceptron (MLP) blocks obtain noteworthy results for the situation recognition task by using CLIP image and text embedding features and it even outperforms the state-of-the-art CoFormer, a Transformer-based model, thanks to the external implicit visual-linguistic knowledge encapsulated by CLIP and the expressive power of modern MLP block designs. Motivated by this, we design a cross-attention-based Transformer using CLIP visual tokens that model the relation between textual roles and visual entities. Our cross-attention-based Transformer known as ClipSitu XTF outperforms existing state-of-the-art by a large margin of 14.1\% on semantic role labelling (value) for top-1 accuracy using imSitu dataset. {Similarly, our ClipSitu XTF obtains state-of-the-art situation localization performance.} We will make the code publicly available.Comment: State-of-the-art results on Grounded Situation Recognitio

Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Emotional Justification

Theories of emotional justification investigate the conditions under which emotions are epistemically justified or unjustified. I make three contributions to this research program. First, I show that we can generalize some familiar epistemological concepts and distinctions to emotional experiences. Second, I use these concepts and distinctions to display the limits of the ‘simple view’ of emotional justification. On this approach, the justification of emotions stems only from the contents of the mental states they are based on, also known as their cognitive bases. The simple view faces the ‘gap problem’: If cognitive bases and emotions (re)present their objects and properties in different ways, then cognitive bases are not sufficient to justify emotions. Third, I offer a novel solution to the gap problem based on emotional dispositions. This solution (1) draws a line between the justification of basic and non-basic emotions, (2) preserves a broadly cognitivist view of emotions, (3) avoids a form of value skepticism that threatens inferentialist views of emotional justification, and (4) sheds new light on the structure of our epistemic access to evaluative properties

A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects of human-robot interaction. Following a historical introduction, and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of recent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminating to a unifying discussion, and a forward-looking conclusion

From Monologue to Dialogue: Natural Language Generation in OVIS

This paper describes how a language generation system that was originally designed for monologue generation, has been adapted for use in the OVIS spoken dialogue system. To meet the requirement that in a dialogue, the system's utterances should make up a single, coherent dialogue turn, several modifications had to be made to the system. The paper also discusses the influence of dialogue context on information status, and its consequences for the generation of referring expressions and accentuation

Ultimate-Grounding Under the Condition of Finite Knowledge. A Hegelian Perspective

Hegel's Science of Logic makes the just not low claim to be an absolute, ultimate-grounded knowledge. This project, which could not be more ambitious, has no good press in our post-metaphysical age. However: That absolute knowledge absolutely cannot exist, cannot be claimed without self-contradiction. On the other hand, there can be no doubt about the fundamental finiteness of knowledge. But can absolute knowledge be finite knowledge? This leads to the problem of a self-explication of logic (in the sense of Hegel) and further, as will be shown, to a new definition of the dialectical procedure. The stringency of which results from the fact that always exactly that implicit content is explicated that was generated by the preceding explication step itself and is thus concretely comprehensible. At the same time, a new implicit content is generated by this act of explication, which requires a new explication step, and so forth. In the dialectical procedure reinterpreted in this way, dialectical arguments are not beheld, guessed at or even surreptitiously obtained, but are methodically accountable. Thereby dialectics is understood as a self-explication of logic by logical means and thus as a proof of the possibility of ultimate-grounding in the form of absolute and nevertheless finite – and thus also fallible – knowledge

Contextual Sensitivity in Grounded Theory: The Role of Pilot Studies

Grounded Theory is an established methodological approach for context specific inductive theory building. The grounded nature of the methodology refers to these specific contexts from which emergent propositions are drawn. Thus, any grounded theory study requires not only theoretical sensitivity, but also a good insight on how to design the research in the human activity systems to be studied. The lack of this insight may result in inefficient theoretical sampling or even erroneous purposeful sampling. These problems would not necessarily be critical, as it could be argued that through the elliptical process that characterizes grounded theory, remedial loops would always bring the researcher to the core of the theory. However, these elliptical remedial processes can take very long periods of time and result in catastrophic delays in research projects. As a strategy, this paper discusses, contrasts and compares the use of pilot studies in four different grounded theory projects. Each pilot brought different insights about the context, resulting in changes of focus, guidance to improve data collection instruments and informing theoretical sampling. Additionally, as all four projects were undertaken by researchers with little experience of inductive approaches in general and grounded theory in particular, the pilot studies also served the purpose of training in interviewing, relating to interviewees, memoing, constant comparison and coding. This last outcome of the pilot study was actually not planned initially, but revealed itself to be a crucial success factor in the running of the projects. The paper concludes with a theoretical proposition for the concept of contextual sensitivity and for the inclusion of the pilot study in grounded theory research designs