On the role of domain ontologies in the design of domain-specific visual modeling langages

Domain-Specific Visual Modeling Languages should provide notations and abstractions that suitably support problem solving in well-defined application domains. From their user’s perspective, the language’s modeling primitives must be intuitive and expressive enough in capturing all intended aspects of domain conceptualizations. Over the years formal and explicit representations of domain conceptualizations have been developed as domain ontologies. In this paper, we show how the design of these languages can benefit from conceptual tools developed by the ontology engineering community

Context-aware Captions from Context-agnostic Supervision

We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of "siamese cat" and "tiger cat", we generate language that describes the "siamese cat" in a way that distinguishes it from "tiger cat". Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination.Comment: Accepted to CVPR 2017 (Spotlight

Image-based Text Classification using 2D Convolutional Neural Networks

We propose a new approach to text classification in which we consider the input text as an image and apply 2D Convolutional Neural Networks to learn the local and global semantics of the sentences from the variations of the visual patterns of words. Our approach demonstrates that it is possible to get semantically meaningful features from images with text without using optical character recognition and sequential processing pipelines, techniques that traditional natural language processing algorithms require. To validate our approach, we present results for two applications: text classification and dialog modeling. Using a 2D Convolutional Neural Network, we were able to outperform the state-ofart accuracy results for a Chinese text classification task and achieved promising results for seven English text classification tasks. Furthermore, our approach outperformed the memory networks without match types when using out of vocabulary entities from Task 4 of the bAbI dialog dataset

Reasoning About Pragmatics with Neural Listeners and Speakers

We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics. Like previous learned approaches to language generation, our model uses a simple feature-driven architecture (here a pair of neural "listener" and "speaker" models) to ground language in the world. Like inference-driven approaches to pragmatics, our model actively reasons about listener behavior when selecting utterances. For training, our approach requires only ordinary captions, annotated _without_ demonstration of the pragmatic behavior the model ultimately exhibits. In human evaluations on a referring expression game, our approach succeeds 81% of the time, compared to a 69% success rate using existing techniques

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, user-facing artificial intelligence systems will require similar skills in pragmatics: relying on various types of context -- from shared linguistic goals and conventions, to the visual and embodied world -- to use language effectively. We survey existing grounded settings and pragmatic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. We present recommendations for future grounded task design to naturally elicit pragmatic phenomena, and suggest directions that focus on a broader range of communicative contexts and affordances.Comment: Findings of EMNLP 202

Modeling Business Models: A cross-disciplinary Analysis of Business Model Modeling Languages and Directions for Future Research

Modeling languages for business models are a powerful and flexible means of representing and communicating knowledge related to business models. More than fifteen years after Osterwalder et al. (2005) clarified the ontology for the business model concept in this journal, we offer a systematic and cross-disciplinary assessment of the literature on business model modeling languages (BMMLs) that facilitate the visualization of this concept. In so doing, we synthesize and organize the knowledge dispersed across different disciplines in which BMMLs have originated and highlight the potential weaknesses in this literature to offer solid insights for future research. Our analysis reveals the existence of 17 BMMLs that have originated in traditional domains such as strategy and information systems, but also emerging domains such as sustainability. We contrast and compare these BMMLs along three dimensions: semantics, syntax, and pragmatics. We also analyze research that has made use of these BMMLs, differentiating between research that is conducted with a given BMML and research that is conducted about a given BMML. We conclude by offering a research agenda in which we illustrate the main challenges associated with the lack of well-accepted semantic, syntactic, and pragmatic foundations of BMMLs and outline opportunities for future research

Teaching Language to Students with Autism

This meta-synthesis of the literature on methods of instruction to students with ASD examines the various methods of teaching language to students with ASD. While each student learns language at his or her own pace, the author has found that certain methods yield results quicker, and these methods need to be examined critically for any literature on their reliability, efficacy, and scientific research. If a student with autism can be taught language quickly, therefore mitigating any further delays in academic development relative to peers, then this methodology should be made accessible to all teachers of such students

Clue: Cross-modal Coherence Modeling for Caption Generation

We use coherence relations inspired by computational models of discourse to study the information needs and goals of image captioning. Using an annotation protocol specifically devised for capturing image--caption coherence relations, we annotate 10,000 instances from publicly-available image--caption pairs. We introduce a new task for learning inferences in imagery and text, coherence relation prediction, and show that these coherence annotations can be exploited to learn relation classifiers as an intermediary step, and also train coherence-aware, controllable image captioning models. The results show a dramatic improvement in the consistency and quality of the generated captions with respect to information needs specified via coherence relations.Comment: Accepted as a long paper to ACL 202