Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature

LTC Newsletter

Spring 2010https://ecommons.udayton.edu/ltc_newsletter/1012/thumbnail.jp

Vereinheitlichte Anfrageverarbeitung in heterogenen und verteilten Multimediadatenbanken

Multimedia retrieval is an essential part of today's world. This situation is observable in industrial domains, e.g., medical imaging, as well as in the private sector, visible by activities in manifold Social Media platforms. This trend led to the creation of a huge environment of multimedia information retrieval services offering multimedia resources for almost any user requests. Indeed, the encompassed data is in general retrievable by (proprietary) APIs and query languages, but unfortunately a unified access is not given due to arising interoperability issues between those services. In this regard, this thesis focuses on two application scenarios, namely a medical retrieval system supporting a radiologist's workflow, as well as an interoperable image retrieval service interconnecting diverse data silos. The scientific contribution of this dissertation is split in three different parts: the first part of this thesis improves the metadata interoperability issue. Here, major contributions to a community-driven, international standardization have been proposed leading to the specification of an API and ontology to enable a unified annotation and retrieval of media resources. The second part issues a metasearch engine especially designed for unified retrieval in distributed and heterogeneous multimedia retrieval environments. This metasearch engine is capable of being operated in a federated as well as autonomous manner inside the aforementioned application scenarios. The remaining third part ensures an efficient retrieval due to the integration of optimization techniques for multimedia retrieval in the overall query execution process of the metasearch engine.Egal ob im industriellen Bereich oder auch im Social Media - multimediale Daten nehmen eine immer zentralere Rolle ein. Aus diesem fortlaufendem Entwicklungsprozess entwickelten sich umfangreiche Informationssysteme, die Daten für zahlreiche Bedürfnisse anbieten. Allerdings ist ein einheitlicher Zugriff auf jene verteilte und heterogene Landschaft von Informationssystemen in der Praxis nicht gewährleistet. Und dies, obwohl die Datenbestände meist über Schnittstellen abrufbar sind. Im Detail widmet sich diese Arbeit mit der Bearbeitung zweier Anwendungsszenarien. Erstens, einem medizinischen System zur Diagnoseunterstützung und zweitens einer interoperablen, verteilten Bildersuche. Der wissenschaftliche Teil der vorliegenden Dissertation gliedert sich in drei Teile: Teil eins befasst sich mit dem Problem der Interoperabilität zwischen verschiedenen Metadatenformaten. In diesem Bereich wurden maßgebliche Beiträge für ein internationales Standardisierungsverfahren entwickelt. Ziel war es, einer Ontologie, sowie einer Programmierschnittstelle einen vereinheitlichten Zugriff auf multimediale Informationen zu ermöglichen. In Teil zwei wird eine externe Metasuchmaschine vorgestellt, die eine einheitliche Anfrageverarbeitung in heterogenen und verteilten Multimediadatenbanken ermöglicht. In den Anwendungsszenarien wird zum einen auf eine föderative, als auch autonome Anfrageverarbeitung eingegangen. Abschließend werden in Teil drei Techniken zur Optimierung von verteilten multimedialen Anfragen präsentiert

Characterizing and Evaluating Users' Information Seeking Behavior in Social Tagging Systems

Social tagging systems in the Web 2.0 era present an innovative information seeking environment succeeding the library and traditional Web. The primary goals of this study were to, in this particular context: (1) identify the general information seeking strategies adopted by users and determine their effectiveness; (2) reveals the characteristics of the users who prefer different strategies; and (3) identify the specific traits of users' information seeking paths and understand factors shaping them. A representative social tagging system, Douban (http://www.douban.com/) was chosen as the research setting in order to generate empirical findings.Based on the mixed methods research design, this study consists of a quantitative phase and a qualitative phase. The former firstly involved a clickstream data analysis of 20 million clickstream records requested from Douban at the footprint, movement, and track levels. Limited to studying physical behavior, it was complemented by an online survey which captured Douban users' background information from various aspects. In the subsequent qualitative phase, a focus group gathered a number of experienced Douban users to help interpret the quantitative results.Major findings of this study show that: (1) the general strategies include encountering, browsing by resource, browsing by tag, browsing by user/group, searching, and monitoring by user/group; (2) while browsing by resource is the most popular strategy, browsing by tag is the most effective one; (3) users preferring different strategies do not have significantly different characteristics; and (4) on users' information seeking paths these exist two resource viewing patterns - continuous and sporadic, and two resource collecting patterns - lagged and instant, and they can be attributed to user, task, and system factors.A model was developed to illustrate the strategic and tactic layers of users' information seeking behavior in social tagging systems. It offers a deep insight into the behavioral changes brought about by this new environment as compared to the Web in general. This model can serve as the theoretical base for designing user-oriented information seeking interfaces for social tagging systems so that the general strategies and specific tactics will be accommodated efficiently