BUILDING EFFECTIVE AND SCALABLE VISUAL OBJECT RECOGNITION SYSTEMS

Ph.DDOCTOR OF PHILOSOPH

Beyond the lens : communicating context through sensing, video, and visualization

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 101-103).Responding to rapid growth in sensor network deployments that outpaces research efforts to understand or relate the new data streams, this thesis presents a collection of interfaces to sensor network data that encourage open-ended browsing while emphasizing saliency of representation. These interfaces interpret, visualize, and communicate context from sensors, through control panels and virtual environments that synthesize multimodal sensor data into interactive visualizations. This work extends previous efforts in cross-reality to incorporate augmented video as well as complex interactive animations, making use of sensor fusion to saliently represent contextual information to users in a variety of application domains, from building information management to real-time risk assessment to personal privacy. Three applications were developed as part of this work and are discussed here: DoppelLab, an immersive, cross-reality browsing environment for sensor network data; Flurry, an installation that composites video from multiple sources throughout a building in real time, to create an interactive and incorporative view of activity; and Tracking Risk with Ubiquitous Smart Sensing (TRUSS), an ongoing research effort aimed at applying real-time sensing, sensor fusion, and interactive visual analytic interfaces to construction site safety and decision support. Another project in active development, called the Disappearing Act, allows users to remove themselves from a set of live video streams using wearable sensor tags. Though these examples may seem disconnected, they share underlying technologies and research developments, as well as a common set of design principles, which are elucidated in this thesis. Building on developments in sensor networks, computer vision, and graphics, this work aims to create interfaces and visualizations that fuse perspectives, broaden contextual understanding, and encourage exploration of real-time sensor network data.by Gershon Dublon.S.M

Effects of Logic-Style Explanations and Uncertainty on Users’ Decisions

The spread of innovative Artificial Intelligence (AI) algorithms assists many individuals in their daily life decision-making tasks but also sensitive domains such as disease diagnosis and credit risk. However, a great majority of these algorithms are of a black-box nature, bringing the need to make them more transparent and interpretable along with the establishment of guidelines to help users manage these systems. The eXplainable Artificial Intelligence (XAI) community investigated numerous factors influencing subjective and objective metrics in the user-AI team, such as the effects of presenting AI-related information and explanations to users. Nevertheless, some factors that influence the effectiveness of explanations are still under-explored in the literature, such as user uncertainty, AI uncertainty, AI correctness, and different explanation styles. The main goal of this thesis is to investigate the interactions between different aspects of decision-making, focusing in particular on the effects of AI and user uncertainty, AI correctness, and the explanation reasoning style (inductive, abductive, and deductive) on different data types and domains considering classification tasks. We set up three user evaluations on images, text, and time series data to analyse these factors on users' task performance, agreement with the AI suggestion, and the user’s reliance on the XAI interface elements (instance, AI prediction, and explanation). The results for the image and text data show that user uncertainty and AI correctness on predictions significantly affected users’ classification decisions considering the analysed metrics. In both domains (images and text), users relied mainly on the instance to decide. Users were usually overconfident about their choices, and this evidence was more pronounced for text. Furthermore, the inductive style explanations led to over-reliance on AI advice in both domains – it was the most persuasive, even when the AI was incorrect. The abductive and deductive styles have complex effects depending on the domain and the AI uncertainty levels. Instead, the time series data results show that specific explanation styles (abductive and deductive) improve the user’s task performance in the case of high AI confidence compared to inductive explanations. In other words, these styles of explanations were able to invoke correct decisions (for both positive and negative decisions) when the system was certain. In such a condition, the agreement between the user’s decision and the AI prediction confirms this finding, highlighting a significant agreement increase when the AI is correct. This suggests that both explanation styles are suitable for evoking appropriate trust in a confident AI. The last part of the thesis focuses on the work done with the \enquote{CRS4 - Centro di Ricerca, Sviluppo e Studi Superiori in Sardegna}, for the implementation of the RIALE (Remote Intelligent Access to Lab Experiment) Platform. The work aims to help students explore a DNA-sequences experiment enriched with an AI tagging tool, which detects the objects used in the laboratory and its current phase. Further, the interface includes an interactive timeline which enables students to explore the AI predictions of the video experiment's steps and an XAI panel that provides explanations of the AI decisions - presented with abductive reasoning - on three levels (globally, by phase, and by frame). We evaluated the interface with students considering the subjective cognitive effort, ease of use, supporting information of the interface, general usability, and an interview on a set of questions on peculiar aspects of the application. The user evaluation results showed that students were positively satisfied with the interface and in favour of following didactic lessons using this tool

Dimensions of Access to Traceability Information for US Beef Cattle Producers: Merging Information Frameworks for Assessment and Visualization of State Web-Based Resources in an Effort to Strengthen National Security Connections between Government and Cattle Farming Operations

US consumers eat a lot of beef. The nation’s beef cattle production industry is a multi-faceted, complex supply chain which makes it an area rich for discussion about information practices, yet vulnerable to problems such as disease and terrorist attack. This research looks at cattle identification and traceability information resources that are accessible to beef cattle producers through two web channels: the state cooperative Extension website and the state Department of Agriculture website. This is a state by state content analysis of all fifty states to look at the topics, types, formats, quality, and interactivity of the available resources. By merging two information frameworks, one with theoretical attention to components of access to information and one with applied attention to government information valuation measures, the research demonstrates an analysis process that connects state cattle producer demographics for comparison with aspects of the available cattle identification and traceability information from that state. This includes visualizing the nation as a whole and comparing state-based similarities and differences, illuminating areas of strengths, weaknesses, and gaps in contextually congruent information for the producer and stakeholder populations

Semantics-driven Abstractive Document Summarization

The evolution of the Web over the last three decades has led to a deluge of scientific and news articles on the Internet. Harnessing these publications in different fields of study is critical to effective end user information consumption. Similarly, in the domain of healthcare, one of the key challenges with the adoption of Electronic Health Records (EHRs) for clinical practice has been the tremendous amount of clinical notes generated that can be summarized without which clinical decision making and communication will be inefficient and costly. In spite of the rapid advances in information retrieval and deep learning techniques towards abstractive document summarization, the results of these efforts continue to resemble extractive summaries, achieving promising results predominantly on lexical metrics but performing poorly on semantic metrics. Thus, abstractive summarization that is driven by intrinsic and extrinsic semantics of documents is not adequately explored. Resources that can be used for generating semantics-driven abstractive summaries include: • Abstracts of multiple scientific articles published in a given technical field of study to generate an abstractive summary for topically-related abstracts within the field, thus reducing the load of having to read semantically duplicate abstracts on a given topic. • Citation contexts from different authoritative papers citing a reference paper can be used to generate utility-oriented abstractive summary for a scientific article. • Biomedical articles and the named entities characterizing the biomedical articles along with background knowledge bases to generate entity and fact-aware abstractive summaries. • Clinical notes of patients and clinical knowledge bases for abstractive clinical text summarization using knowledge-driven multi-objective optimization. In this dissertation, we develop semantics-driven abstractive models based on intra- document and inter-document semantic analyses along with facts of named entities retrieved from domain-specific knowledge bases to produce summaries. Concretely, we propose a sequence of frameworks leveraging semantics at various granularity (e.g., word, sentence, document, topic, citations, and named entities) levels, by utilizing external resources. The proposed frameworks have been applied to a range of tasks including 1. Abstractive summarization of topic-centric multi-document scientific articles and news articles. 2. Abstractive summarization of scientific articles using crowd-sourced citation contexts. 3. Abstractive summarization of biomedical articles clustered based on entity-relatedness. 4. Abstractive summarization of clinical notes of patients with heart failure and Chest X-Rays recordings. The proposed approaches achieve impressive performance in terms of preserving semantics in abstractive summarization while paraphrasing. For summarization of topic-centric multiple scientific/news articles, we propose a three-stage approach where abstracts of scientific articles or news articles are clustered based on their topical similarity determined from topics generated using Latent Dirichlet Allocation (LDA), followed by extractive phase and abstractive phase. Then, in the next stage, we focus on abstractive summarization of biomedical literature where we leverage named entities in biomedical articles to 1) cluster related articles; and 2) leverage the named entities towards guiding abstractive summarization. Finally, in the last stage, we turn to external resources such as citation contexts pointing to a scientific article to generate a comprehensive and utility-centric abstractive summary of a scientific article, domain-specific knowledge bases to fill gaps in information about entities in a biomedical article to summarize and clinical notes to guide abstractive summarization of clinical text. Thus, the bottom-up progression of exploring semantics towards abstractive summarization in this dissertation starts with (i) Semantic Analysis of Latent Topics; builds on (ii) Internal and External Knowledge-I (gleaned from abstracts and Citation Contexts); and extends it to make it comprehensive using (iii) Internal and External Knowledge-II (Named Entities and Knowledge Bases)

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

Welcome to EVALITA 2020! EVALITA is the evaluation campaign of Natural Language Processing and Speech Tools for Italian. EVALITA is an initiative of the Italian Association for Computational Linguistics (AILC, http://www.ai-lc.it) and it is endorsed by the Italian Association for Artificial Intelligence (AIxIA, http://www.aixia.it) and the Italian Association for Speech Sciences (AISV, http://www.aisv.it)

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

Welcome to EVALITA 2020! EVALITA is the evaluation campaign of Natural Language Processing and Speech Tools for Italian. EVALITA is an initiative of the Italian Association for Computational Linguistics (AILC, http://www.ai-lc.it) and it is endorsed by the Italian Association for Artificial Intelligence (AIxIA, http://www.aixia.it) and the Italian Association for Speech Sciences (AISV, http://www.aisv.it)