Context-based multimedia semantics modelling and representation

The evolution of the World Wide Web, increase in processing power, and more network bandwidth have contributed to the proliferation of digital multimedia data. Since multimedia data has become a critical resource in many organisations, there is an increasing need to gain efficient access to data, in order to share, extract knowledge, and ultimately use the knowledge to inform business decisions. Existing methods for multimedia semantic understanding are limited to the computable low-level features; which raises the question of how to identify and represent the high-level semantic knowledge in multimedia resources.In order to bridge the semantic gap between multimedia low-level features and high-level human perception, this thesis seeks to identify the possible contextual dimensions in multimedia resources to help in semantic understanding and organisation. This thesis investigates the use of contextual knowledge to organise and represent the semantics of multimedia data aimed at efficient and effective multimedia content-based semantic retrieval.A mixed methods research approach incorporating both Design Science Research and Formal Methods for investigation and evaluation was adopted. A critical review of current approaches for multimedia semantic retrieval was undertaken and various shortcomings identified. The objectives for a solution were defined which led to the design, development, and formalisation of a context-based model for multimedia semantic understanding and organisation. The model relies on the identification of different contextual dimensions in multimedia resources to aggregate meaning and facilitate semantic representation, knowledge sharing and reuse. A prototype system for multimedia annotation, CONMAN was built to demonstrate aspects of the model and validate the research hypothesis, H₁.Towards providing richer and clearer semantic representation of multimedia content, the original contributions of this thesis to Information Science include: (a) a novel framework and formalised model for organising and representing the semantics of heterogeneous visual data; and (b) a novel S-Space model that is aimed at visual information semantic organisation and discovery, and forms the foundations for automatic video semantic understanding

Interdisciplinarity in the Age of the Triple Helix: a Film Practitioner's Perspective

This integrative chapter contextualises my research including articles I have published as well as one of the creative artefacts developed from it, the feature film The Knife That Killed Me. I review my work considering the ways in which technology, industry methods and academic practice have evolved as well as how attitudes to interdisciplinarity have changed, linking these to Etzkowitz and Leydesdorff’s ‘Triple Helix’ model (1995). I explore my own experiences and observations of opportunities and challenges that have been posed by the intersection of different stakeholder needs and expectations, both from industry and academic perspectives, and argue that my work provides novel examples of the applicability of the ‘Triple Helix’ to the creative industries. The chapter concludes with a reflection on the evolution and direction of my work, the relevance of the ‘Triple Helix’ to creative practice, and ways in which this relationship could be investigated further

Computergestützte Inhaltsanalyse von digitalen Videoarchiven

Der Übergang von analogen zu digitalen Videos hat in den letzten Jahren zu großen Veränderungen innerhalb der Filmarchive geführt. Insbesondere durch die Digitalisierung der Filme ergeben sich neue Möglichkeiten für die Archive. Eine Abnutzung oder Alterung der Filmrollen ist ausgeschlossen, so dass die Qualität unverändert erhalten bleibt. Zudem wird ein netzbasierter und somit deutlich einfacherer Zugriff auf die Videos in den Archiven möglich. Zusätzliche Dienste stehen den Archivaren und Anwendern zur Verfügung, die erweiterte Suchmöglichkeiten bereitstellen und die Navigation bei der Wiedergabe erleichtern. Die Suche innerhalb der Videoarchive erfolgt mit Hilfe von Metadaten, die weitere Informationen über die Videos zur Verfügung stellen. Ein großer Teil der Metadaten wird manuell von Archivaren eingegeben, was mit einem großen Zeitaufwand und hohen Kosten verbunden ist. Durch die computergestützte Analyse eines digitalen Videos ist es möglich, den Aufwand bei der Erzeugung von Metadaten für Videoarchive zu reduzieren. Im ersten Teil dieser Dissertation werden neue Verfahren vorgestellt, um wichtige semantische Inhalte der Videos zu erkennen. Insbesondere werden neu entwickelte Algorithmen zur Erkennung von Schnitten, der Analyse der Kamerabewegung, der Segmentierung und Klassifikation von Objekten, der Texterkennung und der Gesichtserkennung vorgestellt. Die automatisch ermittelten semantischen Informationen sind sehr wertvoll, da sie die Arbeit mit digitalen Videoarchiven erleichtern. Die Informationen unterstützen nicht nur die Suche in den Archiven, sondern führen auch zur Entwicklung neuer Anwendungen, die im zweiten Teil der Dissertation vorgestellt werden. Beispielsweise können computergenerierte Zusammenfassungen von Videos erzeugt oder Videos automatisch an die Eigenschaften eines Abspielgerätes angepasst werden. Ein weiterer Schwerpunkt dieser Dissertation liegt in der Analyse historischer Filme. Vier europäische Filmarchive haben eine große Anzahl historischer Videodokumentationen zur Verfügung gestellt, welche Anfang bis Mitte des letzten Jahrhunderts gedreht und in den letzten Jahren digitalisiert wurden. Durch die Lagerung und Abnutzung der Filmrollen über mehrere Jahrzehnte sind viele Videos stark verrauscht und enthalten deutlich sichtbare Bildfehler. Die Bildqualität der historischen Schwarz-Weiß-Filme unterscheidet sich signifikant von der Qualität aktueller Videos, so dass eine verlässliche Analyse mit bestehenden Verfahren häufig nicht möglich ist. Im Rahmen dieser Dissertation werden neue Algorithmen vorgestellt, um eine zuverlässige Erkennung von semantischen Inhalten auch in historischen Videos zu ermöglichen

Tematski zbornik radova međunarodnog značaja. Tom 3 / Međunarodni naučni skup “Dani Arčibalda Rajsa”, Beograd, 10-11. mart 2016.

In front of you is the Thematic Collection of Papers presented at the International Scientific Conference “Archibald Reiss Days”, which was organized by the Academy of Criminalistic and Police Studies in Belgrade, in co-operation with the Ministry of Interior and the Ministry of Education, Science and Technological Development of the Republic of Serbia, National Police University of China, Lviv State University of Internal Affairs, Volgograd Academy of the Russian Internal Affairs Ministry, Faculty of Security in Skopje, Faculty of Criminal Justice and Security in Ljubljana, Police Academy “Alexandru Ioan Cuza“ in Bucharest, Academy of Police Force in Bratislava and Police College in Banjaluka, and held at the Academy of Criminalistic and Police Studies, on 10 and 11 March 2016. The International Scientific Conference “Archibald Reiss Days” is organized for the sixth time in a row, in memory of the founder and director of the first modern higher police school in Serbia, Rodolphe Archibald Reiss, PhD, after whom the Conference was named. The Thematic Collection of Papers contains 165 papers written by eminent scholars in the field of law, security, criminalistics, police studies, forensics, informatics, as well as by members of national security system participating in education of the police, army and other security services from Belarus, Bosnia and Herzegovina, Bulgaria, China, Croatia, Greece, Hungary, Macedonia, Montenegro, Romania, Russian Federation, Serbia, Slovakia, Slovenia, Spain, Switzerland, Turkey, Ukraine and United Kingdom. Each paper has been double-blind peer reviewed by two reviewers, international experts competent for the field to which the paper is related, and the Thematic Conference Proceedings in whole has been reviewed by five competent international reviewers. The papers published in the Thematic Collection of Papers contain the overview of contemporary trends in the development of police education system, development of the police and contemporary security, criminalistic and forensic concepts. Furthermore, they provide us with the analysis of the rule of law activities in crime suppression, situation and trends in the above-mentioned fields, as well as suggestions on how to systematically deal with these issues. The Collection of Papers represents a significant contribution to the existing fund of scientific and expert knowledge in the field of criminalistic, security, penal and legal theory and practice. Publication of this Collection contributes to improving of mutual cooperation betw

Conceptual modeling of multimedia databases

The gap between the semantic content of multimedia data and its underlying physical representation is one of the main problems in the modern multimedia research in general, and, in particular, in the field of multimedia database modeling. We believe that one of the principal reasons of this problem is the attempt to conceptually represent multimedia data in a way, which is similar to its low-level representation by applications dealing with encoding standards, feature-based multimedia analysis, etc. In our opinion, such conceptual representation of multimedia contributes to the semantic gap by separating the representation of multimedia information from the representation of the universe of discourse of an application, to which the multimedia information pertains. In this research work we address the problem of conceptual modeling of multimedia data in a way to deal with the above-mentioned limitations. First, we introduce two different paradigms of conceptual understanding of the essence of multimedia data, namely: multimedia as data and multimedia as metadata. The multimedia as data paradigm, which views multimedia data as the subject of modeling in its own right, is inherent to so-called multimedia-centric applications, where multimedia information itself represents the main part of the universe of discourse. The examples of such kind of applications are digital photo collections or digital movie archives. On the other hand, the multimedia as metadata paradigm, which is inherent to so-called multimedia-enhanced applications, views multimedia data as just another (optional) source of information about whatever universe of discourse that the application pertains to. An example of a multimedia-enhanced application is a human-resource database augmented with employee photos. Here the universe of discourse is the totality of company employees, while their photos simply represent an additional (possibly optional) kind of information describing the universe of discourse. The multimedia conceptual modeling approach that we present in this work allows addressing multimedia-centric applications, as well as, in particular, multimedia-enhanced applications. The model that we propose builds upon MADS (Modeling Application Data with Spatio-temporal features), which is a rich conceptual model defined in our laboratory, and which is, in particular, characterized by structural completeness, spatio-temporal modeling capabilities, and multirepresentation support. The proposed multimedia model is provided in the form of a new modeling dimension of MADS, whose orthogonality principle allows to integrate the new multimedia modeling dimension with already existing modeling features of MADS. The following multimedia modeling constructs are provided: multimedia datatypes, simple and complex representational constraints (relationships), a multimedia partitioning mechanism, and multimedia multirepresentation features. Following the description of our conceptual multimedia modeling approach based on MADS, we present the peculiarities of logical multimedia modeling and of conceptual-to-logical inter-layer transformations. We provide a set of mapping guidelines intended to help the schema designer in coming up with rich logical multimedia document representations of the application domain, which conform with the conceptual multimedia schema. The practical interest of our research is illustrated by a mock-up application, which has been developed to support the theoretical ideas described in this work. In particular, we show how the abstract conceptual set-based representations of multimedia data elements, as well as simple and complex multimedia representational relationships can be implemented using Oracle DBMS

Exploring the Utility of Patient Stories on Social Media for Healthcare Quality Improvement

This thesis explores the phenomenon of patient stories on social media. This phenomenon represents the intersection of two phenomena: patient experience and social media. Healthcare experience refers to the interactions of a patient with the healthcare system members, including the nurses, physicians, and staff, and the resultant emotional and behavioural effects of these interactions on patients, including patient satisfaction, patient commitment to health, and patient adherence to treatment plans. Social media refers to the internet-based applications that enable people to communicate, interact, publish, and exchange all types and formats of information, including text, pictures, audio, and video. Patient stories on social media refer to patients' posts that describe their healthcare experiences. This thesis aims to assess the utility of patient stories on social media for healthcare quality improvement and explore the health system and policy factors that can positively or negatively affect this utility in the healthcare system in Ontario. The thesis is comprised of an introduction chapter, a theoretical perspective chapter, four studies presented in chapters 3 to 6, and a conclusion chapter. Additional material is provided in several appendixes, including a definitions section in Appendix 1.A. The first study seeks to understand the perspectives of healthcare providers and administrators in Ontario regarding the factors affecting the patient experience. Qualitative data were collected between April 2018 and May 2019 by interviewing 21 healthcare providers and administrators in Ontario. Interviewees included physicians, nurses, optometrists, dietitians, quality managers, and policymakers. The study findings show that there are two perspectives on patient experience: the biomedical perspective, which prioritizes health outcomes and gives high weights to healthcare experience factors that can be controlled by healthcare providers, while ignoring other factors, and the sociopolitical perspective, which recognizes the impacts of healthcare politics and the social context of health on patient experience in Ontario. The second study explores the perspectives of healthcare providers and administrators on patient stories on social media and whether they can be used for evaluating healthcare experiences. Data were collected between April 2018 and May 2019 by interviewing the 21 healthcare providers, and administrators in Ontario noted in study one. Study findings show that several barriers prevent healthcare providers from realizing the benefits of social media, including the professional healthcare standards and codes of conduct, the time and effort required to process these stories, and the significant number of stories on social media, which also increase the time needed to process these stories. The third study analyzes the social media policies of the healthcare regulatory authorities, which are the regulating and licensing bodies in Ontario, and explores how these policies encourage or discourage the use of social media by healthcare providers. The study uses document analysis and qualitative content analysis to analyze social media policies and guidelines of some healthcare colleges in Ontario issued between 2013 and 2019. The study findings show that in the healthcare system in Ontario, social media is perceived as a source of risks to the healthcare professions and professionals, and therefore, policies are developed to mitigate those risks. Healthcare regulators emphasize that the codes of conduct and the professional standards of healthcare extend to social media, despite the distinct context of social media. The study found no systematic recognition of patient stories on social media as a source of information that requires the attention of healthcare professionals. The fourth study analyzes patient stories on the Care Opinion platform, which is an online platform that enables patients to post stories about their healthcare experiences and enables the providers to respond to these stories. The study explores the elements of healthcare experience in these stories, the characteristics of the stories that receive responses from healthcare providers, and the association between the satisfaction level of the patient expressed in these stories and the likelihood of receiving a provider response. The study collected 367,573 patient stories from the Care Opinion platform that were posted between September 2005 to September 2019. The study uses topic modelling (Latent Dirichlet Allocation), sentiment analysis, and logistic regression to analyze the data. Data analysis identified 16 topics in these stories. These topics can be grouped into five categories: communication, quality of clinical services, quality of services, human aspects of healthcare experiences, and patient satisfaction. Stories that describe healthcare experience of a family member, or reflect patient thankfulness, gratitude, or satisfaction with communication are associated with a high likelihood of receiving a provider’s response; however; the sentiment score of a story, which I used as a proxy for patient satisfaction, was insignificant. The thesis concludes by identifying several barriers that impede the use of patient stories on social media for quality improvement. These barriers are the beliefs and priorities of healthcare providers, the social media policies of the healthcare regulatory authorities and professional healthcare standards and codes of conduct that restrict patient-provider communication, the time and effort required to process patient stories, and the credibility of patient stories

More than meets the eye: the conceptual essence of intrinsic memorability

In a world where sensory threads weave an endless tapestry of multi-modal data, the human brain stands as the masterful weaver of meaning. As we wade through this tempest of input, our brain spins these threads into an intelligible internal representation and holds on tight to what it deems important. But what, exactly, makes certain threads more important than others? And how can we predict their significance? Memorability is the tensile strength of the threads that tie us to the world. It is a proxy for human importance, indicating which threads the human brain will curate and retain with exceptional fidelity. This research investigates these multisensory threads by exploring the influence of audio, visual, and textual modalities on predicting video memorability, and how the interplay between them can influence the overall memorability of a given piece of content. The findings suggest that, while visual data may dominate our sensory experience, it is the underlying conceptual essence that truly holds the key to memorability. This thesis leverages state-of-the-art image synthesis techniques to distill and examine this essence, creating surrogate dreams of video scenes to facilitate the disentanglement of conceptual and perceptual elements of memorability. The work also leverages human EEG data to explore the possibility of a moment of memorability—a moment of encoding that corresponds to a remembering moment—which we expect to exist due to the temporal nature of the world and the natural encoding limits of our brains. The previously murky relationship between the two core means of remembrance---recognition and recall---are reconciled by conducting a novel video memorability drawing task. The research sheds new light on the nature of multi-modal memorability, providing a deeper understanding of how our brain processes and retains information in a complex sensory world. By uncovering the conceptual essence that lies at the heart of memorability, it opens up new avenues for predicting and curating more meaningful media content, and ultimately deepen our connection to the world around us

Language and compiler support for stream programs

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 153-166).Stream programs represent an important class of high-performance computations. Defined by their regular processing of sequences of data, stream programs appear most commonly in the context of audio, video, and digital signal processing, though also in networking, encryption, and other areas. Stream programs can be naturally represented as a graph of independent actors that communicate explicitly over data channels. In this work we focus on programs where the input and output rates of actors are known at compile time, enabling aggressive transformations by the compiler; this model is known as synchronous dataflow. We develop a new programming language, StreamIt, that empowers both programmers and compiler writers to leverage the unique properties of the streaming domain. StreamIt offers several new abstractions, including hierarchical single-input single-output streams, composable primitives for data reordering, and a mechanism called teleport messaging that enables precise event handling in a distributed environment. We demonstrate the feasibility of developing applications in StreamIt via a detailed characterization of our 34,000-line benchmark suite, which spans from MPEG-2 encoding/decoding to GMTI radar processing. We also present a novel dynamic analysis for migrating legacy C programs into a streaming representation. The central premise of stream programming is that it enables the compiler to perform powerful optimizations. We support this premise by presenting a suite of new transformations. We describe the first translation of stream programs into the compressed domain, enabling programs written for uncompressed data formats to automatically operate directly on compressed data formats (based on LZ77). This technique offers a median speedup of 15x on common video editing operations.(cont.) We also review other optimizations developed in the StreamIt group, including automatic parallelization (offering an 11x mean speedup on the 16-core Raw machine), optimization of linear computations (offering a 5.5x average speedup on a Pentium 4), and cache-aware scheduling (offering a 3.5x mean speedup on a StrongARM 1100). While these transformations are beyond the reach of compilers for traditional languages such as C, they become tractable given the abundant parallelism and regular communication patterns exposed by the stream programming model.by William Thies.Ph.D

Higher level techniques for the artistic rendering of images and video

EThOS - Electronic Theses Online ServiceGBUnited Kingdo