Bridging the Semantic Gap in Multimedia Information Retrieval: Top-down and Bottom-up approaches

Semantic representation of multimedia information is vital for enabling the kind of multimedia search capabilities that professional searchers require. Manual annotation is often not possible because of the shear scale of the multimedia information that needs indexing. This paper explores the ways in which we are using both top-down, ontologically driven approaches and bottom-up, automatic-annotation approaches to provide retrieval facilities to users. We also discuss many of the current techniques that we are investigating to combine these top-down and bottom-up approaches

Mind the Gap: Another look at the problem of the semantic gap in image retrieval

This paper attempts to review and characterise the problem of the semantic gap in image retrieval and the attempts being made to bridge it. In particular, we draw from our own experience in user queries, automatic annotation and ontological techniques. The first section of the paper describes a characterisation of the semantic gap as a hierarchy between the raw media and full semantic understanding of the media's content. The second section discusses real users' queries with respect to the semantic gap. The final sections of the paper describe our own experience in attempting to bridge the semantic gap. In particular we discuss our work on auto-annotation and semantic-space models of image retrieval in order to bridge the gap from the bottom up, and the use of ontologies, which capture more semantics than keyword object labels alone, as a technique for bridging the gap from the top down

The Museum on the Edge of Forever

This article argues that understanding any space or site relies on a knowledge of its fourth dimension - the timescape. It will explore this by situating the investigation in the museum - a place of heightened contrivance which could easily be shallowly interpreted as "mere style". It will defend a new method of investigating museum temporality which combines both phenomenology and literary theory, and will replace the idea of geo-epistemology with geochronic epistemology: an understanding of context and situation which takes on time as well as spatial location. In so doing, it moves on from notions of the museum as a place out of time, situating it in the networks of meaning, power and politics in which we have lived and are living. Thus, "the whole space of the exhibition" as Lyotard said, "becomes the remains of all time": the Museum on the Edge of Forever

A conceptual framework for studying collective reactions to events in location-based social media

Events are a core concept of spatial information, but location-based social media (LBSM) provide information on reactions to events. Individuals have varied degrees of agency in initiating, reacting to or modifying the course of events, and reactions include observations of occurrence, expressions containing sentiment or emotions, or a call to action. Key characteristics of reactions include referent events and information about who reacted, when, where and how. Collective reactions are composed of multiple individual reactions sharing common referents. They can be characterized according to the following dimensions: spatial, temporal, social, thematic and interlinkage. We present a conceptual framework, which allows characterization and comparison of collective reactions. For a thematically well-defined class of event such as storms, we can explore differences and similarities in collective attribution of meaning across space and time. Other events may have very complex spatio-temporal signatures (e.g. political processes such as Brexit or elections), which can be decomposed into series of individual events (e.g. a temporal window around the result of a vote). The purpose of our framework is to explore ways in which collective reactions to events in LBSM can be described and underpin the development of methods for analysing and understanding collective reactions to events

Analysis of prepositions: near and away from Frames of reference.

XXII Jornades de Foment de la Investigació de la Facultat de Ciències Humanes i Socials (Any 2017)Traditional strategies and procedures to learn a foreign language include the study of rules of grammar and doing exercises such as filling the gaps, repetition of words, drills, memorization of irregular verbs and sentences which may express usual expressions of everyday life. Even if the array of exercises is adequate, polysemy in prepositions causes difficulties in choosing the proper preposition conveying the meaning required by different contexts. Two prepositions of the horizontal axis (near and away from) are taken into consideration in this paper. Approaching the problem from the theory of polysemy and understanding, the use of these prepositions is explored along the dimensions of function, topology – which is the study of physical space–, and force dynamics – introduced in studies such as Navarro (1998)–, as well as the notion of frame of reference (Levinson, 2004). Then, the different senses and uses of these prepositions of the horizontal axis are systematized, explained and examples are used to illustrate the difficulties in learning a language and the doubts which students may have in some situations

Metaphor, Objects, and Commodities

This article is a contribution to a symposium that focuses on the ideas of Margaret Jane Radin as a point of departure, and particularly on her analyses of propertization and commodification. While Radin focuses on the harms associated with commodification of the person, relying on Hegel's idea of alienation, we argue that objectification, and in particular objectification of various features of the digital environment, may have important system benefits. We present an extended critique of Radin's analysis, basing the critique in part on Gadamer's argument that meaning and application are interrelated and that meaning changes with application. Central to this interplay is the speculative form of analysis that seeks to fix meaning, contrasted with metaphorical thought that seeks to undermine some fixed meanings and create new meanings through interpretation. The result is that speculative and metaphorical forms are conjoined in an interactive process through which new adaptations emerge. Taking this critique an additional step, we use examples from contemporary intellectual property law discourse to demonstrate how an interactive approach, grounded in metaphor, can yield important insights

Predicting Visual Spatial Relations in the Maltese Language

Also published in Symposium Melitensia Vol. 15 (2019) p. 115-126In this paper, the automatic detection of spatial prepositions between objects depicted in an image is studied. The ultimate aim is to incorporate the findings in a system that automatically describes images in a natural language. Whereas the explicit prediction of spatial relations in images has been previously studied in English and French, the work reported in this paper addresses relations in Maltese, an understudied language. A dataset consisting of images, spatial prepositions in Maltese, and a number of geometrical and language features is assembled from previous works. A number of predictive models are developed and the results are evaluated in terms of agreement with human-selected prepositions. The relative importance of the features in predicting the various relations is discussed and the paper concludes with a discussion on future work.peer-reviewe

Designing assisted living technologies 'in the wild' : preliminary experiences with cultural probe methodology

Background There is growing interest in assisted living technologies to support independence at home. Such technologies should ideally be designed ‘in the wild’ i.e. taking account of how real people live in real homes and communities. The ATHENE (Assistive Technologies for Healthy Living in Elders: Needs Assessment by Ethnography) project seeks to illuminate the living needs of older people and facilitate the co-production with older people of technologies and services. This paper describes the development of a cultural probe tool produced as part of the ATHENE project and how it was used to support home visit interviews with elders with a range of ethnic and social backgrounds, family circumstances, health conditions and assisted living needs. Method Thirty one people aged 60 to 98 were visited in their homes on three occasions. Following an initial interview, participants were given a set of cultural probe materials, including a digital camera and the ‘Home and Life Scrapbook’ to complete in their own time for one week. Activities within the Home and Life Scrapbook included maps (indicating their relationships to people, places and objects), lists (e.g. likes, dislikes, things they were concerned about, things they were comfortable with), wishes (things they wanted to change or improve), body outline (indicating symptoms or impairments), home plan (room layouts of their homes to indicate spaces and objects used) and a diary. After one week, the researcher and participant reviewed any digital photos taken and the content of the Home and Life Scrapbook as part of the home visit interview. Findings The cultural probe facilitated collection of visual, narrative and material data by older people, and appeared to generate high levels of engagement from some participants. However, others used the probe minimally or not at all for various reasons including limited literacy, physical problems (e.g. holding a pen), lack of time or energy, limited emotional or psychological resources, life events, and acute illness. Discussions between researchers and participants about the materials collected (and sometimes about what had prevented them completing the tasks) helped elicit further information relevant to assisted living technology design. The probe materials were particularly helpful when having conversations with non-English speaking participants through an interpreter. Conclusions Cultural probe methods can help build a rich picture of the lives and experiences of older people to facilitate the co-production of assisted living technologies. But their application may be constrained by the participant’s physical, mental and emotional capacity. They are most effective when used as a tool to facilitate communication and development of a deeper understanding of older people’s needs

Utilising semantic technologies for intelligent indexing and retrieval of digital images

The proliferation of digital media has led to a huge interest in classifying and indexing media objects for generic search and usage. In particular, we are witnessing colossal growth in digital image repositories that are difficult to navigate using free-text search mechanisms, which often return inaccurate matches as they in principle rely on statistical analysis of query keyword recurrence in the image annotation or surrounding text. In this paper we present a semantically-enabled image annotation and retrieval engine that is designed to satisfy the requirements of the commercial image collections market in terms of both accuracy and efficiency of the retrieval process. Our search engine relies on methodically structured ontologies for image annotation, thus allowing for more intelligent reasoning about the image content and subsequently obtaining a more accurate set of results and a richer set of alternatives matchmaking the original query. We also show how our well-analysed and designed domain ontology contributes to the implicit expansion of user queries as well as the exploitation of lexical databases for explicit semantic-based query expansion