Multi modal multi-semantic image retrieval

PhDThe rapid growth in the volume of visual information, e.g. image, and video can overwhelm users’ ability to find and access the specific visual information of interest to them. In recent years, ontology knowledge-based (KB) image information retrieval techniques have been adopted into in order to attempt to extract knowledge from these images, enhancing the retrieval performance. A KB framework is presented to promote semi-automatic annotation and semantic image retrieval using multimodal cues (visual features and text captions). In addition, a hierarchical structure for the KB allows metadata to be shared that supports multi-semantics (polysemy) for concepts. The framework builds up an effective knowledge base pertaining to a domain specific image collection, e.g. sports, and is able to disambiguate and assign high level semantics to ‘unannotated’ images. Local feature analysis of visual content, namely using Scale Invariant Feature Transform (SIFT) descriptors, have been deployed in the ‘Bag of Visual Words’ model (BVW) as an effective method to represent visual content information and to enhance its classification and retrieval. Local features are more useful than global features, e.g. colour, shape or texture, as they are invariant to image scale, orientation and camera angle. An innovative approach is proposed for the representation, annotation and retrieval of visual content using a hybrid technique based upon the use of an unstructured visual word and upon a (structured) hierarchical ontology KB model. The structural model facilitates the disambiguation of unstructured visual words and a more effective classification of visual content, compared to a vector space model, through exploiting local conceptual structures and their relationships. The key contributions of this framework in using local features for image representation include: first, a method to generate visual words using the semantic local adaptive clustering (SLAC) algorithm which takes term weight and spatial locations of keypoints into account. Consequently, the semantic information is preserved. Second a technique is used to detect the domain specific ‘non-informative visual words’ which are ineffective at representing the content of visual data and degrade its categorisation ability. Third, a method to combine an ontology model with xi a visual word model to resolve synonym (visual heterogeneity) and polysemy problems, is proposed. The experimental results show that this approach can discover semantically meaningful visual content descriptions and recognise specific events, e.g., sports events, depicted in images efficiently. Since discovering the semantics of an image is an extremely challenging problem, one promising approach to enhance visual content interpretation is to use any associated textual information that accompanies an image, as a cue to predict the meaning of an image, by transforming this textual information into a structured annotation for an image e.g. using XML, RDF, OWL or MPEG-7. Although, text and image are distinct types of information representation and modality, there are some strong, invariant, implicit, connections between images and any accompanying text information. Semantic analysis of image captions can be used by image retrieval systems to retrieve selected images more precisely. To do this, a Natural Language Processing (NLP) is exploited firstly in order to extract concepts from image captions. Next, an ontology-based knowledge model is deployed in order to resolve natural language ambiguities. To deal with the accompanying text information, two methods to extract knowledge from textual information have been proposed. First, metadata can be extracted automatically from text captions and restructured with respect to a semantic model. Second, the use of LSI in relation to a domain-specific ontology-based knowledge model enables the combined framework to tolerate ambiguities and variations (incompleteness) of metadata. The use of the ontology-based knowledge model allows the system to find indirectly relevant concepts in image captions and thus leverage these to represent the semantics of images at a higher level. Experimental results show that the proposed framework significantly enhances image retrieval and leads to narrowing of the semantic gap between lower level machinederived and higher level human-understandable conceptualisation

A Semantic Web Annotation Tool for a Web-Based Audio Sequencer

Music and sound have a rich semantic structure which is so clear to the composer and the listener, but that remains mostly hidden to computing machinery. Nevertheless, in recent years, the introduction of software tools for music production have enabled new opportunities for migrating this knowledge from humans to machines. A new generation of these tools may exploit sound samples and semantic information coupling for the creation not only of a musical, but also of a "semantic" composition. In this paper we describe an ontology driven content annotation framework for a web-based audio editing tool. In a supervised approach, during the editing process, the graphical web interface allows the user to annotate any part of the composition with concepts from publicly available ontologies. As a test case, we developed a collaborative web-based audio sequencer that provides users with the functionality to remix the audio samples from the Freesound website and subsequently annotate them. The annotation tool can load any ontology and thus gives users the opportunity to augment the work with annotations on the structure of the composition, the musical materials, and the creator's reasoning and intentions. We believe this approach will provide several novel ways to make not only the final audio product, but also the creative process, first class citizens of the Semantic We

Applications of the ACGT Master Ontology on Cancer

In this paper we present applications of the ACGT Master Ontology (MO) which is a new terminology resource for a transnational network providing data exchange in oncology, emphasizing the integration of both clinical and molecular data. The development of a new ontology was necessary due to problems with existing biomedical ontologies in oncology. The ACGT MO is a test case for the application of best practices in ontology development. This paper provides an overview of the application of the ontology within the ACGT project thus far

Standpoint Logic: A Logic for Handling Semantic Variability, with Applications to Forestry Information

It is widely accepted that most natural language expressions do not have precise universally agreed definitions that fix their meanings. Except in the case of certain technical terminology, humans use terms in a variety of ways that are adapted to different contexts and perspectives. Hence, even when conversation participants share the same vocabulary and agree on fundamental taxonomic relationships (such as subsumption and mutual exclusivity), their view on the specific meaning of terms may differ significantly. Moreover, even individuals themselves may not hold permanent points of view, but rather adopt different semantics depending on the particular features of the situation and what they wish to communicate. In this thesis, we analyse logical and representational aspects of the semantic variability of natural language terms. In particular, we aim to provide a formal language adequate for reasoning in settings where different agents may adopt particular standpoints or perspectives, thereby narrowing the semantic variability of the vague language predicates in different ways. For that purpose, we present standpoint logic, a framework for interpreting languages in the presence of semantic variability. We build on supervaluationist accounts of vagueness, which explain linguistic indeterminacy in terms of a collection of possible interpretations of the terms of the language (precisifications). This is extended by adding the notion of standpoint, which intuitively corresponds to a particular point of view on how to interpret vague terminology, and may be taken by a person or institution in a relevant context. A standpoint is modelled by sets of precisifications compatible with that point of view and does not need to be fully precise. In this way, standpoint logic allows one to articulate finely grained and structured stipulations of the varieties of interpretation that can be given to a vague concept or a set of related concepts and also provides means to express relationships between different systems of interpretation. After the specification of precisifications and standpoints and the consideration of the relevant notions of truth and validity, a multi-modal logic language for describing standpoints is presented. The language includes a modal operator for each standpoint, such that \standb{s}\phi means that a proposition $\phi$ is unequivocally true according to the standpoint $s$ --- i.e.\ $\phi$ is true at all precisifications compatible with $s$. We provide the logic with a Kripke semantics and examine the characteristics of its intended models. Furthermore, we prove the soundness, completeness and decidability of standpoint logic with an underlying propositional language, and show that the satisfiability problem is NP-complete. We subsequently illustrate how this language can be used to represent logical properties and connections between alternative partial models of a domain and different accounts of the semantics of terms. As proof of concept, we explore the application of our formal framework to the domain of forestry, and in particular, we focus on the semantic variability of `forest'. In this scenario, the problematic arising of the assignation of different meanings has been repeatedly reported in the literature, and it is especially relevant in the context of the unprecedented scale of publicly available geographic data, where information and databases, even when ostensibly linked to ontologies, may present substantial semantic variation, which obstructs interoperability and confounds knowledge exchange

A COMPARATIVE STUDY ON ONTOLOGY GENERATION AND TEXT CLUSTERING USING VSM, LSI, AND DOCUMENT ONTOLOGY MODELS

Although using ontologies to assist information retrieval and text document processing has recently attracted more and more attention, existing ontology-based approaches have not shown advantages over the traditional keywords-based Latent Semantic Indexing (LSI) method. This paper proposes an algorithm to extract a concept forest (CF) from a document with the assistance of a natural language ontology, the WordNet lexical database. Using concept forests to represent the semantics of text documents, the semantic similarities of these documents are then measured as the commonalities of their concept forests. Performance studies of text document clustering based on different document similarity measurement methods show that the CF-based similarity measurement is an effective alternative to the existing keywords-based methods. Especially, this CF-based approach has obvious advantages over the existing keywords-based methods, including LSI, in dealing with text abstract databases, such as MEDLINE, or in P2P environments where it is impractical to collect the entire document corpus for analysis

Sharing Semantic Resources

The Semantic Web is an extension of the current Web in which information, so far created for human consumption, becomes machine readable, “enabling computers and people to work in cooperation”. To turn into reality this vision several challenges are still open among which the most important is to share meaning formally represented with ontologies or more generally with semantic resources. This Semantic Web long-term goal has many convergences with the activities in the field of Human Language Technology and in particular in the development of Natural Language Processing applications where there is a great need of multilingual lexical resources. For instance, one of the most important lexical resources, WordNet, is also commonly regarded and used as an ontology. Nowadays, another important phenomenon is represented by the explosion of social collaboration, and Wikipedia, the largest encyclopedia in the world, is object of research as an up to date omni comprehensive semantic resource. The main topic of this thesis is the management and exploitation of semantic resources in a collaborative way, trying to use the already available resources as Wikipedia and Wordnet. This work presents a general environment able to turn into reality the vision of shared and distributed semantic resources and describes a distributed three-layer architecture to enable a rapid prototyping of cooperative applications for developing semantic resources

A Call for Executable Linguistics Research

PACLIC / The University of the Philippines Visayas Cebu College Cebu City, Philippines / November 20-22, 200